Resolved Apple CalDav issues with PostgreSQL startup

Today I noticed that my phone could no longer create any new calendar items. With Server.app I noticed that the Calendar (and AddressBook) services were no longer running and when checking their status, it took forever for the panel to load. Enabling the service again also took forever to not start (and unfortunately without any error message).

After some digging I found that the PostgreSQL server the Apple CalDav service uses internally was no longer running and issues starting In the logfiles in /var/log/caldavd/postgresql/ I found messages like:

2015-03-14 12:59:33.665 CET [689] LOG:  unexpected pageaddr 0/5DC82000 in log segment 000000010000000000000061, offset 13115392
2015-03-14 12:59:33.665 CET [689] LOG:  invalid primary checkpoint record
2015-03-14 12:59:33.679 CET [689] LOG:  unexpected pageaddr 0/5DC7C000 in log segment 000000010000000000000061, offset 13090816
2015-03-14 12:59:33.679 CET [689] LOG:  invalid secondary checkpoint record
2015-03-14 12:59:33.679 CET [689] PANIC:  could not locate a valid checkpoint record

I suspect these were caused by a crash a few days ago of my NAS that serves the iSCSI disks where the postgres data is stored. I spent a lot of time today to look for a solution (including trying to restore a backup and set it up from scratch, which all failed). In the end I found a clue in the manual page of pg_resetxlog:

DESCRIPTION
pg_resetxlog clears the write-ahead log (WAL) and optionally resets some other control information stored in the pg_control file. This function is sometimes needed if these files have become corrupted. It should be used only as a last resort, when the server will not start due to such corruption.

This pretty closely matched my situation so (after making a backup of the DB folder) I executed the following command in the folder where Server.app stores it data (by default that is /Library/Server/Calendar and Contacts but in my case that’s /Volumes/Data/Library/Server/Calendar and Contacts as I store all data on a RAID5 container on my NAS)

sudo -u _calendar pg_resetxlog -f Data/Database.xpg/cluster.pg/

After running this command the PostgreSQL for Services started again and my Calendar (and AddressBook) services were running again. So far it looks like I did not lose any data apart from a calendar entry that I had added on my Macbook in iCal.I am glad it is resolved, but I have to look into how backups are made so that the next time I at least know that I can get my calendar and contacts back…

Crashplan stopped backing up due to corrupt cache

Today I noticed that Crashplan running on my NAS was no longer backing up any files and that the backup set was 0 Mb and only 2 files (which should have been a few 100k files and > 350Gb). Rescanning the fileset didn’t help, neither did removing and adding the folders again.

After a little digging I noticed in the logs entries like (log message was a log longer but I only included the relevant part of it):

com.code42.exception.DebugException: BSM:: SET-1: Exception adding source file...skipping - fileStat=FileStat[/volume1/photo, exists = true, fileType = 1, length = 0, lastModified = 1419895925000, lastAccess = 1425690867000, created = 1419895925000], com.code42.backup.manifest.FileManifest$CorruptFileManifestException: CORRUPT FMF ENTRY FixedPortion[entryPosition = 186768232, fileId = 00000000000000000000000000000000, parentFileId = 00000000000000000000000000000000, fileType = 0, version = Version[timestamp = 0, sourceLastModified = 0, sourceLength = 0, sourceChecksum = null, fileType = 0]

Googling for this message did not render any result unfortunately and this part of the Crashplan system is rather obscure (nothing to debug, messages are limited. The only thing I could think of to try to resolve it was to drop the cache Crashplan maintains (in the cache subdirectory of the Crashplan installation). It turns out that this was sufficient as after a restart the cache was rebuild and the the scan resulted in the expected number of filed.

The steps I performed were:

  1. Stop the Crashplan engine
  2. remove all files in the Crashplan cache/ subdirectory
  3. Start Crashplan
  4. Enforce a rescan of the fileset in [Settings] –> [Backup] –> Verify Selection [Now]

Since I had removed the folders from the backup set I feared that I had to upload all data again to my external backup targets, but Crashplan was smart enough not to need that.

Cyrillic (and other language) support for the Pebble Watch

The stock firmware of the Pebble Watch only seems to support English and other western languages out of the box. As I sometimes receive messages and notifications in Cyrillic, it was annoying that these were not displayed correctly (i.e. all the unknown characters were replaced by a rectangle)

On the Pebble site itself there is nothing mentioned on how this can be enabled, so it looks like it is not supported out of the box. Fortunately the smart people of PebbleBits have found a way around this and offer modified firmware versions with support for additional character sets and also a small number of other patches. They state clearly that their site is not operated by or affiliated with Pebble in any way but it offers a very interesting Firmware Generator, which offers support for a number of language sets:

  • Symbols, Emoji
  • Latin-based: English, Croatian, Czech, Danish, Dutch, Estonian, Finnish, French, German, Hungarian, Icelandic, Italian, Latvian, Lithuanian, Norwegian, Polish, Portuguese, Romanian, Slovak, Spanish, Swedish, Turkish
  • Cyrillic: Belarusian, Kazakh, Macedonian, Russian, Ukrainian
  • Other: Greek, Hebrew, Thai, Vietnamese

For modern firmwares (v2.5+) there seems to be sufficient place to add a number of languages, older firmware versions have some limitations. In addition to adding characters, it is also possible to apply a number of patches to the firmware itself:

  • Disable default watchfaces
  • Disable default main menu entries
  • Display phone number instead of contact name on incoming call
  • Additional quick launch options for apps
  • Change buttons layout of stock Music app (my favorite as it adds the option to control the volume)
  • Translate the interface to a another language

After the selection is made, the custom firmware can be downloaded. This should be done from a smartphone that is connected to the Pebble watch, the site works great on a mobile phone, but also provides a QR code that can be scanned by the phone to download the firmware. The installation is seamlessly done by opening the file with the Pebble app, which guides you through the process.

So far I have noticed that the watch works fine with the patched firmware and supports (in my case) Cyrillic notifications perfectly now.

For reference: here is a  link to the configuration I use.

Got a Pebble watch from Дед Мороз

pebble_watchLooks like Дед Мороз did not want me to completely get rid of Pebble as he brought me a Pebble Smart Watch for New Year!

The Pebble Watch is already on the market for quite some time and well supported by iPhone and Android applications. I will probably need some time to really find out what it can do, the notifications and music control are already nice features on top of being a nice watch. I know that Runkeeper also supports it, so something to check out the next time I go out for a run.

Once caveat I already found is that it does not support all characters in notifications out of the box and Cyrillic does not seem to be supported at all… something to dive into.

Happy New Year!

Happy New Year 2015Happy New Year and best wishes for 2015!

A new fresh year and wish you all the best for 2015, let it be a great year in which things only get better again for all.

As you can see also a new fresh website for my blog. I have moved over to WordPress as it suited my needs better and allowed me to simplify my setup (as I am hosting it myself).  During the course of last year I ran into small problems and issues with my setup of the Pebble blogging software I used. I still think Pebble was a great platform when I started using it a long time ago. There have hardly been any updates to the platform, which did not really bother me from a functional perspective, but did make me wonder whether there are really no security issues with it (or that they were just not found & fixed as it has a very small usage footprint). Besides the anti-spam mechanisms it had turned out not to work so I ended up manually removing spam comments just too often. Last but not least it required too much system resources as it requires a Java Servlet container that I do not need for anything else (anymore).

Therefore I decided last year to migrate my own and a few other blogs I host to a single WordPress 4 Network Instance. Migration was not flawless and I am still in the process of migrating the articles from last year (doing that manually) so expect some more content to be added the coming days/weeks. The design is still quite basic, I will clean that up once I find the time for that.

Let 2015 be a very good and productive year!

Piwigo LDAP module blocks upgrade to 2.7 :-(

Today Piwigo 2.7 became available (for info on what’s new refer to their announcement). While upgrading I noticed that the Ldap Login module I depend on is not supported and according to this announcement on the Piwigo forums probably never will be.

I did try to perform the update and patch the LDAP Login plugin, but had to five that up after spending more than an hour on it as it turned out that something had changed in the way authentication is handled in Piwigo. Since I was running out of time I had to leave that for now as it turned out not to be a simple fix.

For me this is bad news as I am depending on LDAP integration so for now I cannot upgrade. Since the 2.7 version is still very fresh (i.e. only announced today) I will just defer the upgrade to see if there is any movement on this plugin. If not, I will have to look for another solution unfortunately….

Will document any alternative solution here as well once I find it to help others with a similar dependency.

Clean Photo Album permalinks with Piwigo

I am playing for some time now with Piwigo to replace my Menalto Gallery3 online photo gallery. Key reason to look at another solution is that after the move from Gallery 2 to 3 the project (which took ages as it was a major overhaul of the code), the projects seems to have stalled.

So far I really like Piwigo as it has everything I need including iPhoto integration and a (simple) iPhone app. LDAP support is available through a plugin that is basic but suffices for my need. However, one of the key gaps for me was that it did not have any way to generate nice and simple URLs to albums that you can share easily (verbally). Although it was possible to define permalinks for an album, the URL remained ugly in my opinion.
Today I hacked a small patch together for the Piwigo 2.6 codebase that changes the URLs for photo albums to something like:

http://photo.mydomain.tld/albumname

which is exactly the way it worked for me fine (like I had with Gallery3). This only works for albums with a permalink defined, default album URL will retain the /category/<albumid> format, which is fine for my situation.

Steps to obtain more clean album URLs are:

  1. Apply this patch: piwigo-url-patch
  2. Add the following mod_rewrite rewriting rules to your Apache configuration
    RewriteRule ^/category/       /index.php/%{REQUEST_URI}               [L]
    RewriteRule ^/[^.]+$          /index.php/category/%{REQUEST_URI}      [L]

Again, in my setup this worked, I am still testing this so any feedback to improve is welcome. I did notice that occasionally the patch results in a / too many in URLs generated by piwigo, but that is silently ignored and does not affect the functionality.

To actually use the patch, define a permalink under [Administration] –> [Albums] –> [Manage] on the [Permalinks] tab.

Change Gitlab homepage using Apache’s mod_rewrite

For some time I have been looking for a way to share public projects easily using GitLab. With the Public Project option of GitLab this was already possible for some time, but it did not work quite as I would like to (i.e. I would like http://gitlab.mydomain.tld to be the URL for all public projects). Due to the way GitLab is setup, the default URL will redirect the user to the login page, which does provide a link to the Public Projects page, but was not quite what I want.

Of course, as GitLab is open source, I could change the code directly, but as I would have to do that after each upgrade of GitLab (which is monthly!) I did not want to do that. Today I found a way around changing the code by using the following mod_rewrite rules to my Apache configuration (I placed this in the <VirtualHost> configuration but should also work from a .htaccess file):

# Redirect /users/sign_in to /public unless it has a local refferer
# This makes the public projects page the homepage instead of the login page
RewriteCond   %{HTTP_HOST}@@%{HTTP_REFERER}    !^([^@]*)@@https?://1/
RewriteRule   ^/users/sign_in$                 https://%{SERVER_NAME}/public/          [R,L]

This is inspired by a blog post on referer checking from the Apache .htaccess file. To get to this solution I just had to realize that an internal redirect by the application clears the referrer and apply the opposite logic to intervene when this happened (no referrer implies a redirect, when the user clicks on a link the request will have a referrer). How this works is:

  1. The user visits http://gitlab.mydomain.tld/
  2. GitLab redirects this request to its sign_in page
  3. The browser requests the sign_in page, as this was a redirected page the referrer will be empty
  4. The above mod_rewrite rule kicks in and redirects the user to the public projects page

For me this setup works as I expect. The only caveats are that users with browsers setup not to provide a referrer (e.g. for privacy reasons) may no longer be able to login and that a direct link to the sign_in page won’t work (the user will be redirected to the public projects page and has to click the sign_in button). For my setup both are no issue, let me know through the comments if there are other issues or perhaps solutions for this.

Login issues after upgrade to GitLab 6.5

I have been playing around with Gitlab, the open-source self-hosted Github clone for a while now. I plan to use it to publish the scripts and small programs I did over the last few years and will still create later this year.

After the upgrade to Gitlab version 6.5.1 (which was a breeze BTW thanks to their excellent upgrade script) I noticed I could no longer login. to the server. In the logfile log/production I found messages like:

Started POST "/users/sign_in" for 2001:XXX:XXXX:X:XXX:XXXX:XXXX:XXXX at 2014-02-02 13:53:46 +0100
Processing by Devise::SessionsController#create as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"XXXXXXXXXXXXXXXXX", "user"=>{"login"=>"XXXXXXX", "password"=>"[FILTERED]", "remember_me"=>"0"}}
Can't verify CSRF token authenticity
Redirected to https://gitlab.mydomain.tld/
Completed 302 Found in 123ms (ActiveRecord: 7.3ms)
Started GET "/" for 2001:XXX:XXXX:X:XXX:XXXX:XXXX:XXXX at 2014-02-02 13:53:46 +0100
Processing by DashboardController#show as HTML
Completed 401 Unauthorized in 1ms
Started GET "/users/sign_in" for 2001:XXX:XXXX:X:XXX:XXXX:XXXX:XXXX at 2014-02-02 13:53:46 +0100

 
This turned out to be a known issue with the installation of Gitlab. Since Gitlab is only supports NGinX while I am running it on Apache, I needed to dig a bit further for the solution. The problem was caused by a security enhancement in Gitlab 6.5 in combination with HTTPS. Since the SSL processing is handled by Apache, which uses mod_proxy to connect to GitLab only using HTTP, cookies no longer worked properly. The solution was pretty simple, it required the following statement to be added to the Apache Virtual Host configuration:

RequestHeader set X_FORWARDED_PROTO 'https'

Please note that this does require mod_headers to be enabled, if this is not enabled, issue to following two commands to enable it:

sudo a2enmod headers
sudo /etc/init.d/apache2 restart

Factory reset after firmware upgrade of TP-Link SG3424

Since about a year I have a TP-Link SG3424, a manageable 24-ports gigabit switch that so far has proven to be table and reliable and was a good deal when I bought it compared to similar devices. It’s not a Cisco switch, but honestly I do not know what I am missing out on given the feature set is has. So far it covered my needs and has been operational without any noticable issues or hiccups. The only drawback (for me) so far was that it lacked IPv6 support, but that’s optional on an internal network anyway.

Today I noticed that TP-Link released a new firmware version in December for the SG-3424 switch that added IPv6 support and is available from their support site’s download page. I followed the installation guide, which was completely correct, but a bit incomplete as I had a few surprises during my upgrade:

  1. The installation manual instructs you to change your network settings to the 192.168.0.X network. As I have an operational environment using another IP range, I happily ignored this and was able to perform the upgrade. However, after the upgrade, I could no longer reach my switch until I changed the IP address of my laptop to the 192.168.0.X network. This is not an issue for someone that knows what he is doing, but should have been mentioned in the installation manual in my opinion…
  2. After the firmware update, the switch was back at its factory settings and lost all configuration settings (including the logins, so I had to login with admin/admin again). This is bad TP-Link, even if your update process requires a full reset, PUT IT IN THE UPGRADE MANUAL at least…
    Fortunately I have a tendency to make backups before I change things and did create a backup before the update , but it was a surprise one would not expect.
  3. When restoring the configuration, all settings were restored correctly EXCEPT the logins and password of the admin user. Again, not a blocker, but a surprise and something NOT DOCUMENTED.

All in all minor things that one can resolve pretty easily, but not something you would expect from a company that positions the product at the professional market… Cisco-like 10+ upgrade guides are a pain to get through, but a 2-pager with only half of the information doesn’t help either.

Apart from these glitches, the upgrade was worthwile as the switch now has IPv6 support, which functions great and has (as always) all features one can imagine (like being able to add manual addresses next to the auto-negotiation addresses, support for RA and DHCP6, support for IPv6 on all services like SSH and SNMP) appear to work fine.