Leaping seconds and looping servers

At around 2012-06-27T09:30:00+1000 several servers had their time go backwards by precisely 10 hours. Ie the time zone difference. These servers were all running a fully patched Debian stable. There are distributed around the country, all running ntpd with 4 debian.pool.ntp.org upstream servers, presumably different servers in each case. Most of them are little more than an internet gateway and so were running very little in the way of non-Debian software. But two of them are application servers.

That day I discovered our how well our in-house software copes with the day being set to yesterday. Turns out it causes lots of transient problems as it whinges about the entered dates being wrong. The most serious thing was a SIP E1 gateway running an embedded linux stopped receiving calls. That took out the companies entire phone system, but a reboot fixed that.

I am pretty sure it was ntpd that was at fault, as "ntpq -c peers", showed then being out by offset by precisely 10 hours from their upstream servers.

Setting the date manually and restarting ntpd fixed the problem, but at the time I was at a total loss at to what the cause might have been. Then I read about the leap second, and then found out that ntpd can start adjusting for it days before hand. Anyway, it made for in interesting departure from the normal daily routine.