The first law of the Time Lords

tardisclock

Let me share a war story with you (one that I swear is true).

In the year 2000 an Ansett passenger jet, loaded with passengers, was preparing to push back from the terminal in Sydney.   But there was a timing problem…  the control tower insisted they were trying to depart too early…  three minutes too early.

What resulted was a dispute about the actual time, which concluded with the control tower politely suggesting Ansett ring 1194 (the phone number in Australia to get the talking clock).   One phone call later and Ansett could see the problem:   Their mainframe was 3 minutes fast.   Seriously!

6137824379_759475932e_z

The root cause of this problem was simple:  while Ansett used sysplex timers (Mainframe timing devices) to co-ordinate time between their mainframes,  nothing had been setup to co-ordinate the sysplex timer time with an external source.  And sadly the time had drifted.

IBM Sysplex Timer

IBM Sysplex Timer

The solution was simple.   We attached a modem to the sysplex timer and it began calling the NIST in Boulder Colorado to check for drift (this product had no concept of NTP).  Meanwhile the Sysplex Timer had to very slowly (and it took a while) drift it’s reported time back to reality.

So Ansett had broken the first law of the Time Lords:  Always sync your clocks.   Pretty well every product in your data center can sync with NTP and there is no excuse not to use it.

So given my very strong views on this,  I am curious, have I missed something?   Are there reasons to NOT use NTP?   Have you seen objections?   Please share your thoughts.

And Ansett?  Sadly no longer with us.  The airline failed in September 2001, their planes sold off or broken down for scrap.   A sad day for Australian aviation and all the people who worked for that fine airline.

Ansett-767-N407AN-070405-01-12

Advertisements

About Anthony Vandewerdt

I am an IT Professional who lives and works in Melbourne Australia. This blog is totally my own work. It does not represent the views of any corporation. Constructive and useful comments are very very welcome.
This entry was posted in Brocade, Cisco, IBM and tagged , , . Bookmark the permalink.

5 Responses to The first law of the Time Lords

  1. Sean says:

    Nice pics and a great story :-)
    Reasons not to use NTP? People don’t want to manage “fancy” services, “never done it and never had a problem”, normally it’s “automatic” in the Windows world so when Windows people get to manage Linux/UNIX hosts they tend to lose sight of the need to configure NTP client (and sometimes even when they do it may not work properly with Windows NTP servers), etc.
    Cure: run into some serious trouble – like those “aviators” did – and then you start minding the time real “soon” :-)

  2. In the 90th a banking customer run bis sy splex timer based on my swatch time set during installation w/o modem for years and nobody was aware ….

  3. co·plex·it says:

    I’ve read of only one reason for discouraging widespread usage of NTP for internet-connected devices: there was at least one case of a small-scale router model that was configured to use the same particular NTP server address by default, and legend has it that to this day there are still tens or hundreds of thousands of those routers in service, always regularly querying the same address. An unintended DDoS of sorts, due to the badly thought-out default NTP settings and handling. (I could probably dig up something on the actual story behind that but am too lazy now.)

  4. Pingback: The first law of the Time Lords | Aussie Storage Blog | Storage & Beyond

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s