NIST was 5 μs off UTC after last week's power cut | Jeff Geerling

If you were 5 microseconds late today, blame it on NIST. Their facility in Boulder Colorado just had its power cut for multiple days. After a backup generator failed, their main ensemble clock lost track of UTC, or Universal Time Coordinated. But even if you used the NTP timing servers they run , they were never off by more than 5 microseconds. 5 μs might seem insignificant. But it is significant for scientists and universities who rely on NIST's more specialized timing signals . But no, you don't need to panic. And yes, they have it under control now. But I thought I'd go over what happened, what it means, and what we can learn from NIST's near-outage. Video This blog post is a lightly-edited transcript of my most recent YouTube video: What happened The NIST campus, which distributes Internet time on six of the most popular NTP servers, lost power last Wednesday . The power company was forced to cut power because of wind gusts over 100 mph (160 km/h). Power lines were coming down and they didn't want to risk starting a wildfire. The whole campus was locked down for safety, so nobody could enter or exit. They have backup generators. And those were working... but apparently one of the generators failed after a couple days. Specifically, the generator that powered the main ensemble clock that's used by the NTP servers. Things were dicey last Friday, and they couldn't get any more staff in to fix it. It got to the point Jeff Sherman, the Group Leader for NIST's Time Realization and Distribution Group, considered shutting down the backup generator that powered the time servers. That would've prevented them from sending out inaccurate time, which would be worse than no time at all for a lot of applications. NTP's designed so you have multiple servers you look at, and if one fails, it won't cause you to lose time. And luckily for NIST, they have another building in their Boulder campus with more clocks, and that building could transfer time back to the one that had the power failure, if they needed to. But yesterday Jeff posted another update : power was restored, and apparently there were still some staff on-site who saved the clocks. They were able to re-route emergency power after the main backup generator went down. Battery backups, I'm assuming some big UPSes, were able to bridge the gap, until they got the backup backup power going. When all was said and done, their monitoring showed deviation from UTC was less than 5 μs. Seeing all that, Jeff and the team at NIST decided to keep their time servers online. But why would they do that, if they were off? Well, time scales are important here. If you're on a Mac like I am, go in the Terminal and run sntp time-a-b.nist.gov . This or a command like ntpdate on Linux gives back an error bound, that shows latency between your computer and the NTP...

Preview: ~500 words