Leap Second' and Violent Storms: How the Internet Died Twice This Past Weekend

You might or might not have noticed but the Internet died this weekend -- twice for some websites. Adding one tiny second -- a leap second -- to the international timekeepers' clock caused a bit of a blip on many websites. It took several down for nearly an hour, which can be an eternity in the digital world.

As Wired puts it, some sites were just not "able to cope" with the additional second:

Many computing systems use what’s called the Network Time Protocol, or NTP, to keep themselves in sync with the world’s atomic clocks, and when an extra second is added, some just don’t know how to handle it.

The “leap second bug” hit just as the web was recovering from a major outage to Amazon Web Services, an online operation that runs as much as one percent of the internet. Some operations, including Google, saw the leap second coming and prepared for it, but others weren’t so diligent.

Sites that were notably affected included Mozilla, Reddit and Gawker.

The second was tacked onto the clock at midnight universal time Saturday, June 30, going into July 1. That's 8 p.m. EDT Saturday. Universal time will be 11:59:59 and then the unusual reading of 11:59:60 before it hits midnight.

A combination of factors, including Earth slowing down a bit from the tidal pull of the moon, and an atomic clock that's a hair too fast, means that periodically timekeepers have to synchronize the official atomic clocks, said Daniel Gambis, head of the Earth Orientation Service in Paris that coordinates leap seconds.

The time it takes the Earth to rotate on its axis - the definition of a day - is now about two milliseconds longer than it was 100 years ago, said Geoff Chester, spokesman at the U.S. Naval Observatory, keeper of the official U.S. atomic clocks. That's each day, so it adds up to nearly three-quarters of a second a year.

Timekeepers add that leap second every now and then to keep the sun at its highest at noon, at least during standard time. This is the first leap second since January 2009 and the 25th overall. Gambis said the next one probably won't be needed until 2015 or 2016.

Although it wasn't necessarily expected that there would be a noticeable change on computers, it is clear some sites had technical issues. Gizmodo has more on how the crash was handled by some sites:

The end result was pretty ugly. Sites were down for hours, as administrators worked to clean up the havoc wreaked by one little stray second. Gizmodo crashed around 8:00pm EDT, was down for roughly 45 minutes before regaining functionality. But not everyone was affected.

Earlier this year, official timekeepers from across the world discussed whether to eliminate the practice of adding leap seconds. They decided they needed more time to think about the issue and will next debate the issue in 2015.

Google was one of the sites, Wired pointed out, that was prepared for the addition of leap seconds. In a blog post last year, the company explained that it had developed a method to handle these changes:

The solution to this challenge drove a lot of thinking to develop better ways to implement locking and consistency, and synchronizing units of work between servers across the world. It also meant we thought more about the precision of our time systems, which have a knock-on effect on our ability to minimize resource wastage and run greener data centers by reducing the amount of time we must spend waiting for responses and rarely doing excess work.

By anticipating potential problems and developing solutions like these, the Site Reliability Engineering group informs and inspires the development of new technology for distributed systems—the systems that you use every day in Google’s products.

In addition to the leap second, several sites also experienced weather-related outages. Netflix, Instagram and Pinterest's servers went down due to severe storms in the Washington, D.C., area late Friday night with functionality not being restored until well into Saturday.

(Related: Violent storms leave millions without power during massive heat wave)

Netflix, Pinterest and Instagram are customers of Amazon Inc.'s web services division. The unit provides web services and data storage facilities that are commonly used for "cloud computing."

So, if you were wondering what was up with the Internet this weekend, there you have it.

The Associated Press contributed to this report. 

Featured image for this post via Shutterstock.