• Sources: primary, discussion
  • Summary: Michael Bacarella writes that contractor dust set off the smoke alarm in Jane Street's datacenter at 1 New York Plaza. The alarm cut the room's air conditioning and filled the sprinkler pipes, whose heads open on heat. He shut down every device by hand, and the temperature stopped at 143.6 F, short of the sprinklers' 150 F trigger. Afterward he wrote an OCaml monitor that alerts when the average temperature across the server fleet rises. The post drew 375 points and 185 comments on Hacker News.
  • Why it matters: Fire suppression interlocks can turn a false alarm into a cooling failure, and fleet-wide temperature averages catch that faster than per-server alerts.