(This is part of an open ended series of posts where I write down random things I feel are sharable from the years of mailing lists I’ve not caught up on…)

How useful is a subscription to a SRE mailing list when you have 284 messages dating back to November of 2019? That’s an excellent question. But that’s what the SRE Weekly folder has. So obviously this will be a multi-part one as well.

  • The silence of the racks is deafening, production gear has gone dark – so which wire do we cut? – this story reminds me of when we had an air conditioner fail in the machine room, which lead to the glue holding the clamp (that held the fire extinguisher to the wall) to come unstuck dropping the canister on the EPO switch with enough force that it triggered it. So there was a BANG! and then all our connectiving to our test machines went away.
  • Teaching “The Smell” – Part of maturing in this industry is developing this skill. The actual trick is to learn how to identify the smells from both the technical and business persepective. Spoiler; an app is never 100% healthy. But its also only rarely out.
  • How to Avoid Cascading Failures in Distributed Systems – Some anti-patterns to check against your systems. Because everything is a distributed system these days.
  • Dispatch – sometimes it is ‘fun’ to go through an archive like this. I clicked the annoucement for Netflix’s incident management system being open sourced … and the repo has officially been archived as of 3 months ago. An entire open source project was born, matured and died in how long I was ignoring my mail.
  • Building a more accurate time service at Facebook scale – I love things that happen at this sort of scale. Amazon builds their own electrical hardware to run their data centers for example…
  • Google’s Technical Writing Courses - ‘Every engineer is also a writer’
  • Why you can’t just ask “why” – this is interesting, but what’s really interesting is the field of Naturalistic decision-making. That is a very cool rabbit hole to have discovered.
  • What happens when you update your DNS? – Remember, its always a Security Group. And if it isn’t then it absolutely will be DNS. But do you know how it works?
  • Change Advisory Boards Don’t Work – Posting without finger pointing. But I assure you, I could. And yes, I do realize it depends on what you are optimizing for, and yes the authors have skin in the game for more deploys. I just happen to agree with them.
  • That Moment – New goal; don’t be on call when firefighters have to respond.
  • On Call Shouldn’t Suck: A Guide For Managers – How is this the first Honeycomb / Charity Majors article I’ve linked to in this.
  • ‘your microservice-based architecture is secretly a monolith in disguise’ – I just really like this turn of phrase. And, yup. Long live the intentional monolith!
  • Here are 4 Ways SRE Helps New Employees Onboard – Onboarding is hard. Like really hard. I might be biased in this thinking as we had an onboarding product in our suite when I was at HP so got the sales pitches, but they make sense and nothing I’ve seen since has proven they weren’t onto something.
  • Sharp tools for emergencies and the –clowntown flag – I love this on so many levels. Reminds me of when support once asked a customer for the ‘validator spew’ during an investigation. Yeaaah. I’ve named a lot of things, but this wasn’t my best one. (Though might be my earliest.) ‘Adam’ is its own dialect.
  • Type in the exact number of machines to proceed – same person. Feels like a companion to the ‘smell’ notion way above. It is like a smell trigger. Also, do I need an RSS reader? Probably. Do I need another thing to feel guilty for not reading? No, I do not.