Brainiac Corner with Ryn Daniels

Ryn Daniels is an operations engineer at Etsy who got her start in programming with TI-80 calculators back when Geocities was still cool, and these days has opinions on things like ‘the cloud’ and ‘devops’. She lives in Brooklyn with a perfectly reasonable number of cats, and in her spare time can often be found homebrewing, rock-climbing, or hand crafting knitted server koozies for the data center. How did you get from stork to brainiac (i.e. what do you do today and how did you get there)? I started out playing around with computers and trying to figure out how they worked from a young age, then teaching myself html/javascript/css and the basic LAMP stack in junior high so I could create websites for my terrible angsty goth poetry or Neopets or whatever it was kids were into back then. Because both my parents worked in tech, I originally studied creative writing and art in college as my way of rebelling, but eventually my love/hate relationship with building my own gaming PCs somehow turned into a CS degree. After a few years working in workstations R&D in the middle of corporate America, I got it into my head that I was going to move to New York and work for a startup, for reasons that I’m sure made sense at the time. That risk ended up paying off eventually though - employee turnover at the first startup I worked for led to me accidentally becoming their sysadmin by virtue of being the only person who had spent more than an hour using linux before. As it turned out, I really enjoy operations, and have been honing those skills ever since. These days I’m an operations engineer at Etsy, working on monitoring, configuration management, and automating many of the things. I’m also co-authoring Effective DevOps with Jennifer Davis, due to be published by O’Reilly in February 2016. What is in your group’s technology stack? The site itself is a fairly typical LAMP stack (the MP being mysql and php). On the operations side of things, we have Chef, Nagios, Ganglia, Statsd/Graphite, and ELK, as well as a plethora of lovingly handcrafted open source tools - of special note being knife-spork for Chef, opsweekly for on-call tracking, and nagios-herald for adding context to alerts. Who would win in a fight between ninjas and pirates? Why? Which side has the fewest yaks to shave? Ninjas can be stealthy and maybe sneak away from the yaks, but pirates could escape the yaks by sailing away in their pirate ships. Unless yaks can swim. Can yaks swim? These are the questions that keep me up at night. Which is a more accurate state of the world, #monitoringsucks or #monitoringlove? Our world is all about the #monitoringlove. Remember, Laurie’s Law states: “If it moves, graph it. Even it it doesn’t move, graph it anyway because it might someday.” Monitoring is something we take very seriously at Etsy, as well as alerting and how alert fatigue can impact the well-being of our engineers. Just a couple weeks ago I helped organize a monitoring hack week because over time, our monitoring had grown to be messier than we would prefer. At the end of the week we had significantly less alert noise, better tooling and visibility around monitoring newly-built hosts, on-call sleep retrospectives, and a dashboard to show the on-call person what is most likely to wake them up that night, among other things. It wasn’t just ops participating either - people from other engineering teams came with their own monitoring questions and projects they were working on. We’re all about #monitoringlove - and making sure we’re well-rested! In six words or less, what are the biggest challenges your organization faces? “Scaling automation and culture. Also yaks.” A challenge with any growing organization or team is how to keep the best parts of the culture they have built while continuing to strengthen and diversify. Automation, as wonderful as it can be, abstracts away and hides often critical pieces of software and infrastructure; it is often more difficult to troubleshoot when it goes wrong than it would be without automation, and this tends to only get worse the more automation you have. Growing organizations find themselves with increasing complexity on both the human and technical sides of things, and we are not immune to these challenges. Also, there’s yaks. There’s always yaks. What’s the best piece of advice you’ve ever received? Don’t be afraid to celebrate your successes - especially in delicious ways. To me, this has meant both learning how to trust in my own skills and knowledge as well as knowing when to say yes (and no) to things - it’s been a powerful combination. I dealt with a fairly nasty bout of impostor syndrome for a few years, so being able to even recognize, let alone celebrate, successes has been very important to me. And learning what things I should be saying ‘no’ to allows me to say ‘yes’ more wholeheartedly to other things, giving those things more of my attention and care and being more successful with them than if I was only putting in partial effort. What principles guide your expertise in your given domain?

• Never stop learning.
• Blameless post-mortems are what it’s all about.
• Boring technology is underrated. It doesn’t matter how many new features your shiny new framework has if it’s always taking your site down.
• Automate as much as is reasonable, graph everything, and carry a large pair of yak shears.
• Do not taunt the on-call gods.
• Not every problem is worth the time and energy it would take to solve it - you have to know which yaks to shave and which to let go.
• Have you tried turning it off and on again?

What is your vision for system administration in 5 years? More automation, but a more mature role for automation. Knowing what and how to automate will give human operators the ability to focus on things that actually require human skills and attention. There will be less 'traditional' system administration and more reliability and operations engineering as our industry and its requirements continue to evolve. It won’t be enough to be a “rockstar” if you’re also an asshole - people who can communicate well and who practice empathy for their peers and the community as a whole will be the most valuable people to hire. Also, drones will be providing 24/7 kitten delivery services and we’ll all be taking flying cars to our data centers on the moon.