Aaron and I in our Microsoft MVP regalia at the 2013 Microsoft MVP Summit
Kevin (KK): We understand you’re in a new position at Stack Overflow. Stack Overflow is renowned among the IT crowd, so I certainly understand being drawn there. But could you tell us that story in your own words? Aaron (AB): I’ve been a user of Stack Overflow for a dozen years, and I’ve always thought it was the best platform for Q&A. A lot of people think of Stack Overflow as a programming website, but there’s much more to it. They have the best model I’ve come across for encouraging and motivating people to write high-quality answers and help each other solve problems. It doesn’t matter if it’s programming or grammar or electrical or baking—just about any problem you can have in real life, there’s probably a Stack Overflow site for it. When the opportunity presented itself, I felt like I could really help improve things and keep their services stable and performing well. I want to help empower all our users by using the best possible platforms and practices in the back end to support this community’s ability to solve problems on the front end. KK: All that data, all that content sounds like it represents a tough chore to manage, to keep up with, and so forth. Does Stack Overflow have DBAs? Or is it the DBRE who keeps up with all the databases? AB: I don’t think there’s a person who’s called a DBA. We’ve got site reliability engineers (SREs) and database reliability engineers who form the “who keeps the site running” team. I’ll be honest, the application side is a much more powerful part of that than the database side. The application side has been strenuously engineered over a dozen years to be the most efficient and well-performing thing it can be. And there are tons of caching mechanisms that make it so that when you ask for things, you’re not hitting the database directly. You’re never even querying the database memory, never mind the database disk. From the application perspective, when you’re on the site, those pages are fast because we’ve had amazing engineers for 12 years who have made it that way. We’ve had people who have tuned that for more than a decade and made it the fastest thing it can possibly be. KK: The DBRE role seems to be an evolution of the DBA. How do you map that into your overall career? Is it a natural progression, or is it basically more of the same? And do you think we’re going to see a lot more evolution toward that role across the broader landscape? AB: I’ve taken an interesting career path, and I definitely wouldn’t call it traditional. I started as a web developer, but I quickly became an operational DBA at a startup that was doing advertising and website building. We were building a website for a company that wanted to start doing e-commerce. They saw e-commerce was going to be a thing, so they came to us and asked if we could build a shopping cart and a database. I was the only one who was interested in doing it, so that’s basically how I started getting into databases. I started with Microsoft Access, and we quickly learned that it wasn’t the most appropriate thing to use. And the only reason we used SQL Server at the time was because I was already an MVP for building websites with Active Server Pages and we got an almost-free version of SQL Server. This was a $150, unlimited SQL Server 6.5 enterprise license. That was hard to turn down—every other option was thousands of dollars—so that was a viable option for a startup. But my first DBA work was as an operational DBA, and as you know, I spent some years as a community person. I was a product manager for Plan Explorer® when I was at SentryOne, and then I shifted back to operational work. I think what has really happened in the industry is we’ve shifted away from a single database, single application mindset. We’ve gone from, “how do we solve individual performance problems and keep replication running?” to “how do we keep availability groups (AGs) in sync across a seemingly limitless number of apps and service-based offerings?” KK: What’s the difference between a DBA and a database site reliability engineer? With SREs, I think about the four golden metrics—latency, errors, concurrency, and saturation. And the whole idea of observability instead of monitoring, such as automating resolutions, is intriguing. So that is an enormously different thing than what DBAs typically do. DBAs are like, I will resolve the problem; I just want you to tell me there’s a problem. What other differences are you seeing? AB: I feel like the DBA does the day-to-day things, and the DBRE takes the things we learn as DBAs and turns them into strategies and directions. One analogy I’ve thought of is the guy who’s sticking his finger in the dam to prevent leaks. That’s the DBA to some extent. Sometimes, they’re adding cement and fixing the problem in a better way than just plugging the leak. The DBRE is more like the person who’s observing what’s happening with the dam and planning the next one to be better. And sometimes, it’s not designing the next dam. Sometimes, it’s building a canal to route some of the water away from the dam. It’s not always necessarily “add more things to SQL Server.” Sometimes, it’s “don’t even put that data into SQL Server in the first place,” and making those decisions of where data should live, how users should access it, and how often it goes in the database versus not touching the database at all. KK: As a DBA, it sounds like instead of just looking at what happens in the database, you have more of an application performance management (APM) approach. You’re looking at what’s happening on a particular app—is it hitting our caching? Is the data properly cached? Then you’re looking at how things get to the database—considering several other parts of the workflow. Do you think that’s an accurate summary? AB: Yeah, I think so. Sometimes that comes into play, but it really depends on what it is. At Stack, certain parts of an application have very specific routes through caching layers, so they’re never going to the database. While we have other parts that need to go to the database—for a lot of writes, you can’t not go to the database. If you need a change to be seen by all the things, you can’t pull MySQL shenanigans with eventual consistency. You’ve got to do it right and put it in the right place. There are workflows where we really have to fine-tune those paths and make sure those are as efficient as possible. KK: What tools are you using to get started in your new role? Have you changed your tooling much as you moved from DBA to DBRE? AB: I’ve used SQL Sentry® most of my career, and when I went to Wayfair, they used SolarWinds® Database Performance Analyzer (DPA), so I switched over. At Stack Overflow, I’m back on SQL Sentry, so it’s a very familiar toolset. There’s not really a whole lot else I’ve gotten into. The main tools I use are my keyboard, monitor, brain, and collaboration. The main tooling that’s changed are the systems through which we communicate and make decisions as a team. KK: The kind of thing I hear from people attending your sessions is, “Aaron is so informed. How does he stay up with all the latest?” AB: I read a lot of blogs, and I pay attention to things smart people are posting about. Over my career, I’ve found that blogging on a topic forces you to become an expert in it—as does answering questions on Stack Overflow. You must have the right answer, or you’re going to get pummeled. You’re going to get downvotes, and on the blog, you’re going to get comments. Being the person writing or teaching a concept forces you to learn everything you can about it, so you become an expert in that thing. And I’ve always made a point of trying to do that. I haven’t always succeeded. There are lots of things about SQL Server that I don’t understand, and people will talk circles around me about them, but I’m OK with that. KK: This question has come up in different ways earlier, but again, people want to work at a cool company like Stack Overflow. What sort of advice would you give to someone who says, “I’m willing to change jobs to make those aspects of my career happen. Because you have inspired me to speak more, to write more, to say to myself, ‘I could be a DBA or DBRE at Stack Overflow.’” What kind of advice would you have for somebody who wants more than what they’ve got? AB: You always need to have the attitude that you’re not solving this specific problem. You have to step back and look at things from at least one level higher. Not just “how do I solve this problem” but “how do I change something to make this problem not a problem at all?” What you want to do is make yourself scalable. You want to be able to solve problems before they’re problems. It’s such a simple thing, but I’ve found over my career that we can get stuck in the processes we’re doing. We get in this mindset of saying, “I know how I fixed this last time; I can just do that again.” Shrinking databases, defragmenting indexes, kicking bad plans out of the cache. You repeat those processes because it’s the known path—even though it’s annoying and it sucks. It’s harder to step back and say, “Well, is there a better way?” Oftentimes, there is. At SentryOne, I didn’t solve a lot of database problems, but I got exposed to our customers who had massive scale. Answering questions on Stack Overflow was a way to supplement that, to keep me fresh. Some people go on Stack Overflow and are really good at a MERGE statement (which should never be used, by the way). They’ll go and seek out all the MERGE questions, and they’ll answer them, and all the answers will be the same. You’re not growing by doing that; you’re repeatedly answering the same question. I did that for a long time with string splitting problems. I learned string splitting and string concatenating the old-style way. I know those; I’ve memorized those. I can write an answer in a minute and solve the problem, but I’m not growing when I do that. Donate some time answering questions in the areas in which you’re proficient, but take some time to help people solve problems that challenge you as well. KK: You had mentioned reading and keeping up with Stack Overflow as a great way to keep your skills sharp. Who else do you read, or what podcasts do you listen to? In my case, any time Paul White writes something, I’m definitely going to read that. If Conor writes something, I’ll read it too, and I do keep up with the SQL Skills blogs and a few others. Who are some of your touchstones that you try to make an effort to read? AB: SQLSkills for sure, Paul White, Itzik Ben-Gan. I’m an editor at SQL Performance, and I manage the content for Itzik, Paul White, and Paul Randal. So, I’m able to stay on top of that a little bit easier because I already have to read it. Every time I review one of those posts, I learn something, and that’s valuable. There are a few other people in the community that I read every time they put something out. Erik Darling is one of them, along with Joe Obbish—who writes on Darling’s blog, Darling Data. Joe has a recent post that I’m referencing right now for a solution with a query using a columnstore index. Nick Craver was an architect at Stack Overflow and works for Microsoft now. He just becomes perfect at everything he tries. He has posts about electricity and office organization and all kinds of technology things. I posted on SQLblog.org about one of my laptop stands—I put SSDs on the back and meticulously wired everything. I would never have cared enough to do that before I met Nick. He’s the kind of person who motivates you to make your workspace and even nontechnical things in your life better. KK: You’ve had a very enviable career. At times in your life, you were a technology evangelist—a lot of people think it’s great to be the rock star on the stage—but you’re also performing on the technologist side of the house at the highest levels. What career advice do you have for our audience? AB: The best piece of advice I’ve ever heard was just a few days ago. Someone I was having a casual conversation with at Stack said, “Don’t sell your happiness.” Don’t stay in a job because of the money or the perks if you’re miserable. Do the work you want to do. I stayed as a technology evangelist probably for longer than I should have, and I missed out on some of the actual technology pieces of things that I could have been contributing toward or learning. I was making decent money, and I had this fear of change. I didn’t want to rock the boat or fight for a better situation, and I didn’t want to leave. Really, all you’re doing in those cases is selling your happiness. I’m realizing now what I lost by selling my happiness for so long. I’m glad I know it now and not in another ten years. KK: Aaron, it’s been so good to catch up with you. I really appreciate your time, and thank you so much for chatting with us.
