Announcer: This episode is brought to you by SolarWinds Database Performance Management Solutions, designed to help manage database platforms running on-premises, in the cloud, or in hybrid environments. Visit solarwinds.com/solutions/database-solutions to learn more.
Thomas: Welcome to SolarWinds TechPod. I’m your host, Thomas LaRock, Head Geek here at SolarWinds. And on this episode, we’re going to explain who DBAs are and why they’re disappearing. Joining me today in this episode of TechPod, SolarWinds Head Geek, Kevin Kline. Welcome, Kevin.
Kevin: Thanks, Tom. Great to be here.
Thomas: So Kevin, let’s just dive right in, and let’s talk about how does somebody even become a database administrator? How does that even happen? Because it’s not like you go to school for this, right?
Kevin: Yeah. There’s not a college degree for database administration. You can get a CS degree, you can also kinda get in the ballpark of databases with say a management information systems degree, but what we find happening is that you just had one or two optional courses you could take on databases. And so universities aren’t graduating people who are great at administrating, managing, and performance tuning databases. That means you got you have kind of a two-stage career. In my case, I was a programmer for things like Fortran and Oracle, and some of the character-mode systems back in the ’80s. Then I went forward with things like PowerBuilder, if you remember that, and Delphi, and finally, Visual Basic and that sort of thing.
And I was always just someone on the team who was better at SQL than the average bear. So when the time came that we had to have a DBA, I was asked if I would take that on. So I had spent six years as a developer before actually being a DBA. And what I find is that that’s extremely common. People started as something else, maybe a sysadmin, a network admin, a server admin, or a developer, and then later on, they say, “I really enjoy the data side of things. Let’s take a look at this as a career.”
Thomas: So, I do remember PowerBuilder quite well actually. And I would say how someone becomes a DBA, I usually find that there are one of three things that they have done, either on purpose or maybe accidentally, or just helping out. And one of those are, they took a backup, they maybe did a restore, or they reset somebody’s password. And there you go, congratulations. You’re now the new database administrator. The old guy just quit. Don’t worry about why he might’ve left. And you’ve done a restore before, right?
Kevin: Yes, indeed.
Thomas: So, like you said, people fall into the role because they’re maybe doing these types of tasks already, or they just get assigned the role simply because they’re considered the most competent, trustworthy, or they’re just available, or maybe all three of those things. I’m guessing you’ve seen a lot of that over the years.
Kevin: Yes, exactly. In my earliest IT job as a developer working with Oracle, I got drafted into the role of backup DBA, which at that time meant, if our DBA needed to take a vacation, then I was the person who made sure the backups were available and that they worked, and that’s all. And as you said, reset a password or two. That’s all it took for me to be considered the backup DBA. So yes, exactly what you said.
Thomas: So I had a similar route where I actually requested to learn a little bit about database administration, and the DBAs there saw that as an opportunity that they could maybe have a day off. So if they did take a vacation day, that I would be somebody that would be there in order to help, like I need to restore prod to test or, so I can’t log in, things of that nature. I can recall, it’s been a long time, but I recall I would step in, they would take the day off. I’d step in, I do those tasks. And then they were nice enough to send emails to my boss, say, “Hey, just let to let you know, I took a day off. Tom handled a handful of our requests.”
Thomas: He even looked at some performance tuning things, like, “Just to let you know, we’re fully staffed and we have backup when necessary.” But that’s all it took, because when they left, they were like, “Well, we need a DBA. What’s Tom doing? And can we have Tom from your team? Can Tom come over to our team now? Because we need somebody to do this, I think.” And I also found that I was tasked with some of these things simply as my role as a database developer. So, in that case, it was a matter of, “Well, I’m trying to optimize the system that we have.” And so you start learning about, not just the queries you’re writing, but the structures of the indexes that you’re using and how this might speed things up, if you try this, that, and the other.
Thomas: So I find that in a lot of times, in a lot of different roles, you have a team of people, say a team of data engineers, but one of them might be doing these more of an administrative role than the others. So data engineers, data analysts, data scientists, these are teams of people where you find that one or two of them are actually handling a lot of DBA duties. Would you agree?
Kevin: Absolutely. And just like you said, when I was doing a lot of that database development, and this again will date me as to how old I am, but I was the first person on the team to figure out how to do a subquery. It was quite new in the ANSI 86 Standard, right? And so people were like, “How did you get the second result set into your report?” So, they’d say, “Okay, Kevin, you’re going to have to train the other developers to use these innovative techniques,” like a function call or a subquery.
Kevin: And so, again, just as was the case with you, the DBAs found that this was a way to vent some of the pressure that was building up on them to get stuff done, because they had a lot of things to do and it took a lot of time. And so if they could just say, “Have you talked to Kevin first about this query tuning before you bring it to me?” And they’re like, “Oh, no, I haven’t done that.” They’re like, “Oh, talk with Kevin. He probably knows the answer.”
Thomas: Now, how about mistakes, as part of that natural growth? Now for me, in my example was I had talked about, we’re trying to optimize a system. A lot of it was, or a good chunk of it was a reporting system. Of course, you want a query that can build a report fairly quickly, process the data. And so for me, I said, “Hey, I think I want this thing that’s called a covering index.” It’s going to touch, every column of the query is part of this index and it’s going to speed things up. And so I’ll say, the junior DBA at the time, “I went about with these structures. I’m creating, I’m altering the schema, I’m adjusting things, I’m trying to optimize.” And of course, my mentor, who is an actual DBA, he points out that the table already had a handful of indexes on it.
Thomas: And he’s like, “So, you’ve probably made your … I’m sure your query is running faster. And I’m also sure that everybody else’s queries are running slower.” But as he put it, he thought I would say, “It could. We don’t know yet. We don’t know. It could have made things slower.” So it was mistakes that along the way. I think it was nice to have that safety net of there was somebody there to guide me. I was also allowed to make these mistakes and learn from them. And I think that was kind of crucial, when somebody is becoming a DBA, is having that ability to kinda screw things up.
Kevin: And you bring up a really good point. I have similar stories about ridiculous mistakes I made just from ignorance. But one of the things that really does shine through for people who were effective database administrators, managers of these information systems, you have certain characteristics that that mentor is on the lookout for. For example, when I was a developer, in a way, conceptually for me as a developer, it’s like building something fun and cool with Legos. I really just love to build stuff. And so to roll something out that couldn’t pass the QA, to try to roll it out and then QA pushed it back, that was like, “Your Lego design is not good enough, Kevin. Go away and make it better.”
Thomas: I agree completely. And you struck upon this thing of some personal growth of understanding how you had to have kind of alter your behavior, so to speak. And that was certainly part of it for me as well along the way, because no matter how much experience you might have in say, customer service roles, when you become the database administrator, and I like how you said nemesis, and we’re going to come back to that later towards the end. DBA was the nemesis. The thing about it is it’s a different type of customer service than I think any other, like other industries, you work in the restaurant hospitality thing, that these are, to me, almost standard and basic, but you can have those skills, but they don’t necessarily apply when somebody’s in front of you and they’re telling you what is wrong. They’re telling you data doesn’t match. They’re telling you, you’ve screwed up. You restored the wrong database, because the data they want isn’t in there. So it’s your mistake.
Thomas: And you’re like, “Wait a second here.” And it’s really easy to become defensive when you’re basically getting beat up for stuff. And so I tell people, as a DBA, you have to remember, it’s not your fault, but it is your responsibility. Once you start to understand that it’s not your fault, but you are going to be responsible for fixing it, it can help change your perspective and the interactions you have. Let’s talk a little bit more about the traits though. Would you say a DBA is going to be good with data in general?
Kevin: Yeah, that’s a fundamental. And you want a DBA who can, maybe they’re not really even extremely good at databases, but they’re good at data. So somebody who knows how to, if I were to hire a junior DBA and they came in and showed me some beautiful spreadsheets they made and how they extracted some information from it that was a little bit unexpected, but very useful, that would be a person I’d say, “I want to talk more with you. Let’s take this interview further. Let’s go to the next stage.” As opposed to, sometimes I encounter DBAs who are like, “Yeah, here’s what I do. I do backups, I do DBCC checks and I do restores.” “Okay. That has value, definitely. That has a place.” That kind of stuff is being commoditized, especially in the cloud, where they automatically take backups and they automatically have geo redundancy and things of that nature. So yes, data savvy is going to be a very important thing.
Thomas: And I think, so for DBA, they have to at least have the appreciation of data, but I see it as two different views, because to me, there’s database administration or the … When I was coming through the ranks, when I was a production DBA, you really had database administrators, you had database developers. So you really just had these two groups, but the developers were really the ones that might be working more with the actual data. So these developers were also maybe more like a data analyst or an engineer, and sometimes a data scientist.
Thomas: So I may have not understood the data as well as they did, but in my role as a DBA, I was the one that helped protect it. I was the one to make sure I could recover the work they have been doing. I was the one that helped them understand why data types would matter. I was the one that helped them understand when they kick off three jobs at the same time, and one’s doing an update, one’s doing a delete, and one’s doing a select, why they all might run slower than if you did them at different times. So I would say you have to be good with data and databases in general. You have to build up that experience over time, but where does that come from? Do you get that? Can you get the same experience just by using Excel Power BI?
Kevin: No, unfortunately. The single piece of advice I give to junior DBA is, they’re like, “Tell me one thing I can do that’s going to improve my prospects as a DBA. Just one thing.” And I always say, “Get a mentor. Seek one out. Find somebody whose career you want to emulate.” And like you mentioned, Tom, you had someone to help coach you through those hard times. And there is literally no replacement for that. It is incredibly valuable. Now, today we do have some additional options. We’ve got lots of videos you can watch, for example. So you can go to YouTube and learn massive amounts. But if you have a mentor, it’s many layers more than just learning a technical thing. A lot of times they can advise you about how to interact with different kinds of personalities, and watch out for this area of risk, or watch out for that area of risk.
Thomas: So, yes, you’re not going to learn that from spreadsheets. In the same way, I think there is one big benefit from the database developer path into database administration, which is scripting becomes pretty important as a DBA. We spent a lot of time writing TSQL scripts, and in the last 10 years or so, PowerShell scripts. And so if you don’t have really any understanding about how to program, and I’ve talked to quite a few DBAs who say, “Yeah, that’s me. I just use the GUI.” That’s going to be an inhibitor. There’s going to be times when you’re going to find you have to work way harder rather than smarter to get something done.
Thomas: So I would tell somebody, if they were looking to become a DBA, you need to have some level of coding, some coding experience. And it’s mostly because you have to be familiar writing what we’ll just call generically code. And that could be, in our case, TSQL, it could be PL/SQL. You have to have a comprehension of what you’re doing and what you’re using for the tools for the job, but when it comes to the scripting stuff, well, there are a lot of tools out there that generate code for you. Like in SQL Server Management Studio, I can right click and I can generate, and I can create scripts of my databases and stuff.
Thomas: And I don’t really need to remember a lot of syntax, but I need to have an understanding of what would happen if I run this thing. And it’s this whole thing of you don’t have to be the world’s best database developer in order to transition to be a DBA. I didn’t need to have that level of skill. What I needed to do is say, there were certain basic functions that I had to be able to focus on, but let me ask you this. Do you think there’s somebody is listening to this that isn’t the DBA yet, but they might become one? And my question to you is, do you think they will ever use an entity relationship diagram?
Kevin: Well, I pray that they don’t have to. Entity relationship diagrams are for database builders, right? You’re constructing and designing a database. So nowadays, what I find is more and more often, when you and I were back in the enterprise, in the classroom, we didn’t have to go through a buy versus build process, a decision process very often, because there just weren’t that many products that we could buy that would do what we wanted. So in our time, it wasn’t uncommon for us to have to not only know and read entity relationship diagrams, but also write them or use, we often had tools for that. Things like ERwin or ER/Studio. And they were expensive.
Kevin: But nowadays, I would say very few DBAs would need to go to the extent of buying one of these very expensive tools and writing their own ERDs and things like that, because there’s already products you can buy for inventory control, or for time management of your consulting staff, or just a huge plethora, panoply of products you can buy that do all that for you. So, my hope is, as time goes on, more and more of that will be available for you off the shelf. And you don’t have to spend all that time building things. Now what you can do is you can spend your time adding value to the business, like helping the different dev teams write really insightful reports, that sort of stuff. So it’s possible, but I would say the number of people today who need to know ERD stuff is small, 15% maybe, and declining over time.
Thomas: So, I don’t think a person hearing this today that might become a DBA, I don’t think they’ll ever look at an ER diagram ever. I don’t think they’ll ever, they’ll never print one out and put it in their cube. They’ll never know the moment or the joy of you want to make a change to a datatype and you’ve got to go and try to figure out where it is everywhere on this sheet of paper and drawing lines and say, “Oh, it’s here, it’s here and it’s here.” They’ll never have that wonderful experience, but I don’t think that’s a bad thing either. And like you’ve just mentioned, there’s so much more software available that you can just purchase to fill the need. And that means, and this was true even for me back in the day, we were the DBA but we were really just the administrator for a third-party product.
Thomas: And we’ll mention just some of the, you were by default, the SharePoint admin as well, because that ran on SQL server, or you’d be the BizTalk admin because there’s certain things that that needs, or app dynamics, all these other tools. So you were still the DBA, but you also have these limited number of things you can do. Like if SharePoint, you can’t do anything. They’re like, “Just don’t touch it. Don’t touch it at all.” Which is always a warming thing when you’re in that meeting with Microsoft and it’ll resist the urge to touch SharePoint in any way. But here’s the thing, is I’m the DBA and somebody is expecting me to make things better. And so what tasks can I do for these third party apps?
Thomas: So, because these companies are able to buy these tools off the shelf now, I feel that there’s less and less of these tasks that you end up doing. And I think as we get more and more involved with the two clouds, as I call them, with Azure and AWS, the more that they offer and the more that’s automated, is the less there is for a typical traditional database administrator to do. So let’s talk about some of those common tasks. Let’s think. We’ve mentioned a few, backup, restore, password resets, right?
Thomas: If you’re going to be a DBA, I would tell you that your number one job is recovery. If you can’t recover, you can’t keep your job. Update the resume, leave town, right? It’s, somebody is going to come to you, I don’t care where your data is, I don’t care what structure it is, I don’t care if it’s 1,000 Excel spreadsheets pulled together with a PowerShell script. Somebody thinks that’s a database. You’re the DBA. If you can’t recover them to whatever point in time they need, you cannot keep your job, your role. You just can’t.
Kevin: Absolutely. In fact, that’s the only time I’ve known a DBA to be fired on the spot, was a really important database couldn’t recover it. And the CIO said, “Go home, pack your desk up. We’ll find somebody who can fix this.” I have a story to tell too, of the early days when I was transitioning from developer over to DBA. We had spent a huge amount of money, this great big enterprise that I worked at. And we went to Compaq computer company. If you remember the good old days with Compaq in the ’90s. And we spent probably a couple hundred thousand dollars running enormous benchmarking tests to size exactly what kind of servers we were going to get. And these are the kinds of servers that if you ever read the TPC benchmark reports, it says, “This server costs $300,000.” And because it had 900 hard disks in it, the hard disk array costs $300,000, just top of the line equipment.
Kevin: And this whole time, having actually just recently transitioned to database administrator from database developer, I was thinking about, “Oh, my God, what are we going to do when one of these servers crashes?” Because it’s a given. All of the equipment in the ’90s had a meantime between failure. Nowadays, it’s more just hard disks and SSDs have it. But back then, it was like it’s a given. This is going to crash at some point. And it did, probably about six or eight months later. And what really shocked me about that experience was I knew the CIO or the deputy CIO would call. And I expected them to say, “What are you going to do about my server, this thing that we spent hundreds of thousands of dollars on?”
Kevin: And instead of that, the deputy CIO said, “What are you going to do about my data? How are you going to get my data up? How fast is that going to happen?” And it really spun my head around, because we had thrown all this money down to figure out the right servers and then to buy the right servers, and all this time I was thinking that was the asset. But in fact, no, that’s like the bottle that this Rothschild, very expensive wine comes in. The data is the wine. That’s what’s so valuable. The bottle is incidental, the hardware. That was a huge lesson for me.
Thomas: Absolutely true. I’ve said it before, data is the most critical asset any company owns. I mean, somebody might argue and say, “No, the code,” but code is just data as well. It’s just data is the most critical thing. And if you can’t recover, that’s a problem. So when we talk about the common tasks for a DBA, I tell them, if you’re going to be a DBA, the first thing is to focus on the recovery plan. Backup is a part of a recovery plan. And your recovery plan, the two things you want to know is your RTO and RPO, right? Recovery time objective, recovery point objectives. That’s how far back can I recover and how long will it take me to physically … You wouldn’t have told that deputy CIO, “Oh, I’ve got the data, but it’s going to take three days.”
Thomas: Yeah. But he might say, “Okay, three days is a long time, but all my data? Okay. Then it’s secured, I feel better,” because the hardware is the hardware. And I also try to tell people hardware can be replaced, and so can people, but the data, that’s almost irreplaceable. You might think, “Oh, I can just go recreate this. I can get to the source.” These are the questions you have to ask as the DBA, when somebody tells you, “Don’t worry about it, don’t worry about it. It’ll be fine.” No, no, no, no, no. You need to ask. You have to ask those follow-up questions and say to yourself, all right, if things really go wrong and you think you can just go download it, the data set again, but what if that’s not there? Now what?
Kevin: Massive problem.
Thomas: Yeah, you don’t have to worry about backups on the server.
Kevin: Well, and related to that, Tom, you mentioned RPO and RTO. This is one of those maturing points in your life too, if you’re coming up from a different starting place into the DBA world, because this is where you have to present options to the decision makers, because your decision makers probably are going to say, “I want you to be able to recover a database failure immediately within three minutes and I don’t want to lose ever more than three minutes worth of data.” And a lot of us are like, “Oh, that’s great. Do you realize how much you’re asking me to spend to achieve that goal? So yeah, we could dump transaction logs every three minutes and we could have a very powerful system that we could spin up as a secondary or a standby, but that means we have to buy an entirely second set of servers, or third set of servers to make this happen. This means we have to have massive amounts of storage to have all of these backup files from point in time.”
Kevin: So, these are things you’ll learn to negotiate. You’ll start to say, “You know what, boss, great idea, love what you’re thinking.” So as Wayne from Wayne’s World used to say, “I love what you’re doing here, but let’s consider, if we have to have no more downtime than three minutes, that means we have to investigate, not just backup and recovery, but high availability.” That’s a whole new world, in terms of the technology required and the experience required. If you ever spend time on one of the Q&A websites, like Stack Exchange, or SQLServerCentral or those kinds of places, what you see is high availability is a constant point, a struggle. People are always trying to figure out how the heck do I make this work?
Kevin: And so, again, it’s the thing that you’re shifting from a person who builds stuff to a person who makes sure things are safe and secure. And that takes a different mindset. If I was interviewing a junior DBA and they said they lived in an apartment, I don’t know if I’d be breaking rules, but I’d say, “Do you have renters insurance?” Because of, they said, “No,” I’d be like, “Hm, that’s a warning sign.” We want people who constantly think about the risks and dangers, and build methods to mitigate those. And as you’ve got right here, backup is only part of recovery. So you got to think about that stuff systemically.
Thomas: And we’ve mentioned RTO and RPO, and I want to make sure I mention before I forget, is you have to test those constantly, because the business says, you just use the example, three minutes. Three minutes of data and it can’t take more than three minutes to recover that, and it’s putting me back. But as data volume will grow, those may no longer be realistic. And so when you talk about high availability, I do remind people, you may not have to spend a whole lot of money to build a system that has all the resiliency that you need. It just depends on the volume of data. Helping you with eight gigabytes versus eight petabytes.
Thomas: So if you have a large volume of data that you need this tight RTO, RPO for, then the price skyrockets, but if you’ve got something that it doesn’t have to be that large to be incredibly valuable, but you’ve got to test. You’ve got to stay on top and make sure that the RTO and RPO always fit into what the business requirements are. So other tests. So we’ve talked a lot about recovery. Again, job one, recovery, right? And we mentioned the password resets are a thing, of course. You should be using a windows authentication, but in general, the idea of doing, I mentioned password resets it’s, say archaic, but the idea is there are lots of simple mundane tasks that you’ll end up doing as part of being a DBA.
Thomas: One thing that we’ve touched upon, data engineering as well, to me, that’s all the pipes, that’s how the data moves from one to another, the way that gets done. But part of that, let’s talk a little bit about query tuning or just database development in general. I mentioned this, because SolarWinds has the SolarWinds query report that just came out, all about database priorities and pitfalls. Half of the respondents said that performance and query tuning is among the top three skills that they should improve. And as somebody who has done, I’ve tuned a query too my time, I will tell you, I would always feel that I could improve upon that, but this is going to be one of those tasks that I think as a DBA, you’re going to get called upon to do time and time again. Agree?
Kevin: Absolutely. In fact, I double down on that, Tom. The reason for that is although probably only 20% of the market, in terms of enterprise shops, are all in on the cloud, it is accelerating adoption of the cloud, whether that is AWS or whether that is Azure, or what have you. And when we have on-premise SQL server, if we have a poor performing query, we just have angry users, but in the cloud, if you have a poor performing query, you have actual expenses that go up, because you pay for CPU, you pay for data egress, and things like that. And so if you’re a DBA today and you’re saying, “No, I don’t know how to tune queries very much.” I would encourage you to really invest some learning time in that because it’s important today already, but it’s getting more important with time, because the cloud makes it an actual cost that you can see the bad SQL is costing you money.
Thomas: And then in our survey report, they said the top skill that they could improve upon is performance tuning, but this is followed by security management, data engineering comes third. So the security management second to performance tuning, and for me, I look at that and I think to myself, there’s the opportunity as the DBA to talk to somebody about the queries they’re writing and to think, “Hey, could I help? This is sensitive. We should be using data masking. Maybe those credit card numbers should be encrypted.” Things of that nature is where you’re going to have to really show your value as the administrator of this data and help people understand what can be done, what the cost benefit risks are for all these decisions around the data itself.
Thomas: I would stress highly that the common tasks of these, the recovery and of query tuning and data engineering, that’s where you’re going to spend the bulk of your time most likely as a DBA, but there’s a lot more. And I know that you’ve done, you’ve talked about this quite a bit, all the different tasks. And Kevin, let’s just run throughout some of these other tasks. For example, I’ll start. Let’s just name one. How about alerting?
Kevin: Yeah, this is essential. If you want to achieve anything besides firefighting boat in your career, you have to get where you have at least a baseline of monitoring and observability, and alerts that are based on that. If you don’t do those, you’re just going to be a firefighter, just responding to the latest user complaint in terms of speed and performance of the different applications you’re running.
Thomas: So, how about another one, well, I’m going to say maintenance, but to me that’s, let’s say indexes, statistics, fragmentation. You get something else to look at, right?
Kevin: So like you said, indexes and statistics need to be maintained and updated. There’s also certain kinds of preventative maintenance you need to look for, for corruption. We call them DBCC checks. There’s also things like suspect and torn pages. There’s memory dumps that happen. And so you need to have some watchfulness. You need to have some observability and alerting on those things too, so that you can tell, “Hey, this particular table hasn’t had its indexes and statistics updated in a year. We better refresh those.”
Thomas: So also in our SolarWinds Query report, the respondents said, one third of them said that they spend a third of their time just doing maintenance, which I actually thought was amazingly low when I thought to how I used to be a production DBA, I spent far much more time of my time on maintenance tasks. And to hear that it’s a third, I thought, “Wow, that’s great. That’s a lot of extra time for myself. What would I do with all that free time?”
Kevin: That’s right.
Thomas: Well, I’ll tell you, well too, I got a long list of tasks that DBAs would be involved with. I mean, how about, I don’t know, tracking down, locking, and blocking and just transaction behaviors in general?
Kevin: Yeah, exactly. One thing I do find, particularly because of the nature of the accidental DBA, we’re not really taught anywhere in that process of a consistent troubleshooting checklist. If you’re a pilot and some alarm goes off, you can pull a book out of the cubby hole in the cockpit, and it tells you exactly what you need to do. There is a checklist of steps you follow. And if you’re flying solo compared to a co-pilot, there’s different steps under those circumstances too. We don’t have that in the database administrative industry. And so what happens is a lot of people just kind of, “I’m used to using Profiler, I’ll use Profiler.” And somebody else will say, “I’m used to using perfmon, I’ll use perfmon.” But that’s not a systematic, guaranteed way to achieve actionable results.
Kevin: And so, again, that’s the kind of thing that you and I both evangelize at conferences and things. Here’s my checklist to solve problems, whether that be locking and blocking, whether that be some I/O, subsystem problem, or CPU and NUMA, and memory problems, what have you. There’s a way you can approach these, that guarantees that you can get to the root cause pretty quickly, and without panicking. I was talking with one junior DBA. It’s like, “What’s the first thing you do when you have to troubleshoot a problem you really don’t know anything about?” And they said, “At all earnestness, I panic.” I’m like, “Okay, well, that’s probably not the best first step, but I can relate.”
Thomas: So one thing that has come across for me through the years, when I first started, database server was something located just a few feet from me, and it was all encapsulated in this one system, so to speak. It was a physical server, direct attached storage type thing. And over time, things got a little more abstracted. So virtualization, so like you just said, you got to get to the root cause. I started understanding that I needed to learn more about all of the different layers that were between people and their data. So virtualization was one. We had a SAN. We had a storage area network at the time, and understanding how SANs operated. And all of this, I couldn’t just write a store procedure, call myself a DBA and be done. I had to learn a lot about a lot of different layers in order to really be an effective DBA as well.
Thomas: So, of course, the latest and greatest has been understanding things like NUMA nodes and how that architecture. Now we’re talking about the physical architecture of the CPU sockets and how that can relate to performance. So there’s just so many of these little, I’ll just call them rabbit holes, that you can dive deep into. I don’t think you’re going to find anybody that is a true expert in all of these areas. You’re going to find a handful of people out there that have some decent level of knowledge over the years in a lot of these areas. And someone like yourself, of course, right? You have a lot of experience across all this, but for somebody starting out, I would never tell a DBA, a junior DBA, “Yeah, there’s 27 things that you’ve got to learn.” It’s more like, “There’s 27 things you’re going to have an opportunity to learn about over time as it becomes necessary.” So otherwise, it would be overwhelming to give this list to somebody and say, “Here’s everything.”
Kevin: And to add to what you’re saying too, Tom, that’s a great insight. To add to what you’re saying though, there’s one thing about databases, database administrators, that’s very different than when I was a developer or as an administrator or a help desk person. If you’re a network administrator, every TCP/IP packet that you care about is just a TCP/IP packet. You don’t have to do something different from the email that goes out on TCP/IP compared to the user connections to a SharePoint server. It’s all TCP/IP. But what’s really different for data people is, let’s say I work at a trucking company as a DBA. Okay. My data will require certain levels, like you said, RPO, RTO, service level agreements, things like that. But if you, Tom, are working as the DBA at a very important radiology center and your data is about people’s lives on the line, everything changes.
Kevin: Now you do have to know about CPU and NUMA nodes. You do have to know about how the queuing system works to the SAN. You do have to know about high availability. So here I am, as a DBA, a commodity trucking company, I don’t have to learn all that stuff because a nice $4,000 server can carry the workload for each of my databases. But you, because your data is so different, you have to be a different kind of DBA, and more knowledgeable and more capable. And you don’t get that with sysadmins, or network admins, or email admins. For them, their little niche is just that niche. And if they managed a huge email server or a medium-sized one, it wouldn’t make that much difference. They’d have more requests to restore Outlook inboxes, but that’s about it. But when it comes to data, it can completely change the dynamic based on what kind of data you are responsible for.
Thomas: So we’ve talked about where DBAs come from, the skills they’re likely to have, the tasks that you would do in the DBA role. I want to kind of round out the episode. I want to talk a little bit about the acronym itself, DBA. Most people in the world, they see DBA, it’s Doing Business As, right? So the first one I’ll say, DBA, I’m assuming you’ve heard all this Kevin, but DBA stands for Do Bout Anything.
Kevin: Absolutely. Like you said, the DBA maybe stands between end users and virtual machines, stand between the SAN and the database and users. All they know is that things are slow, right? So you might get called in the middle of the night to fix some very important workload problems with queries and things like that, but then you also might be asked in the middle of the day, it’s like, “We just designed and deployed this database production and it’s not working like we thought, even though we test it in QA. Figure this out.”
Thomas: Oh yeah, that’s the thing. You’re the catch-all, you’re the person, you’re the senior administrator. You really become. And you’re the one that can open up the manual, read it, and then understand and say, “I know what this is saying to me. I know what I need to do next.” You’re the person they come to for just about anything. And you’re going to tune a query, you’ll do some forensics root cause analysis. You’ll do it all at 2:00 AM as needed, and then you will also work in the morning and start on the next thing. And because you’ve developed, you have a unique set of skills. You’re like Liam Neeson. You have a unique set of skills, and everybody knows that. And you can do just about anything.
Thomas: All right, the next one, and this really, this described me as a production DBA. It was grueling at times, but DBA stands for Don’t Bother Asking, because the job, the role is tough. So for me, like you mentioned before, with alerting and monitoring and having, if something goes wrong, you need to be able to find and get to the root cause fast. So for me, I had a build process. If we were building a database server, it was going to be built the same way as any other database server. I was not a fan of having one-offs. I was not a fan of trying to go to a server, parachute in and figure out, what have you done to this thing? No. If somebody was having an issue, I wanted that server to be as vanilla as anything else.
Thomas: So, when somebody comes to me and they say, “Oh, I’m working on this thing. I need the …” “You know what? Flip the bit, make that database trustworthy.” And you’re like, “Whoa, whoa, whoa, whoa. I got thousands of databases. I’m going to make this one trustworthy. What are we doing here?” “Hey, hey, stop getting in the way of our job, Mr. DBA. Just do what we’re telling you to do.” And I’m like, “No, no, no, no, no, no, no. We turn on that trustworthy.” And unless you have a real good reason.
Thomas: And right now the good reason isn’t just because you want to try out some new feature or something. No, we’re not doing that. We’re going to keep things stable because that way, when something goes wrong and you need me to come in and solve it quickly, I have a better chance of doing it. I don’t want to be sitting there going, “Hey, why is this so different? Oh, my God, what did we do? Why is this one so odd?” No. No, no, no, no. So Don’t Bother Asking.
Kevin: I would tell a different dev team managers, what I am looking for is stability and predictability. And so because you have asked for, I don’t know, you’ve asked for Itanium CPUs that automatically injects instability and unpredictability into this situation. So absolutely, right on board with you.
Thomas: All right, the last one. This is why, I remember hearing this at, I was in, I think in MVP summit. DBA stands for Default Blame Acceptor.
Kevin: Oh, my gosh. Yeah. Whenever I would get these triaged requests, I’d lean over to the rest of my teammates and I’d be like, “Oh, we’re going to play the blame game again,” because they blamed us. But now it’s our job to prove exactly where in the system, with your experience with virtual machines. And it turns out there was some noisy neighbor on there. So one machine that had dynamic memory and ballooning and dynamic hard disk, and it was sucking up all the resources on that virtual host and preventing the other guests from getting any access. We had virtual machine administrators, but we were the ones who had to troubleshoot that.
Thomas: Well, it’s easy to blame something you don’t understand. And this is why I’ve come to realize in my career. It’s why people will blame the database. They have no idea what’s happening. I sent a query and I expect my data to come back, and quickly. They have no idea, it’s just magic. It’s just all the same, creating a Word document, a spreadsheet, querying a database. It’s just magic. I type and I get back what I need. So yeah, you have to learn that you are going to ultimately be blamed for something. Remember, it’s not your fault, but it is your responsibility.
Thomas: So these different acronym meanings, I think summarizes a lot about the role that the people that do it, you’re going to find yourself, at some point, you’re going to be the don’t bother asking. You’re like, “No, just don’t bother.” You’re going to have to be the Default Blame Acceptor, and you’re also going to have to realize you’re the one that do bout anything. And once you start to understand the facets of the role and how you got here, and you really do understand you have a unique set of skills and you are the DBA.
Kevin: One quick question, what was your internal private term for you and your fellow DBAs? So like in my case, we always called ourselves the plumbers, because it was a thankless job, nobody cared except when things were broken, and then everybody remembers your name. So that’s what we called ourselves. How about you? Did you have a little nickname internally for you?
Thomas: We did not. And now I feel like that was a missed opportunity in my career, but what we did refer to ourself, what we did do was we basically referred to ourself like Batman. And we had a red telephone and the cake dish, because essentially, that’s how we felt was our role, was that somebody needed help and there was a direct line and we had to answer it. Batman doesn’t get a day off. Right? That’s just the way it is. So no, I really feel I missed out on that because that wouldn’t look great on my LinkedIn profile. I was a member of the plumbers. I don’t know. I want to thank you, Kevin, for joining me for this discussion today.
Kevin: My pleasure.
Thomas: Hey look, the DBA role is complex, and pitfalls will exist everywhere you turn, and it’s difficult to survive and to thrive. On the next episode of the Disappearing DBA Series, well, we’re going to help those accidental DBAs become DBA survivors. And if you enjoy the SolarWinds TechPod, we’d love for you to follow, rate, and review this podcast. Thank you for listening. And until next time, I’m Thomas LaRock.
