Announcer: This episode is brought to you by SolarWinds® database performance management solutions designed to help manage database platforms running on-premises, in the cloud, or in hybrid environments. Visit solarwinds.com/solutions/database-solutions to learn more.
Thomas: A good DBA is hard to find, and this is because a good DBA, they evolve into having three or even four full-time jobs. And the secret is automation, and automation is the reason that DBA is disappearing. Welcome to SolarWinds TechPod. I’m your host, Thomas LaRock, SolarWinds Head Geek. Joining me today in this episode of TechPod will be my fellow Head Geek, Kevin Kline. Say hello, Kevin.
Kevin: Hi Tom. How you doing?
Thomas: I’m doing great. And also joining us is Chrissy LeMaire. Hello Chrissy.
Chrissy: Hey, thank you so much for having me, Tom and Kevin.
Thomas: Well, it’s great to have you with us. Do you want to tell us a little bit about yourself?
Chrissy: My name is Chrissy LeMaire. I’ve been SQL Server DBA mostly for the past 20 years. I am also the creator of DBATools, which is a PowerShell automation tool kit that actually makes SQL Server database administration super fun. And I kind of feel bad for DBAs of other database management systems because they don’t have that SMO and those libraries to make it just one command to do things like export your entire SQL server instance for disaster recovery or perform an instance to instance migration. PowerShell is fun, SQL Server is fun. I’ve been doing it a long time, and I’m really happy to be on the podcast. Thank you so much.
Thomas: Oh, we are extremely happy to have you here. And one of the things about DBATools, that project itself, when I first started coming across it, I realized that this was a way to really allow just about any DBA out there to automate at scale, like you just mentioned, exporting your entire instance for a DR purpose with just a click of a button and a couple of PowerShell scripts, things that would have just taken a larger amount of time from a DBA.
Thomas: So let’s just get right into basically explaining how you are part of the reason DBAs are disappearing. And it’s because of this automation. It’s because of people like you that have built these tools, these wonderful tools. So the SolarWinds Query report with database priorities and pitfalls that we’ll link to as well. That revealed that some of the top priorities for database management strategies, that it was really all about efficiency. They talked about automation, improving performance, reducing costs, everything’s about cost, cost benefit, and risk. So talk to us a little about your experience, a wealth of experience, decades of experience, and about how automation has really allowed you to just achieve and do more.
Chrissy: I used to dread even migrating just one SQL server instance, because if you have, let’s say you have SharePoint, it has 30 databases. It has how many SQL server service accounts? And then you have all of those jobs associated with it. And even that, that would be so much right-clicking, right-click, explore, right-click, backup, restore. That’s a whole bunch of downtime. And with PowerShell, all of that it’s now just one single command.
Chrissy: One of my favorite things that ever happened was whenever I was, I was a DBA at NATO Special Ops, and we had to migrate. We did a bunch of migrations. We were upgrading. We were keeping on the bleeding edge. And whenever it first started out, it was like, oh man, this takes so long to prepare. It was always a thing. It was a thing with the AD people. It was a thing with storage people. It always had to be coordinated. It was always just such a time intensive task.
Chrissy: And after PowerShell and DBATools, I remember I was on like probably the 20th minute of migrating. And then since I had gone to the grocery store, I picked up some cheese and bread. I came back and I was waiting, and the storage guy said, Hey, what’s taken so long? After 20 minutes to migrate an entire SQL server instance. And I was like, that’s the best thing that I’ve ever heard because it increased his expectations that this is just going to take as much time, pretty much as it takes to copy over that data. And so I blamed it on his storage of course, and the migration was done pretty quickly after that.
Chrissy: That’s some of my experience with automation. It really changed my life as a DBA. It made it far more relaxing. I knew that in the middle of the night, if I got a call that it, even if you had 20 different SQL servers or 200 even, it’s not a big deal because you just pipe in those SQL server names, you do what you need to do, and it’s all just simplified commands. It’s very quickly. It’s super well tested. And so actually since then, since DBATools, I’ve been a far more relaxed person and automation in general. Go on Kevin.
Kevin: If I could pipe in on that, my experience was a little bit different, but I had a similar kind of epiphany. What had happened was it was in the very early days when we had been building our first SQL server apps. We were actually running on an OS/2 version of SQL server because Windows wasn’t out yet. So we had gotten some of it to work properly and we’re still building the app. I was actually still officially a developer, but responsible for the SQL servers. And we were having issues late one night after a SQL server 4218 came out or whatever it was. And at that time I was only using the GUI long before SQL server management studio.
Kevin: But what had happened was it was probably 2:00 AM. I think every DBA has this story. 2:00 AM, you’re really tired, you’re worn out, and you click the wrong button. And I blew up the recovery process I was trying to do. And so I called my wife and I said, I’m going to be another hour and a half, maybe two hours at this point. And at that point, I kind of said to myself, I have to write these as scripts because once I tested it and it works, I will never make this mistake again as, even if I’m really tired, like I am now. And so that’s when the epiphany hit me, scripting automation, scripting automation, tested, as you said, with your PowerShell scripts.
Chrissy: I have a question about testing back then. I believe with the current era of CI/CD and DevOps, testing has really become a focus. And for me it feels like a very natural part of the process. It’s even built into my entire development time, where you’re given testing time back then, or time to develop tests. And were there even testing frameworks for what you were trying to do?
Kevin: No, as far as I know. We weren’t really given time for testing. In fact, it was always a perennial problem that project managers would say. The first thing they’d say if we were running behind was, you know what, let’s cut the QA time period from a month to two weeks. So we can make up on dev time. And I would always say to the project managers, it’s like you’re making that decision based on the idea that we won’t find anything during the QA cycle. When in fact we will have to fix everything we find in that timeframe and that’s why we need a month. So yeah, it was not a respected part of the process, I guess you could say.
Chrissy: What you’re insinuating is that now it actually is. I feel that way as well. You don’t have to justify it as hard nowadays to say that this is actually the time. I always talk about tests because they did change my life as the maintainer of an open source project, where we used to literally have someone poor thing. Like if we got a poll request, he would actually take that code and then he would execute it against his singular machine until we started using something called Pester, which is available within PowerShell, that the things that he would execute, we just added them to a script. And now every single time that makes a change to DBATools, we have thousands of tests that run both in Windows and now on Linux. So that is thoroughly tested, and I know that whenever somebody makes a change, it’s not going to break the tool set and introduce instability and make people kind of more apprehensive of using DBATools. I love tests.
Thomas: In my experience, when it came to the testing phase, I was very similar to Kevin where they would cut down time in other areas. First of all, the testing was just sort of expected. I would say it was more of a unit test mentality back in the day, and it wasn’t even the true unit tests because things weren’t really independent. But it was just the idea of it looks good enough, move forward. But can you imagine, if you’re putting up a building and you fall behind and then you just sort of say, well, we’ll just cut down on the amount of time we let the concrete set. Right?
Kevin: Right.
Thomas: And yet that’s how, and the real problem in my opinion is that you’re talking about industries and companies that have software engineering as a component of their operations, but they aren’t software companies. And they think they can just speed up certain parts of it by just throwing more hours at it. And that is not necessarily going to be the case all the time.
Thomas: Let’s transition a little bit more and talk about how that automation, and in my experience, the automation has really freed up my time to take on other tasks. So I would automate away a lot of different things in terms of mostly data migrations, backups, restores, but there is a lot of other things going on. Say, for example, if a person left the company, I built a process that would go and look for their login against instances, not just inside of a database, but a Windows login if they were ever already paying to the server back in the day. So we had to go through and say, all right, here’s their ID. Show me everywhere they are in the company, everywhere they’ve touched in the past 90 days, because we have to go and remove access manually from all that. Well, at first manually, and then automate the other way.
Thomas: So these were things that you ended up building, but then what would happen is one day somebody shows up and goes, well, we’ve got this app that’s coming in. It’s called BizTalk or it’s called SharePoint, or it’s called AppDynamics. And that runs on SQL Server. And you’re the database, you’re the DBA. So you’ll just administer this. I remember being the SharePoint admin for two months, sitting there, staring at it, and finally going to my boss, why am I the admin of this again? This has nothing to do with database administration. You need a SharePoint admin. This is a second job. So the automation, but of course I had the time to take it on. I had the time to take it on and do these things. But before too long, you end up doing a lot more than just focusing on your data, your database instances. Has that been your experience Chrissy as well?
Chrissy: Absolutely. I was actually the SharePoint administrator for this law firm that I worked for, and I loved it. At the time I wasn’t using PowerShell, but I was using VBScript. And we had, not that many SQL servers, but I really actually loved it because I’ve always had the developer in me and I was making web apps and it was so exhausting with classic ASP. And then eventually with SharePoint, it seemed very natural. And I was expected to do that. I created some super cool extra nets for some really awesome companies. But then I realized, oh my God, I am super overwhelmed. And what was really great is we were able to bring in a SharePoint consulting company who just evaluated the environment and said you actually need four people now for the size of this form, plus your database administrator. So I’ve absolutely had that experience as well.
Thomas: This is where I’ve tried to point out, once you have those three or four full-time jobs, when you go to look for a new job, let’s say you get one, because now you have four times the opportunities to land a new job. But you go somewhere else and then they have to backfill your position. And they’re like, so what’d you do here? Well, you did all these things. And then they write up this job description, which becomes one of those horrible job descriptions we always laugh about. Like, look, they’re advertising for four or five jobs. And I look these days and I say, yeah, that’s because they’re replacing somebody who’s been there 20 years and they’ve automated the way so much that they can do it all and they can do four or five jobs somewhat efficiently.
Thomas: So Kevin, I’m going to transition a little bit here, and I want to talk about some of the other tasks that we see getting automated the way, and that’s going to be maintenance work. So in our query report, the respondents said a third of their time was taken up by maintenance tasks. So in my opinion, that’s less than what it used to be. And I would think it would be a good thing. But do you ever envision a point in time where database maintenance tasks might just disappear altogether?
Kevin: That is a great point. And we do see that as an emerging trend across the entire database vendors spectrum. So Oracle autonomous database, that sort of thing. So yes, I think we will get to a point in which the systems kind of commoditize all of those hours spent on doing corruption checks and suspect pages and updating statistics and all those things. Over the last few years, we’ve already seen a couple incremental improvements in the way SQL Server can do some of those preventative maintenance sorts of activities. And the other thing too that I’m thinking about as well is that, and in fact, Chrissy I know you had read SQL mag back in the 90s.
Chrissy: Love it. And the 2000s as well. And I got the compiled help CHIM. I was a big fan of that. My company paid for it and it was permanently open on my desktop. Big fan.
Kevin: So I remember writing an article back in the day about how your backups are useless unless you’ve guaranteed that they’re corruption free because SQL Server will let you back up corrupt database all day long, but it won’t let you recover one. And so the article was about, first check for corruption. And if that fails, don’t even bother backing it up. And if it does succeed, then go forward with a backup. And now we have great kind of community driven tools like Ola Hallengren scripts, which I know you utilize in your DBATools.io commandlets. And so it was one of those things where we were thinking about working smarter instead of harder all the time. And that by extension is also something that Microsoft and the other major database platform vendors are also doing. That kind of work when you connect to an Azure SQL database, as opposed to an on-premise database is kind of assumed.
Kevin: And so we’re starting to see that trickle down into the product that is installed. Tom likes to call it an earth server, as opposed to a cloud server. One in your data center close to you rather than in someone else’s cloud. So we’re going to see that part of the job commoditized. And if you think of a DBA, either you as a DBA or perhaps someone on your team as being the person who primarily pushes backups around and does the preventative maintenance and fights a fire or two as they come up, I think only one of those three fighting fires when they happen will be something that DBAs five years from now have to spend a big portion of their time on.
Chrissy: I actually think of that as great news. Even though I am a DBA, I feel like SQL Server is home, even presenting at SQL Server conferences, I’ve since branched out and it always feels so good coming back home. But that sounds like I didn’t know about that. And it sounds like great news to me as a super on-prem kind of person, because that was boring. Even though Ola made it way easier to do all of those type of maintenances, you still have to schedule your stuff and stagger all of your schedule. So I’m actually very excited to hear that that’s coming in the future. And I do want to throw out one thing. I probably may be referenced one of the articles that you wrote. We have one command called Test-DbaLastBackup, which is for me, back in the day would have been sent, and now it’s something that I use regularly, even during audits. So an auditor will come in and they’ll say, have you tested your backups? And with one command now, PowerShell via DBATools will go through a list of your databases. It’ll grab all the last full diffs and transaction logs, build that together, restore all of those under a different name. And if you want, on a different server. And then it’ll run all of the checks and then drop it and report if that is a backup that has been verified to actually work for backup and restore.
Chrissy: So that is part of automation. To me, that’s like so much more fun instead of actually building out all of this using Dynamic TSQL or anything else that you have. Now it’s just PowerShell. It’s free. It’s one line, and you take all of those processes that are super uninteresting, in my opinion. I love that PowerShell and automation in general, what I hear over and over is that people are just a lot happier and they’re having a lot more fun as they’re using these tools.
Thomas: So Chrissy, I wanted to ask, do you have usage metrics on the commandlets that are downloaded most often? I was wondering if that existed or not.
Chrissy: I should really be a lot better reading our web stuff. I have this other website that I did. I put a lot of SEO effort into in it, and it kind of burned me out on looking at stats and responding to stats. But the last time that I looked, it’s primarily the commands that really started DBATools, which is the backup, restore and all of the copy commands. It seems like it’s used heavily, primarily for migrations. And from there, I’m trying to think off the top of my head, because we do not collect the telemetry. I think that that would be a really hard cultural thing for people to adjust to. So I can just look at my web stats, and I should. The availability group commands are actually quite popular. What I tend to do is I look at the presentations that people are doing and kind of build from there what’s our most popular outside of the migration commands.
Thomas: And I know most of the DBATools, of course, they’re all maintenance related or ancillary in some way for the most part. You’ve got a lot in there. And every time I look at that, it’s mostly maintenance focused. And so the reason I ask is simply knowing which ones were the top of funnel say, so backup and restore. Really it’s about this migration of data or objects, but the backup and the restore to be at the top of mind, like you just said, being the ones most hit just from your stats, really underscores to me the direction that the industry is going. So when Microsoft comes to you and they say, we have Azure, and we’re going to do all this for you. And it’s just all going to go away for you.
Thomas: That’s why I think, just however many years ago, a lot of this stuff was still being done manually. And that’s part of the reason why I say the DBA is disappearing, because a lot of those traditional tasks, let the machines do what they’re good at. And we can let machines do the backup and the restore and the checks for all of this, and we don’t have to spend our time on that. We can spend that time on the human tasks. And that’s where I try to head with this topic and say, the data professionals are always going to be there, but the job, the things that they’ll have to do will change over time.
Kevin: Tom, I always say that the indispensable DBA is not the one who’s really good at doing those routine and rote sort of tasks. The indispensable DBA is the one who adds value to the new products and data services that the company needs. So that’s a person who is able to formulate insightful opinions from the data. They’re able to add to the development process, either by writing good SQL code or developing really good indexing schemes for the database. That’s where the value add is for a data professional. And so if you’re like, I’m the best damn backup taker in the whole 40,000 person enterprise, well, that kind of bragging rights is not going to carry you very far.
Chrissy: Well, I have actually found is over the course of time, even though I do consider myself 20 plus year DBA, I’m a developer with automation. This has freed me up to do the other things that I really enjoy doing. And so I actually take on system engineering roles. So I am actually no longer a SQL Server DBA. I never believed your article, Tom, where you’re like, no, the DBA is disappearing. And I’m like, I can find a lot of jobs and just go to indeed.com or wherever, and I didn’t believe you until they literally did.
Chrissy: It kind of became, it used to be database administration. Sometimes that still happens, but what’s happening at my current role is that I started as a security person who primarily secures machines. And as part of that, it was really wonderful that I know how to secure a SQL Server machine. But I’m enjoying that type of, I do a lot of development. I work with other products like Tenable and I get to work with like REST APIs and things like that. And I spend a lot of my time developing, but in general, what I’m finding is that my SQL Server skillset is lending to this new job that I’m in. And I do love that I have some SQL servers to take care of. It’s just in a different way.
Thomas: I think it was six, seven years ago when I started talking publicly about the shift in the role of a database administrator. I was seeing it a lot with our customers and I just had the general sense for myself, for my own data journey, so to speak. But what I did about five or six years ago is I decided to go get some data because I figured most people would respond to data and facts. And I went to the Bureau of Labor and Statistics because I know as a younger man, I remember the BLS being like, you know what, DBA job growth 10% year over year, because there was a point in time where I said, I want to be DBA because they make a bunch of money. And it’s a great job. It never changes. And there’s 10% growth rate. There’s always going to be your job here.
Thomas: Well, so I decided to go back and I looked at the BLS data, year over year. Every year they projected a 10% job growth. And that seemed weird to me because by now we should have a couple million DBAs out there and we don’t, we don’t. Every year they kept saying 10%. And then their projections are 10 years out. So if you have 20 years’ worth of data, you can have a pretty good idea of how accurate these projections have been over time. And I know the nature of how they do their study and things change and what the outlook is, but I couldn’t just blame the BLS for bad analysis because the BLS is working with what is input, and what is input underscores how data collection is hard.
Thomas: So when you go to these companies and you say, how many DBAs do you have? And now somebody goes, I don’t know, he’s kind of not really DBA. No, they really just do development. They’re not really a DBA. We used to have five. We only have three now. And then, oh, actually two, because the other one is really doing more SharePoint. And so you start to understand how they’re collecting their data and doing the analysis. And then on top of that, I say, well, there’s also this other data. You know what it tells me? DBAs on average are old, experience. DBAs have a lot of experience these days. And a lot of those, like 40% of the current DBAs today might be retired by 2030.
Thomas: So disappearing DBAs isn’t always just about the job changing. It is physically people aging out of that role. And the BLS data contains that entire story for me. And that is when I started writing and I published one to LinkedIn, including the data and trying to share that analysis to get people to understand, this job role that you have. If you’re a DBA right now is fluid. It is changing constantly. And however you identify with it.
Thomas: So in Chrissy’s case, she could have identified as somebody who really into security or development, and then she gravitates towards that particular role. It’s almost like being a DBA isn’t the end of the road for you, so to speak. It’s where you’re going to jump to the next great thing that you want to do. DBA lets you have a ton of experience working with data that you can then leverage for what you want to do next. And for me that’s been data science. For other people at security, for other people it’s different things.
Thomas: So Kevin, share some thoughts on the BLS data and the role itself.
Kevin: One thing I just wanted to throw out there, Tom, thank you, is just raw numbers. The last time I took a look at the BLS data myself, I was really surprised to see that they said, well, there’s middle 300,000 people in the USA who earn a living as a DBA. And like you said, I’d expect that to be like a seven figure kind of number. There’s got to be a million of them or more. And then I started to think about it. And I remembered a keynote at a past summit in years past, Microsoft said that they had about 3 million developers who were working hard on data related kinds of activities. And then at another point, they had said that they had 300 million users of Excel, the kind of data analyst types of people.
Kevin: And so when I sat back and started to think about those numbers, I said, okay, well maybe they’re not actually specifically accurate because some people are kind of, it’s a Venn diagram and your overlap is only so much, but it does kind of look like a proper funnel for skillsets, where the broad users of data, there’s 100 times more of them than DBA. There’s 10 times more of them than the developers. And so that kind of made a little bit of sense for me.
Kevin: And the other thing too, having worked on the Oracle side of the house. And Tom, I know you spent some time on Sybase, when you talk to people who are working on those non-Microsoft platforms, they’re older. The average DBA who’s an Oracle person is middle or late 50s. The average person who’s a SQL Server DBA, I would say is probably in their middle 40s. So it has a difference in terms of who’s going to retire when, how much pressure there is to fill these jobs. And as we could talk about for a great deal of time, I think we covered it in the last episode, actually, a TechPod. There’s so much pressure that most companies are willing to accept what they call an accidental DBA.
Chrissy: I’ve seen that across the board. We’re calling them our systems engineers. It’s a generic systems engineer with a whole bunch of different roles, including that one. First of all, I don’t know if it’s nostalgia. I hate what y’all are saying. I’ve really enjoyed being a SQL Server DBA with a magazine dedicated to me entirely. And it is hurting letting that go. But I wonder, so I am mid 40s and I’m right in the SQL Server DBA role there. What are the 35 year-olds doing currently for on-prem, or does on-prem not exist for the 35 year-olds?
Kevin: Oh, it definitely exists. At least in my role as a customer advocate and talking to a lot of customers, I do see quite a few 30 somethings, probably a quarter, 25 to 40% of the people that I talk to are that age. However, the other thing that has happened too, over the years, and we can all kind of complain about this, is that when I was working with SQL Server 7, I felt like I was an expert on every aspect of that database platform. I knew everything, top to bottom. When 2000 came out and they had the analysis services in there and integration services and reporting services, it was at that point that I started to say, okay, I’ve got to start picking and choosing what I know well and what I don’t.
Kevin: What I’ve seen is many of those younger people come in through a development. In your case, your experience, a lot of people come in as CIS admins, but with many of my customers, I see more people coming in as a developer, and they probably actually wanted to go into like building iPad and Apple apps or something like that. That’s where the cool activities were going to be. And then we’re like, oh, there’s not quite as many Apple developer jobs out there as I thought, so let’s see what else I can do. Then they go into enterprise development. And from there, they become an accidental DBA or move into the database space. That’s where the younger people I’m seeing are coming from into the business.
Chrissy: Whenever I go into an organization, it’s unexpected generally that I’m a SQL Server person. But I’ll raise my hand and I’ll be like, I know how to take care of these servers. And whenever I get to them, they are very poorly maintained. The index that the integrity checks aren’t running and the indexes. And so do you all believe that in the near future, that that’s going to get better because of the things that the database vendors are introducing? Because right now I would like to see a little better posture as far as just databases. It’s kind of breaking my heart as a SQL Server person to see.
Kevin: That’s a great point, an excellent point. In fact, I do a lot of webcasts. And basically, 80% of my webcasts are trying to get people to stop making mistakes that people have been making since 1990. It’s shocking. SQL injection was a big problem back then. It’s still a flipping big problem today. Like you said, indexing was an issue then, still a big issue today. I think that’s a really good insight Chrissy, because I don’t think that the database vendors are going into this from the perspective of, most of our customers won’t have someone to do the proper care and grooming of these database platforms. I think if they changed their mindset and said, there’s nobody who’s going to be around to take care of this. I think that would do the database world a world of good.
Chrissy: Agreed entirely. I hope that’s where it goes. What do you think, Tom? You smiling.
Thomas: I have so many opinions. I’m going to start with some data, back to the BLS data. One of the things I’ve found, for example, in one year, putting down database administrator was actually two words, data base administrator, or three words. Then very next year was just two words, database administrator. And one year you saw computer scientists and then it became computer programmers and things like that. So the job titles and how they collected it actually changes over time. So it’s really hard when you try to do analysis over a longer period of time.
Thomas: So what I did find though, trying to go through all that was the growth in the related fields. So even though the DBAs, the number of DBAs were stagnant, say at 110,000, roughly a year, despite being the projected growth, every year roughly about the same, 110,000, what you saw were an increase in systems analyst or programmers or mathematicians or data analysts, and those numbers were growing.
Thomas: Let me ask a rhetorical question. If somebody shows up to work and they spend most of their day working in Excel, would you call that person a DBA? And most of us would say, no, that’s not a DBA. That’s like anything else, however, what if that person is administering the one or two SQL instances? They’re in control of it. They don’t know anything about index rebuilds, stuff like that. They’re doing the best they can, they’ll learn as they go, but they’re really primarily the ones responsible for the backup and the recovery of the data, but they spend all day using Excel.
Thomas: Now, what do you think they identify as when the BLS wants that survey? They’re like, no, I’m a data analyst. I’m not a DBA.
Kevin: Right.
Chrissy: Great point.
Thomas: Now, will that stuff get taken care of for us? Well, I think they’re actively building that out if I think of what they want to do in autonomous database. And if I think what the folks in Redmond are doing for Azure SQL database. They’re actively trying to help that data analyst not have to spend 10% of the time worried about rebuilding stats. They want to fix that problem. So that person can focus on the data analysis because there might be 300 million users plus, 300 million users of Excel in the world right now, and yet there’s still a dearth of people that can actually analyze data properly.
Kevin: Absolutely.
Thomas: So there is a greater need in the world for people that can analyze data, then there is for database administration. And I’ll give you an example of one particular task. It is quite possible for me to roll forward a transaction log and pick out a particular transaction and restore that to a point in time. How much time should I spend on that particular task versus just a straight up restore, going back to a point in time being done with that, or having that automated done, wait bad page, let me revert, do this. All that can be done for me. I don’t need to actually do that work myself. I shouldn’t be spending my time. I’m recovering this one loss transaction. It should already be recorded somewhere else in the ether for me and taken care of if there’s a problem.
Thomas: Anyway, that’s where I come from when I think about this stuff that we’re doing and that what we should be doing and what the machines can be doing for us.
Chrissy: If any database vendors are listening, I would like to throw out that I forgot to mention log backups because I literally shrank 150 gig log for a 16 meg database just recently.
Kevin: Wow.
Thomas: And it’s still running.
Chrissy: Yeah, because it wasn’t just index and integrity checks, it was log backups as well. And so if anybody from that world is listening, some defaults perhaps to fix that issue. And that’s why I sat there saying like, thank goodness that there is somebody who used to be a DBA on staff. So even though we’re disappearing and that’s not my title, I really appreciate the experience that I have that I know exactly why, because the person was like, we’re running out of space and I’m like, yeah, you’re probably not backing up your log files and that was exactly the first thing that happened.
Thomas: Right,
Kevin: Right. They wouldn’t even know where to begin to look to fix that problem.
Chrissy: They would just schedule nightly shrinks.
Kevin: Yeah. Oh my goodness. A follow on thought to what you were saying about the BLS data, about how all these ancillary positions, data analysts and data scientists and those kinds of things. What I think a lot of enterprises are realizing now is that it’s related to the growth of web 2.0, where even a slow website is, it’s almost as if the website is down, oh, it’s taking six seconds for this page to load. Forget it. I’m not even going to use it. Executives now have enough experience being used to getting an immediate response from a Google query. And they’re thinking, okay, well, why is my database application taking two minutes to answer this question?
Kevin: At the same time, what we have though is data is a foundational skill, and there’s now a kind of a foundational realization that data is the real asset of the enterprise. There was this kind of, now it’s famous in our IT community, a headline from the Economist Magazine back in 2017 that said data is the new oil. It’s the valuable fungible asset of your organization. And so there are executives now who are starting to realize that data is extremely important. And if they have even a moderately successful team, database administration team, what they’re now realizing is, oh, we need more data analysts because we’re not using our data nearly as much as we could. And we’re not making decisions based on data like we did. So that’s why I think we’re seeing a big growth in things like those jobs, a big growth in things like Power BI, because visualizing that data is one of the best ways to make it actionable.
Kevin: So even though DBAs are under stress, there’s not enough of them. And in a lot of cases, they’re disappearing, titles are changing, enterprises themselves are becoming much more aware of the importance of data, the significance of it, it’s true value and are trying to do more with it. In fact, if I were to talk about significant trends we need to pay attention to, I would say, have you noticed that there’s now a couple dozen new data governance startups that have shown up on the scene? And that’s because all these companies are saying, oh, we got all this great data we can use. Oh no, we got three versions of this sales data or this customer data, which one do we use? And is the data of inequality.
Kevin: So again, I think we’re going to continue to see more job types spin up that didn’t exist 12 years ago about things like making sure there’s data quality, making sure the data is of high integrity, those sorts of things. So that’s a great point, Tom, that this is a foundation upon which so much else in the IT enterprise is built upon.
Thomas: I wanted to mention when we talk about the disappearing DBA, I’m not talking about with the exception of the people who are going to retire, but the role, the DBA person, that role, the data professional, that as you just said, Kevin, that remains. It shifts though. The job titles change, the roles change, the tasks change. I think right now, what we see, what I’ve seen is I see a rise in people who have the title of site reliability engineer, but what is it you do? Well, I really focus on the databases. I make sure that we can recover all of our data. But now they’re SREs. I’m sure if I look for SREs, if I go into the BLS data, I’ll see the SREs are rising over time and the DBAs remain stagnant.
Thomas: So I think that there’s a lot of things that the DBA 20 years ago would have just been taken care of, then now it’s becoming a little more specialized as companies recognize the value of the data. The example I give about rolling forward that log transaction, for me, I have an RTO and RPO. How much time can I spend on trying to get that one thing versus look, I’ll put you back to where we were 10 minutes ago. Sounds great. And I can do that in five minutes, or I can look for the one thing and it can take me an hour. You have these trade-offs, cost benefit and risk of everything here. When I talk about disappearing, really rings alarm bells for some people, oh my God, you think what I’m doing now isn’t good. I’m not going to have a job and tell you, no, no, you can have a different job. You’re still going to be a data professional, but you’re not going to worry about those backups and restores and index rebuilds and all of that stuff. You’re going to use DBATools and it’s going to take care of everything for you. All of this exists for you that you don’t have to knock yourself out unless you enjoy that sort of thing, I guess.
Thomas: I also think cloud service providers accelerate this change a little bit. I think the cloud service providers are doing good work in this area to automate away a lot of these things. And we know from that query report that we did, a lot of the respondents say they’re shifting to cloud database as a service regardless of platform. It doesn’t matter to them, whoever they’re using, that platform is available somewhere. And they’re going to shift to using that and they’re going to build cloud-native apps. And these are the skills and the skillsets that are coming that I think may be the next wave.
Thomas: So final thoughts, Chrissy, anything?
Chrissy: Well, I did want to highlight one thing that it’s funny you had mentioned going back to that singular transaction. And I think that whenever we talk about the disappearing DBA, we’re talking about probably a lack of depth. We have somebody who can do just the superficial stuff. And that includes me. For me, I got really into PowerShell. I got really into automation. I got really into SharePoint. I love SharePoint. I love REST APIs. I love all of that. And I was actually writing a, I’m writing a book along with three other co-authors called DBATools in a month of lunches. And I was assigned the restore chapter. And in order to do that, I had to go through the restore DBA backup commands. And that is, y’all will probably be horrified. But think of me kind of as a generalist with a kind of a specialization in SQL Server, I didn’t know that you could name transactions. So you have a DBA who has to know that you can name a transaction and then you also have the developer who has to know that they need to name their transaction to roll back to.
Chrissy: And so I do think that even though we’re going to this way that I find is a little bit more superficial and exciting because I do like to do generalization stuff, that is kind of painful in a way that me as a DBA, I didn’t know, to tell the developer that I don’t work with because I just work with SharePoint. I don’t know if they’re naming their transaction, they probably aren’t. I think that that’s something that would have probably occurred heavily in 2008, that you had a developer that knew about name transactions, that you had a DBA that knew that they could restore to that transaction point that you’re talking about. So I think that we are suffering a bit of a loss. Maybe that’s not nostalgia, even though it is a place that I appreciate going, I think that we’re both gaining and losing at the same time.
Kevin: Very dynamic. I want to play off of something that Chrissy just said in terms of in-depth, knowledge in-depth. And so this is something that I have always advocated DBA spend their time doing. And the way I like to illustrate the need for being in-depth is to compare DBA to several of these other kinds of very common engineering roles or admin roles in the enterprise. So if you talk to a network admin, for them, TCP/IP packets are just TCP/IP packets. It doesn’t matter if you work for the country’s biggest parking lot company or the country’s biggest dialysis company. It’s just TCP/IP packets. If you’re an exchange admin, they’re just emails. Doesn’t matter what kind of data, what kind of company.
Kevin: But if you work as a DBA at the country’s largest parking lot management company, everything is different for you than the DBA of the country’s largest dialysis company. Because the DBA at the medical company is responsible for data at which people’s lives depend. People will die if you mess this stuff up. And so that means your RTOs and RPOs, your backup recovery, all of that stuff is now suddenly very, very different for you than it is for the DBA down the street.
Kevin: And so, if you, as a DBA, and this is something else I find happens really frequently, is that the DBA who is used to just pushing backups and doing restores and those kinds of things, if you don’t really know very much about your business, you are not the DBA that that business needs. They need for you to maybe not be a domain expert on everything that company does, but you need to know at least the basics of what satisfies a customer. Many times you’ll hear people talk about data tells a story, so you need to know what those stories are so that you can properly emphasize your time spent and your prioritization. It’s going to impact the enterprise architecture for your applications in your database. It’s going to have enormous impact and kind of cascading effects throughout the entire system.
Kevin: And so, when I had that realization, some of the things I started to do back in the 90s was to write jobs that would keep track of how busy we were, because the first thing I realized is I don’t even know when these systems are at peak busyness. When do I have the most customers on? And one of the things that, that influenced heavily once I learned that was, okay, it turns out that we are actually really busy on Saturdays. For some reason, our people will work a lot of hours on Saturday. So let’s not do major upgrades on Saturdays, let’s do them on Friday night when all those people are out drinking and not on the database systems, so that when they come in on Saturday, it’ll be done.
Kevin: So knowing what your business is and the story that your data tells is extremely important. And that is an aspect of the kind of depth you were talking about, Chrissy, that true mission critical DBAs need to be able to know.
Chrissy: I love that. And I just want to say that what I heard you say is that while the role may be disappearing, it absolutely is still essential and will never actually disappear because you will need SQL Server DBAs at hospitals and other life affirming organizations and things like that. So that was actually very encouraging for me who really does want this role to stay around forever.
Kevin: It’s not going to completely disappear because, well, for example, and again this is the 90s, so it’s very different today, but at that time we had one database we believed, we evaluated it to earn the company about $140,000 an hour. And that’s small change compared to what big companies like Wayfair or Amazon, those kinds of companies generate the revenue they generate, but there will always be DBAs for those systems where it makes more money than a DBA costs, particularly if it makes more money in an hour than a DBA costs for a year. It is changing, but it does have longevity there.
Thomas: I want to thank you both for taking the time today to join us here in TechPod. Chrissy, thank you so much.
Chrissy: Thank you both so much for inviting me. I really had a wonderful time and I feel like I’m with family.
Thomas: And Kevin, same to you, thank you so much for joining me here again. This was fun three-part series we did.
Kevin: Thanks Tom.
Thomas: The DBA role is disappearing, but it’s not going away. It’s evolving as it always has. Database vendors are automating a way the tasks that machines should handle well, and that frees up the humans who spend time on tasks that humans do better than machines for now. If you enjoyed SolarWinds TechPod, we’d love for you to follow, rate and review the podcast. Thank you for listening. And until next time, I’m Thomas LaRock.
