Announcer: This episode of TechPod is brought to you by SolarWinds Upgrade Resource Center. For valuable upgrade resources like, Orion Upgrade Advisor, upgrade planning checklist, educational resources, and more. Please visit support.solarwinds.com/upgrade-resource-center.
Leon: IT folks have a hot and cold relationship with the idea of upgrades. If it’s our phone or a game console, most of us want to be first in line. But when it comes to our business environments, we’re pretty lukewarm. I know plenty have experienced IT practitioners who are almost proud of how long some of their systems have gone without a patch or an update. It’s like the whole thing about not rebooting your Novell Server for months at a time, I don’t know. To be blunt about it, this is one of the dumbest things that we could possibly do as tech pros. So with me today to talk about all of this are, Chrystal Taylor.
Chrystal: Hello.
Leon: And Mike Driskell.
Mike: Hey.
Leon: Hey. Okay. So, since the topic of this episode is upgrades, I want to do introductions like we normally do. But usually, I do some shameless self-promotion where we talk about our Twitter handle and stuff like that. Instead, I want you each to introduce yourself by talking about the upgrade system you like the most. And I can see the confused looks on your faces. So, I’ll go first and hopefully it’ll be a little bit clearer when we go through there. Okay. So, my name again is Leon Adato and I’m the Head Geek at SolarWinds. Yes, that really is my job title. And it’s no secret that I am a Linux fan boy. So, one of the reasons that I happen to love Linux so much is the upgrade process.
Leon: With other operating systems, Windows, when you try to do upgrades, it could be multiple reboots, you could do an upgrade and then another upgrade and then another upgrade. And then by the way, this other thing, because it’s piece by piece and it is sequential. Whereas with Linux, when you do an upgrade, you’re getting all the bits at one time. And usually, they work even without a reboot. It’s really no fuss, no muss. So, I really loved that. That’s my favorite upgrade system. So hopefully the introduction thing is clear. Now, Chrystal, how about you go next?
Chrystal: So, my name is Chrystal Taylor. I am also Head Geek here in SolarWinds. And previous to that, I’ve been about 10 years into being a SolarWinds consultant and administrator. That was fun. And my favorite upgrade thing lately, to probably very few people’s surprise, is going to be about gaming. And one of the things that I am super impressed with this time around is that there’s not really any need to upgrade to a series X if you have a gaming PC, they have done wonders with the upgrades from console to PC. And so, you could get the same value out of a console as you would a PC. But since I already have a PC and I can piecemeal and upgrade my PC and parts, which is also wonderful. And I did a PC upgrade earlier this year, that was very exciting. You can do all of that and you still get to get all your X-Box stuff. And I’m still waiting on my PS5. One day I’ll be able to buy one, but not today.
Leon: Right? Today’s not that day. Okay. Great. All right. And wrapping up the intros the voice on this episode, who has any actual credibility since he’s not a Head Geek, Mike Driskell, welcome to TechPod.
Mike: Thank you. And I actually would probably want to sit here and talk to Chrystal more about gaming upgrades as I sit here next to my PC, waiting to buy a new video card. But I’ll do the shameless plug and say, we probably should talk about SolarWinds upgrades.
Leon: All right. So, what do you like about the SolarWinds upgrade? Like why is it one of your favorites?
Mike: I like how far it’s come to be honest. So, I’ve done several upgrades and the most recent one that I did was just two days ago and literally went to the page, clicked the button, and we’re all there and done and having access to whatever the latest and greatest feature might be or fixing a previous bug. I’m never one to fall too far behind on code.
Leon: Got it. Okay, great. And we’ll probably get into that a little bit more as the episode progresses.
Chrystal: Yeah, absolutely. And I do want to say we might not all be Head Geeks, but we are all THWACK MVPs.
Mike: Well, former MVPs, since we joined SolarWinds.
Leon: Right, we had to give it up. Although I prefer to think of us as MVP Ameritas. I really don’t ever want to get rid of my MVP status, so I’ll hang onto it however I’m able to do so. Okay. So, with introductions out of the way, where I want to start is with, I think what every IT person listening to this episode is probably thinking, which is the honest-to-God truth is that upgrades are a pain in the ass. They really are. They’re just difficult. So, before we talk about how they’re wonderful or why they’re necessary, I think it’s important to be transparent about it and say, why do IT people put off doing upgrades?
Leon: And I’ll start off with maybe one of the biggest ones, they’re risky. Upgrades don’t always go well. And the worst part about an upgrade that fails is that not only do you not have an upgrade like =that didn’t work. But now you have possibly an unusable system and you have all this extra work to do to put it back just to the way it was before you even started. And it makes you think, why did I bother? Why didn’t I just leave it the way it was? Leave well enough alone. So, I think that’s one of the things that a lot of IT folks have been burned once or twice. And that’s why they’re there just cold to the whole thing. What are some other reasons?
Chrystal: They can be time-consuming. And oftentimes you have to do it in the middle of the night. I once was on an upgrade that took three days with engineer’s swapping out and that was the roughest upgrade I’ve ever been on. But it can happen.
Leon: Was that like Assassin’s Creed? Or Halo or? Okay, what else? What are the other reasons why you would not want to?
Mike: Change control can definitely be a thorn. And many of our customers sides having to… I personally remember having to get basically a letter from a dead relative in order to be able to conduct any form of change. And those can be a pain in themselves.
Chrystal: Yeah. Even getting the time to present to the CAB is problematic in some companies.
Leon: Right. And presenting a monitoring change to people who have no concept what monitoring is, but are just nervous about the whole thing. I used to have to change controls in a bank. And there is no more risk averse environment than banking. I’m sorry, but I’ve worked in healthcare, I’ve worked in factory environments, it’s just really hard. So, I’m also going to say that a lot of times IT folks don’t see a compelling reason to do an upgrade. Let’s face it bugs fixes isn’t something that really has me super excited to get up. As you said, Chrystal, at 1:00 AM in the morning on Sunday morning, like, Woohoo. I get to do this. That is not how I feel about bug fixes. So, with those things on the table, it’s important for us to go back and talk about how they’re not as big a problem as we think they are a lot of times.
Leon: So, let’s start off with the whole concept of expense, that they’re expensive. It takes time and resources, and it’s really hard. And I’m going to start off with the there’s this old phrase. “If you think education is expensive, try ignorance.” Which I turned around and when I was trying to talk to people about monitoring it all. I would tell people if you think that a monitoring solution is expensive, wait until you have to pay for the first system failure because that’s usually an order of magnitude or two greater. In fact, that was the whole point of my keynote at THWACKcamp last year was that, a $5,000 monitoring system they said no to, but then the failure cost a million dollars. So, like you saved $5,000, sure you did. And I think the same thing here that if you think it’s a lot of work to upgrade, just try dealing with a system that has completely collapsed on itself because it hasn’t been patched in years.
Chrystal: Yeah, absolutely. And I’m sure Mike can attest to this as well. Doing hundreds of upgrades over the years, there are some that are not even worth doing. Once you’ve reached a certain delta between current version and the version that you’re on. The amount of steps it’s going to take, if you’re going to have to go through two more OS’s, probably just start over. You’re going to be better off. The pain that you would go through to upgrade a system that’s let’s say is on a 2003 server, been there, done that, horrible don’t want to do it again. When you have to move 2003 to 2016, just start over. There are times that you have to do that and really making that assessment is important early on before you figure out it’s halfway through that it’s gone bonkers because you’re doing that step to step, to step to this server and that server in this server, especially with SQL involved in having to do those SQL upgrades as well and transitions as well. And adds a layer of complexity. So sometimes the answer is start over. Port over what you can database wise and start over.
Leon: I call that the Newk It From Orbit option. Again when you talk about the expense, the cost of doing it, are there any things that you would say to IT folks who were like, “No, it’s too expensive.” And you as an experienced IT person. A seasoned professional. We don’t say old around here. So, like, how would you argue that?
Mike: Yeah. It’s like the whole, like you said bug fixes aren’t compelling. I didn’t want to spend four hours at the dealership either to get a safety recall from my airbag fixed. But I also don’t want that blowing up in my face when I was just driving down the road without even being in an accident, which is what the recall was. So, like Chrystal said, you don’t necessarily want be doing the stepped upgrade and getting many versions behind where it is going to be more painful. But you also don’t necessarily need to do every single one. Maybe it’s an upgrade once a year to keep you pretty much in line with, so you’re only plus or minus one version back of code.
Leon: That’s actually really smart is just knowing that, unless there’s something compelling that you need to upgrade for but then when you do need to upgrade, let’s say there’s that compelling feature that comes out or whatever it is then it’s not that big a lift to do, I like that. I also want to say that this is not a surprise for software vendors, whether we’re talking about operating system people or software vendors like SolarWinds or whatever it is that we know upgrades are a pain. We know they take too long. And once again, using Linux as an example. I run on a Debian-based system. I run on Ubuntu and it’s like sudo apt update, sudo apt upgrade, the end. And really it takes like 45 seconds and my upgrade is done. And the same thing I know for SolarWinds ones, Mike, you were talking about it before. It’s like, single-click, walk away, come back half a cup of coffee later and it’s finished.
Chrystal: Yeah. Shameless plug here earlier this year when we did the THWACKcamp. We did a live upgrade within 20 minutes with additional polling engines and several modules. So, people were doing it during THWACKcamp. They were watching glad THWACKcamp and went and upgraded their Orion systems because they didn’t realize how much it had changed. And having been around SolarWinds software for 10 years. I definitely want to attest to the amazing work that our teams have been doing to improve that upgrade process. It is night and day from when I started working with SolarWinds, better and faster.
Leon: We’re all nodding. You can’t see us, but we’ve all been doing using SolarWinds as users and then as consultants and stuff for a while. And we’re all thinking back a few years when it wasn’t hard compared to other software. I’ve done UNIX system upgrades and things like that, where every piece required an engineering degree. It wasn’t like that, but it was a little shaky every time like you just weren’t sure. And it has become so smooth and the fact that the installer tells you before you even start, “Hey, you need to install this version, or you need to make sure you back this thing up, or would you like me to do it for you? And it’s delightful.
Mike: And now you can actually even do that pre-planning week or two before you were going to do the upgrade. You can go ahead and have it checked to see what outliers it might find so you can prep for it.
Leon: Right. And the pre-staging also. I don’t mean for this TechPod to become a commercial for Orion or for SolarWinds, but you can actually, I know that half the time that I was upgrading was spent just moving bits from one machine to another machine. And that wasn’t, that doesn’t require a change control. Moving bits typically doesn’t require change control. So, the ability to pre-stage your bits and put them on to all the systems where they need to be, and then simply pull the trigger on the upgrade itself is a huge time saver. It’s really big. Leon: Okay. So, I think that takes us to one of the pieces that makes me the most frustrated, which is when somebody says, “Well, I don’t need to upgrade because I don’t need any of the new features. And I’m just going to go full Lex Luther in the Superman movie. And just in the microphone, I’m going to shout, but I won’t because our sound engineers will hate me, but I’m just going to shout Wrong! Wrong, that’s not true! You do need, of course you need the features, right?
Mike: Yeah. Every week I talked to a handful of customers and I’ll hear them say they have this use case they need to solve. And a lot of times it’s stuff that we already do. Like when we came out with the Network Insight, for example for Palo Alto or ASA or Nexus devices. I’ll get people saying they want to be able to monitor their VPN status. Or maybe it’s something that we do with Active Directory for SAM. But a lot of times the features already there, and they aren’t aware of it because they’re three or four revs behind.
Leon: Yeah. I see this when I’m at conventions. Back in the day we’re recording this…
Mike: When we could go places?
Leon: Yeah, we’re recording this in December of 2020, still in the middle of the nightmare hellscape pandemic. But we remember fondly the days in which people got to, traveled on airplanes and got together in large convention halls. And back in those days, people would come to the booth routinely and they’d say, “So what’s new?” And I showed them some things like, “Oh my gosh, is this new?” And I would say, “Well, if you’re talking about like geological time, sure it’s new, everything’s new. But it’s been out for like two and a half years.” And like, “I didn’t even know that.” And because they’re not doing the upgrades, or even when they do the upgrades, they don’t realize what capabilities are added to it. I love that, like “I need to do this thing.” “You have it.” “What, I have it?”
Chrystal: And honestly, I’d like to throw out there, consider that while you may not be able to use a particular feature, somebody else might. So especially when it comes to monitoring, you get new things added. And maybe right now your network team isn’t using the same monitoring system as you. But if you could take it all under one umbrella and it can get them the things that they need in a faster, easier way than what they’re using. It might be easy to consolidate and save your business money.
Mike: Yeah. Just to touch on that. Again, like I was saying with having the feature already there. A lot of times we’re on calls with customers and there might be the manager or the director on the phone and they’re actively looking for a tool. And they’re considering buying something and then when I tell them that they already have it, and they don’t have to spend another dollar. A, that’s great, but then B, they’re also thinking to themselves, “Why are we four revs behind? And that’s where unfortunately, we hang up the phone. And I’m not the one having that conversation to explain that to their boss, but it’s reality.
Leon: Right. And I’ll close up this section by saying that I was with a client who was evaluating multiple pieces of software. And every time they said, “Well, we’re looking at software X”, I’m like, “Well, the software you have does that.” “No, it doesn’t.” Look that’s because you’re four versions behind. It was added three versions ago? “Well, but we’re looking for software that does Y.” “Well, the software you have does that.” Again, what they finally realized was Mike what you were saying. But also, they said, it’s actually not fair for us to do an RFP for a new software. If the software we already own isn’t on the latest version. So, we don’t even… It suddenly was this light bulb moment. And I think that’s another reason why saying, “I don’t need those features.” Maybe you don’t think you do, but if you’re ever in a situation where you’re feeling like, “Oh, I’ve got to go shop around for some new software to do XYZ.”
Leon: If your current software isn’t current, how can you even possibly know that? So, I want to pivot over to the whole bug fixes aren’t compelling. Mike, I really like that example of safety recalls on cars. Like those are also not like, well, that’s not exciting. It’s not as exciting as buying a new car.
Leon: I said, we were talking about it before we started recording. It’s like having a Barry White Whitney Houston duet, just sexy singing in your ear, “There’s nothing to update here.” It’s like, it makes me so happy.
Chrystal: Actually, I’m writing a blog about this right now which is essentially, this is about the conversation. The conversation that you should be having before you’re actually do the software upgrade, the part where you’re doing all of your planning, where you’re considering the risks, where you’re considering bugs, where you’re considering costs, impact, all of that stuff, your recovery plan, because as we all know, something can go sideways. So, you need to have all of that stuff in place ahead of time and do an assessment to figure out, is it going to be more cost effective for us to just start over? Are we that far behind that we need to just start over? This is the conversation. And all of the information that you uncover, while you’re doing this, while you’re having this conversation, while you’re doing these assessments, all of that is fodder for the conversation with the business to say, “Hey, we really need to do this change right now,” all of that’s fodder, because they’re going to ask you these questions. And if you already have answers for them, that it makes it all the easier for you to convince them that now is the time, now is when we should be doing this.
Chrystal: Okay. All of that is part of the conversation that you should be having ahead of time. And that includes new features of course, but that also includes all of that other stuff. How much time is it going to take me to do this? If I wait six months, how much additional time is it going to take to do this? How much additional materials, what are the resource requirements? Because as software adds new features and becomes more robust, obviously the resource requirements changed as well. So, all of that is things that you can be planning ahead of time. In my previous life, I had a template that I built that forced our engineers to go out and do all of this assessment ahead of time to ensure that we had the smoothest upgrade possible. Because honestly, something can still go wrong even if you do all the prep work, something can still go wrong. Murphy’s Law is still there, there’s still a chaos, there’s still random things in your environment, something can go wrong.
Chrystal: But if you can minimize the risk of something going wrong or minimize the risk that not doing this upgrade or not moving forward on your software is going to cost more money. If you could get a feature by upgrading the software, you already own, to go back to your point earlier. If you can upgrade and get a feature that you already own, isn’t it going to cost you less than buying a whole other software to do that one thing or two things that you can do. And makes sure you’re doing those assessments ahead of time, because honestly it can save the company money, it will save you time in the long run. And the company always likes to hear that they can be saving money somewhere.
Leon: Well. And also, I think that the point that you made about, it will take you longer later. Like, “Hey boss, this upgrade is going to cost is going to cost us two hours of our time.” “Oh, we just don’t have two hours, we’re going to put it off until later.” “All right. It’ll take four later.” “What, why?” Because you have more to do.
Chrystal: The longer you put it off, the longer it takes. Remember earlier when I talked about that three-day upgrade horror story okay, I’m just saying the longer that you wait, the longer you put it off, the more changes are going to be involved, the more work is going to be involved in the longer it takes and the more risks that are introduced as well.
Mike: Or when your operating system goes unsupported. And then the software you’re using for monitoring isn’t going to work on that going forward with new versions. But it all comes down to a planning a lot of times, it’s that whole measure twice cut once. And as somebody who bought their first set of power tools this past summer and built a desk, I can attest that I might want to measure three times in the future. But you definitely have to plan it out to try to mitigate that risk. Although, I liked hearing the story about the guys on THWACKcamp upgrading live during the sessions. Hopefully they submitted change requests for those.
Chrystal: It was amazing.
Leon: It was heartwarming. They were like, “Oh my gosh, this is easy enough I can do it now. I just asked my boss they said I can do it.” Like I was showing them this. And environments are different sizes and some people have different change controls. But it was really fun to hear. And we’re like, “Tell us how it’s going.” There was a few of us, Chrystal and I especially we’re on chat and we’re like biting our nails. Like, “Please tell me how it goes,” because what we don’t want to hear is like, “Yeah, I started it and my whole system’s down now, what do I do?” So, they were like, “No, this is great.” It was really kind of cool.
Leon: And that’s our view from a SolarWinds standpoint, but other software has a lot of the same things. I also, in the idea of planning, I just want to reinforce that if you’re monitoring is watching mission critical or business critical systems, then by definition, you’re monitoring is business critical also. And therefore, all the things you do for your business-critical systems, like having a lab, having an environment that you test things in, you should be doing for monitoring also. And we have lots of conversations about, how to build a lab and how to do it for not much cost or even free and things like that.
Leon: But regardless of whether we’re talking about a monitoring solution or your operating system or whatever, you should have a system that you can test the upgrade on just so that to expect that you know is going to ask for this information, “Oh my gosh, it’s asking me for the login to the database, I didn’t have that in handy right now,” or whatever it is like, oh, that’s what the upgrades going to look like. Any good it person is going to have done a dry run of it, during the day without the change control, because it’s a test system all that. But they know what they’re getting into so that when showtime comes, they have it. I have to reinforce that having a lab system is imperative in any reasonable IT context. I don’t know if there’s anything to add to that.
Chrystal: No, I mean, just that we acknowledged that it’s not always possible for you to have your own lab environment to do these dry runs. But, where possible, it is nice to do a dry run ahead of time especially when you can get your software licenses, and you can get evals of software licenses for most software for 15 days or whatever. So even if it’s not SolarWinds, whatever software you happen to be upgrading consider just throwing a VM up somewhere and building it in an eval mode and then doing the upgrade in that way, if you can to save on costs.
Mike: I see a lot of customers do that with just evals. Maybe they’re not big enough to justify a full on always-on lab environment. So, they’ll spin up a VM with the trial software, test out the upgrade, and then plan for their production upgrade.
Chrystal: Yeah. And just one last addition, if it is a software like SolarWinds that has a robust database, consider replicating the database to that test system when you do that trial run. It’ll make all the difference in the world to your final run.
Leon: Yeah. Upgrading real data like stuff yeah, absolutely. Okay. So, this has been a really good conversation. I want to finish up with the lightning round, I love this part. Any last thoughts? Any final things from anything we’ve talked about that you want to share? Chrystal, you get first dibs.
Chrystal: First dibs. So, if it’s your first time upgrading and you’re not sure or anything, there’s lots of forums out there SolarWinds has great, the THWACK forum is great, I’ll always promote it, we were all THWACK MVPs so we’re all for it. There’s always forums out there for most software that you can go talk to other people who’ve been through what you’re going to go through. Reach out to the communities because communities are great and most people are good when in those places especially THWACK. So, if you ever need advice or you’re looking for somebody’s experience. It’s probably out there somewhere. Go look it up.
Leon: Yeah. THWACK is always a Reddit for something, there’s usually a Stack overflow for something you can find an environment and just putting out there, “Hey, I’m doing the upgrade from X to Y. The other thing is that when you say things like that, the engineers I’m looking at Mike, while I say this. The engineer like especially if your leading edge of upgrades. Engineers are always interested to hear what those experiences are like. And so, you may end up getting a little bit of extra attention or extra love from somebody saying, “Really, you’re going from what, to what? I’d like to hear about that. Tell me how that goes?”
Mike: Yeah. I think, one other thing I would point out is, we talked about bug fixes we talked about compelling features. But something to also consider is doing the upgrade on a regular basis, whether that be six months or yearly. So, you don’t fall back behind, maybe there isn’t a bug fix that impacts you or maybe there isn’t a feature you need. But if you don’t want to end up being four or five, six versions way and have to, as Leon put it, nuke it from orbit, maybe look at planning for an upgrade window once a year.
Leon: Yeah. I like it. I really like that. And yeah, it’s good IT hygiene, right? You brush your teeth, you wash behind your ears, you patch your systems.
Mike: And your back up your data.
Leon: And you back up your day.
Mike: Because every IT person backs up all their data every time.
Leon: You should be. One of our other Head Geeks, Tom LaRock is famous for saying that, “Improving performance will get you a promotion, but being able to restore, we’ll keep your job.”
Mike: That is true advice.
Leon: Yep. Okay. Chrystal, Mike, thank you so much for taking time to talk about upgrades today. Really appreciate you making some time for this episode.
Mike: Absolutely. Thank you.
Chrystal: Yeah. Thanks for having us.
Leon: The shameless self-promotion part. If people want to find out more about you and the work that you’re doing, or hear more of what you have to say, where can they find you out on interwebs?
Mike: The easiest place to actually find me is either in an online game or on THWACK. The handles just @MDriskell.
Chrystal: I also can be found in games frequently, but I am pretty active on Twitter @ChrystalT87. And I am also on THWACK of course, @ChrystalT, and on Instagram and I think that one is @geek_chrystalt. But don’t hold me to that one.
Leon: Just look for her, you will find her. I’ll round it out. You can find me on the Twitters, which I say like that because it horrifies my children every. @leonadato. And you can also find me on THWACKTHWACK @AdatoLE, yeah THWACK.com there. For everyone at SolarWinds, I want to thank you our listeners for carving time out of your busy day to listen to this episode. Thank you and see you soon on another episode of TechPod.
