In this episode, Head Geeks Patrick Hubbard and Leon Adato, along with Product Manager (and Kung Fu master and support rock star) Kate Asaff cover the best way to plan and prepare for a SolarWinds product upgrade, as well as provide hints on how to make your upgrade experience stress-free, simple, and FAST!
Episode Transcript
Hi, I'm Leon Adato. I'm Kate Asaff. And I'm Patrick Hubbard, and we know what you want. You want to see an upgrade from start to finish, with all the works and all the time that it's going to take to do that. And we don't want to run out of show. So we're going to go ahead and just start that off right now. So Kate, kick us off. Okay. No, this is going to be really hard. We're going to record the whole thing. You start the installer like this. Okay, so this is going to be really complicated. So just watch it carefully, Patrick. Okay? How long does this take? Depends on the environment. [Whimsical music] I never understood why Night Crawler had to smell at the same time. Like I get the teleporting, I get the fact that he looks sort of demonic and all that stuff, but what's the scent? Like, why does he have to smell like a ****? [Whimsical music] What color is it? It's red. It's red. What color should we do on this other side? [Upbeat music] And we are done. Okay, but you didn't really do anything. I mean, you started the installer, you clicked okay, and then it just kind of finished. So what are we going to do with the rest of the 25 minutes of this episode? Geeks, just start the episode. Okay, so once again, welcome to SolarWinds Lab. With us today is Kate Asaff, former application engineer, Kung fu master, and product owner. Welcome. Thank you. We just showed you what a typical NPM 12 upgrade looks like, but it's not always sunshine and puppy dog. Or sandwiches and hair curlers. Sometimes things do go wrong. That's right. Now for the vast majority of you, you have little trouble with installs. You've been doing them for years; it's not an issue. But if you have a really complicated environment, or maybe lots and lots of modules, you may occasionally run into issues. So with a little bit of preparation, you can actually avoid most of the gotchas that do come up from time to time, and that's what we're going to show you how to do on this episode. I think it's important that we go over this in an organized way because honestly, that's the key. I think the number one reason why people end up on a support call at 2 a.m., is because they see how simple the upgrade is and they think that means they don't have to use the same amount of discipline they would for an upgrade of Exchange or something. Right, as monitoring systems go, SolarWinds is crazy easy to manage, but it is still enterprise-grade software. That's true, so how about this: first, we're going to talk about the things that you need to do before you start. What's the French word for that? Mise en place. Mise en place for Orion. And then we're going to talk about how to assess the environment. Make sure that you have all the resources that you need. How to plan out the actual upgrade, and then we'll talk about the tools and processes during the upgrade itself. And then last, we'll finally cover a couple of other advanced scenarios. Right. And meanwhile, if you have questions, you can ask us in the chat window that you should see over there. Now, if you don't see a chat window it means you aren't watching us live. To do that, head over to lab.solarwinds.com and sign of reminders or to catch up on past episodes, or leave us a comment about what you'd like to see us cover in future shows. Now that's the organized approach I'm talking about. So before we begin, I think it's really important to talk about some of the— we talked about gotchas. The gotchas that people run into because they don't realize, although it's in our admin guide, it's in our release notes, but it's one of those things that if you're not reading carefully, the fine print, and I'll start it off. You have to do the install as local administrator, not domain administrator, not Bob who has admin privileges. You actually need to do your installs as the local administrator. And that's super important, because permissions issues are a huge sticking point for a lot of upgrades. So when you say permissions, what does that really feed into? Well, first of all, like you said, administrator account, because nobody has the same rights as the local administrator does. Domain admins are still bound by GPOs and other scary little things that can lock you out of where you need to get. Or worse, lock out a single file. For example, when the installer's laying the files down, there might be one area that's protected differently by GPO or a combination of different policies, that's really hard to come up with beforehand. Which leads me to folder permissions, which was the next thing I was going to say is, you never know. One folder has weird permissions that the other one doesn't. We try to lay down some bits there. So if I've got an installation that is maybe three or four versions old and I know for a fact that I was installing as me sometimes, and then I had a contractor or somebody else who installed, and I have no idea—is there a way to adjust that? Or am I talking about building a new machine? So we have a permissions checker utility you can run that will just go through and tell you every file or folder that we need to access and whether or not it has the correct permissions. But it only works on local administrator accounts. So you want to make sure that's how you're logged in before you run that. The other thing is, it's not a bad idea to just kick off the installer. Click run as administrator. Make sure that it's not going to run into any problems. Okay, so that's two ways. And I was just going to say the other thing that tends to cause file permission issues, is maybe you've installed other security software on the system to be able to harden it. That one, especially if you have regulatory compliance issues, can do that as well. So in that case, when you're running as local administrator, you're going to be able to confidently lay down the bits. Right, although, I'm going to say something, and I want everyone just to take a deep breath and don't panic: you need to turn off any keyloggers, you need to turn off any of your defender software, you need to turn off antivirus. Not on the server, not forever, just for the install. Just suspend it. That is a big one, antivirus especially. Right, and in the old days, also the defraggers sometimes got really aggressive about things. What's a defragger of which you're speaking? Sorry, okay yeah. It's one of those things that people who don't run Linux sometimes have to deal with. So anyway… I'm on a SAN and virtualize, buddy, I don't have any of those problems. Okay. Well, good. So again, I'm not saying forever and I'm not saying that's even remotely a best practice, but during an install or an upgrade, you want to turn those things off, just so they're not in the way. Right, you may successfully cruise right through it. You may have done it a million times before. But again, this is all about doing everything that you can to be prepared so that you don't have an issue later on. And the other thing about it is that I have seen in the past: it's not the install that has a problem, but it is the unintended downstream consequences that, you know, a week later, two weeks later, something just isn't quite working right. And it may work fine on the primary poller but it doesn't work on the additional poller, or whatever it is. So this is just best practices across the board. Now we did talk about this. We talked about this in two different episodes. Episode number 36, which was the support spectacular, where we went over some of this. And also on episode 38, which was the December episode last year, getting your environment squeaky clean for 2016. And lab 44, we showed the new installer that is a part of NPM 12 and beyond, and we actually talked to the product management team about that as well. Right, so those are all places that you can get some additional information. So moving on, because I think we sort of hit that. And they want more talking. Right, absolutely. No, they want to see us get our hands on stuff. So how to link the SolarWinds [stammering] your installation to the customer portal? And the reason why this is useful is it's not certainly critical for an upgrade, but it is really useful to do because you get better insight into the modules that you own. You get the notifications, the message about patches or upgrades that affect you specifically. So, that's a really useful thing to have done, actually, not just do but have done before you actually get to an upgrade. Just makes it super easy for you, too, when you're ready to do that upgrade, to get all your bits if you have the link right there and readily accessible for you. Okay, so what you're talking about there is the ‘integrate with your customer portal.’ So go ahead and click that. And I'm actually going to just jump in here. We'll just do-si-do. And you're logging on here with your email-based individual log on. Yes. I know that there are some people who still use their SWID. Don't. Don't do that. And the main reason to do that, we've talked about on the show probably four or five times already, but the main reason for setting up an individual user account here is, well, first of all, you get to have a password that we don't know. So it's better security for you. The other thing is that you can have multiple user accounts on your SWID with differentiated roles. So probably, you have a team of people that are all using the products. That way, you can give them different permissions to either be able to open help desk tickets, for example, but maybe not download software or download software--track who's got access to what. But the other big benefit is things like access to unlimited training in the customer success center. So the live training classes that are offered through that program, you need to have an individual log on into the portal to be able to access those. So getting your team advanced training, and that sort of thing, and having access to those features of the portal, all made possible by having individual logons. And the customer success center's something we're going to take a closer look at here in just a couple minutes. So what you can see on the side here is that my account is now linked with this installation and I can see which of my licenses are active, which are expiring. And Kate, as you mentioned, you can download your bits if you need to, and things like that. As long as we're talking about linking— I know it's again, not precisely upgrade-specific, but I do want to take just a quick second and talk about linking your THWACK account to your customer portal ID. You can see where the theme is here, we're all coming into a single point of unification. So here on the THWACK page—I'm actually on my profile page— if you go to actions. Go ahead and just click there and there. And go to link, what was it? Customer portal account. Link to customer portal and then it would open it up, and you put in your username and password right there, and that would link it. Just let me go ahead and do that again. Here we go. I'm just trying to figure out how you got it unlinked after we ran through this once. There is an option to do that. You actually can. It's under actions, also. And why would we do that? I think that if you are changing roles, or if you are moving from one. As one opportunity for the next, as many of you do, and you tend to stick with SolarWinds products from opt to opt. This way, you can break that association, maintain all of your points balance. It doesn't become property of the SWID that you attach it to it remains associated with you. So it's a temporary binding that just makes everything easy. Which is super important, because we all know you're saving up for that pool table. Oh absolutely, or the corn hole, or the fridge, the THWACK fridge. As you can see, we're very laid back about this. We're not really excited at all, but your account is connected there. And at this point, once again, you will see the modules that you own, the products and information. You can actually hit download, check your license details right from your THWACK environment, as well. Okay, so the next thing that I want to get into is actually hands on, is I want to take a look at how to make sure, before you upgrade, that all your ducks are in a row. And this comes down to the right way to back up, the right way to be ready to fix a rollback. And we'll take a look at that in a second here. So here we are on the database. Always starts with the database. Right, you can start with the database. And in fact, you can start with the database live, even if you're collecting data, and I want to do a backup. And doing a backup is the absolute most important step that you can do before you upgrade because if you need anything to revert, rollback, any kind of problems happened, this is the key to restoring you to a good working state. Well, it's also a good thing to do just every now and then, anyway. Yes, ideally you should be doing this at least once a week, but... You can technically lose, let's say the VM or the server that's actually your Orion platform server, and then rebuild it from scratch and attach it to your previous database. So of the pieces that are most critical— I mean, there might be a couple configurations that would change a little bit, but generally speaking, everything is in the database. So this is the thing that you want to make sure you protect. It is definitely the first one. If you have multiple modules— I mean, having typically run with half a dozen, at least, SolarWinds modules, I really don't want to have to rebuild. But we're going to talk about that in a minute. So backing up a database, even if you're not a DBA— and I'm going to channel Tom here, just few feet taller here— is not that hard, and in fact, as a DBA, even an accidental DBA, job number one is backing up the database. Job number two is knowing how to do a restore. That is what you will keep your job as a DBA. And you just right click on the database. Here, we've got the SolarWinds Orion, or I also have a separate database for NCM, that was once upon a time. I was going to say, this is an old server. It is. It has seen some days. It has seen some things. And under tasks, backup. And at that point, you just have to pick a location for the backup. And you have that. And that's just your... Save it somewhere maybe a little bit different. It doesn't need to be on your RAID 10, you can squirrel it away. You're not going to be accessing; it's okay. Right, it can be on cheap storage. It does not have to be there. And you're not doing a disconnect copy reconnect. You're just doing a straight backup here. Right. Okay, so that's backing up the database. Just want to make sure everyone understands that it's not that hard. You can even connect to the database using your SolarWinds credentials. You don't need to ask the DBA for your own administrator account or whatever. So this is pretty simple. But that's just the database. And you're using the standard SQL workstation tools here. Correct. Or you could use command line if you wanted to, if you're more comfortable with that. But you're just using standard Microsoft SQL Server tools. SQL Manager Studio. And SQL Manager Studio is a free download from Microsoft, so if you're not familiar with it, you can grab it, throw it on your Orion server, and manage your database from there. And it didn't used to be. It was always tucked in the installer for SQL Server, and it was kind of a pain. Now it's a little bit easier to get to. It's been free forever. [Laughter] Okay, but that's the database. Now how about that other stuff? You have some opinions about backing up the... I like to snapshot the server itself too, especially for major version upgrades. And the reason for that is the installer's doing a couple things, right? You have the installer and then the configuration wizard. So the installer lays down all the bits and then the configuration wizard sort of heals it, looks at your database, unpacks, move things around. It changes database scheme, everything else. So it does all of the magic on the database. So it is possible, with a major version upgrade, and again, we aren't really seeing this with NPM 12, but it is possible that you could have something go wrong. File permission or something else where a file gets missed, like SWIS stops working. Because maybe the XML files that represent the schema, that are used by information service to then map everything in the UI, one of those files doesn't get loaded and all of the sudden you get resources that don't show up, or you get the 17 777 error. Right. And one of those resources can take a little walk and then you have no web access. Right. So for major version numbers and most of our customers now, I mean you guys are running us on either VMware or Hyper V, take a snapshot of the server. So the sequence there is backup the database, shut down the services, take a snapshot. And then continue with the rest of your upgrade. You probably aren't going to need it. But then you've got one so that worst case scenario, especially if you have a really large environment with over a hundred thousand elements, where maybe something happens during that maintenance window--you have an easy way to rollback to at least where you were if something doesn't go right. And you get half an hour in and you want to just roll back to where you were. Then you're just going to start your server backup, bring the database back up again, and you're ready to go. Yeah, and let's say, if you don't have 100,000, if you have only a mere piddling 50,000 elements or even 5,000 elements, it's still not a bad idea. And the snapshots are too easy to do to not be disciplined about it. And remember, it sounds like this is a lot of moving parts, but one of the things that you'll discover is, if you go to the sequence that we're describing here, and especially if you practice it in your lab, and we're going to talk about that a little bit later, this gets to be really, really quick. And you can do this really quickly and don't need to plan a huge maintenance window to do it. Well, and you can do— oh, so, the database backup is cheap and easy and fast. And you can do that right before. Taking a snapshot of the bits, you can do a few days in advance. You can have this. And in fact, I know some people that's just part of the normal operating thing--is they take a snapshot every week or whatever it is and they have it. Another thing to do is diagnostics. Now I'm not going to show that because we have ‘diagnostic-ted’ you guys to death. We'll put links in the notes. We've done it in a couple different labs. But one thing: that's super important to run those before you kick off the upgrade because let's say something happens during it. You want to be able to give support as much information about your system pre and post. Snapshot? Be able to diff. Right, before, after. And we've talked about this in the support spectacular episode. Where we said take a snapshot before, and then you have, "this is a known, good running," which takes me to sort of the files to have on hand. So let's go ahead and flip back here for a second. So as far as having the files on hand. Earlier we upgraded this from 11 53 to 12 O, but I want to make sure I still had the old bits. Now some people compulsively keep older versions. And you can see here I have my old SolarWinds folder here with 11 53 install, but you can get the other ones from your customer portal. You can go to the download manager and open up the product list. I can go and say that I want network performance monitor. There we go. I want the SLX license, because that's what I have. And here now, I can go down to which version and you can go pretty far back. Back to the Stone Age. Practically, exactly, back to 2011, 10.1.2, if you need to. [Sighs] Good times, good times. And you can pull back the bits that you need. Right, and the point there is, if you're going to practice this, especially in your lab, that way you can look down at the bottom, see the version that you have, get exactly the same bits that you have laid down, and then practice exactly what that upgrade's going to look like. Right, and we're going to talk about practicing a little bit later on the show. However, I know, Kate, you have some strong feelings about rollback. It's a pretty standard thing in an enterprise. When you do a change control, you're asked what's your rollback plan, and I always dutifully said, "I have the old bits." And I can say it, but that's not exactly how you would prefer we go. Well no, I think it's a good idea, as I said, always have a database backup on hand. But the thing is, if you run into a problem, we need to fix it so that you don't run into that problem again. So the best thing to do if you run into a problem is open a support ticket and let us take a look at it. Because the chances are, we can get it fixed pretty quickly, and then you won't have to go through a change window to upgrade again. Okay, very good. And again, in the support spectacular episode, we talked about the process of opening a support ticket. You know, first run diagnostics, then open a ticket, then proactively upload the diagnostics. That's all in there in that episode, so I'm not going to cover that again. But yeah, we would prefer not rollback. It's the thing that you tell your change control team. You have to tell them that to get your window. But just don't plan on ever doing it. But getting the bits for the practice is definitely something you want to do. So, another thing that you want to do before upgrading, and this is just standard things, is read the release notes. Early in the morning, it's my favorite. Or maybe late at night when I'm trying to go to sleep. [Laughter] Either one, with a beverage. Hey, but you remember back when these things used to be giant text files that were included in the zip library and asterisk, deliminated. Right, they were hard to read. And they were hard to find. They were really difficult to find. And now in the customer success center, you can see that when you're in the page for Network Performance Monitor, release notes is right there. So I can just click on that and get to not only the current, but the past release notes. Because even in a corporate environment, I remember getting into conversations, "When did that get fixed?" "What version was that updated?" You know, I remember it was a patch or an upgrade, or whatever it was. Here, you can go back and look at them. And that is important to do to know major— like a NPM 12 upgrade. You're doing it for the features and everything. But when you got a hot fix or even a dot dot dot rev, those are the ones where you want to look and say, "Is this really fixing anything that I need?" We always like to have our customers up to the most current, but we also understand that the compelling reason to open a change window this weekend, right now may not be there. Because you never know, there might be some awesome features you didn't even know about. Right, and that's all over the place. What we spend most of our lab episode showing people features that they've been paying for on maintenance for years and didn't know was in the product. And looking at the release notes is one of the easiest ways for you to find out what's available. Yeah, recently I found out SAM 6.2.4, the latest release. There's a lot of really amazing things that we included in that one revision, but you may not know because it's a dot dot dot. It's a 6.2.4 release. So it's really worth it to take a look at those things. So some other places to do some research. Because again, you want— especially with the major left of dot revs, you want. Like NPM 12, I want NetPath so bad. I'm not going to wait for dot one. Must have. Right, but you still want to do due diligence. So another place a check is THWACK. We've been talking about along the product form, looking at what other people are doing. But there's another place in there that's even more important. Actually, let me navigate to that real quickly. Is under the product forums, let's go to NPM. That main page—again, because we've updated and really dressed up the THWACK pages a lot, there's a lot of interesting things just on that main page that you may not go to because everything's in your stream. So not only that, but also the beta and release candidate pages. One of the best ways to find out about your upgrade is to participate in the beta and release candidate programs. I thought the reason that it was good was because you earn a ton of THWACK points. That is exactly—we're back to corn hole and the fridge, and the THWACK Snuggie. And all of those. And the socks. The socks. That is definitely why you do that and you work with UX. That's right. And in fact, you may have noticed that we've been talking about the SWUG events, the SolarWinds User Groups that we've been going to. THWACK has made some adjustments in the way that the points are given. So that we've actually trended more toward really incentivizing participation in UX and the beta programs. You can earn about—if you're start to finish on the product you can earn up to about 6,000 points. So it's a chance to really push aLTeReGo down. Really, someone will eventually take it. Plus, when you join the beta, you get to give development engineering your feedback before anybody else and we would love to have you. Thank you for bringing it back on track. Yeah, the beta and the RC. First of all, you have a chance to really affect the direction of the products. You have a chance to say, "I want the functionality work like this not like that." You really of a chance to give feedback that's going to matter and the other thing is that RCs are really solid, a solid software. Now I understand that enterprise, especially larger enterprises, have a governance process that will not allow something that is labeled as an RC to be put into production. However... It's fully supported. It is fully supported. It is production software, production quality, production grade. It gets production support. Yup, absolutely. You can open a ticket. Our developers will help you. We definitely want to... It is an express route to support because if you're participating, certainly for those of you who've been involved in betas, but RCs as well, I mean we want to know if there's an issue. So that we can get the golden build out there. As fast as possible. So when you open a ticket, if you're part of one of those programs, those are expedited. So my point with all of this, with the RC and the beta, is that the best time to be ready to do an upgrade is actually in the months before the upgrade, when the beta is announced, when the RC is announced. Get it installed in a lab, at a practice environment, which we're going to talk about in just a minute here. But get that involved and then you have a wealth of experience, and you can go to your change control, your governance groups, and say, "Nope, I've been doing this for two months." And that is worth, at least in the enterprises I've worked, that is actually worth a lot of credibility in terms of getting a change window. Oh, you've already done this upgrade three times? Fantastic, then we know what we're up for. And then you get to go brag on THWACK, like all the RCs. Like when the people are in beta, users are in beta, they weren't talking about-- they're not supposed to talk about things like NetPath. But once they get into RCs, and they start leaking it on THWACK and like, "I'm using this incredibly cool thing." "This is really great." So, you do get some bragging rights, too. That's wonderful. As you are planning, one of the things that you can do, also, is take a look at the order of upgrading and the process. You know, if you have multiple devices or multiple modules. And a really great way to do that is with the upgrade advisor. A lot of people aren't aware that we put this out almost a year ago now. The two versions of the upgrade advisor. Right, there are two conversions. So from the customer portal— again, at the customer portal, you go to support and product upgrade advisor. Now from here, what you can do is describe your environment. So let's say that I'm running Microsoft 2012 R2, my database is running on SQL 2012 SP3. I've got NPM version 11.5.3. I've got another product— oh sorry, add another product here. I've also got, let's say, SAM 6.2.3. We'll add one more just for fun. NCM version—I'm doing this on purpose, don't hit me or get the duct tape, version 7. And I want to go to NPM version 12. So now, I can check and make sure the compatibility, but also it will tell me how to do that upgrade. And if you forget what products you have, look at the bottom of the page. More importantly, it's not going to work because that's not possible. Yeah, this is impossible. I think it was the NCM 7 that really messed it up, so let me go ahead... See, you wouldn't let me tell you. I know, I know. So, we're just going to go with fairly recent versions. And there we go. So here, it's telling me that this is compatible and it also tells me the path. You know, before starting, you want to upgrade NPM from 11.5.3 to 12 and that's it, because I said, I just want to upgrade 12. If I also wanted to upgrade SAM or whatever, it would be in there. But I would say the previous screen was not a fail. Because you figured it out first, that you needed to make this upgrade first. So you saved yourself a bunch of time. Now, it would've blocked that, the installer would've blocked that when you went to do it, but you now are not in the maintenance window, discovering that, "Oops I can't really make that upgrade." Right, okay, and that's a really important point is that this is the web-based upgrade installer but when you run the actual install on your box, it's going to run a local version of this and it's going to look at the bits you have on your server. And so this is a nice way of double-checking. Or pre-planning. Like part of the pre-flight. I like to do that, like if you're planning your upgrade, you know a week out, because you don't want to do it months out, because we will probably have new versions of this, especially if you have a lot of module. But a week out, run through this and think of it as sort of like an upgrade simulator. Right. Or if you're going to practice, then you can go ahead and say, "Well, I just want to have all the bits on hand." "I don't want to have to do download." It's also good even after the fact. If you get that message that you can't do it and you have an environment where you can't get to the internet from your servers. So if you've got that, then again, you can go to the upgrade advisor on internet-connected device and pull down the installers that you need. Which takes us to the practice first. What are some ways that you can practice doing the install? Because I get that a lot when we're at trade shows and things is, "Well, that's wonderful, but I just have my installation." "So it's a one-way trip and that's all I've got." That's not all you've got. You got some other options. I'll take volunteer suggestions. So you showed earlier that you can get all the early older versions from the customer portal going all the way back to ancient times when we monitored dinosaur attacks. [Laughter] Exactly. Dinosaur availability. Yeah. One thing you can do is download the older version, put it on a VM-- it comes with a 30-day trial license--and run through the upgrade there and see if you run into any problems. And then you can get them solved before they even happen. So then I can put it on my cheap PC, my laptop, and I can install on that? Exactly. That's what you recommend? Well. Rather have you put it on a VM that's already your current server. Well, I was going to say the other thing is it's a chance to practice doing a restore. Because you're doing database backups anyway, so unless there is some overriding reason why you don't have room to do it, I start with—I restore my most recent snapshot, set up the version that I have, and then practice the install. So I also know how long it's going to take up my data, and I got to practice doing a restore as well. Right, you can restore the database. You can restore, actually, one of those VM snapshots if you want to. But you can use it on a spare VM or you can install the other one. Now, sometimes I get some folks who say that they don't have a license for another instance or another SQL Server. So in those cases, you can download— we'll have in the show notes, that you can download SQL Express. It is free. Free as in puppies and beer and rainbows and all that, that you can download SQL Express, install it on, again, the same VM or another VM. And you can use that because you actually don't care how much data it's going to hold. For a small test. For a small test, right. If you have a 100 gigabyte database that you've backed up. Yeah, don't use. Well, if you're not worried about testing the database. If you just want to test the install, the actual install, that's where it comes in most handy. And again, we'll have that link in the show notes, to do that. The other one, sometimes people want to be able to see the upgrade along with sort of live monitoring information. And this goes back to lab episode 43, when we talked about how to install NPM inside of GNS3. So we have this wonderful partnership with GNS3 now, and you can install NPM inside with some routers and switches and things like that. And you can have a monitored environment on, let's' say 11.3, and then you can do the upgrade on that box and then see what the monitoring implications are with live data. You can take interface down, and you can take a router down, and watch all that happen. So that's another way that you can test. I typically use that when I'm testing advanced custom alerts that I've created to make sure that the workflow is working the way I expect. Which is a great idea because then you're not spamming your ops group and especially if you're doing that early in the morning or late at night, they will appreciate it. Yeah, few alerts and the alert simulator. "This is going to affect how many?" [Laughter] Those are a lot of numbers, and two commas, oh. Hold on a second. Oh, I did base this alert action on a custom property that I changed here in the lab. I'm probably not going to do that in prod. Right, okay, so those are some ways that you can test out the environment and practice, and also some more research tools. I want to move us along to actually doing the deed, actually doing the upgrade. But you know, it's more than just one server. I've got a primary poller; I have three or four APEs, additional polling engines. I've got an additional web server. I've got a bunch of stuff. So what are the guidelines that we can give folks in terms of should I do leaf and branch? You know, start at the furthest point away or what do we recommend here? So the best thing to do is to start on your primary poller because the additional pollers are going to do a version check when they install and if they're all different, then nothing's going to work. I think trunk and leaves. Yeah, you start from the center, start in the juicy center, and then work your way out to the crunchy shell. Yes. The other thing is you always want to start if you're running with NPM, you want to start with that first. That will get your core upgraded, the main upgrade going, and then you can add all the other modules as satellites. And then, my favorite tip, you don't have to run the config wizard after every install. I just learned that while we were prepping for this show. I was very methodical about it, and I'd install one thing and I'd run the config wizard and then I, you know, it's a lot of time. You can do it that way and as much fun as it is to watch it run through the database schema four, five, 10 times. On 17 polling intervals. Yeah. No, no, no, back when I built all of our demo systems, I have laid down many, many, many copies going a long way back with all of the modules and have probably done 200 installs just laying all the bits down first and then running the configuration wizard. I can't imagine how long that would've taken doing at one at a time. Oh, I can tell you. I have old notes from things. I've lost years of my life on some weekends doing that. So yeah, so again, you only have to run config wizard once at the very end if you have multiple modules, that's an important thing to know. Okay so, now this takes us to some of the advanced upgrade scenarios. Some of the things—we talked about just the install and hopefully at this point, everyone has a really good comfort for the upgrade process in general. But, you know, I've had this installation going for two or three years, the hardware's getting old and tired. I need to move things, I need to migrate it, the polling engines, I think, is less of an issue. Or you originally started monitoring 2,000 elements. And now you're at 20,000. Maybe you've just grown and so many of you do that. You start with may be a smaller license or one module and pretty soon, you're now monitoring a lot, and it seems magical for a really long time. And then all of the sudden it's like, hey, my ride's getting kind of slow. I don't know what the problem is. And we almost—it's sort of the painter's house syndrome. Like the tool that's closest to us, a lot of times, we just assume that it's going to be able to do anything and we wouldn't worry about hardware. But the reality is just like a database server or Oracle or any other application, you need to think about the resources that are made available and I think I know where you're going is back to the customer success center and being able to use the sizing guide. I review the sizing guide every single time I do a major point upgrade. It's just a good habit to get into and that'll let you go through and it actually has some pretty easy-to-use, little calculator to say, I got this many elements, nodes. It'll walk through and remind you what hardware you, let's say, would have deployed if you were starting at the scale that you have grown to. As long as you're talking about the sizing guide, let's take a look. So here we are on the customer success center. Yeah, and for those of you who have not checked out the customer success center, definitely do, there's a link for it on the solarwinds.com page and everything that you're going to look at here, there are layers and layers of product-specific information, everything from quick start guides to in-person training, and it's all here. Quick videos, a bunch of other stuff, definitely check this out. So here we are on now NPM, getting started. This entire thing is the quick start guide. So instead of walking you through a 1,700-page PDF. It was glorious reading. It was the best of times; it was the worst of times. It was great bedtime reading. It was. That's right. But we've extracted the things that you guys have said that you use most as a part of the admin guides and then added onto that some additional information, training, how-to videos, the rest of it. So it's all put together in this getting started guide. And things like the first thing here, which is "plan your production deployment." Right. And so here, what we've done is you've grouped your employment into small, medium, and large--which is kind of arbitrary--and we just said sizes. But you can see that it has a lot to do with the licensing levels, and then we give our suggestions for both the polling engine and also the database server. So you can go here to get an idea of where you want to be with. And we also keep this up to date. Like, you'll notice that some releases actually improve performance. Maybe polling ability per CPU actually goes up or maybe the database isn't used as much, maybe there's more caching or more optimization. So we keep this up to date. So maybe it's been a while where something shifted for where you thought you needed lots and lots of database, but maybe now you actually are caching a little bit more, and maybe more memory would help. So regularly going back and looking at this, at least for major upgrades, is handy because the math gets recalculated for major releases. Right, so that's how you can plan things out. So I want to wrap up by talking about— all right, so we've done all the planning, we've been real careful, we practiced the bits and everything, but something still didn't quite go right. You know something is off, and again, in earlier lab episodes, we talked about how to open a ticket, but I want to see it. I want to take a look and have just an idea of what's going on. So I'm going to switch places with you again, Kate. And where should we be looking, as experienced SolarWinds administrators, as people who've been using this and we want to do a little bit of hunting and catching our own food? Where should we be checking things out to have an idea of what's going on before we open that ticket? So all of your log files live in C program data which doesn't have the space in it but I do that literally every time I type it in. Wouldn't it be nice if there weren't spaces in any panel? Yeah, it's not what I said, what I meant. C program data--SolarWinds is where all of your log files live. Recommend starting with the logs directory, which is pretty easy. Orion is where the bulk of the new stuff gets written and depending on where your error happened, but most of the time, you're probably going to see something in the configuration wizard. You want to start with this log. And just kind of scroll down and see, okay. If it failed on a SQL query, it will tell you what it was trying to do. If it had a file permission error, it will tell you what file it was trying to access. This is probably your best bet for anything that would go wrong. The other good one to start with is the Orion web log. So let's say the config wizard finished without a problem but your website isn't coming up. You're getting an error. This is a good place to look to sort of see okay, what was I trying to do when I hit that error. And when you generate diagnostics, it packages all these up as part of it. Yeah, which is something else I was going to save. I'm looking at logs on a server. A lot of times, I'll just run a diagnostic to get them all in one place to make it easier for me to find. But if you just want to check a couple of quick files, pretty much everything is here in C program data SolarWinds. And go back just for a second. The upgrade wizard log, the confederation wizard. That log has been really helpful for me in the past. I don't do it now, because I use SWIS for everything but for some of you who have been messing around with the database schema, if there is something that breaks because of something that you've customized in the database, you will usually see the alerts in there. That's also true. Also, when the installer starts to lay down all of the different MSIs, they all have their individual log files. So if you don't even get to the config wizard, you can check these little MSI logs and see. And that'll be file permissions or folder structuring. Excellent. Although if you're logged in as machine admin and you ran the permissions checker, you shouldn't have a problems. Hopefully that's the case. Well I think that's the whole thing about all this, is that if you've been you know organized about this, you shouldn't have these problems at all. But you know where to go look, you know whether it's a critical failure or just a little hiccup where honestly, re-running configuration wizard cures a lot of ills. That's very true. That may be all you need. And you don't need to have that panic moment at two o'clock in the morning during a change window just to rerun it. But this way, you have an idea of what happened and you can go back in and take a look at it again. My guess is that, if our customers follow these instructions, their upgrade success is practically guaranteed. Well, we pulled this information from the support calls we've gotten over the years, so I'd hope so. And that's true. I mean, there've been a couple of upgrades over the years that maybe weren't the smoothest thing ever. [Cough] 9.5 But in true SolarWinds fashion, we've worked hard to learn from past experience, and create a much better experience now. Most of you have seen the forum. If you haven't, go out there and look at the NPM forum for NPM 12, the upgrade seems to be a lot smoother. So don't take our word for it, go and look and read what people are saying and we think that this is actually going to be something that's going to make a major difference going forward for all future releases. And the truth is that most of the time, most upgrades over the years have gone smoothly. It's true. Having used SolarWinds products since, oh, back to 2003, I actually have never had to rollback an install. But you were also pretty organized about them right? For sure. Okay, well there you go. So if you've been putting off an upgrade, we hope this episode gave you everything you need to feel confident to move forward. For SolarWinds Lab, I'm Patrick Hubbard. I'm Leon Adato. And I'm Kate Asaff. Thank you for watching.
