As you might imagine, it’s a bit of a busy time here at SolarWinds HQ. There is A LOT going on, not the least of which is encouraging our customers to upgrade before March 8.
(Why March 8? Keep reading and I’ll explain in a moment.) First, I want to emphasize the important part of that sentence:
“encouraging our customers to upgrade before March 8.”
All of us—from sales to product management to the leaders of the THWACK community to your
humble Head Geeks—have been up-talking upgrades for weeks.
In fact, the Head Geeks are SO VERY committed to encouraging folks to get onto the latest version, we’ve been pitching in to help our support desk family. For several weeks now, we’ve been grabbing tickets out of the queue to answer common questions, dig into weird problems, and help escalate issues. We’ve done this because all those blog posts, emails, and phone calls resulted in ticket volumes of historic proportions. While all of the Head Geeks love a good mystery and all those “not working as expected” tickets are like catnip for us, we stayed on task and limited ourselves to picking up one particular type of ticket: the ones asking for help upgrading their environment.
Because of this focus, we all noticed how a few common themes came up in ticket after ticket.
And so, in the name of public service (as well as helping you avoid the need to open a support ticket yourself), I thought I’d share some of the most common issues I’ve seen, along with the most commonly successful solutions.
Before I get into the details, there are some key points I’d like you to walk away with:
- Upgrading is not only useful, it’s necessary. The upgraded software will ensure you’re protected from any of the recently discovered vulnerabilities.
- Upgrading is also the only way to avoid A LOT of noise you would otherwise hear from your antivirus tools on March 8. That’s the day we revoke the old signing certificate we used on our binaries, in favor of the new one we started using in December 2020. If you don’t upgrade, your security tools will begin to flag SolarWinds software.
- If you want detailed specifics on SolarWinds security measures; on the versions and tools affected; and more, please see our Security Advisory Page.
With all the preamble out of the way, let’s dig into the upgrade fun!
Look Before You Leap. Or Upgrade
You know what they say about an ounce of prevention. This is the checklist I’ve started giving customers who open tickets either because they want to “know before they go,” or have tried an upgrade and hit an unexpected snag:
- Get the red out
Before you start, it’s always a good idea to go to Settings, My Orion Deployment, and click the “My Deployment Health” tab and click into any red items. Follow the instructions there to resolve them. They are red because they’re BAD. You can be fairly sure those suckers are going to cause you upgrade heartburn.
- Local Administrator or bust
While this may not immediately screw up your upgrade, it’s the #1 cause of weird lingering issues post-upgrade. If you’re upgrading using one of the local installers (more on that later) you will need to remote desktop (RDP) into the Orion polling engine. And when you do that, you need to log in as Local Administrator. Not domain admin. Not “Bob who has admin privileges.” Actual local admin.
- Check permissions
Largely because of issues relating to doing upgrades when NOT logged in as local admin, some files and folders don’t get the correct permissions. Therefore, it’s always a good idea to run the permission checker: C:\program files (x86)\solarwinds\orion\orionpermissionchecker.exe. Run a check, and if it finds errors, let the tool fix them.
- Your Orion database’s one and only
Another common issue I’ve been seeing is the install hanging at xx%. Digging through the logs shows the database is never “free.” Digging further shows something—a backup routine, logfile shipping, or even a forgotten SDK applet—is repeatedly (if not continuously) connecting to the database, leaving the installer unable to sever all connections and do the work of updating tables, schema, etc.
- Show respect to your parent
Parent poller, I mean. I’ve seen a lot of folks under the impression that you should upgrade the failover or secondary devices first. BIG NOPE. If you have SolarWinds HA, make sure you upgrade the active polling engine before you upgrade the secondary/failover engine. If you have additional polling engines, upgrade the primary FIRST. Then you can upgrade the additional polling engines, either via the console-based upgrade utility, or using the downloadable “Scalability Engine” installer.
- HA is no laughing matter
Read the docs on the process of upgrading your HA environment. There are a few gotchas hidden in there that you want to be clear on. Example: once you upgrade the active, the secondary won’t be able to join the HA pool until you upgrade it. This is normal.
- Hit pause on antivirus
I *know* there are people who are going to misconstrue this, so let me be 100% crystal clear: NOBODY IS SAYING YOU CAN’T HAVE ANTIVIRUS ON YOUR SOLARWINDS SERVERS!YOU *WILL* BE TURNING ANTIVIRUS ON AFTER THE UPGRADE, RIGHT?! <piercing angry stare>
Ahem. Sorry, I get a bit emotional about this. If you have especially aggressive antivirus and/or anti-malware tools running, they can block the installer. Please pause them for the duration of the install. Then turn them back on.
TURN. IT. ON. AGAIN.
Whitelisting For Fun and Profit
Another major grouping of tickets I’ve seen has to do with the upgrade—either Web console based or using the local Online Installer—getting stuck when it can’t reach an external resource. While it’s a truth bordering on a law of IT physics that corporate server should have unrestricted access to the internet, it’s equally true most corporate systems need SOME access. Exactly which external resources are needed, and for what operations, can be a bit of a head scratcher. Luckily, THWACK MVP and all-round cool frood Mark Roberts did the legwork for you, by publishing a SolarWinds URLs he’s had to whitelist, and what modules or operations they relate to. You can find that post here:
https://thwack.solarwinds.com/product-forums/network-performance-monitor-npm/f/forum/88937/orion-urls-for-firewall-whitelisting
However, I don’t want to keep you in suspense. The list of URLs you need specifically for upgrades are:
The Work of the Work
With all those caveats, hurdles, best practices, and preparations aside, we arrive now at the main event: DOING the upgrade. I wish you good luck and Godspeed.
Let’s say things didn’t go off as planned. Despite your careful preparation, you ran into issues. And I KNOW some of you are running into issues, because I’ve answered approximately 100 tickets this month about it. Let’s look at ways you can gracefully recover and avoid opening a
second change control window to get the work done.
The Web-Based Install Is Cool. The Local Install Is (More) Reliable
I’m not bashing the web-based install. As the saying goes, “60% of the time, it works every time.” And from what I’ve seen, our numbers are more like 80+% of the time. To be honest, the web-based install should be your first stop on the upgrade parade.
BUT there are times it doesn’t work (that’s why people open tickets). Which is when the local install should be your next step. Here’s how you get what you need:
- Log in to customerportal.solarwinds.com.
- From the home screen, scroll down to the NPM installer for 2020.2.4 (Jan 25) and click “download.”
- From the next screen, download the Online installer (which often appears second, so read carefully).
- This should download SolarWinds-Orion-NPM.exe (about 97Mb)
- Log in to the Orion polling engine as local administrator.
- NOT domain admin. NOT “Bob who has admin privileges.”
- Copy the installer file to the poller.
- Shift-right-click the installer and choose “Run as Administrator.”
- Let the installer run and have it update everything (database, services, and website).
When All Else Fails, Try Offline
Despite all of this, there are STILL times when the installation fails. Something can’t get to something to run something else.
In those cases, my next piece of advice is to download the OFFLINE installer and use that. This is a 4.7Gb .ISO file with all the bits in one self-contained package. You get it from the same place as the online installer. But once you have it, you need to mount it as a drive. To do that, you have some options:
- If you are kicking it old school (as the hep kids say), you burn the ISO to a DVD, insert it into the server, and get jiggy with it.
<record scratch>
OK, maybe I’m being a little more retro than is realistically possible. Or even desirable.
- You can also make the ISO available to your virtualization manager and add it as a mounted drive for your Orion poller VM.
- Finally, you can copy it to the Orion polling engine (whether it’s a VM or physical box) and mount it through the operating system. Here’s a guide to walk through it:
Once the ISO is mounted, you can run the installer from there.
The Mostly Unnecessary Summary
The point of this blog (and the ones from my
fellow Head Geeks) is to get you past most of your upgrade woes, both now while upgrading is absolutely essential before the March 8 deadline, and in the future. Hopefully, you didn’t even really need most of this blog, and are just reading it for the entertainment value.
But if, on the off chance, you have tried everything here and still have no joy, there’s one more thing you can do:
Open a support ticket. I’ll see you there.
