Are your network device configurations anything but standardized? Does standards compliance, whether internal to your company or external like HIPAA or SOX, keep you awake at night? Join Destiny Martin, along with Head Geeks Patrick Hubbard and Leon Adato, as they dive deep into SolarWinds® Network Configuration Manager (NCM) customizations, including creating policy checks, rules, and templates, and using them to help ensure that your network devices are safe, stable, and standardized.
Episode Transcript
Hello, I'm Destiny Martin, and welcome to SolarWinds Lab. I have a feeling you've been wanting to say that for a really long time. Oh, and that's Patrick and Leon. Actually, you've been waiting over nine years to say that. She's one of the original Geeks, as in all the way back to Tulsa and the founders' original Geeks. Oh, so there are people who are around here from before your time? Well, not that much old, he's really old. I think I know what you're trying to say, but yes, today's show is all about NCM, and in particular, some really deep magic for configuring custom security audit policies... From scratch. And from scratch. And even some automated radiation actions... From scratch. Of course, from scratch. You guys, seriously, I watch this show, too. And you keep dodging all of the NCM questions. I'm going to finally fix that. Well, that's because you're sort of a THWACK community celebrity. No, no, I'm a celebrity. Destiny is beloved. I don't know about that. Trouble? Yeah. Okay, well, you spent half a decade managing the MIB database that powers most of our products. Oh, that little thing? Okay, do you guys want to see how to create your own audit policies and remediation scripts, or not? Of course, that's why you're here. Okay, so how about this? Destiny and I will walk through the "how-to," and then you can come back for the bit that you want to talk about. Oh, the NIST database integration. Thank you for not saying API. Oh API, yes, but it is an API that automatically imports the NIST national vulnerabilities segues, and then scans, not just configs, but the hardwares and operating systems end. So you really are compliant all the way from the silicon, all the way up to the configs that you make, including the OS. [SOUND] Okay, Destiny, so set this up for me. I don't work in legal, I'm not a hospital, I'm not a bank—why do I care about compliance policy reports? Well, I'm glad you said legal first, because compliance reporting actually can help you with standardization of your configs. So say for instance, if you need an actual banner that has your copyright on it, we can actually tell you if it's there or if it's not. And if it's not there, we have automation remediation. Oh, got it. So, if my—for example, my bonus is based on the fact that all my devices are provisioned correctly according to the audit group. Yes. Which we all love. Then I can make sure that my assets are protected. Yes, and that is where your NCM comes in, because we're going to make sure that you are protected, you get the money in your pocket, and you're saving money by using NCM. Awesome, let's take a look at that. Okay, so now to go on to the policy reports. And actually, what we can do now is automatic remediation. Okay. So let me explain a little bit about policy reports, because I know they can be a little bit confusing. Sure. So, I'm going to go into the compliance portion of the NCM. And this is what presents itself. I can then manage my policy reports. That way, I can show you where everything is coming from. On your Manage Rules—think of this as kind of a pyramid, because you start with your rules, and that's where you're going to set your scripts or your actions that are going to actually take place. All right. So once I go in our roles, I want to add a new role. So I'm going to open up a page that we have currently going on here. And I'm identifying this role, and I'm saying it's a banner change. Say, since it's 2015, I need to change all of my legal banners to 2015. So we're compliant, and I'm not getting audited, and I'm saving my money. By accessing this, I put it in an "In House Rules," created a new folder. That way, I can keep it in context of what I'm using in my office versus what's out of the box. I'm going to put this as a "critical," because I don't want my bonus leaving. So I want to make sure we're on top of this. Anything with my paycheck is critical. Yes, and instead of leaving it "as string is not found," because I want to know if this legal banner is there. I'm going to say, is it found? Because if it is, I want it to actually change it to the new legal 2015 banner. Now, how I can also get this to go through is, I can check this new box that we have, and it'll automatically execute the script when violation is found. Nice. Yes, so think about that. I can also, if I wanted to, go to the ‘reg’ expression that I'm looking for. And instead of doing it as found, I can say, what if it's not found? And then, any time my banner is not presented on any of the network devices, it would automatically put in the new actual banner that we need. So, that is something that with automatic remediation, that people can kind of use for standardization of their configuration files. And when you create a rule, how it gets into a policy, is we'll go back to the Manage Rules, is you're then able to pick and choose, and if you notice, there's about 44 different pages of these of rules that are already created for you. And you can click as many as you want, and what you do is you actually add these into your policy. So you start with your rules, and then you roll up three, four, five, ten, however many you want, and then you make that a policy. And then from the policy, you then roll up how many policies you want. Maybe it's just one. Maybe it's six different policies and you're manipulating them around. You roll that into your report. So that's why I say to everybody that it kind of reminds you of hierarchical pyramid, because your rules are down here, and that's where the scripting comes from. And some people get confused about that, and they want to know, "Well, how do I know it's actually doing what I need it to do?" And it's when you drill into it, you can see the rule. It has the remediation. You apply those rules to a policy. You're not doing a rollout policy. Within there, the rule is actually non-compliant. It's not rolling out all three of those rules that are in the policy, it's the ones that aren't compliant. Got it. So you don't have to worry about it just running a whole bunch of scripting on there. And the thing I like about the rules, also, back at that screen, was that if you're not comfortable with regular expressions--which a lot of network engineers typically aren't--that's more of a programming kind of thing--and a lot of the network pros haven't gotten into that yet-- although it's a really good skill to have--is that you can do straight strings. You can do, "I'm just looking for A, B, C, D, E," just like that. So you can mix and match the way that you're doing it. And something that we can also do since you said that, is you can also do an advanced config search now, for which you can do a block search. So you can actually grab your config files, like what you're trying to do. And it must contain reg expression or find, and you can put the string in there. So we kind of take the hard part out of the equation and help you to actually put it in your own code. But with the Wizard Interaction Point that people that kind of don't want to go in with the rejects formulate, you can actually do it this way. And it makes it a lot easier, and it saves you some time, especially if you're not into rejects. I'm not a huge fan of it. I can get by with a lot of Reg Helper. But, I mean, this is something where you can come in here and you'll learn and it helps you to mold out what you're looking for when you're getting into your config files. All right, that's really, really useful, and I can see it being applicable in a lot of different situations, making sure that your interfaces all have the correct naming standard, for example. Or your ACLs, if they're absent, or even if they're present. Right, or if your passwords, make sure the default password is out. Mm-hm, validate correct quality of service shapers. Right. You definitely want to make sure they're on the right routers and where they need to be. Right, near and dear to our heart in the monitoring space is to make sure that you have the proper SNMP stings and that public and private are gone. Yes, and also, you want to make sure that you're sending to the right syslog server. Sure, okay. So, of those that we just talked about, I'd really like to see the QoS. I know it's becoming a bigger deal for a lot of people. For a lot of our IT pros, they set the QoS and then it's been six months, and they don't remember it or whatever. So, having something that would automatically check for that might be really useful. So tell me more about what we can do in that space, with QoS. So, with the QoS in the shapers, I mean, you're going to start noticing a lot of drops and things. And if you don't need them on there, take them off. There's not a need to have them on there. And always, it can also help you with your voice over IP, so you don't have dropped calls, things of that nature. So, what you want do is make sure where they're at, and if they're not supposed to be there, because we can actually choose nodes that were not supposed to be on and tell it to take it off. Excellent, so why don't we dive into that line and try it out. Okay, so to get into the quality shaper. When I'm looking at this view, if you notice, this is the network path troubleshooting view. This is found on our demo as well, and right here you can actually create the view. We can actually see that there's 433 discards from BOWAN, we see the utilization—that's high. We can see that there is a change made just from this view. So I'm going to actually click and see what it is. And I can see that there is shaper that was applied incorrectly. Right, which again, we were talking about how QoS is one of those things you apply once and a lot of us have to look it up, we don't do these every day, so this is an easy mistake to make. Correct, and so if I was to come in here and I'd seen that this was placed incorrectly on one of our devices, I would want to know where else is this being found at. So a really easy way, and what I wanted to show you, is we can actually highlight what we don't want. And we can right click and copy it, and then we can go over to the policy and create an actual rule for this. So, to show you how easy it is with the regex, and also with the advanced config search, we can obviously set up our rule name in our description. And then we can add a parenthesis, must contain, and then you copy what you were looking for, because we don't want that to be within there. So we want to say, "is that string found?" Right. We're looking for it so we can get it removed. Mm-hm, and then all you have to do is check your parenthesis, and this validates your regex. So when you’re doing this and say you’re not very with your regex or anything. Like stated before, this validates it before you even get started, so you know if it's going work or not. That's awesome. So then I'm going to go look down at the remediation. I already have it on there. We need to say, configure the terminal, no service policy, and exit out of there. I don't want this to be automatic this time. Because I'm going to run this every once in a while as a check. And I may want this on some of the nodes. Right. But what if I only wanted to run this on just a couple of nodes instead of running this report against every one of our configs? Okay. A lot of people don't know you can actually change that. It's not in the rules; it's in the policies. Ah. So, once I have this set up the way that I want to do it, I'm going to hit submit. I can then go into the policies that we have this added in, to be able to apply it. So, I'm going to go back into the managed, through your NCM settings, and Manage Policy Reports. Now I'm going to look at the policies. I've added these into the actual policies that we have, as a QoS implementation, already. You would just add it in for whichever one that you were wanting to work on. And I can click in here and actually show you where you can pinpoint the nodes that you are wanting to do. You can do this on a dynamic query— that's what I have done here, because I'm only running on certain routers that I have, that are the Austin. And I can actually go into core, or do anything more important, or grouping, or custom property. It's really up to you. Right, and this would be also one of those places where we talked earlier about custom properties. To use custom properties to say, this one does get packet shaping, this one doesn't get packet shaping. Queue as packet shaping, yes or no. Or packetshaper group two, three, four. So you could have multiple different configurations that are automatically scanned, or grouped, at least, when you want to apply them. And it's very simple. A lot of people got confused when they correct the rule, because they thought, if I actually run the report, it's going to go against all my configs. It's when you go to that next level up, when you roll it into a policy, where you pick and choose where and what you want to do on this. So this particular rule, I'm only going to use on the Node Caption that has the AUS on either side. So, by using that, the other policies within here in rules, they are actually going to be ran on everything else. So you can pick and choose with your rules which ones go to where, but still have one policy, because you're wanting to run one report. Really useful. Definitely. And then you'll be able to run your report and then you'll be able to remediate using the script that we put on there. And that's how easy it is to get these set up for this. And now that we were able to actually take off the QoS shapers, now let me show you a cool trick of how to put them on, if you want to, using the config change templates. Now, something that I want to talk about with this is that a lot of people think config change templates, and they're going, what is this scripting nightmare? Well, just keep going. Hit the execute, and it's actually a wizard. It's an interactive wizard, and you can pick and choose nodes. We're not asking you to create scripts. People have already done that and you may have somebody in your facility that can create the script for it. Simple to learn, we actually have it in the admin guide. But before we go there, let me show you how to make it simple and actually useful for your company. Excellent, let's go ahead and take a look. All right. For the config change templates, the easiest way to start with it is with a template that you know you are going to create for your policy. So, I'm going to use this QoS policy on the IoS routers and I'm going to click into this and tell it to define my variables in a run. I don't need to modify it because somebody's already done the scripting behind it. This is actually Kevin Sparenberg, who works for us currently and this is one that he uses. So we're going to pick our devices. Notice there's no scripting enabled or anything that's being needed. And I can say either all of the Cisco, or choose. I'm going to choose these two because they're the Austin cores that I want. When I go to the next, now I get to choose the interfaces. So once again, there's no scripting here, and we're still in the config change templates. It is a very intuitive wizard, and it's very interactive. I'm going to click on the interfaces that I want. Say okay. Now I get to choose: do I want the 10-megabyte policy on the branch office? Do I want the 20-megabyte policy? Probably not the one megabyte broken policy. [LAUGH] No, we don't want that one. But it's nice that you said that, because in this policy that we can create, we can actually have this set up to show if we have that on there for testing. So we have this one available just to test our policies to make sure that they're working correctly as a shaper. So that's a nice way to have these within here. So, I'm going to choose the ten, and I'm going to hit NEXT. Now it's going to show me, make sure that it's on the interfaces. And then it shows me the show commands in the new window. So this is everything that that's doing in the background to get those. Now for the policy, it's already created my script for me. I'm going to grab this. I see. So you're really just using the ply to get the full config push that you wanted to do, and using that as your starting point. Mm-hm. Sneaky. And then I'm going to go into the compliance. Let's just add one from the policy report we were working on earlier. Go into the QoS implementation. Let's edit this report. Notice if you go up in the tier that it like, from now I'm on the report but now it's saying, do you need to create a policy? So it'll take me to the policy portion of it to create a new policy. If I'm in the policy, it'll remind you if you need to create your rules so that's very handy, too, because you can stay within where your working at and it takes you back and lets you know where the chain is. So back to that pyramid idea of it's going to walk you back up or down the pyramid to make sure you have all the pieces. Correct. Like it! So, I can say config change template here. I will actually go to the strings, say that I'm going to look for, say AUS. And I'm going to put on here core. I can then name the rule. Got to give it a name, give it a description. Obviously on lab, we always give real quick things just to keep things moving along. But you're going to want to really give well-defined descriptions and rule names so that you can keep things sorted. Definitely. So now I have my QoA shipper as an actual rule, which then I can add to my policy. And I'm going to use that same one on the implementation. Edit it. Then I can choose it. Now I can roll it in to my report. Here's a key information for you: when I go through here and I add things to a report that's existing. If you notice this time here, it says last update was 6/16/2015. Well, I've updated the rule sets. If I go in this report right not it's not going to show that rule. You have to actually update it and you can now just click on the report you wanted and hit update selected. Now it's waiting on it, so it's going to go out and it's going to re-gather the rule setter for the report. Then it's going to run it against what we have chosen it to run against. Right, and that's one of the things that I noticed about policies, also. Just the canned policies, is that I'll run the policy compliance report, I'll see that I have a security violation, I'll hit the fix it button in the execute repair action. And then I go back to the report and it's not fixed because I forgot that I have to actually re-gather the config off of the device and then re-run the checker right there. So that's just a reminder that it's not quite instantaneous, you still have to pull the config back and do all that stuff. And that allows you to adjust and work with your reports without going one-on-one and pulling everything from all these devices. So you can adjust these, and move them around, then update it once, instead of doing it manually, and you're just constantly going out and downloading and gathering your information. So now that I go into the report, It shows the violations here. Yes. So I can click on to it and tell it to execute on this node. And now, this is where it comes in, because we chose to use that config change template, because it knows we're going to go and find the script for this, like I just showed you to do. So now I can either, if I have one that is saved I can load it, that was already available, which I have one here. Or I can use one that I've already grabbed, that I just created from the change report. Right. So, either you can copy paste or you can pull from the library of fixes that you've got. And you can validate it, and you can actually run through and do the execute script. And when you execute it, it will go through there, and get everything done and saves it for the next time when you want to do that. The reason for the config change to be able to use that is so that once you get your policy—say you're working in there and you need to figure out the script that's going to work the best for the selected nodes that you need. Instead of sitting there and trying to figure it out and script it out, you can go to your config change template, go through the wizard, and get it set up. And then at the end, it'll show you all the commands that you need to place within there. Now some will use this as just a script, and some will still use the CLI or the config change template, so when they see the violation, they can actually click on it, execute it, and choose the scripting of what they want. Because you can change your policy around. You may want to change the script a little bit. That's amazingly useful. I mean, definitely some high-end stuff. But I think for a lot of our customers and people that are watching, it will really change the way they manage their entire inventory. Definitely. And I've heard a lot of customers tell me that the config change templates are scary to them because they don't know how to script. And that's because you're hitting the actual advanced modify. That is for scripting, if you are a pro at it. If you know how to script and you know the language, you're fluent in it. Go for it, do the advanced modifying. But if you don't, learn from the people on THWACK that already have them. Learn from the ones that we already give you available. And then you can use the intuitive wizard, and you're just point click in choosing what you need to actually execute the scripts there. And there's no scripting there, you're just drop down, click, and go. And you brought up a good point about using the expertise on THWACK. We have the Content Exchange, which if you haven't checked it out, you definitely need to go to THWACK.com. Look at the Content Exchange there might be a few, hundred, things from Destiny herself. You know those are pre-written, so if they're not the actual solution that you need, they will at least get you more than halfway to where you want to go. Definitely, and if you have any questions about when you're creating these, or the scripting, and the language that goes through there, it's in your admin guide. There's also plenty criteria and available learning education on THWACK, and I'm in there, too. So if you want to send me a message and shoot me over what you're doing, I can usually adjust it and work with you. And, I mean, that's we're we all here for. We want to make sure you're able to use the product to the best of your ability. Okay, thanks again. You're welcome. [NOISE] That is really cool and I'm pretty sure we haven't gotten to that level of detail on the show before either. Well, if you guys would pay attention to the suggestions in chat and on the homepage, you'll see other topics, too. And where would that be? Really? lab.solarwinds.com You can even sign up for reminders to be there to chat live with us during the livestream. And you can post suggestions, if you typically catch us online afterward. We really do base our content on what you want to see, so be sure to let us know. Yeah, they were saying NCM, NCM, NCM, so now you get this show, right. All right, so talk to us about NIST. Okay, so the National Vulnerability Database—we will scan about midnight is when it is out of the box. And you can do this manually, too, if you want to. But we will check your firmware, your iOS, and your ASAs, and tell you any common vulnerabilities that are exposed. Yeah, that's awesome, and the great thing is, you have control over when that happens and the results of those scans. Yeah, and the point I want to make here is that, this extension now, the NIST extension, allows us to do the final, the last mile, if you will, of vulnerability and compliance. Because, you know, we were doing the hardware and saying if that was compliant. We were looking at the config. And now we're getting right into the silicon and finding out if your iOS or your code is also compliant. So it really sort of gets the entire router stack. Router stack. I can't believe you said that. Do we need to put a TM around that? Absolutely. Okay. Legal will make us. Another great thing, though, is that you can actually alert once you start doing things, and it shows you on the report if you made any remediation steps. So it's definitely your tax dollars at work, you might as well use it. Yeah, the database is there, you might as well consume it. Right. Okay, so let's start how to make sure that the database updates are happening. And then you can actually show us the results of the scans. All right. Okay, so the basics. Let's start with the Settings page on how to make sure that the data's been pulled out. Okay, so what you want to do here is actually go to you NCM Settings and you want to be logged in as an administrator or an engineer. And I'm going to go to NCM, scroll down to where you see the firmware vulnerabilities. Go ahead and click into that. And you'd actually check this box. I've already checked it because I've already been using it. And you can see it by default, it runs at 12:30 a.m. You can also change that, if you want. Okay, yeah, probably everything doesn't need to happen at exactly 12:30, sort of pick a random time. Exactly. And also, if you are behind a closed network, we actually sent you the link so that you know how to do this manually and download the database. Which is really handy because, especially so many of the NCM customers, I mean, you guys are actually in secure networks, where you may even need to sneaker net it over as a file. Definitely. So, we can go ahead and submit. Now once you have this, you can run it manually as well. You'll actually see the vulnerability of firmware resource in your config summary. And that will show you which ones are vulnerable. So we can click any of these. These are your CVEs. So this is your common vulnerabilities that are exposed. And we can see that this sends the link that'll actually take us to the website, the actual vulnerability, so that we know how to remediate the situation. Now what's neat about this is that you can actually put in comments and you can actually change things around. And put, if you're not applicable for your company, maybe it's a low priority for your company, then you can actually change this around. In the all nodes, you can apply this to, or select the nodes that you want. Maybe some of them, it needs to be remediated because they are forward facing, and some of them are in the back, and it is not a big deal, so you can change them over to a waiver. And we can just submit that. When you do this, you can also look at the reports. And it shows you where and what has been changed—so, if we've changed it from potential, if we changed it from remediated, we can see what has happened. So we can go into the reports and we can go into the audits and it's the vulnerability state change. And that's in the Orion reports. As opposed in the NCM report section. Correct, and that's when we're able to see when the remediation plan was set, if it was successful, the vulnerability. This one changed from potential vulnerability to remediation planned. And that's how you're able to track. And what's also nifty about this is that you can set up your alerts and the difference is, instead of going into your "insium" settings, you want to go into the main settings. Because it's a regular alert like anything else. It uses the core, so it's able to actually use your custom properties between anything, especially if you have other software components. So, it really help when you're trying to zone in on your alerts. What, like if you have a particular area like a HIPAA compliant section of your network? Or maybe you want to do really rapid remediation versus everything else in the network, so you can actually decide which portions you might want to take an alert on, and which ones you wouldn't. Exactly. Custom property, like give me an alert. [LAUGH] There you go. And a lot of people use custom properties to put in like, if they have an escalation manager, what the names are, and things of that nature. I'm going to go into the Manage Works. Then, you're going to go ahead and you can set up. I'm going to change the grouping here. I want to put this by the object type. I’m going to go to an auditing event, the NCM audit. Now I can see those vulnerability state changed. I can enable this; I can show you what it actually consists of so that you know anything that gets changed in the vulnerability. So from potential, or if you're going to use it as remediation, a waiver, it triggers. So we're able to report on it and we can also sit here and send this out to people that need to know. Is this informational? Is this critical? We can set all this up and we can see that the trigger actions will actually show you what it's doing is an NCM audit. And so, it's the date and time, the action. And you can add to these. We know for the intelligent alerts, you're able to adjust these to fine tune them to your company's needs. But it's nice to know that if you're wanting to keep track and you're trying to track changes with your vulnerabilities, we have a way to do that through the reporting and also with your alert actions. And this report, I'm sorry, this alert, comes out of the box. It's one of those ones that we include automatically. So it's not something that they have to go out and configure themselves. Well, what I like about it too, is it's the vulnerability state change. So, you may have become vulnerable through no fault of your own, and just the database is identifying a new vulnerability. Those are the ones that are really, really important. I mean, hopefully they're not all zero day. But at least it's been discovered. You think you're okay. You've been policy scanning for a long time, and now a new vulnerability has been discovered. Those are the ones you're particularly going to want to take action on. Definitely, and what's even better is that everybody can customize them. So yeah, it's out of the box, it gets you started, but you can get as complex as you want. And that's pretty much what we're going to do with your vulnerability scanning. Oh, that's awesome! And Destiny, thanks so much for being on the show this week. It's my pleasure. I'd love to come back again. Is this the part where we get to talk about all the fun we had at Cisco Live? Hey, what happens in San Diego, stays in San Diego. Or we post it on THWACK. Or both. In most cases, although you were pretty smart to leave those pictures off of a Geek Speak. The pictures? Oh, those. We've had pretty good time out there. Remember everyone; please keep sending your suggestions to our home page, which is, of course, lab.solarwinds.com. We look forward to seeing you for the next live broadcast. Destiny, where can viewers catch up with you? You can find me on THWACK as Dez. And you can find me on Twitter as dez_sayz, and I think that's it. Let's wrap it up, guys. I have work to do. You bet. I'm Patrick Hubbard. I'm Leon Adato. And I'm Destiny Martin. And thank you for watching SolarWinds Lab.
