Home > SolarWinds Lab Episode 49: Automation Domination at Your Scale

SolarWinds Lab Episode 49: Automation Domination at Your Scale

Watch as the Head Geeks explore automation domination that scales to suit your needs. They will cover Virtualization Manager’s recommendations engine, and Network Configuration Manager’s compliance report automation.

Back to Video Archive

Episode Transcript

Welcome to SolarWinds Lab, I'm Destiny Bertucci. I'm Kong Yang. And I'm Chris Paap, product manager for SolarWinds Virtualization Manager. Thanks for having me on this episode. [Destiny and Kong] Welcome Chris! We have a packed show today. We're covering automation domination, at your data center scale with Virtualization Manager and Network Configuration Manager. The virtual "data" center abides. Automation is certainly a hot topic for IT pros. And there's so many different facets to including automating your monitor, and also to implement management with discipline. Agreed. That's why we'll start with a conversation around what is automation, and some best practices for monitoring automation. And we'll dive into Virtualization Manager 7 and walk you through the new features, including recommendations for automation. Don't forget about Network Configuration Manager, the compliance reporting automation with Connie and me. That's plenty of goodness right there. And we'll end with not one, but two more guest appearances. Chris and Destiny, the big question that we're going to peel the onion on is: What is automation? You know in its basic form, automation is packaging any workflow that's repeatedly taking up time and creating it in a way where you can wash, rinse, repeat. Taking out any variable out of that that could cause changes in your environment, so you get a structured response every time you run that. Consistency, right? Correct. Yeah, I'm going to have to say that a lot of people shy away from automation, but they don't even realize that they're doing it. Because the manual operations that you're doing that get automized, is like logging into a router and downloading config. That's automation. We're already doing that with Network Configuration Manager. So you've got to take it a little bit further. Something that I used to always say is, if I have to do it more than once, and I can create a script for it, it's probably not going to be done three times by me. Right? Like, you've got to script it out and get that done because that's just an extra time on my hands of manually doing something when I could script something out that's repetitive. I completely agree with that. From a system's engineering perspective, a lot of things that we do from a scripting perspective, that we automate, are things that are time consuming, things that take us away from long term projects and goals, that are just nit-picking and putting out fires or that help us get ahead of fires. And then from a Virtualization Manager perspective, automation rolls right into recommendations, right? Like taking away those small things that kind of take time up all day, and providing a workflow for you to solve a problem. Something that I like with Network Configuration Manager is automating a lot of those security policies and the things that need to be out there in place for standardization. Because when you're actually thinking about it, there's certain things that have to be on a basic level, right? The things that you want, and you want to make sure that they're there. So when you have things come on your devices on your network, you want to be able to run a compliance report that actually says, "This is out of compliance." "You don't have these basic functions there." And then we can automatically remediate these. And this doesn't have to mean, oh, we're developing some crazy software networking-driven type of remediation. We can literally just say that hey, if we're supposed to be sending Syslog over here so we can track an alert upon these devices, then we can automatically place that on there if the device comes on and it was accidentally forgot, or it left outside of the config. So there's way that you can use automation that aren't necessarily changing the scope of your device or changing the routing, or doing anything. But you're making sure you're able to monitor it immediately when it's out there by having basic things that are available. So what I've heard, and to quickly summarize what you have just stated. It's two things, right? You got scale, where you can scale the things that you do and touch the number of devices when you have a one-to-n type scenario of doing the same things. So you get that consistency of pieces in there. But also, I heard policies and best practices into the templates, right? So that you have that high level of consistency, so when you are doing these things multiple times at a time, you don't have to worry about the level of quality. Now my next question is around automation monitoring. Who watches the automation piece so that it completes correctly? Or does it efficiently and effectively, versus, "Oh yeah, I scripted it," or "I did this template." And I've enabled this workflow and all of a sudden, it's like, I don't have to worry about it. But we know better, right guys? That's right. Common scenario: somebody has a manual PowerShell script that they run every morning, or they expect it to run every morning. And then a week later, it's not running and it's someone that did something innocuous, like removing snapshots and then they run out of space because all the snapshots are on there. What's actually managing that? When you come to scale, you hit it on the head as you get bigger and as you grow your enterprise is hopefully growing with you. And you usually don't have time to watch every single thing out there. So I think that's key. What's going to be triggering you to know that that script did run, or if it had unintended consequences, such as somebody getting in and modifying it. Absolutely. Or when you have many of the same network devices, right, that you have to manage. And you have to make sure that they're on the same firmware or software package in there, and have similar configurations so that they can handshake with all the connected elements. Yeah, and so being able to have some kind of auditing, to be able to have some kind of reporting functionality that lets you know that the basic function levels of scripts being ran. Or in my case, is there protocols in place or not in place? You need to have a weekly auditing report. You need to have some kind of accountability within your— any kind of monitoring system or management that you have so that you know and have a way to verify and check that yes, these things are implementing and being done. Absolutely agree. So, the two things that I've heard from both you guys is optimization is needed with automation, because if you don't have a purpose to reflect efficiency and effectiveness, then you'll really not quite sure if you've automated goodness, right? Because if you automate something that's bad, it's just going to propagate that badness into your data center environment at that scale. And then the other piece is, a key piece: reporting. You have to show good. You have to be able to show and show to decision makers that the decision to automate these workflows has resulted in something that's positive, right? And accountability is teachable. So, you can have your accountability and have your reports. You can have the information that comes on there. So when new people come in and maybe they don't understand, or maybe you're bringing them up to date on your policies--things that you need to have done--you're able to show them the reports previous and how to compare, and how things that need to come across there. So it's a teachable thing that you can do, and it's very easy, especially when you have software that actually allows you have the accountability and reporting available. And what I love about it is it breaks silos. Because you have to break silos to go across all the different constructs and layers. And we're going to show you that with our next two demos. Chris, oftentimes we ask when a new release comes out, what's new? So what's new in Virtualization Manager 7? So, besides the numbering on it, we've gone to a 7.0 from VIM and VMan. We're really excited about this release, we've got a lot of great things and we expand on what we do today great. Our all-active virtualization is what people really leverage in the environment today. We still have our Hypervisor virtual alerts that are right out of the box. No additional configurations necessary. So as soon as you set it up and you poll your vCenter, Hyper-V environment, you should get these without any additional configuration. And this is great for the virtualization admin who's a generalist, right? Who's been thrown the responsibilities of virtualization on top of everything else that they're doing. Correct, and if you're a veteran of THWACK, you know that we have some posts out there that some very intelligent contributors have added out there, and you know what to look for when you're monitoring within like co-stopper CPU. But this kind of puts it into more of a readable format and pops it up, so you don't have to be an expert, and you can learn from it as well. Awesome, and that's just right off of the Virtualization Manager landing page, right? That's correct. And once you integrate— so, our features, what we're moving to and the general trend within Virtualization Manager is we're migrating all our features set into Orion, native Orion. We do have an appliance that's installed, it's very simple setup, that's installed into your HyperReader or VMware environment and then you integrate that within the settings of Orion. So you go in there, and so this data will show up. So our all-active virtualization alerts that we see on the home page, on the virtualization summary page, can be found also within the appliance, but we're rebuilding all our goodness now, all our new features set, is on Orion. And so from here, the benefits, to kind of expand on Orion, are alerting, web-based alerting, or reporting. So in addition to the active alerts and alerting system native to Virtualization Manager, you also mentioned the sprawl feature. Correct. So, a lot of our customers go into Virtualization Manager today and one of the features that they get the most value out of is our sprawl functionality that VMAN brings to the table. This is kind of an aggregation of a lot of different counters, and it gives a very easily readable format to show you not just top ten, which we do use, but it shows some issues that are in your environment that are common in most virtualization infrastructures. That you can easily see what's being affected and then address those with a general recommendation and be able to take action on them. Yeah, so. What we're looking at is resource capacity planning type issues, right? Over- and under-allocation of vCPUs, vRAM, and so forth. But also, to your point on issues that vradmins face a lot, you also covered things like snapshots, right? So right, you touched on it. So right sizing of resources, reclamation of resources, the things that kind of sneak up and bite you. You know things that a lot of the virtualization engineers go out and they run these crazy custom scripts in the morning or at the end of the day to run. For example, Snapshot disk usage. Once it gets so big after a while, you tend to have a performance impact of your virtualization server, or your VM. So to speak. So we kind of identify things have gotten out of control. And those are things that you didn't do anything to, but maybe backups didn't clear them up, or backups created a snapshot and it's just been growing over time. Or, an admin thinks that snapshots equates to backups. Right, which is very common. And in our example here, that's actually a real example that's in our lab environment, of all places. But orphaned VMDKs is an example of that, as well, where just over time, you have the option in VMware to either remove the VM from the tree or to actually delete the whole VMDK file. And so it's a great way to reclaim space you didn't have— that you didn't know you have, by going in and finding out what's just been orphaned and what's not existing in the environment. Absolutely. And an orphan case comes up when somebody in vCenter deletes a VN but actually doesn't release the actual storage. Correct. It's the option of removing from inventory. So they remove it so they don't see it in your virtualization tree within vCenter, and the actual flat VMDK file still exists on your storage. Now the interesting thing is this third notion of actions that one can take, correct? Like the "delete the data store file." And in the other example, deleting the snapshots, right? But even alongside that, there's other management actions such as changing the amount of virtual resources that you give to a virtual machine depending on its own contention or whether it's wasting resources. Absolutely. So, before our seminal release, we did already have management actions. What our customers did generally use them for outside of a one-off of modifying memory or CPU, is they generally set a threshold, and they'd set an alert to the threshold. So if they had an alert on an important VM, where they were running at a high CPU utilization, a high memory utilization, they would go and set that alert and the trigger would be your management action to change CPU memory resources. And so, you can set that within your alerts as a trigger, and they would be able to modify that on the fly. The great thing about this is you see the recommendation into it. But you also see the steps that we provide to doing it. So, know exactly that we're going to power off the VM, we're changing the config, and then we're going to power the VM on again before you take action. Correct. And the key still here is it's still up to the engineer to provide their know-how, their contextual knowledge of their infrastructure, to be able to make this change. So it's not flipping a switch and it automatically does everything for you. You still have to set it up, and you still have to key in on doing it. So there's no accidental way of modifying your whole infrastructure. Now that segues right into what's new with Virtualization Manager 7. Correct. Yeah, so what we're excited about and what's with 7, is we spent a lot of time and effort into developing a recommendation engine that takes these actions that we have and then it provides a remediation to go out and make a recommendation that the engineer, whether it's a generalist or the expert, can understand. So what we don't want to do is something where right off the bat, it's very complex. We want to stay inside the structure of what VMAN provides right out of the box. And so, what we do is, we provide it in the context of Virtualization Manager to provide an easy step-by-step guide of intelligent actions to take. Not just that you need to move a VM, for instance, but like where to move it to. And take into account what we do best. We do monitoring very well. And so we know what your historical trends are going to be, and what your workloads, so we're not telling you to move a VM that you're just going to have to move back later, is a perfect example. I love how you mentioned context. Because context is needed, especially when you're automating. Without the context of optimizing so that you can get a return on investment, an ROI on the efficiency and the effectiveness of your automation actions. There is no real value to that, right? You need to be able to optimize and report off of your automation piece. So, with that, what does the Virtualization Manager recommendation engine provide? Or allow one to do? So Virtualization Manager will address your environment in two ways. One is, we'll provide active recommendations. Which are the recommendations to address an issue that's already occurred. So these are an after-the-fact or firefighting type of response. They're designated when we're looking at the All Recommendations page here; they're designated with "now" label. When you go into them, it'll give you more details of what's going on in your environment, as well as identify the active alerts that will be resolved by taking these steps. And then, as I've kind of repeated before, we want to be able for the admin to go in and validate these. And we want to give all the corresponding data that we've made this decision on. And so, as you go into this, yes, you have the alerts that have already triggered and so that will be resolved, and you can actually link back to them, you can go back to the alert. But you can also see the statistics that are part of this. Whether it's CP load or memory usage of whatever we're providing on the recommendation. We're not asking you to trust us. We're actually asking the engineer to validate their setup and their infrastructure against this. So this provides context, basically baselining one's virtual data center environment. Correct, but the key here is, it's taking into account like, intelligent trends, right? So we all know that not every VM is running at 100% at all times, right? Unless it's just been severely under-provisioned. What ends up happening is, you have workloads that are, VMs that are designed to accommodate a workload, whether that's two to four in the morning, and the rest of the time it's at 2%, that normalizes out. But we identify with our trends like, hey, that's when it needs its resources. Or, if a bunch of VMs are spitting up at the same time, like they're all doing back-end processes, in the middle of the day or at the end of night, we know that even though that host doesn't do anything now, it's about to get a lot of load on it. Very much like this SQL database that ramped up at the end of month, end of quarter. Correct. It's a perfect example. And it's ramping down again in there. So you've mentioned the "active now" alerts. What are the other ones? So, VMAN provides predictive recommendations in order to get ahead of an issue, versus active recommendations are an after-the-fact hot fix, or after-the-fact problem resolution. This is preventative work that can get ahead. And the idea is to free up your time. So instead of waiting until capacity reaches 100%, let's head that off. Or if we know that a host is going to be over-provisioned and it's Monday, and we know by Friday, all the VMs are going to spit up to their maximum capacity and set your world on fire, so to speak, we want to get ahead of that and provide you recommended steps so you don't have that issue. So active is reactive, and predictive is proactive. Absolutely. So let's take a look at one of the predictive. Yeah and also, you can easily clear these out. One thing I want to point out is you can filter these out. And the idea being is stuff that's going on in your environment you probably want to address first. So you can filter out by active, or if, once you get those taken care of, or if they're not issues that you really can take care of, you've got to wait for still a change management window and an open window of maintenance. So an example of a predictive recommendation would be one without the "now" designation on it. And if we select this one, we're at critical level is red. It's still following in line with what Virtualization Manager, or all of our Orion products do. Red is critical, yellow is warning. And in this one in particular, it's talking about the memory utilization of a VM that's been higher than the 70% threshold for more than 50% of the time. So the key here is we're not hitting on a spike. So you're not going to have where you bounce back and forth. Where it spikes for 30 seconds at a high CPU utilization and then it drops off. Like we're doing something that actually is consistent, and it actually needs to be addressed, and so this way, you're not getting inundated with noise. So it's not an asynchronous anomaly that may just be blip in the system. This is repeatable, consistent, over time. Here it says 70% for more than 50% of the time over the last 30 days. Correct. But, you gave a perfect example. Say if as we're monitoring it, and what looks like a blip over one day or two days. If we notice a pattern in there, that that blip is something that happens as a cyclical pattern, then we'll identify it like hey, well, we need to make sure that this pressure, this CPU memory pressure, is addressed because this a true workload that's occurring. And so then, we'll provide a recommendation for that. Perfect. So, on this particular one, Virtualization Manager is recommending to increase the memory on this VM because of the 70% for more than 50% of the time over 30 days. It's asking to go from 1 gig to 2 gigs in there. And it's telling the steps, right? Because it kind of follows SolarWind's mantra on the product side. Do no harm. Correct. And then, and to that, "do no harm," out of the box by default, we power off the VM for any CPU or memory change. Regardless of if your environment is hot-add capable. Reasoning behind that is: We don't know your particular application on it. So we know that the host itself, the hardware, and we know that the hypervisor will support hot-add. But depending on the application that you put on it, most of the time, they don't wipe or remove over CPU, or add over CPU on the fly. It might not be graceful, and it might not be able to account for that, because it has some dependency with the OS that's been placed on the hypervisor. And to that point, if you don't want to power off, if you know that it's all hot-add capable, you can easily uncheck that. And you'll see that the power off step goes away once you do that, it's dynamic. And then also, most people do have a maintenance window. Let's imagine this is a SQL server that we are going to be powering off, that has a bunch of applications that it's a back end for. And we have to schedule that maintenance window. What you can then do is schedule the recommendation window, and then you have an actual time figure that you can choose, as well as a date picker and as well as a time picker to choose when your maintenance window would be. Perfect. And it fits along the lines of our other edict, and our modus operandi, right. Our MO. Which is, "trust but verify." Because you know your environment, you know when, like you said, your maintenance window is going to occur. And so you can schedule ahead of time to take these actions. You also know when you're going to get hit with a Black Friday rush, or end of quarter, end of month, and you're going to need more resources to finish off the business intelligent queries in there. And so, from here... You can apply the recommendation. We also have the "more actions" button, if you select that. This is something that, a perfect example that happens in your environment. Say you set this up and we're providing recommendations for it. And at this point in time, it's a known issue; you just don't want to do anything with it. Say this is a pre-test, or pre-production, like I air quote that. But that's before it's gone in production, and you don't really want to take action on it until it's gone in production. Or it's something that you're taking down, or you already have a maintenance window scheduled for that. And you do want future recommendations on that VM, but you just don't want it for this one at the particular time. You can actually select "Ignore Recommendations," and you can give it a timeframe once you do that. And so no recommendations will show up for that, I think the default is three days. And so this way, you could actually identify what is noise at that point, but in the future you want to get it for. So it's just like our maintenance window, when we set the maintenance window on our Orion products for alerts. This way you don't shut it off for a long time, but this way you kind of give yourself an out if you forget about it. Wonderful, Chris. And this, I mean, the recommendation engine is an exciting step for VMAN. To step forward, right? Whether you are an accidental virtualization admin. Because I know my friend, Thomas LaRock, fellow Head Geek, loves the accidental DBA. In this case, the accidental vradmin. Or whether you're a seasoned virtualization expert. The recommendation engines can help you, can provide you a tool to enhance what your delivery is as you are continually delivering services to your organizations, and meeting your stakeholders' service level agreement. Correct. It's another knowledge point that you can key off of, whether you are an accidental admin, virtual admin, or whether you are the expert, such as you, a vExpert. And that's kind of— we built that into the product. We have strategies which we're trying to accomplish with our recommendations. And we can actually filter on those. Whether it's storage capacity, host performance, workload balancing, or VM sizing. So once you're going through these, you can identify the importance of it to you, or whether or not, while you're going through it, "do I need to make that move?" And this is another data point that will push you in the right direction. It'll say like, based on the information and historical data, you can confirm your thoughts on your environment. So Chris, you talk about the four strategies that VMAN 7 incorporates into the recommendations engine. What is the basis for VMAN's strategies and its knowledge base? So it's in two basis, right? We have the context of an extension of our alerts that people leverage heavily today. And so the reactive now recommendations, the active recommendations is what we call them, are an extension of that. Taking that knowledge and the historical data of what people are using in their environments set in thresholds, and how do you fix that. So essentially, here's a problem, how do I fix it in my environment specifically? And then, based on the data that we're taking in and the intelligent training that we do, we then take that data and provide, based on best practices from VM or Hyper-V, and based on the counters they provide to us through their API, we then make a recommendation on what to do in that environment. So this data that we're polling is not data that we generally calculated on our own. There are some calculations involved, but from VMware. It's their API that's used today. And from HyperV, it's their WI information that's passed on to us. So we're taking that data and then we're trending it and then providing those calculations to be proactive and get ahead of any issues in your environment. Awesome. So, VMAN 7 encapsulates the active and also now, does predictive. Exposes that out, such that one can start to expand the automation of management actions within their virtual data center environment. Correct. The idea is to have your environment optimizing at its best, free the admin up so he's able to do strategic work in their environment, so those day-to-day, that we talked about, workflows. Everybody's environments different, but there's things that are more important. They tier in their importance in an environment. So their low-risk items, they may set up to automatically kick off and deal with it every day. And then their top-tier applications, they still probably want a hand holder. Making sure that it doesn't bring down their entire environment because of all the dependencies on it. So, it's going to be offloading some of their day-to-day work. And then being able to make quick decisions in a highly dynamic environment with very intelligent background that they can verify. Awesome. And for those of you in the community, you can engage with us on the VMAN product page on THWACK. That's correct. As well as VMAN 411 page, also on THWACK. Request new features. Requesting features at Product 411 page, we'll be having new content going up. Kong and I will be working on that. But as you go through, put it through its paces, and if you see things that are just different in your environment, reach out to me directly. I'm usually the one answering anything on our product page for VMAN. It's me or another product manager that works close with me. So we'll get back to you as soon as we can. Perfect. Thanks for joining us, Chris. Absolutely, thank you. So Connie, I want to show our viewers some basic to more advanced automation with Network Configuration Manager. Sounds good to me, Des. So tell me what you'd consider basic automation. So in the ‘basic-est’ form, you can think of just logging into a device and backing up a configuration. That's automation with NCM. We can also look at, if you have syslog, and you want to actually be able to use that in your real-time change notifications, we can use that as well, because we can actually roll out those changes. So it can go all the way from there to automatic remediation. Now something that I want to do is kind of baby step you into using the config change templates, so that you're able to see how scripting works. And then also use it as a validation. And then use those in your compliance. So we'll kind of jump around, but I want you to realize that there's a lot of moving parts within NCM that's at an intermediate level, that I just feel like we're missing. So I want to help baby step us in to getting comfortable with them and just using the out of the box. Okay. Sounds good? Yeah, sounds good. All right, let's jump right in here. So I'm going to go into my dashboard here, and go into the configs, and go to Config/Change Templates. So when using a real-time change notification, you'll use things like syslog or yrap. So we're going to do it with syslog here, and it's real easy to search for these, so I'm just going to hit syslog. And then, I already have this one out of the box. Enable syslog. So I'm going to go define variables and run. Now this where people kind of stop and freeze. And this where we're wanting to kind of open this up for you. So you're going to watch me go back and forth here and show you where you're actually going to execute something, all the way to like, hey, I'm not executing, but I can see what's going on here. So let's not be scared. We'll do this together. So I can pick the vendors, and I'm going to hit next. It's not doing anything yet. I'm defining the variables. So this is this actual syslog server that I'm going to tell these devices to send all the logging to. So I'm going to pick the box. I'm going to say next. Now, now that it says execute, that doesn't mean that I have to hit that and just wonder what is going on. I can actually look over, as you see here. Here's the devices and the commands. So what is the script doing? This is actually automating out each one of these to these different devices instead of going individually to them. So let's see what commands it's actually giving to these. You can click on the Show Commands window; it'll populate in another window and show you exactly what the actual login events are. So we don't have to manually log in and write, "Configure terminal," and log on, and this is the IP, and this everything that we're going to do to each individual one. It gathers the information from you because you tell it, and then you're able to take it and it'll script it out individually to them and run this at either now, or you can schedule it for a week ahead, a month ahead, one time only; however, you're wanting to do this. But what's key is these are the actual scripts that it needs to run. This is the command line interface, commands of which that it's using. So what I'm going to do is I'm just going to copy this and we'll come back to this. Because we know that's where we're wanting to send this information to, right? Okay, this is the command logs. So I'm going to exit out of this. I can actually schedule this, like I said, or I can execute it to run one time, however we're wanting to do it. And like I said, they'll basically—just like you do when you set up any kind of a job, It's just a scheduler, goes through there, you set it up. So, now. Since I copied that, where's another place that I could probably use this? If I'm doing standardization, let's use compliance reports. Because for auditing, like when we're talking about accountability especially, you want to have a report, right? And a lot of the times, we want to have a report that we can grab and schedule to run, and it shows us everything that's laid out. And it also shows you what's new. So let's look at the compliance reports, and how I can turn this into something that's kind of a little bit fun to use with it. So I'm going to go into Compliance. Now remember we go Rules, into Policies, into Reports, kind of like a pyramid, right? So any rule you want to figure out and put together, you just start setting these up. And this is anything with your actual business. If you want to use one of these reports out of the box and just kind of skim off of it, you can do that too. So we're going to go to Manage Policy Reports. I'm going to go into the rules, because that's where I start from. I'm going to search—see, there might be some in here that have syslog. And there is. So we have syslog with Cisco, or we also have it with Juniper. So I'm going to hit the Cisco one, and I'm going to hit edit. So this is telling me that it's going to look for SNMP server, enables trap syslog. Enables syslog error message notifications. So this is what we're trying to make sure that we're doing here. So what I can say is, all right, if it doesn't find this on any of those devices, I'm going to right click and paste. But I pasted this from the Config Change template, right? So I used a wizard within Network Configuration Manager, created it based upon the variables that was set up, grabbed the script, and now I'm using that same script from the Config Change template, that's validated and verified, and put it into my compliance report. So now, I can actually automatically remediate this violation, if it's not found. Why would I need to do that? Because if we have devices that come up, right? Like, if we get new devices on our network, if we get things that come across there. You may configure it differently than what I configure it. Or maybe you're sending this syslog to what you thought was the IP address, but here I am monitoring and managing it. So that whole, "how are we actually monitoring the automation" part of it? I can run a compliance report to make sure that you're busting your ****, right? And you're getting all the network devices out there. But maybe we just haven't told you, "Ahh, I switched to a new Kiwi Syslog server, and the IP address is actually this address." Or you know, something else that's going on there. And we just didn't connect on that. But in my compliance report, I'm automatically verifying this. So I can run this every day, I can run this weekly, I can run this monthly. But what this is going to do is any time when I click this "Automatically Execute This Script," when it's found, it'll automatically change the logging so that it'll actually go to the Kiwi Syslog server of which that I set up, that you didn't know about. But instead of, it's kind of one of those "CYA" moments. You know, we're just making sure that something is verifying our automation that we have out there. So that we know that we have those real-time change notifications, and we know the things that are coming across. Now when we're doing scripting, a lot of times people get scared because they're like, well if I don't know what's right, what is it going to do, right? Scripting scares a lot of people. Exactly. But we have that actual Configuration Wizard-type thing, with the Config Change templates, right? Right. So I would say that was the best place to start. But let me show you when it doesn't go how you want it to look. So everything happened, right? Like, "Ahh!" You know, I don't know my scripts, or maybe this is a violation, now what? So let's look at how easy they've actually done the wizarding, I guess you could say, behind it. I like to make up my own words. And so we'll look at the syslog. I've already downloaded one that was incorrect. And it's this one. So if I go here and say Define and Run, it's errors, automatically. So this is a great thing. So don't be scared or worried if you're like, "Ahh! It's ‘error-ing’ out and I don't know what's going on here." That's perfect. We have checks and balances in place so that you're not doing something that's going to mess your network up. You're not doing something that could possibly cause a problem on your side, like the wrong interfaces, or the wrong things that are going on that you're not using. We actually have these built in. So what we're going to do here is edit the template because once it airs, it'll actually allow you from that location, you know. You know, the retreat, and we can do something about it. What's great about it is that we also have our error detection within there. So we can scroll down here and it automatically tells me that it's line 62, character 38. So I go across here, it already has it highlighted here. And if you're new to it, you can just kind of look at the script. So this is something that I learned when I was in school and was doing stuff. When I was going through my Python training, they were like, "What's different?" Like, you know, just look at this box and tell me what's different. So if we're looking up here at IP addresses, and we're looking at stuff after an equal sign, what do you notice that's different between those two? I see single quotes. So let's try that, right? So that's a great observation. There's the difference there. There's the delta. So let's move this. Let's put it as a single quote. And I can say ‘Validate,’ and there you go. So you don't have a scripting degree, right? No, no. But you just figured that out by just looking at it, and being able to have those checks and balances, right? Yes. So that is the perfect part. Do not be scared. Play around with it. You can use this to help you get better at your scripting. And if you go out to THWACK, then especially download those, because the great comment fields, they allow you to see work with it, right? We can go from this script and look at it, go to the defying variables, and you see how they play with each other so that you know, oh, okay. So now, I can do it based on vender. That's what the node context, so I can go to the description, I can go to the label. And you baby step yourself into it so that you understand what's going on with these. I think that's pretty fun. It is fun. And it's safer, and it makes things a lot easier. Oh, I think it makes things a lot easier. Let's get Kong back in here so that we can actually kind of finish this one up. Wow, automation can have many connotations. It does break down silos. But to automate correctly, it requires a circle of trust. We heard circle of trust, so we came by. So did you have fun automating against my demo environment? You didn't break anything, did you? Wait, we haven't even covered automating application monitoring and app templates from SAM. That's a whole other episode entirely. P.S., I think Leon already has dibs on it with you. Can't wait. So for SolarWinds Lab, I'm Destiny. I'm Chris Paap. I'm Connie Dowdle. I'm Kevin Sparenberg. I'm Kate Asaf. And I'm Kong. Thanks for watching SolarWinds Lab.