One of the Orion® Platform’s most powerful features is the ability to get alerts when you need them, and more importantly, where you need them. In this episode, we focus on alert integration for Service Desk and internet live collaboration platform Slack.
Head Geeks Patrick Hubbard and Leon Adato, along with Destiny Bertucci, will demonstrate everything you need to know for making webhook calls to REST APIs, how to call services form SolarWinds® Orion alerts, and even some advanced customization and PowerShell. Take your alerting to the next level with help desk integration and live interactive notification.
We’ve even created an Orion and Slack integration eBook as a companion to this episode. It includes full examples, troubleshooting, and advanced ideas to get your imagination going.
Episode Transcript
Hello, and welcome back to another episode of SolarWinds Lab. You know, I don't think I've started us off in a while. No, I don't think you have, but you're in a really good mood, so we'll let you. I'm Leon Adato. I'm Kong Yang. And I'm Destiny Bertucci. And I'm Patrick Hubbard. And of course I'm in a good mood, because this episode marks three years of SolarWinds Lab. True, and I've been watching it from the beginning. Leon, that's how you ended up here, too many comments in the live chat. That's true, and Sparenberg also. But seriously, in this episode, we're going to do something awesome with alerts and external system integration. And many of you are watching live, which is really good, because you're bound to have questions. So ask them in the chat box over here to the right. Or do it the way that I do it. Pull the chat tab into another monitor and go full screen for both video and chat. You do it on your Surface? He does, but if you're not watching us live, be sure to swing by our homepage at lab.solarwinds.com. And you can sign up for reminders for our next event and you can ask us questions live. It also gives you a chance to check out our back catalog of over 40 episodes. That really is a lot of shows. So what's harder, writing the show, or setting up all the demo systems? You're kidding, right? Yeah, I'm absolutely kidding. I'm playing Patrick's rhetorical question game. Of course, setting up the systems is by far the most work—I mean, fun, FUN! Nothing but fun. So, what are we doing today? We're going to integrate Orion alerts with Slack and service desk using REST. Viewers have been asking for that forever. Well, I've been asking for even longer than that. [LAUGH] That's true. Okay, so here's how we're going to do it. First, I'm going to set up cURL and make some rest calls to Slack form the command line to make sure that it's working. And then we're going to do it the right way with a PowerShell script, so that cURL is not required. No cURL at all? Not even a hint of starnexty goodness? No, no, no. We're going to show cURL because it's fast, and it's really easy. And you can also make sure that it's working, but you're really going to want use a PowerShell script because although sending your team alerts in Slack is really easy, service desk integration is a lot more complex. It has authentication and more fields in the JSON, so you’re really going to need a reusable script. And you're going to show all that? I am going to do that, and then we're going to have cake and a toast to three years of SolarWinds Lab. Nice. Well, I'm going to let the three of you guys handle that, and I'll be back at the end of the show for the toast and the cake. The cake is a lie. I wouldn't be too sure about that. Okay, so I know I go on and on about APIs and Rest and JSON, but what did you ask upstairs earlier? I asked you how long did it take you to set this up? Yeah, and he was pretty shocked because literally what I'm going to show you would have taken me less than five minutes to set up. Now, I am a programmer and I do play with the stuff all day long. But the point here is: you can totally do this and all the resources that you need are already out on the internet. We'll have some links below to show how you to do this, especially the set up for Slack. And some of the other things that you're going to need to do, but this is really straightforward. And it's a great example of how flexible Orion actually is in terms of alerting. So don't worry about, "I'm missing a particular integration for one package." It's more open than that, so that you can integrate alerts with just about anything you can think of that needs to receive alerts. We're doing this one, because—actually, you did request this—and it is kind of fun, and I think you'll get a lot out of it. And I think you're going to enjoy this because you've been doing alerting forever [[LAUGHS]] Pretty much. And have watched us go from just basic alerts... Up, down, up, down. ...to now, finally, web-based alerting. You know, I hate to ask this, do we still do the serial out for the modem for pots? Awesome. It's still on there. That's good. I haven't looked in a while—I forgot about it. Okay, now this is how this is going to work. I'm going to set a cURL on this machine as the agent that's going to send the REST of the command out. You guys are supposed to remember REST, right? So it's restful service URL, I can send things out to it. Somethings going to happen. And that's going to sit on top of something called a webhook. You guys probably already know what webhooks are. You guys know what webhooks are, right? Yes. Webhook is just a URL on the internet that is attached to some sort of a process. And in this case, it's connected to a Slack receiver that's going to then do something with it. So the first thing that we're going to need to do is tell Slack where to send this, right? So, what we're going to do here is, we're going to take a look at our system now. Thanks again for setting up this machine for the next episode of Lab, it's really handy. I was able to piggyback on that, so that's all right. My pleasure. Okay, so here we are with this live dashboard. And remind everyone what this is. So Slack is, at its first blush, it looks like an instant message platform. But it is actually more than that. It's a way for workers to collaborate. There are add-ons that will let do everything from silly things like getting cute cat pictures from the internet to logging ideas. You can say slash idea and type your idea out and it will file it in a way where all the ideas are together and searchable. So it really is a collaborative work environment that goes way beyond instant messaging. Lots of teams find it to be very, very useful. And Destiny, you really like organization, right? Yeah, and the way that it keeps everything segregated in its own terms. It's very easy to search and get things that you need out of there quick. And the main thing is that lots and lots of you have been using it, you've been asking us to show how to do this. And I thought this would be a great thing, to do that, and satisfy you at the same time. Yes. So, here's what we're going to do. So the first thing that we need to do, if you're familiar with this, you know what I'm doing, I'll just tell you: I'm basically logged in here to Slack, I'm looking at a channel which is... A realm, a category. I think more pub sub, right? It's just a receiving endpoint. So everyone is subscribed to that. Anyone who is subscribed to this channel is going to be able to receive messages to that channel. So, we are all on this one, for example. And the idea being, I get an alert of a particular type. I would like for it to show up here. And then we're going to talk back about it back and forth. Decide whether we want to create a trouble ticket or not. We might do a lot of things. Just in terms of quick troubleshooting and kind of bat it back and forth so that we don't have to just throw it into the help desk system. And then use that. Right. To then take 15 minutes for what we could solve in two seconds. Right, and this goes into something I talked about a lot of times, which is that you've got the monitoring system, which is collecting information. You have the alert trigger, which is what's going to tell you. But then you need something that's an effective poke in the shoulder. Now for some systems, a ticket environment is the poke in the shoulder. But really, most of the time, you need an interruption--something that comes up and tells you, and not just you in your instant message, but a group so that everyone can say, "oh, I got it," "nope, I got it." That kind of thing, so that's what we're looking at here. So back to the DevOps-type mentality, if we were all on a team of network engineers and SysAdmins and we're wanting to keep it to date, we would be attached to this channel. Mm-hm. So hence, we're able to get all the alerts, and we're on the same team there. That's right! So the only trick is, how do I publish this into this channel? So there's a couple of different ways you can do that. The easiest one is—and you need basic write permission to the channel to set this up—so, you just do Google add webhook for Slack. It'll take you to this page. Click on "incoming webhook integration." Of course, by the time we show this show, they'll change the UI and this'll all be different. But it'll still be easy, so it doesn't matter. Choose the channel that you want to add the integration to and then click "add incoming webhooks integration." And then right here is the webhook. This is bad CSS. It breaks across, but that is actually one string. And then copy and paste that somewhere where you will know that you'll keep track of it. Now if you forget that, in the feed it will say, "Hey, somebody added a webhook integration to this channel." And if you click on the link here that says incoming webhook, it will tell you what that webhook URL is. So, as we do these demos, that URL is not going to work anymore. It also lets you shut those down, so that if for some reason you hand out one of these URLs and it gets out there—because these things really are public—you can pull them out of... Maybe you have it on your lab episode 40 show and people are trying to then send you messages. That's right. "Help me!" The other thing is, I'm going to publish all these scripts out there, so you guys can see them, and I'm just going to leave the old ones in there, so you could see what they look like. Okay, so here's how this works. We've captured our URL. So the next thing we need to do is get cURL set up on Orion. Now the way that I did that. I mean, you guys have obviously done this, download cURL for Windows. I set it up. And thank you again for this very Linux-y structure. I'm still trying to get user bin here figured out, but SW files, I just set it up in here, and there's the bin directory with cURL in it and then all the goodies that it needs. So, the other thing you're going to want to do is remember that path, because you're going to need it. You could, if you wanted to, add it to the path. But I don't like to do that, especially in scripts. I like to call the explicit path so I know exactly what it is and it's not some question where, or which instance, is it executing. PowerShell being another great example of that right? Okay, so cURL is set up. I'm ready to go with this first example. So, first, I'm going to need an alert to hook into, right? Oh, I'm sure we can find one of those. Well, I'm going to make one from scratch, just so we can remember what those steps look like. And the way that I'm going to do that here is, I'm going to come over here to my interface, right? I'm going to say settings, alerts, and then I'm going to add mine. So, I'm going to add a new alert. And I'm going to give it a name. So, I'm going to call this one 00 just to keep it at the top. Mm-hm. We're going to see a bunch of your 01s in here, I think, in just a minute. Correct, it's my standard MO. 00, and then let's do Slack update on interface status change and I'm not going to do a description, we know what this thing does. The idea being, anytime an interface changes status, I would like it to send me a note. Generally, I'm not going to care, but every now and then it's going to go down, and I'm going to want to know what that is. I'm going to evaluate the trigger condition every minute, I'm going to make the severity warning, and the response team is Geeks, and there's no limitation on this category, so I say next, and we'll look here for interface, and what we're looking for is "status has changed." I'll say select. I'm not worried about events in a filter. Normally I might do that, especially for escalation or condition or something else, but I'm not going to worry about that. I'll say next, and I'm going to set my trigger condition. I want an alert on an interface. And then the field that I'm going to use is— instead of a field, let's actually look at events. And remember: events actually store data in the database, and then they look for a change in that value, as opposed to a regular field, which is actually going to parse out just the value, and evaluate right then when it executes. This also means I'm not going to have to do a reset condition as well, because I don't care. Any time it flops, I want to know. So, I'm going to say status has changed, say select. I'm not going to worry about matching filters. I'll say next here. Just keeping it simple. Keeping it simple. I mean, this is really a good one, right? So, now I got a reset condition, I don't care about a reset condition because it's never going to reset. Just going to toggle back and forth on values, time of days, all the time, and then my trigger actions. So, here's where this gets fun. So, I can display a bunch of messages when this is triggered, which I'm not. I could lead default, but if I wanted to, I could put a bunch of variables there. So this one I'm going to say, execute an external program. Now you might ask yourself, why not execute a BB script? And the answer is: this works much better. You could do the same thing in a BB script if you wanted to, but the capabilities of PowerShell are just much, much greater, and there is one thing that we're going to have to do to do that, because you're going to have to call PowerShell and pass in the script. So it adds a little bit extra to it, but it's not a big deal. So, we'll say configure the action, and now we're going to do this. The action is notify Slack, notify API, and we're going to set the program. Now, I'm going to cheat here just a little bit, because I have actually gone and set up a script to do this. So, I took a little while of playing with this, as I mentioned before but, this really is not a big deal. Let's walk through what's going to happen, because I'm going to paste this whole thing into that line as a command line. It's all executing on a line. And I'm just going to say, it's not a bad— it's not cheating, and it's not a bad idea, because if you have a script that does a set of actions and takes input, you can use it in multiple situations. Whether that is having SolarWinds Orion write to Slack or having some other external source write to, it's not a bad idea to have it modular like that, that's still a very DevOps... Something I like about it is that when we post scripts and do things, people who are just kind of getting their feet wet with things, it gives you a model or a template to kind of go over. So then you're thinking in variable terms, which then helps you when you start using the SDKs and stuff. Because you're like, variables, replace, get out. Makes it a little bit easier to dive into the scripting. Alright, so let's look at what that looks like. The first thing up here is that path to—that I showed you before—to where my cURL executable is. I have the full path to it. And then I'm going to tell it, I want to do a post action, I'm going to want to URL encode the data as it goes and then I've got my variables here for the payload. Then the last thing that I've got is the URL that it's going to execute to, which again, is that webhook URL, so that it's going to receive that action and then Slack is going to do something with it. Now, this is a little hinky in that payload does not actually mean body in this case, but cURL is using it that way, so this payload is essentially a wrapped, escaped JSON string inside of a DOS string. So there's a lot of back slashery here, and some other things that are kind of ugly, but I'll walk you through that. I'm feeling excited about the template now. [[LAUGHS]] Yeah, exactly. I'm glad somebody else did this. Exactly. No, but seriously, it only took a couple of tries to get this working. Okay, so payload equals, and then there's a couple of different variables you can set. You can set the channel, which in this case is going to be lab bot, right? That's where that thing's going to go. Username—which believe it or not, there's no authentication here. That's one thing you're going to want to make sure of, because those webhooks are wide open. There is something that we can do with actually creating a bot which would do authentication. This is not that. So that's just a name, Orion bot, I can put whatever I want in there. That's not actually a username. I just called it that. I should call it like, Orion lab bot. I think that would be really cool. Coming soon to a lab near you. That's right. So then, the next thing is text. And so text is the actual payload instead of payload, because in this case that is the section inside of the JSON object that it's building. And you can see my back slash equals here is my escape for all of my escaped quotation marks. So, I'm just going to say alert on an interface and then I'm going to have some information. So, if you look at these things, like this one right here, these things look familiar, don't they? Mm-hm. Mm-hm. That's just a normal... What would that be? That's a parameter, it's a variable, right? Mm-hm. So, the way I got those, I clicked on the insert variable here, and then I went and found what I was looking for. So, first one was a search for URLs and I found a whole bunch of them, right? So, one of them is acknowledge URL. So I clicked on that, it puts it right down here, and then I can just cut and paste it right from there, and stick it into that script. If you want you, can just say insert variable, it will put it here and you can copy it and paste it out of there, but that's an extra step. But that's a good point, though, because just like they say, when people are trying to get into scripting and try to use our variables, we're showing you right here what the actual variable is so that we can copy/paste. We're not asking you to create it or try to figure it out. I mean, we actually had the functionality in front of you. Well, and a lot of people will ask me, "Well, I don't know which variable I'm supposed to use, there's so many of them." So what I always tell people is set up an alert action, which is just write to file, write to a log file, and just throw a bunch of them in there, and call them what they are. So, alert URL, colon, and put the variable in there. Name, IP address, DNS, sysname, whatever. Put them in there and just run a fake alert once, and see what gets populated. Then you know, and then you can use it from that point forward. And if you're using SWQL Studio, it's going to be able to list all these for you. So you can explore them that way. The other thing is that this variable picker is smart. You'll notice I've got global all variables alerts and interface. Well, why is node not up there? Because this isn't a known alert. This isn't a known alert, so that'll help make it easier. It's not an application alert either, or an NCM alert, so we're only giving you the stuff that's appropriate for the scope that you're doing. That's right. And then the search is also partial search, so you only get what you need anyway. That's how you do it, that's how I got all of the, everything that you're going to see here. I used this wizard to get all of those, because I want it to be easy and fair and not go use my Circle Studio. Okay, so I've got my variables. Mm-hm. What did I do with that in the script? Let's take a look at that. Okay, so I've got my interface, and you could see here, I just kind of broke on this line, and break that over here. So it's the dollar sign, bracket, and then whatever's inside of it is a variable. This first thing: the "n" is where the base object, and the "m" is what I'm looking for. Right, so SWIS entity called caption. SWIS entity called status. SWIS entity from alerting here. That's an alert detail URL. An acknowledgment URL And then this icon thing here is, I want to give it its own icon just so it looks a little bit different, and you will see what that looks like. But the last thing here, again, is that webhook piece. And one more thing here, and this has nothing to do with SolarWinds, this is just the way Slack works, is you have to escape URLs for links, if you want to be able to override them with text. And that's what this greater than symbol is here. So basically it's, this is going to be the URL, and then the pipe says here for the name. So I'm going to get the detail link, I'll get a node link, and then I'll also get an acknowledge by clicking here link. And it should be good to go. And I'm going to take this guy. So I'm going to grab this. Come back over here. And I'm going to paste it right in here. So, send to cycle IPI. I click next. Reset action—don't need a reset action. And I'll click next. Summary, gives its name, all the trigger conditions. I can edit it if I want, I don't. So I'm going to say submit. Okay, so now I'm going to test the action. The easiest way to do that, is I can come over here to my action manager. I'm going to click on this guy, send to Slack via API, and click test. But let's do an all-up test, just to save a little bit of time, but that would be the quick and easy way to test it. Right. But the other thing is that, remember, this thing has got URLs in it, I can actually show you what it looks like in the test here. It's got URLs that link back to the alert instance, and because the alert instance is not going to be populated. The text that comes back won't be complete. Right, and this goes back to the conversation that we have on THWACK a lot, which is about why is the test button not working, or how come I'm not getting it, whatever. Remember that any of those test buttons are going to do a simulated alert. But if—like you said—if there's no down, I can't give you the details of the down that doesn't exist, or if there's no actual alert, I can't give you an acknowledge. Because I didn't actually create an alert, which is why—in real life—when people want to test, I always tell people: just change one variable of the actual alert. Instead of saying over 90, say under 90. Right. Whatever it is, and even if you have to limit the alert by saying "only for this one node," so that you don't get 500 of them or whatever. A real test is one where you just change that one variable and then you do a reset, and let the alert keep happening to make sure. Because that's when you're going to have everything fully populated. Just a little PSA. Yep. That's absolutely right. We'll come back and do it, and I'll show you how to do it through the command line. Because, there's a chance of also blowing this up when you get into advanced scripting. Let's take a look at it. I'm going to fire it, actually, with the console and this is going to give us a chance to talk about polling status. Because what I am going to do for this guy, I'm going to take this guy down right, this one over here on this talk box. I am just going to say ‘unmanage.’ I'm going to put it in an unmanaged state. Now I would love it if it would fire an alert that says "the status has gone to unmanaged," but that's not going to happen. No. Right, because unmanaged means I'm not polling you anymore. Exactly. So, now it's showing that it's gone into unmanaged. It's not polling, the alert is not going to fire, nothing is going to happen. So to get it to fire, in this case, because of the way I wrote that looking for the status change, it's going to detect that it's gone to unmanaged on the first poll. And then there's going to be an actual poll and then the alert condition's going to fire again, and it's going to show us that it's gone to an up. So, I'm going to click on ‘remanage.’ All right. So there we go. So this one has told us that it's gone to an unknown state, which will be the status when we're unmanaged, and then turn it back to managed again. Right? It doesn't know because it doesn't poll yet and you can see that I've got a couple of little options here, right? These URLs I've put in, right? So I can click here and go get the details for that alert. I can also click on the nodes, get the node details, and I can acknowledge it right here if I want to. And now, we've got an answer that it's come back to the upstate. So remember, what it did was: it did a poll, discovered it was in an unknown state because we'd remanaged it. The polling interval was firing on the background on its own, because remember, the alert poll and the actual device poll are different. So then it caught up with the device poll, noticed that the status had changed, because it's an event, not working for just a field, and send us the message, right? That's pretty cool. Very cool. I'm pretty excited, actually. There is kind of a plethora of things you can do with this. Yeah, there are a million things. And the other thing that struck me, as we were talking about this, is that while you want teams to interact with the alerts, and you want them to know about it, one of the questions that I get asked a lot is, "can I get an FYI?" And I am diametrically opposed to FYI alerts. If you don't have responsibility, you don't get a ticket. And a lot of organizations, all they have is tickets. Well, you want to know who's got the baton, who's working the problem. But this would allow you to send a notification, just an FYI, your system, your interface, your application, your whatever, is having an issue, but that's not the ticket. The ticket goes to the team that's actually responsible for it. So it's just another way to spread the information around. That's right, and like in this case I just said: FYI this is a lab machine, don't worry about it. So everyone else on the channel saw that I'm saying I've got it, and then you guys can all go back to whatever you were working on. I vouched for this, this is not an issue, and there is also a paper trail, you can prove it to your boss later. Okay, but that was cool. But we can be way cooler than that. Of course we can. So we really want to do this as a script. So the way that I'm going to do this. I'm going to use PowerShell for it. And coming back here to my files, I just created this little directory here for me. So, I created a PowerShell script and dropped it into that directory. And that's important because we're going to need the full URL to it when we go drop it back into our alert. Alright, let's take a look at what this script is going to look like. I'm going to do it a couple of different ways. The first way is, I'm just going to take what we already had, and I'm just going to wrap it up just as it is now into PowerShell, and just simplify it a little bit. And then we're going to do it the right way. So, the first version is going to be with cURL. So what I've done here is just broken out a couple of things. First, I defined all of my parameters. That makes it just a little bit easier. It makes sure that they're defined positionally, and it's easier to track. I definitely recommend using parameters by name here, don't use them positionally out of the array. And then I went ahead and set a couple of other things. Like, one of them is, I defined where cURL is. Makes it easy for me to check that right here and I can reuse it pretty easily as well. I'm going to, that icon, that light bulb icon you saw. Mm-hm. I went ahead and gave it a URL for that, so I can easily change that. I've defined my webhook that I'm using here for the update, the channel that it's going into, and the user name. So, it makes it really easy for me to change any of these without getting down here and messing with all of this once I finally got this piece working. Because if you misplace individual characters, it won't work, and I'm going to show you how we're going to debug that in a second. So, I basically built all that up into my argument list and then I call the same thing before, which is cURL. And then I pass in the argument list that it needs. Including the string with the actual text. And this, believe it or not, is going to work. So I'm going to grab this guy. Why do I say "this guy"? That's so strange. This thing. This thing. I'm going to grab this entity and I'm going to come back over here and I'm going to I'm going to paste it in here, so I'm just going to keep reusing the same form over and over again, so I don't have to make any changes. And that's the other thing. You'll see how easy this is to edit once you get it working the first time. So, I'm going to save that, and we're going to go through the exercise again. Now, the first thing is that we are going to make a small change to the text that we're going to place inside our alert, right? Because we're going to need to call the script with all of this stuff. So, the first change is, we're going to have to change the execution string from what we have now, to calling this a PowerShell script. Okay, so this is another one where you will need to experiment a little bit better. But you can already see how much easier this is to read, right? So, this is the text that I'm going to place inside the execute command in the Orion alert to get it to fire and call my Geek Slack sender PS1 script on the server, right? So, I've got PowerShell.exe. It would be cleaner to actually put the full path to the PowerShell here, so could you could specify the version number, and know that it's not going to change on you. This is going to just use the latest and greatest on the machine, which could blow you up, but I was just being lazy. I'm going to specify the file that we're going to pass in here, which is this Geek Slack sender 1 PS, that's a follow up we saw just a second ago. And then this will look really familiar to you. This is a list of a quoted values that are all coming from the variables that are coming out of that Orion alert, right? So the exact same ones: the caption, status, alert details, the details URL and the acknowledge URL. And believe it or not, that's it. And the other thing is, I'm never going to have to change this once I get this piece working, if I want to change the functionality of the alert itself. So the way that we do that is, I'm going to say edit action. I'm going to come here, make sure I go with that, put that back in there, and say save changes. Now I am going to test it here for a really important reason, because it's not going to work. Oh good. That's something we can agree on. And why? [LAUGH] It's not going to work, because there is a bug in the script. Now I'm not going to know that it's not going to work, because it tells me that the action executed successfully. What we haven't told him is that we switched his script for Folger's coffee. That's right. [LAUGH] Well, the reason that it didn't work, and you'll notice that nothing has come up over here, right? That's a good point to mention, because a lot of people will say, "But the action executed!" But they have to understand: It did execute the action, but the action itself is flawed. That's right. It still failed. And here's how you can check that. So, I'm going to come back over here to my edit action. And basically, I have a bug right here in this path. I'm just going to grab this again. And I'm going to do it this way, instead of cutting and pasting, because I'm trying to debug this. I could take it from my original source and put it in here. But I don't want to do that. I want to know exactly what's on the machine. So, I'll come back over here to my box, and then, what do I have back here? Ah! A command prompt. I'm just going to paste it right in here and execute it. And it's going to say, "I can't find that." So, even though it gave me an okay in the test button, it never showed up in Slack, there was a bug. So, don't be afraid to test these things. And, in this case, because it's just calling that directly, go ahead and call that in the command window and see what the results are. And this is one of things that we keep coming back to on Lab often, is knowing how things work at the command line, at its most atomic level. Even though it's not something you're going to do every day, really does allow you to do a faster and more thorough sense of debugging, whether you're starting with something like Ping or doing an SNMP walk or doing code debugging like this. Knowing how it's working right at the basic level lets you be able to drill down and see what is going on. And not be so reliant on hoping that the person who wrote the interface to the tool also exposes the alert, that kind of thing. Exactly. Alright, so, I've got that thing saved, and I put the letter "e" back in the path. I'm going to click test. It's going to say again. And it's a good point to make. You do need to give it some information about what it's actually testing against, right? Yes. Because it has no concept of that. Right. So I'm going to specify that. It says, hey, successful. But ah, what's this? [[GASP]] It worked, and it worked really fast, because I tested it instead of waiting for the poller to do it. Yes, yes. So now, this is being executed from a script. Now the cool thing here is, I'm going to be able to substantially increase the capability of what I'm doing without making any more changes to the Orion alert. All I'm going to be doing from now on is just editing that script. I know the script works, I know the PowerShell call works, so now it's just about testing the details of what I have inside of my script. And that's important because if you do edit with an alert and it is actually being alerted upon, every time you edit it, save it, it will re-alert on those devices. It's going to re-trigger, right? So doing wrappers, especially for the scripted stuff, is extremely useful because then you're not going to risk those re-triggers and you can make as many edits in real-time as you need to. That's right. And also, I can do really cool things with things like conditionals. So here is my new script and I'm just going to go ahead and walk you through what this thing does. And so we got the same things, and I'm going to put these up again. And you'll notice that this one, I actually have commented, because I'm expecting that our audience is going to see this and I don't want it to look just too totally hacky. So I broke out what all these lines do, if you're not familiar with PowerShell. So, I've still got my definition and for all of my parameters. And now there's a lot less stuff in here. I've got my webhook and then I have this Git date. And I've broken it out into a formatted time, because I would like to have a timestamp. Especially because it will put multiple posts under one glyph, if they come in back to back, and it's kind of nice to have a timestamp. The other one is, I'd like to use emojis to give me up/down statuses. And the cool thing is, there's this great emoji cheat sheet that they'll link you out to. And you just use the regular colon escape for that, right? So, I've got a default empty emoji—if it doesn't know what it is, it's just not going to have anything. But if it knows, in this case, if it's unknown, I'm going to get a thought balloon. If it's up, I'll get a nice green heart, and if it's down, I'll let you figure out what that is. Okay, so then the next thing I'm going to do is, I'm going to dynamically build my JSON payload using arrays—native arrays—inside of PowerShell, right? So I create an empty array with my JSON and then I create .channel. Hey, lab bot .username, that's Orionbot, .icon.url. Much cleaner and easier to read. And then down here, I do my JSON.txt, so again, these are all those elements we saw all crammed together before, just broken out. And it lets me do things like, I've got an escape here, so I can do things like this double escape quote in. For new line, right, a little bit cleaner, and I've broken out all of my other elements. Much, much easier to read. So the next thing I am going to do is use my array again to build a web request, so I can do my URI, which goes back to my webhook, content type. It does like it if you tell it is expecting JSON, the message, and then the body. I am going to convert my previous structure directly to JSON and then I'm going to send it using the invoke web request right here, right? Mm-hm. So I'm just going to grab this, and again, I am not making any changes in the UI. I'm just going to come back over and drop it right in here, and save it. Because I'm not changing the file name, so I don't have to make any changes on the alert. And we're going to fire this thing. Alright, so I'm going to come back over here by manage nodes. We'll put this thing in unmanaged stat. And now we're going to put it back into remanage, so that we can fire the alert. And we'll come back over here and see what happens. Anticipation. [[LAUGHS]] There it is, right? It fired off the emoji. It said that I got a thought balloon, told me that that the state was unknown, and I'm using some formatting there, as well. Now check this out, because the poll is going to do its magic again. Okay? The next thing you're going to see is... that! Is that incredibly cool? It's right there and it's in my mobile app as well. So it's coming up on the dashboard. I can actually see it right here on my phone. And we can deal with it and that means that our lunch hour didn't get interrupted. Our lunch hour did not get interrupted. I like that. We got to stay at lunch a little bit longer. It was really, really handy. And, of course, you can see it right here on our main dashboard as well. I would have said, I think we have a problem here, or I have a bad feeling about this. [[LAUGH]] But instead, it's totally booted us back up. And again to your point before, I can organized this. I can sort this. I can redispatch this. And you know what? Let's take a look at the details on that. I'm going to click on the detail link. It's telling me that I'm in a warning state. That the alert's been triggered. And I can also see my alert history here as well. So make sure that all those elements are happening. I can add notes, if I want to, so that I can easily track that. If I had a question about the node it was connected to, I'm going to click node, and where's that going to take me? Exactly: the node details page. And I can see all of that. Especially if there's something else that's being caused by that. Or usually, you wouldn't be alerted on an interface. It might be a node, or an application, or something else, where you'd want to see all the other dependencies to try to figure out what the root cause was. That would be really handy. But in this case, I know it doesn't matter. It's just fine, so I'm just going to acknowledge that and I click there. Ah, there it is acknowledged by admin. Very nice. So handy. So, something I think is really neat about this is even if you do the basic scripting, the very first one that you did. The basic side of it. You already have valuable information, quick and easy set up. Yes, you can get engrained and keep adding and kind of become a master scientist at it. But from the get-go, from just what you're posting out there, that's valuable in its own self. Right. I can just feel the wheels turning in everyone's head. Right, well, the mad scientist stuff. Speaking about it, I realize I'm sort of stealing your line, but Swis and Swickle, and the Orion SDK. Which was our last episode. I can see that because Slack and any of these interactive DevOps type, work-sharing environments will respond to requests. You can actually say OrionBot, what's the CPU on server xyzabc123? And it will use the SDK to go back and to get that. Those are all things that can be written in as listeners. After we build that, we're going to do a show on that. We're going to have to. That would be really, really cool. Yeah. I'm going to be signed up for that one. And that brings us to Service Desk. I use Web Help Desk myself. I've been using it for a long time. I like it, and most people have. There's about ten different tools that you guys are all using, but the number one request that we get for documentation on integration is Service Desk. So, I'm going to walk you through how I think this works. I've tested it against the dummy endpoint, but I don't have access to Service Desk. So I'm going to attach my script that I'm going to show you. And please don't just hate me online if it's completely terrible. But let's go back and forth with it. It's a start. It's a conversation. Yeah, go ahead and download it if you have Service Desk. And let's experiment with it and get that thing straight. And whoever comes up with the best one, we'll throw out a Fun THWACK and give you all kinds of cool stuff and honors. Okay, so with Service Desk, there are a couple of things that it needs to do. It's got a lot of, it's got authentication that we didn't see before in the last webhook. The documentation for it is somewhat limited. It's available, it's out there kind of, but it's mostly in their community. And the things that you might want to do, like conditionally decide whether or not you're going to add a new instance directly in Service Desk as an object or instead just alert. Being able to do conditional processing, and then still do notification, you're definitely going to use a script for all of that. So, let's walk through how this script works and again, I want somebody to download this and test it for me. So, this one is my script for Service Desk. Here we go. This compiles and runs. I have a testing point and it's behaving itself. So, this one is a little bit different. I've added a couple of other things, like one of them is, I added the Orion alert ID object. I really need that because I'm going to use that and tie the Orion alert into my Service Desk ticket. Okay. Right. So again, the format for their URL. There's a couple of different versions of it, and obviously you would hope that it would be on HTTPS, but basically, it's a time-kept service. So, it's a time-kept server instance and then, they use their ARAPI. There's actually a special WAR file that you download and you can Google for it out there. It's the ARAPI WAR file. Do this on a test system first because you can quickly blow up your Service Desk and make people really unhappy. But once you restart time that gives you the other side of the API to be able to process this. So, once it knows the name of where it is. So, it's got the host, it's got the ARPI, the server instance, and then the schema, which is also going to be URL escape. So in this case it's HPD. I guess that's their help desk base. And then help space desk. Now, when you send information to it, you can do things like create new, add, update, delete. You can do basically anything in those defined values and again, everyone's service desk definition will be different. They'll have some custom fields and some base fields. Refer to your documentation for your instance, but the basic format is reference ID. And then a set of field IDs which is basically a bunch of two polls, names, and value pairs, and then you send it to that URL. So the way that I did that was I build, in this case, a set of objects. So this is a short description, details URL, no details, acknowledge URL, the same thing we saw before. I give it a submit date and time. And I'm parsing out my timestamp here instead of doing it with text. I give it a status. I'm going to give it the submitter, in this case. It tells me it's coming from the API. And then I'm going to roll all of those things up and then into a structure with myID_, and I'm going to use the Orion ID alert. And then pass in this bunch of hash prompts. So again, this is a substantially more complex script than what we were doing before. And it's a reason that you could never pass this in on that execute program command line. Right. So, now I need to do authentication. In most cases, it's using basic auth. So basically I'm doing a user and pass. And then, you guys are familiar with secure strings, right? Special type of memory objects. And then there are some things that require them. So, in this case, this PCS credential requires a secure string to be passed in for the password. So I give it a username and a password, and then I pass that into my PCS credential method, and that gives me a fully qualified credential. I'm going to pass that into the web request. So then this looks pretty familiar, right? I've got my webhook, which is that URL that we saw before that we're going to go to. Again, it's a JSON application. I'm going to do a post. I give it the credentials, so basic auth will be set up. And then I'm going to pass that hash that was created here. This is a little different. Last time I actually did it pretty much with just string concatenation. But here, because I am making so many changes, it was easier to actually build it using the prop add method. So we've got a little bit better control over it, and then I'm going to do the same thing I did before. I'm going to do an invoke web request and then pass that object in. Now, the thing that's different on this one is that it's going to respond with the reference number for the item that you just added and then an AR identity ID, which is basically the reference to identity. So if you wanted to take that and then use Swis to then go update the notes field, for example, so that you could find it again or stick a URL in that would be able to go open that in Service Desk. That would be really handy. And so, I added right here, a variable called R = so that the output from this web request will then be passed into this variable. So you would want to do something with that here and I don't have it in this. But I am hoping to see what some of you guys can do with it, so again. It's a little bit more complicated, and if I wanted to switch, I would just copy and paste this, and come over here, and put it inside my script file. And I would instantly go from taking that information and posting it up to Slack to putting in the service desk. So it gives me the ability to reroute where those integrations go without making changes again to my Orion alert. This is awesome. Right. It's something that people have been asking for a long time and repeatedly. "How do I integrate this with my ticket system?" And whether it is Service Desk or some of the other ones, the process remains pretty much the same. The details may change, but this is something that's really going to make a difference for a lot of people. It's not the simplest thing on earth, but it's really useful. And we totally went from beginner to advanced there. So, I mean, we're taking you on the journey and for once, we're actually diving in. [[LAUGH]] And going all the way through it, so that you can walk the steps with us. That's the best part. But I was not an expert in any of this, before I sat down and looked at the docs, right? Slack gave me all the information, I went out and looked at the Service Desk community for the details that I needed for Service Desk. So this is just based on my understanding of Orion, of NPM, of networking and alerts, and my own business's workflow, and my needs. The actual technology to make this happen? Pretty straightforward. Yeah, if you're not afraid to just try a little scripting, you can get there. And especially if you're not afraid to script, he provided it. Okay, I admit cURL is not the way to go long term, but it is a great way to get started. So, did you guys demo how to restart a vSphere VM using the management APIs? No, but that's another great example of doing something really useful with an alert. We have lots of users out on THWACK executing all kinds of actions and alerts. Automation is a good thing. Yeah, and we've seen several requests for things like that on chat as well. So if you're not with us live today, then be sure to visit the SolarWinds Lab homepage at lab.solarwinds.com, and sign up for a reminder. And check out dozens of other episodes. Yes, and there are dozens of episodes, because this is our third anniversary of SolarWinds Lab, so where's the cake? I thought the cake was a lie. [[NOISE]] Here's the cake. [[MUSIC]] It's been another great year for the show and we'd like to raise a toast. To the video crew, to the teams that help get the word out, and most of all to you, our viewers. Thanks so much for being such a great and sharp audience. Geeks rule! Geeks rule! [[MUSIC]] Alright, I'm Leon Adato. I'm Kong Yang. I'm Destiny Bertucci. And I'm Patrick Hubbard, and thanks again for watching SolarWinds Lab.
