SolarWinds Lab Episode 76: Joe vs. The NetFlow Volcano

Whether you’re curious if flow monitoring might be helpful, or you’ve been coalescing IPFIX streams of network-activity goodness for years, IT’s reliable, old hand has learned a few new tricks. In this episode, Head Geeks^™ Patrick Hubbard and Leon Adato are joined by Principal Product Manager for NetFlow Traffic Analyzer, Joe Reves, to revisit the basics and rediscover why network engineers use flow monitoring in the first place. They'll discuss how it’s configured, and examine the differences between NetFlow, J-Flow, sFlow, and other protocols. Find out when to use basic flow monitoring, traffic sampling, 95th percentiles, and traffic-based rules to see what’s really happening on your network. Times have changed and, believe it or not, flow monitoring has, too.

Back to Video Archive

Episode Transcript

So Tom really does take a nap back there. Yeah, I mean he's got the teddy bear. And that's also why it smells like bacon. Oh. Um. So, Joe, what are ya, what are ya doin' here. Somebody probably burned the fish casserole up on the PM pod again. Nah, I'm just looking for a quiet corner to update the NTA roadmap on CLACK. Oh! That's right! It's an RC now, right? Yeah, I've been runnin' it for a while. It's pretty amazing. Okay, so my dev environments stuck in approval hell to get the upgrade to SQL 2016. So, I’ve got to ask, is it worth this much hassle? Totally worth it. Oh, it's absolutely worth it. I mean, think about all the new technology support, right? Azure, IPv6, Palo Alto Firewalls. You can also show NetFlow from a local source, right? So, if you want to get people up and running on the team, maybe they're new to NetFlow, you can do that just right out of the box. We released a bunch of NetFlow Free Tools. They're absolutely going to love that. So, why don't we do this, let's walk through, I mean, we've been talking about some of the classic NetFlow use cases for years on Lab at this point, why don't we walk through some of those and show how they're easier to do, they're sort of quicker to get started with using those new features, sort of remap those new features into those classic use cases. Perfect. And meanwhile, I'm going to have to get on IT's case to get the sequel upgrade going. I'll get it done. 2016 is worth it. 2016. Okay. [electronic whirring] So, I think the best way to roll through this is to first talk about those big technologies that we can just talk over, so let’s start off with Azure. What, why did we do that, what's that all about. So, the Azure support is designed to give people another option to deploy into the Azure environment and also Okay. to be able to consume and support the native Azure SQLdatabase service. Okay. Right, and it also means that you don't have to install that separate flow storage engine in another VM. So, it eliminates the need to set up a separate container for that. Okay, great. And it continues with the idea of supporting RDS and, you know, all of that. Exactly. Choices Fantastic. Now, I have had an unrequited love for IPv6 for a long time, something like two decades. So, why now? What's that? So, IPv6 is starting to become a reality for a number of our customers. So, education customers, Yay! large distribution customers, some of our federal government customers, they're all actually rolling out and deploying now. And they need visibility to the other half of the traffic in their network. And so, is that IPv6 for the sender and the collector as well? That's right. So, we're able to consume IPv6 flow records, also to consume IPv6 over an IPv6 endpoint. Pretty handy if you're dual STACK by default and your using NetFlow as a security tool to actually look at traffic, and oops, we are emitting IPv6 on that interface. Absolutely. Now, I know that we're going to see a lot of our UX improvements as we go through the rest of these demos, we're going to see where things have been tweaked and stuff like that Mm-hmm but it sounds like you've been working really hard with our UX team to make that happen. We have. We've built an epic for UX improvements. The UX team has actually been interviewing customers about things that annoy them about NTA, we've been adding those stories into that epic, and we've been working through those, kind of, one by one. And so, we've delivered about 10 of those, kind of, minor improvements over the last two releases. The Use-A-Build-A-Buddies have been busy. I'm glad you said that Absolutely. I don't think I could've. [Joe laughing] I can't always get it right. All right, so we were joking about the SQL 2016, but I think it's worth mentioning again that we have deprecated the 2012, that you really need to be moving to 2016 as fast as you can. Right, and, I mean, you've had the option for a while to maybe run an earlier version, but you didn't get all the features. This is the first time that we said, look, you really do need to be on 2016. And it's simply because column store database is allowing us to check off so many of those top requested feature requests. So, when you go to the, for example, the “What Are We Working On Now” page in THWACK, since this one is in beta currently. It should be in RSE shortly, but when you look at what we're releasing, we're able to now start accelerating the releases for features for NTA, due in large part to that move to 2016. Right, okay. So, there's one really amazing and sort of work changing feature, which is in support of Palo Alto, now you can see how a configuration change affects a flow. Can we take a look at that? Absolutely. So, let’s take a look at the Node Details page for one of the Palo Alto firewalls here. And you'll notice on the slideout, there's a number of new options here including an option to take a look at the policies that are configured on this particular box. So, let's take a look at the list of the policies that we have. If we look at the policies that we've collected from the configuration, of the device. Mm-hmm. If we zoom in on those, there's a couple different ways to filter those and to sort those. Let's zoom in on one of those and take a look at the Policy Details page. On the Policy Details page, one of the things that we'll see, which is contributed by NTA, is a top conversation on policy. And this particular widget shows us the conversations that are directly related to a particular policy. So, the impact of this is that if I want to make changes on that policy, I can see which conversations are affected immediately. I love the way this extends. I mean, policies are never, in production, end up being what you thought they were when you were imagining an ideal [Joe laughing] policy, right? And so, like with CBQoS, right? You want, you have to be able to watch the traffic, the sort of cause and effect, as you tune it over time. And the idea of using NetFlow as a way of, not necessarily perfecting policies, but actually optimizing them in a real-world environment makes that process a lot faster. It's going to make a huge difference for a lot of the folks who are using this right now, is it's that safety check when you're planning out your change controls. Absolutely. So, I think I really want to hear about this local source. This is something pretty revolutionary. How did we come up with it? What is it? Let's, I want to take a look at it. Okay. So, the local NetFlow source came about as a result of the experience that most of our customers have when they trial this software. Usually they'll install it, they'll navigate to the NTA homepage, No data available. No data available. Miserable experience. Right Wah-wah! So, we want to be able to put some data in there. Well, so this now, this installs sort of by magic on the collector. So, wherever there's a collector, it's set up. Right. And is it enabled by default, or turned off? So, for new installations, it's going to be installed and enabled by default, for upgrades, it's going to be installed but disabled by default. You can go in and turn it on if you want to use it. So, there's a switch, you can turn it on and off. And so, when you're testing you turn it on and then when you start driving a lot of traffic, you want to make sure you, maybe, decrease one more, cut off one more sender, you just turn it off and you're good to go. Gives you some choices about where your sources come from. So does it just, and it just looks like anything else in the interface? Exactly. Exactly. Okay, well, I want to see it. All right. What does it look like? Well, let's take a look at it. So, let's take a look at NetFlow Sources, over here. And of the things that you'll see that you haven't seen before is the NetFlow source with a little Windows logo next to it. I've never been so happy to see that logo. I have to tell you, it's pretty exciting. So, local NetFlow source here, not much traffic on this right now, this is a Lab machine, but you can click into that source and you can see traffic that is unique to your particular machine, which is kind of cool. So, when we say this is unique to the machine, this is actual NetFlow coming off of the interface on the poling engine itself? That's right. So, what we've done is we've taken all of the interfaces on that machine, we have combined them into a virtual interface, so what you're going to see is traffic that is sources from, or destined to, this particular machine. So, just those like ingress/egress on any other interface. Exactly. So one of the questions that this answers, that's really interesting, is tell me about my network management traffic. A lot of customers ask the question, what kind of network management traffic am I driving? SNMP, Flow, that sort of thing. This gives you the ability to actually see what that looks like. Especially for a distributed collector. Right. So, to be able to enable or disable this particular source, you'll go into Manage Sources here and you've got a simple checkbox here. For your Windows interface, you'll be enabling that or disabling that with a checkbox. And if you have sFlow or jFlow sources, they'd show up here too. Absolutely. So, some of the UX improvements include easier navigation options, right up here. So, you'll be able to select the time period. At this point you can select ingress or egress and if you want a filter based on IPv4 or v6 traffic, you can make that selection here. One of the things you'll see is some IPv6 traffic here. And our conversations, you'd be able to see the addresses that are involved in those conversations. Nice. Pretty cool. Great, and so, basically while you were adding the selector for v4 or v6, you went ahead and redesigned that selector in general. Exactly. And you'll also see that in the flow navigator here, so, and IP version selectors here and the flow navigator as well. So, the thing that's interesting about local source, is that this data is almost personal. I mean, I know what my polling engine is doing, so, when I look at this NetFlow data, it's familiar, right? Yeah, it's interesting that you should say that. When we had the conversation with the development team about first developing this feature, Mm-hmm we were trying to solve the problem that new customers have when they try to evaluate the NTA software. That is, they install it, they navigate to the NTA homepage, and they see no data. And so, over the evaluation period, they were required to go and try to enable flow data from an external source that they may not have control over. So, what we wanted to be able to do was to give them a sense of how the product worked. And I asked the development team if they could pick up flow data from the local interface of the server. So, we took a weekend trip, we went out, did some team building, we came back, and some of the senior developers dropped me off and said, hey, we need to go into the office. They went in on a Saturday afternoon, they were there again on Sunday, on Monday they came, and they sat me down and they showed me a machine that they had installed with a local NetFlow interface enabled. And they said three things to me. The first things they said to me was I've worked on this product for a long time, but when I first turned this on, it changed the way that I thought about the product. And I thought, that's pretty cool. The second thing they said to me was I know what's running on my server. The observation that you made. I know exactly what's generating this traffic, this data that I'm looking at is personal to me. Hmm I know where it comes from. And the third thing they said to me was I can't believe we haven't done this before, which I had to agree with, right? One of the things I love about that story too is that you, about it being personal, is that there's a parallel, I think also with IPAM in a way. Like, I sort of Mm-hmm learned IPv6 by using the interface to define IPv6 subnets in IPAM. So, I kind of, the tool helped me understand better how IPv6 actually worked. And so, in this case, we all know, or I think if you're watching this episode you know, NetFlow is extremely valuable, but you're not sure exactly how. Like, you're going to have to Right. learn how to interpret it, how to read the NetFlow tea leaves. And so, looking at data that you know by heart, that is personal, you can map your understanding of that traffic to the charts, to the breakouts, to the reports, and then you can take that and extrapolate it to all of the other nodes on your network. Exactly right. So, that kind of emotion is, I think what was the impetus behind the Free Tools also. We just want to make IT folks lives better, even when we release a tool that is free like puppies and beer. Yeah, sort of NetFlow Lager. NetFlow Lager. That was awful, I s- That's terrible. It was good. I want to drink it now though. All right, so let's talk about NetFlow, the Free Tools. What are they? What do they do? Like, what's that all about? Okay, so the Free Tool bundle includes a NetFlow replicator, a NetFlow generator, and a NetFlow configurator. And, for those of you who might've worked with, maybe, support and some other areas and been part of betas, you've might've heard us talking about some of the internal test tools that we use. And this actually came from the code that we developed internally to do high-rate flow replications and some other things where we needed to actually send net flow to 10 different destinations, not just two. That's right. Like a lot of our great ideas, we were solving a problem for ourselves and then we just kept on helping other people solve the same problem. So, there's three of them, right? There's three altogether. So, the replicator, we added an interesting feature to that that allows you to sample the traffic that you forward. So, some folks are looking at that as something they can deploy remotely and then they can reduce the traffic because of our WAN link or maybe comes out of a public cloud environment. Or you can't enable sampling from the sender, so you can sort of That's right. convert it. That's right. So, the replicator allows you to send to multiple destinations. Some folks are using that to send to their production NetFlow platform plus a security tool, Mm-hmm to be able to do security investigations kind of on the fly. Nice. Okay. And what's the next one? So, the next one is the generator. And the generator basically simulates sources of NetFlow traffic. So, that allows you to describe where the traffic is coming from, and it'll simulate that source of NetFlow traffic. It allows you to describe the endpoints that are involved in the conversation and the type of flow that you're getting. And folks are using that to verify things like their firewall rules and make sure their firewall rules are configured correctly end-to-end. They're also using that just to verify that they're getting flow at their collector, and they can see data come off and run. And that'll spoof the IP of the transmitter, so it looks like a regular transmitting node and then your endpoint data lines up with interface data. Exactly. Nice. And the last one. Yep. The last one is a configurator. It's a very simple tool that we have re-wrapped. It's one that we have released a number of years ago. It's a very simple configuration tool for NetFlow version 5. So, for IOS based devices, Cisco IOS based devices, we'll go in, we'll configure NetFlow version 5, get that up and running immediately. If you want a more sophisticated configuration management capability, to be able to set up NetFlow across all of your devices, then NCM is your ticket. I have a feeling that you've talked about this before because looking at the screenshots here Right, right. This slide, what's this logo in the upper right-hand corner? Hmm? And SWUG down here? So, you've been talking about this at SolarWinds user groups. We have, so, and we'll be talking about it at the user groups through the remaining part of the year as well. So, Yep. customers are reacting very positively to having a set of test tools in their environment. They really like that. And, you know, we're enthusiastic about getting those out to as many customers as possible. Great, and- And you're, well I was going to say, and you're also working on the documentation now for the support center as well. That's right. So that if you go out to support.solarwinds.com, you'll be able to get lessons, how-tos, and everything else for these new features soon-ish. Right. Right. Right. In the link to pull down these Free Tools is in the show notes, and you'll be able to pick those up everywhere, just look us SolarWinds Free Flow Tools Bundle and you'll be good to go. The Flow Tools Bundle is one of my favorites. Flow Tools. All right, yes, yes. [all laughing] All right, moving on now. What's the next thing that we have to talk about? Okay, so, we've covered new, we've talked about UX, we've talked about some of the foundation component changes. Let's go back over some of those classic use cases that all of you deal with on a regular basis, the whole point of why you're using NetFlow to analyze traffic and how they're different using some of the enhancements in this particular release. Fantastic. All right, let's start with the absolute classic, bandwidth hogs. I think I got an email on that. [sighing and laughing] Okay, but to be fair, it is the canonical example of what NetFlow, like, it is the 80/20 rule of what you will do with NetFlow. My interface is pegged, what is pegging that interface? Fair enough. So, the usual way that the usual word flow that we use to investigate bandwidth hogs is a very reactive flow. We get a helped us ticket, we see high volumes on a particular interface and we go, and we investigate what's going on in that interface. Now, after we've done that a couple of times, it would be really cool to be able to set an alert and to get on a Yes. more proactive basis for some of these bandwidth hogs that we're seeing in the environment. So, let’s take a look at how we would normally walk through a bandwidth hog investigation. So, our application data, down here, may be in our top 10 applications, we may be able to see volumes in our Top 10 Applications, here. Let's take TLSSSL port 443 web traffic as an example. Okay. Let's see where this particular volume in traffic is visible. It's visible on our own local NetFlow interface. It's also visible on our Palo Alto box on this interface. So, let's drill into that and take a look at that. So, in the Details page here for this particular application, one of the things we're seeing is HTTP protocol. We're seeing it as it traverses through this particular node on this particular interface. And then we've got our application view down here. That's enough context for us, then, to be able to create a flow alert. A flow alert. So, that's completely new in this version? That is completely new in our last version. Right, we're starting Right. to see customers actually deploy that into production now. And that's the primary reason for going to SQL 2016 because that Yes. gives the performance for the back-end to be able to do these queries on traffic banks. Right. So, we've had the capability for some time to be able to alert on the total aggregate volume of traffic on a particular interface. What's new is our ability to pick out an individual application Mm-hmm and describe a threshold for that application's traffic and alert on that. Okay, so how do we set that up? So, the new box here, the Create a Flow Alert box, summarizes for us that the alert's going to trigger on flow data that we're getting from this particular Palo Alto device on this interface. We've got the ability to customize the alert name a little bit. So, we could call that Leon's Favorite Alert. Uh-huh. But you’ve got to put an exclamation point first. Absolutely. [laughing] Absolutely. Sort it right to the top, yeah. We can select the severity for that particular alert. So, we can look and set the severity at serious. And then we've got a couple of different trigger conditions here. So, the classic bandwidth hog is application traffic is going to exceed a particular threshold, we're looking at ingress or egress traffic here. So, we can pick ingress or egress. Traffic that is greater than a particular volume here. This is a lab instance, it's very low traffic, but we've got the ability here to go ahead and type in a static value here and then select the unit. The alert's going to look over the last five minutes to make sure that we're looking at an average of traffic over the last five minutes. And then we can either create the alert immediately, in which case it's just going to log, we can go back look at it in logs or look at it in Log Analyzer if we're sending alerts to Log Analyzer, or we can open it in the Orion Alert Wizard. Now, when we do that, what we're going to see in the Alert Wizard is that we've already jumped to step seven of the Alert Wizard. All of this stuff is pre-populated already, but if I wanted to go back, for example, and attach a trigger action, Right. if I wanted to add an email alert or something like that, I could do that here. Mm-hmm, or an actual configuration change, or, I mean, you can be very, very, not reactive, but programmatic about how you handle some of these alerts. One of the bandwidth hogs that people diagnose on a repeated basis is running backup traffic in the middle of the day. So, if I wanted to add a time of day filter here, this would be the place to do it. And then I'd be able to specify a schedule here. So, when I jump to Summary for this particular alert, I'm ready to go. There's the SWQL. There's the SWQL that actually generates the alert. This alert would be immediately triggered on zero objects at this point, right. Mm-hmm. And then I can go ahead and submit that, and I've created my new alert. Perfect. Well, what I really like about this though is that you can start to use social engineering for remediation with this, right? Uh-huh Because, you know the old joke we always made about if you ACL out YouTube, no one will sit with you at the cafeteria? [laughing] At lunch, right. So, but being able to do say, for example, if, like the, the top remote offices buy YouTube traffic, being able to send a targeted email to that office that says congratulations, your branch is number one in YouTube viewership for today Ooh and then you watch it fall off again, right? So then you can actually use that traffic mix data to do kinder remediation using social engineering as well. And so often, you want to be able to use that kind of burst bandwidth or unused bandwidth on an interface Right. when it's available, but otherwise, make sure that you can adjust it back down. And so, that's something that's another great reason to do the 2016 upgrades to be able to get this kind of behavioral-based alerting. I love that. So, this leads to something that was really hard to do before. Can we go back to the NetFlow application details page? Right. So, over here, where you would've set the, when the flow is greater than a certain amount, you can set it for when flow is less than a certain amount. And what that means is that you can monitor for the absence of something. So, when I have an application where I'm expecting a certain volume of traffic, I'm expecting active directory traffic, I'm expecting application flow traffic and all of a sudden it disappears, I can catch that here, right? Exactly. By direct observation. So, what we'll do is we'll go in and set this to less than or equal to and then specify a volume here and, and so that could be a very low volume, right? It just needs to be a trickle. Yeah, so, that could be less than or equal to 1K, it's going to check it over five minutes and then you can create an alert and generate an alarm based on that. I think that's something that's going to be very useful for a lot of folks as well. This was actually one of the primary use cases for being able to do per application alerting. This was the thing that people were most interested in because they weren't aware when their application went off the air. So, you could also use it because you can generate reports off of this data. That's right. You could also see am I getting, are my backups running. Because if you have a day where that expected backup traffic didn't happen, then you'd be able to have a page that actually showed days completed and where that traffic was seen. And you'd be able to notify someone to remediate that immediately. So, in between knowing when something has gone too high and knowing when something has gone too low, there's this other weird class of traffic that might be present but only in small amounts because it actually wants to get lost in the noise. Forbidden needle in the haystack. Yeah, it shouldn't be there, but it might be, right? So, port 0 traffic is a good example of that. So port 0, that's completely impossible, nothing could ever po—awww! Nobody's there. Yeah, exactly, right. So, port 0 traffic, the ability to monitor port 0 traffic was an ask from one of our ISP customers. And it's either horribly misconfigured communication STACK or it, more often, it's malicious traffic. Okay, and so you can monitor that and know when it exists at all because that's a red flag. Exactly. Okay, [speaking all at once] So, if you ever see it in the traffic flow, I want to know immediately. Bad things are happening. So, let's take a look at how to find that traffic. So, the way that we find this traffic is we'll go ahead and use the Flow Navigator to set the context for where we want to see this particular traffic. So, we'll look at a detailed view here for a particular interface. This is the interface for a device that faces the public internet. Mm-hmm. And then we'll take a look at the application that we're interested in searching for. And in this case, that's our port 0 application. One of our UX improvements was to go ahead and alphabetize this very long list of applications Can you... you can actually start typing and get to the application that you're interested in. But, in this case, it's a lot simpler. Port 0 is port 0. We'll type it in zero here, we'll add that filter, and then we'll go ahead and submit. Tell me that's generated data. This is generated data. Okay, good. Fortunately, right. [laughing] But, there's a lot of interesting information on this particular screen, you can see that we're looking at port 0 traffic on this particular interface and this device, application port 0 only, right, but we see a number of different endpoints here, a number of different conversations. Which you would expect if there was scanning activity. Exactly. This is, this is what evil on your network looks like. Yeah, so once we have this context here, then we can go ahead and create a flow alert. And, again, we have the summary up here, the data that we're seeing on this particular interface, the alert name, so we could call that Malicious Port 0 Alert. It's a pretty serious thing, if we see this at all. Maybe critical. And then we can set a relatively low threshold here for ingress traffic. So, let's say something like 5kbps. Presence of traffic at all. Could you might expect every IP address, especially in this case with a firewall example, is getting scanned all the time, you're looking for a targeted attack, for example. That's right. That's right. Yep. And one of the things I like here too, I mean, we use, for NPM we use PerfStack a lot. So, sort of visually put together the data that represents a real issue, right? So, you have sort of the customer issue across the top or the affecting issue across the top and then root cause eventually somewhere in that STACK. So, it's the same thing here with this, with the flow wizard, right? Because you were actually building out a visual representation going back to this is personal data that I understand and saying yes, I want an alert on this. So, it allows you to visually confirm the source of the data that's tied to the alert. So, when I create this particular alert, maybe I want to do a notification, for example, to my security folks. Yep, definitely. So, for the last trick, the last demo. How 'bout traffic that we don't have right now, so we don't have any examples or context, but we know it probably could pop up later and we really just want to know as soon as, so we can shut it down? So, how 'bout Dropbox traffic? There we go. But that's a great example of when I really think we ought to talk about NetFlow as a security and governance feature because if you want to say we have a policy though shalt not run Dropbox clients on your machines, watching that at the firewall is a really handy way to figure out where they are and then immediately identify the in-point and go have a conversation about it. And saves the work from trying to monitor the entire surface of the bubble, every device. So, it's a much more efficient way. All right, so how do we go about monitoring a thing that doesn't exist right now? So, let's start by creating a custom application. Okay. So, for that, we'll go to the Settings page here, All Settings. And we'll take a look at Product Specific Settings for NTA. What we're going to do is to create a group of IPs that this traffic is sourced from or destined to. Okay. And then we'll use that in our definition of a custom application. Okay, so there's a little bit of work here because we're going to have to find every IP address for Dropbox That's right. Or the range. Or the range, right. But you know, you only have to do that once. So, we have to have looked this up. So, the IP address that we got was 162.125.248.1. So, let's add a new group. And we'll call this Dropbox. [computer keys clicking] And our address range here, we'll go ahead and type that in. [computer keys clicking] And we'll just enter a single address here. So, that is also 162.125.248.1. Okay? And we can enable this to be present in the Top 10 IP Groups resource if we want to. We'll go ahead and add that. Actually, we don't need that one. So, we've got this one in here. We'll hit okay. And then we'll enable that. We'll go ahead and hit submit. And in this case, you're actually creating an IP group for a remote address, but you could do that for internal addresses as well. Absolutely. Right. So, if you don't want traffic from one section of your environment to another section of environment, this is how you could go about setting it up. Or you just want to categorize it. Effectively get a tag on it. Right. Right. Exactly. So, now we have the group, now we'll use the group in the creation of a custom application. So, let's take a look at Application and Service Ports here. And what we'll do is we'll go ahead and add a new application that doesn't exist now. We'll call that, also, Dropbox. [computer keys clicking] Dropbox runs over our web protocols 80 or 443 and so, in this case, we're interested in sources within our network destined to Dropbox. Mm-hmm Our group, this going to run over TCP. And we'll go ahead and add that application. And then, very important, submit the changes. Very good. Verify the changes first, then spread it out. It's a step that a lot of people forget and then they're frustrated that they can't find the application. All right, so now let's go back over to our NTA Summary. And pop out our Flow Navigator and let's take a look, again, at our interface that faces the public internet. So, we'll look at a detail here for an interface that faces our public internet. And then let's look at our application that we've just created. So, we'll use that type ahead here and find Dropbox. We'll add that filter and apply that filter. Ooh, aah. Alphabetical type in. Now, what would we expect to see here? No data. And that's a No data right now. and that's a good thing in this case. Very good thing. Yeah. No, if we want to, we can go ahead and create an alert based on that and we can say Right. if I do see any of this type of traffic, then, I'm going to go ahead and add a threshold here over the last five minutes. I can go ahead and open that in the Alert Wizard and I can go ahead and open add an action to that just as we've done before. That's fantastic. And this also gets you out of the game of network protocol whack-a-mole. Exactly. Where you have to wait for the problem to occur and then set up your alert for it, you can proactively, predicatively go and say and I shouldn't see this and this and this and this and get it, all your ducks in a row when you start out. So, this could be a whole set of protocols that we're concerned about from a security perspective in terms of data leakage out of the network, or it could be other times of non-business traffic that we expect not to see. Fantastic. So, I just want to be clear that this is only the stuff that's in NTA. So, when you look at the entire suite of network monitoring solutions, we There'd be enough for a whole episode, which is exactly what the next episode is about next month. Right, we know that once we open the topic to NPM and NCM and all the rest of the gang, it's going to suck all the oxygen out of the room, so we wanted to make sure that NTA had its own time to shine. That's right, and that's based on your feedback and, of course, one of the great ways to give us feedback is to interact with us live. And hopefully, over here to the right, you see the chat window so that we can take your questions live on air, but it you don't, well that means you're not live with us. So, swing by our home page, which is lab.solarwinds.com, take a look at upcoming episodes and makes sure you're with us live so that we can talk to you in real time. Right. That about it? That's it. Awesome. So for SolarWinds Lab, I'm Patrick Hubbard. I'm Joe Reves. And I'm Leon Adato. Thanks for watching. [electronic music]