In this episode, the Head Geeks showcase SolarWinds Security Event Manager (SEM, formerly Log & Event Manager) to address a critical aspect of IT security and compliance: Active Directory security. AD changes, such as adding users to privileged groups, escalating privileges, and changing user accounts, can create security holes.
The Head Geeks show how to install a SEM agent on a domain controller, edit the domain controller audit policy to log key events, and show how SEM can then monitor all AD events in real-time. Finally, they discuss why you need to monitor file system changes on the Domain Controller itself.
Episode Transcript
Hi, I'm Leon Adato.
And I'm Lawrence Garvin.
And I'm Thomas LaRock.
And I'm Patrick Hubbard and thanks for joining us on another episode of SolarWinds Lab. We have a huge show for you this time. And as much as we try to keep these things under 25 minutes—and we really, really do— this one may go just a little bit long.
That's because we have a lot going on. First, we have all of the Geeks here today.
Hi.
Thank you, Thomas and Leon, for flying and joining us.
That's right.
Thank you.
Thanks for having us.
And second, we are finally, at long last, doing a show on—
Wait for it.
Is it Log & Event Manager?
Yes, exactly.
Ah, awesome.
We're going to do a show on the LEM. You guys have been asking for it on chat for a really long time. And we thought we were going to do LEM and EOC together, but we started doing prep to actually put both of them on the same show and realized that there was so much to talk about with LEM that we're going to save EOC for another day.
In fact, we also found out that doing the EOC prep that that was potentially another 20 minutes all unto itself, as well, so.
That's right.
Right. So here's what we're going to do today. On today's show, we're going to roll up all the questions that we've seen on chat and everything about LEM. We're going to do some how-tos and we're actually going to solve a couple of real issues that customers have when learning SIEM platform.
Yeah, and why are we even doing this show again? Because you guys have been asking questions about this for a really long time at our homepage, which, of course, is lab.solarwinds.com.
So make sure you come by. If over here on the side, you can see a big chat window and you happen to be chatting live with all of us, well then, you probably are at our home page and you even signed up for notifications, so you knew this show was coming. If you did not know that, well then, come by lab.solarwinds.com, sign up, and you'll know about upcoming episodes before they happen. Some of the special things that we do, an occasional— I don't know, we give things away sometimes, all kinds of cool stuff. Yeah, so first, let's actually start this off by de-jargonifying SIEM.
Oh, I know this one, it's security information and event management.
That's right, and if it sounds like two things put together—information and events—kind of jammed together, it sort of is.
Yeah. Because there's this thing called SEM, S-E-M, Security Event Management. That's real-time event detection, analysis, notification—well, not really analysis, notification, detection, those sorts of things, working on things that are known and responding to those things as they occur.
Right, and then you have SIM, which is Security Information Management, which you can think of as the storage and analytics over the long term, with the log data that you've been receiving.
That's right. So, SIEM then is a tool that actually does both. It handles events in real-time and it can raise alerts and take actions, but it also has access to the historic data, so that it can actually spot issues that can only be detected over time.
Right, things like say, Active Directory shenanigans.
Yeah, yeah.
Right.
So then really, the SolarWinds acronym LEM is actually a more direct way to explain it. But why do we take the word "security" out if this is really a full-on security product?
Yeah.
Because back in the day, we had this thing about three-word product names. [Techno music] Okay, so how many of us know how to use LEM? How many of us are familiar in normalizing events? How many of us have 10 years’ experience literally doing everything that you can possibly do with the product?
No.
Nah.
Right, and me either. And since you guys have waited so patiently and added so many comments, please do LEM, please do LEM, over here in the chat, which you should definitely be looking at if you're watching us live. We've actually brought in an expert who, in fact, has more than 10 years, not just with SIEM products in general, but this product specifically. And, in fact, he has more than six years’ experience with this product before SolarWinds even bought Tri-GO, which actually was the originator of this LEM technology.
You know, I was thinking about, just the other day, I mean, LEM is completely focused on security event monitoring for longer than some other generic event collection tools have even been around.
That's true, and so with that, please welcome a true LEM guru, Rob Johnson. [Cheering] Rob, thanks for joining us.
You bet, thank you for having me.
Yeah, and we're not trying to bust your chops and make you sound old and crusty or anything. But you really do have a lot of experience with this product.
True, and really, it's the years of development in the Log & Event Manager technology. And for many of you, this might be a new addition to the product line, but really the platform is about, is as mature as the Orion platform as well.
Just not as sexy as applications.
You know, MPM predates SAM.
Yeah, well what about Patch Manager?
Do I need to bring Cirrus into this?
[Everyone but Will] Network Configuration Manager.
Okay, so Rob, lots of folks know you actually from THWACK. Probably a lot of them are actually over here in the live chat, as a matter of fact, because it's a chance to chat with you live, but you've sort of been the unofficial Head Geek of LEM since you've joined the company. And I figured that you'd be the master to walk us through some of the real deep dive how-tos in terms of being up and running quickly and effectively with LEM.
You bet, and really, if you're coming to this from the Orion platform, this might be very new. Orion, or Virtualization Manager, you know at that point. So it might be new and it's just a different way that LEM processes information, right?
Like, how do you say?
So Orion and Virtualization Manager and some of the other Orion products, they typically rely on a polling process, so they go out, they grab information. They pull it back and then you set up your alerting according. Well, LEM focuses on that information at network speeds, or in true real-time, meaning we don't have to wait for it populate a database. We don't have to wait for it to populate tables, for that matter. We basically process it in line and then alert from there.
Right, and so it's really that whole concept of getting used to events and so, an event flow as opposed to polling and aggregation, so the only thing that's sort of, a little bit the same in the other platform products might be Syslog and NPM. Or SAM—or to some extent, NetFlow.
Exactly, and really, you just have to change your thought process a little bit here when it comes to developing the filters, the rules, and the actions. A lot of the logic is actually very similar in all the products, but it's just a different way of processing the data.
Okay.
So I made a comment the other day about Active Directory. But it's actually a great place to start, because a decent number of viewer questions about LEM were from users having to deal with detecting Active Directory security issues.
Correct, and you know, really, Active Directory's probably one of the most popular tools in your IT environment that you need to monitor, whether that be for security, compliance, or just having a general idea of what's happening on your demand controllers.
All right, so let's do this. Tom and I are working on a segment for the THWACKcamp Holiday THWACK-tacular, and we could stand here while you guys do all this demo, or we could go out and do some on THWACKcamp.
Okay, well, I think I got this under control. So we're going to just take care of you guys right now.
He's got the phone.
Oh.
Just—no, I finally got the app. [whirring]
Whoa.
Yes.
Now I, too, have the Digital Conveyor app, ho, ho, ho.
Yeah, nice Die Hard reference, very good.
Yeah, all right, so yeah, I think it was one of the best ones.
The first one was really the best. [Techno music]
One of the most common security issues that you guys have talked about, especially with Active Directory, is adding users to privileged groups where they effectively get escalated because of special privileges on those groups they really never knew about.
Ah, right, it's the old out-of-sight privilege escalation trick.
Exactly. Right, so they get to take advantage of all the escalated privileges, and could potentially do damage within your network. But I think it's more important to take a look at those groups that are not quite so visible. Administrative groups are very obvious. Every server has them. Everybody uses them. But you also have to create groups that have maybe very specific privileges and to specific things within the environment. And those are even more important to monitor, because they're not so obvious and can even create more damage on some very critical systems in your network.
Right, because you have something where it's maybe designed to use around an edge case, or maybe only for administrators to use, and then it gets applied and nobody really has any real awareness of that. Or maybe you're just in a hurry trying to close the ticket. It's Friday at 4:30 in the afternoon. You've got somebody pounding on you, saying, "Hey, I've got to have access to the server." And they've in the past, never been a problem. You have no reason to think they're asking for anything they're not supposed to and so you just add them to that one special group, but not really fully understand the consequences of doing that.
Right, or maybe they're not taking a shortcut. Maybe a group like BSGF with a description like "Bob's Special Group Fix 2004."
Oh, right, okay.
You know, it doesn't make it easy to understand the permissions it's going to escalate. And it's an innocent mistake.
Right.
Right. Okay, so here, let's do this. I know that there's an out-of-the-box configuration in LEM to monitor and manage exactly this kind of example. But you still have to install the AD agent itself, and then you have to edit the policies and everything to use it. So, how about we demo how to configure this example? But also, let's pretend like the filters and the rules, and the searches—they're not there. And then we'll see end to end how to formulate a security event that we want to catch, and then how to hook up the plumbing in LEM, and how to set up the routing, if you will, inside the whole console. And then we'll just see how it'll make it all work.
Sure thing.
Okay, let's take a look at it. [Techno music] All right, so the first thing we want to do is install the agent on the system, because we have to get logs over to the Log and Event Manager virtual appliance.
On the Active Directory server.
On the Active Directory server. So that's what we're going to go through right now. I'll double click on the Setup utility and we'll just walk through the steps. So one of the things it's going to ask you is for the Manager IP address, and I always like to point this out, because it can be kind of confusing sometimes. The manager IP address, or any time in the context of LEM, that you hear the word "manager," it means the Virtual IP Address or host name.
Got it—the destination for all the information.
Exactly, exactly. So if you choose to accept the agreement, click the accept option right there. And then click next, and now it's going to ask you to install a utility called USB Defender. Now this is—quick little digression, but if this is a physical box, this can be really handy, because it can also let you know if unauthorized USB devices are attached to that system.
Ah, that's great.
So we're going to uncheck this. This is actually a virtual machine M on here. So we'll uncheck and we'll hit next, okay. Now next it's giving us a summary telling us, is it going to install this system? So we're going to go ahead and hit the install button right here. And now goes through the install process.
So this is actually Plinko. [Laughing]
You know, it's like one of those tests that you, or those puzzles that you see at the Cracker Barrel, which the little—that's exactly what I think of it when I see that.
But it's actually a propagation metaphor, right?
Sure.
Okay.
Okay, and then now what it'll do is, it'll start the agent for me. So if there are events being generated in the OS, you're immediately going to see those in the Log and Event Manager web console when it connects.
When you get all the way through here and you're green, you know you're good. You're not in some indeterminate state where maybe it's forwarding or it's not. You either succeed through the installer or you don't.
Correct. So, we'll hit next here and then done. And now we have an agent set up. The next step is to set your auditing policy up so that you're actually generating the events that you want to monitor, right? In a lot of cases, this auditing is not default in an operating system. There'll be some basic things that'll be enabled. Like, you'll see logins and log offs. Or maybe even system, like processes and services that are having trouble, errors and warnings. But you won't necessarily have all the auditing you need to like monitor privilege escalation, like we talked about earlier. So, that's where we're going to go next. We're going to set up the audit policy to generate those particular events. Now it's really simple. On a demand controller, you can either do the local security policy, you can do your default domain policy, or you could do your default domain controller policy. But no matter where you go, the actual audit setup is the same.
Okay.
So we're going to jump into the local security policy here. And we do this by going to the Start menu, Administrative Tools, and it's right there in front of you. So you just come up here, select Local Security Policy, and it'll bring us to where we need to go.
And on 12, you'd actually just type out the name for audit security policy, and you pull it up. Or local security policy, and that'll give you link. You'll comment. It'll open the same way.
Exactly, exactly, okay. Now within the security policy, you'll see different options in here. We're specifically going to focus on local policies and then Auto Policy. Now, once we select Auto Policy, you'll see a number of different options here. Now, this is for your basic Audit Policy setup. And for the most part, you won't need to go beyond this point when it comes to setup. But I always like to mention that you do have an Advanced Audit Policy Configuration option there. That's going into a lot more detail. You'll definitely want to reference, either Tech Net or Microsoft 4-specific descriptions, as to what additional auditing that you can enable.
And that would be to either expand or to limit the number of messages you're being sent to improve the overall messaging quality.
Exactly, exactly, yup. So as you can see by default, right here, Process Tracking and System Events are turned on for Success and Failure. What this means, that if there is a successful process event or a failed process event, it'll generate an event in each way. And that's important thing to remember, especially when you enable auditing. If you're not so concerned about successes, then don't turn them on. If you're more concerned about fails, then only select that option when you set up you're auditing.
And that's a key piece just to point out for everyone out there, is that one of the fears that I know a lot administers have when turning on a Log and Event Management System of any kind, especially here, is they don't want to be inundated with lots of quote, unquote, noise. But everyone's noise is different. So, the fact that you can control it right here and say, you know, I only want to know the failures—I really don't care when it succeeds— for any of these, is really critical. You can control the volume on this.
That's exactly right. So in this case, what we do, we're specifically concerned about privilege escalation, right? Well, privilege escalation really is in two different areas within the Windows Audit policy. One is Privilege Use—okay, so we can double click on this, and it just gives us a couple of simple options. Do we want to log Success and Failure? Right? Or you know, one or the other? In this case, we're going to do both, so we click okay. And now that's set up.
Why would you want to log failure? Oh, so and so who doesn't have credentials to do something, is trying to go in and get administrator grips.
Exactly, one of the earliest indicators of malicious activity is several failed attempts.
Right.
All right. And so this is a way to set that up. And then the next thing we want to do is monitor Account Management. So this specifically has to deal with groups and users, all right. So any type of changes that are made to those accounts. This is the audit policy that you want to enable.
And the administrator actions are actually there.
Exactly, so we'll hit Success and Failure on both of those, we click okay, now just like that, we've set up auditing. If we wanted to go a little bit deeper and do things like logons and logoffs, or start looking at file access and things of that nature within the OS itself, you can enable these other options as well. But for the purpose of this demo, this is exactly what we need to do.
And if that looks familiar to you, the log on, when you install the auditing for User Device Tracker, that's how it actually knows which users on which port, because it's actually sending those log on event messages to EDT.
Right, okay, now that completes the demonstration for installing the agent, and setting up the audit policy within the Windows operating system.
That's great.
Okay. So next, what we want to cover is actually using that data. All right, so now we're going to jump over to the LEM Console. So web-based console, as many of you probably already know, or those of you that are new and taking a look at this, this is a web-based console. And basically, there are several different ways that we can see this information. We can look at it in real-time. We can search for it historically. And we can use it for automated notification, which is probably the most critical piece. I want to know when privilege escalation occurs in my network.
And the thing that's really different here, and the thing that we—I think, that took me a little while to figure out, was that you're looking at the actual information being sent from the LEM server out here to the client. So, it's not occasionally polling for these messages that are sitting in a database. You're seeing them as they actually come in. So when you're on the Monitor tab here, you're getting them live, directly from the server and that is one thing that is very different. And when you start doing things like, hey, click on this and turn, incorporate this into a rule. That's something that you really have a chance to play with this, and see the events actually happening in real-time.
As Rob mentioned earlier, this is real-time. This isn't near real-time; this is happening right now.
Right, you're not looking at the results of a database poll. These are coming in and are actually being pushed out to the web client from the server.
Which really changes the whole concept of monitoring, where before it's always, something happened on my network—I have to go back into the database. I have to figure it out and then I'm going to put new policies in place to counteract it, right? With LEM, you can start to think a little bit more proactively. Because now you're seeing these events as they happen, so you can literally look at it and say, "Wow, I need to take care of that action immediately."
Yeah, that's novel. I've never seen that before. Well, let's dig in and find out what that was.
Or for people who are familiar with the Orion and NPM product set, it's the way that you think about traps and Syslog messages coming in, not the way you think about any of the polled data. You know, we've seen it before, but it just wasn't as complete or as in depth as this is giving us here.
If the Trapped and Syslog data came directly to the web client.
All turned on, right.
Exactly.
Cool, now you'll see when you log into a web console, there are all kinds of out-of-the-box filters, rules, reports, you know, whatever you need there.
You gave me the number on that sometime.
It's several hundred. 800+ filters, reports, and rules that come out of the box. So a lot of what I'm going to talk about today, you can probably accomplish very quickly with the out-of-the-box activity.
And you should RTFM, but of course, I don't do that.
Yeah, we're IT guys. We don't do that.
No, and so that's actually the way that I learned it, was I hooked into a couple of basic events. Then I went and found a couple of the out-of-box examples actually that were listed in the manual. And then I tore up some of the existing ones and sort of saved them as new and then started playing with them and deconstructing them and figure out how they worked. So, that was really helpful, starting with that as a base.
Sure. Now, one of the first things that I always like to mention when demonstrating the product, especially when you're talking about events, is event normalization. Because basically what we're doing, is we're taking information, and you'll see this here as I bring up the Event Details. So like in this particular case, I got New Group Member. All right, so we normalize the data to make it easier to process information across multiple device types, right. So this could be Linux, could be Windows, could be Macintosh—it doesn't matter. It's still a New Group Member, right? But what we do is, we parse that information and we place it into these fields. So if I was to emphasize anything within the product to focus on as you learn, as you start to get comfortable, is Event Names and Fields. The more comfortable you are with that, everything else in LEMs becomes very easy, because they're all using the same information.
And I mean, it's sort of the exact opposite of an SNP Trap message, right, where I'm going to go out, I'm going to look at Oids. I'm going to try to figure out what it is that I'm looking at. In this case, by the time that you've actually normalized it into this event details section over here, it's now a logical object. It's pure and clean from whatever back-end system sent it or Gorp had to be accessed in order to pull that information forward, so then everything else that you're going to do inside of Log and Event Manager is much, much simpler because this is where you've—I don't want to say homogenized, because you haven't. But where you have normalized it, you cleaned it, stripped all of that ugly extra context off.
Right, it takes a whole, that additional learning process of understanding raw data, right. Where's this located? Where do I find this? How is this formatted? Right, it kind of takes the regular expression out of it. Not to say that regular expressions aren't handy, but this can definitely make it easier when you're processing a lot of data. So what we're going to do is, we're going to build a quick filter here monitoring for administrative activity, okay. So we go to the plus sign over here and we click New Filter, and that brings up the filter interface. Now pay attention to this, especially you out there. Pay attention to this, because what you'll see, when I jump from here and I go into Build a Search. When I go from there and go to Build a Rule, the interface is the same, so the whole logic behind what you're doing does not change. That's why it's emphasized. Pay attention to fields within an event, because they'll make a difference everywhere on you.
Okay.
Okay, so what we're going to do is, we're going to go in and build a filtering monitoring for administrative activity or Active Directory specific information, right?
Sure.
Okay. So, I'm going to go to my Search bar up here for my event taxonomy, where all my events are listed. And I'm just going to type what I need, especially if I want to look for a singular type of event, like a group change. So we'll hit Group. We just start typing it in—you can see it starts to populate all the group events that I want to monitor.
Okay.
Okay. This is one way of doing it. In this case I can say, well, I want to look for any deleted groups. So basically, all I need to do is just drag that into the Conditions box over here, and you'll see it starts to set up my logic, basically. Now it's and/or Boolean logic that we use, right. So that is indicated over here on the right side of the group. If I mouse over, that blue line with the triangle, it's an ‘And.’
So right there, it just "anded" with itself.
Exactly. If I click on it, it just switches to an ‘Or.’ So if I want to add additional criteria, I can just drag and drop what I need.
So you can make it inclusion area or exclusion area, really, and again, the click just toggles.
Yup, exactly. Yup, so now if want to leave it there, I can do that. That's going to show me all deleted groups that happen within Active Directory. But what I can also do is again, we come down to the fields here. I can say, well, I really only want it for the group name administrator, okay. So I'll scroll down here. Over on this side, okay, and I'll drag Group Name into the Conditions box. Now I'm replacing the original group in there, because I want to focus on a specific field. You'll see when I do that, it still lists the Event Name, Deleted Group—the dot indicates that there's going to be a field used, and then we hit Group Name, and now we have our options.
So basically, it turned it into a property on that object for you, and then you didn't have to create the conditional connecting them up.
Correct, yeah, it builds that connection.
And it's just one line, so it's way easy to read that way.
So we have a Deleted Group where the Group Name field equals, now we simulate contains by using wild cards. Right, so those of you from Orion may be familiar with contains, is not contained, is like, you know, that type of thing. LEM kind of simplifies that whole process. Right, so basically, if I use an asterisk in there, I can say—I can just type in Admin with a trailing asterisk, and now I'm saying Deleted Group, where the Group Name contains—equals, in this case—Admin. So what those, the wild cards will do, is look at for any variation of Admin. So Administrator, Administrators, you know, whatever that group name is.
So that's great, because you're also not worried about regex. This is just wild card replacement and you move on.
Correct. So it's simple. Now, the other approach that you can take is using a group. So let's say I want to monitor all of my administrative activity in one filter, okay? So now what I'll do is, I'll look, I'll go down to Event Groups, and you'll see different Event Groups that are placed in here. Now these common groups— honestly, I found over the years of working with a product, you rarely have to replace them. You can create your own groups. You can go in and create your own event groups, basically consolidating several types of events into one category.
Sort of a pre-normalization library.
Exactly, exactly, but our library's pretty large as far as that goes, so you can really take advantage of what's already in place there. And that's what I'll do here. I want to look at Autoable Group Events. So I will grab Autoable Group Events, and I can do the same thing. I can either drag and drop the whole thing in here, right, or I can jump down and grab that same group name field. Okay, and say I want to look at any group event that's associated with the Admin group. And you see how I just dropped it over the top and it just replaced it, so I didn't have to retype any information in there. What you want to look for is dragging over, just look for the whole thing highlighted in orange. Once it does that, it's just going to replace it, but it'll leave the text in place.
And that was the other thing that I spent a lot of time doing, just sort of experimenting, right? So then I drag it over and see what my results were. Drag it over and see what my results were, and that's how you actually can figure out what a lot of these different event groups contain.
Exactly, exactly, so that really—that's how you set up the logic. Now if you have a console up and running, let's say you're in a NOC or you have eight screens on your desktop—a lot of us IT guys do, Right, and you want a console just up that you glance at every once in a while. Well, this can also get your attention from here, right. So we can go to the notifications area down towards the bottom right here. And we can select some simple notifications. Like a pop up, so it will merely pop up saying "hey, you've got an alert here." "You need to check it out, right?"
Sure.
Or you might play and audible alarm, and this can be an MP3. So you have some flexibility there. So if you want some other rock bank playing or something in the background or a siren going off, you can put those types of sounds in there as well. All right, so in this case, we'll just display a pop-up notification. Now there is some simple additional logic. Really, time and frequency. So don't—throw me the first pop-up when the event happens, but don't show it again until it happens 15 times afterwards, right, or occurs, that event shows up 15 more times, right? So you can set that up in the display area right here. Alright, so that can help slow—especially if you know what's going to generate a lot of events, right, but you just want to keep an eye on it. It's a good way to use that particular pop-up, and you can do that for some of the other notifications, as well.
Okay, and so this is one thing, just to clarify one thing. This is one of the things that got me stuck at first, because coming from working with Orion platform products in particular, you go into the trigger condition, and then you have your action list, and it's sort of immediately bolted. Well, in this case, I was like, well, where's the notification option that's actually going to execute an action? And so, notification is something that you can do on top of a condition that's executing. So, the actual action is something that's going to happen when we get to rules.
Correct. Okay, so now that we've got this filter set up, we can go ahead and save it, and it will then create that filter. In this case, I didn't name it, so bad me. But name your filter. Name your filter, go in, and now any new information that occurs, that meets that criteria that we defined in the filter should populate an event within this area here.
Okay.
Okay. All right, so that shows your covers the ability to filter out events as they're happening. And again, I remind you, this is real-time. All right, so you'll actually see this data flowing in. And one of the early indicators of an issue is you see a bunch of data flowing in after you built a filter, because that could be something's wrong on that system. Right, so that's—and if you don't see an event showing in there right away, it's not really a big deal. It just means that you don't have that happening on your network, and that's probably a good thing when it comes to privilege escalation or changes in Active Directory.
Right.
Okay. So now, the next thing we want to do is talk about, how we can use that same type of information and turn it into an action.
Okay.
Turn it into something as simple as "I want an email." Or it may mean something else: "let's reverse the change that we've made." So one of the advantages of the agent that gets installed on Windows systems is what's called Active Response. Meaning, if somebody's added to a group, I can leverage LEM's active response capability to automatically remove them from the group.
And send an email to the administrator to say that they got added.
Exactly. So you're notifying and you're remediating all at the same time, and you can have as many actions as you want, and they all occur at the second that that event is detected.
And it's really handy, because you could actually, for example, do both. Going back to talking about using an edge case, right. So I'm going to automatically roll back and I'm going to send an email to the administrator. So that then, if people call and complain because that was actually okay and should be allowed, then I will just go in and delete the action to do the rollback. But I'll leave the notification, so that I can continue to be aware of the fact that those are going on forward, but I can actually say, "No, that's an edge case that I'm willing to allow." So you can start with a more aggressive policy of using actions and then actually pull them back.
Right.
Right, while still maintaining the audit trail.
That's right.
So you always can go back and say, oh yeah, Rob did make that change yesterday.
Yes, oh, and the person made that change to the action is also in the audit log as well.
Exactly, exactly.
All right.
All right, so let's jump over to the Rules configuration. So when we go into rules, again, lots of out-of-the-box rules in here that you can probably use, but I think it's—like you guys said— it's more important to see the nuts and bolts here. So we're going to build that rule from scratch. So we hit the plus sign over here. And it brings up our interface. Now, notice the correlation area here. Looks exactly like the filter utility before. That's the key—you're using the same logic. There's some additional capabilities, like the actions and notifications in here, but the same concept, or, the concept is the same as what we did in the filter.
It's still going to be Event, Object.
Exactly.
Got it.
Okay, so in this case, we want to know when anybody is added to a privilege group, all right. So we're going to focus on a singular event and it's called New Group Member. So we can start typing in ‘New’ and you'll see it just lists all of our events that we can choose from. There's our New Group Member. But again, I want to be a little bit more granular here. And I want to focus on a specific group name. So now, we'll go down here and we'll look at the Group Name. Okay, and we'll drag that in. So now that I've got the Group Name entered into the correlation window in here, now I need to tell it what I want to look for. So now in this case, I want it to look for, you know, anybody added to the Administrator Group, so that's what I'll do. I'll just type in Admin here and again, I'm using a shortened term and wild cards to cover any variation of Administrators. So if somebody goes and changes it to Admins or Administrators or something like that, I'm covered.
Got it.
Okay. Now the correlation time, again, allows me to determine when I get notified. Do I want notified after three events, two events, those types of things.
And if you want to be notified anytime, you basically just remove that.
Yup, or just leave it as Default. If you leave it as Default, it's basically real-time.
Okay.
So now that we've got that defined, we can go down and define what action that we want to apply. So obviously, the most popular one, send me an email, all right. So we'll go down here, we'll select dozens of actions in here that you could choose from, but we want to send an email, so we'll grab the Send Email Message. We'll drag it into the Actions box. Now once we do that, we need to select a template. What do we want to tell people, right? Or what do I want to know about? So I'll go in here and I'll just select Account Modification, okay. Then select Users, who I want to receive this.
And these are users you've already defined in the system.
Exactly, right, so I've selected my template. I've selected the users. One key thing to remember is you need to populate these white boxes with Event Information.
Okay.
Okay. So you go back up to your Correlation window, and then in turn, the event that you're focusing on, and just match them up. Okay. So in this case Date, we know that's going to be detection time, or the Date Time stamp, basically. So we'll go to section Time. And basically, you just go through and you drop the appropriate fields into place. That tells LEM that it's going to pull that information directly out of the log and place it into the body of the email.
Okay.
Okay. Now we can add other actions here. So we can go in and we can go down to the Actions. So let's say I want that user removed from the group. So, same thing.
Just roll back.
Just remove the user and put that action right in here. And now we tell it that we want it to remove that user from the group. So now, we've added an email action, as well as a "Remove a User From Group" action. So it's going to notify me that this happened, and whoever else I want to know about it, and remove that user from the group. So that way, they cannot take advantage of those escalated privileges. This happens in real-time. So basically, we've gone from filtering the data in real-time to setting up a rule to automatically notify us of this particular event and reverse the change.
While I'm asleep, on the weekend?
Exactly.
I know, it's really handy, it's great, especially once you actually get that email from someone who says, "hey no, but I really needed it." Oh okay, open a help desk ticket. Go through the proper channels.
Where was your change control?
Where was your change control?
And now you have record, exactly. So that covers how you can set up the auditing, or install the agent, set up the auditing, and then use a filter to see that information as it happens, and use a rule to respond automatically for you.
Excellent. [Techno music] Okay now, you make that look really easy. And I admit I've been in IT for…
Ever.
A really long time, I was going to say. But it did take me a little while to make that switch to start thinking about events as logical objects, and then apply them to the execution flow in the processing chain that LEM is actually using to then get to an action or an alert.
Right.
And that meant that the first couple of weekends, especially coming from a network configuration, typical network management polling background. There were a couple of weekends where I was playing with it, where that learning curve felt a little bit steep, but once I got used to it, it was really, really easy to actually start working with the event chain.
Yeah, that's the key, is that once you get used to it and get your head wrapped around it, it is very, very simple.
Yeah, no scripts, for starters.
Okay, and when Lawrence gets back, I know the first thing he's going to say is, "yeah well, it's all for not if someone messes with your ID controller."
Yeah, that's right, that's exactly what he's going to say, only he'll have his hands in his pockets when he does it.
Oh, sorry. [Laughing]
You know, that's exactly right, though. As important as it is to monitor for privilege escalation, it's just as important to monitor the system itself. A lot of security issues source from people accessing the operating system, making back-end changes that then allow further access to the rest of the network.
If that's the monitored system of choice, and you're actually watching the log, then you can't have people messing around with the system. So what you're really talking about is file change monitoring.
Exactly. All of those systems contain files. Change those files that can create a chain reaction.
Okay, so you'd recommend that as part of what you're doing with an Active Directory of security management. All right, well, can you show us how that works?
I will.
Okay. [Techno music]
So the next piece we wanted to cover, as we discussed in our—or, as we talked about in our earlier discussion—was the file integrity monitor. Right, so the ability to monitor the file system itself on that server, because it's just as critical. So it's actually really simple to do that in LEM. And it's embedded in the agent. So we've already installed the agent. We've already done the auto policy, and we've got a rule set up, but let's monitor the file system. So basically, the way that we do that is, we jump over to Manage, and then Nodes. Now that we're in Manage Nodes, we select the agent, or the domain controller, that we want to configure. So I'll go down and I'll grab domain controller here. I'll go to Connecters. You'll see hundreds of connectors in here. There are several different things, additional log sources, that you could grab if you want to get more visibility, but really, what we're going to focus on is file integrity monitoring or FIM. So just type in FIM and that'll take you to what you need to see. Okay, now on the system, we can monitor two things. We can actually monitor the registry, which can be a critical part of it as well. Or we can monitor the actual file system itself. So I'll monitor the file system here. So basically, we'll go in and edit this template. Now when we do that, when we go in and edit the template, you'll see some existing file monitoring templates embedded in here.
Okay.
Which include one for Windows Server Monitoring. So this is already pre-configured for you out of the box. Now it doesn't mean you can't edit it to monitor specific files or folders. But in this case, you'll see it's monitoring all of our system-related files down here. So batch files, DLLs, Hosts, INIs, the Boot INI--all the most critical files within the operating system. So really, all we need to do is just add this selected monitor and then, you know, if we want to go in and edit and add more information, we can just hit Edit Monitor. But in this case, we're really done, all right. So we go in and hit ‘Save.’
It's now going to pipe those file change events over with everything else.
Yup, it is. Now here is what's really cool about this. And what I don't think a lot of people are aware of is that this is not using Windows File Log. This is embedded in the agent, so you, because everybody knows who—or I should say, everybody who has configured Windows File Editing, understands how incredibly noisy that can be. Well, we've helped pair that down by creating this FIM utility that you can have.
Well, also anything that's trying to get around auditing is going to get around the Windows File Auditing itself and not going to get around this. This is a separate piece.
Exactly.
And in fact, remember the episode we did not too long ago, where we were talking about optimization for LEM, and we talked about narrowing down the number of Windows events that you were actually sending. This is what were talking about.
Exactly, exactly. All right, so that pretty much covers the File Integrity Monitoring piece and especially monitoring the file system itself for the operating system. [Techno music]
That was great. I really like LEM. And it was nice to have a how-to for our users by a real master.
And it was no problem, and I really think our users are going to get a lot out of the product.
And thanks again for coming by the show. I mean, you've been using the product forever. And you're going to stick around for the live chat also, right?
Absolutely.
Which of course, you can participate in because if you're sitting here watching at our homepage, which is lab.solarwinds.com Over this on this side, you have the big chat window, and of course, we're all live chatting with you right now. If you happen to watching this recorded but would like to participate, go to lab.solarwinds.com and sign up for reminders for upcoming episodes in addition to, of course, being able to provide feedback to say things like, "when are you finally going to do a show on LEM?" and then you get to see Rob. So definitely come by the homepage, lab.solarwinds.com and check it out. So, what do you think? Let's get the lads back in here. All right, cool, all right. Let's see if I can see the guys.
Are you zapping me out?
No, no, no, no. I'm just going to bring them back in here.
All right. [Whooshing]
I don't think that's how that was supposed to work.
Pretty cool, though. [Whooshing] That was awesome.
Yeah. I think Digital Conveyor app doesn't like Frodo S very much. Let's try this one more time. [Whooshing]
Well, did you guys have fun?
Yeah, we were sitting comfortably at our desks, so that was good.
Yeah, good. Well Rob, thanks again for coming by. We really appreciated having you on the show, and—
No problem at all, no problem at all.
Good, good, so stick around for the chat. Okay, so I think that about wraps it up for this episode. Tom, you want to take us out?
Sure. I'm Thomas LaRock.
And I'm Lawrence Garvin.
I'm Leon Adato.
I'm Patrick Hubbard.
And I'm Rob Johnson, and thanks for watching SolarWinds Lab. [techno music] SolarWinds Lab Episode 20: Why You Should Monitor Active Directory (AD) Events from Domain Controllers
May 10, 2018 |
Video
In this episode, the Head Geeks showcase SolarWinds Security Event Manager (SEM, formerly Log & Event Manager) to address a critical aspect of IT security and compliance: Active Directory security. AD changes, such as adding users to privileged groups, escalating privileges, and changing user accounts, can create security holes.
The Head Geeks show how to install a SEM agent on a domain controller, edit the domain controller audit policy to log key events, and show how SEM can then monitor all AD events in real-time. Finally, they discuss why you need to monitor file system changes on the Domain Controller itself.
Episode Transcript
Hi, I'm Leon Adato.
And I'm Lawrence Garvin.
And I'm Thomas LaRock.
And I'm Patrick Hubbard and thanks for joining us on another episode of SolarWinds Lab. We have a huge show for you this time. And as much as we try to keep these things under 25 minutes—and we really, really do— this one may go just a little bit long.
That's because we have a lot going on. First, we have all of the Geeks here today.
Hi.
Thank you, Thomas and Leon, for flying and joining us.
That's right.
Thank you.
Thanks for having us.
And second, we are finally, at long last, doing a show on—
Wait for it.
Is it Log & Event Manager?
Yes, exactly.
Ah, awesome.
We're going to do a show on the LEM. You guys have been asking for it on chat for a really long time. And we thought we were going to do LEM and EOC together, but we started doing prep to actually put both of them on the same show and realized that there was so much to talk about with LEM that we're going to save EOC for another day.
In fact, we also found out that doing the EOC prep that that was potentially another 20 minutes all unto itself, as well, so.
That's right.
Right. So here's what we're going to do today. On today's show, we're going to roll up all the questions that we've seen on chat and everything about LEM. We're going to do some how-tos and we're actually going to solve a couple of real issues that customers have when learning SIEM platform.
Yeah, and why are we even doing this show again? Because you guys have been asking questions about this for a really long time at our homepage, which, of course, is lab.solarwinds.com.
So make sure you come by. If over here on the side, you can see a big chat window and you happen to be chatting live with all of us, well then, you probably are at our home page and you even signed up for notifications, so you knew this show was coming. If you did not know that, well then, come by lab.solarwinds.com, sign up, and you'll know about upcoming episodes before they happen. Some of the special things that we do, an occasional— I don't know, we give things away sometimes, all kinds of cool stuff. Yeah, so first, let's actually start this off by de-jargonifying SIEM.
Oh, I know this one, it's security information and event management.
That's right, and if it sounds like two things put together—information and events—kind of jammed together, it sort of is.
Yeah. Because there's this thing called SEM, S-E-M, Security Event Management. That's real-time event detection, analysis, notification—well, not really analysis, notification, detection, those sorts of things, working on things that are known and responding to those things as they occur.
Right, and then you have SIM, which is Security Information Management, which you can think of as the storage and analytics over the long term, with the log data that you've been receiving.
That's right. So, SIEM then is a tool that actually does both. It handles events in real-time and it can raise alerts and take actions, but it also has access to the historic data, so that it can actually spot issues that can only be detected over time.
Right, things like say, Active Directory shenanigans.
Yeah, yeah.
Right.
So then really, the SolarWinds acronym LEM is actually a more direct way to explain it. But why do we take the word "security" out if this is really a full-on security product?
Yeah.
Because back in the day, we had this thing about three-word product names. [Techno music] Okay, so how many of us know how to use LEM? How many of us are familiar in normalizing events? How many of us have 10 years’ experience literally doing everything that you can possibly do with the product?
No.
Nah.
Right, and me either. And since you guys have waited so patiently and added so many comments, please do LEM, please do LEM, over here in the chat, which you should definitely be looking at if you're watching us live. We've actually brought in an expert who, in fact, has more than 10 years, not just with SIEM products in general, but this product specifically. And, in fact, he has more than six years’ experience with this product before SolarWinds even bought Tri-GO, which actually was the originator of this LEM technology.
You know, I was thinking about, just the other day, I mean, LEM is completely focused on security event monitoring for longer than some other generic event collection tools have even been around.
That's true, and so with that, please welcome a true LEM guru, Rob Johnson. [Cheering] Rob, thanks for joining us.
You bet, thank you for having me.
Yeah, and we're not trying to bust your chops and make you sound old and crusty or anything. But you really do have a lot of experience with this product.
True, and really, it's the years of development in the Log & Event Manager technology. And for many of you, this might be a new addition to the product line, but really the platform is about, is as mature as the Orion platform as well.
Just not as sexy as applications.
You know, MPM predates SAM.
Yeah, well what about Patch Manager?
Do I need to bring Cirrus into this?
[Everyone but Will] Network Configuration Manager.
Okay, so Rob, lots of folks know you actually from THWACK. Probably a lot of them are actually over here in the live chat, as a matter of fact, because it's a chance to chat with you live, but you've sort of been the unofficial Head Geek of LEM since you've joined the company. And I figured that you'd be the master to walk us through some of the real deep dive how-tos in terms of being up and running quickly and effectively with LEM.
You bet, and really, if you're coming to this from the Orion platform, this might be very new. Orion, or Virtualization Manager, you know at that point. So it might be new and it's just a different way that LEM processes information, right?
Like, how do you say?
So Orion and Virtualization Manager and some of the other Orion products, they typically rely on a polling process, so they go out, they grab information. They pull it back and then you set up your alerting according. Well, LEM focuses on that information at network speeds, or in true real-time, meaning we don't have to wait for it populate a database. We don't have to wait for it to populate tables, for that matter. We basically process it in line and then alert from there.
Right, and so it's really that whole concept of getting used to events and so, an event flow as opposed to polling and aggregation, so the only thing that's sort of, a little bit the same in the other platform products might be Syslog and NPM. Or SAM—or to some extent, NetFlow.
Exactly, and really, you just have to change your thought process a little bit here when it comes to developing the filters, the rules, and the actions. A lot of the logic is actually very similar in all the products, but it's just a different way of processing the data.
Okay.
So I made a comment the other day about Active Directory. But it's actually a great place to start, because a decent number of viewer questions about LEM were from users having to deal with detecting Active Directory security issues.
Correct, and you know, really, Active Directory's probably one of the most popular tools in your IT environment that you need to monitor, whether that be for security, compliance, or just having a general idea of what's happening on your demand controllers.
All right, so let's do this. Tom and I are working on a segment for the THWACKcamp Holiday THWACK-tacular, and we could stand here while you guys do all this demo, or we could go out and do some on THWACKcamp.
Okay, well, I think I got this under control. So we're going to just take care of you guys right now.
He's got the phone.
Oh.
Just—no, I finally got the app. [whirring]
Whoa.
Yes.
Now I, too, have the Digital Conveyor app, ho, ho, ho.
Yeah, nice Die Hard reference, very good.
Yeah, all right, so yeah, I think it was one of the best ones.
The first one was really the best. [Techno music]
One of the most common security issues that you guys have talked about, especially with Active Directory, is adding users to privileged groups where they effectively get escalated because of special privileges on those groups they really never knew about.
Ah, right, it's the old out-of-sight privilege escalation trick.
Exactly. Right, so they get to take advantage of all the escalated privileges, and could potentially do damage within your network. But I think it's more important to take a look at those groups that are not quite so visible. Administrative groups are very obvious. Every server has them. Everybody uses them. But you also have to create groups that have maybe very specific privileges and to specific things within the environment. And those are even more important to monitor, because they're not so obvious and can even create more damage on some very critical systems in your network.
Right, because you have something where it's maybe designed to use around an edge case, or maybe only for administrators to use, and then it gets applied and nobody really has any real awareness of that. Or maybe you're just in a hurry trying to close the ticket. It's Friday at 4:30 in the afternoon. You've got somebody pounding on you, saying, "Hey, I've got to have access to the server." And they've in the past, never been a problem. You have no reason to think they're asking for anything they're not supposed to and so you just add them to that one special group, but not really fully understand the consequences of doing that.
Right, or maybe they're not taking a shortcut. Maybe a group like BSGF with a description like "Bob's Special Group Fix 2004."
Oh, right, okay.
You know, it doesn't make it easy to understand the permissions it's going to escalate. And it's an innocent mistake.
Right.
Right. Okay, so here, let's do this. I know that there's an out-of-the-box configuration in LEM to monitor and manage exactly this kind of example. But you still have to install the AD agent itself, and then you have to edit the policies and everything to use it. So, how about we demo how to configure this example? But also, let's pretend like the filters and the rules, and the searches—they're not there. And then we'll see end to end how to formulate a security event that we want to catch, and then how to hook up the plumbing in LEM, and how to set up the routing, if you will, inside the whole console. And then we'll just see how it'll make it all work.
Sure thing.
Okay, let's take a look at it. [Techno music] All right, so the first thing we want to do is install the agent on the system, because we have to get logs over to the Log and Event Manager virtual appliance.
On the Active Directory server.
On the Active Directory server. So that's what we're going to go through right now. I'll double click on the Setup utility and we'll just walk through the steps. So one of the things it's going to ask you is for the Manager IP address, and I always like to point this out, because it can be kind of confusing sometimes. The manager IP address, or any time in the context of LEM, that you hear the word "manager," it means the Virtual IP Address or host name.
Got it—the destination for all the information.
Exactly, exactly. So if you choose to accept the agreement, click the accept option right there. And then click next, and now it's going to ask you to install a utility called USB Defender. Now this is—quick little digression, but if this is a physical box, this can be really handy, because it can also let you know if unauthorized USB devices are attached to that system.
Ah, that's great.
So we're going to uncheck this. This is actually a virtual machine M on here. So we'll uncheck and we'll hit next, okay. Now next it's giving us a summary telling us, is it going to install this system? So we're going to go ahead and hit the install button right here. And now goes through the install process.
So this is actually Plinko. [Laughing]
You know, it's like one of those tests that you, or those puzzles that you see at the Cracker Barrel, which the little—that's exactly what I think of it when I see that.
But it's actually a propagation metaphor, right?
Sure.
Okay.
Okay, and then now what it'll do is, it'll start the agent for me. So if there are events being generated in the OS, you're immediately going to see those in the Log and Event Manager web console when it connects.
When you get all the way through here and you're green, you know you're good. You're not in some indeterminate state where maybe it's forwarding or it's not. You either succeed through the installer or you don't.
Correct. So, we'll hit next here and then done. And now we have an agent set up. The next step is to set your auditing policy up so that you're actually generating the events that you want to monitor, right? In a lot of cases, this auditing is not default in an operating system. There'll be some basic things that'll be enabled. Like, you'll see logins and log offs. Or maybe even system, like processes and services that are having trouble, errors and warnings. But you won't necessarily have all the auditing you need to like monitor privilege escalation, like we talked about earlier. So, that's where we're going to go next. We're going to set up the audit policy to generate those particular events. Now it's really simple. On a demand controller, you can either do the local security policy, you can do your default domain policy, or you could do your default domain controller policy. But no matter where you go, the actual audit setup is the same.
Okay.
So we're going to jump into the local security policy here. And we do this by going to the Start menu, Administrative Tools, and it's right there in front of you. So you just come up here, select Local Security Policy, and it'll bring us to where we need to go.
And on 12, you'd actually just type out the name for audit security policy, and you pull it up. Or local security policy, and that'll give you link. You'll comment. It'll open the same way.
Exactly, exactly, okay. Now within the security policy, you'll see different options in here. We're specifically going to focus on local policies and then Auto Policy. Now, once we select Auto Policy, you'll see a number of different options here. Now, this is for your basic Audit Policy setup. And for the most part, you won't need to go beyond this point when it comes to setup. But I always like to mention that you do have an Advanced Audit Policy Configuration option there. That's going into a lot more detail. You'll definitely want to reference, either Tech Net or Microsoft 4-specific descriptions, as to what additional auditing that you can enable.
And that would be to either expand or to limit the number of messages you're being sent to improve the overall messaging quality.
Exactly, exactly, yup. So as you can see by default, right here, Process Tracking and System Events are turned on for Success and Failure. What this means, that if there is a successful process event or a failed process event, it'll generate an event in each way. And that's important thing to remember, especially when you enable auditing. If you're not so concerned about successes, then don't turn them on. If you're more concerned about fails, then only select that option when you set up you're auditing.
And that's a key piece just to point out for everyone out there, is that one of the fears that I know a lot administers have when turning on a Log and Event Management System of any kind, especially here, is they don't want to be inundated with lots of quote, unquote, noise. But everyone's noise is different. So, the fact that you can control it right here and say, you know, I only want to know the failures—I really don't care when it succeeds— for any of these, is really critical. You can control the volume on this.
That's exactly right. So in this case, what we do, we're specifically concerned about privilege escalation, right? Well, privilege escalation really is in two different areas within the Windows Audit policy. One is Privilege Use—okay, so we can double click on this, and it just gives us a couple of simple options. Do we want to log Success and Failure? Right? Or you know, one or the other? In this case, we're going to do both, so we click okay. And now that's set up.
Why would you want to log failure? Oh, so and so who doesn't have credentials to do something, is trying to go in and get administrator grips.
Exactly, one of the earliest indicators of malicious activity is several failed attempts.
Right.
All right. And so this is a way to set that up. And then the next thing we want to do is monitor Account Management. So this specifically has to deal with groups and users, all right. So any type of changes that are made to those accounts. This is the audit policy that you want to enable.
And the administrator actions are actually there.
Exactly, so we'll hit Success and Failure on both of those, we click okay, now just like that, we've set up auditing. If we wanted to go a little bit deeper and do things like logons and logoffs, or start looking at file access and things of that nature within the OS itself, you can enable these other options as well. But for the purpose of this demo, this is exactly what we need to do.
And if that looks familiar to you, the log on, when you install the auditing for User Device Tracker, that's how it actually knows which users on which port, because it's actually sending those log on event messages to EDT.
Right, okay, now that completes the demonstration for installing the agent, and setting up the audit policy within the Windows operating system.
That's great.
Okay. So next, what we want to cover is actually using that data. All right, so now we're going to jump over to the LEM Console. So web-based console, as many of you probably already know, or those of you that are new and taking a look at this, this is a web-based console. And basically, there are several different ways that we can see this information. We can look at it in real-time. We can search for it historically. And we can use it for automated notification, which is probably the most critical piece. I want to know when privilege escalation occurs in my network.
And the thing that's really different here, and the thing that we—I think, that took me a little while to figure out, was that you're looking at the actual information being sent from the LEM server out here to the client. So, it's not occasionally polling for these messages that are sitting in a database. You're seeing them as they actually come in. So when you're on the Monitor tab here, you're getting them live, directly from the server and that is one thing that is very different. And when you start doing things like, hey, click on this and turn, incorporate this into a rule. That's something that you really have a chance to play with this, and see the events actually happening in real-time.
As Rob mentioned earlier, this is real-time. This isn't near real-time; this is happening right now.
Right, you're not looking at the results of a database poll. These are coming in and are actually being pushed out to the web client from the server.
Which really changes the whole concept of monitoring, where before it's always, something happened on my network—I have to go back into the database. I have to figure it out and then I'm going to put new policies in place to counteract it, right? With LEM, you can start to think a little bit more proactively. Because now you're seeing these events as they happen, so you can literally look at it and say, "Wow, I need to take care of that action immediately."
Yeah, that's novel. I've never seen that before. Well, let's dig in and find out what that was.
Or for people who are familiar with the Orion and NPM product set, it's the way that you think about traps and Syslog messages coming in, not the way you think about any of the polled data. You know, we've seen it before, but it just wasn't as complete or as in depth as this is giving us here.
If the Trapped and Syslog data came directly to the web client.
All turned on, right.
Exactly.
Cool, now you'll see when you log into a web console, there are all kinds of out-of-the-box filters, rules, reports, you know, whatever you need there.
You gave me the number on that sometime.
It's several hundred. 800+ filters, reports, and rules that come out of the box. So a lot of what I'm going to talk about today, you can probably accomplish very quickly with the out-of-the-box activity.
And you should RTFM, but of course, I don't do that.
Yeah, we're IT guys. We don't do that.
No, and so that's actually the way that I learned it, was I hooked into a couple of basic events. Then I went and found a couple of the out-of-box examples actually that were listed in the manual. And then I tore up some of the existing ones and sort of saved them as new and then started playing with them and deconstructing them and figure out how they worked. So, that was really helpful, starting with that as a base.
Sure. Now, one of the first things that I always like to mention when demonstrating the product, especially when you're talking about events, is event normalization. Because basically what we're doing, is we're taking information, and you'll see this here as I bring up the Event Details. So like in this particular case, I got New Group Member. All right, so we normalize the data to make it easier to process information across multiple device types, right. So this could be Linux, could be Windows, could be Macintosh—it doesn't matter. It's still a New Group Member, right? But what we do is, we parse that information and we place it into these fields. So if I was to emphasize anything within the product to focus on as you learn, as you start to get comfortable, is Event Names and Fields. The more comfortable you are with that, everything else in LEMs becomes very easy, because they're all using the same information.
And I mean, it's sort of the exact opposite of an SNP Trap message, right, where I'm going to go out, I'm going to look at Oids. I'm going to try to figure out what it is that I'm looking at. In this case, by the time that you've actually normalized it into this event details section over here, it's now a logical object. It's pure and clean from whatever back-end system sent it or Gorp had to be accessed in order to pull that information forward, so then everything else that you're going to do inside of Log and Event Manager is much, much simpler because this is where you've—I don't want to say homogenized, because you haven't. But where you have normalized it, you cleaned it, stripped all of that ugly extra context off.
Right, it takes a whole, that additional learning process of understanding raw data, right. Where's this located? Where do I find this? How is this formatted? Right, it kind of takes the regular expression out of it. Not to say that regular expressions aren't handy, but this can definitely make it easier when you're processing a lot of data. So what we're going to do is, we're going to build a quick filter here monitoring for administrative activity, okay. So we go to the plus sign over here and we click New Filter, and that brings up the filter interface. Now pay attention to this, especially you out there. Pay attention to this, because what you'll see, when I jump from here and I go into Build a Search. When I go from there and go to Build a Rule, the interface is the same, so the whole logic behind what you're doing does not change. That's why it's emphasized. Pay attention to fields within an event, because they'll make a difference everywhere on you.
Okay.
Okay, so what we're going to do is, we're going to go in and build a filtering monitoring for administrative activity or Active Directory specific information, right?
Sure.
Okay. So, I'm going to go to my Search bar up here for my event taxonomy, where all my events are listed. And I'm just going to type what I need, especially if I want to look for a singular type of event, like a group change. So we'll hit Group. We just start typing it in—you can see it starts to populate all the group events that I want to monitor.
Okay.
Okay. This is one way of doing it. In this case I can say, well, I want to look for any deleted groups. So basically, all I need to do is just drag that into the Conditions box over here, and you'll see it starts to set up my logic, basically. Now it's and/or Boolean logic that we use, right. So that is indicated over here on the right side of the group. If I mouse over, that blue line with the triangle, it's an ‘And.’
So right there, it just "anded" with itself.
Exactly. If I click on it, it just switches to an ‘Or.’ So if I want to add additional criteria, I can just drag and drop what I need.
So you can make it inclusion area or exclusion area, really, and again, the click just toggles.
Yup, exactly. Yup, so now if want to leave it there, I can do that. That's going to show me all deleted groups that happen within Active Directory. But what I can also do is again, we come down to the fields here. I can say, well, I really only want it for the group name administrator, okay. So I'll scroll down here. Over on this side, okay, and I'll drag Group Name into the Conditions box. Now I'm replacing the original group in there, because I want to focus on a specific field. You'll see when I do that, it still lists the Event Name, Deleted Group—the dot indicates that there's going to be a field used, and then we hit Group Name, and now we have our options.
So basically, it turned it into a property on that object for you, and then you didn't have to create the conditional connecting them up.
Correct, yeah, it builds that connection.
And it's just one line, so it's way easy to read that way.
So we have a Deleted Group where the Group Name field equals, now we simulate contains by using wild cards. Right, so those of you from Orion may be familiar with contains, is not contained, is like, you know, that type of thing. LEM kind of simplifies that whole process. Right, so basically, if I use an asterisk in there, I can say—I can just type in Admin with a trailing asterisk, and now I'm saying Deleted Group, where the Group Name contains—equals, in this case—Admin. So what those, the wild cards will do, is look at for any variation of Admin. So Administrator, Administrators, you know, whatever that group name is.
So that's great, because you're also not worried about regex. This is just wild card replacement and you move on.
Correct. So it's simple. Now, the other approach that you can take is using a group. So let's say I want to monitor all of my administrative activity in one filter, okay? So now what I'll do is, I'll look, I'll go down to Event Groups, and you'll see different Event Groups that are placed in here. Now these common groups— honestly, I found over the years of working with a product, you rarely have to replace them. You can create your own groups. You can go in and create your own event groups, basically consolidating several types of events into one category.
Sort of a pre-normalization library.
Exactly, exactly, but our library's pretty large as far as that goes, so you can really take advantage of what's already in place there. And that's what I'll do here. I want to look at Autoable Group Events. So I will grab Autoable Group Events, and I can do the same thing. I can either drag and drop the whole thing in here, right, or I can jump down and grab that same group name field. Okay, and say I want to look at any group event that's associated with the Admin group. And you see how I just dropped it over the top and it just replaced it, so I didn't have to retype any information in there. What you want to look for is dragging over, just look for the whole thing highlighted in orange. Once it does that, it's just going to replace it, but it'll leave the text in place.
And that was the other thing that I spent a lot of time doing, just sort of experimenting, right? So then I drag it over and see what my results were. Drag it over and see what my results were, and that's how you actually can figure out what a lot of these different event groups contain.
Exactly, exactly, so that really—that's how you set up the logic. Now if you have a console up and running, let's say you're in a NOC or you have eight screens on your desktop—a lot of us IT guys do, Right, and you want a console just up that you glance at every once in a while. Well, this can also get your attention from here, right. So we can go to the notifications area down towards the bottom right here. And we can select some simple notifications. Like a pop up, so it will merely pop up saying "hey, you've got an alert here." "You need to check it out, right?"
Sure.
Or you might play and audible alarm, and this can be an MP3. So you have some flexibility there. So if you want some other rock bank playing or something in the background or a siren going off, you can put those types of sounds in there as well. All right, so in this case, we'll just display a pop-up notification. Now there is some simple additional logic. Really, time and frequency. So don't—throw me the first pop-up when the event happens, but don't show it again until it happens 15 times afterwards, right, or occurs, that event shows up 15 more times, right? So you can set that up in the display area right here. Alright, so that can help slow—especially if you know what's going to generate a lot of events, right, but you just want to keep an eye on it. It's a good way to use that particular pop-up, and you can do that for some of the other notifications, as well.
Okay, and so this is one thing, just to clarify one thing. This is one of the things that got me stuck at first, because coming from working with Orion platform products in particular, you go into the trigger condition, and then you have your action list, and it's sort of immediately bolted. Well, in this case, I was like, well, where's the notification option that's actually going to execute an action? And so, notification is something that you can do on top of a condition that's executing. So, the actual action is something that's going to happen when we get to rules.
Correct. Okay, so now that we've got this filter set up, we can go ahead and save it, and it will then create that filter. In this case, I didn't name it, so bad me. But name your filter. Name your filter, go in, and now any new information that occurs, that meets that criteria that we defined in the filter should populate an event within this area here.
Okay.
Okay. All right, so that shows your covers the ability to filter out events as they're happening. And again, I remind you, this is real-time. All right, so you'll actually see this data flowing in. And one of the early indicators of an issue is you see a bunch of data flowing in after you built a filter, because that could be something's wrong on that system. Right, so that's—and if you don't see an event showing in there right away, it's not really a big deal. It just means that you don't have that happening on your network, and that's probably a good thing when it comes to privilege escalation or changes in Active Directory.
Right.
Okay. So now, the next thing we want to do is talk about, how we can use that same type of information and turn it into an action.
Okay.
Turn it into something as simple as "I want an email." Or it may mean something else: "let's reverse the change that we've made." So one of the advantages of the agent that gets installed on Windows systems is what's called Active Response. Meaning, if somebody's added to a group, I can leverage LEM's active response capability to automatically remove them from the group.
And send an email to the administrator to say that they got added.
Exactly. So you're notifying and you're remediating all at the same time, and you can have as many actions as you want, and they all occur at the second that that event is detected.
And it's really handy, because you could actually, for example, do both. Going back to talking about using an edge case, right. So I'm going to automatically roll back and I'm going to send an email to the administrator. So that then, if people call and complain because that was actually okay and should be allowed, then I will just go in and delete the action to do the rollback. But I'll leave the notification, so that I can continue to be aware of the fact that those are going on forward, but I can actually say, "No, that's an edge case that I'm willing to allow." So you can start with a more aggressive policy of using actions and then actually pull them back.
Right.
Right, while still maintaining the audit trail.
That's right.
So you always can go back and say, oh yeah, Rob did make that change yesterday.
Yes, oh, and the person made that change to the action is also in the audit log as well.
Exactly, exactly.
All right.
All right, so let's jump over to the Rules configuration. So when we go into rules, again, lots of out-of-the-box rules in here that you can probably use, but I think it's—like you guys said— it's more important to see the nuts and bolts here. So we're going to build that rule from scratch. So we hit the plus sign over here. And it brings up our interface. Now, notice the correlation area here. Looks exactly like the filter utility before. That's the key—you're using the same logic. There's some additional capabilities, like the actions and notifications in here, but the same concept, or, the concept is the same as what we did in the filter.
It's still going to be Event, Object.
Exactly.
Got it.
Okay, so in this case, we want to know when anybody is added to a privilege group, all right. So we're going to focus on a singular event and it's called New Group Member. So we can start typing in ‘New’ and you'll see it just lists all of our events that we can choose from. There's our New Group Member. But again, I want to be a little bit more granular here. And I want to focus on a specific group name. So now, we'll go down here and we'll look at the Group Name. Okay, and we'll drag that in. So now that I've got the Group Name entered into the correlation window in here, now I need to tell it what I want to look for. So now in this case, I want it to look for, you know, anybody added to the Administrator Group, so that's what I'll do. I'll just type in Admin here and again, I'm using a shortened term and wild cards to cover any variation of Administrators. So if somebody goes and changes it to Admins or Administrators or something like that, I'm covered.
Got it.
Okay. Now the correlation time, again, allows me to determine when I get notified. Do I want notified after three events, two events, those types of things.
And if you want to be notified anytime, you basically just remove that.
Yup, or just leave it as Default. If you leave it as Default, it's basically real-time.
Okay.
So now that we've got that defined, we can go down and define what action that we want to apply. So obviously, the most popular one, send me an email, all right. So we'll go down here, we'll select dozens of actions in here that you could choose from, but we want to send an email, so we'll grab the Send Email Message. We'll drag it into the Actions box. Now once we do that, we need to select a template. What do we want to tell people, right? Or what do I want to know about? So I'll go in here and I'll just select Account Modification, okay. Then select Users, who I want to receive this.
And these are users you've already defined in the system.
Exactly, right, so I've selected my template. I've selected the users. One key thing to remember is you need to populate these white boxes with Event Information.
Okay.
Okay. So you go back up to your Correlation window, and then in turn, the event that you're focusing on, and just match them up. Okay. So in this case Date, we know that's going to be detection time, or the Date Time stamp, basically. So we'll go to section Time. And basically, you just go through and you drop the appropriate fields into place. That tells LEM that it's going to pull that information directly out of the log and place it into the body of the email.
Okay.
Okay. Now we can add other actions here. So we can go in and we can go down to the Actions. So let's say I want that user removed from the group. So, same thing.
Just roll back.
Just remove the user and put that action right in here. And now we tell it that we want it to remove that user from the group. So now, we've added an email action, as well as a "Remove a User From Group" action. So it's going to notify me that this happened, and whoever else I want to know about it, and remove that user from the group. So that way, they cannot take advantage of those escalated privileges. This happens in real-time. So basically, we've gone from filtering the data in real-time to setting up a rule to automatically notify us of this particular event and reverse the change.
While I'm asleep, on the weekend?
Exactly.
I know, it's really handy, it's great, especially once you actually get that email from someone who says, "hey no, but I really needed it." Oh okay, open a help desk ticket. Go through the proper channels.
Where was your change control?
Where was your change control?
And now you have record, exactly. So that covers how you can set up the auditing, or install the agent, set up the auditing, and then use a filter to see that information as it happens, and use a rule to respond automatically for you.
Excellent. [Techno music] Okay now, you make that look really easy. And I admit I've been in IT for…
Ever.
A really long time, I was going to say. But it did take me a little while to make that switch to start thinking about events as logical objects, and then apply them to the execution flow in the processing chain that LEM is actually using to then get to an action or an alert.
Right.
And that meant that the first couple of weekends, especially coming from a network configuration, typical network management polling background. There were a couple of weekends where I was playing with it, where that learning curve felt a little bit steep, but once I got used to it, it was really, really easy to actually start working with the event chain.
Yeah, that's the key, is that once you get used to it and get your head wrapped around it, it is very, very simple.
Yeah, no scripts, for starters.
Okay, and when Lawrence gets back, I know the first thing he's going to say is, "yeah well, it's all for not if someone messes with your ID controller."
Yeah, that's right, that's exactly what he's going to say, only he'll have his hands in his pockets when he does it.
Oh, sorry. [Laughing]
You know, that's exactly right, though. As important as it is to monitor for privilege escalation, it's just as important to monitor the system itself. A lot of security issues source from people accessing the operating system, making back-end changes that then allow further access to the rest of the network.
If that's the monitored system of choice, and you're actually watching the log, then you can't have people messing around with the system. So what you're really talking about is file change monitoring.
Exactly. All of those systems contain files. Change those files that can create a chain reaction.
Okay, so you'd recommend that as part of what you're doing with an Active Directory of security management. All right, well, can you show us how that works?
I will.
Okay. [Techno music]
So the next piece we wanted to cover, as we discussed in our—or, as we talked about in our earlier discussion—was the file integrity monitor. Right, so the ability to monitor the file system itself on that server, because it's just as critical. So it's actually really simple to do that in LEM. And it's embedded in the agent. So we've already installed the agent. We've already done the auto policy, and we've got a rule set up, but let's monitor the file system. So basically, the way that we do that is, we jump over to Manage, and then Nodes. Now that we're in Manage Nodes, we select the agent, or the domain controller, that we want to configure. So I'll go down and I'll grab domain controller here. I'll go to Connecters. You'll see hundreds of connectors in here. There are several different things, additional log sources, that you could grab if you want to get more visibility, but really, what we're going to focus on is file integrity monitoring or FIM. So just type in FIM and that'll take you to what you need to see. Okay, now on the system, we can monitor two things. We can actually monitor the registry, which can be a critical part of it as well. Or we can monitor the actual file system itself. So I'll monitor the file system here. So basically, we'll go in and edit this template. Now when we do that, when we go in and edit the template, you'll see some existing file monitoring templates embedded in here.
Okay.
Which include one for Windows Server Monitoring. So this is already pre-configured for you out of the box. Now it doesn't mean you can't edit it to monitor specific files or folders. But in this case, you'll see it's monitoring all of our system-related files down here. So batch files, DLLs, Hosts, INIs, the Boot INI--all the most critical files within the operating system. So really, all we need to do is just add this selected monitor and then, you know, if we want to go in and edit and add more information, we can just hit Edit Monitor. But in this case, we're really done, all right. So we go in and hit ‘Save.’
It's now going to pipe those file change events over with everything else.
Yup, it is. Now here is what's really cool about this. And what I don't think a lot of people are aware of is that this is not using Windows File Log. This is embedded in the agent, so you, because everybody knows who—or I should say, everybody who has configured Windows File Editing, understands how incredibly noisy that can be. Well, we've helped pair that down by creating this FIM utility that you can have.
Well, also anything that's trying to get around auditing is going to get around the Windows File Auditing itself and not going to get around this. This is a separate piece.
Exactly.
And in fact, remember the episode we did not too long ago, where we were talking about optimization for LEM, and we talked about narrowing down the number of Windows events that you were actually sending. This is what were talking about.
Exactly, exactly. All right, so that pretty much covers the File Integrity Monitoring piece and especially monitoring the file system itself for the operating system. [Techno music]
That was great. I really like LEM. And it was nice to have a how-to for our users by a real master.
And it was no problem, and I really think our users are going to get a lot out of the product.
And thanks again for coming by the show. I mean, you've been using the product forever. And you're going to stick around for the live chat also, right?
Absolutely.
Which of course, you can participate in because if you're sitting here watching at our homepage, which is lab.solarwinds.com Over this on this side, you have the big chat window, and of course, we're all live chatting with you right now. If you happen to watching this recorded but would like to participate, go to lab.solarwinds.com and sign up for reminders for upcoming episodes in addition to, of course, being able to provide feedback to say things like, "when are you finally going to do a show on LEM?" and then you get to see Rob. So definitely come by the homepage, lab.solarwinds.com and check it out. So, what do you think? Let's get the lads back in here. All right, cool, all right. Let's see if I can see the guys.
Are you zapping me out?
No, no, no, no. I'm just going to bring them back in here.
All right. [Whooshing]
I don't think that's how that was supposed to work.
Pretty cool, though. [Whooshing] That was awesome.
Yeah. I think Digital Conveyor app doesn't like Frodo S very much. Let's try this one more time. [Whooshing]
Well, did you guys have fun?
Yeah, we were sitting comfortably at our desks, so that was good.
Yeah, good. Well Rob, thanks again for coming by. We really appreciated having you on the show, and—
No problem at all, no problem at all.
Good, good, so stick around for the chat. Okay, so I think that about wraps it up for this episode. Tom, you want to take us out?
Sure. I'm Thomas LaRock.
And I'm Lawrence Garvin.
I'm Leon Adato.
I'm Patrick Hubbard.
And I'm Rob Johnson, and thanks for watching SolarWinds Lab. [techno music] 
