Announcer: This episode of TechPod is brought to you by THWACK.com, the SolarWinds community for IT pros. That’s where you’ll find the monitoring for managers form, where we look at monitoring from a manager’s point of view. Join the conversation at THWACK.com/m4m.
Leon: Paul Guido has worn several hats during his IT journey. But now he’s made the leap from deep-dive individual contributor to managing teams of the same kind of IT pros he was and still is. There are lessons in that journey, not just for IT practitioners considering the next step of their career but also for managers who came to the job from a different and non-technical direction. I’m Leon Adato , and today on TechPod, I’ll be talking to Paul to find out what insights and advice he has to help managers do a better job of leading their IT teams. Paul, welcome to TechPod.
Paul: Yeah. Thank you very much, Leon. I really hope I can live up to that intro, to say the least.
Leon: No. I have no doubt. I have no doubt. So before we dive into this incredibly helpful and necessary conversation, I want to do some shameless self-promotion and give you a chance to talk about yourself, what you do day to day and where people can find you, if they want to hear more about how you think about life.
Paul: Yeah, no. Long time ago I did ham radio and in part of my ham radio career, I actually helped build networking in the 1980s, late eighties, and made-
Leon: With rocks and pigeons and smoke signals.
Paul: Actually, no. Even better than that. We took radios out of garbage trucks in Dallas. It would be bought by the pallet and that would be really good data transmitters and modified them to make that actually work.
Leon: Oh my God.
Paul: We were able to shoot signals 60, 65 miles to the next hop terrestrially and built a network throughout South Texas. It was nice, but in the process I learned frame tracing and all of these other skills that can really handy later. I did all this while I was still doing construction and carpentry and maintenance and stuff like that. So I wasn’t even in IT when I did all that, but that would help kind of accelerate my career when I did actually start getting into computers for a living.
Leon: Got it. Okay. If people want to find you out on the interwebs, where can they do that?
Paul: Twitter is a good spot @Radioteacher. And also I’m on THWACK as Radioteacher. What’s that?
Leon: I said it’s good to be consistent.
Paul: Yes, it is. I helped taught have some ham radio classes for the local radio club here in San Antonio back in the early 2000s and Google was giving away email addresses. I grabbed one. So there you go. But also like you were saying, I moved into management and manage a great crew of people that are very dedicated to security. It’s really nice. It’s a different management style that you need for knowledge workers. I have seen other types of management styles over the years. You just can’t use a hammer to order people around this isn’t drill sergeant work. This is work to support your team and to make sure that they are best positioned to do the best work that they can, and to make sure that they are taken care of. I don’t need to take care of me. I need to take care of them.
Leon: Yeah. The old joke about the beatings will continue until morale improves is definitely not true here. So before we get into some of the lessons learned, I want to start off just by framing, you were an individual contributor, you worked in networking, you worked in, obviously, you got into monitoring. You’re a SolarWinds MVP on THWACK.com. So obviously that’s where you’ve got your technical chops, but tell me about your transition to management when you made the leap to the dark side. Really, I mean we like to joke about it here at SolarWinds, but really management is just as valid and just as important as any other part of the team. So tell me about that moment when you became a manager and also how your job is different today than it was as an individual contributor.
Paul: Well, I mean, even when you’re doing individually contributions to different things, you have to work on teams, whether you’re working on a project, hardly anything ever happens for one person in a vacuum. You have to be able to get together and work as a team to make things happen. Whether you’re working with a vendor or an outside resource or whatever. That’s true when we were doing our VMware rollouts, our SAN rollouts. These things are very complex mechanisms that require a lot of expertise. You have to be able to work with the people that they come in to do that type of work. And so my management background started even before IT. I did construction for a living. I had to manage jobs. I’d take care of the books, take care of the payroll.
Paul: So managing those job sites, getting all the proper people there in the proper order is such a big deal to make things run smoothly. So back in the eighties, I was doing that, moving that forward, doing that once again, working with the construction of building the networks and the equipment that goes into the sophisticated networks today. Whether it’s once again, SANs, whatever the parts and pieces that you’re getting from the data, whether it’s in or out or through or processed whatever. And securing it, right?
Paul: One person can’t secure a network. You have to have everybody working together. Everybody, even frontline employees, everyone has to work together to get that done. So, that started my whole management end of it, kind of continues forward today. I get to work with some really talented, really sharp individuals that are experts in their field, and it is so wonderful going to work every day to work with these teams.
Leon: That’s fantastic. Okay. So let’s get into the nitty gritty again, looking at things you are a manager who has a technical background, but thinking about the folks who came to management without the benefit of that experience, what are some of the tools or terms or things you need to know to lead folks who do monitoring when you can’t subnet in your head automatically, and you haven’t been around long enough to know when SNMP was a year-old and things like that. When you don’t have the benefit of that, what are what’s some of your advice or insight you can share with non-technical managers who find themselves leading teams of monitoring engineers.
Paul: In some ways they have a couple of advantages. Being a technical person, and then moving into the management field, a lot of the times I opine way too much about “back in my day we used to it this way.”
Leon: We used to write the network with magnets bit by bit and we… Okay.
Paul: We talked about, no, no, no, this is a five meg hard drive. Full height five and a quarter. Those kinds of things. It’s nice to have the history of that information but it’s not necessarily pertinent to the roles that you’re doing. So non-technical managers don’t generally have those kinds of problems. Right? Where technical managers, we have to unlearn some of the conversations that we’re having with our teams and stuff. So that’s always an interesting thing from a technical point of view. The non-technical managers normally they’ve had experience being brought up into that area. So, maybe they managed a smaller group of non-technical teams, but when you start managing technical teams or high-end teams, let’s say you’re in doing risk or fraud or any of the bookkeeping, finance, whatever, very technical teams, very intelligent workers. You have to be able to manage them with finesse. You need to be able to look for the constraints that they’re having, getting their work done. You have to be able to remove those constraints so they can work as efficiently as possible.
Leon: Right. One of the things that I’ve always said about managing, again, those highly skilled technical workers is that you don’t have to be the best technical person in the room. And in fact, you shouldn’t be if you are leading them. What you should do is you should make sure that you’ve surrounded yourself with really good solid technical people. You should tell them what the business goals are. You should ask them how to achieve those business goals, how the technology can help achieve those business goals. You should believe the answers they give you and then you should do everything you can to keep the floor clean. Meaning like you said, remove those obstacles for them to execute what they just told you. I find that bad managers of technical teams will somehow short circuit some part of that process either they surround themselves with untrustworthy folks who don’t have the business’s best interests at heart or they don’t communicate what the business goal is. So that the IT folks are working on completely the wrong thing or they ask what to do but they don’t believe them “or, or, or…” That kind of thing.
Leon: So you really just summarized really well the thing that a non-technical manager can do and be really effective in part in the process. You also mentioned that again, as somebody who leads the monitoring folks, when we were preparing for this, you had talked about how knowing what’s going on, and actually you started what I’m effectually calling the managers garden of monitoring terms used differently or wrong answers only. And you gave me two, one, was observability. Now, observability to an IT person, to a IT pro means something very specific about logging and tracing and poll values and things like that. But you interpret observability as a manager to mean something completely different.
Paul: Well, I mean, from a IT, and from a security point of view, for me, observability is I cannot manage what I don’t know about. So if there is a hidden part of the network or if there’s equipment on the network that I’m not aware of, it’s difficult to be able to give it a lifecycle. It gets built, it gets maintained, maintained as the thing that gets forgotten or ignored quite a bit. And then at the end, when it’s past its useful life or it cannot be patched or it has vulnerabilities you can’t do anything about, it’s got to be decommissioned. And how do you decommission that in an orderly manner and bring in, let’s say, a new piece of equipment.
Paul: You bring in the new piece of equipment that starts the whole cycle over again, but the first part needs to be, how are we going to manage this? What are we going to do? What people is it going to take? What processes? How has that thing going to be configured? So it’s the most secure as possible. Big things these days which most people didn’t think about in 2005. So important in 2021.
Leon: Right? And I would argue that the concept of observability as a manager extends to the staff, that observability means that as a manager, I have a vested interest in knowing how you’re doing and what you’re doing. And that if I don’t know in the same way that you just said, if there’s a piece of equipment that hits the network, and I don’t know about it, I can’t put a lifecycle on it. If I don’t know what’s going on with you, that doesn’t mean oversharing, but it means I need to know if you are struggling, whether it’s struggling just generally speaking or you’re dealing with a health issue or you’re just struggling with a concept or an idea or a particular task. If the manager doesn’t know it, they can’t bring any workflow technique to bear.
Leon: I was not a manager. I was a team lead on a monitoring project and I had assigned some work. And three weeks later I found out that one of the team members had made zero progress on a particular thing. He was really struggling with this one task and he was stuck at the very beginning but he was so proud about being able to do things himself that he wasn’t going to tell me about it until I basically held his feet to the fire.
Leon: And I said, you do realize that I’ve been covering your work for three weeks. And I could have answered this question in about five minutes. And that was when I told the rest of the team, look, if you are struggling with something for more than two hours, you’ve Googled it, you’ve asked, you whatever, you get two hours after that, you need to throw up a flag. That does not mean you’re weak. It does not mean that you failed. It does not mean that you’re a loser. It means you need help. Right? And that’s also observability.
Paul: And that’s one of the things you also when I’m working with the team, I make it quite clear over and over again. I don’t care when you call me, call me 24 by seven. If you have a problem, I’d rather you call me and let’s work together on this than either you suffer in silence or you’re concerned about something or there’s something bugging you, I want to make sure that once again, the people are taken care of. That they, like you’re saying, that that kind of problem goes away. At least gets worked on by a team, not an individual that needs to fret about it.
Leon: Right. The other manager’s garden of monitoring terms wrong answers only that you gave me was unknown unknowns. Now, again, as an IT pro when I say unknown unknowns, that sort of fits into the same observability, it’s a term that Charity Majors, who’s CEO of Honeycomb.io talks about all the time, the high cardinality events. Right? We talk about that from a monitoring standpoint but you’re using it differently as a manager. What do you mean from a management standpoint of unknown unknowns?
Paul: It goes hand in hand with the observability. Having those systems out there, having pieces and parts out there, depending on how you work with the other teams. If they’re not communicating to your team that what’s going on in the network, you have to be able to find that information or get that information or build the trust bridges with the other teams.
Paul: So you can find the unknown information that you really need to know. So that’s kind of deal. You got to scour your resources internally to find those things from a security point of view, one of my focuses now, primary focus, is being able to find that information and stuff. And once again, you have to work with the other teams. I am very fortunate. The teams I’m working with today, everything’s about openness about all the information that they have. They’re glad to share it with everyone. I guarantee you that is not the case at many organizations out there. Everybody wants to be a fiefdom. They don’t want to share. They want to keep their information or their… For whatever silly reason they might have. They don’t want to work with the other teams. Fortunately, I think a lot of companies are not putting up with that anymore out of their managers and they’re making them go away.
Leon: Right. And I think there’s an unreasonable, and I’m going to label that, unreasonable fear that talking about the dirty laundry will somehow expose personal weakness or failure. And it would be a cause for censure or loss of credibility.
Leon: I want to pivot though because although we say we’re talking about non-technical managers, I don’t want to imply that you can slide into the job knowing nothing and never learning anything. So if you’re talking to somebody who doesn’t have a technical background, where can they educate themselves appropriately? Where can they find the appropriate level of new knowledge and new information so they can be even more effective in what they’re doing?
Paul: Well, one of the things is YouTube but my favorite right now especially from a security point of view, but security does so much more than just say security. There’s other ways to find out what’s going on at a group called BSides. BSides is a sub kind of small regional security conferences that happen throughout the world. Last count, I think there’s about 400 of these that happened every year. A lot of them have gone virtual during the pandemic. Some of them are starting to open up back again as physical events, regardless. There is a number of videos out there from all of these conferences that you can find online.
Paul: Just about any subject you were thinking of, you probably could do at a BSides plus you get to network with people in your area, regardless of what they’re doing. It’s probably a good idea of having those kinds of networking aspects built into your regime. Right? You want to talk to people in your field and the best way to find them, find local people, find an organization that’s doing what you’re doing, whether if you’re a manager in your PMP, get with the local PMP group and work with them. If you’re a security person, there’s ISSA, there’s ISC2, local networking, local mentoring is the best way to try and gather more knowledge. Not only do you network with people that are bright in area, you have someone you can call for advice.
Leon: And also, as we were preparing for this, you mentioned a book that is fairly iconic in IT circles right now, what was that one?
Paul:
The Phoenix Project. And The Phoenix Project, though it’s primarily around the writing of the code and maintaining of the code for this organization and the fast, quick processes that happen there. It goes back on the Toyota process is a wonderful book for any kind of manager to read not just for those types of processes but everyday work that you do. And I think I’ve already used the word constraint. Right? What constraints is keeping somebody? What impedance is keeping somebody from getting their job done in a quick and judicious manner? What as being a problem for them that we can fix? Right? And some of those things they put in the book, they’re simple fixes. Right? Oh, we do this instead of that or we bring another machine in that does this other thing that’s a constraint and things move much better.
Paul: Those are the kinds of deals that you can learn and hopefully take to heart using a board with post-it notes to find out what’s going on. What do you have in the pipeline? How many people you can actually feed into a problem and still manage the problem? Right? Sometimes adding more people doesn’t help. Sometimes not having enough people is bad. Right? So you got to find those balances and stuff. And The Phoenix Project goes through that in a nice personable storytelling kind of way. It is I think they have something called The Cybersecurity Canon. It’s a list of books that you should read. And The Phoenix Project is one of them. There is a number of other books out there as well. I think we’ll probably get some links. We’ll put it here in the end of the deal. This year they actually added to the Canon “Code Girls” which is about code breaking in the United States using I think 10,000. I haven’t read the book yet. I’m dying to read it. The 10,000 women recruited from colleges and it is one of the most unsung World War II stories out there. So…
Leon: That’s going on my list. I want to add one too, Accidental Empires by Robert X. Cringely, which is not really his name…
Paul: That’s awesome… That’s an old book. It’s a great book.
Leon: And here’s why. Again, we’re talking about ways that non-technical managers can educate themselves and be familiar. Again, we’re not trying to turn managers into CCIEs, you know, network engineers, but Accidental Empires goes all the way back to the 1960s in the Xerox P-A-R-C Palo Alto Research Center, the PARC Project. And it talks about how technology started there and how it has expressed itself, even down to today. It gives you a historical understanding of the people and the technology and how it started and how it has grown and changed over time. And it’s told in an incredibly entertaining and fun way. So it’s an easy read, but again, as a non-technical manager who is in charge of technical people, it gives you a framework and a foundation to relate to your team and to have a better understanding of the entire genre.
Leon: So again, if you’re writing down these titles: Stop it, put your hands back on the wheel, pay attention to the road. There will be show notes and you’ll be able to get all these links along the way. I want to-
Paul: And by the way, in that book, that I’m pretty sure that story’s in there. The one about Intel’s chip problems that they were having with production. How they had such a low yield, that is a wonderful story and it has so many lessons in that one story that everybody should be familiar with it. I won’t spoil it, but it is so important to understand what everyone does and what the company actually does for a living.
Leon: Yeah. And it’s not a delightful surprise that you find out at the end, but it is delightful that you find out that the fix to the problem of low yield and a lot of bad chips coming out and why that was happening and how easy it was to fix. But it was very personal. Okay.
Paul: I had a friend in Compaq at the time and he remembers taking sleeves of processors and throwing them in dumpsters.
Leon: Yeah…yeah.
Paul: Because once they found out what the problem was, they literally trashed the entire lots that was made during that time period.
Leon: Right. Because you just couldn’t trust a single one. Okay. I want to pivot a little bit. We talked about what’s important to leading folks who do monitoring, meaning being the leader who sets direction, but I also think that there’s an important thing for managers to understand about what it takes to lead the engineers themselves. How do you reward or encourage or incentivize this very specific type of knowledge worker?
Paul: Provide them as much education as they can consume. Give them time to be able to do things that they enjoy. Making sure that they have really great work-life balance. I’m very fortunate that the organization I’m working with feels very strongly about that. The previous place I work, I probably pulled three to four all-nighters at least three every year. I did my first all-nighter about a week and a half ago with this company after about 14 months. And so the reason I did it is so I could let my team sleep. And if I needed somebody, I could call them. It just so happens I didn’t need anybody. The system weren’t ready for them to come online and start doing their job.
Paul: So once they started showing up about 6:30, 7:00 in the morning, I went to bed. But, but once again, I’d rather inconvenience myself, make sure that they’re well rested and fresh and ready to go that next day because one of those deals telecom people messed us up and delayed us 10 hours. Circuits are, circuits are hard.
Leon: They’re really hard. And I just want to emphasize that again, sometimes you, Paul, can cover for things because you have that technical background, but what you’re talking about, wasn’t a technical cover. It was just a staffing being on the floor, being ready to go kind of coverage that and it emphasizes something you said to me before we started recording, which is just showing the team that you have their back, that you’re willing to get in there and do what you can. You’re not going to do what they can, you’re going to do what you can. And also I think in a very IT way, you’re a firewall, I think as a manager.
Paul: Absolutely. And so whenever we have discussions with different audit type functions or anytime else we’re interfacing with other teams, I definitely would like to be there to make sure that everybody has an equal role in equal access at the conversation for my team, for their team, their management, our management, everybody’s working together to try and solve whatever problem we’re working on. It’s really interesting some of the different audit situations that happen especially in the financial services area that you want to make sure that everybody is taken care of. It generally, if somebody is oversharing and an audit, it’s me. I’m working on that. I really am working on that. But that’s the kind of deal is we want to make sure that we’re providing the information that they request and that the information that they request is a reasonable expectation as well.
Leon: Excellent. And the thing I want to say just to close out this section is that everything that we just described are things that any manager can do. This is not about technical acumen. This is not about knowing that one secret GRE Frankel command or being able to load a UNIX system from tape or whatever. These are techniques that non-technical folks who are leading IT, again, specialized knowledge workers can do and can do well on day one, as long as you’re prepared that you have the mental framework, that this is how the job is going to be done. But I do want to move to another section. We were talking about managing the team, I won’t say, managing down, but managing the team, but then there’s the other side of management, which is managing up dealing with the upper leadership, whether that’s C-level or VP-level executives.
Leon: And I’m going to ask the same question that I started the previous section off, I’m going to ask it, but in a different, in this upward direction, what are the tools or terms or concepts that a non-technical manager would need to represent monitoring in the boardroom? So now you’re walking into the, what I actually call oak avenue or mahogany row, all those hoity-toity leaders who never get down to visit the troops, and you’re going to represent these people. You’re going to represent your team, your monitoring engineers, and monitoring as a concept. What are some things that non-technical managers should know or be able to do that successfully?
Paul: It’s funny is a lot of the board members and upper management at my previous organization. And even at this organization, I got to know very, very well. Matter of fact, one of the board members I actually worked for a few years, years even prior. So it’s really funny being able to personally know the board, personally know upper management that happens a lot, especially when you’ve been there a couple of decades not to have that kind of timeframe of their current company, but I know a lot of personal connections that we have between those types of… And that’s the kind of deal, once again, you want to build personal relationships with the upper management, with the board. When you also, you want to know the goals and the values that you have as an organization, and you want to make sure that whatever you’re expressing to them that follows those goals and those values.
Paul: So if you’re talking about procedures and policies, you want to say that these are in line with what we’re trying to do as an organization, to help our customers, to help our members, to help our employees out, to make sure that everyone does well. And it’s easy for them to do their jobs, make, once again, get rid of constraints, make things a much simpler process. Then if you also, you want to make sure that if there’s a problem out there, you want to make sure that you’re providing solutions for them as well. You want to say we’re finding these issues, but we want to go ahead and make sure that they don’t happen in the future. We’re going to put these procedures in place. We’re going to go back and make sure that those procedures are audited. Make sure that things go well. Right? It’s really nice because then you get some really great feedback from them as well and work with that as well. So, two-way conversations.
Leon: If you’re representing monitoring projects. So the people piece is, again, something that any manager technical or not can do. The leadership, obviously doesn’t have to think that you are the strongest technician on the team. However, when you’re talking about projects, there is a level of detail that a non-technical manager might find daunting. So what are some things that that manager can do to represent a monitoring initiative or project as they present them to leadership?
Paul: Well, once again, I look at it from the goals of the organization. So I want to make sure that what we’re trying to do from a monitoring point of view, it lines with the organization. So I’m trying to, let’s say, give really good uptime on our systems to make sure that people can go ahead and get to their financial information whenever they need to in a secure manner. To do that, we have to be able to monitor the equipment and monitor the systems, monitor the data centers, monitor the traffic, all of that information needs to be aggregated up.
Paul: So what do you do to get that across is you can say, look, I’ve got 20 tools that does this monitoring for us or I can consolidate those into a single tool or at least a fewer tools out there to try and get that information faster, better, better troubleshooting, better proactive work. So we can see problems before they get to become problems that affect the people that want to get to their access, whether it’s internal, external, whatever processing wise. That’s really important to have a holistic tool that can look all the way through the stacks. Let’s say you have a server that’s running on a VMware host that the host has got a VMDK file that it’s looking at. That’s sitting on a SAN. Inside the SAN it has different LUNs and those are made up of particular disks.
Paul: You find out that you have a constraint on a particular disk inside of the SAN. The SAN tool might be able to tell you that it might be something that’s minor for the SAN itself, but if you look at it all the way through, from the VMware server, all the way down to the SAN, then you see where the issue is. And that’s the deal finding a tool that has those kind of holistic views or multiple tools in a single pain of glass, that kind of deal. That’s the kind of thing. If you can communicate that without using all those words, I don’t know, VMware and SAN and LUN and Disk. That’s the trick. Right? Being able to either draw it out and say that it’s not just a server, it’s a server that has all of this stack of turtles underneath it right Holding it up. And that turtle on the bottom. That’s that’s the thing that we need to look at but the problem is how do we see that? And all the turtles on top.
Leon: I’ll also add that as a manager who is presenting monitoring, to your credit, the word holistic monitoring solution in an organization, it’s important to differentiate the difference between a management tool and a monitoring tool. Lots of individual technologies, routers, switches, servers, specific operating systems, firewalls, load balancers, et cetera, et cetera. They all have very particular management systems that are necessary. And most of those management systems also come with a monitoring component to it that nobody should ever talk about getting rid of. The difference is which tool is the single source of truth, which is, or the primary source of truth.
Leon: You may use the management tool as a secondary or even tertiary source of truth as a sanity check, but we want to make sure that there is a primary source of information that is tied in to your point, to all the other sources of information that you can even if it is at a lighter level, slightly less detailed level, but it’s able to connect to everything.
Leon: I’m not a big fan of the term single pane of… in the… glass. But I think that avoiding what’s called swivel chair integration, where you have 27 tools on four screens in front of you, and you just swing your head wildly from side to side and take it all in that you have a unified source of data that where every data point from each component can be related to every other point. I think that’s a really important thing and a very important differentiation a manager can make because you’re going to get pushback. Well, how come we need this monitoring tool if we already have the X and the Y and the Z and the all that.
Paul: The other thing that the board hears, which definitely came out a lot in the last six months or so is, well people should use our open source tools for this type of monitoring. And a lot of them were considering that let’s say SolarWinds, for example, that it only was a way to manage routers and switches or it’s only a way to manage servers in the data center, or it’s only a way to manage whatever. Most of those kind of looks at things were very narrow because they didn’t understand what the products were capable of doing when everything works together. I’m very fortunate. I’ve got to see everything working together and know how much of a help that is to the organization. And open-source tools, once again, you do a lot of head swiveling, they are point solutions. They’re not holistic. They can’t dig as deep as things go.
Leon: And the other thing I want to add to the whole open source thing is that one of the things that I hear upper level managers say is why don’t you use this open source tool it’s free. It’s only free if my time is worthless.
Paul: Well, the other free tool that I hear about is well, including an RX license from whatever company that may be, we get these free monitoring tools. And so we’re gonna use these free monitoring tools. And I have had for years had to fight that at a previous organization, that was very difficult. When we finally cracked that nut and we were able to get the licenses that we needed to use the tools that we wanted to use, things got a lot better. Once again, they didn’t really realize the constraints they were putting on us. They drank the Kool-Aid of that corporation and when they said, oh, no, you don’t need that we’ve got this covered they’re wrong. And then when I actually told them that that company actually uses a third-party product to do their managing and monitoring it was like, don’t mess with my facts.
Leon: And I think not attributing malice where uneducation adequately explains the idea, there’s probably a good faith effort to save money, to be fiscally responsible. And-
Paul: Absolutely. There’s not an unlimited supply of money in anybody’s budget that I’m aware of.
Leon: Right. And so we already have monitoring, it came for free with XYZ. On the surface, it sounds like a perfectly valid statement. But to your point, you have to have facts as a manager, you have to have facts. You have to know the question is coming and you have to be ready for it, and you have to have calm, cool, reasonable facts that point to the business case, the business justification, for using the tool that is non-free even though the other one wasn’t particularly free, either this, the cost of it was hidden inside all the other stuff. Okay. So I also want to talk about the… And this takes us to communicating the value of monitoring. We’ve talked a little bit about it already but there’s some other ways that you as the manager leading the monitoring team and the monitor initiatives can frame monitoring as necessary, important, useful, what are some of the things that you’ve discovered along the way that have helped you do that in the boardroom?
Paul: Well, I mean it’s funny the products that, let’s say, you didn’t normally monitor. Long ago, there was a situation where the monitoring solution told me that I had let’s say 12 machines that I know were next to two racks next to each other, all went down at the same time. And this was about two in the morning. So got some clothes on, grab the car, went over to the office, open up the data center. And as soon as I opened the door, it hit me with the problem was the HVAC systems were down. And the heat hit me in the face. It was about 120 degrees in that room. And I’ve found out that the reason that those systems went down is because circuit breakers work on temperature. And so the circuit breaker that was running those systems was about as close to its limit, once the heat went up in the room, that circuit breaker flipped.
Paul: And so that’s why those systems went down. Open up the doors called maintenance, everything else, maintenance, of course, it’s like, hey you don’t have to monitor temperature, we monitor temperature. And I’m like, so why weren’t you here before me to take care of this problem? Right? And so we ended up getting monitoring for the temperature sensors or temperature sensor monitoring in the data centers and had that directly fed to IT as well as maintenance.
Paul: And also we monitored humidity and we found some odd things happening there humidity wise that they were able to get corrected as well. But it’s one of those fields relying on… You want to check the checkers. Right? That also was something that we had to once again, relate up and say, we were going to need these other products. And of course management is, hey, maintenance has got that problem. But once again…
Leon: But do they really?
Paul: That was great until the power went out on these systems and the heat went up and everything else, and we were causing stress to systems. We don’t need to cause stress to. So we ended up getting that and getting it in place into multiple locations. So not just that one data center, but multiple data centers, we could keep monitored that way.
Leon: So those two stories the free software and also the circuit breakers make lousy thermometers story point to something that managers leading monitoring teams are going to have to do, which is combating FUD (fear, uncertainty, and doubt), and also mis- and disinformation that may be in the organization. And again, I’m not going to attribute to malice what may be completely explainable by just lack of education or opinions or whatever. But how as a non-technical manager, would you advise, sorry, how would you as somebody with a technical background, advise non-technical managers to go about combating these poorly formed opinions about monitoring solutions and things like that?
Paul: The best thing, once again, is to keep the lines of communication open, educate the best you possibly can, keep everyone informed with what’s going on. And also the current postures that you have. Right? So you want to make sure that it’s harder for FUD to take hold if good information is being processed prior to that. Right? It’s so hard for someone once they have an opinion about something to change their opinion. So you want to make sure that as soon as you can to try and get the information out there so they can make informed decisions. So when they see the FUD coming in, they can hopefully short circuit some of those things.
Leon: And I think it also speaks to being approachable, both externally, as well as internally. You want your team to know that you’re approachable and that they can, I’ll say, tell you anything. We’re all grown ups here. We don’t not anything, anything but that they can tell you, hey, I’m not going to hit my mark. I’m not going to hit my deadline on this. You want your staff to always feel comfortable that they can tell you that with enough time to recover but at the same time you want external teams, the network team, the server team, the leadership team, to know that they can always approach you with an idea, a thought, hey, I heard this thing. I overheard on the airplane. I read in a magazine article, whatever it is, and that they’re not going to be dismissed. They’re not going to be discredited or, sorry, give me a second, disrespected.
Leon: That they’re not going to be dismissed or disrespected for offering that, that you’re always open to having a open, honest, sincere conversation about that. And that way, these kind of weird rumors or opinions have less chance to form because you’re brought in early on.
Leon: Right. And again, if you don’t have that openness and that communication only bad things can come from that. If you can’t have that kind of openness and communication, it’s an opportunity to ask yourself why. And again, none of these things are technical issues. Any manager of any team anywhere has the skills to be able to do this. So, the last part that I want to address is advice to new managers. So we’ve been talking about people who may have been managers for years, and simply moved into leading a monitoring team. But now we’re talking about people who are new to the managing game again, non-technical and I want to know, what do you wish you’d known when you transition from being an individual contributor? What are the things that would have given you a leg up to the process?
Paul: Yeah. Once again, you just would definitely want to hire to the job, hire to the culture. You want to make sure that they’re going to fit in very well with the team. You want to find out what they’re most comfortable and what they’re happiest doing, and you want to make sure that they’re fed as much of that information as they can to do those things that really brings satisfaction to them. It’s really interesting. You can see people it’s like, I don’t like doing this, but I don’t have to do it very often. Right? That’s good. Let’s get the bad stuff out of the way, the stuff that they don’t enjoy as much. But the most thing is we want to try and educate them the best we can. We want to make sure that they’re well taken care of. Those are the things that it’s really helping me as a manager and in helping the organization as well.
Paul: That’s the main thing. I’ve seen some of the previous let’s call it almost a stress that they’ve had from managers that say didn’t get along. And because of that teams didn’t get along. And one of the hardest part as a new manager at this organization, was to try and get those teams, working with our teams and let them know that it’s going to be okay. No, one’s going to yell at you. If someone yells at you, you need to just, please, please let me know because I want them to yell at me.
Paul: If I’m telling you to do something and supposedly that got you in trouble, that’s not your problem. That’s my problem. They need to come talk to me about it and stuff and we’ll work it out. We’ll get this taken care of whether… I’m willing to be wrong, that’s one of the things I like telling people. And it’s true. If I’m willing to have made a mistake on something and at least tried something and to move things forward in one way or another. And I have met too many people that are not willing to be wrong. Right? My code’s good. Well, it’s not my code. Well, if you walk onto a phone call and you’re saying that that’s probably not a great attitude, because then you’re never going to be part of the solution to find the problem.
Leon: Right. I also like the point that you made about asking the team to identify what for them is broccoli and what for them is cake, because it’s not going to be the same for every person, and it’s not going to be the same for every team, but being able to identify those things and for you as a manager to understand that and be able to… No one can live without having a little bit of broccoli. Okay, you got to eat your broccoli. But if you, as a manager, understand that this is not the delightful, pleasing opportunity that you see it to be or another employee, another staffer sees it to be that for this person, it’s really hard and it’s really tedious and they don’t like it. Then you’re not liable to accidentally keep assigning that task to that person that you can feed people the fun, the cake that they want so that they find fulfillment.
Leon: I liked that idea of identifying those things. I want to get your take on something that’s pretty consistent in the IT world. And that’s the brilliant jerk, the jerk genius, the person who, who does write perfect glowing code, who does have lots and lots of solid answers who can fix almost anything, but is honest to gosh, a pain in the rear end to work with. Nobody likes talking to them. Going back to the Phoenix Project, it was never completely framed this way, but I think it was Brent. Brent was always the constraint. Right? He was the character who he coded everything and he did everything. And if he wasn’t available, things didn’t happen. I think it was Brent. I don’t like Brent.
Paul: And I don’t know how well it would be to have someone like that on your staff. And unfortunately, there are many people like that on different staffs out there. I’ve found that they are a pull on the productivity and creativity of the rest of the team.
Paul: And sadly enough, sometimes you just have to let them go. Sometimes you can do your best to reeducate them show them their weaknesses and let them hopefully grow past the problems that they’re having. There’s a manager here in this area who has gone through extensive types of training like that, and they are much better for it. But when I originally met them that was a really interesting working with them. I mean, at one time I was taking apart out of a plastic bag that had a Ziploc type seal on it, and it was a part I was putting on a server that it was never coming off. And basically they jumped on me because I tore the bag open and what if they need to return that? And my thought was, this is like $90,000 that you’d bought for this thing for, it’s not like cans of soup at the grocery store here you should hopefully you’ve spec’d this, what you really wanted. And that part goes there.
Leon: Right. It goes there. If it needs to go back, they’ll send us another bag.
Paul: I promised them at the time that I got their back and if at that part needs to get returned, I will make sure they accept it without the plastic bag.
Leon: It’s an interesting take, but what I’ve heard over the course of time is that usually if you have these jerk geniuses if they’re not, I will say easily trainable. If they can’t be shown the impact that their behavior is having on everyone else around them, then it’s often not worth the minimal benefit that they bring to the organization. So it’s worth again, to those new managers who are thinking, oh, but I have this linchpin person that I have to… There’s a difference between being curmudgeonly. In fact, I will go so far as to say that it’s good for teams to have people who are contradictory, who if somebody says we should do X someone on the team should say, but why. Not letter Y, but why are we going to do it that way? Why not this other way? Are you sure?
Leon: Not because you want someone who’s constantly challenging the idea just for the sake of challenging it, but really if your team can’t justify a decision to your team, what hope do you have of justifying a decision to the rest of the organization? Because they’re going to ask, they’re going to challenge you on this. So better to go through the exercise of defending a decision internally and playing that game and being able to completely explain everything before the rest of the organization has their shot at it. There’s a difference between that and being a jackwagon.
Paul: Well, and the deal is, it’s almost like gravity. Right? It’s this pull that’s pulling the organization, let’s say in the wrong direction. And so you really need to find people that are gravity in the opposite direction that they’re pulling it towards the goal. They’re pulling it towards the mission of the organization. And that’s the deal and sometimes if you can’t have them move forward, you just have to let it go. And that happens even back when I was doing construction at the same thing, doesn’t matter if it’s IT or construction or a subcontractor, it’s all the same thing.
Leon:
Okay. Let’s move on to the lightning round, which is just any final thoughts. Anything you want to add, any last words that you want to leave the audience to remember you by?
Paul: Well, thank you. Support CyberPatriot. Cyberpatriot.us, I believe is there a URL and CyberPatriot is a nationwide competition of middle school and high school students that learn about cybersecurity. In the process they also learn about doing some of the basic things needed that we need in our organizations. Now I’ve had people tell you, well, they’re not all going to be cybersecurity professionals. And I go, absolutely not. Some of them are going to be doctors and lawyers, and some of them are going to be whatever position they’re going to be in, but at least they’ll have a cybersecurity mindset going forward. Right?
Leon: Look, we enroll our kids in little league baseball and they play football and soccer and stuff like that. And yet we know they’re not going to end up being the next Pele or whatever. It’s okay to just enjoy a activity and learn what you can from it and bring it to the rest of your life.
Paul: There’s also scholarships out there. The city of San Antonio has a huge event every year where they are thrilled with the latest teams and present them awards for the teams that are there. It’s a big deal here in San Antonio. We have probably more teams than just about anywhere else in the country going into the competition. We’re very fortunate. We had third place team win. This past year did just wonderful and scholarships come out of this thing. I’m not talking about little scholarships, I’m talking Ivy League scholarships and other organizations and universities that are there handing out full rides for places that can do if you’re one of the best in the areas.
Paul: We had a girl that was placed number one in the State of Texas for the Linux image. Doing all the parts and pieces of it and getting it done in the shortest amount of time possible. And she literally had full rides to four or five different organizations or universities out there. And that’s some of the things that people just don’t realize that it’s out there, they need mentors. Why? Because we have skills that can help these children get better at doing what they’re doing and what are they doing? They’re doing the standard stuff that we do every day as defenders of organizations and builders of organizations and stuff.
Paul: So get out there and do that. Also support your local BSides, support your local organizations that you have that you, that you’re in. It’s not all about RSA and these big things. Find local people.
Leon: DEF-CON! Yeah.
Paul: Yeah, exactly. Find your local version of these things and work with the people the best you can. So that that’s the thing I’d really like to see more mentoring. No matter of fact, if you have people working for you, please have them go mentor these kids on a regular basis, just take an hour or two hours every two weeks, a couple of people, they go out to the schools and you will be building the workers that will take care of your organization in 10 years.
Leon: Not only that but the people who are doing the mentoring get an incredible… There is an incredible return on that investment of time that is almost impossible to predict.
Paul: Yep. I mean, you look at the sports people at these schools, very few make it to national sports, but a lot of these kids are going to be in the organizations locally taking care of your stuff.
Leon: Okay. One more chance to just hawk your presence on the interwebs, where can people find you, if they want to hear more about what you have to say?
Paul: @Radioteacher. I haven’t been real vocal lately, but I will be soon. Also there’s some things out there on BSides San Antonio, BSides SATX, I have some videos up there of different things that I’ve done with the organization. BSides by the way, San Antonio, unfortunately, it’s going to be tomorrow. And so this won’t come out until afterwards, but the videos will be up soon.
Leon: Fantastic. Okay. Paul, thank you for taking all this time and sharing your life and your experiences with the TechPod audience.
Paul: I’m thrilled to do it and thank you very much for giving me the invitation.
Leon: And once again, I want to thank everybody for taking time out of their busy day to listen to TechPod. Have a great day.
