Thomas: Hello good morning, good evening and welcome to this SolarWinds Secure by Design webcast. Tonight’s topic is Securing the Software Development Build Environment. My name is Thomas LaRock. I am a Head Geek here at SolarWinds and before we get started, I’m gonna go over a little bit of housekeeping for everyone. So first to have you understand that today where I am it is March 16th. It might be March 17 where you are, but we are recording this session live. There’s an attendee poll that is also live to the lower right. If you could take a moment to simply answer that quick question let us know who you are, it will help us. And the on-demand recording and slides will be made available by Thursday, March 18th. Everyone who has registered for this webinar will get an email with a link so you can download the slides and watch the recording. Lastly, there will be some time for a Q&A at the end. We have a handful of questions have already been submitted. If you have a question, please use the Q&A section to the right as well. You can enter your question, if we have time to answer it tonight, we will do so if not, we do promise to follow up with you and get you the answer. All right, the agenda for today. First, we’ll go through some introductions, so I’ll introduce my guests here in a moment. I will speak to the Secure by Design Resource Center. A quick reminder for everybody about the content that we have there. Then we’re gonna get into the meat of the presentation. We’ll talk about the ever-changing landscape, principles to securing the enterprise, characteristics of a secure build system. Then, as promised, the Q&A and we will wrap up with the thank you. So without further ado, the introductions. First I’m gonna introduce Lee McClendon. He is the SolarWinds senior vice president for research and development. Lee, why don’t you talk about yourself for a minute?
Lee: Sure, thanks Thomas for the introduction and thanks for hosting. I also I wanna thank everyone in the audience for attending today. This is the second or third in our series of Secure by Design webcast. It will be an ongoing thing that we do throughout the year to provide more updates on what we’re doing to make our customers and the software community at large more secure. As Thomas said, I’m the SVP of Research and Development. So I lead the engineering team and the UX design and research team as well. I started my career as a software engineer which is a good background for actually understanding the technical challenges with creating secure build environments. I’ve also spent time in product management and general management roles which gives me a lot of empathy for the problems that our customers face every day and their need for secure software and vendors that they trust. I hope today’s webcast is very informative for each of you and that you take away some key learnings and some insights into the challenges and solutions that are involved in building software securely. We’ve been working closely with a few partners at SolarWinds on our forensic investigation from the attack that we encountered back in December. One of those is KPMG and today we have Caleb Queern here to represent KPMG in this webcast with me. And so I’m gonna let Caleb tell me a little bit about himself and about his background as well.
Caleb: Thanks so much Lee, thanks Tom. My name is Caleb Queern. I’m a director in KPMG cyberpractice. I’m out of Austin, Texas and my role in KPMG is to help large organizations as they attempt to quickly build software with the least risk, soonest, so that they can grow their business and ultimately I do have a prosperity and growth and all those good things that they’re really looking for. But my focus is security DevOps and application security within KPMG cyber.
Thomas: Okay, thank you both for the introductions and thank you again for spending the time with us today. So the Secure by Design Resource Center. This is a dedicated landing page that you can find over at SolarWinds.com. Just go through SolarWinds.com, Secure by Design Resources. Again, you’ll get the slide deck, a link to email to you. So you can just click on the link when the slides do get in your possession. This resource center contains every piece of information or content that we have curated since the incident I’ll say. So what you’re gonna find there are the previous webcasts that we’ve done. You’re gonna find blog posts. You’re gonna find resources such as links probably to the security bulletins. Everything that you need for your path forward as a customer you will be able to find in the Secure by Design Resource Center. If you have not been there to check it out, you should go there and have a look. All right, let’s talk a little bit about what we say here is the ever-changing landscape. I’m gonna ask Lee, maybe set the context here about where we are, the current situation, how COVID has kind of affected things a little bit say even in the software world and a little bit about the common industry practices that we have and what we need to do going forward.
Lee: Sure, so one of the things that’s really clear is that sophistication is a way to describe a lot of what’s happening, whether that’s the increasing sophistication of the threat surface, COVID-19 as Thomas mentioned is a big part of that where a year ago all of a sudden IT teams had to start worrying about most of their end users being outside of the firewall and securing their mobile devices and making sure that they could communicate securely and secure their their internal IP as well. And so that is something that not just because of the hybrid IT but things like COVID can really amplify that and make that threat surface increase and become more sophisticated. At the same time adversaries are becoming more sophisticated in our first Secure by Design webcast, Alex Stamos who is a consultant for us right now talked about the evolution of people who had to worry about nation state adversaries. Back in the early days, it was banks and government entities that were targeted around defense. Those people had to really worry about this. And over time this got to the point where today as Alex said, every Fortune 500 company or vendor to those companies is a target for this level of attack. And from one of the things that really stood out to me as Alex description of telling that adversaries, coming to work every day, getting their coffee, and spending their whole day trying to figure out how to break into your environment. One of the things that we’re seeing with this increased sophistication is that these attacks are focusing on the supply chain both actual attacks and theoretical ways of breaking the supply chain. These are things that are increasing right now as our weak point and the typical software development life cycle. There was a one recently talked about the dependency confusion package problem that came out a few weeks ago and is another version of the supply chain attack as well. All of the sophistication in the threat surface, the adversaries, and just the way we build software means that we have to change the way we build software for the benefit of our customers and for the security of our customers.
Thomas: Okay, is there anything you wanna add to this slide here or the ever-changing landscape?
Caleb: Yeah, thanks Tom. I think a metaphor that comes to me when I think about this is that we’re learning new things regularly about how attackers do what they do. And that informs how we need to defend ourselves at larger organizations. And the rate at which new things are on our doorstep as it worked to kind of defend against, suggest that we should probably increase the frequency at which we plan our investments in security. And here’s what I mean by that. You know, in a lot of large organizations there’s an annual budgeting cycle where they say, our priorities for this year are A through D but a lot can happen in a year and sometimes you need to be a little bit more agile about it. I know that’s a loaded term when we talk about software development but it’s actually appropriate way to think about this. So we do more valuable things in smaller chunks more frequently. And we should think about that when we plan how we defend ourselves because things will come up and we need to be agile in how we respond and kind of react so that we’ve balanced as possible when it comes to risk reduction and then just driving business outcomes at the same time, because a year can be a long time. So that would be one thing to consider to compliment Lee’s comments about the current situation.
Thomas: Absolutely, I think that the idea of everything that’s happened in the past year the increase in the say but not just the threats but the attack threats all that, but what we have at the bottom there this right now for all of us especially here at SolarWinds I know Sudhakar has talked about this before, but we see this as our opportunity to learn, improve, iterate, and share for the safety and security of all environments. We’re talking about something for the industry not just for SolarWinds itself, but for everybody because we all know that security is a shared responsibility. So let’s segue a little bit to the principles of a secure enterprise. So Lee, since you had mentioned what Alex had presented last month, by the way and if you haven’t seen that webinar again go to the Secure by Design Resource Center and you can watch that very first episode we had with Alex Stamos but maybe you could review some of the key points that Alex had mentioned. And talk to us a little bit about how to secure the enterprise.
Lee: Sure, and I think some of these key principles on securing the enterprise are important as we talk about the software development of built systems as well because ultimately that’s where these attackers have to get a foothold in your environment to attack your build system and so some of the key things that Alex talks about were making sure that you complete your transition to cloud identity, trying to avoid that half cloud, half on-prem scenario that a lot of people find themselves in. Then also auditing those connection points between your on-prem and cloud service providers, so that you understand what security permissions and privileges are being given, the cause of this connections between the two, and then really aggressively using modern authentication. So MFA, just-in-time permission for your admins, that sort of thing. So those are important even in the software development life cycle because it’s protecting your enterprise infrastructure in your authentication mechanisms, which is something that software environments depend on as well. Also there was some talk about the supply chain, and again these are new things, but the thing that’s really starting to come to the forefront is that these things are becoming more sophisticated even though the subtleness of the attack on SolarWinds was something that really stood out. People spending a lot of time being very careful to reach an end goal through the supply chain. I know one of the quotes that I really remember from that first one is, Alex saying that, in his experience working with hundreds of companies on security issues, he didn’t think that he had seen a build process that was perfectly designed against this level of adversary. And so there’s some things that we need to do to stand up to those adversaries and prevent them from impacting our software throughout the industry. And we’re gonna cover some of those principles today. During that same talk, we also we’ve talked here about people, the importance of making sure that we’re really paying attention to people’s side of the security. That’s still where a lot of the vulnerabilities and initial attacks happen right now. Then also the infrastructure is important, but for today we’re gonna focus on the software development side of things specifically the build system. There are a lot of parts of the software development life cycle that we will cover in future talks but for today we’re gonna talk about characteristics of build systems and then later we’ll have some follow ups to go even deeper into these topics that we’re talking about today. One of the things I think that stands out to me is back in the day when I first started writing code, when we talked about build systems, it wasn’t much more than a bunch of make files and some scripts to actually make them generate the software that we were running and today we’ve evolved a lot into continuous integration systems, package management, configuration management, and at the same time there’s been a lot of advances in security the trustworthy competing initiative from 20 years ago that really kick started a movement to really make sure that our software was more secure. Patching often changed the way developers thought about the code that they wrote and made it secure. So we’ve had this significant change in the way software is built, in a way software is secured but there’s still a lot to do in the area of the secure build system and some of those principles and characteristics are what we will talk about today. It’s also important to remember that this does not just apply to companies like SolarWinds that are selling software as the main part of our business. If you have an internal team that’s developing your own internal software, these same rules and principles that we’re using will apply as well. And as we go through these topics today, keep in mind that we mentioned earlier in the talk that we wanted to better everyone, the whole software community and ecosystem. And so this will be one of a series that we’ll be sharing the actual implementation of what we’re doing so that the community can get the benefit of that and we can make supply chain security problems a thing of the past.
Thomas: Caleb, I’m interested to hear you share your experience with KPMG and what you’re seeing in the field and we have principles of security enterprise. So how many of these principles are actually in practice?
Caleb: That’s a loaded question because the answer of what it looks like depends on who you ask. And I think a lot of organizations, a lot of large organizations haven’t yet even fully formed their ideas on what that means, because I joke sometimes that if you had 10 people in a room and you asked just what DevOps is, you get 15 answers. Much less like how do you secure that? So one of the ways we think about it at KPMG is, if you were to imagine the SDLC being mapped left to right maybe on a whiteboard, and then at each step of the SDLC, what are the things that the capabilities or behaviors or controls that you can put in place to reduce risk across that SDLC all the way from planning to monitoring and production. That’s a pretty big list. There’s dozens there’s 30, 40 things you could put on those. And a lot of large organizations still have yet to probably put a very formal approach or framework around which investments they make where and why? Because I think there’s some things that have been in place ’cause they’ve always been in place that’s the way we’ve always done it. And people are starting… it’s kind of trite these days to talk about shifting left because it’s so overused, but I mean people are shifting left so that we are focusing more on prevention rather than detection to response to problems like that’s great. We want to build things well by design, you know secure by design as SolarWinds puts it. But long story short, I think one of the things that our audience should take away today as they listen is historically we focused on making great code and how people access the code but we’ve never really put a ton of thought as an industry into the infrastructure itself that we use to get that code into a package or in production and that build pipeline is where we’ll focus today.
Thomas: Yeah, it’s funny how you mentioned that the infrastructure, right? So Lee, you’ve been in the middle of the SolarWinds forensic investigation and the efforts to develop this next generation secure build system. So let’s have you walk us through some of the key questions that your team has asked or was asking when you were starting down this path?
Lee: So certainly the first thing is making sure that we have a secure design, that’s the starting point for having a secure system but one of the questions that quickly comes up after that is how can we ensure that this system is working as designed both so we can prove it to ourselves that’s something that we obviously care about, third parties that we’re asking to take a look at what we’re doing or to audit us. We want to be able to prove it to them and ultimately to prove it to customers who wanna know that how the software is being built, that they’re consuming that they’re bringing into their environment. And so one of the things we realized is that we need to have metadata stored in a very secure fashion for every step in the build pipeline. And that means when we do a code check in where recording who actually made that change, who’d made the commit. We’re actually storing information about the files including hashes of the files that were checked into the source code repository. If we’re doing code scans for static analysis or open source vulnerabilities, making sure that we record the logs of that, what was scanned, what’s the output, what security principle did that scan run under at the time for builds running, giving the build output log and what context did that run under in, what are the hashes from the compiler of the actual files that were compiled and that allows us to look at the actual check-in of the codes as well. So it’s really important that we have this metadata. It will allow us to do some monitoring but also be able to show what was actually happening in that built environment. And Caleb when you and I have talked about this, these things I’ve talked about belonging to a category of auditing characteristics that you refer to as automated governance. And I know from your background at KPMG, could you give us a little bit more detail about what exactly is automated governance? Why is it a part of the secure build process and what are some areas we can improve in this area?
Caleb: For sure, for sure. Thanks, Lee. So the name itself automated governance, you know governance is a very loaded term. It’s just like maybe DevOps it means a million things to a lot of people. So here’s what we mean here when we talk about automated governance. So it’s a name given to, I’ll call it a framework that was put out in a whitepaper I’ll say 18, 24 months ago, by some thought leaders and very large organizations who build software a lot of software that helps power many of the things that we take for granted in our daily lives across industries. If you do a search for automated governance whitepaper, it’ll probably turn up. But here’s the idea, like why did they set out and start putting pen to paper? And the problem statement that they had in mind was internal audit often has traditionally and stereotypically can have a bit of heartburn when it comes to DevOps and modern ways of building software. We know that one of the traits of modern building and modern delivery of software is a higher frequency of deployment to production, for example or at least compilation and CXD all those great things happen faster than they used to. And the internal audits legacy approach to looking at this would be to stop by once a year, do the best to make sense of some of the config and controls and maybe risk reduction capabilities that make people feel good that the right things are happening as the software is being built. And that really doesn’t really, you know has a hard time cutting it these days with how fast we build things. I mean they still are looking for things like segregation of duties and developer access to prod, but things move so fast. So what’s some platform by which folks like that, with that use case and that need to feel good about the process what can we put together with that? We can all look at and agree that there’s a record of what happened. And so on the screen, we have kind of an example, it’s not comprehensive, it’s representative of some of the different steps along the way and then in the separate development life cycle and some of the outputs or metadata or signals or artifacts all of those great things that emanate or radiate from each of these steps, you know maybe they were collected somewhere before but if they were it certainly wasn’t in the same place, maybe they’re in their own little silos where if you went and dug around you could find it but it wasn’t readily available for anybody who needed to see it on-demand, much less on a kind of an annual cycle and that internal audit use case. So this automated governance approach comes from that whitepaper meant for internal audit 18, 24 months ago, but now I think is a great way for our security use case to say, you know what, what did happen as the software passed through the platform here? What makes us feel good that the right things happen and there’s nothing that should raise an eyebrow. That’s the long and short of it, is putting it on the same place in collecting it all in a secure way that we can also feel good that hasn’t been tampered with or meddled because as Lee pointed out everything that gets in there, there’s gonna be hash along the way and we can build hashes of hashes of hashes and see that if any of those don’t add up well, you know it suggests there is some meddling. So it’s very secure way to just look altogether at the same thing about how the code went through this process.
Lee: So it seems like really just bringing a level of rigor around things that are standard practice in many other industries but just having a survey come into the software development life cycle today.
Caleb: Yeah, I mean, you’ve said it a couple of times already this event that happened to SolarWinds is a great opportunity to make this the default right for the community whereas before, it never really rose to the surface but now we have an opportunity to take this forward.
Thomas: So on that note Caleb, let’s talk a little bit about say pipeline security monitoring, right. Now we’ve established that being able to audit the build process is fairly important. And we want to ensure that each step in that build process is being performed as designed and by the appropriate security principle. And it’s necessary to have that detailed telemetry and logging of the build process but as with any security system, right, that audit trail is only useful if a trained person or systems actually looking at the data. As a former DBA, as a person that works with data all the time and knowing it’s readily available, it’s of no use if nobody’s ever looking at it. So, talk about some of the key components that are for that build pipeline that should be monitored and where are the best ways of performing that monitoring?
Caleb: Sure, yeah good question Tom. So one of the ways we think about it is to your questions Tom, like what are the questions you might ask at each step along the way that if you knew the answer to would make you feel better that the right things are happening and there was no funny business by anybody along the way. So imagine a spreadsheet you could put together where each row is a question, and then you can start mapping those questions to each piece of the SDLC and then you can say, what is the level of effort to answer this question by gathering all of the logs or metadata artifacts… every time we do that, that’s a lot of work and some of those questions take more work than others. And then you may also prioritize these questions by how much actual risk reduction value that information would give you. So let me give you a couple of examples. In security operations like legacy security operations on the enterprise, I think we’re all pretty familiar with the use case of VPN connection like a virtual private network connection coming into the enterprise and you wanna know that for example, this user always comes in to the VPN from the same place and suddenly they’re connecting to the VPN from a place they’ve never been in their lives. It’s 5,000 miles away and yesterday they were not there like that use case is pretty straight forward because all of the logs to answer those questions are probably in the same place but let’s imagine I’m coming back to the SDLC and I wanna know whether the developer who checked in this code to Github or Bitbucket wherever on this date was using multifactor authentication and had all of the other kinds of identity things in place. Suddenly we’re talking about collecting logs and metadata from a bunch of different systems. So the level of effort for that security monitoring about the bill process and the code goes up. So in any case, the point is you can create a long list of these questions that would make you feel better for real time monitoring. One of the other things that I think our audience should consider when they think about build pipeline security monitoring is legacy security monitoring and alerting would go to a security operation center. I think in this context, the alerts should go both to the dev teams and the engineering teams as well as your security folks, right? As opposed to just the people who may be receiving alerts from their SIM or some other platform like that, wouldn’t it be great when these bells go off or the alert fires that, a message pops into your real-time chat platform where all of the devs hang out together, that’s the type of thing where not only, they can maybe go take a look and see if anything really went wrong or suspicious, but also we always like to educate developers a little bit more about security and building this into their daily, maybe not daily, but the regular routine would help communicate that and can make it more of the forefront for developers.
Thomas: Yeah, so this is all great information and the next topic here is something near and dear to my heart ’cause it’s reproducibility. And again IT pro here if somebody shows me an error, and one of the first things I wanna do is reproduce that error. That’s the only way I’m gonna out what’s really happening, but when it comes through the build pipeline that might mean something slightly different. So, talk about a reproducible build and how it helps make things more secure.
Lee: Sure, so when we talk about reproducible builds, we’re not talking about the functionality that an end user sees. You know, if I’ve build on one system and build the exact same software later, it generally will have the exact same functionality but if it’s not set up in a certain way, I will not get a bite-by-bite match for the binaries that are shipped to those customers. And so that’s really what we were talking about, we say reproducible builds it means I can build a software now, I can build it on another system, I can build it later and the two executable or packages that are produced will be bite-by-bite equivalent. That’s does kind of tough to do if you think about file versioning if you have your system set up just to increment the version each time or the way you pass files to an insert compiler or linker, timestamps. So we have answers there was a lot of different things that can affect that. And while there won’t be noticeable to an end user, they’re really important in the context of security. And the reason for that is that we can actually have something like what we’re showing here on the screen which is essentially three different build pipelines building from the same source code but the advantage of that is we can go have these in different cloud service providers or data centers or wherever we wanna put these different build pipelines. We can have them running under a different authentic indication mechanisms. So they’re all separate in that regard, they could all have a hardware-based multifactor authentication and because of that, we can actually build and then do the comparison at the end of the output to see if they match. If they don’t match, we know that something happened between that source code and when we actually got the output. And so it’s good way to detect things that may be manipulating things after source code in the build environment and your build agents that sort of thing and it gives you an extra layer of detection there. So Caleb, now we’ve talked about what that actually what a repeatable build does what are the challenges that you see development teams encounter when they’re trying to get to repeatable builds? And do you have any other thoughts about the concept of the security techniques around having reproducible builds?
Caleb: Yep, I think it’s early days. I mean the concept is one of those things that sounds relatively straight forward the more you think about it and the more you dig in and you say well, yeah, that’s probably pretty tricky just like you said to me there’s some things that would make it tough to pull off but it’s I think a relatively known problem space however, there’s not like a package out there or some sort of tool or system that anybody could use to pay for or download and really make this easy today at scale. I’m not aware of very many large organizations, Fortune 100, Fortune 500 that really do this I know there are some right, the real kind of the people that really come to mind top of mind large IT and technology companies that we think of. They may be dabbling here, but that’s a very small sliver of everybody who’s building software across the planet these days and they still have yet to I think maybe make this production ready across their portfolio. This isn’t something that’s very common, is one point. However, the value is very high like I think the likelihood that an attacker, what happened at SolarWinds was a lot of work. It was a ton of work, a ton of patience. To do that twice in two separate builds systems is extremely hard, much less once. So if you can perform reproducible builds, the probability that an attacker pulls off that same very difficult compromise in both places is extremely low and impossible, absolutely possible, but it’s worth probably considering reproducible builds in all of the work that you have to do as a security decision maker and it can’t be off the table anymore. This needs to be prioritized in your backlog appropriately because now we know it’s a thing, and this is one of the strongest controls to prevent this type of attack.
Lee: Yeah, and of all the different things that we’re talking about here today. Like anything else in security it’s important to have layers and build multiple security things together to make it harder and that was one of the concepts Alex covered, in the first one of our Secure by Design webcasts, is the importance of making sure that you’re making it hard at each step an adversary would have to take to compromise your system. And so all of these things are just pieces that fit into that larger secure build system that we’re talking about.
Thomas: Okay, so now we’ve talked about the auditing of the build and we’ve talked about the reproducibility, let’s talk about pipeline consolidation. And what I mean here is looking at a company like SolarWinds or any growing software company, you end up at some point with a portfolio of products, right? And usually these products are obtained through MNA, right? Mergers, acquisitions and so that means you might have some products and say the newer ones are using newer technologies and approaches. Now this may lead to a scenario where that software company has many products with many different ways of building those products throughout their portfolio. So what kind of problems does this fragmented system cause, and how is this much of a problem when you’re trying to create that build system? It’s a question for you Caleb, speak a little bit about the importance of pipeline consolidation.
Caleb: Yeah, thanks Tom. This is one of my favorites. To some degree in other technologists apart and we love the idea or actually tinkering and making changes and building things that solves problems. This one is more about how we plan and strategize the business. So here’s what we mean by that to your point a moment ago, Tom, over time without a lot of governance, we’re all gonna find ourselves with entropy and a lot more kind of things in the environment and we started off with whether it’s by acquisition or just lack of kind of rules of the road internally that would kind of keep a straight path. And here’s what I mean by that, there are organizations out there who have been building software for 20, 30, more years. And in some of those organizations they have whether it’s architectural guidelines or some other sort of governance that says, we have a portfolio of applications and we have decided that we are going to focus our development efforts around one framework or one language and we’re gonna say 75% of our app portfolio on this, and then we’re going to experiment, we can’t not be ready for the next big thing that’s coming ’cause change happens so we’ll devote another 15% of our portfolio of apps to allowing other things for experimentation so we can be prepared for the next thing. And then the remainder of our portfolio can be on that thing we’re all coming off of together. But whatever that mix looks like, there’s a thoughtful intentional of approach to minimizing complexity that the business has to manage. And so what we’re talking about here is just doing the exact same thing with the build pipelines because some organizations, I mean I’m aware of organizations who were intentional about this even in auto manufacturing they’re consciously trying to reduce the number of build pipelines. So there’s less variety and it’s easier to implement security capabilities among other things, you have fewer places that you need to implement say, you know, certain type of security scanning, right? You don’t need to go to 30 or 40 different teams and ask them to do it, you have four or five maybe pipelines that you need to go to. So I mean, you can imagine the benefits and agility that brings you because those new capabilities will come and you don’t need to recreate the wheel across a huge portfolio. That’s really all we’re talking about here. Does that make sense Tom, Lee?
Thomas: Absolutely yes.
Lee: Yeah, I think like a lot of things in security, simpler systems are easier to understand. The fewer systems you have, the more you can invest in making the few that you do have really secure. So that’s a very good principle to keep in mind.
Thomas: Yes, keep it simple. Isn’t that? That’s everything in life, right? Just keep it simple. So, let’s begin to wrap up here. I’m gonna ask Lee, Caleb if you have any last thoughts any last words of wisdom, maybe Lee you could speak to some of the characteristics of a secure build here that we have on the slide.
Lee: Yeah, so certainly one of the things we’re really focused on over the last few months is this building a great software build process and build pipeline is extremely secure. Leveraging the research that’s out there today. There are a lot of interesting research, a lot of the ideas we’ve talked about today some of those have come from different research articles or reports that we’ve read and so really focusing and leveraging that to produce something that’s going to be great for SolarWinds and great for the industry. Also leveraging some of the open source work that is out there. There’s some really interesting projects that I expect us to use and that we’ll contribute to. So I think that’s another important part of this as well. And then, again, just continuing to share our learnings. A lot of what we’ve talked about here, you’ve heard Caleb talk about the companies that he sees at KPMG and the number of them doing some of these things well, and the others who haven’t invested in that. What we wanna do by sharing is lessen that investment for other companies ’cause the goal again is to make SolarWinds customers safe but also to make the software industry safer. The supply chain attacks on the build system are things that need to be improved and will be improved and we intend to be at the forefront of that, sharing, learning from others, contributing back all for the good of customers who are using the software that we and other companies sell. And so again I think that leading by example, hopefully will exert a little bit of positive peer pressure on other companies contributing to them hopefully it makes that burden to implement some of these things less for them and also makes it so that customers are demanding this and having this kind of expectation from vendors that they will have secure pipelines that we can talk about and share building materials and other things with customers to show exactly how this is working and how we’re taking care of them and making software that is secure by design
Thomas: And Caleb any last thoughts from you?
Caleb: Yeah, Lee did a great job wrapping up. I’m excited about the point that we’ve used a few times in our conversation today, of turning this into an opportunity to increase the dialogue and the awareness around this specific part of the challenge of building software well and fast and securely these days so that we can grow the business. And when we do that we provide opportunity and prosperity for others so I don’t know, it’s an exciting time as always in securing DevOps.
Thomas: All right, so let’s transition over to the Q&A portion real quick. I have a handful of questions here and I’m gonna start with you Lee and I’m gonna give you an easy one to start with this. This viewer wants us to speak a little bit about the changes we’ve implemented to strengthen our security controls. Is that something you wanna touch upon?
Lee: Yeah, I’ll talk generally about some of that. So certainly like with any security incident, you go back and start investigating everything, how can you reduce that attack surface? And so there are things that we’ve done to shrink the attack surface certainly basics of incident response or rolling credentials and other things like that. Looking at all of the permissions not just within our software environment but throughout the company to find out, does this person need these permissions? Do they need them for a certain amount of time? Is there another way to accomplish this? So I think a lot of those basic incident response in addition to the things that we’ve talked about here for securing our build pipeline are important things that we’ve already done. And we’ve also we’ve got some stuff on the Secure by Design website on things that we’ve done already to secure our software that we’re shipping to customers, but I think just generally there’s a huge effort and investment on security right now for basic incident response stuff that you would expect also for the build pipeline and making sure that our software is safe.
Thomas: Yep, thank you for that answer Lee. Next one for you Caleb, another easy one for you. What are the minimum requirements around source code reviews?
Caleb: Yeah, so I can interpret that a couple of ways. So I’m gonna pick one and just go with it. When we apply scrutiny to source code what does that look like these days? Is how I’ll take the question. I think most organizations there’s even organizations like when they say, what is your application security program? And they’ll say, Oh, you know we’re good to go, we have apps side covered. We are doing static application security testing and that usually can be as a warning flag when they equate risk reduction across the SDLC with just that single point in time snapshot of how the code looked as it flew by, or maybe even wasn’t even in the automated build process, it was like a manual effort they had to go catch the dev team and grab the package and then run it through the scanner. That’s still unfortunately all too common. So what does good look like these days as we apply scrutiny to source code? I would say at a minimum stack application security testing. If you can automate dynamic application security testing in the build process, that’s great too. Clearly software composition analysis which is a fancy way of saying, how do you feel good about the open source library and components that are being incorporated? I think we would all acknowledge that the percentage of software that gets built these days is increasingly open source to some degree and we stitch it all together. You know, that differs depending on what we’re talking about but it’s not a smaller mass that relies on open source. So software composition analysis that is table stakes these days and the application security tooling community has been actually very fast in baking this in to legacy products and there’s a bunch of new contenders that can do this very well and a developer friendly way. So I would encourage our audience to consider those among others.
Thomas: So Caleb, I’m gonna have a follow-up question for you since you mentioned the open source. So if a product has lets say incorporates some sort of open source project inside of it, how would you safeguard that type of project once it’s bundled say into a product?
Caleb: Sure, so the SCA or software composition analysis tooling that I hinted at a moment ago is I think step zero and some of those are open source. I think WASP the open web application security project has its own maybe it’s pulling these days. So, start there with an SCA tool. The thing I didn’t mention and meant too, okay, I think the opportunity to do now is the software building of materials, right? Because that offers some of the things that Lee was talking about earlier that your transparency into what you’re getting, not only as the developer who’s taking that open source library or component from who knows where it came from over the years of being built, but also the end user because a software creator or software company can provide the S-bomber software building materials. They can hold it up and say here’s everything that’s in the product you’re about to get. And they can also do their own due diligence and say, looks good I understand the risk that comes along with some of these packages or not because there’s resources online that will say, this version of that has these three or four critical high, medium, and low vulnerabilities or whatever the readout is, but I would strongly encourage the adoption of the software building materials that is also an underappreciated capability but it’s a growing muscle out there that I’m excited about in the community.
Thomas: That’s great advice I had not heard of that. Yeah, that software building of materials. It seems so common sense once you said it but yeah, that sounds like a great initiative. Lee, we’re gonna finish with you something of course as an IT pro I struggle with and the question deals with reconciling the metrics and KPI’s and development that are mostly revolving around your build being on-schedule and on-budget delivery against software security basically getting it out the door, don’t worry about security. How do you reconcile those metrics?
Lee: Well, that last part is obviously not the right answer. There’s other ways to look at this as well. It’s not just the business need of software you’re producing or the security. Are there architectural needs? Maybe you have a product that if you don’t make some changes it won’t continue to perform at the level it is. So I think the main thing for me and all of these scenarios is to get all of these out on the table for discussion and make sure that everyone is discussing all of those, not just the business side of things but also the security needs of the product, the architectural needs of the product and make sure that all of those are considered together. Generally, I’ve found that when there are problems with the assets, when people aren’t talking about all those or someone makes an assumption or it’s not brought to the table where somebody is arguing strongly and advocating for doing something a certain way. And so I think that’s the thing that I would encourage is within development environments make sure that you have healthy conflict among all the different parties and that means people who are advocating for specific things make sure that you have someone advocating for great architecture, great security, great user interface design as well as the needs of customers. And generally when I’ve seen a good software design is because those things are all being brought together, considered together, debated. And sometimes the answer may be that you delay an architectural feature or you delay something a customer’s requested because something else is more important. But I think if you get these all out on the table and have a plan for when you are gonna tackle them I think that is generally the way that I’ve seen that work well
Thomas: Agreed, it seems like it was a lifetime ago when I was myself the junior developer and security was more of the afterthought for everything, right? Business needs, first, don’t worry about that. We don’t have time to do that now, we’ll fix it later, but we know that’s never the case, right.
Lee: That is true and again it happens in so many areas of software development as well. There’s a lot of constituencies and they’re all important. So they should all be debated and prioritized against each other. Since you gonna end up making the best software that solves the needs of the customers this solving business problems is one of those needs. So is having a great interface and something that performs as well as something that is very secure.
Thomas: Absolutely, so that brings us to the conclusion. I have some resources to share. Again, you will get an email with a link to the slide deck so you can download and make it easy to just click through these. We mentioned the Secure by Design Resource Center earlier. I’m not sure if we mentioned the Orion Assistance Program. We spent some time on that in the last episode of Secure by Design. I would encourage you to watch that or just go to that link and you can get more details about the links for a security advisor as well inside the slide deck and the information about the new digital code signing certificate. I think that was pushed out earlier this week or last week and also a link for our trust center. And with that, we are at the end. I wanna thank Lee and Caleb for both of you joining me tonight here. I also wanna make sure we again thank the audience for showing up we know they have their choice of webinars these days and we’re very thankful that they chose to spend time with us today. So again Lee, Caleb before we sign off, just wanna say thank you and have a good night.
Lee: Thank you Thomas, thank you Caleb.
Caleb: Take care.
