Thomas: Welcome to this SolarWinds Secure by Design series. Today’s topic, Our Plan for a Safer SolarWinds and Customer Community. My name is Thomas LaRock, I’m your host today. I’m a Head Geek here at SolarWinds. And with me I have the privilege of being with my CEO, Sudhakar Ramakrishna. Sudhakar, would you like to say a few words about yourself?
Sudhakar: Absolutely, good morning, everyone. And thank you for taking time to join us today. Alex, whom I’ll introduce in just a minute, and I are looking forward to this discussion. We’d like to make it worthwhile for everyone to attend this forum and get a few key principles on what we are doing at SolarWinds and sharing it with the broader industry to make us all safer and secure going forward. Just by way of quick introduction, my name is Sudhakar Ramakrishna, I joined the company exactly a month ago on January 4. And in my background, I have experience working at various companies in the areas of security, cloud collaboration, mobile technologies, virtualization, and networking, so it spans the gamut, and it gives me a vantage point to look at deployments, enterprises, technologies as systems, as opposed to point products, because as you will hopefully recognize as we go through the conversation today, it is the confluence of all of these things that must be kept in mind as we think about enterprises that are secure by design. Without further ado, I’d like to pass on to my partner Alex, and give you a quick introduction to Alex.
Alex: Thank you Sudhakar, my name is Alex Stamos. I’m currently a partner with the Krebs Stamos Group, I’m also the director of the Stanford Internet Observatory and a lecturer in the computer science department at Stanford. In my history, I’ve come from the computer security world for over 20 years, and I was the chief information security officer at Yahoo and Facebook. And I’ve spent the last couple of decades dealing with adversaries like this. And so I’d like to give you a little bit of background on who we’re dealing with and what kind of the situation is, so you can understand, then, the recommendations we’re gonna have in this webcast for dealing with this level of adversary. And then you’ll hear from Sudhakar, about how SolarWinds is themselves specifically putting these recommendations into practice and dealing with the issues that have been tracked. So what we’re talking about here since mid-December has been a discussion of an intrusion into at least dozens, probably hundreds of American businesses and government agencies by an organization called the SVR. The SVR is one of three intelligence agencies in Russia that are all active and have active cyber-intrusion capabilities. The agency we often hear about, over the last couple of years has been the GRU. The GRU is the military intelligence office, they report up through the uniformed officers in the Kremlin, and the GRU has spent a lot of time and energy trying to undermine the NATO Alliance as well as to do direct attacks against adversaries of theirs such as Ukraine. And so when we talk about the GRU, we often hear about very destructive attacks, such as pieces of malware like NotPetya, that caused lots of damage, as well as intelligence gathering that is then used rather quickly as part of disinformation and misinformation attacks. But there turns out to be two other agencies in Russia, that we have to be aware of, and those are the FSB and the SVR. Those two organizations, unlike the GRU, are the descendants of the KGB. So after the Soviet Union fell, the KGB was split up, and the domestic intelligence capabilities, effectively the secret police function of the KGB, was taken over by the FSB. The FSB does have very good hackers, but their activity is often focused on understanding what is going on within the borders of Russia as well as within their near abroad, such as the former Soviet states, like Ukraine. FSB also does a lot of hacking around the oil and gas industry.
Alex: And then there’s the SVR, the SVR is what used to be called the First Directorate of the KGB. They are the foreign intelligence agency, probably most equivalent to our CIA in the United States. Their job is to understand what’s going on around the world, especially in the countries Russia considers adversaries, such as the United States. They’re the ones that would have been running spies in the Cold War, and living in the suburbs. If you’ve ever seen “The Americans,” one of my favorite TV shows, those would be people who now would work for the SVR. The SVR has incredibly good cyber operations teams, but the difference between them and, say, the GRU, is that their goal is often the gathering of intelligence that is not used immediately, and for which they do not use for destructive attacks. And that’s what we’ve seen so far here is that one of the reasons that this campaign has been able to last for well over a year is because they’re incredibly subtle about the intrusion into all these companies. But then also of covering their tracks afterwards, and of gathering up information, exfiltrating it in a very careful way. And so that’s one of the interesting differences here. And one things we have to keep in mind, as we deal with these adversaries, is not all adversaries will do the kinds of things that GRU does, that will make them pop under your radar, to this kind of work. Finding the SVR requires a very deep hunting capability.
Alex: But overall, whether we’re talking about the SVR or another actor, one thing you have to keep in mind is that offensive cyber has become the primary way that states both try to affect geopolitics, in a way that doesn’t include kinetic action. So it has become the standard mechanism by which many, many countries succeed in their efforts, on the American example, the best example would be Stuxnet, which allowed the alliance of the United States and Israel to affect the Iranian nuclear program in a way that did not cause any cost of life, and did not have destructive effects on the civilian population of Iran. And so, you know, that kind of use of cyber, has become extremely common. It is also absolutely the number one way that intelligence gathering happens these days.
Alex: And so one of kind of the lessons that I’d like everybody to take away from discussion today, is that the number of companies that now have to consider themselves at playing at this level, who have to understand who these intelligence agencies are, the People’s Liberation Army in the Ministry of State Security in the People’s Republic of China, the Lazarus Group in North Korea, APT31 in Vietnam, the number of companies that need to play against this level of adversary has massively grown. 20 years ago, the only companies that had to think about, this kind of capability, where you have a dedicated team sitting in a government office who have months and months, perhaps even years to break into single targets, would have been members of the Defense Industrial Base. And then you had to add to that, in the mid-2000s, you had the large banks, oil and gas industry. Famously in 2009, Silicon Valley was brought into this, to the Aurora attacks were still here in Silicon Valley, we started to understand that we were playing at this level with the People’s Liberation Army.
Alex: But in 2020, 2021, it turns out that almost every Fortune 500 company, and anybody who’s a vendor to the Fortune 500, is now a legitimate target of attack, for this level of attacker. And so to protect against that, you have to really change the way you do security. If you expect that somebody every day is gonna come in, get their coffee, and they’re gonna spend the entire day trying to think about how to break into your company, and they’re gonna do that for months and months, to defend against that level of attacker, you have to act differently. And that’s what we’re gonna talk about today.
Thomas: Thanks for that wonderful background, Alex. It’s really, it’s like you said, there’s a lot to unpack as to what’s really happening. So, a quick overview of the agenda today. We’ve already taken care of the welcomes and what our focus is for today, we’re gonna spend some time talking about our principles to secure the enterprise. We will then discuss our plan for the safer SolarWinds and customer community, we’ll bring it to a close, we’ve prompted for some Q&A, I have a list of questions to ask both Sudhakar and Alex at the end, and of course, the thank you. So, Sudhakar, I wanna start with you a little bit. I want you to just tell me, why is this so important to you? Why are we here today?
Sudhakar: Tom, that is a very pertinent question, as Alex gave in his introduction, this is something that I feel very strongly about. In some ways, I have a very unique vantage point, because I came into the company just about a month ago, and had the opportunity to not only assess what we have, but also focus on what we need to do going forward, not just from a SolarWinds standpoint, but what can we contribute to the broader community. But before we think about the changes, it’s important to understand what’s happening in our landscape. And some of these trends have been going on for quite some time, except that with situations like COVID, these trends have accelerated. Most of us, for instance, work from home, and therefore devices and peripherals that we use are completely fragmented and unique to the individual as well, that puts an additional stress on IT organizations and IT systems in terms of provisioning, in terms of supporting, the needs of the users in their environments. So that’s an element of hybrid IT. On the other side, if you think about infrastructure, applications, they are being distributed as well. Some for economic reasons, and many for need-based reasons. So in that world, what happens is the job of IT, the job of securing enterprises, becomes exponentially more complex. That requires a certain and different approach to how you secure your enterprise, because what you cannot do, is to simply clamp everything down, and even if you try to clamp everything down, when you’re being fought by adversaries who have many more resources, you simply cannot maintain your defenses.
Sudhakar: Consequently also what happens either due to organizational challenges or organizational boundaries, the way things are set up, cloud configurations, Active Directory configurations, application permissions, tend to be very fragmented. So oftentimes, those themselves increase the threat surface, as we like to call it, in enterprises, and they have to be managed and understood as well. And as we just heard, threat actors are becoming increasingly sophisticated, and are much more well-resourced.
Sudhakar: So as I assess the common industry practices, including at SolarWinds, what we have continued to believe that will be highlighted here is that many of the vendor breaches are happening because of different threat vectors, right? And it is very difficult for one single enterprise using traditional practices of be it, IT management, software development, and deployment, to be able to thwart all of these challenges. And so this presents a very unique opportunity, to learn, to improve, to iterate, and share for the safety and security of all environments. So the pledge we have taken at SolarWinds is how do we create an enterprise that is secure by design in every sense of the word. And there are three elements of that which you will see.
Sudhakar: First and foremost, I like to always start with people, because security research indicates that, while there are a lot of challenges, technologies, techniques, presented to support the security needs of enterprises, many security challenges emanate because of either things like configuration problems, or human errors, clicking on a phishing attack email for instance. So there’s a very strong element in security and securing of people in terms of our behaviors, in terms of the techniques that we implement, and the principles that we adopt.
Sudhakar: Two is obviously the infrastructure that we rely on, not just premises based, but broadly speaking, hybrid IT and hybrid infrastructure. It could be virtual engines, it could be cloud instances, it could be multi-cloud instances, and then not to mention applications and people’s identities. And as a software developer, and as a software company, re-looking at build systems, our software development life cycles, which I’m now dubbing them as secure development life cycles, are all elements of what we call Secure by Design.
Sudhakar: These are things that I’ve had the experience of implementing in other companies, but there is really an opportunity to set a completely new and better standard at SolarWinds. But in doing so we felt it is not sufficient, to simply use the principles, practices that we are familiar with, or we project them to be, and instead make it a community effort. Part of making this a community effort, was why I asked Chris Krebs and Alex to see if they can come and work with us to accelerate this journey, and broaden the scope of this, and ensure that not only SolarWinds is safe, but the broader security industry, the broader community is safe as well. So what I’ll do next is turn it over to Alex to walk through a few basic principles along these three pillars. And I’ll also try to comment on what we have already done here at SolarWinds, or what we plan to do on a go-forward basis, so Alex.
Alex: Okay great, so what I’m gonna do is talk about some of the basic things you should keep in mind, based upon what we know from the forensic work that has happened inside of SolarWinds. There’s two great companies doing that, CrowdStrike and KPMG have done fantastic work on the incident response here. And so I’m basing this upon their fantastic output and I thank them for it. And what we know of the open reporting of the intrusions into other companies as part of this. So, this will be changing, also, if you want the details, we’ll be publishing a whitepaper, probably not appropriate for me to get to exact registry keys and the like today. But we’ll talk about like the overall principles people should keep in mind if you’re trying to defend against this kind of attack.
Alex: The first is that we find ourselves in interesting place where many large enterprises have, are kind of halfway moved in the cloud, when it comes to an identity management perspective. One of the great security improvements over the last couple of years has been people giving up their on-premises Active Directory and Exchange servers, and instead moving to Azure Active Directory and Office 365. I think for the most part, this is a very positive thing. But what we see right now is that lots of companies are kind of halfway moved, they have an Azure Active Directory environment, but it’s running in a hybrid mode. And it is connected up possibly to multiple domains that are hosted on on-prem domain servers, and then some cases, you have two-way trust here.
Alex: And so unfortunately, this halfway move means that, you are often exposed to vulnerabilities, both against the cloud instance of Azure Active Directory, as well as kind of the traditional escalation paths and vulnerabilities that exist in domain servers that are still held. And so one of the things that needs to be, company should think about here is really accelerating their transition to cloud identity, where their cloud identity provider, which is often Azure Active Directory, but can also be the Oktas and OneLogins and the like, that cloud identity provider is that one source of truth, and to think about the different kinds of escalation paths that still exist and whether or not they allow escalation on-prem, that becomes escalation in the cloud. This was a problem in the attack that we’ve seen, against SolarWinds, was the ability of the attackers to do things locally, that then allowed them to go back and create persistence in the cloud. And so completing that transition, and thinking very carefully about which attack surfaces you’re providing and reducing it to only the on-prem or cloud attack surface, is one thing people should do.
Alex: Next, it’s really important to audit both your on-prem systems, but also your cloud providers, and to focus on the web of trust that exists between your systems and these providers. SolarWinds obviously, is doing everything it can to secure its own systems and, but it will also be working with its customers, so that they can understand the security parameters around their on-premise software, and how they can properly secure it. But the other things that are coming out, is that there’s a bunch of different trust mechanisms that people didn’t really understand. We have this golden ticket issue, as well as a number of situations where cloud service providers were used to enter into enterprises, because those cloud service providers needed very high privileges for some initial work, and then those privileges were not removed when that work was completed. And so having a very good understanding of who you were relying upon, and which of your service providers have security privileges inside of your network, and then making sure to de-provision them, as soon as possible, is a critical thing that you’ve got to do.
Alex: Next thing to consider is kind of the basics around authentication for all of your users, and then really aggressive use of modern authentication and authorization technologies for your administrators. So this obviously means multi-factor authentication rolled out for all of your users. This entire campaign was caught, apparently, because of the use of multi-factor authentication within another company, that they got an alert that then allowed them to do an analysis. Having that kind of MFA, not only prevents exploitation and the escalation of privilege into higher level accounts, but it also creates another stream of data that you can use to monitor and alert on, and respond quickly. Having then for all of your user risk-based authentication, which is something you can get sometimes from these identity providers built in, sometimes through third parties is really critical.
Alex: And then for your administrators, conditional access, and just-in-time permission management, is a critical thing. You do not need to have dozens and dozens of people with continuous capability to have effectively global administrator access to your Azure AD tenant. You should have one or two break-glass accounts that are never used, and when they are used, it sets off alarms across the building. But then the admin accounts and again, they should be dedicated admin accounts or administrator, so not the accounts they’re using to browse the web, or to use email, they should have, you’ll probably their name dash admin accounts, in Azure Active Directory as well as on-prem. Those need to be set with very strong MFA, preferably hardware token-based MFA.
Alex: And then should have the ability to request permissions and have those permissions granted via a two-person system, just in time for them to do their work. And then those permissions need to be de-provisioned very quickly after their work is completed. This again, is something we see the SVR and adversaries of that level are very good at getting into these accounts and then once they have access to those accounts, if you’re not doing this kind of just-in-time provisioning, if you’re not doing conditional access, then they have the run of the place. And so you wanna make it extremely difficult that there’s lots and lots of opportunities to detect that the adversary is in your network, because they are constantly running up against a conditional access policy, or having to ask people to escalate their privileges on their behalf.
Sudhakar: Alex, good points all. So, as we’ve looked at, from a SolarWinds environmental standpoint, the types of things that we’ve been thinking about is not just the audit aspect of it, which is very crucial to understand, what are the policy islands? Where might configuration gaps and inconsistencies arise? And to Alex’s point, how do we consolidate them? The principle focus that we’ve had is to ensure that we build a complete enterprise where privileges are granted on a least privileged access basis, in other words, flipping some of the old paradigms on their head, and not having too many administrative accounts. And then doing the notion of just-in-time permissions, both in terms of the infrastructure, as well as the build systems, which you will hear about shortly.
Sudhakar: What this really does is reduce the attack surface that I was talking about in the context of setting the stage for this discussion, as well as eliminates or reduces the time potential for a threat actor to inject anything malicious into our systems, and to the degree that they’re able to do anything, that it actually compresses the amount of time that they can actually cause damage. So Alex, those are some of the changes that, as you know, we’ve been already implementing at SolarWinds, so why don’t we get to the next principle, and then continue the conversation.
Alex: So, now talking about software development, I think, clearly, this incident has greatly raised awareness of the supply chain security issue with both on-prem and cloud products. Now, this issue is not new, right? There have been a number of cases of the supply chain being attacked to access different enterprises, probably the most interesting nation state examples, the attacks against the RSA secure ID tokens, which were then used for intrusions against the Defense Industrial Base. But certainly the scope and scale and success of this specific attack, makes it stand out.
Alex: The other thing that stands out about this attack, is the incredible subtlety of the mechanisms by which the SVR was able to insert themselves into SolarWinds build chain. This was not an implant that was inserted into the source code, and then made its way through the build environment, the mechanism here was the insert of a custom piece of malware that was written specifically for this purpose that was inserted to a build system, that watched the build steps happen, and then replaced one file, for on the order of something like 100 milliseconds, allowed that file to get compiled, and then deleted it and cleaned up its tracks with that implant now making its way down through the process.
Alex: Defending against that level of supply chain attack, where you’re not just inserting code on the front end, or you’re not replacing DLLs, or EXEs on the back end, but you’re inserting in the middle of a very complex build process, requires building software in a very different way. And so this is something that we’re making recommendations inside of SolarWinds, but is clearly going to become a need for all enterprise software makers, because the truth is, in my experience working with hundreds of companies on security issues, I don’t think I’ve seen a build process, that is perfectly designed against this kind of level of adversary. And so some of the things, that we’re gonna have to think about.
Alex: First, your build systems for at least your release software, are going to have to be separate, isolated, administratively separated, and preferably created just in time with very little opportunity for persistence. So one of the things that happened here, was the attack on SolarWinds’s IT infrastructure allowed the attackers then to get into their build systems. But the truth is in the vast majority of enterprises, the people who are doing administrative IT, are separate from often a DevOps team, or an engineering success team, who is maintaining the build pipeline and running the build pipeline. Those are different groups, and don’t necessarily have to have access to each other’s systems.
Alex: And so building your build pipeline that is completely administrative separated, that’s on a completely different source of truth for authentication and authorization, and that is only accessible to a very small number of trusted people, is a pretty critical thing. And the best way to do this is by building preferably a cloud-based system, where you are creating containers or instances just in time based upon configurations that are visible to everybody. Those configurations are checked in to something like a GitHub or another kind of repository, that everybody has access to, that you have tightly controlled access to both the repository and then to the build config systems, and then preferably on systems that are born just for building the production release, and then are torn down and never seen again, by the time the release is signed and pushed out, this is quite difficult.
Alex: It’s difficult because build systems are generally not built, kind of thoughtfully, honestly. Build environments in complex companies, both for internal software and for ship software, grow organically as the product grows. And so it does mean you have to be very deliberate about thinking about how you do builds, because having a build system where individual engineers cannot tweak tiny little things to make it work can be difficult. That being said, once you do it correctly, it will greatly increase the efficiency of your software development process, because you will not have those weird little moments at which making the, shipping a patch, requires somebody to go in and to change one config, in a way that does not persist later and does not learn from later.
Alex: The next thing you should think about is, recording forensic artifacts throughout your build. This is not a standard thing that people generally do. Usually, what, you have these build pipelines, they’re on systems that are doing builds continuously, they’re doing CI/CDs continuously. And so as a result, you end up churning all the forensic artifacts on these systems, you’re writing down temporary files, deleting them, creating log entries very quickly. And those log entries are not tied usually to the cryptographic identity of individual files.
Alex: So building a build process where you collect forensic artifacts all the way, meaning something like a SHA-256, of the source code file, of the intermediate build steps of the temporary files created, and then of the output files and then recording that in something like a SIM, or a one-way log aggregator, is going to be a critical thing, because that would allow you to go back. And if there’s a situation where you want to check this, you have the forensic artifacts in a way that is secure against attackers cleaning up their tracks, where you can try to find the place where things deviated.
Alex: There have been some interesting research projects where people have worked on kind of cryptographically signing all of this stuff, I think these are really interesting things to work but, as of right now, if you’re trying to do this, pretty much any build system that allows you to do arbitrary scripts, will allow you to call in and to collect your own cryptographic artifacts, either by signing or by generating hashes and pushing it into a one-way logging system.
Alex: And then finally, in the build process, you need to really focus on the accountability for each piece to ship code. And so as part of your security teams exercise in this area, a really smart thing to do is to have them pick out a specific function in a DLL, a specific function in a shipped executable. And then to work backwards, who was responsible for writing this code? And does the code that was written actually match up the object code that we are shipping out to our, that we signed and shipped out to our customers? Turns out that that is not that easy, right? There’s a bunch of different security skills that are required to go backwards along that pipeline, and to figure out where did the code come from.
Alex: And so that’s an exercise I recommend any enterprise software maker do with their security team is ask their security team, “If this function right here turns out to be malicious, what capability do you have to figure out what was there?” and when you do that exercise, you will figure out, these are the things we need to change in our build process for visibility, and these are the vulnerabilities we discovered when we went through this process, where we think that if somebody was able to get a global admin or a domain admin, or the equivalent in the Unix system, if somebody was able to get administrative privileges, they would have been able to insert something in this place, that would’ve been extremely difficult for us to figure out, either in real time or through post-hack investigations.
Alex: And then, when you talk about security overall, when you do a secure development lifecycle, and you integrate security, you need to do it early into the process, and everybody’s a big fan of security training, but I think one thing that you really need to think about, is training your engineers on the positive things they need to do for security, not just about vulnerabilities. There’s way too much focus when we talk about security training for engineers of just giving them the laundry list of here’s what SQL injection is, this is what cross-site scripting is, this is what a privilege escalation is, this is what a buffer overflow is, this is what a format string bug is.
Alex: That’s great for them to understand those things, but what’s more useful is to build a software development lifecycle, where there are well-supported, well-audited functions for the most dangerous things that happen in the applications you build. Authentication, input validation, output escaping, and the like, and that those functions are then required. Your training focuses not just on the vulnerabilities, but in the mechanisms the engineers can use to reduce the attack surface and to make sure that those vulnerabilities are never created in the first place, and to reduce the ability of them to make mistakes.
Alex: And so that’s I think part of the SDLC process that needs to change a little bit here, is less focus on visual vulnerabilities, more focus on eliminating attacks on entire classes of vulnerability through the appropriate use of libraries and the reduction of attack surface.
Sudhakar: It is a philosophy of a design-based focus versus a test-based focus after the fact. Because if you take a test-based focus after the fact, that is a situation where it’s almost impossible to be able to build that into the fabric, and create an interlocking set of systems that reduce the challenges that we and others have faced. So as it relates to the build environments, some of the things that are going on to improve the integrity and as I like to say, the non-repudiation of the code that we deliver is not just the binary compatibility in one way, but we also are de-compiling staff, to make sure that it traces back to the source code. That is one aspect of it.
Sudhakar: Two is going back to the reduction of the threat surface and providing just-in-time permissions, ensuring that the least privileged access model applies, into our source code as well as our build systems themselves. And then the other process that we are taking, which I think is above and beyond what is normal, is having parallel build and parallel pipeline systems, and having check sums and hashing mechanisms across each one of those, and ensuring the integrity across multiple environments. The implication of that is that, even if one environment is compromised, the threat actor would have to move laterally across multiple environments to ensure that they are doing exactly the same things in exactly the same sequence, the probability of that is highly unlikely.
Sudhakar: And so that is a next level of security and safety that we’re injecting into our supply chain. And to your previous point, highlighting forensics or recording forensics at each one of the steps in the software development lifecycle, such that we can go back and learn and iterate, going back all the way to how we set the stage for this discussion. There is obviously one other critical set of principles that we need to adopt, so why don’t you go into that next?
Alex: Great, thank you. And so the last one is thinking about the people and the processes you have in place to prevent against this kind of intrusion. Modern defense is best thought of being threat focused, of thinking about what are the capabilities my adversaries have, and what can I do to protect against every step of them? And the difficult truth that everybody has to accept, is that if you have an organization of any complexity, more than a couple of dozen employees, and you have on-prem and cloud software, and you have all the normal kinds of complexity that exists in a modern enterprise, and you go up against an adversary like this, you will not be able to stop them from initially getting into your network. It is effectively impossible. Remember, we’re talking about adversaries that have full-time research teams, full-time R&D teams, building brand-new malware, kits, NC2 systems, and who have months and months to get into your enterprise.
Alex: Stopping the initial intrusion in the initial execution of malicious code in your environment is impossible. It doesn’t mean we don’t wanna try to make it hard, but you can’t build all of your defenses with the expectation of just keeping the attackers out. What you have to do is you have to expand the breach mentally, of what are the steps my adversary would wanna take, to successfully have a campaign against my company? Whether that’s to put something malicious into my products that I ship, whether it’s to steal data from my company, whether it’s to break my systems and to disrupt my operations on behalf of a competitor of mine, you have to think about all of those different steps, and then put protections in at each of those steps. And the framework I like to use for this is the MITRE ATT&CK framework, the enterprise compromise framework, the MITRE team has done a really good job of documenting the different kinds of steps different adversaries take along this. And so we have to shift right, a lot of our protections.
Alex: So one of the things you got to do, is you often, when you go through and you think through all the steps in the kill chain or in the MITRE ATT&CK framework, think about for each one of those steps, what are we monitoring? What kinds of logs are we collecting? What kinds of alerts are being generated? And what is our response capability for each of those steps? And you can do this both just mentally, by sitting down and thinking through it with your team, the other thing you can do is do exercises where you express all this. But the goal is for each of those steps to not expect that you can make it impossible, but to raise the difficulty of the step happening, and then to also raise the capability to monitor and detect, and then if you’re gonna do that, you have to have the ability to respond.
Alex: One of the great ways to try to figure this out, is to do red team and tabletop exercises. So, when you’re doing a red team, you’re usually doing an actual technical intrusion, where people are simulating a specific attacker skillset, and you’re seeing what kinds of information was gathered along. And so I’m a big fan of these kinds of red teams. And when you do the red team, it’s important that the blue team does not know it’s an exercise, right? So it’s a great idea to do this in a way where only a couple of executives in the company know that it’s exercise, everybody else has to act as if it’s a real intrusion, and then when you’re all done, when you’ve called it, you’ve said, “Okay, that was an exercise, great work, everybody,” and you do the post-action review, that you very carefully step through, every documented step of what the red team did. And a good red team, is spending half their time writing notes, they’re not just hacking away your stuff, but they’re keeping notes on every single step they took, and then work with the red team to understand, “Okay, great, at this step, you were local admin on this machine, and then you became a domain admin on the domain controller. Let’s talk about the three or four intermediate steps for that privilege escalation.” And then you look at those intermediate steps and you’re like, “Could we have locked that? Could we have loaded on that? Could we have responded to it?” And for each of those steps, take notes for yourself of what could we have done better, and then make that part of your security planning for the next six months.
Alex: And then you can also do tabletop exercises with executives to make sure that your response to that technical work would have been appropriate. And then what you really need to focus on, is having an internal team that can handle most the vast majority of this kind of activity. There are situations in which you can have like a managed security service provider do some of this work. But the truth is against this level of adversary, you need to have a pretty tight response time, that within 12 to 24 hours, on one of these alerts, you want to have an analyst who is diving in, and who is arms deep in the evidence that you have preferably collected into one central location and that is available to all your analysts on demand.
Alex: And so the way I usually think about this is you want to build an internal team that can handle kind of 98% of your activity, and that the vast majority of the alerts you get, of the different situations you face, can be handled internally end to end without calling on external folks. But then for the top 2%, when it’s like, oh, man, this is gone, this looks real, right? And we’re gonna have to do something here that we normally don’t do, maybe we’re gonna have to reverse engineer Windows kernel implant, maybe we’re going to have to do some really deep network forensics, because it looks like this might be a C2 mechanism command or control mechanism that’s very subtle, then you already have relationships with trusted third parties that you can bring in reasonably quickly, so you’re not caught in a multi-day kind of contract negotiation situation, or waiting for somebody to get back to you on a Monday, a salesperson, so that you can finish your response.
Alex: And so thinking about by doing the red team exercise and the tabletop exercises, you will start to figure out, what you wanna handle in house and what you think is appropriate to handle outside, and then that will help you figure out what kinds of parameters you have for who those outside vendors are.
Sudhakar: Alex, in the one month or so that I’ve been here, I’ve been talking to a number of customers and partners throughout the world, and including the authorities, both in terms of what we have learned, as well as what we are doing. So I’m highlighting some of those things here. In terms of the concept of red team and concept of expanding our monitoring capabilities across the entire event chain.
Sudhakar: At the same time, we have a very large and diverse customer base. So when we think about forming red teams and having parallel streams, et cetera, they may apply to one set of customers. But a large set of customers also are looking to us, to ensure that we provide guidelines, both in case of hardening guidelines across their environments, as well as be able to automate a lot of these things. So what we are doing is first doing it at SolarWinds, and then leveraging that experience to ensure that our hardening guides are simple and yet complete to provide customers the ability to not only protect themselves, north-south so to speak, but also east-west.
Sudhakar: So we may have guidelines across the environment for instance, help how to set up your firewall rules in cases where you want to turn on internet access on an Orion® software platform for instance. Those types of capabilities should further reduce the ability for a threat actor to do damage in a customer’s environment, but also provide you the insights required for monitoring, whether you have large teams to be able to do this or not. I consider that to be an obligation that SolarWinds has, in the context of us delivering not only powerful and affordable solutions that we have known to deliver over the years, but also powerful, affordable, and secure solutions. And as a result of that, not only do we implement the principles that we’re talking about here today, but also continue to build on it going forward. Across the board, we are looking at tools, techniques, procedures, which will translate into our product and automate it as best as we can, provide true support in a hybrid IT world, through our solutions, but also through our learnings and practices.
Sudhakar: But most importantly, I want to thank our customers, Partners, and community members who are on this call, because over the years, and while I have not been here for a long time, I have learned the history of how we have grown with and through the community of learning from you, but also sharing with you. And one of the things that I am actively trying to do at SolarWinds and across the industry, is to raise the level of collaboration, in terms of these types of best practices such that together, we can make faster progress and defend ourselves better and deliver better experiences to IT professionals and end users. So in that regard, you can expect that we will continue to be transparent. And to the degree that you need help with us talking to either you or I’ve seen some of the chat messages about your board, and others both in terms of our learnings, as well as what we are doing to progress further, we are at your service. So with that, I’d like to pass it back to you, to get some Q&A and listen to the audience’s feedback.
Thomas: Absolutely. A wonderful presentation to both of you today, great information for our customers and for the attendees. I think this is exactly what they needed to hear from us today. And, Alex, your depth of experience, I think is just unmatched, but I also wanna make sure we call out Sudhakar, your efforts, I can’t even imagine what it was like to step in as a new CEO and having this happen at the same time. I don’t even wanna go into my kitchen when it’s dirty, and think about all the stuff I have to do. So thank you for your efforts and leading us for these past eight weeks. And in bringing on Alex and Chris, just wonderful. So thank you.
Thomas: So let’s get to some questions. The first question we have, what degree of persistence, if any, do the threat actors still have in SolarWinds or other affected parties? I think Sudhakar, that might be one for you.
Sudhakar: So based on the evidence that we have seen in the investigation that we have conducted to date and the changes and the improvements that we have made, we do not see the threat actor in our environment, we do not believe they exist. But as you all know, this is an ongoing process, this is not a point-in-time type process. But we have done everything we are aware of, to ensure that our environment is safe and secure.
Thomas: Great, so next one, what is SolarWinds’s plan for checking code that is put into their application?
Sudhakar: So we covered this through the discussion today, there’s a number of initiatives that we have already implemented. And one of the key things that I highlighted, was the practice of having multiple build pipelines in multiple environments with just-in-time access, with a heavy amount of call it, forward and reverse checking of source to binary, binary to source, what that does really is greatly reduce the possibility of compromise, consistent compromise across multiple environments, which is really what we are trying to eliminate by having different administrative domains and controls in different sets of the build pipeline.
Sudhakar: Those things are in process, which we believe are far above and beyond what is an industry norm in terms of software development lifecycle, because it is typical that you will have one pipeline, set of engineers working on that, you create a binary, sign it with a certificate, and that’s your mark of integrity and non-repudiation, because you signed it with your certificate. Obviously, in the types of sophisticated supply chain attacks that we’ve seen, simply signing it with a certificate is not enough. And as you’ve seen in other instances of other vendors reporting, their certificates are also getting stolen. And I think Alex mentioned the RSA example in one instance.
Sudhakar: So the reason why we are doing it in multiple discrete with different permissions is to significantly eliminate for the same attack to happen in multiple environments, to the same files. And so that is a unique thing that we are doing, which we plan to again publish as a whitepaper, for the broader software industry to potentially follow to increase not just the integrity of what we build and deliver, but as I’ve been trying to emphasize, the notion of non-repudiation, to ensure that it is true and complete.
Thomas: Can you put together a comprehensive step-by-step outline for setting up hardened security recommendations, not just the end result, but the actual steps needed? I think we’ve kind of answered that already, but maybe elaborate again.
Sudhakar: I’ll touch on that, because I think this is a very important thing, and that has come up in pretty much all my customer conversations over the last month or so. So, just for context, we do have config and hardening guides today in as part of our standard offerings. What we are in the plans of doing now next is to take those and not just make it product specific but make it more environmentally specific. So, as we think about those guides, we should think about a customer in the context that complete deployment, as opposed to simply let’s call it Orion or any one of our SolarWinds products. That is the next evolution that you will see.
Sudhakar: In addition to that, as it relates to our own products to focus on some of the principles that we discussed, in terms of least privileged access, do we need complete admin access in every instance? And how can we enable a customer to escalate the access as needed, versus doing it by default? Those are some of the things that we harden by default, so to speak, as well as provide guidelines to harden their own environments.
Thomas: How are you reviewing and prioritizing, creating product security patches for publicly or internally known or reported security vulnerabilities?
Sudhakar: As I first highlighted in my very first blog on January 7, in the context of Secure by Design, one of our goals was to not just drive transparency, but also leverage outside parties, ethical hacking communities, such that we are able to identify as many issues as fast as possible. Clearly, what we have done from a product prioritization, engineering prioritization standpoint, is to look at some of these challenges first, and ensure that we provide a heightened level of focus from a release planning standpoint, as well as prioritization standpoint.
Sudhakar: That will be an ongoing effort, and once we kind of get through the initial push, let’s say, on these principles that we have created, then we build a more regular cadence, similar to let’s say, Microsoft having like a patch Tuesday, every Tuesday, you can expect a bunch of patches related to security. So we’ll build a cadence along those lines, which is the reason why I even call the software development lifecycle, which is SDLC, and more focus it on secure development life cycles.
Thomas: When should we expect a stable version of SolarWinds that does not need further updates and security patches against SUNBURST and SUPERNOVA?
Sudhakar: That’s a good question, what I can say here and Alex you may have a point of view here as well is, the code that is out there, which we have tested, re-certified, is free of SUNBURST already. SUPERNOVA, as you know, is unrelated to SolarWinds code itself, it’s a separate third-party issue. However, what I can say is that this is going to be an ongoing process. We are going to learn, just like any other software vendor out there. I mentioned patch Tuesdays just in my response to the previous question. We will keep learning, we’ll keep iterating, and we’ll keep being transparent. I think those are some of the things that we are committed to doing as part of the secure development lifecycle practice at SolarWinds, as I mentioned earlier, this is not like a one-and-done type thing, this is part of our fabric going forward, and we expect and hope that it is the part of the fabric of the industry going forward.
Alex: Yeah, if I may add, what you want to see here is you wanna see a regular set of updates that continuously improve the software. Like Sudhakar said, the specific issue here has been closed with the latest Orion release. So it is important for customers to get up on the latest version of the Orion Platform, and SolarWinds is working out deals with folks, to try to get them up there even if they’re out of maintenance and such, so you can talk to your customer rep at SolarWinds about that.
Alex: But the truth is, like, software is made by humans and is therefore fallible. And any reasonably complex software product, even not that complex, were posted in the last couple of weeks, we just had a bug in Sudo, like one of the most basic command line apps upon which security rests on all Unix systems, has had a vulnerability in it for a very long period of time. Like this is just the reality of software, and so what you wanna see is for companies to have a strong vulnerability disclosure program, to have a good relationship with the security research community, to intake vulnerabilities quickly, to move them through the process quickly, and then to patch them as reasonably quickly as possible, but on a regular basis, especially when you talk about enterprise software. And so those are all things that we’re gonna be working on with SolarWinds.
Alex: But, the idea that like, there’s just gonna be a version of any SolarWinds product that you’ll never have to patch again is just not true, just as that’s not true for anything that other companies that have thousands of security engineers and billions of dollars to spend on security, is not true for them. The last Microsoft Patch Tuesday had hundreds of patches for interesting security bugs, and that’s what you wanna see, that is what you see out of responsible companies.
Thomas: Okay, so that’s the end of the Q&A, we’re gonna wrap up now, again, I wanna thank you both for your time today, Alex and Sudhakar, just wonderful information for our customers. There’s some key resources here. And again, I just wanna thank you all for your time today this SolarWinds webcast series, Secure by Design, thank you.
Sudhakar: Thank you, all.
