Platform engineering started appearing in 2017 and was identified as a distinct DevOps role in 2019 in the book Team Topologies. Adoption quickly accelerates, and by 2022, there are platform engineering meetups worldwide, and over 6,000 DevOps professionals attend Platform Con, the first platform engineering conference. With such a fast rise in popularity, many technology professionals still need clarification on the role and how it fits into the DevOps methodology.
Definitions vary slightly, but the DevOps community generally understands Platform Engineering as the discipline of designing and building toolchains and workflows to enable self-service capabilities for developers. The output of platform engineering is an Internal Developer Platform (IDP) that acts as an abstraction layer between individual developers and the underlying technologies.
Platform engineering is distinct from site reliability engineering, though the two roles involve automation and contribute to an efficient and performant DevOps team. While Site Reliability Engineering is playing defense and working to improve reliability and resiliency, platform engineering is the offense by providing a golden path for developers to smooth bumps in the setup, integration, and delivery process to accelerate the release delivery lifecycle. The two roles complement each other to empower teams to reduce lead time for changes, change failure rates, and mean time to recovery (MTTR) while increasing deployment frequency and release velocity.
‘You build it, you run it’
Platform engineering is a response to the need for developers to have the freedom to select the configuration of an appropriate infrastructure while addressing the exponential increase in the complexity of cloud infrastructure. One of the key precepts in DevOps methodologies is ‘you build it, you run it,’ originally attributed to Werner Vogels in 2006, which makes the application development teams responsible for managing their own infrastructure. Using this approach, application development teams select the infrastructure environment for the service they build. Application development teams then need to do the infrastructure environment evaluation and selection, configuration and integration, and troubleshooting and support. Now, with containers, microservices, and orchestration layers, simple scripts replace routine deployments with multiple YAML files, Helm charts, Puppet manifests, and Chef recipes. The complexity only increases as applications scale and move to multi-cloud deployments.
Managing the configuration and deployment across diverse environments consumes resources, places a drag on developer productivity, and slows release velocity. There are other drawbacks, such as the increased security risk and the lengthening of ramp-up for new DevOps team members. Enforcement of consistent policies and practices across multiple teams becomes more difficult as the complexity increases.
Instead of building and testing code, teams were spending more time looking for the right information just to get started. “Where’s the API for that service we’re all supposed to be using?” “What version of that framework is everyone on?” “This service isn’t responding, who owns it?” “I can’t find documentation for anything!”
Internal Development Platforms
Cloud-native giants like Amazon, Netflix, Spotify, and Google are the first to identify the challenges posed by the increase in complexity and the ‘you build it, you run it' approach and begin building an internal development platform (IDP). An IDP provides an abstraction layer to standardize developer workflows and remove the complexities of the underlying infrastructure. The IDP offers developers a self-service interface through which they can create, manage, and deploy their applications. It enables developers to spend more time coding and less time configuring and managing environments, allowing organizations to accelerate feature delivery and improve developer productivity.
The IDP offers a centralized platform with a set of battle-tested tools and workflows to make it easier for application development teams to take advantage of the latest infrastructure tooling without the need to develop deep expertise. It also removes the confusion and delays to development schedules arising from too much choice.
Platform Engineering emerges
The team responsible for building and maintaining the IDP is the platform engineering team. Though each organization implements platform engineering in slightly different ways, it generally includes tasks such as:
- Identifying the needs of the application development teams
- Staying up to date on the latest technologies
- Creating a software development environment along with the underlying infrastructure and software tooling
- Automating the release process
- Ensuring infrastructure reliability and scalability
The primary “customers” for the platform engineering team are the application development teams. As they develop and evolve the IDP, platform engineering collaboratively works with the application development teams to select the tooling and then builds in the automation and integrations to create a “golden path” for the application development teams.
In many organizations, the developers have the ultimate flexibility. They can select the level of abstraction they are comfortable with to run their applications. Suppose they are new to the team or not interested in the nitty gritty. In that case, they can select a fully provisioned environment with everything they need to deploy and test their code and not spend time learning about the different container structures. On the other hand, if a developer enjoys playing around with Teraform modules and YAML files, they can also do that.
Adoption of platform engineering hit 51% in 2023, according to a Puppet 2023 State of DevOps report. With the clear benefits of implementing platform engineering and an IDP, it’s easy to understand the popularity. According to the report, over 60% of respondents saw improved system reliability, 59% saw efficiency improvements, and 58% saw delivery times accelerate. An IDP also offers interesting possibilities for teams looking to incorporate new artificial intelligence and big data platforms into their standard workflows.
What's next for Platform Engineering?
2024 promises to be an exciting year for platform engineering. The rapid increase in platform engineering adoption coupled with the astounding speed of advances in artificial intelligence (AI), large language models (LLM), and natural language chat interfaces unlock enormous potential for organizations building an IDP. Development-focused AI interfaces, such as GitHub Copilot and Amazon CodeWhisperer can generate configuration files, auto-complete code snippets, provide context to deployment errors, and deliver training and enablement documentation. While these technologies seem unlikely to replace developers, platform engineering, and documentation writers in the next year, they are poised to become an indispensable DevOps companion.
