Taming the Database: How Observability Spurs Continuous Delivery

For tech professionals it’s intuitive that higher software delivery and operational performance — SDO performance is a new-ish moniker — yields higher organizational performance, or at least substantially contributes to it.  But now, five years of research from the DevOps Research and Assessment (DORA) team have consistently provided this as a data-driven conclusion for business leaders too. And when it comes to the specific technical practices that improve SDO performance, rising decisively to the top is a core DevOps workflow: continuous delivery—discovering what positively impacts it, honing those catalysts, and optimizing.

So what does impact continuous delivery? The usual suspects: deployment automation, continuous integration, branch-based development, loosely coupled architecture, and version control for all production artifacts. But also impactful and tagged in the roundup this past year for their use or lack thereof are: monitoring and observability, continuous testing, integrating data and the database into the deployment pipeline, and integrating security into software delivery. The database is a heavy theme of the latter set.

Bringing the database into the fold of DevOps’ most foundational technical process, continuous delivery, can feel like a complex, even frightening, proposition for a lot of organizations, especially those weighed down by legacy architectures. But know this: it’s doable and improves SDO performance, even for organizations with scaled, distributed data tiers.

Make the Database Part of the Herd

Regardless of the tooling a shop employs for its pipelines—Jenkins, Travis, GitLab, Amazon’s Code suite, etc—automating a software release process with a flow triggered by commits that builds, tests, stages, and ultimately deploys updates is the beating heart of continuous delivery. It’s adaptable for the database and complementary tooling with the right information and models. Baron Schwartz digs deep into schema as code and non-schema code kept in source control as well as data migration automation (and much more) in discussing how to achieve continuous delivery with databases in DevOps for the Database. But here the holistic picture will do. In Accelerate, the DORA team offers some wisdom around the big picture with pipelines and data tiers:

There are a few practices that are predictive of performance outcomes. We discovered that good communication and comprehensive configuration management that includes the database matter. Teams that do well at continuous delivery store database changes as scripts in version control and manage these changes in the same way as production application changes. Furthermore, when changes to the application require database changes, these teams discuss them with the people responsible for the production database and ensure the engineering team has visibility into the progress of pending database changes. When teams follow these practices, database changes don’t slow them down, or cause problems when they perform code deployments.

Of course, there’s a kicker, a big one. All of this working well is predicated on the observability of a system—how well your developers, your teams engaged in genuine DevOps feedback loops, can measure what they want and need to measure (and not what irrelevant predefined metrics force on them) in order to craft and push an appropriate fix, an improvement, or a new feature. Observability is a property of the system. Being able to distinguish the patterns actually playing out and relevant to queries themselves, being able to explore a relevant external viewpoint, like details about a customer-facing workload, are all key. As Baron notes, “Without observability, it’s impossible to safely automate, it’s impossible to have confidence about migrations, it’s impossible to employ continuous delivery, and so on. Each of the capabilities and categories of development rests on a foundation of being able to observe and monitor the database.” In a skilled DevOps environment, observability has to be accessible to everyone on the team because developers are responsible for the database performance of their own application. It’s unreasonable to ask them to manage systems or be accountable for performance improvements if they lack a view of the right landscape.

Everyone’s In the Saddle

All of a team’s engineers can own continuous delivery for the database (or be on the path to doing so) if they can observe and measure particular signals from both external and internal points of view. Let’s break it down:

1. First, a team needs to observe measures of Concurrency, Error rate, Latency, and Throughput (CELT) for customer-facing database workloads, i.e., the “golden signals” of the external view;
2. Second, they need to observe the Utilization, Saturation, and Errors associated with  CPU, memory, storage, and network, respectively, as those are the key resources the database uses to execute workloads.

It’s the CELT metrics for the stream of event data representing user requests (queries, SQL, etc.) that are usually tougher to acquire. Whatever tool(s), instrumentation, or processes you use to do it should, per Baron, enable developers to have immediate, high-resolution, self-service visibility into the database’s externally-focused, customer-centric workload quality of service. That will democratize database responsibilities. That will empower your team to implement authentic DevOps for the database. That will enable them to push fixes faster, to confirm outcomes faster, to work with more agility, and to continually learn. So, if your team gets there, is your DBA out of luck or out of a job? Of course not.

Your DBA is a Strategic Expert, Not Just a Wrangler

Nor a firefighter. This is not to say your Database Administrator (DBA) can’t execute a hotfix in a critical moment, but the role’s primary modus operandi should involve leveraging expertise. If the role is mostly reactive with the proverbial weight of the world on its shoulders, rather than strategic, bottlenecks around legacy IT problems continue to cascade. The strategic thinking an organization can ask of their DBA isn’t fluff and goes beyond obvious product contributions. Is your DBA working with teams to build data validation pipelines and recovery testing automation? If not, those are worthwhile goals. And, there is progress in applying deep learning to the core database problem of query optimization—i.e., letting the database system itself learn how to find the best physical execution path for an SQL query. You want a DBA who’s an expert, who has far more than a cosmetic understanding of trends like this which will be yielding business value faster than expected. That kind of actualized team member and concrete business benefit can’t exist if a DBA is constantly dealing with the fragility of out-dated processes and systems. The “A” in DBA in a high performance organization should mean architect and analyst. Building a genuine culture of data observability and data fluency among cross-functional teams is future-proofing your organization. It’s the present of DevOps for teams reporting high SDO performance and that’s ultimately steering business wins. ______________________________________________________________________