DevOps has great potential; when it’s done right, it offers organizations improved collaboration, quick innovation, better quality, and shorter delivery cycles. But the reality is that some businesses don’t carry the DevOps process through to the data layer. People often say dealing with data is difficult, but the data produced or processed by applications is often more important than the application—so why isn’t this data treated to the same DevOps benefits enjoyed by our applications?
One reason is because doing so can be alarming for application developers and data professionals alike. Application developers may not be comfortable managing databases and data, and data professionals might not be comfortable with the quick pace of DevOps. These concerns are legitimate. Agility isn’t a quality often associated with data platforms, and updating data schemas (or the actual values) can be a slow process. Updates also come with inherent risks, as data loss can occur if something goes wrong when updating schemas. These risks often require planning for extraordinary actions to mitigate them.
The challenge of incorporating data into DevOps has become so significant there’s now an entire community of practices dedicated to it. Often called “Data DevOps,” these practices describe how people can apply
DevOps techniques and tools to data. Still, data and DevOps shouldn’t be considered separate entities; if data isn’t a major consideration in your DevOps efforts, you limit your potential to reap the full benefits offered by a DevOps-oriented business.
What are some of the key benefits of incorporating data into DevOps—and what can happen if you don’t do so?
Shorter Delivery Cycles
Because it focuses on continuous integration and delivery, DevOps helps deliver features to end users faster, meaning there’s a shorter time to value. But if they don’t include data in their implementation, businesses can see slowdowns whenever a new feature impacts the data layer, whether through data changes, schema updates, or anything else.
To achieve shorter delivery cycles, you have to speed the delivery of every part of your applications—and yes, that includes the data layer. Otherwise, you may encounter delays with development efforts and releases when you need to deploy data and schema changes.
Improved Quality
One of the reasons why DevOps offers improved quality for delivered features is because of the higher amount of test automation in continuous integration and continuous delivery (CI/CD) pipelines. For applications, test automation has become a standard process, but automated data testing and validation are also important.
Processes like data integration, analytics, and reporting that are data-centric are critical for modern applications, and like application functionality, automated tests need to cover them.
Another reason why automated data testing is so critical is because data is volatile; in fact, it’s the most volatile part of your system. Though your DevOps pipeline controls the application code, data flows into your business from several separate channels. For most organizations, legacy applications, partners, direct user input, and third-party applications are all sources of data. Your teams need to make sure your applications can handle data that’s collected from multiple sources.
Quick Innovation
Businesses are under constant pressure to quickly introduce new services and features, and DevOps allows them to innovate more rapidly. Because you release more frequent incremental updates, it’s easier to adjust and provide the most compelling features. This benefit, however, depends on two factors:
- Quality: You can confirm quality through data testing and validation, and it offers a safety net for developers to quickly make changes and avoid inadvertently affecting a different part of the system.
- Recoverability: This allows you to recover from features that don’t work the way you expect them to as well as bad deployments.
Recovery is straightforward for application code—all you have to do is deploy an updated version to replace whatever code isn’t working as desired.
Recoverability is a bit more complicated for data, as it can involve restoring schemas, rolling back data changes, and more. It’s crucial to have a plan in place to address data recoverability in your DevOps environment. This also means knowing when something’s broken, which requires you to have visibility and monitoring throughout the entire delivery pipeline.
Improved Collaboration
Another benefit DevOps practitioners enjoy is improved collaboration. One of the core tenets of DevOps is increasing collaboration between those responsible for delivering applications to production.
This collaboration helps ensure no part of the delivery process is left out. It also helps teams consider every aspect of successful application delivery. Data professionals, who create and maintain the data, help others learn how applications interact with data and how data flows throughout the organization. Understanding the data lineage helps keep everyone informed of the potential effects of changes.
Prioritize Data
If organizations want to maximize the value DevOps can offer, it’s just as crucial to prioritize data in the DevOps process as it is to prioritize applications. Don’t allow data to be a blind spot in your organization—your business needs automation, visibility, and transparency for your data
and your applications.
As expertise in DevOps and best practices continue to grow and change, more businesses will start to get value out of
their investments into DevOps.
