As a data professional, I often think about all the things around me generating data and consider where this data is ultimately being used. Have you ever taken a minute to consider exactly how much data is being generated at any point in time? I love collecting data and looking at ways it can be used and living in a “smart” home with appliances logging and reporting usage patterns certainly provides a lot of opportunities.
However, producing data is only part of it—where does the data go? Is it stored in a database of some sort? Is it stored in flat files? Have you considered what exactly a database is? And what your employer considers to be a database? Is a database something beyond simply a means of storing and retrieving data? Can a line of business application today exist without some form of database?
In this blog post, I’ll consider the realities of applications and databases today and what this means for organizations, and the data insights they rely on.
How Do Organizations Use Applications and Databases?
Before we dive into the question about databases and applications, we should consider a few important items:
- Where is the application source code stored?
- Is the “application” a service in a service-oriented architecture?
- Where is the application configuration information stored?
Even basic file systems are a type of hierarchical database, providing storage and access to the files stored on them. Today the word “database” is commonly used to refer to a relational database that stores data in schemas of tables with indexes, yet this is merely one type of database. There are NoSQL databases, object-oriented databases, distributed databases, flat model databases, graph databases, and “big data” databases for implementing one or more of the other database types as a means of providing flexibility in the storage of data while still providing access for analytics.
Given all the above, it begs the question: can you have a useful/meaningful application without a database? It would be as easy to answer this question with “Yes!” as it is today. For example, my smartphone has an application that functions as a level—it doesn’t store any type of state; it simply shows whether the orientation of the phone is level to the horizon or not, using the internal accelerometer. This is an incredibly useful app without a database, but I would be willing to bet somewhere the code is checked-in to a source control system with a database. Even GitHub, one of the most popular source control solutions available, has a database behind it.
While we’re a long way from punch cards on a ring for programming computers, isn’t the code itself simply a database of operations we want performed, providing outputs from inputs? The fact is, data is being generated by simple household items such as water heaters, thermostats, refrigerators, and also our cars for diagnostics purposes. The Internet of Things (IoT) world is an almost unrestrained producer of data.
Examples of When You Don’t Need a Database (Until You Do)
While not all the data gets saved immediately to a relational database, there’s likely some form of intermediate storage format to consider. JSON, XML, delimited text files, GIS, EDI, and HL7 messages all store data, some even acting as databases for specific applications. It might be the sensors or monitors producing the data are merely broadcasting to anyone willing to listen. I have built more than a few Arduino projects with my kids where the code for the Arduino does something like the following:
- Wait for a sensor reading to occur
- Perform an action based on the value from the sensor
- Loop back to 1
This is a totally stateless design that only consumes real-time data; nothing is stored anywhere, and it’s truly an application without a database of any type, performing a highly specific purpose. One of the projects doesn’t even have the source code retained, as my 15-year-old son didn’t believe me when I said he might want to change the behavior or add to it at some point in the future.
Now let’s consider a commercial example of an application not using a traditional database: a simple soil moisture monitor. When the moisture gets too low, the monitor uses a solenoid to open a valve to mist water over a garden, and when a short timer expires, it closes the valve again. This is a simple, real-time solution to a problem that works well—until the owner wants to get some metrics, such as how many times a day the valve gets turned on and how it compares to the same week in a previous year? Perhaps they’re trying to make projections for how much water might be needed in rain barrels to offset the usage?
How do you answer these questions? A typical approach would be to start logging each time the valve is opened, which leads us back to requiring a form of data storage, needing the data to be available for analysis, and usually some form of database as the ultimate destination of the data. Let’s face it—data is power to a business. Businesses pay big money to access data produced by other companies, as it drives the bulk of decision-making processes.
Why Databases Are Important for Businesses
Can you have an application without a database today? Yes, in specific cases, I believe it’s possible, but those cases are becoming fewer and fewer, with even the most basic devices in our everyday interactions logging information for review/consumption later. Logging, monitoring, and reporting are crucial aspects of application design that are sometimes missed during the requirements-gathering stage. In many cases, I’ve seen where this is a complete afterthought, with major limitations as a result. It’s also common for the requirements to change through what’s known as scope creep, increasing the complexity of systems and adding new requirements to previously existing ones.
I would argue it’s extremely difficult to have a true line of business application without a database of one sort or another
somewhere in the architecture. Modern tiered application architectures are more and more complex, utilizing hybrid solutions across different platforms, cloud providers, and even OSes. Have you ever stopped to ask, where’s all this data going? If you call yourself a “full-stack” developer/architect, you should be asking this question all the time.
More than ever, data is one of the biggest drivers of business decision-making, marketing, cost-savings, and ultimately success or failure. Whether the data is stored in a relational database, a NoSQL database, the file system, or possibly another solution not discussed in this article, having an insight into all this information is critical to your business.
How to Achieve Database Observability
Do you know all the interrelated components comprising the entire stack of your most critical business systems? Do you know what information the business considers essential and whether it’s being retained in a meaningful way?
With data continually growing more mission critical, it’s important to have a means of monitoring the full application stack. Learn more about how SolarWinds Observability SaaS (formerly known as SolarWinds Observability) solutions built on the
SolarWinds® Platform are designed to deliver flexible implementation of monitoring entities to help you go beyond single service awareness with dynamic rules grouped into comprehensive dashboards for displaying true “state of the union” observability across an entire IT landscape, including database observability.
