Organizations implementing observability in their digital services architecture should be familiar with OpenTelemetry (OTEL) framework. While
our OTEL guide provides an in-depth examination of the benefits of this open-source framework, the potential security challenges with OpenTelemetry warrant a separate guide.
Security should be a concern for anyone using OpenTelemetry. Unsecured telemetry data is a juicy target for any bad actor looking to exploit weaknesses and dependencies within an organization’s applications or systems. Fortunately, there are ways for DevOps and other teams to improve security while continuing to leverage OpenTelemetry for the instrumentation, collection, and exporting of logs, metrics, and trace data. In this post, I’ll explore common OpenTelemetry security concerns and ways organizations can address them, either manually in-house or by leveraging an observability solution with integrated OTEL security measures.
Common OpenTelemetry security challenges
As a vendor-agnostic, open-source project from Cloud Native Computing Foundation (CNCF), OpenTelemetry can adapt to many different use cases to feed observability dashboards. However, this agility can also result in security vulnerabilities. For example, there are no standard security protocols or authentication mechanisms because no centralized entity drives security and privacy priorities.
The burden of ensuring security often falls upon organizations using OpenTelemetry to instrument their observability strategy. Let’s take a closer look at some potential OTEL security vulnerabilities to be aware of:
- Telemetry data is transmitted online and is unencrypted. Online, unencrypted transmission can increase the likelihood of in-transit or at-rest data being intercepted and tampered with, opening the way for man-in-the-middle attacks. An effective way to keep telemetry data from being intercepted or compromised is spoofing to secure transport channels where communication between the agent and platform occurs. Implementing encryption on either the transport or collector layer is necessary and should be a primary security priority.
- The lack of encryption amplifies the risk of data leaks. Since telemetry data can contain sensitive data on the state of an organization’s infrastructure and application performance, a data leak can give malicious users what they need to exploit and take out mission-critical systems. An effective solution would be implementing metadata masking at every stage, from creation to collection and transmission to a backend service.
- Collector receivers are a vulnerable point. While OpenTelemetry Collectors have basic security functions, more is needed to deter determined bad actors. When exposing collectors on a public network, user access permissions and control restrictions must be set to stop unauthorized end users from writing telemetry data to a backend and protect endpoints.
- Additional security validation is needed. The OpenTelemetry Protocol (OTLP) employs long-living gRPC receivers for which the OpenTelemetry collectors provide settings when authenticating incoming RPCs. Authentication is done via a client-server token, and organizations must deploy an OAuth 2.0 Authorization Server to validate and refresh the token. One effective approach is to layer a secure communication protocol like HTTPS and encryption like SSL or TLS over gRPC receivers, which can further secure the transport layer and minimize the risk of data leakage.
Benefits of using an observability solution with integrated OpenTelemetry security
While organizations can build their own backend and deploy the relevant security measures to secure OpenTelemetry collectors, this can cost valuable financial resources, human resources, and ongoing maintenance. An alternative is to partner with an observability vendor already embracing OpenTelemetry and implementing security measures on collectors, receivers, and the backend, allowing organizations to protect telemetry data at a fraction of the cost and effort.
As part of our
Secure by Design initiative, SolarWinds continuously tests for critical vulnerabilities and issues, provides
best practice guidance on security configurations, and employs strong encryption for traffic to and from the
SolarWinds® Platform, freeing up organizations and their DevOps and operations teams to focus on deploying and delivering key observability milestones. Our SolarWinds agent is designed to securely register with the SolarWinds Platform on installation via API token and obtain an ingestion/refresh token before commencing data transmission back to the platform.
An additional benefit an organization can gain from using an observability solution with integrated OTEL security is gaining access to valuable security and performance learnings. For instance, SolarWinds can analyze metrics and data collected via our standalone Orion
® Collector for insights we can use to optimize the performance and stability of our customer’s applications and systems.
With our integrated portfolio,
SolarWinds Observability SaaS (formerly known as SolarWinds Observability) solutions are designed to help customers achieve lower total cost of ownership compared to using multiple monitoring and management tools. By providing full-stack coverage, our observability solutions can provide comprehensive visualizations built to support real-time troubleshooting and streamlined workflows across your network, cloud, infrastructure, application, and database. Our elevated OpenTelemetry security approach is designed to eliminate the need to maintain separate OpenTelemetry security measures, saving time and resources you can channel toward other key development initiatives, including providing end users with more responsive, stable, and innovative services.
