Modern applications can generate thousands of events in moments. Engineers need those records to diagnose failures, investigate security concerns, and understand what changed in production. Yet storing every routine message can consume a substantial observability budget. log sampling offers one way to reduce the volume reaching expensive downstream systems while keeping selected events available for analysis.
The challenge is choosing what can safely disappear. A successful policy protects rare failures, preserves useful request context, and makes its trade-offs visible. This guide explains practical methods, deployment steps, and validation checks for cloud and Kubernetes teams.
Understand What Gets Kept and What Disappears
At its simplest, log sampling means retaining a subset of emitted log records instead of collecting every event. A 20% probabilistic policy aims to retain roughly one-fifth of eligible records over a sufficiently large population. It does not promise that one in every five adjacent events will be kept, or that any particular incident will appear.
Filtering is different: a rule might remove a known health-check message because its content meets a condition. Sampling selects among otherwise eligible records, often using probability or consistent identifiers. Retention controls how long collected records remain stored. These three decisions can work together, but they solve different problems. Tsuga
Understand the Benefits and Hidden Costs
Selective collection can lower data transferred to indexing and storage systems. Savings depend on where decisions occur and how vendors bill: dropping an event after a chargeable ingestion stage may not reduce that charge. Reduced volume may help search performance, but indexing design, query structure, and backend capacity still determine actual speed.
More importantly, log sampling trades completeness for efficiency. If a rare failure appears once, a random policy may discard its only useful explanation. When messages from different services are selected independently, a retained error can lack the neighboring records that explain its cause. For forensic investigations, regulatory recordkeeping, and certain security workflows, complete coverage may be essential.
The goal is not to keep as little as possible. It is to match collection policy to the evidence each team truly needs.
Five Approaches for Different Production Needs
1. Use Probability for Repetitive, Low-Risk Events
Probabilistic selection retains a configured share of eligible messages. It is easy to reason about and useful for frequent successful requests, routine heartbeat events, or verbose diagnostics. For example, a service might preserve every error while keeping 10% of repetitive informational records.
With log sampling, percentage alone is not enough. If one critical event is mixed into the same random pool, it can disappear. Separate protected event classes first, and remember that sampled counts cannot be interpreted as complete raw counts unless the selection method and weighting are understood.
2. Cap Volume During Sudden Bursts
Rate-based selection limits records within a time window, such as a configured maximum per second. It can protect a pipeline during a runaway logging loop or an unexpected request spike. Depending on implementation, excess records may be dropped or summarized.
This form of log sampling offers an operational ceiling, but burst periods are often exactly when engineers need more detail. Include a counter for suppressed events and preserve important errors separately. Without those safeguards, a quiet dashboard could simply mean that the limiter discarded the evidence.
3. Apply Rules Based on Event Value
Rule-based policies treat categories differently. A team may keep failed payment events, authentication alerts, or deployment changes while reducing routine success messages. Structured fields such as service name, severity, environment, and event type make these decisions more reliable than fragile text matching.
For log sampling to remain useful, write rules in business and engineering terms. Document why each class is protected and who owns the decision. Avoid blanket assumptions that all INFO records are harmless or all ERROR records contain enough context to debug a problem.
4. Use Stable Identifiers to Preserve Context
Deterministic selection hashes a stable identifier, such as a trace ID or request ID, to make consistent decisions. With compatible settings, related records can follow the same selection outcome rather than being randomly scattered across services.
This improves log sampling for distributed investigations, but only when identifiers are reliably propagated and collectors apply compatible policies. OpenTelemetry’s probabilistic sampler can use trace IDs or configured log attributes for consistency. It does not automatically coordinate every independent pipeline or reproduce decisions made later by trace tail sampling. GitHub+1
5. Adjust Policies as Workloads Change
Adaptive approaches change collection rates in response to traffic, budgets, or detected conditions. A service might retain more diagnostic events during an investigation and fewer routine messages during steady operation. This can be valuable for workloads with unpredictable peaks.
Adaptive log sampling requires careful guardrails. Define minimum visibility, maximum pipeline load, and a clear rollback procedure. A policy that changes automatically without explaining its decisions can make trends impossible to compare and complicate incident reviews.
Build a Safer Rollout in Stages
Inventory What Your Applications Produce
Start with a representative week of telemetry. Group records by service, environment, severity, and event type. Identify the largest recurring sources and note which records actually help during incidents. Include input from platform, security, compliance, and application owners rather than letting one team decide alone.
Before enabling log sampling, determine whether certain records have contractual, legal, or internal retention requirements. These obligations depend on industry and jurisdiction. Do not use a generic percentage on audit trails or security evidence without a documented assessment.
Establish a Protected Baseline
Create a set of records that should bypass reduction unless a formally approved policy says otherwise. Common candidates include authentication failures, production exceptions, payment failures, access-control changes, and security alerts. Preserve enough surrounding identifiers to reconstruct what happened.
A safe log sampling design should also retain unsampled operational metrics, such as total requests and error counts, when accurate aggregates matter. Metrics derived from randomly reduced log streams may understate events unless appropriately adjusted and validated.
Choose Where the Decision Happens
Application-side policies can avoid transmitting discarded messages, reducing upstream network and collector load. They also require coordination across language runtimes and development teams. Collector-side policies centralize control but still incur collection costs up to the decision point.
For log sampling within an OpenTelemetry Collector deployment, inspect the supported processors in the exact distribution and version you run. The probabilistic sampling processor supports logs in some distributions, but support and stability differ by signal. A processor intended for traces is not automatically suitable for logs. OpenTelemetry
Keep Trace Decisions and Log Decisions Distinct
Trace tail sampling waits for more of a request’s outcome before deciding which traces to keep. That approach is useful for tracing, but the Collector’s tail sampling processor is trace-focused. Do not assume its decision will automatically govern log records moving through another pipeline.
If your log sampling policy relies on trace context, test whether stored logs and traces can still be correlated after sampling. Shared deterministic keys can help for probabilistic decisions; more complex coordination may require design beyond a default configuration. OpenTelemetry+1
Pilot, Measure, and Provide a Rollback
Begin with a low-risk service or a subset of informational events. Record current ingestion volume, indexed bytes, query latency, investigation success, and the share of protected events before introducing any reduction. Run an incident replay or synthetic fault to check whether engineers still find the evidence they need.
Expand log sampling only when the pilot demonstrates acceptable visibility. Monitor the number of records dropped by policy, broken trace correlations, and changes in troubleshooting time. Keep configuration in version control and make it easy to restore full collection during a serious incident.
Calculate Savings Without Misleading Yourself
Suppose a system emits 100 million eligible routine messages daily, plus 2 million protected events. Retaining 10% of the routine pool yields an expected 10 million routine messages and keeps the 2 million protected events, for about 12 million records total. That is an illustrative record-count reduction of roughly 88%, not a guaranteed invoice reduction.
A log sampling forecast should also consider message sizes, compression, index charges, fixed platform fees, and where records are discarded. A small number of large messages may dominate billed bytes. Compare measured costs before and after rollout, and never equate a 10% selection rate with an automatic 90% reduction in every cost category.
Four Mistakes That Create Avoidable Blind Spots
First, applying one global rate to every severity treats rare failures like routine requests. Second, sampling independently in each microservice can fragment a transaction’s story. Third, forgetting to record dropped-event counters hides how much data is missing. Fourth, assuming that sampled error rates represent actual production rates can distort dashboards.
Good log sampling policies document these risks explicitly. Use structured fields, protected exceptions, consistent request identifiers, and unsampled aggregate metrics wherever possible. Revisit the rules after architecture changes, launches, new compliance obligations, or incidents that reveal missing context.
Know When Full Collection Is the Better Choice
There are workloads where reducing volume is less valuable than keeping complete evidence. Security investigations, audit-sensitive records, low-volume critical services, and difficult intermittent defects may justify retaining everything for an appropriate period. Storage tiering, shorter retention for low-value classes, and query optimization can sometimes manage costs without discarding events at collection time.
Tsuga’s guide to log sampling explains common approaches and the cost-versus-visibility trade-off from an observability platform’s perspective. Its arguments for fuller retention are worth weighing against your workload, budget, and governance requirements rather than treating any provider’s recommended strategy as universal. Tsuga
Frequently Asked Questions
Does selecting fewer logs always make searches faster?
No. Log sampling can reduce the amount of data a backend ingests or stores, but actual search speed depends on indexes, partitioning, filters, and infrastructure. Test representative queries before claiming a performance benefit.
Should error messages ever be reduced?
Usually, teams should protect production errors unless a considered policy addresses an exceptionally noisy class. log sampling applied indiscriminately to errors can remove the only record of a rare failure. Preserve context and count suppressed duplicates when limiting known noise.
How is this different from keeping data for fewer days?
Sampling decides which records enter the retained dataset. Retention decides how long stored records remain available. With log sampling, discarded events generally cannot be recovered later from that destination, even if its retention period is long.
Can a Collector automatically keep logs for saved traces?
Not necessarily. The OpenTelemetry Collector handles logs and traces through separate signal pipelines, and trace tail sampling does not automatically export its decisions to the log pipeline. Consistent identifiers help, but log sampling and trace policies need explicit coordination.
What percentage is a good starting point?
There is no universal number. Begin by protecting critical event types, then trial a conservative rate for high-volume routine traffic. Validate investigations and cost changes before adjusting log sampling more aggressively. Your acceptable rate depends on traffic and failure frequency.
How often should collection policies be reviewed?
Review them after important incidents, major releases, new services, or changes in data requirements. Also schedule periodic audits to compare discarded volume with troubleshooting outcomes. log sampling is an operational policy, not a permanent set-and-forget setting.
Conclusion
Reducing routine telemetry can make an observability system more manageable, but the missing records still carry a cost. Treat log sampling as a deliberate engineering trade-off: protect critical evidence, choose an understandable method, and validate actual outcomes. With clear ownership and a reversible rollout, teams can control volume while preserving the context needed to investigate production problems.

