Logging is the one form of instrumentation nobody has to be persuaded to add, which is exactly why it gets out of hand. Adding a log line is free at the moment you write it. It is not free at three in the morning six months later when you are searching a hundred million of them, and it is not free on the invoice.
Volume is a query problem before it is a cost problem
The obvious cost of high log volume is ingest and retention. The one that actually hurts is that search gets slow and imprecise, so the tool you reach for during an incident becomes the tool you avoid. A query that takes ninety seconds is a query nobody iterates on, and iteration is the whole point.
The worst case is a debug line left on in a hot path. It generates enormous volume, buries everything of interest, and is invisible in review because it is one line of code that looked harmless.
Levels mean something, so use them
A convention worth holding to:
- Error means a human needs to look at this eventually. If it fires routinely and nobody looks, it is not an error.
- Warn means something unexpected happened and the system handled it. Useful in aggregate, not individually.
- Info is the record of what the system did: requests served, jobs run, state changed. This is the tier you will actually search.
- Debug is for development and should not be on in production except deliberately, temporarily, and for one service.
The failure mode is level inflation, where everything becomes an error because errors are the ones people notice. That works exactly once.
Structure is what makes them searchable
A log line that is a sentence is a string you can grep. A log line that is a set of fields is a thing you can query, aggregate, and correlate. The difference is the difference between finding a specific request and finding all requests matching a condition, and only the second one is useful under pressure.
The fields that pay for themselves immediately are a request or trace identifier, the service and version, and whatever your domain's primary key is. With a request identifier propagated across services, tracing a single failure through a distributed call becomes a query rather than an archaeology project.
Cardinality has a ceiling
The advice to add rich context has a limit that is easy to cross without noticing. High-cardinality fields, unique per request, are fine as log fields and disastrous as metric labels. A counter tagged with user identifier will produce a time series per user, and most metric backends will fall over or bill you spectacularly.
The rule I use: unbounded values go in logs, bounded ones can be metric labels. If you cannot state the maximum number of distinct values, it is unbounded.
Retention should match the question
Different logs answer different questions over different horizons. Debugging needs high detail for a short window. Audit needs specific events for a long one. Trend analysis wants aggregates, not the raw lines.
Storing everything at full fidelity for a year satisfies all three expensively and none of them well. Tiering by purpose is more work to set up once and cheaper in every direction afterwards.