Retention, throttling & limits

This page documents the operational envelope of the log pipeline — how long logs stay searchable, the limits that apply at ingest, and how the store is secured.

Retention

The searchable hot window is retained for 30 days. Lines older than that age out of the log store automatically. The 30-day window applies uniformly across all servers and sources in your organisation.

The raw batches the agent ships are also written to a durable object-storage archive, partitioned by date. The archive exists for durability and replay — if the hot store ever needs to be rebuilt or a batch re-processed, the source of truth is intact.

Need longer than 30 days?

Extended and tiered retention is an infrastructure-level concern rather than a per-account toggle today. If your compliance regime requires a longer searchable window, raise it with us — the durable archive already keeps the underlying data, so extending the searchable window is a provisioning change, not data recovery.

Ingest limits

Limit	Value	Behaviour on breach
Lines per batch	1000	The ingest endpoint returns 413; the agent splits and retries.
Request signature clock skew	±300 s	Requests outside the window are rejected — keep host clocks in sync (NTP).
Body encoding	gzip	Batches are compressed on the wire; the endpoint counts lines without parsing each one on the request thread.

The per-batch line cap matches the agent’s own WT15_LOG_MAX_LINES default, so under normal operation the agent never trips it.

Back-pressure & delivery guarantees

The pipeline favours not losing lines over lowest possible latency:

• If the durable archive write fails, the ingest endpoint returns 503 and the agent keeps its file offset and re-ships on the next cycle — delivery is at-least-once.
• The agent checkpoints every source by inode and offset, so a crash or restart resumes exactly where it stopped.
• Duplicate suppression: each line carries a content hash derived from its server, source, timestamp and message, so replays after a retry don’t create duplicate entries.

How the store is kept private

• The log store is never exposed to the public internet. In production it sits behind an internal reverse proxy on a private network; the store’s own port is not published.
• All reads go through the WatchTower15 API, which enforces your organisation scope and the logs.view permission on every query. The browser never talks to the store.
• Agent ingest is authenticated per request with an HMAC signature keyed by a per-agent secret.

Current limits worth knowing

In the spirit of honest docs, a few things the pipeline does not do yet:

• There is no per-agent or per-organisation ingest rate cap beyond the per-batch line limit. Very high-volume sources should be scoped deliberately on the profile rather than firehosed.
• The searchable window is a fixed 30 days (see above).
• Logs from sources other than the agent (e.g. cloud-native log streams) are not ingested today — the agent is the supported shipper.

Related

• Logs overview & shipping — the pipeline and what the agent ships.
• Search & explore — querying within the retention window.
• Configuration reference — WT15_LOG_MAX_LINES and friends.