Skip to content
WatchTower15 Docs

Logs overview & shipping

WatchTower15 includes centralised log collection built on the same zero-privilege agent you already run. It tails the log files you nominate and the systemd journal, ships them over an authenticated, durable pipeline, and stores them in a fast columnar engine purpose-built for logs — ready to search, chart and explore.

Why it’s built this way

Section titled “Why it’s built this way”

Logs are the highest-volume, spikiest data a monitoring platform handles, so the pipeline is designed for durability and cost, not just speed:

  • A columnar log store (VictoriaLogs) keeps the hot, searchable window with roughly an order of magnitude of on-disk compression and a modest memory footprint — you get fast full-text search without an oversized cluster.
  • Object storage sits in front of the store as a durable buffer and replay source, so a burst or a momentary store outage never loses lines.
  • The store is never exposed publicly. All queries go through the WatchTower 15 API, which enforces your organisation scope and the logs.view permission.

The pipeline, end to end

Section titled “The pipeline, end to end”
  agent (tail files + journald)
    │  gzip NDJSON, HMAC-signed
    ▼
  POST /api/agent/logs/ingest ──▶ object storage (durable archive, date-partitioned)
                                       │
                                       ▼
                                  forwarder ──▶ columnar log store (hot, searchable)
                                                       │
                                                       ▼
                                         WatchTower15 API  ◀── your dashboard
                                         (org-scoped, logs.view)
  1. The agent tails each configured source and batches new lines.
  2. It gzips them as NDJSON and POSTs to the ingest endpoint with an HMAC signature (the same auth used for telemetry).
  3. The endpoint archives the raw batch to object storage and hands off a pointer.
  4. A forwarder loads the batch and inserts it into the log store.
  5. You query it through the API — the store is never contacted directly by the browser.

What the agent ships

Section titled “What the agent ships”

Log files and folders

Section titled “Log files and folders”

Which paths a host tails is profile-driven (see fleet & profiles). Set the file and folder list on the profile and every host on it ships the same sources; the WT15_LOG_FILES environment variable is a per-host fallback. Each source is labelled with a log_source — by default the file’s basename — which becomes a first-class field you can filter on.

The systemd journal

Section titled “The systemd journal”

On hosts with systemd, the agent also ships the journal via journalctl, labelled log_source = journald. It reads through the systemd-journal group the service user already belongs to — no elevated privilege.

Tailing behaviour you can rely on

Section titled “Tailing behaviour you can rely on”
  • No backfill on first sight. When the agent first sees a file it anchors at the current end, so enabling logging doesn’t replay gigabytes of history.
  • Exactly-resumable. It checkpoints each source by inode + offset, so a restart resumes precisely where it left off; log rotation and truncation are detected and handled.
  • Whole lines only. A partial final line is held until its newline arrives, so you never see torn log entries.
  • Ordered. Each line is stamped with a distinct nanosecond-precision timestamp, which keeps cursor-based paging stable even for bursts written in the same millisecond.

Log levels

Section titled “Log levels”

The agent classifies each line’s severity by scanning it for the usual level keywords — emergency, alert, critical/fatal, error, warning/warn, notice, info, debug. Lines that don’t match a known keyword are stored with no level (and are still fully searchable). Levels power the histogram and level facets.

Turning logging off

Section titled “Turning logging off”

Set WT15_DISABLE_LOGGING=1 to stop a host collecting and shipping logs entirely while leaving telemetry untouched — useful for a host under investigation or one with strict egress rules.

Section titled “Related”