Observability in one Go binary: how Gotcha is built
Gotcha is a single Go binary over two data stores: PostgreSQL and
ClickHouse. There’s no message broker (Kafka) and no cache (Redis) in the
picture. The same binary, via a --mode flag, becomes an ingest endpoint, a web
UI, an uptime runner, or a remote probe — or runs all of it in one process
(--mode=all). Here’s what it’s made of and why.
What Gotcha is made of
The whole system is three containers in the stock docker-compose.yml: the
gotcha application, postgres, and clickhouse. The two stores split
responsibility by the nature of the data:
| Store | What it holds |
|---|---|
| PostgreSQL | Relational state: organizations, projects, users, alert rules, incidents |
| ClickHouse | High-volume data: events (errors), trace spans, metrics, profiles, uptime check results |
The split is natural: metadata is transactional relational data (Postgres), while the telemetry stream is columnar analytics with heavy write volume and TTL (ClickHouse).
Why no Kafka or Redis?
The ingest path (--mode=ingest) batches events, spans, metrics, and profiles
itself and writes them to ClickHouse, which is built for high insert throughput.
For self-hosted observability that’s enough — no separate broker is needed
between ingestion and storage. Session state and alert computation live in
PostgreSQL and in the process itself, so no external cache is needed either. The
practical result: fewer moving parts, less memory, and simpler operations than
stacks built on Kafka + Redis + several services.
What does the –mode flag do?
The same gotcha binary takes a --mode flag that enables a subset of
functions. This lets you run everything in one process, or spread the load
across separate processes as you grow.
| Mode | What it does |
|---|---|
--mode=ingest | HTTP ingest: Sentry envelope endpoints and OTLP metrics, batching of events/spans/metrics/profiles, alert computation over incoming data |
--mode=web | SSR web UI (templ + htmx): authentication, org/project administration, dashboards, public status pages |
--mode=uptime | Uptime check runner, incident watchdog, performance-regression and metric-threshold evaluators |
--mode=probe | Remote uptime probe: talks only to the central instance over HTTP, with no direct database access |
--mode=all | Everything above in one process — the default for a small self-hosted install |
How it scales
While load is small, --mode=all keeps everything in one process. When ingest
starts competing with the web UI for resources, the same functions spread across
separate processes: several ingest instances behind a load balancer, a
dedicated web, a dedicated uptime. And --mode=probe deploys in another
region or data center to check your services’ availability “from the outside” —
a probe opens neither PostgreSQL nor ClickHouse, it only talks to the central
instance over HTTP. So scaling isn’t an architecture change — it’s running the
same binary with a different --mode.
How data is stored and retention
ClickHouse keeps each telemetry type for a set number of days, then deletes old records by TTL. The defaults are tuned to how “heavy” the data is:
| Data type | Default retention |
|---|---|
| Events, transactions, Web Vitals | 90 days |
| Trace spans | 30 days |
| Metrics (OTLP) | 30 days |
| Profiles | 7 days (the heaviest by volume) |
Each value is configurable via an environment variable (GOTCHA_RETENTION_DAYS,
GOTCHA_SPAN_RETENTION_DAYS, and so on) and applies on the next start as a TTL
on the ClickHouse tables. Details: Configuration.
What the architecture gives you for privacy
Because everything is self-hosted, event data — often the most sensitive logs in a system — never leaves your infrastructure. On top of that, a few protections are built in at the architecture level:
- SSRF protection for outbound requests (uptime checks, webhook alerts) — by default they can’t reach private/loopback addresses, so one user can’t turn a check into a scanner of your internal network.
- Control over error text in external channels — you can send only an anonymized link to Telegram/webhook, without the error body (which may contain personal data).
- Signed session cookies with a dedicated secret; on a non-localhost address
the app refuses to start in
web/allmodes without your own key.
In short
Gotcha is deliberately “boring”: one binary, two databases, one flag to lay it out across processes. That’s a trade-off in favor of simple operations for a self-hosted instance. How to stand it up — Installation; all variables — Configuration; how to connect an app — SDK & Integrations.