What's new

A human summary of each release — what changed and why. The full technical list lives in the CHANGELOG, announcements in the Telegram channel.

1.9.0 · September 28, 2026 — the installer no longer touches the web server or TLS

A breaking release for the Docker-free install: the installer stopped installing and configuring a web server and certificates. Gotcha listens only on 127.0.0.1:8080, and which proxy exposes it is your call. One fix applies to every install, Docker included: under CPU load, requests to PostgreSQL could fail with i/o timeout. No migrations.

• nginx, certbot, EPEL, the SELinux boolean and firewalld are gone from the installer. In their place the documentation has an External access and TLS section: six requirements for any reverse proxy and tested examples for nginx, angie, Apache and Caddy. --domain and --email now refuse with exit code 2 and a hint, while --no-proxy and --no-firewall are accepted and do nothing, so existing automation scripts will not trip over them.
• --base-url is required for a fresh install. The external address must match GOTCHA_BASE_URL exactly, or every POST, including the first sign-up, gets a 403, so the installer no longer tries to guess it. Without the flag it asks for the address on the terminal, and with --yes or without a terminal it refuses before touching the host. Re-running with a different --base-url rewrites the address in /etc/gotcha/gotcha.env, leaves passwords and the secret key alone, and restarts the service.
• Upgrading from 1.8.x: drop --domain and --email from the command. The nginx site and certificate the previous version set up stay where they are and are yours to maintain from now on.
• Behind a proxy, the login rate limit was shared by everyone. With a reverse proxy on the same host, every request looked like it came from 127.0.0.1: the installer never wrote GOTCHA_TRUSTED_PROXIES. It now writes GOTCHA_TRUSTED_PROXIES=127.0.0.1/32,::1/128 on install and adds it to an existing env on a re-run; a value the operator already set is left untouched.
• i/o timeout to PostgreSQL under load — fixed. The check a connection gets before it leaves the pool, added in 1.8.1, waited only 50 ms for an answer. Under CPU load that was too little even for a live connection, and the next query on it failed with i/o timeout. The timeout is now one second: it still cuts off a hung connection but leaves a live one alone.
• The installer fetches what is missing. Base utilities such as tar, curl and runuser are installed instead of aborting the install, and the run ends with a summary: the version, the /readyz answer, the address and what is left to do.

1.8.1 · September 22, 2026 — installing without Docker on AlmaLinux, Rocky and RHEL

A second distribution family for the Docker-free install — and one fix that applies to every installation, Docker ones included: the connection pool could hand the application a connection PostgreSQL had already closed. Version 1.8.0 was tagged but never published: its installer did not work on AlmaLinux/Rocky/RHEL 9, nor on any arm64 host of that family, so everything it contained ships in 1.8.1 instead. No migrations, no new environment variables.

• AlmaLinux, Rocky Linux, RHEL 9 and 10 — amd64 and arm64. install-bare-metal.sh and the manual steps that mirror it now work on a second distribution family alongside Debian and Ubuntu. The two failures that used to abort the install are gone: sudo: PAM account management error at the very first contact with PostgreSQL, and the crash while adding the PGDG repository on arm64 — PostgreSQL signs its aarch64 metadata with a key of its own, and the installer imported the x86_64 one. The key is now chosen by host architecture.
• A 500 right after a PostgreSQL restart — fixed. A request arriving in the first moments after the database restarted could fail with a 500 even though /readyz reported the database as ready: the pool handed out a connection the server had already closed. The pool now checks each connection before handing it out and transparently replaces a stale one. This is the one item in the release that matters even if you are not installing anything.
• runuser instead of sudo. Commands that act as the postgres user now run through runuser: sudo runs the full PAM stack, whose account phase can still be waiting on NSS right after the system user is created. The install now requires runuser rather than sudo; it ships in util-linux, present by default on Debian, Ubuntu and EL 9, but not on RHEL 10 and its rebuilds, which install only util-linux-core and need the full package added. The preflight names it like any other missing command.
• The installer no longer stays silent about firewalld and SELinux. On AlmaLinux/Rocky/RHEL it sets the httpd_can_network_connect SELinux boolean and opens ports 80/443 in firewalld — and when it skips that, it says why: one warning if firewalld is not installed or not running, a different one (with the command to run) if the operator declined the interactive prompt, and nothing at all under the new --no-firewall flag, since that is an explicit choice. Neither the boolean nor the open ports are reverted by --uninstall.
• A failed certbot run no longer aborts the install. On either distribution family the HTTP setup on port 80 stays up, and the certificate can be obtained later with the same command.

1.6.1–1.7.1 · September 17–18, 2026 — installing without Docker, and a number behind “light load”

Three releases in a row about how Gotcha gets onto a server, and about what is written about it before you install. There is now a second way to install the product — onto a plain Debian or Ubuntu host, without Docker; the hardware requirements were re-measured on the current release, because the old figures were taken before the product had either log ingestion or host metrics; and the manual installation page stopped breaking on a question the package installer asks. No migrations, no new environment variables — if you already run it in Docker, nothing changes for you.

• Installing without Docker (1.7.0). install-bare-metal.sh, downloaded from the release page, sets up PostgreSQL, ClickHouse and the application itself as a systemd service on a plain Debian or Ubuntu host. Next to it is a documentation page with the same steps run by hand, for anyone who will not run someone else’s script in production: every command and every config file it writes is there to read. Releases now also ship per-architecture binary tarballs, and alongside the HTML docs there is a PDF bundle of the operational pages — installation, configuration, hardening, backup and restore, upgrade — in each locale: the pages you need exactly when the service itself is not reachable yet, or any more.
• Manual installation no longer breaks on the installer’s question (1.7.1). On a live terminal the clickhouse-server post-install script asks for a password for the default user, and any password set there made the very next documented step — creating the database — fail with 516 AUTHENTICATION_FAILED. The page now installs packages non-interactively, the way the installer script has done all along. The generated password of the ClickHouse gotcha user is printed in the same step that creates it, since the config keeps only its SHA-256 and the password cannot be recovered afterwards. Troubleshooting now covers both authentication failures and how to issue a new password, and the installer itself, when it cannot create the database, names the likely cause instead of a bare error code.
• “Light load” became a number (1.6.1). The hardware requirements and the figures for the small-hardware overlay were taken on an early version — back when the product had neither log ingestion nor host metrics, so the numbers described something other than what you are installing. They now name the stream the 2 GB minimum was verified against: 10,000 events, 200,000 log lines and 5,000 traces per day, four hours of continuous load with no rejected requests. Alongside it: the memory each of the three containers takes, and the share of the disk the data itself occupies.

1.5.2–1.6.0 · September 16–17, 2026 — cheap health probes and truth in the docs

Three small releases after September’s wave: health probes stopped reaching into the databases on every request, the installation guide said the parts it had left unsaid, and the escalations help stopped stating something untrue. No migrations, no new environment variables — upgrading is a matter of replacing the image.

• /healthz and /readyz no longer ping the databases per request (1.6.0). Both endpoints share one probe whose result is reused for up to five seconds. Every request to either used to cost two round trips to PostgreSQL and ClickHouse — and a probe is usually pulled from three sides at once: the load balancer, the orchestrator’s readiness check and the container healthcheck. A noticeable share of a small shared connection pool went to requests that do not even require authentication. The response body gained checked_at, the RFC3339 time of the measurement it carries, so you can tell how fresh the state you are reading is. Response codes and the meaning of both endpoints are unchanged.
• The escalations help said something untrue — check your ladders. The help panel claimed that later steps in a ladder widen the set of recipients. They do not: every step has its own set of channels, unrelated to the previous one, and a step with no channel selected notifies nobody — even when the first step has channels selected. If you configured escalations on the strength of that sentence, your upper steps may have no recipient at all. The text is fixed, and the storm-suppression help was rewritten alongside it.
• The installation guide said the parts it had left unsaid (1.5.2, 1.5.3). How much disk a source build needs on top of the running set, with the actual image sizes, and how to reclaim the build cache afterwards. A new troubleshooting case: ClickHouse fails its health check and the application never starts — look at memory and disk first, because when memory runs short the kernel kills the process and ClickHouse’s own log may hold nothing useful. The operation not permitted section now says what to do when both probes pass: the image itself was faulty, and a clean rebuild clears it. And the Go version the application image is built with is now named, next to the PostgreSQL and ClickHouse versions that were already there.
• Under the hood: the docs can no longer drift from the code in silence. Tests now fail on a mismatch: the Go, PostgreSQL and ClickHouse versions in the documentation are checked against the code, table names against the migrations, organization roles against the type and the schema, telemetry ingest endpoints against the actual routes, and the health-probe cache lifetime against the code. A separate guard catches translation keys nothing refers to any more. None of this is visible from the interface, but it is what makes the documentation state what is true now rather than what was true six months ago.

1.4.0–1.5.1 · September 14–15, 2026 — a correctness and security wave

Four releases in two days, folded into one entry: a full audit of the 1.x codebase and its remediation (1.4.0), the fix for a mistake that wave itself shipped (1.4.1), two more findings closed (1.5.0), and a rare ingest failure traced to its root (1.5.1). The 1.0 contract holds: no environment variable was renamed, ingest endpoints and request body formats did not change, and the five new migrations (0094–0098) only move forward. Four new environment variables all default to the previous behavior.

If you are on any 1.x, upgrade straight to 1.5.1. Do not stop at 1.4.0: for a day it served every page over a kilobyte as a file download — compression shipped with the wrong content type, and 1.4.1 fixed it.

• Two things change on an existing install, both about signing in. Signing in through an organization’s single sign-on no longer silently links to an existing password account by matching email address: anyone who came to own an address in the organization’s domain — a freshly created mailbox, or one reassigned after its previous owner left — could sign in as that address’s existing account without ever knowing its password. Linking is now a deliberate action the account owner takes from their own profile, so anyone who used to get in with a single click now signs in with a password once and links single sign-on by hand afterward. Separately, self-registration and email auto-linking through a generic OIDC provider now require GOTCHA_OIDC_TRUST_EMAIL=true — an arbitrary provider can forge email_verified, so trust it only for a single-tenant IdP you control yourself. Yandex ID and VK ID are unaffected; they confirm the address themselves.
• Check whether your instance ever ran without GOTCHA_SECRET_KEY. Everything encrypted since then was written under a development key — alert channel tokens, single sign-on client secrets, HTTP monitor headers. Re-encrypting after setting the key is not enough: the plaintext has already reached the write-ahead log and your backups, so those secrets need to be revoked and reissued. The condition used to be visible only as a startup log line, which does not survive container log rotation; it is now the gotcha_secret_key_insecure gauge on /metrics and a warning on the form where such a secret is entered.
• Profile parsing is bounded by memory. A crafted profile could inflate a few kilobytes of input into gigabytes of heap. Parsing now runs under a fixed budget, makes one linear pass over the input before decoding, and applies a separate byte ceiling to input whose shape is not confirmed as protobuf. The number of concurrent parses is now a memory budget rather than a count.
• Ingest rate-limits by client address before the key is authenticated. A flood of requests with random keys reached the key cache and the database — which has a small shared connection pool — before being rejected. The “invalid key” signal was also recorded against whichever project id sat in the URL, checked before the key itself, so walking ids could show an owner an alert about an attempt that had nothing to do with them. Alongside: a limiter whose key map overflowed rejected everyone, bystanders included, until entries expired — precisely the outcome it exists to prevent.
• A metric value at the edge of the float64 range could hang the handler that draws its chart forever, burning a core on every page open until restart; a value at the opposite edge drew every point at zero labelled “Infinity”. Ingest, reachable with a project’s public key, now rejects such values. The delivery log no longer shows full error text to people without permission to manage channels, and organization invitations are rate-limited: without a cap, an admin could send instance mail to any external address in any volume.
• Evaluators silently did part of their work under load — now they resume where they stopped. The host, metric rule and SLO evaluators and the escalation scheduler restarted their list every tick, so anything past the cutoff was never evaluated at all, and during an incident storm the scheduler escalated the same oldest incidents again and again while new ones waited indefinitely. How much a tick skipped is now a metric. Neighbouring misses are closed too: an SLO window with no check results is no longer treated as healthy (a monitor that went quiet could close a burn-rate incident and announce “SLO recovered”), a regression whose target left the traffic sample no longer stays open forever, and a missed-check incident is not opened twice when two replicas see the same miss.
• Quota is no longer charged before the body is parsed. A malformed or oversized request cost quota while delivering nothing, and an envelope of only profiles on an instance with profile ingest disabled answered 200 and vanished without trace. The metric and profile writers now count rows dropped on buffer overflow and refund the quota charged for them — the same accounting events and transactions already had now covers all four. Trace ids are set entirely by the client, so two projects can send the same one: resolution is now deterministic, and a project that copied someone else’s id no longer shadows their trace.
• A rare ClickHouse failure under concurrent writes was traced to its root. use of closed network connection happened like this: when a batch’s context is cancelled the client tears down the socket directly, bypassing the pool — but by then the connection had already been returned to the pool and could belong to an unrelated query, which was the one that failed. The writers' flush context is no longer cancellable, and the flush budget is held by timeouts on a separate writer connection.
• What arrived in 1.4.0. A /forgot-password page: a one-hour single-use link to the account’s address, setting a new password ends all of that account’s sessions, its own rate limit, a response that does not reveal whether the address exists, and on an instance without SMTP it says so plainly instead of showing a form that would do nothing. For operators, gotcha set-password --email=<address> reads the password from standard input, so it lands in neither shell history nor the process list. An increase aggregation for rules on monotonic counters: a “10 errors in 5 minutes” threshold now means exactly that regardless of scrape interval, and a counter reset on exporter restart no longer looks like a spike the size of everything accumulated. The retention actually applied to each ClickHouse dataset is logged at startup and exported as gotcha_retention_days{dataset="…"}. A public status page’s address can be changed without deleting the page along with its monitors and history. GOTCHA_SMTP_REQUIRE_TLS refuses to send mail if the server did not offer STARTTLS, instead of falling back to a plaintext connection.
• Pages are served compressed, and the host evaluator stopped asking the database once per host. Only markup goes through the compressor: exports and agent binary downloads pass through untouched, so resumed and range requests work exactly as before. The host evaluator now issues one grouped query per project and threshold kind — host is not part of the table’s sort key, so five hundred hosts meant five hundred queries, each reading the project’s whole window. From a live query log at five hundred hosts: 500 queries and 750,000 rows read became 1 query and 1,500 rows.
• Guides were checked against the code rather than memory, and the gaps were not cosmetic. The host guide twice sent readers to fetch the key that writes metrics but cannot register a host; the self-monitoring guide suggested an alert on a metric the product does not have; the SMTP example used port 465, which this client cannot speak at all, so mail simply hung until timeout. The webhook event reference documented 8 kinds out of 23 — all 23 are documented now, and a guard keeps the list from drifting from the code again. Two limits people kept tripping over are finally written down: revoking an ingest key takes about thirty seconds to take effect because key resolution is cached, and “end all sessions” never could end someone else’s.
• One retraction. 1.3.1 explained exec /usr/local/bin/gotcha: operation not permitted as Docker installed from snap. That explanation rode on circumstantial evidence, and a direct counter-test did not support it: on a host where the symptom reproduces, a container with no-new-privileges starts fine. The troubleshooting section no longer names a cause — it gives two checks that separate “the container’s restrictions were rejected” from “the image or binary is broken”.

1.1.0–1.3.1 · September 8–11, 2026 — ingest stops saying “accepted” for data it may drop

Four releases since 1.0, and three of them exist because a reader opened an issue and was right. None of them breaks the 1.0 promise: no environment variable was renamed, no ingest address or request body format changed, and both new migrations only move forward — upgrading from 1.0 is still “read Upgrade, pull the image, go”.

• A full buffer now answers 503, not 200 (1.2.0). Until this release ingest enqueued a payload and wrote success unconditionally, while the enqueue itself could discard the item: closed pipeline, full queue, exhausted byte budget. The next buffer, the one in front of ClickHouse, is also RAM-only and evicts the oldest records when it fills. The client saw success, stopped retrying, and the data was gone. Now every in-memory buffer on that signal’s path is checked before anything is enqueued, and if any of them is 95% full or above — the point where accepting more means evicting older data — the whole request is refused with 503 and Retry-After: 5, with nothing from it accepted. The check sits before the enqueue and before the quota charge on purpose: a retry creates no duplicates and does not burn the organization’s monthly limit. Refusals are counted in gotcha_ingest_rejected_total{reason="overloaded"}.
• A request that enqueued nothing answers 503 too (1.3.0). The 1.2.0 check left a race: an item discarded after the check still got a success. The enqueue now reports its outcome, and a request that put nothing at all into the queue is refused — retrying it is safe. A partially queued envelope still answers 200: retrying it would duplicate the part that already went through, and the product has no event-id deduplication to catch that.
• A worker waits for room instead of evicting data a client was promised (1.3.0). Before writing into a saturated batcher or span writer, an ingest worker now waits up to two seconds, polling every 50 ms. Two new self-metrics show it: gotcha_pipeline_backpressure_waits_total and gotcha_pipeline_backpressure_wait_seconds_total. If the wait budget runs out, drop-oldest still fires — deliberately, because in monitoring fresh data is worth more than stale.
• Retries no longer burn quota (1.3.0). The charge happened before the enqueue and the refusal was decided after it, so a client dutifully retrying our own 503 paid for every attempt. Quota is now charged only for what actually entered the queue, and anything refused for lack of capacity is refunded. A malformed payload is never refunded: quota doubles as an abuse limiter here, and free garbage would make a flood of garbage free.
• Profiles no longer die under someone else’s quota (1.3.0). The decision on profiles used to be made after events and transactions, so exhausting those quotas dropped profiles silently under a 200. A profile is now a first-class envelope class in the rules, in quota and in overload alike. Visible consequence: an all-profile envelope against an exhausted profile quota answers 429.
• Logs you can subtract from, not only narrow down to (1.1.0). The log filters had positive conditions only — you could narrow down to something, never remove something. In practice the thing you want gone is usually the thing you can name: a recurring nginx warning, a health-check line, one chatty service. Every facet value — severity, service, environment, attribute — now carries an exclude chip next to it, and the body text has a substring exclusion of its own. Filters can be saved, personally or for the whole project, and one saved filter per project can be marked as the default: it applies when someone opens the logs section with no query of their own, and a “show all” link drops it for that visit, so a default can never hide data from someone who did not set it.
• A no-new-privileges switch for hosts that cannot start with it (1.3.1). The app container died on exec with operation not permitted, without writing a single log line, on exactly one configuration: Docker installed from snap, where dockerd itself runs under an AppArmor profile and the kernel then refuses the container’s profile switch. The troubleshooting section of installation now names that symptom outright, along with a one-line check that does not involve Gotcha at all. The real fix is Docker not from snap; GOTCHA_COMPOSE_NO_NEW_PRIVS turns the flag off as a stopgap and stays on by default.
• What is still open, and why. Loss on a hard process kill before flush has not gone anywhere: the graceful path drains the buffers (stop_grace_period: 90s and a parallel drain), but a real crash needs an on-disk log, and that is not coming in the 1.x branch. So the ingest contract is now written down in the documentation in plain words: 2xx means “accepted into memory, written asynchronously after that”, not “stored”.

0.36.0–1.0.0 · September 5–6, 2026 — the first stable release

Four releases, and the one that matters is 1.0. What counts about the number is not how much arrived in the two months since the first tag, 0.1.0, on July 22 — it is the thing 0.x never had at all: a written compatibility promise.

• 1.0: the compatibility contract is frozen (1.0.0). From this release on, these change only in a backward-compatible way: environment variable names, ingest addresses and request body formats, the migration schema, the backup format, the names of self-metrics not marked temporary, the outgoing webhook body, the GOTCHA_AGENT_* contract, and already-published status page URLs. Anything that violates that waits for a major version. Before 1.0 the contract could change between releases, and that was stated out loud; it no longer can. What the promise deliberately does not cover — internal packages, the look of the interface, the names of counters marked temporary — is stated in the same place, along with the deprecation timelines for each item: versioning policy. What it means in practice for whoever runs an instance: an upgrade stops being an event you prepare for — read Upgrade, pull the image, go.
• The organization projects page no longer lists other organizations' projects (1.0.0). Operator precedence in the access predicate — AND binds tighter than OR — left the team-membership branch unscoped by organization, so anyone belonging to a team in another organization saw that organization’s projects on /orgs/{id}/projects as well. Access itself was never widened: every project shown was one the viewer could already open. The predicate is now parenthesized at the source, which also protects the two other queries built from it.
• The three deprecated ingest addresses are removed in 2.0, not 1.0 (0.37.0). /logs, /profiles/pprof and /api/{project}/deployments/ keep working — same headers, same counter, same responses; only the documented removal date moved. The reason is our own deprecation policy: at least one full major release has to pass between announcing a deprecation and removing it, and these were announced on August 31. Breaking someone’s integration with the release named “a promise of stability” would be a strange way to start.
• The “Overview” screen got a documentation page (0.37.0). /docs/overview in both locales: what each of the three status tiles counts and over which window, what the 24h / 7d switch does and does not affect, how the three timeline sections are filled and capped, and when the empty state shows. The screen has been the project’s entry point since 0.29.0 but had no page of its own.
• Rolling the binary back onto a newer schema no longer crash-loops (0.36.0). Startup now shows the prepared schema-version error instead — including with automatic migration enabled, which is the default: the schema-ahead check runs before the migration step, not only after it. This was one of five blockers found by an upgrade rehearsal on a copy of production data — the upgrade and the rollback were driven by hand before the release, not after.
• Dependency updates (0.37.1). golang.org/x/crypto moves to v0.56.0, closing GO-2026-6354 and GO-2026-6355, two denial-of-service advisories in the SSH package. The shipped binary was never exposed to them: the only path reaching that code is the test harness, and Gotcha serves no SSH itself. golang.org/x/text moves to v0.41.0 along with it.

0.32.0–0.35.0 · September 2–5, 2026 — environment variable renames and reliability

Four releases, one of them breaking: upgrading from 0.31.0 or below requires editing your .env first. The rest is data direction on the dependency map, editing rules instead of recreating them, and a broad pass over escalation and export reliability.

• Breaking: 31 environment variables renamed (0.34.0). Ahead of the 1.0 contract freeze, 17 server variables, 3 agent variables and 11 Docker Compose and build variables were renamed. The old names are no longer read: a non-empty value under an old name refuses to start the process and names the new variable, instead of silently applying a default. One agent variable also changed type: the collection interval is now a whole number of seconds (10–300), not a duration string like “30s”. The full mapping tables are in the CHANGELOG and on the Upgrade page. Separately: GOTCHA_SECRET_KEY is now trimmed of surrounding whitespace. If your key was ever copy-pasted with a trailing space, everything encrypted under it at rest (delivery channel secrets, SSO client secrets) stops decrypting after the upgrade — silently, with no error at startup. The recovery path runs through GOTCHA_SECRET_KEY_PREV, which is read verbatim; the steps are on the Upgrade page.
• Breaking: silent misconfigurations now refuse startup (0.34.0). An unrecognized GOTCHA_* variable in the environment is no longer ignored — startup stops and names the closest known variable if one is within a couple of letters (a typo like GOTCHA_HSTS_ENABLE without the D used to pass silently and apply a default). The same now goes for a garbled boolean (GOTCHA_SCRUB_IP=ture), an unrecognized logging level or format, an GOTCHA_SMTP_PORT outside 1..65535, zero or negative buffer limits, invalid export limits, a mistyped DSN (parsed at startup, not connected to) and a whitespace-only value. An instance already running with a broken export limit will not come back up until the value is fixed.
• Data direction on the dependency map (0.32.0). Every dependency on the map and in the table now shows its direction: ← the service reads from it, → writes to it, ⇄ both. The direction is derived from the operation verb of the spans (SELECT/INSERT, GET/SET). The layout was rebuilt — data stores on the left, HTTP dependencies on the right, the service in the middle, per-node metrics (calls, p95, error rate) on the nodes themselves; the map fits up to 16 dependencies, was 8.
• Rules are edited, not recreated (0.33.0). A metric alert rule and a suppression edge can now be edited and switched off without deleting them. Switching a metric rule off closes its open incident — previously such an incident hung forever while the escalation ladder kept sending steps. Host thresholds and suppression rules are edited in modal dialogs, and escalation steps are laid out as a grid, so a wide screen shows the whole ladder at once.
• The heartbeat monitor page stopped pretending to poll (0.33.0). The DNS/TCP/TLS/TTFB phase chart and the recent checks table are gone — a heartbeat monitor never runs a check; the uptime tiles are replaced by the last beat, the grace period and the time the next beat is expected by. The SSL tile is now shown for HTTP monitors only, and the latency heading follows the selected range.
• Keys and secrets in the interface (0.33.0, 0.35.0). Webhook targets in channel lists no longer print the secret path in full: the host and dictionary path segments stay readable while the token is masked. The DSN keys block was rebuilt as cards, and the type of a new key is picked with a segmented control. The “no active key” warning now also shows when every key is revoked, not only when there are none at all. The project page shows when a sender is still using a deprecated ingest address, and key rejections from the last hour (invalid, revoked, wrong project, wrong type) appear on the empty issues list and in the getting-started checklist.
• Escalations, maintenance windows and exports (0.35.0). An escalation step running on two replicas could be delivered twice — a channel now claims its step before it is sent, not after. An incident suppressed because its dependency was down resumes escalation once the dependency recovers, instead of staying silent for good. Uptime incident reminders stop when a maintenance window starts after the incident opened and resume once it ends; pausing a monitor silences reminders and the ladder without closing the incident. The issue export masks the assignee’s email by default, and truncating an export drops the least active issue groups instead of arbitrary ones.
• Metric points from the future (0.35.0). A point timestamped ahead of the server clock is clamped to the receive time — it used to be accepted as-is (up to a day) or dropped beyond that, so a host whose clock ran ahead was invisible to charts and threshold rules. The skew is now visible: the gotcha_metric_points_clock_skew_total counter and a rate-limited warning naming the host. Histograms with more than 512 buckets are truncated consistently, and percentiles over histograms with negative bounds no longer fall back to zero.
• Privacy and access (0.35.0). The “sign in” and “create account” links on the invitation page no longer carry the invite token as a URL query parameter, where it could leak into browser history, reverse-proxy logs and outgoing request headers — it now travels through a short-lived cookie. Erasing a data subject’s information now also removes the spans of that subject’s trace transactions, not just the transactions, and can be safely retried if interrupted. The instance administrator’s account cannot be deleted while other users exist and the role has not been transferred; a race between deleting your own account and someone else registering can no longer leave the instance with no administrator.
• Operations (0.35.0). Database containers are pinned to a specific image digest instead of a moving tag, so a deployment cannot silently pick up a different image between two runs. Stopping the stack gives both databases enough time to shut down cleanly. Startup configuration errors are reported all together instead of one per restart, and the SLO evaluator and the export janitor run under a tick budget and report their own liveness.

0.22.0–0.31.0 · August 27 – September 2, 2026 — project overview, typed keys and hardening

Ten releases in a week: a new way into a project, exports, typed ingest keys, secret key rotation and a long reliability pass over alerting.

• Overview screen and new navigation (0.29.0, 0.30.0). A project now opens on an Overview: a status line (uptime, hosts over threshold, new issues today), open incident groups and deploy markers — instead of dropping straight into the issue list. An organization gets an “All projects” screen in the organization/project switcher, and signing in without a remembered project lands there. The sidebar is split into three tiers (work areas, setup, footer), sections are named after their contents (“Transactions” is now “Performance”, “Performance issues” is “Bottlenecks”), and no page changed its URL. Issue bulk actions moved into a toolbar above the table; inputs and selects share one height in both themes.
• Error and event exports (0.22.0–0.24.0). A background CSV/JSON/NDJSON export of a project’s error groups or raw events, filtered by time range and environment. PII is masked by default — now including stacktrace, breadcrumbs, user_id (a stable per-export pseudonym) and free text; raw exports are for org admins and owners only. The set of matching groups is snapshotted before reading, so a group that gets a new event mid-export can no longer be skipped. Metadata left the file bodies (?meta=1, attributes on the page, a line in the email) — files open with standard tools again.
• Typed keys and the /api/v1 namespace (0.26.0, 0.27.0). DSN keys now have a type — browser, server or agent — with exactly the rights their source needs; a new project gets three keys right away, and a host registers only through an agent key. Keys issued earlier keep working unchanged. Logs, pprof profiles and deploy markers are accepted at POST /api/v1/logs, /api/v1/profiles/pprof and /api/v1/{project}/deployments; the old paths still work, answer with a Deprecation header and are counted by gotcha_ingest_deprecated_path_total — removal in 1.0 (moved to 2.0 in 0.37.0, see the section above). OTLP and Sentry-compatible endpoints are untouched.
• Secret key rotation (0.25.0). GOTCHA_SECRET_KEY_PREV lets you change GOTCHA_SECRET_KEY without losing SSO client secrets, channel tokens and monitor headers: old key next to the new one, restart — everything readable is re-encrypted. Values carry a format version and key id (enc:v2:<key-id>:...), so a rotation can be verified right in the database. The backfill runs on every start and picks up values left in plain text since the dev-key days. Roll forward only: once the first enc:v2 value is written, downgrading below 0.25.0 is not supported, and during a rolling deploy key rotation must wait until every instance is upgraded.
• Hardening (0.23.0, 0.28.0). docker-compose.yml listens on 127.0.0.1 by default — plain HTTP and the unauthenticated /metrics no longer face the internet past the HTTPS proxy (GOTCHA_BIND=0.0.0.0 restores the old behaviour). HSTS is configurable via GOTCHA_HSTS_*. Two unbounded-memory paths in the login-form rate limiters are closed: key length and request bodies are capped, every limiter has a hard cap on distinct keys, and the per-IP check runs before the per-account one. A new “Hardening your install” documentation page: reverse proxy, service endpoints, TLS/HSTS, security.txt and a curl self-check.
• Alerting reliability (0.23.0). Uptime monitors take part in escalation and can be acknowledged, like every other source. A failed “down” delivery is retried on the next check (up to five attempts), and with a dead channel the incident carries a “notification not delivered” badge. Recovery goes only to the channels that received the alert. Notifications name the project in the subject, body and webhook payload. A heartbeat is not credited to link previews and prefetchers — forwarding a URL into a chat no longer silences an alert. The log quota is editable in organisation settings.
• Self-metrics and configuration canon (0.23.0, 0.24.0). New metrics: ingest rejections by reason and signal, key rejections, liveness and tick duration of every evaluator, missing translation keys, ignored heartbeats. Breaking: queue metrics follow the gotcha_<subsystem>_queue_* scheme, and ten environment variables carry their unit or the right subsystem prefix (GOTCHA_RETENTION_DAYS → GOTCHA_EVENT_RETENTION_DAYS, GOTCHA_METRIC_EVAL_INTERVAL → ..._SECONDS and so on) — the old names are not read, and a start with one of them fails naming the replacement. The full table is in CHANGELOG 0.24.0.
• Flamegraph zoom (0.31.0). Every frame is a link: a click expands it to full width, ancestors stay on top as the way back, the “all” row restores the full profile. The zoom lives in the URL (focus=) and can be shared. Labels are tied to the row height and clipped inside their frame — no more text spilling out on tablets and phones.
• Also. Deleting a project deletes its logs from ClickHouse; subject data export/erasure and the backup instructions now cover logs. A layout pass at 360px: page titles, the event-frequency chart, context cards, the public status page, the SDK setup sample. The PostgreSQL connection pool has explicit bounds; stale-data cleanup also runs at startup; a failed ClickHouse migration no longer blocks rolling the version back. Documentation pages for escalation ladders and dependency-based alert suppression.

0.21.0 · August 26, 2026 — incident groups

• A cascade collapses into one card. When a node that others depend on goes down, the signals used to arrive from every direction: hosts behind the gateway fall silent, metric thresholds fire, a monitor’s SLO burns. Now a root availability incident (a silent host or a down uptime monitor) gathers its members around it — incidents of dependent hosts, monitors, host-scoped metric alerts and uptime SLOs. While the root is informing, members stay quiet and do not escalate, and the root’s notification carries a “Dependent nodes: N” line. Once the root recovers, the released members notify again and the escalation ladder restarts from the moment the group closed rather than from their own beginning.
• The root is decided by the facts, not by the order of events. A failure travelling top-down is grouped correctly too: when a node goes silent under an already-down parent, its children’s earlier incidents still join the top root’s group — even when that root is itself a monitor.
• Incident feed (under “Alerts”). One page for open groups with expandable composition, out-of-group incidents across all six sources, and everything that resolved in the last 24 hours. The composition shows who is quiet because the root is informing and who is suppressed by an unreachable parent — two distinct states that can coincide. An incident that outlives its group shows up among the ungrouped ones right away, pointing back to the group it came from.

0.18.0–0.20.0 · August 20–21, 2026 — service recipes

• Service monitoring recipes (0.19.0, 0.20.0). Ready-made recipes for PostgreSQL, nginx, Redis, Docker and MariaDB: an OpenTelemetry collector config, live-data detection, preconfigured charts and one-click recommended thresholds — useful monitoring for a service without writing alert rules by hand.
• Dependency-aware alert suppression (0.18.0). Hosts and monitors can declare dependencies: when a parent fails, its children’s alerts are silenced automatically — one notification instead of a storm, with a settling grace period and storm-free recovery.

0.13.0–0.17.0 · August 19–20, 2026 — smarter alerting

The close of the August sprint: fewer false alarms, more control over who gets notified and when.

• Notification escalation (0.17.0). A ladder of steps per severity: each step widens who gets notified after a configurable delay from when the incident opened — the same for hosts, metrics, regressions and SLOs. Acknowledging an incident stops the escalation; every incident shows a severity badge. The new Escalations screen configures both ladders with a dry-run preview.
• Project-wide maintenance windows (0.16.0). A window now silences notifications from every source — uptime, hosts, metrics, regressions, profiles, SLOs and error alerts — not just monitor checks. Windows can be open-ended for maintenance of unknown duration. Data collection and incident tracking are unaffected.
• Seasonal baseline for the regression detector (0.13.0). The detector can compare a metric against the same window on the same weekday over prior weeks — services with a pronounced daily profile stop ringing on the morning ramp-up. Falls back to the rolling baseline automatically when seasonal history is short.
• Host labels and threshold overrides (0.14.0, 0.15.0). Hosts carry an environment and a role straight from telemetry, with facet filters and grouping. Each of the four built-in thresholds can be overridden per host or per label group: a host → role → environment → project cascade with inherit / override / off at every level.

0.10.0–0.12.1 · August 18–19, 2026 — SLOs and deploy context

• SLOs and error budgets (0.12.0). A service level objective — availability, latency or uptime — on a rolling 1–90 day window. A two-window burn-rate alert opens an incident only when both the long and the short window burn the budget above the threshold. The SLO screen shows attainment, remaining budget, a burn chart and incident history.
• Deployment markers (0.11.0). CI reports a release with a single request; deployments show as version-labelled markers on the charts, and a regression that starts within 7 days of a release gets an “after deploy vX” note — the likely culprit is visible at a glance.
• Dependency map (0.10.0). A service’s external dependencies — databases, caches, outbound HTTP calls — with call volume, p50/p95 latency and error rate. Derived from the traces you already collect, no setup required.

0.7.0–0.9.0 · August 18, 2026 — logs

The third telemetry signal: logs are ingested, stored, searchable and cross-linked with everything else.

• Ingest (0.7.0). OTLP/HTTP (POST /v1/logs) and newline-delimited JSON (POST /logs), authorized with the project key. Severity is canonicalized to a six-level scale; logs get their own monthly quota and retention, and the body passes the same URL scrubbing as error messages.
• Logs screen (0.7.0). Time, severity, service and environment filters, full-text search, a volume histogram, facets with counts, attribute filters with autocomplete, “show older” pagination.
• Cross-linking (0.9.0). “Logs around this event” on an error page, “logs for this trace” on a waterfall, “host logs” on a host card.

0.5.0–0.6.2 · August 14–17, 2026 — host monitoring and the native agent

• Hosts section (0.5.0). System metrics for your servers — CPU, memory, disk, network, load average, process count — kept separate from application metrics. Four built-in thresholds (disk, memory, load, went-silent) open incidents out of the box; onboarding hands you a ready-made collector config with the key already filled in.
• Native gotcha-agent (0.6.0). A single dependency-free binary installed with one command served by your own instance — works in closed networks with no outbound internet. Runs as an unprivileged systemd service and buffers about an hour of undelivered metrics. The OpenTelemetry Collector remains a fully supported alternative.
• Security (0.5.1). Rebuilt on Go 1.26.6, closing seven Go standard library vulnerabilities.
• Self-contained builds (0.6.2). Go modules are vendored: the image build no longer reaches the internet, make up-rebuild works in air-gapped networks.

0.4.12 · August 12, 2026 — operator tier for teams

A cumulative release (includes the docs-only 0.4.11). The full story is in the blog.

• Operator tier. Project team members run day-to-day monitoring without org-admin rights: monitors, the heartbeat ping token, maintenance windows, status-page content, issue alert rules and metric alerts. Alert channels stay owner/admin: a channel’s recipient and secret are credentials, not an operational setting; the team sees the channel list and delivery log with recipients masked.
• Security. Monitor headers (e.g. an Authorization header on an HTTP check) are encrypted at rest, the same way channel secrets are. A status page’s public address is an opaque /status/p_... key instead of a slug: names can no longer be probed or squatted; old links keep working via 301.
• Reliability. Ingest: memory protection while ClickHouse is idle, honest queue/quota accounting, per-org attribution of dropped data. A corrupted channel secret is no longer treated as live — alerts are not lost on that path. Fixes for uptime check consensus, maintenance windows and OAuth invitations.
• UX. A Copy button on event pages — the event context on the clipboard in one click, as Markdown or plain text.
• Docs. The SDK section covers the CMS extension family (WordPress, Joomla, 1C-Bitrix, Drupal, OpenCart, MODX, Tilda); an AlmaLinux/RHEL installation guide.

0.4.10 · August 5, 2026 — stabilization of the 0.4.x line

The outcome of two full product audits (operations, security, accessibility, i18n) and the 0.4.5–0.4.10 patch train:

• Operations. Compose stack with memory ceilings and healthchecks; a hardened app container (read-only FS, no-new-privileges). Liveness and readiness are separate: /healthz and /readyz. --migrate-force recovers an interrupted migration; the schema carries a backward-compatibility marker — rolling back a release no longer requires restoring from a backup. Retention is configurable per data kind: metrics, profiles, incidents.
• Restricted networks. GOTCHA_TELEGRAM_API_BASE points Telegram delivery at your own Bot API server; GOTCHA_NET_MTU sets the container network MTU for tunnel setups.
• Correctness. Transaction quota is charged for what is stored, not what arrived; regression durations fixed (µs were read as ms); alerts, retries and throttling are computed by the database clock.
• Security. Removing a member from an organization revokes their teams' project access too; invite-mode registration requires the token from the link; deeply nested OTLP/JSON is rejected; the redirect guard is hardened (CodeQL report).
• Notifications & UX. One operator-chosen notification language (GOTCHA_LOCALE); “inside the perimeter” channels with full event details; keyboard navigation, contrast and screen-reader support across the UI.

0.4.0 · July 27, 2026 — time windows and explainable perf issues

• One time-range control (1h / 24h / 7d / 30d / custom) on every chart page.
• A perf-issue page explains “what’s happening / how to fix” for its kind (N+1, slow query, HTTP flood) — with the full SQL and the code location.
• Full request details on issue pages: method, URL, params, body, headers (PII-scrubbed).
• Tooltips for every metric: p50–p99, failure rate, Apdex, Web Vitals.

0.3.x · July 24–25, 2026 — reliable uptime and small servers

• Per-check retries for uptime monitors — transient blips no longer raise false incidents.
• Latency chart with DNS → TCP → TLS → TTFB phases; a distinct “partial” availability state.
• A docker-compose.small.yml overlay for minimal servers (2 vCPU / 2 GB).

0.1.0 · July 22, 2026 — first public release

The initial set: errors (Sentry SDK protocol), distributed traces and Web Vitals, OTLP metrics, CPU profiles (flamegraphs), uptime monitoring with status pages, alerts via email/webhook/Telegram, organizations and RBAC, SSO (OIDC, Yandex ID, VK ID), PII scrubbing and SSRF protection on by default — one Go binary with PostgreSQL and ClickHouse.