Monitoring

searchcode enterprise publishes Prometheus metrics on a listener of its own, ships a Grafana dashboard and a set of alert rules built against those metrics, answers health probes on the API listener, and can send traces to your own OTLP collector. None of it needs an agent, a sidecar or a vendor account.

The metrics listener

/metrics is served on a third port, separate from the API and MCP listener (server.listen) and the UI listener (server.uiListen).

It is a separate port rather than a route for one reason: repository names appear in metric labels, and they are often sensitive. A NetworkPolicy or firewall rule can say "this port, from the Prometheus namespace", but it cannot say "this path". Give Prometheus the port and nobody else.

The endpoint is unauthenticated, so restrict who can reach it at the network. Validation refuses an address that one of the other two listeners already uses, and the bound address is announced at startup on stderr as searchcode metrics listening on http://…/metrics. See Ports, names and what sits in front of each listener for where each listener should be reachable from.

Turning enabled off is a decision about scraping, not about exposure: the listener is already separate.

Per-repository series

indexing.metricsPerRepo controls whether the per-repository metrics carry one series per repository. Each repository costs about two series and a few hundred bytes of scrape, so set it to false above roughly 2,000 repositories.

When it is false the repo label is kept and every series collapses onto repo="all". The label set never changes with the setting, so a dashboard query that groups by repo keeps working either way. Lag collapses to the maximum (the stalest repository is the question); counts are summed.

The estate gauges are sampled every ten seconds off the request path, so a scrape reads a snapshot and never waits on a clone. /api/v1/status reports the age of that snapshot.

What is exported

The list below is the whole set. Nothing else is published: there are no go_* or process_* series.

Licence and metering

See Licensing for what each meter does.

Freshness and ingestion

searchcode_files_skipped is a gauge, not a file count, despite the name. For timeout, too_large, minified, binary, excluded and excluded_default it counts files. For gitignore, ignore_file and other it counts skip events, and a whole skipped directory is one event. Each scan replaces a repository's counts, so the value falls when a repository shrinks or an exclusion is narrowed. Alert on the value, never on increase(). A non-zero timeout or minified count is how you find out files are missing from search.

excluded is something you configured; excluded_default is the built-in exclusion set (dependency, build-output and cache directories). They have different remedies, which is why they are counted apart.

Capacity

Disk is sampled every thirty seconds, never per write. A check on every write puts a system call on the ingestion path, and a slow or failing filesystem is exactly where it would cost the most. /api/v1/status, /metrics and the admin page all read the same cached sample, so they cannot disagree. On a platform with no way to measure the filesystem the series are omitted rather than reported as zero, because a disk reported 0% full is a false all-clear.

Access and credentials

Code-host connections

partial is the phase worth alerting on. A failure is loud everywhere else; a connection quietly missing one organisation of three is not.

Search index

A repository without an index is still searched, correctly, by reading its files. The index makes search faster; it never decides what matches.

Analysis

HTTP and usage

route is the registered route pattern, never the request path. Paths carry repository names and file paths, so a per-path label would be cardinality driven by your estate.

Advisory data

An instance that has side-loaded no advisory bundle reads 1 here, which is correct: the compiled-in seed is a handful of entries and is not coverage.

The shipped dashboard and alerts

Two files are shipped with every release and built against the list above:

• deploy/grafana/searchcode-dashboard.json — one dashboard, in rows for licence and metering, freshness and ingestion, capacity, access, usage and adoption, HTTP, search index and analysis.
• deploy/prometheus/searchcode-rules.yaml — Prometheus alerting rules. Check them with promtool check rules deploy/prometheus/searchcode-rules.yaml.

The Helm chart carries the same two files and renders them when asked: serviceMonitor.enabled, prometheusRule.enabled and grafanaDashboard.enabled (a ConfigMap labelled grafana_dashboard for the Grafana sidecar). All three are off by default. See Install with Helm.

The alerts

The two licence alerts guard on > 0: a community instance and a licence with no expiry both report zero, and without the guard both would page as long expired.

Every alert about ingestion says the same thing in its description, and it is the property to remember when one fires: serving is unaffected. A failed fetch, a paused scheduler or a full disk stops the estate growing, never answering.

Health endpoints

All of these are on the API listener (server.listen) and need no credential, because a probe runs before any credential exists.

Two properties look like oversights and are not:

• /readyz does not wait for repositories. The instance is ready as soon as it can serve, and it serves while the estate is still cloning, reporting what is not ready yet as a coverage gap. Gating readiness on ingestion would mean a large estate never passes its probe and the orchestrator restarts it forever.
• /livez touches no storage. A liveness probe that reads a slow disk restarts the process and makes the disk slower.

The estate itself — per-repository state, analysis coverage, index coverage, the licence and the meters — is on /api/v1/status, which authenticates.

Tracing

OTLP tracing is off by default. It is one of the three things that can make the process open an outbound connection, and it points at your own collector, never at a vendor.

Traces are sent as OTLP over HTTP with a JSON payload, posted to /v1/traces on the endpoint (appended when the endpoint names only a host). Every span carries the same request identifier as the audit record and the log lines for that request, so one request can be followed across all three.

enabled and endpoint are restart-only on purpose: opening an outbound connection is not something an edited file should do a second later without the deployment noticing. sampleRate adjusts how much of an already-permitted stream is sent, so it reloads. A disabled tracer installs nothing at all; there is no half-armed state where a collector is dialled to find out it was not wanted.

Trying it locally

deploy/observability/ is a separate Docker Compose project that brings up an instance with Prometheus, Grafana and Jaeger already wired to it:

cd deploy/observability
docker compose up -d --build

Grafana is on http://localhost:13000 with the shipped dashboard loaded, Prometheus on http://localhost:19091 with the shipped alert rules, Jaeger on http://localhost:16686, and the instance's UI on http://localhost:18081. The shipped dashboard and rules are mounted rather than copied, so what renders is exactly what you get. It uses its own ports so it does not collide with an instance already running on the usual ones. It is for looking, not for production.

Related

• Sizing for what the capacity numbers mean for hardware.
• searchcode doctor for diagnosing an ingestion failure an alert reported.
• Log codes for the code on every log line.