Troubleshooting
Symptoms first, then the cause and the fix. Each fix names the setting or the command.
Start with searchcode doctor. It runs the startup checks against a stopped
instance as well as a running one, reads per-repository failures from disk, and
names the probable cause of the failures it recognises. searchcode doctor bundle
produces the redacted document to attach to a support ticket. Both are described
in searchcode doctor.
Two more things to have to hand:
- Log lines carry a
uniqueCode. Search your logs for it rather than for message text, which changes between releases. Log codes has the list and explains how to resolve one. - Every HTTP response carries an
X-Request-Idheader. The log lines and the audit record for that request carry the same identifier.
It will not start
A startup failure prints one plain sentence on stderr first, naming the file and the probable cause, before any structured log output. Read that line before anything else.
| Symptom | Cause | Fix |
|---|---|---|
| The first line names an unknown key and a line number | The configuration has a key the schema does not know, usually a typo or a key in the wrong block. Unknown keys are refused rather than ignored, so a misspelt authz block cannot silently grant nothing |
Correct the key. searchcode config validate checks a file without starting anything, and the configuration reference lists every key |
| It names a secret as unreadable | A *File path or *Env variable does not resolve. No secret can be written inline anywhere in the file |
Check the path is readable by the service user, or that the variable is set in the service's environment, not just in your shell |
| It says the audit sink could not be opened | audit.output is file or syslog and the destination is not writable or reachable. This is deliberately fatal: an instance that serves without recording what it denies is worse than one that does not start |
Fix the path or audit.syslogAddress, or use audit.output: stdout, which needs no writable path beyond dataDir |
| It refuses proxy mode at startup | auth.mode: proxy without auth.trustedProxyCIDRs, auth.sharedSecretFile or auth.sharedSecretHeader. All three are required |
Set them; see signing in |
| The data directory check fails | instance.dataDir is missing, read-only, or full. The check creates and removes a file, so a read-only mount fails here even when the permission bits look right |
Give the service user a writable directory with free space |
| A listen address cannot be bound | Another process holds the port, or two of server.listen, server.uiListen and observability.metrics.listen name the same address |
Free the port or move one listener |
A licence problem never stops it. An absent, expired or unverifiable licence
runs the instance in community mode and says why on /api/v1/status, the admin
page and in the log. searchcode license show prints what this build decided
about the licence file. See Licensing.
Signing in through a proxy
The listener roles are set out in Ports, names and what sits in front of each listener. The UI listener sits behind your sign-in proxy; the API and MCP listener never does.
| Symptom | Cause | Fix |
|---|---|---|
| Every UI request is refused with this request did not come from a trusted proxy | The request reached the UI listener from an address outside auth.trustedProxyCIDRs. Trust is decided on the direct peer address, never on X-Forwarded-For |
Put the proxy's own address (or its pod or node range) in auth.trustedProxyCIDRs, and nothing broader |
| the proxy shared secret is missing or wrong | The proxy is not sending the header named by auth.sharedSecretHeader, or the value differs from auth.sharedSecretFile |
Configure the proxy to add the header with the exact value. Watch for a trailing newline in the secret file |
| the trusted proxy forwarded no identity | The proxy authenticated the user but sent no X-Forwarded-User. A request with no identity is refused, never treated as anonymous |
Turn on identity headers in the proxy (for oauth2-proxy, pass_user_headers, or set_xauthrequest behind an auth-request proxy) |
| the identity headers are ambiguous | An identity header arrived twice, which means a client-supplied value survived alongside the proxy's | Make the proxy overwrite, not append, identity headers. The request is refused rather than one value being picked |
| Opening the UI's own port directly in proxy mode returns 403 | Working as intended: only the proxy may reach the UI listener | Go through the proxy's address |
| Signed in, but the person sees no repositories | Their groups match no binding. Groups are taken from the proxy's X-Forwarded-Groups and replaced on each sign-in, never merged |
See who sees what |
The searchcode auth: line printed on stderr at startup names the mode, the
trusted networks and authz.adminGroups. Check it first when people cannot sign
in.
An agent cannot connect
| Symptom | Cause | Fix |
|---|---|---|
| The client cannot find the server, or registers and then fails | mcp.oauth.publicUrl is unset or names the wrong address. It is the API listener's public URL and becomes the OAuth issuer, so a token is checked against it |
Set it to exactly the URL agents use. Restart: it is restart-only |
| The browser approval page fails with the proxy shared secret is missing or wrong | mcp.oauth.uiUrl is unset, so the client was sent straight to the UI listener instead of through your proxy |
Set mcp.oauth.uiUrl to the address people sign in at, and restart |
| Every call returns 401 although the key is right | Something in front of the API listener is authenticating, or is stripping Authorization |
Nothing that authenticates may sit in front of the API listener, and Authorization must reach it unmodified |
| Claude Desktop's custom connector cannot add the server | The custom connector is reached from the client vendor's side, so the MCP URL must be reachable from the internet | Use the mcp-remote form in Connect an agent client, which connects from the user's machine |
The client says server disconnected with the mcp-remote form |
Node.js is missing, or the key in the env block is mistyped or lacks Bearer |
Install Node.js; check the key, keeping Bearer and its space. Do not put a space after Authorization: in the argument |
| Tools are connected but do not appear in the desktop client | The client was closed, not quit | Quit the client completely and reopen it |
| An agent that worked last week now has to sign in again, in proxy mode | OAuth refresh happens on the API listener, which never sees your proxy, so it relies on the person having signed in to the UI within the last seven days | Open the UI in a browser once; the agent's next refresh succeeds |
| A key is refused as unknown | The key was revoked, expired, or was minted on another instance | Mint a new one on /account. Keys are shown once and stored as a digest, so a lost key cannot be recovered |
Which credential is actually being used is covered in Which credential do I use?.
Tools work but every answer is empty
The usual cause is access, not data. A principal whose groups match no
binding is authorised for nothing, and with authz.defaultAccess: deny (the
default) that is an empty answer, not an error.
- Open
/accountas that person. It shows their groups, the repository groups those reach, and how many repositories they can reach. Zero means no binding matches. - An administrator or auditor can open
/admin/explainto ask why a named person can or cannot see a repository. - Fix the binding, or the group the identity provider sends. Bindings reload without a restart.
See Who sees what.
A repository the web UI shows but an agent cannot find. An API key keeps the groups its owner had when it was minted; the operator's bindings still apply live, but a later change to the person's groups in the identity provider does not reach the key. Mint a new key. A service account uses the repository groups declared for it in configuration, which may be narrower than any person's.
Search finds nothing the agent expected. Check for a filter typed into the
query: a OR b path:src means a OR (b AND path:src). Use the request's
top-level path, language and repo parameters to filter the whole query. See
Searching the estate.
A repository is not searchable yet
| Symptom | Cause | Fix |
|---|---|---|
| A coverage banner says a repository is still cloning or being analysed | First ingest is still running. The instance serves throughout and reports the gap rather than an empty result | Wait. Sizing has first-ingest times |
| The repository facet marks it as failed | Its clone or fetch failed, and it is retrying with exponential backoff (indexing.retryBackoff, capped at indexing.retryBackoffMax). The last good state keeps serving |
Read the error on /admin/repos or in searchcode doctor. Once you have fixed the cause, Retry failing on /admin/repos brings every failed repository forward instead of waiting out the backoff |
| Search works but technologies, findings or dependencies are missing | The deep analysis pass has not finished for that repository. Answers that depend on it carry a gap until it does | Wait, or raise analysis.deepConcurrency if you have idle cores |
| History, blame and hotspots are empty | The repository is a directory source, which has no commits, or it was cloned with fullHistory: false |
Use a git source with full history for those features |
| A repository from a code-host connection never appears | The connection's sync is partial, paused or failed. A rate limit pauses it, and an organisation that returns zero repositories usually means the token cannot see it (for example SAML enforcement) |
Check the connection's state on the admin page. searchcode doctor --check-connections tests each token with one API call |
| A repository disappeared after a code-host change | A sync that completed with no gaps no longer lists it, so it was retired. Its clone is kept for a week before it is deleted | Restore it at the code host, or check the connection's filters |
Clone failures
| Error | Cause | Fix |
|---|---|---|
x509: certificate signed by unknown authority |
A TLS-inspecting egress proxy, or an internal git host with its own certificate authority. The image trust store does not carry it | Mount your CA bundle: Helm values caBundle, or set both GIT_SSL_CAINFO and SSL_CERT_FILE |
Authentication failed, could not read Username, or a 401/403 |
No credential, or the wrong one, for a private repository | Declare it under credentials (ssh with privateKeyFile and knownHostsFile, or https with username and passwordFile or passwordEnv) and reference it with repositories[].credential |
Host key verification failed |
The SSH host is not in the knownHostsFile |
Add the host's key to that file |
| Clones to internal hosts fail as if DNS were broken | http_proxy is set and internal hosts are being sent to the corporate proxy |
Add the internal hosts or the cluster's service range to noProxy |
| A large repository is abandoned part way | It exceeds indexing.maxRepoSizeBytes, which is checked during the clone |
Raise the limit, or clone it shallow with fullHistory: false and depth |
| Repository not found, for a repository that exists | Most code hosts answer a private repository the token cannot see exactly as they answer one that does not exist | Check the token's access to that repository |
Disk
| Symptom | Cause | Fix |
|---|---|---|
The admin page and /api/v1/status warn about disk |
Usage is at or above instance.diskWarnThresholdPercent. A warning only; nothing stops |
Add space, or see Sizing for what uses it |
| New repositories stop being ingested | Free space in dataDir fell below indexing.minFreeDiskBytes, so ingestion paused. Serving is unaffected and ingestion resumes by itself when space returns |
Free or add space |
| Disk use is much larger than the source code | Full history is the default, and history is most of the disk for an old repository | Set fullHistory: false for the repositories that do not need blame, history or hotspots. Sizing covers the trade-off |
Memory and CPU
| Symptom | Cause | Fix |
|---|---|---|
| The process is killed for memory during first ingest | The analysis store holds roughly 1.4 KB per analysed file, plus transient buffers in the deep pass | Raise the memory limit using Sizing, or lower analysis.deepConcurrency |
| First ingest is slow, and a CPU limit is set | The deep pass is CPU-bound, so throttling it only makes it slower | Set a memory limit and no CPU limit |
| The machine is busy for a long time after start | First clone and analysis of the estate. Cloning is usually the longer half and is bounded by your git host | Expected. indexing.cloneConcurrency (default 4) bounds clones; analysis.niceValue lowers the workers' priority |
The first queries after a restart are slow
Search builds a small index per repository in memory and never writes it to disk, so a restart rebuilds them. Until a repository's index is ready, search reads that repository directly: correct, just slower. Queries speed up as the indexes come back. Nothing needs doing.
If every query is slow, check that search.index.backend has not been set to
cs, which turns the indexes off. The default is caisson.
Still stuck
Run searchcode doctor bundle > bundle.json and attach it to a support request
with the X-Request-Id of a failing request and the uniqueCode of the log line
you are asking about. The bundle carries an allowlisted subset of your
configuration and never a secret value; searchcode doctor lists exactly what
it contains. If the problem started after an upgrade, see
Upgrades.