Enumerating a code host with `searchcode discover`

searchcode never calls a code host. There is no GitHub app to install, no OAuth consent screen, no privileged token sitting on the server, and no background job reconciling your estate against an API. The instance reads one thing: the configuration you committed.

searchcode discover is how that configuration gets written without anybody typing two thousand clone URLs. It runs wherever you run CI, enumerates an organisation or group, and writes a YAML fragment. A human reads the diff and merges it.

scheduled CI job  →  searchcode discover  →  git diff  →  pull request  →  review  →  merge  →  indexed

The workflow is the feature. A tool that prints YAML at you is not worth shipping; a new repository appearing in a reviewable diff with a real approval gate is. That is why this page carries working jobs rather than a flag reference.

What it produces

# generated by: searchcode discover --provider github --org acme
# review before merging — searchcode never reads a code host at runtime.
# Every repository needs at least one group; `searchcode config validate`
# fails while any is unclassified.
repositories:
  - url: "https://github.com/acme/billing.git"
    groups: [payments]            # suggested from topic sc-group:payments
  - url: "https://github.com/acme/platform-api.git"
    groups: []                    # UNCLASSIFIED — assign a group or config validate will fail

Four properties are deliberate.

It writes a conf.d/ fragment, never searchcode.yaml. Policy — your authz bindings, your repository groups, your auth mode — is hand-authored and stays in the main file. Generated volume lives beside it. A two-thousand line repository list merged into the same file as twenty lines of bindings makes the diff that matters unreviewable. discover refuses to write over the file --config names.
It is idempotent, and there is no timestamp. Re-running against an unchanged organisation produces byte-identical output, so git diff shows only what actually changed. A generator that produces a fresh diff every run is one nobody reads by the fourth time.
Unclassified repositories are emitted, not omitted. A new repository appears in the diff with an empty group list and a comment. Silently leaving it out is how a repository never gets indexed and nobody finds out for a year. searchcode config validate fails while any entry is unclassified, so the gate is the merge rather than somebody's attention.
It never writes a credential. Repositories reference a credential by name. The token that generated the file is not in it and cannot be, so the output is safe to commit by construction.

Group suggestions from topics

A repository topic of the form sc-group:<name> is read as a suggestion:

  - url: "https://github.com/acme/billing.git"
    groups: [payments]            # suggested from topic sc-group:payments

Two limits, and both matter:

The name has to already be a declared repositoryGroup in your config. A topic naming something you have not declared produces an unclassified entry with a note saying which topic was ignored. The generator never invents a group.
A human accepts the line in the pull request. Nothing about a topic reaches a running instance.

This is the whole of what an earlier design did at runtime, minus the part where anybody who could edit a repository's settings could choose its audience with no review.

The token

One read-only metadata token. It needs the ability to list repository names and nothing else — not repo scope on GitHub, not read_api on GitLab, and emphatically not the per-user OAuth tokens an ACL-sync integration needs. That is a materially smaller ask of a security team, and reviewers notice.

It is supplied by reference, never inline:

--token-file /run/secrets/discovery-token
--token-env  GH_TOKEN

There is no --token flag. A token on a command line lands in the CI job's logs, in shell history, and in the process table of whatever else is on the runner. With neither flag given, SEARCHCODE_DISCOVER_TOKEN is read.

Because the token lives with the CI job rather than with the server, compromising the running instance does not expose it at all.

GitHub Actions

Put this in the repository that holds your searchcode configuration.

# .github/workflows/searchcode-discover.yml
name: searchcode discover

on:
  schedule:
    - cron: "17 6 * * 1"        # Mondays, 06:17 UTC — off the hour on purpose
  workflow_dispatch:

permissions:
  contents: write               # to push the branch
  pull-requests: write          # to open the PR

jobs:
  discover:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install searchcode
        run: |
          curl -fsSL -o searchcode \
            https://searchcode.com/download/searchcode-linux-amd64
          chmod +x searchcode

      - name: Enumerate the organisation
        env:
          SEARCHCODE_DISCOVER_TOKEN: ${{ secrets.DISCOVERY_TOKEN }}
        run: |
          ./searchcode discover \
            --config searchcode.yaml \
            --provider github \
            --org acme \
            --skip-archived \
            --skip-forks

      - name: Validate the result
        run: ./searchcode config validate --config searchcode.yaml || true

      - name: Open a pull request
        uses: peter-evans/create-pull-request@v6
        with:
          branch: searchcode/discover
          title: "searchcode: repositories discovered on ${{ github.event.repository.updated_at }}"
          commit-message: "searchcode discover: refresh conf.d/repositories.yaml"
          body: |
            Generated by `searchcode discover`. Every entry marked UNCLASSIFIED
            needs a group before this can merge — `searchcode config validate`
            fails until each one has one.
          labels: searchcode

Two things about that job.

config validate runs with || true in the discovery job only, because a newly discovered repository is unclassified by definition and the point of the pull request is to get it classified. Run searchcode config validate without the || true as a required check on the pull request itself:

# .github/workflows/searchcode-validate.yml
name: searchcode config validate

on: [pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          curl -fsSL -o searchcode https://searchcode.com/download/searchcode-linux-amd64
          chmod +x searchcode
          ./searchcode config validate --config searchcode.yaml

That is the gate. It is a plain exit code, so any CI system enforces it.

The DISCOVERY_TOKEN secret is a fine-grained personal access token or a GitHub App installation token with Metadata: read-only on the organisation. Nothing else.

GitLab CI

# .gitlab-ci.yml
stages: [discover, validate]

searchcode:discover:
  stage: discover
  image: alpine:3.20
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
  variables:
    # Set SEARCHCODE_DISCOVER_TOKEN as a masked, protected CI/CD variable.
    # A read_user / read-only-scope token is enough.
    GIT_STRATEGY: clone
  before_script:
    - apk add --no-cache curl git
    - curl -fsSL -o /usr/local/bin/searchcode https://searchcode.com/download/searchcode-linux-amd64
    - chmod +x /usr/local/bin/searchcode
  script:
    - |
      searchcode discover \
        --config searchcode.yaml \
        --provider gitlab \
        --org acme \
        --include-subgroups \
        --skip-archived
    - git config user.email "searchcode-bot@acme.example"
    - git config user.name  "searchcode discover"
    - git checkout -B searchcode/discover
    - git add conf.d/repositories.yaml
    - |
      if git diff --cached --quiet; then
        echo "no change"; exit 0
      fi
    - git commit -m "searchcode discover: refresh conf.d/repositories.yaml"
    - |
      git push \
        -o merge_request.create \
        -o merge_request.target=$CI_DEFAULT_BRANCH \
        -o merge_request.title="searchcode: discovered repositories" \
        -o merge_request.remove_source_branch \
        "https://oauth2:$CI_PUSH_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git" \
        HEAD:searchcode/discover

searchcode:validate:
  stage: validate
  image: alpine:3.20
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  before_script:
    - apk add --no-cache curl
    - curl -fsSL -o /usr/local/bin/searchcode https://searchcode.com/download/searchcode-linux-amd64
    - chmod +x /usr/local/bin/searchcode
  script:
    - searchcode config validate --config searchcode.yaml

CI_PUSH_TOKEN is a project access token with write_repository. It is separate from SEARCHCODE_DISCOVER_TOKEN on purpose: one can push to your config repository and the other can list project names, and neither needs the other's rights.

Set the schedule under CI/CD → Schedules; the rules: block means the discovery job runs only from a schedule and never on an ordinary push.

Flags

Flag	Meaning
`--provider github\|gitlab`	Which host. Defaults to `discovery.provider` in the config file.
`--org NAME`	The GitHub organisation, or the GitLab group path (`acme` or `acme/platform`).
`--base-url URL`	GHES or self-managed GitLab. Defaults to `discovery.baseUrl`. `https://ghe.example.com` and `https://ghe.example.com/api/v3` both work.
`--token-file PATH`	File holding the token.
`--token-env VAR`	Environment variable holding it. Default `SEARCHCODE_DISCOVER_TOKEN`.
`--out PATH`	Where to write. Default `conf.d/repositories.yaml` beside the config file.
`--include GLOB`	Keep only repositories whose `owner/name` matches. Repeatable; the globs union.
`--exclude GLOB`	Drop repositories whose `owner/name` matches. Repeatable, and applied after `--include`, so it always wins.
`--skip-archived`	Skip archived repositories.
`--skip-forks`	Skip forks.
`--include-subgroups`	GitLab: enumerate nested subgroups. Default true.

Globs are path globs: * matches within one path segment and ** crosses them. acme/* matches acme/billing; acme/** also matches acme/platform/gateway, which is what a GitLab subgroup estate needs.

Filters are flags on a run, never configuration the server reads — the header comment in the generated file records which ones were used, so a reviewer can see why a repository is missing.

Self-managed hosts

GitHub Enterprise Server serves its API under /api/v3; --base-url https://ghe.example.com is enough and the prefix is added for you. Self-managed GitLab serves /api/v4 at the host root, so --base-url https://git.example.com is enough there too.

A rate limit is waited out rather than failing the run — including GitHub's 403-with-X-RateLimit-Remaining: 0 shape, which is what a large organisation actually meets. A host asking for a delay longer than a minute makes the run stop and report how far it got instead of holding a runner, and it is never retried into, because hammering a secondary limit is what extends it.

A partial enumeration writes nothing. Whenever the run could not see everything it asked for — a rate limit part-way through, or an organisation that enforces SAML SSO and answered this token with a filtered list — it prints what it could not see and exits non-zero without touching the fragment. The reason is worth stating: the generated file is the estate, so a fragment written from a partial answer would delete every repository the run could not see, and the pull request would carry that straight to a merge.

What the server does with all this

Nothing, until you merge it. Then it clones over git and indexes. Repository access is git itself, image pulls are your container runtime, authentication is your proxy. Discovery is this page — a CLI, run by you, and the server makes no code-host call on its behalf.

If you use this page and nothing else, the searchcode process makes no outbound connection at all. Three things can change that and each is configuration you write: a code-host connection, OTLP tracing to a collector you name, and a remote syslog audit sink. All three are absent or off in a default install.

The other way to define an estate

A connection is the same enumeration held by the server instead of by you: configure it once and the repository list stays current without a scheduled job. The trade is that the server then makes the outbound connection this page's CLI makes, to one host you declare — which connections.egress: internal-only can restrict to a code host inside your own network.

Keep using this page when you want repository membership reviewed in a pull request. Use a connection when the alternative is somebody remembering to re-run a job.

Enumerating a code host with searchcode discover