CI/CD Guide¶

One-stop reference for the six workflow files that govern every code change, release, docs update, and supply-chain check in this repo. Read this when you're debugging a failed run, adding a new job, or preparing a release.

🗺️ At a glance¶

Push / PR to main, weekly cron,                v* tag push
manual dispatch
       │                                     │
       ▼                                     ▼
 ┌───────────────────┐             ┌──────────────────────┐
 │  ⚙️ ci.yml         │             │  🚀 release.yml       │
 │  (merge gate)     │             │  (publish pipeline)  │
 └─────────┬─────────┘             └──────────┬───────────┘
           │                                  │
    change detector (code / packaging flags)
    ┌──────────────────┬──────────────────────┬────────────────────────┐
    │ packaging-only   │ docs/human (no code) │ code change            │
    │ ──────────────   │ ────────────────      │ ──────────────────     │
    │ 📦 validate-pkg  │ 📄 docs-drift        │ 🧹 lint                │
    │                  │ 🐍 python-drift      │ 🧪 test (race+coverage)│
    │                  │ 🧪 test-python       │ 🔬 e2e (STDIO+HTTP+OAuth│
    │                  │ 📦 validate-pkg      │ 🐳 docker-smoke        │
    │                  │                      │ 🏗️ build (5 platforms) │
    │                  │                      │ 🛡️ security (vuln+gosec│
    └──────────────────┴──────────────────────┴────────────────────────┘
                                              🏗️ GoReleaser · Build & Publish
                                          │
                           ┌──────────────┼────────────────────┐
                           ▼              ▼                     ▼
                  🐳 docker-sign   📋 mcp-registry   🐍 pypi
                  📦 packaging     (all parallel)

Two more files run outside this diagram: codeql.yml runs GitHub's CodeQL deep static analysis on push-to-main, PRs to main, and a weekly schedule; provenance.yml and scorecard.yml (below) cover supply-chain security.

📄 Workflow files¶

File	Trigger	Purpose
`.github/workflows/ci.yml`	push to `main` or a `v*` tag, PR to `main`, three weekly crons, `workflow_dispatch`	Gate: prevents broken code from merging
`.github/workflows/release.yml`	push of a `v*` tag, `workflow_dispatch` (manual re-run)	Publish: builds, signs, and ships a release
`.github/workflows/docs.yml`	push to `main` (docs and landing paths only)	Deploy: builds the mkdocs site to GitHub Pages
`.github/workflows/codeql.yml`	push to `main`, PR to `main`, weekly	Security: deep static analysis via GitHub CodeQL
`.github/workflows/provenance.yml`	called by `release.yml` (`workflow_call`)	Supply chain: generates SLSA Level 3 build provenance for release artifacts
`.github/workflows/scorecard.yml`	push to `main`, weekly	Supply chain: runs OpenSSF Scorecard, feeds the README badge

⚙️ ci.yml — The merge gate¶

📡 Triggers¶

Every pull request targeting main
Every push to main or a v* tag
Weekly crons (Mondays, UTC): 07:00 soak test, 08:00 trust/GEO/panel eval, 09:00 multi-pod Redis session continuity
Manual: Actions → ⚙️ CI → Run workflow. Inputs: run_python_live (live SDK tests), run_soak_test (sustained-load soak test), run_trust_eval (trust/GEO/panel eval), run_multipod_session_test (multi-pod Redis session continuity)

🔍 Change detector (`changes` job)¶

The first job classifies every PR. It reads the list of changed files and outputs two flags:

code=true — at least one file outside the harmless/packaging allowlist changed → full CI runs
packaging=true — every changed file is under packaging/** → packaging-only mode (only changes, validate-packaging and test-gate run)

Harmless files (skip heavy CI, not packaging-only): - *.md, docs/**, decks/**, assets/** - LICENSE, .gitignore, mkdocs.yml, overrides/** - .github/**_TEMPLATE*, .github/ISSUE_TEMPLATE/**

Anything else outside packaging/** → code=true → full CI runs.

Why? Branch protection marks skipped required checks as passing. A docs-only PR goes green in seconds without getting stuck on checks that legitimately have nothing to test. A machine-generated packaging PR only runs validate-packaging — the one check that actually matters.

🔒 Always-run jobs (every PR except packaging-only)¶

These run on every PR — except packaging-only ones, which can't cause the drift they detect:

Job	What it catches	Skipped on packaging-only?
🧹 lint	`gofmt -s` + `golangci-lint` — formatting and static analysis	No (gated on `code=true`)
📄 docs-drift	`docs/TOOLS.md` ↔ registry drift; tool annotation coverage	Yes
🐍 python-drift	Python client (`models.py`/`client.py`) not regenerated after Go schema change	Yes
🧪 test-python	Python SDK unit + integration tests (mock HTTP, no binary needed)	Yes
📦 validate-packaging	`PKGBUILD` / `.SRCINFO` / `flake.nix` version + hash consistency	No — with `changes` and `test-gate`, it is one of the three jobs that run on packaging-only PRs

Rule: If you change a Go tool schema, run make gen-python-client before pushing. If you change docs/TOOLS.md, the matching tool definition must change in the same commit (or vice versa). These jobs enforce both.

⚡ Fast-fail ordering in the code path¶

🔍 changes ─► 🧹 lint ─► 🧪 test ─► 🔬 e2e
                      └──────────────► 🐳 docker-smoke
              └──────► 🛡️ security
🔍 changes ─► 🏗️ build   (parallel, independent of test)
🧪 test ─► 🧬 fuzz
🧹 lint ─► 🪟 windows-pipe
🧪 test ─► 🚦 test-gate   (always runs)

lint gates both test and security — a formatting error surfaces before expensive compute starts.
e2e and docker-smoke wait for test — a unit test failure blocks the heavier suites.
build runs in parallel with everything else — cross-compilation failures surface alongside test feedback without waiting.

🧪 Code-only jobs (skipped on docs/packaging PRs)¶

Job	What it runs
🧪 test	`go test -race` — unit + integration tests with race detector
🔬 e2e	Security + lifecycle E2E suite (STDIO, HTTP, OAuth) — network-free
🐳 docker-smoke	Builds the Docker image and drives MCP over HTTP end-to-end
🏗️ build	Cross-compile: linux and darwin on amd64 + arm64, windows on amd64 only
📦 validate-mcpb	`make validate-mcpb`: validates `mcpb/manifest.json` against the mcpb schema and checks bundled assets
🪟 windows-pipe	Runs on `windows-latest` after `lint`: the profile browser starts over DevTools pipes with no TCP debug port. Not a required check
🚦 test-gate	Runs `if: always()` after `test`. Reports under the name branch protection requires, so docs-only PRs pass
🛡️ security	`govulncheck` + `gosec` — vulnerability and security scanning
🧬 fuzz	`make test-fuzz FUZZTIME=20s` — native Go fuzz targets over untrusted-input parsers (`internal/documents`, `internal/content`, `internal/scraper` SSRF checks)

🐍 Manual-dispatch only¶

Job	How to trigger
🐍 python-live-e2e	`Actions → ⚙️ CI → Run workflow` with `run_python_live=true`

python-live-e2e hits real external APIs — too flaky on rate limits for a required gate.

🐢 Weekly schedule + manual-dispatch¶

Job	How it runs
🐢 soak-test	Automatically every Monday at 07:00 UTC (`schedule` trigger), or on-demand via `Actions → ⚙️ CI → Run workflow` with `run_soak_test=true`
🎯 trust-eval	Every Monday at 08:00 UTC, or on-demand with `run_trust_eval=true`. Suites skip cleanly when credentials are absent. See TRUST_EVAL.md
☸️ multipod-redis-session-continuity	Every Monday at 09:00 UTC, or on-demand with `run_multipod_session_test=true`

soak-test (#480) is network-free but runs for an extended window (default 45min on manual dispatch); it drives the real HTTP binary under moderate concurrent load and asserts /metrics shows no unbounded goroutine/heap growth — see make test-soak. Being network-free (no external API rate-limit risk), it's safe to run unattended on a schedule, unlike python-live-e2e.

🚀 release.yml — The publish pipeline¶

📡 Trigger¶

Any tag matching v* pushed to the repo.

git tag v1.38.0
git push origin v1.38.0

🔑 Required secrets and variables¶

Name	Kind	Purpose
`GITHUB_TOKEN`	Auto	Release assets, Docker GHCR, PR creation
`DOCKERHUB_USERNAME` / `DOCKERHUB_TOKEN`	Secret	Docker Hub push
`HOMEBREW_TAP_GITHUB_TOKEN`	Secret	Homebrew tap PR
`SCOOP_BUCKET_GITHUB_TOKEN`	Secret	Scoop bucket PR
`WINGET_PKGS_GITHUB_TOKEN`	Secret	WinGet PR
`CHOCOLATEY_API_KEY`	Secret	Chocolatey push (optional — degrades gracefully)
`MACOS_SIGN_P12` / `MACOS_SIGN_PASSWORD`	Secret	macOS Developer ID signing
`MACOS_NOTARY_ISSUER_ID` / `MACOS_NOTARY_KEY_ID` / `MACOS_NOTARY_KEY`	Secret	Notarization
`AZURE_TENANT_ID` / `AZURE_CLIENT_ID` / `AZURE_CLIENT_SECRET`	Secret	Windows Authenticode (jsign)
`AZURE_SIGNING_ENABLED`	Var	`"true"` to enable Windows signing
`PYPI_PUBLISH_ENABLED`	Var	`"true"` to enable PyPI wheels
`AUR_SSH_KEY`	Secret	SSH private key for AUR push (optional — skips gracefully if absent)
`PACKAGING_UPDATE_ENABLED`	Var	`"true"` to enable AUR/Nix manifest generation + upload
`NIXPKGS_FORK_GITHUB_TOKEN`	Secret	PAT (repo scope) used to push branch to the NixOS/nixpkgs fork and open the PR
`NIXPKGS_ENABLED`	Var	`"true"` to enable the nixpkgs submission job

📦 Job dependency graph¶

[🏗️ GoReleaser · Build & Publish]
     │
     ├──► [🔏 Build Provenance]         (SLSA, calls provenance.yml)
     │
     ├──► [🐳 Sign Docker Images]
     │
     ├──► [📋 Publish → MCP Registry]   (independent of docker-sign)
     │
     ├──► [🐍 Publish → PyPI]           (if PYPI_PUBLISH_ENABLED)
     │
     ├──► [📦 Update Packaging Manifests] (if PACKAGING_UPDATE_ENABLED)
     │         └─ attaches PKGBUILD/.SRCINFO/flake.nix/package.nix to release
     │         └─ pushes to AUR via SSH (if AUR_SSH_KEY set)
     │
     └──► [🐧 Submit → nixpkgs proper]   (if NIXPKGS_ENABLED)

All downstream jobs run in parallel after the core release job completes. A failure in any one job does not block the others.

⚡ Fast-fail ordering inside the core release job¶

Steps run in cheapest-first order so an early misconfiguration surfaces before slow Docker builds start:

0. Resolve tag ref (override for manual dispatch) — resolves RELEASE_TAG for workflow_dispatch runs
1. Set up Go
2. Set up Python + verify Python client drift  ← fast fail
3. Verify macOS signing chain                  ← fast fail
4. Set up Chocolatey CLI (only if key set)     ← fast fail
5. Log in to Docker Hub + GHCR
6. Set up Docker Buildx + QEMU
7. Install cosign + Syft
7a. Clear existing release assets (idempotent re-run) — deletes existing assets so GoReleaser can re-upload on workflow_dispatch retrigger
8. Run GoReleaser  ← the slow step (cross-compile, sign, push)
9. Build + upload .mcpb bundles
10. Generate + attach SBOM
11. Sign checksums.txt with cosign (keyless OIDC)
12. Upload dist binaries artifact (for PyPI)

🔬 Core release steps in detail¶

Step	What it does
Verify Python client drift	Fails fast if `make gen-python-client` was not run before tagging
Verify macOS signing chain	Checks p12 contains leaf + intermediate + Apple Root CA (prevents AMFI SIGKILL on launch)
Set up Chocolatey CLI	Installs `choco` under mono; `push` subcommand is non-fatal (Chocolatey moderation can 403)
Run GoReleaser	Cross-compiles 5 platforms; signs macOS (quill+notarize); signs Windows (jsign/Azure); pushes Docker multi-arch; updates Homebrew/Scoop/WinGet; publishes GitHub Release
Build .mcpb bundles	Assembles MCPB bundle archives
Generate SBOM	Syft produces a full SPDX JSON; attached to the release
Sign checksums.txt	Cosign keyless OIDC signature; `.sig` + `.pem` attached to the release
Upload dist binaries	Artifact for the PyPI job to wrap into wheels without re-running GoReleaser

🐳 [docker-sign] — Sign Docker images¶

Fetches the image digest from GHCR and signs it with cosign using GitHub's OIDC identity. Creates a publicly verifiable Sigstore signature.

📋 [publish-mcp-registry] — MCP Registry¶

Depends on release directly (not docker-sign) — the registry only needs the GitHub Release to exist, not the Docker signature. Re-syncs the version string and publishes via mcp-publisher + GitHub OIDC authentication.

🐍 [publish-pypi] — PyPI platform wheels¶

Downloads the cross-compiled binaries from the release job artifact, wraps them into platform wheels, smoke-tests the manylinux wheel (import check), then publishes via Trusted Publishing (OIDC — no token secret). Gated on vars.PYPI_PUBLISH_ENABLED == 'true'.

📦 [update-packaging] — AUR + Nix publishing¶

Fully automated — no manual steps needed.

1. Run scripts/update-packaging.sh <VERSION>
   └─ Fetches checksums.txt from the new release
   └─ Generates packaging/aur/PKGBUILD         (version + hex SHA256)
   └─ Generates packaging/aur/.SRCINFO         (version + hex SHA256)
   └─ Generates packaging/nix/flake.nix        (version + SRI hashes, 4 platforms)
   └─ Updates packaging/nixpkgs/package.nix    (version string only — hashes via nix)
2. Attach PKGBUILD, .SRCINFO, flake.nix, package.nix to the GitHub Release as assets
3. Push PKGBUILD + .SRCINFO to AUR via SSH (gated on AUR_SSH_KEY secret)

Gated on vars.PACKAGING_UPDATE_ENABLED == 'true'. The AUR push step is additionally gated on the AUR_SSH_KEY secret — absent key means the step skips cleanly, manifests are still attached to the release.

No PR to this repo. Checksums only exist after GoReleaser builds the binaries, so the manifest generation must happen post-release. The generated files are published directly — as release artifacts and to AUR — without needing to land in main. Nix flake users pin to a release tag (github:zoharbabin/web-researcher-mcp/vX.Y.Z) and get the correct flake.nix from that tag's checkout automatically.

🐧 [submit-nixpkgs-pr] — nixpkgs proper¶

Initial submission is automated; future bumps are handled by the nixpkgs-update bot.

This is what gets nix profile install nixpkgs#web-researcher-mcp working for users — no flake, no pinning to this repo. Two modes, chosen automatically per run:

Package not yet in nixpkgs → installs nix, computes src hash and vendorHash (using lib.fakeHash trick — build with empty hash, extract real hash from the error, repeat for vendor), forks NixOS/nixpkgs, pushes the derivation to a stable branch, opens or updates the one PR against NixOS/nixpkgs master.
nixpkgs' go_1_27 older than go.mod's go directive → skipped with a workflow warning (the build runs with GOTOOLCHAIN=local and cannot succeed). Re-run the job once nixpkgs bumps Go.
Package already merged → no-op; the passthru.updateScript = nix-update-script {} in the derivation signals the nixpkgs-update bot to open version-bump PRs automatically on every new release tag — zero maintenance.

The branch name (nixpkgs-web-researcher-mcp-init) is version-independent by design: every pre-merge release pushes to the same branch and reuses the same open PR (checked via gh pr list --head) instead of opening a new PR per release. The bot-authored commit carries an Assisted-by: Claude Code (claude-sonnet-5) trailer and the PR body ticks the automation/AI policy checklist item, per CONTRIBUTING.md#automationai-policy in nixpkgs.

1. Check if pkgs/by-name/we/web-researcher-mcp/package.nix exists in NixOS/nixpkgs
   └─ If yes → exit 0 (nixpkgs-update handles future bumps)
2. Install nix via DeterminateSystems/nix-installer-action
3. Compute real src hash  (nix-build with lib.fakeHash → parse error output)
4. Compute real vendorHash (same trick with src hash known)
5. Patch packaging/nixpkgs/package.nix with both hashes
6. Fork NixOS/nixpkgs (gh repo fork — idempotent)
7. Clone fork, force-push branch nixpkgs-web-researcher-mcp-init, commit (with Assisted-by trailer)
8. Look for an existing open PR from that branch — update its title/body if found,
   else open a new PR: zoharbabin:nixpkgs-web-researcher-mcp-init → NixOS/nixpkgs master

Gated on vars.NIXPKGS_ENABLED == 'true' + secrets.NIXPKGS_FORK_GITHUB_TOKEN.

📚 docs.yml — Docs deploy¶

📡 Trigger¶

Push to main touching any of: - README.md, ARCHITECTURE.md, CONTRIBUTING.md - docs/**, decks/**, overrides/**, landing/** - assets/logo-final-transparent.svg, assets/favicon.ico - mkdocs.yml, .github/workflows/docs.yml

Steps¶

Checkout
Install mkdocs-material from .github/pip/docs-requirements.txt (hash-pinned)
Assemble site_src/ from root + docs/ files (with cross-link rewriting)
mkdocs build -d publish/docs
Assemble the landing page at publish/ root, add redirect stubs and robots.txt
Deploy publish/ to the gh-pages branch via peaceiris/actions-gh-pages (force_orphan)

All docs/*.md links are rewritten from ](docs/FOO.md → ](foo.md so the published site URLs are clean. Root policy files (SECURITY.md, CODE_OF_CONDUCT.md) point to GitHub because they are not published to the site.

🔍 codeql.yml — Deep static analysis¶

📡 Triggers¶

Push to main
Any PR targeting main
No path filter
Weekly schedule: every Monday at 06:00 UTC

Runs GitHub's CodeQL engine with security-extended,security-and-quality query suites. Results appear in the Security → Code scanning tab. Findings must be addressed or dismissed before they accumulate.

🛠️ Adding or changing a workflow¶

Adding a job to ci.yml¶

Runs on every non-packaging PR (e.g., a new drift gate): add needs: changes and if: needs.changes.outputs.packaging != 'true'.
Only relevant when Go code changes: gate it on needs.changes.outputs.code == 'true' and add needs: [changes, lint] (so formatting failures fast-fail first).
Must run even on packaging PRs: omit the packaging guard and add needs: changes only if you need the outputs. Only validate-packaging falls into this category.

Adding a post-release distribution channel¶

Add a new job to release.yml with needs: release.
Gate it on a repo var (e.g., vars.MY_CHANNEL_ENABLED == 'true') so an unconfigured fork is a clean no-op.
Don't swallow the actual failure signal. A bare || echo "::warning::..." reports the job green even when the publish never happened — a bare warning once hid failed publishes across several releases (#673). If a failure is expected to be benign (e.g. "already published"), still capture the command's exit code and its output, and only downgrade to a warning after confirming the specific known-benign case (grep the output for the expected message); anything else must fail the job loudly.
Document the required secret/var in the table above.

Cutting a release¶

# 1. Merge your feature branch to main via PR
# 2. Bump version
echo "1.38.0" > VERSION
bash scripts/sync-version.sh
make sync-lenses
make gen-python-client
git add VERSION server.json .claude-plugin/plugin.json python/ internal/search/lenses_embed/
git commit -m "chore: bump VERSION to 1.38.0"
git push origin HEAD  # via PR

# 3. Tag + push
git tag v1.38.0
git push origin v1.38.0
# → release.yml starts automatically
# → update-packaging job generates PKGBUILD/.SRCINFO/flake.nix and attaches them to the release

🐛 Debugging a failed run¶

Symptom	Where to look
`gofmt` failure	Run `make fmt` locally, push again
`golangci-lint` failure	Run `go tool golangci-lint run` locally
`docs-drift` failure	`docs/TOOLS.md` section out of sync with the Go tool registry — update the matching `## Tool N: \`name``section in`docs/TOOLS.md` (or update the tool definition to match the doc)
`python-drift` failure	Run `make gen-python-client`, stage + commit the regenerated files
`validate-packaging` failure	Version mismatch across PKGBUILD / .SRCINFO / flake.nix — run `scripts/update-packaging.sh <version>`
GoReleaser failure	Check secrets are set; check `.goreleaser.yml` template syntax
Python drift fails during release	Tag pushed before `make gen-python-client` — rebuild on a new patch tag
macOS chain verify fails	Re-export the signing p12 with full chain (leaf + intermediate + Apple Root CA)
`update-packaging` assets not attached	Check `vars.PACKAGING_UPDATE_ENABLED == 'true'`; check job logs
AUR push skipped	Expected when `AUR_SSH_KEY` secret is absent — add the secret to enable
AUR push fails	Verify SSH key is registered on aur.archlinux.org; check package name matches
PyPI publish failure	Check `pypi` environment is configured with Trusted Publishing OIDC
Docker sign failure	Check GHCR image exists; check cosign OIDC token permissions
MCP Registry warning	Non-fatal; check `mcp-publisher` OIDC credentials, may already be published
Release workflow didn't fire on tag push	GitHub infrastructure glitch — use `Actions → 🚀 Release → Run workflow` and supply the tag (e.g. `v1.37.5`) to retrigger manually

🔗 See also¶

CONTRIBUTING.md — commit format, branch naming, PR process
docs/DEPLOYMENT.md — Docker, Kubernetes, env vars
docs/SECURITY_AND_COMPLIANCE.md — security rules that apply to CI changes
.goreleaser.yml — GoReleaser pipeline config (build matrix, signing, publishers)
scripts/update-packaging.sh — packaging update script invoked by the CI job

CI/CD Guide¶

🗺️ At a glance¶

📄 Workflow files¶

⚙️ ci.yml — The merge gate¶

📡 Triggers¶

🔍 Change detector (changes job)¶

🔒 Always-run jobs (every PR except packaging-only)¶

⚡ Fast-fail ordering in the code path¶

🧪 Code-only jobs (skipped on docs/packaging PRs)¶

🐍 Manual-dispatch only¶

🐢 Weekly schedule + manual-dispatch¶

🚀 release.yml — The publish pipeline¶

📡 Trigger¶

🔑 Required secrets and variables¶

📦 Job dependency graph¶

⚡ Fast-fail ordering inside the core release job¶

🔬 Core release steps in detail¶

🐳 [docker-sign] — Sign Docker images¶

📋 [publish-mcp-registry] — MCP Registry¶

🐍 [publish-pypi] — PyPI platform wheels¶

📦 [update-packaging] — AUR + Nix publishing¶

🐧 [submit-nixpkgs-pr] — nixpkgs proper¶

📚 docs.yml — Docs deploy¶

📡 Trigger¶

Steps¶

🔍 codeql.yml — Deep static analysis¶

📡 Triggers¶

🛠️ Adding or changing a workflow¶

Adding a job to ci.yml¶

Adding a post-release distribution channel¶

Cutting a release¶

🐛 Debugging a failed run¶

🔗 See also¶

🔍 Change detector (`changes` job)¶