BenchDB

Workflow guide

Follow one benchmark result through BenchDB.

This page walks one result from your benchmark harness to a CI report. Each step links to the reference page with the details. Your harness still runs the benchmarks. BenchDB takes over once a result file exists.

Start the guide Open quickstart
  1. 01

    Your harness runs the benchmark

    Measure with any tool.

    Go benchmarks, pytest-benchmark, Google Benchmark, and shell scripts all work. Run them on a laptop, a CI runner, or dedicated benchmark machines. BenchDB only needs the result written as JSON.

    Write results to files before you upload them. If the upload fails, you can retry it without running the benchmark again.
    Read the concepts →
  2. 02

    Result metadata

    Record where the number came from.

    A result names the benchmark case in tags and the unit in stats. It also records the run, the repository commit, the machine, and a timestamp. BenchDB puts results with the same case, context, machine, and repository into one series. Keep tag values stable between runs, or each change starts a new series. 

    {
  "tags": {"name": "decode-batch", "rows": "1m"},
  "run_id": "ci-1842-1",
  "github": {
    "repository": "https://github.com/example/project",
    "commit": "8f3c29a"
  },
  "machine_info": {"name": "linux-arm64"},
  "stats": {"unit": "ms", "data": [82.1, 81.7, 82.4]}
}
    Read the result contract →
  3. 03

    Upload

    Upload files with a reporter token.

    Each JSON file holds one result object or an array of them. The CLI reads the token, rejects unknown fields, expands quoted globs, and uploads files in parallel. Set the number of parallel uploads with --jobs. 

    export BENCHDB_SERVER_URL=https://benchdb.example.com
export BENCHDB_TOKEN=<reporter-token>

benchdb results submit "bench-results/*.json" \
  --server "$BENCHDB_SERVER_URL"
    Keep the token out of command lines and payloads. Set BENCHDB_TOKEN in CI. Save the JSON lines the CLI prints; they hold the ID of each stored result.
    Submit your first result →
  4. 04

    Investigate

    Go from a run to the raw values.

    The Runs page shows whether a run uploaded every result. A result page shows the raw values and metadata. The benchmark's trend page charts every machine over time. Compare puts two results side by side.

    Run→Result→Series→Compare
    Learn the investigation workflow →
  5. 05

    CI

    Report on the commit in CI.

    After the upload, benchdb ci report finds the results for the repository and commit and picks a baseline run. Each row is regressed, improved, stable, insufficient, errored, missing_baseline, or not_comparable. A row regresses only when the result is outside the benchmark's recent history, more than 2% from the baseline, and larger than the benchmark's minimum change (30 ms for timings by default), so one noisy pair of results does not fail the build. 

    benchdb ci report \
  --server "$BENCHDB_SERVER_URL" \
  --repository "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY" \
  --commit "$GITHUB_SHA" \
  --format markdown \
  --output benchdb-report.md
    Choose selectors and baselines →
  6. 06

    Operate

    Run one binary and PostgreSQL.

    The benchdb binary serves the API and the dashboard. PostgreSQL stores recent results; the server moves older results to Parquet files on a persistent volume. Back up both together. Each deploy runs benchdb migrate before benchdb serve starts. Mint reporter tokens with benchdb admin tokens create. Monitor /api/ping and /metrics. Schedule alert evaluation with access to the server's archive volume, and delivery as a separate job.

    DeployOne application image, PostgreSQL, and a persistent archive volume
    Migratebenchdb migrate before the server starts
    Observe/api/ping and Prometheus /metrics
    AutomateReporter tokens and scheduled alerts
    Read the operator reference →

Next step

Upload a result with the quickstart.

It builds the CLI, sets a token, and uploads one result you can open in the dashboard.

Open quickstart