QAmcp GitHub

Test your MCP server on every change

Add one file to your GitHub repository. Every pull request gets a comment with the score, and the build fails if the server got worse, before your users' AI apps notice.

Sheet 2 of 6 · Set it up

One file, three steps

Create the fileIn your repository, add .github/workflows/qamcp.yml with the contents below. Change the url to your server's link.
Commit itThat's the whole setup. GitHub runs it for free on public repositories.
Open a pull requestQAmcp tests the server and comments the score. If the score is under 80, the check turns red. Later runs update that same comment instead of adding new ones.

name: QAmcp
on: [pull_request]
permissions:
  contents: read
  pull-requests: write   # to comment the score
jobs:
  qamcp:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: QAmcp/qamcp-cli@v0
        with:
          url: https://your-server.com/mcp
          fail-under: 80

Sheet 3 of 6 · A server that needs a key

Keep the key in GitHub's secrets

Make a test keyCreate a key for your server that can only read, just for testing.
Add it as a secretRepository → Settings → Secrets and variables → Actions → New repository secret. Name it MCP_TEST_TOKEN.
Send it as a headerAdd these lines under with:. The key goes only to your server, and GitHub hides it in the logs.

          headers: |
            Authorization: Bearer ${{ secrets.MCP_TEST_TOKEN }}

Is your server a program instead of a link? Use stdio: npx -y your-mcp-server in place of url.

Sheet 4 of 6 · Catch what changed

Compare every run with the last one

With a baseline, QAmcp remembers the last report and fails the build if the score dropped by more than 5 points, a check started failing, or a tool was removed. Tools that changed are listed in the report and the tools-changed output, so you can decide what to do. That's how you catch a broken release before your customers do.

      # keep the last report between runs
      - uses: actions/cache@v6
        with:
          path: qamcp-baseline.json
          key: qamcp-${{ github.ref_name }}-${{ github.sha }}
          restore-keys: |
            qamcp-${{ github.ref_name }}-
            qamcp-main-
      - uses: QAmcp/qamcp-cli@v0
        with:
          url: https://your-server.com/mcp
          fail-under: 80
          baseline: qamcp-baseline.json
          save-baseline: qamcp-baseline.json

The first run has no baseline yet, and that's fine: it just saves one. The full example, with uploading the reports as files, is in examples/qamcp-ci.yml.

Sheet 5 of 6 · Any other CI

GitLab, CircleCI, Jenkins or your own script

Run the same command anywhere Node.js 20 or newer is installed. The exit code tells your CI what happened.

npx qamcp-cli https://your-server.com/mcp --fail-under 80 --baseline base.json --save-baseline base.json --junit report.xml

Exit code	Means
0	Fine: every rule you set passed
1	A rule you set failed: score too low, a check failed, or the server got worse than the baseline
2	Wrong options in the command
3	Could not test: the server is down, needs a login, or asked us to slow down

Sheet 6 of 6 · All options

Everything the GitHub check accepts

Option	What it does	Default
`url`	Link to the MCP server to test (use this or `stdio`)	none
`stdio`	A command that starts the server as a program	none
`headers`	Headers to send, one `Name: value` per line. Put keys in secrets.	none
`fail-under`	Fail if the score is below this number	none
`fail-on-fail`	Fail if any check failed	false
`baseline`	A saved report to compare with (a missing file is fine on the first run)	none
`max-drop`	How many points the score may fall below the baseline	5
`save-baseline`	Where to save this run's report as the next baseline	none
`timeout`	Seconds allowed per request	20
`comment`	Comment the score on pull requests	true
`junit`, `markdown`, `json`	Where to write each report file	qamcp-junit.xml, qamcp-report.md, qamcp-report.json
`extra-args`	Any other command-line option, like `--look-only` or `--owner`	none

It also gives the next steps in your workflow three outputs: score, gate (passed, failed or not-tested) and tools-changed.