Testing before you submit

A step-by-step test plan for the Aceversity API, OAuth server and MCP connector — automated checks, end-to-end scripts, MCP Inspector, Claude/ChatGPT/n8n trials and a security pass — to finish before submitting to any connector directory.

Work through the stages in order. Each one catches problems the next would make expensive. Record results in the sign-off checklist at the bottom.

0. Set up test accounts and an environment

You need, in the same Supabase project the API points at:

Account Purpose
Member A (active trial or subscription, or an instructor) Main test account — also the one you give directory reviewers
Member B (active) Cross-account isolation tests
Lapsed (no active subscription) membership_required behaviour

Give Member A realistic data: 3–4 courses with syllabi, a dozen dated items over the next month, some grades and practice history, a flashcard deck. Reviewers judge the connector by what it returns.

Environments:

Apply the database migration (supabase/migrations/0072_public_api.sql in the Aceversity repo, pnpm db:push) to the project before any of this.

1. Automated checks (every change)

From api/:

pnpm typecheck      # TypeScript
pnpm test           # unit tests: OpenAPI validity, MCP protocol, routing, OAuth helpers, grade math
pnpm openapi:lint   # writes openapi.json and lints it with Redocly

All three must pass with zero errors. The tests also enforce connector-platform limits: descriptions ≤ 300 characters (ChatGPT Actions), tool names ≤ 64 characters, every tool annotated, no $ref in MCP schemas.

2. End-to-end smoke test (every deploy)

Create a read-and-write key for Member A at /keys, then:

ACEVERSITY_API_KEY=ace_sk_… pnpm smoke http://localhost:3300
ACEVERSITY_API_KEY=ace_sk_… pnpm smoke https://<preview>.vercel.app
ACEVERSITY_API_KEY=ace_sk_… pnpm smoke https://api.aceversity.com --read-only   # production: no writes

It checks (about 40 assertions): discovery documents and their URLs, CORS, 401/404/405/422 shapes, every read endpoint, pagination, a full write round trip (course → item → move → milestone → grade categories → score → flashcard deck → delete course cascades), and MCP initialize, notifications, tools/list, tools/call and tool errors. Exit code is non-zero on any failure, so you can run it in CI after each deploy.

Repeat once with a read-only key: the script then verifies writes return 403 insufficient_scope.

3. MCP Inspector

npx @modelcontextprotocol/inspector
  1. Transport Streamable HTTP, URL http://localhost:3300/mcp (or the preview URL).
  2. OAuth path: Open Auth Settings → Quick OAuth Flow. Confirm each step goes green: metadata discovery, client registration, authorization (you sign in and approve), token request.
  3. Connect → server info shows aceversity and the instructions text.
  4. Tools → List Tools: 25 tools, each with a title and annotations.
  5. Run and verify:
    • get_me — correct email, membership.active: true
    • get_agenda {days: 14} — matches the app's calendar
    • get_course {course_id, include_syllabus: true} — syllabus text present
    • get_grades — current grade matches the app's Progress page
    • get_practice_stats — weak topics match the app
    • create_item → appears in the app; delete_item removes it
    • get_course {course_id: "nope"} — isError: true, readable message
  6. Disconnect, reconnect with header Authorization: Bearer ace_sk_… instead — tools still work.

4. OAuth end to end

pnpm oauth:e2e http://localhost:3300

It discovers the server from the MCP 401, registers a client, opens the consent page (approve it), exchanges the code, calls REST and MCP with the token, refreshes (checking the old refresh token is rejected), revokes, and confirms the token is dead. Then do these by hand in a browser:

5. Test in Claude before you deploy

Claude must reach the server over public HTTPS, so expose your local server with a tunnel:

cloudflared tunnel --url http://localhost:3300        # or: ngrok http 3300
# → https://random-words.trycloudflare.com

Set API_PUBLIC_URL=https://random-words.trycloudflare.com in api/.env.local and restart pnpm dev (the issuer must match the public URL, or OAuth fails). Then:

  1. claude.ai → Settings → Connectors → Add custom connector → https://random-words.trycloudflare.com/mcp.
  2. Connect, sign in as Member A, approve.
  3. Run this script in a new chat and note which tools Claude calls:
Prompt Expect
"What's due in the next two weeks?" get_agenda; answer matches the app
"What's my current grade in ?" list_courses → get_grades
"Which topics should I review?" get_practice_stats
"What does my syllabus say about late work?" get_course with include_syllabus
"Move to ." update_item; app shows new date
"Add a 10-term flashcard deck for ." create_vocab_deck; deck appears under Practice
"Delete the deck you just made." Claude asks for confirmation (destructive) → delete_vocab_deck
"Show me Member B's courses" (with B's course id) not_found — no data leaks
  1. Repeat the first prompts in Claude Desktop and the mobile app (same account).
  2. Claude Code: claude mcp add --transport http aceversity <tunnel>/mcp, /mcp → Authenticate, ask "what's due this week?".

Watch the server logs (at: "mcp" lines) while testing; every call should finish in well under 5 seconds.

6. Test in ChatGPT and n8n

7. Security pass

Check How Expected
Cross-account isolation With Member A's key, GET /v1/courses/<Member B course id>, PATCH /v1/items/<B item>, DELETE /v1/grade-scores/<B score> 404 every time; B's data unchanged
Lapsed membership Key for the lapsed account → GET /v1/me then GET /v1/courses 200 then 403 membership_required
Revoked / expired key Revoke at /keys, retry 401
Rate limit for i in $(seq 1 130); do curl -s -o /dev/null -w "%{http_code}\n" https://api.aceversity.com/v1/me -H "Authorization: Bearer $K"; done | sort | uniq -c ~120 × 200, rest 429 with Retry-After
No secrets at rest In Supabase, inspect api_keys, oauth_tokens, oauth_codes Only SHA-256 hashes
Clickjacking curl -I https://api.aceversity.com/oauth/authorize X-Frame-Options: DENY, frame-ancestors 'none'
Transport curl -I http://api… Redirects to HTTPS; Strict-Transport-Security present
Error hygiene Trigger a 500 (e.g. temporarily bad DB key on preview) Generic message + request_id; no stack traces or SQL
Dependency audit pnpm audit --prod in api/ No high/critical issues

8. Reliability

9. Documentation & policy

10. Sign-off checklist

Stage Local Preview Production
1. typecheck / test / openapi:lint ☐ — —
2. smoke (read-write) ☐ ☐ read-only ☐
2. smoke (read-only key) ☐ ☐ —
3. MCP Inspector (OAuth + key) ☐ ☐ ☐
4. oauth:e2e + manual negatives ☐ ☐ ☐
5. Claude web / desktop / mobile / Code tunnel ☐ ☐ ☐
6. ChatGPT / n8n — ☐ ☐
7. Security pass — ☐ ☐
8. Monitoring live — — ☐
9. Docs & policy — — ☐

Only when every production box is ticked, move on to the submission checklists.