Skip to content

M7 Cutover Runsheet — Operator Execution Document

What this is. The fill-in-as-you-go execution document for cutover day. The cutover checklist is the reference textbook; this is the surgical checklist. Print it (or copy it into the runbook log), fill in every blank, initial every step, and staple it to the sign-off.

What this is not. Engineering verification. The engineering preflight was re-run 2026-07-05 and recorded in the checklist's Preflight record section — all local Go/No-Go rows green at head 9ca626280828. If more than ~2 weeks or any migration has passed since the last preflight, re-run it before scheduling cutover day (any engineer: checklist §1 rows 3, 5–8, 10, 14, 15, 17, 18).


0. Header — fill in before starting

Field Value
Study code ____
Study UUID ____
REDCap project ID (production) ____
REDCap project ID (staging) ____
Production API base URL ____
Staging API base URL ____
Cutover date/time (start) ____
Coordinator (executes) ____
Study admin (approves) ____
Engineer on standby (unblocks) ____
DB admin contact (PITR/rollback) ____

Abort criteria — read aloud before starting. Stop the cutover and page the engineer if ANY of the following occurs. Do not improvise past a stop.

  • Any Go/No-Go row is ❌ (Part B).
  • The audit-chain verifier reports valid: False on the production DB — this is a tamper signal, escalate to security, not just engineering.
  • The live DET (Part D) does not produce a processed webhook event within ~30s and the cause is not identified within 15 minutes.
  • Any smoke output contains a magic-link token or plaintext secret.
  • The token-migration dry-run reports a rotation needed on production (row B3).

Rollback options per action are in checklist §10. A full point-in-time restore is a DB-admin action — do not attempt it from this runsheet.


Part A — Production environment verification (~15 min)

Run on the production infrastructure. Engineer may drive; coordinator records.

# Action Expected Done (initials/time)
A1 On EVERY app + worker container: printenv ENVIRONMENT production on all ______
A2 curl -s <prod-api>/api/health 200; environment field reports production ______
A3 printenv DEV_MODE REDCAP_ENCRYPTION_ALLOW_PLAINTEXT_READS ENABLE_API_DOCS all empty or false ______
A4 Confirm REDCAP_ENCRYPTION_KEY set and ≠ JWT_SECRET_KEY; SESSION_STORAGE_BACKEND=redis; SMTP_HOST set all true ______
A5 curl -sI <prod-api>/docs and /redoc and /openapi.json all 404 ______
A6 On the production DB: alembic current vs alembic heads identical revision ______
A7 Privilege separation (checklist row 18): from an APP connection run ALTER TABLE app.audit_logs DISABLE TRIGGER ALL; fails with insufficient privilege; SELECT current_user shows the app role, not the owner ______
A8 Audit chain over the production DB (checklist §3.3 snippet) valid: True, rows > 0 ______
A9 Token at-rest check: checklist §4.1 SQL api_v2 = t, secret_v2 = t for the sponsor study ______

Part B — Go/No-Go matrix walk (~10 min)

Walk checklist §1 top to bottom. Rows 3, 5–8, 10, 14, 15, 17, 18 are covered by the recorded engineering preflight if it is current (see header note); re-initial them from that record. Rows 1–2, 4, 9, 11–13, 16 are live checks:

Row Check (abbrev.) Done (initials)
1–2 Environment + encryption key (= A1–A4) ______
4 Webhook secret encrypted (= A9) ______
9 DET signature: wrong signature → 401, correct → 200 (verified in Part D) ______
11 Per-study admin exists (checklist §5.2 SQL; ≥1 row, ideally 2) ______
12 Coordinator dashboard "Failed Sync" tile renders (with synthetic failure) ______
13 Patient-portal stale-data flag (/api/portal/data-status) ______
16 API docs disabled (= A5) ______

Go/No-Go decision: all 18 rows ✅? → proceed. Any ❌ → abort, record the row number and finding: __________

Part C — Staging smokes (~20 min)

# Action Expected Done
C1 python scripts/phase1_staging_smoke.py --api-base-url <staging-api> --app-origin <staging-app> --expect-docs-disabled --expect-public-participant-routes-disabled --portal-email <real staging account> every line [PASS]; no magic-link token in any response body ______
C2 Study config cross-check (checklist §5.1 SQL): integration_mode='redcap', explicit consent_mode as configured ______
C3 Sites mapped for every REDCap DAG (checklist §5.4 SQL) one Site row per DAG ______
C4 Anchor-date policy set (checklist §5.3): sources ordered, flag_for_review on completed visits as configured ______

Part D — Live DET end-to-end on staging (~20 min)

Checklist §6.5–6.7. Use a real-but-non-PHI test record.

# Action Expected Done
D1 Fire a DET with a WRONG signature 401 ______
D2 Save a record in staging REDCap (correct secret) triggering the enrollment instrument within ~30s: webhook_events row status='processed' (checklist §6.5 SQL) ______
D3 Participant created with enrollment_date set (§6.5 SQL #2) one row, correct code ______
D4 ScheduleVersion v1 active (§6.5 SQL #3) version_number=1, is_current=true ______
D5 Re-fire the SAME payload second row status='duplicate', duplicate_of_id set; NO second participant/schedule ______
D6 Failure path: break the staging REDCap URL, submit a session sync_status='failed'; CRITICAL log with fallback_used: false; portal has_failed_sync: true ______
D7 Keep failing ≥5 times breaker opens (error_type='circuit_open'), REDCap not called ______
D8 Restore URL, wait cooldown (default 60s), submit sync succeeds, breaker closes, stale banner clears ______

Part E — Production cutover sequence (~30 min)

Checklist §7, in order. Each step assumes the previous succeeded.

# Step Time done Initials
E1 Freeze writes on the staging study ______ ______
E2 alembic upgrade head on production; re-verify A6 ______ ______
E3 Production env vars set (Part A verified); server + Celery workers + beat restarted ______ ______
E4 Go/No-Go matrix re-confirmed green (Part B) ______ ______
E5 Production REDCap project configured: token + webhook secret via portal (never raw SQL), DET URL → production ______ ______
E6 POST /api/studies/{id}/redcap/sync-eventsVisitWindow rows created ______ ______
E7 Per-study admin provisioned (≥1, ideally 2) ______ ______
E8 ONE production DET smoke with a non-PHI test record; verify the D2–D4 trail on production; then withdraw the test participant (status='withdrawn' + audit note — do NOT delete) ______ ______
E9 Study status → active ______ ______
E10 Coordinator notified; enrolment open ______ ______

Part F — Post-cutover handoff (first hour + first week)

  • [ ] First hour: re-run A8 (audit chain over production — now with cutover rows in it); confirm valid: True.
  • [ ] First hour: SELECT * FROM app.webhook_events WHERE status='failed' — expect zero rows.
  • [ ] Monitoring owner for days 1–7 assigned: ____ (watch the checklist §8 signal table daily).
  • [ ] Failure-mode runbook (checklist §9) bookmarked by the coordinator.

Sign-off (checklist §11)

All parts A–F complete, no unresolved aborts.

Role Name Signature/initials Date/time
Engineer on standby __ __ __
Study admin __ __ __

Record the sign-off durably — either in the project runbook log (attach this completed runsheet) or as an audit-log entry:

# from a server shell (production), one-time:
from app.db.models import AuditLog
db.add(AuditLog(
    user_id=<study admin user id>,
    study_id=<study uuid>,
    action="cutover_signoff",
    resource_type="study",
    resource_id=<study uuid>,
    audit_metadata={
        "runsheet": "docs/guides/m7-cutover-runsheet.md",
        "engineer": "<name>",
        "study_admin": "<name>",
        "preflight_date": "2026-07-05",
    },
))
await db.commit()

The audit-log route is preferred: it lands inside the tamper-evident hash chain, so the sign-off itself becomes part of the record it certifies.

After sign-off: M7 ships. Subsequent sponsor studies reuse this runsheet; if a step must change for a different sponsor, change the runsheet (and the checklist it references), not the per-study process.