M7 Cutover Runsheet — Operator Execution Document¶
What this is. The fill-in-as-you-go execution document for cutover day. The cutover checklist is the reference textbook; this is the surgical checklist. Print it (or copy it into the runbook log), fill in every blank, initial every step, and staple it to the sign-off.
What this is not. Engineering verification. The engineering preflight was
re-run 2026-07-05 and recorded in the checklist's Preflight record section —
all local Go/No-Go rows green at head 9ca626280828. If more than ~2 weeks or
any migration has passed since the last preflight, re-run it before scheduling
cutover day (any engineer: checklist §1 rows 3, 5–8, 10, 14, 15, 17, 18).
0. Header — fill in before starting¶
| Field | Value |
|---|---|
| Study code | ____ |
| Study UUID | ____ |
| REDCap project ID (production) | ____ |
| REDCap project ID (staging) | ____ |
| Production API base URL | ____ |
| Staging API base URL | ____ |
| Cutover date/time (start) | ____ |
| Coordinator (executes) | ____ |
| Study admin (approves) | ____ |
| Engineer on standby (unblocks) | ____ |
| DB admin contact (PITR/rollback) | ____ |
Abort criteria — read aloud before starting. Stop the cutover and page the engineer if ANY of the following occurs. Do not improvise past a stop.
- Any Go/No-Go row is ❌ (Part B).
- The audit-chain verifier reports
valid: Falseon the production DB — this is a tamper signal, escalate to security, not just engineering. - The live DET (Part D) does not produce a
processedwebhook event within ~30s and the cause is not identified within 15 minutes. - Any smoke output contains a magic-link token or plaintext secret.
- The token-migration dry-run reports a rotation needed on production (row B3).
Rollback options per action are in checklist §10. A full point-in-time restore is a DB-admin action — do not attempt it from this runsheet.
Part A — Production environment verification (~15 min)¶
Run on the production infrastructure. Engineer may drive; coordinator records.
| # | Action | Expected | Done (initials/time) |
|---|---|---|---|
| A1 | On EVERY app + worker container: printenv ENVIRONMENT |
production on all |
______ |
| A2 | curl -s <prod-api>/api/health |
200; environment field reports production | ______ |
| A3 | printenv DEV_MODE REDCAP_ENCRYPTION_ALLOW_PLAINTEXT_READS ENABLE_API_DOCS |
all empty or false |
______ |
| A4 | Confirm REDCAP_ENCRYPTION_KEY set and ≠ JWT_SECRET_KEY; SESSION_STORAGE_BACKEND=redis; SMTP_HOST set |
all true | ______ |
| A5 | curl -sI <prod-api>/docs and /redoc and /openapi.json |
all 404 | ______ |
| A6 | On the production DB: alembic current vs alembic heads |
identical revision | ______ |
| A7 | Privilege separation (checklist row 18): from an APP connection run ALTER TABLE app.audit_logs DISABLE TRIGGER ALL; |
fails with insufficient privilege; SELECT current_user shows the app role, not the owner |
______ |
| A8 | Audit chain over the production DB (checklist §3.3 snippet) | valid: True, rows > 0 |
______ |
| A9 | Token at-rest check: checklist §4.1 SQL | api_v2 = t, secret_v2 = t for the sponsor study |
______ |
Part B — Go/No-Go matrix walk (~10 min)¶
Walk checklist §1 top to bottom. Rows 3, 5–8, 10, 14, 15, 17, 18 are covered by the recorded engineering preflight if it is current (see header note); re-initial them from that record. Rows 1–2, 4, 9, 11–13, 16 are live checks:
| Row | Check (abbrev.) | Done (initials) |
|---|---|---|
| 1–2 | Environment + encryption key (= A1–A4) | ______ |
| 4 | Webhook secret encrypted (= A9) | ______ |
| 9 | DET signature: wrong signature → 401, correct → 200 (verified in Part D) | ______ |
| 11 | Per-study admin exists (checklist §5.2 SQL; ≥1 row, ideally 2) | ______ |
| 12 | Coordinator dashboard "Failed Sync" tile renders (with synthetic failure) | ______ |
| 13 | Patient-portal stale-data flag (/api/portal/data-status) |
______ |
| 16 | API docs disabled (= A5) | ______ |
Go/No-Go decision: all 18 rows ✅? → proceed. Any ❌ → abort, record the row number and finding: __________
Part C — Staging smokes (~20 min)¶
| # | Action | Expected | Done |
|---|---|---|---|
| C1 | python scripts/phase1_staging_smoke.py --api-base-url <staging-api> --app-origin <staging-app> --expect-docs-disabled --expect-public-participant-routes-disabled --portal-email <real staging account> |
every line [PASS]; no magic-link token in any response body |
______ |
| C2 | Study config cross-check (checklist §5.1 SQL): integration_mode='redcap', explicit consent_mode |
as configured | ______ |
| C3 | Sites mapped for every REDCap DAG (checklist §5.4 SQL) | one Site row per DAG | ______ |
| C4 | Anchor-date policy set (checklist §5.3): sources ordered, flag_for_review on completed visits |
as configured | ______ |
Part D — Live DET end-to-end on staging (~20 min)¶
Checklist §6.5–6.7. Use a real-but-non-PHI test record.
| # | Action | Expected | Done |
|---|---|---|---|
| D1 | Fire a DET with a WRONG signature | 401 | ______ |
| D2 | Save a record in staging REDCap (correct secret) triggering the enrollment instrument | within ~30s: webhook_events row status='processed' (checklist §6.5 SQL) |
______ |
| D3 | Participant created with enrollment_date set (§6.5 SQL #2) |
one row, correct code | ______ |
| D4 | ScheduleVersion v1 active (§6.5 SQL #3) |
version_number=1, is_current=true |
______ |
| D5 | Re-fire the SAME payload | second row status='duplicate', duplicate_of_id set; NO second participant/schedule |
______ |
| D6 | Failure path: break the staging REDCap URL, submit a session | sync_status='failed'; CRITICAL log with fallback_used: false; portal has_failed_sync: true |
______ |
| D7 | Keep failing ≥5 times | breaker opens (error_type='circuit_open'), REDCap not called |
______ |
| D8 | Restore URL, wait cooldown (default 60s), submit | sync succeeds, breaker closes, stale banner clears | ______ |
Part E — Production cutover sequence (~30 min)¶
Checklist §7, in order. Each step assumes the previous succeeded.
| # | Step | Time done | Initials |
|---|---|---|---|
| E1 | Freeze writes on the staging study | ______ | ______ |
| E2 | alembic upgrade head on production; re-verify A6 |
______ | ______ |
| E3 | Production env vars set (Part A verified); server + Celery workers + beat restarted | ______ | ______ |
| E4 | Go/No-Go matrix re-confirmed green (Part B) | ______ | ______ |
| E5 | Production REDCap project configured: token + webhook secret via portal (never raw SQL), DET URL → production | ______ | ______ |
| E6 | POST /api/studies/{id}/redcap/sync-events → VisitWindow rows created |
______ | ______ |
| E7 | Per-study admin provisioned (≥1, ideally 2) | ______ | ______ |
| E8 | ONE production DET smoke with a non-PHI test record; verify the D2–D4 trail on production; then withdraw the test participant (status='withdrawn' + audit note — do NOT delete) |
______ | ______ |
| E9 | Study status → active |
______ | ______ |
| E10 | Coordinator notified; enrolment open | ______ | ______ |
Part F — Post-cutover handoff (first hour + first week)¶
- [ ] First hour: re-run A8 (audit chain over production — now with cutover rows in it); confirm
valid: True. - [ ] First hour:
SELECT * FROM app.webhook_events WHERE status='failed'— expect zero rows. - [ ] Monitoring owner for days 1–7 assigned: ____ (watch the checklist §8 signal table daily).
- [ ] Failure-mode runbook (checklist §9) bookmarked by the coordinator.
Sign-off (checklist §11)¶
All parts A–F complete, no unresolved aborts.
| Role | Name | Signature/initials | Date/time |
|---|---|---|---|
| Engineer on standby | __ | __ | __ |
| Study admin | __ | __ | __ |
Record the sign-off durably — either in the project runbook log (attach this completed runsheet) or as an audit-log entry:
# from a server shell (production), one-time:
from app.db.models import AuditLog
db.add(AuditLog(
user_id=<study admin user id>,
study_id=<study uuid>,
action="cutover_signoff",
resource_type="study",
resource_id=<study uuid>,
audit_metadata={
"runsheet": "docs/guides/m7-cutover-runsheet.md",
"engineer": "<name>",
"study_admin": "<name>",
"preflight_date": "2026-07-05",
},
))
await db.commit()
The audit-log route is preferred: it lands inside the tamper-evident hash chain, so the sign-off itself becomes part of the record it certifies.
After sign-off: M7 ships. Subsequent sponsor studies reuse this runsheet; if a step must change for a different sponsor, change the runsheet (and the checklist it references), not the per-study process.