Metricis Administrator Guide

This guide covers deployment, configuration, operations, and maintenance of the Metricis platform.

Table of Contents

1. System Requirements
2. Deployment Options
3. Configuration Reference
4. Database Setup
5. Background Workers
6. Notification Providers
7. REDCap Integration
8. Security Configuration
9. Monitoring & Logging
10. Backup & Recovery
11. Troubleshooting

---

System Requirements

Minimum Requirements

Component	Requirement
CPU	2 cores
RAM	4 GB
Storage	20 GB SSD
OS	Ubuntu 22.04+, Debian 12+, or macOS 13+

Recommended Production Setup

Component	Requirement
CPU	4+ cores
RAM	8+ GB
Storage	100+ GB SSD
Database	Dedicated PostgreSQL 15+
Cache	Dedicated Redis 7+

Software Dependencies

• Python 3.11+ - Server runtime
• Node.js 18+ - Client and portal builds
• PostgreSQL 15+ - Primary database
• Redis 7+ - Message broker and cache
• Nginx (recommended) - Reverse proxy

---

Deployment Options

Option 1: Docker Compose (Recommended)

# docker-compose.prod.yml
version: '3.8'

services:
  server:
    build: ./server
    environment:
      - DATABASE_URL=postgresql+asyncpg://postgres:${DB_PASSWORD}@db:5432/metricis
      - REDIS_URL=redis://redis:6379/0
      - JWT_SECRET_KEY=${JWT_SECRET_KEY}
    depends_on:
      - db
      - redis
    restart: unless-stopped

  worker:
    build: ./server
    command: celery -A app.celery_app worker -Q reminders -l info
    environment:
      - DATABASE_URL=postgresql+asyncpg://postgres:${DB_PASSWORD}@db:5432/metricis
      - REDIS_URL=redis://redis:6379/0
    depends_on:
      - db
      - redis
    restart: unless-stopped

  beat:
    build: ./server
    command: celery -A app.celery_app beat -l info
    environment:
      - REDIS_URL=redis://redis:6379/0
    depends_on:
      - redis
    restart: unless-stopped

  db:
    image: postgres:15-alpine
    environment:
      - POSTGRES_DB=metricis
      - POSTGRES_PASSWORD=${DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    volumes:
      - redis_data:/data
    restart: unless-stopped

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - ./client/dist:/usr/share/nginx/html/client
      - ./portal/dist:/usr/share/nginx/html/portal
      - /etc/letsencrypt:/etc/letsencrypt:ro
    depends_on:
      - server
    restart: unless-stopped

volumes:
  postgres_data:
  redis_data:

Deploy with:

# Build and start all services
docker-compose -f docker-compose.prod.yml up -d

# View logs
docker-compose -f docker-compose.prod.yml logs -f

# Apply database migrations
docker-compose -f docker-compose.prod.yml exec server alembic upgrade head

Option 2: Manual Installation

# 1. Install system dependencies
sudo apt update
sudo apt install -y python3.11 python3.11-venv postgresql-15 redis-server nginx

# 2. Create application user
sudo useradd -m -s /bin/bash metricis
sudo su - metricis

# 3. Clone and setup
git clone https://github.com/your-org/metricis.git
cd metricis

# 4. Setup Python environment
cd server
python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# 5. Setup database
sudo -u postgres createdb metricis
sudo -u postgres psql -c "CREATE USER metricis WITH PASSWORD 'secure-password';"
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE metricis TO metricis;"

# 6. Apply migrations
alembic upgrade head

# 7. Build client and portal
cd ../client && npm install && npm run build
cd ../portal && npm install && npm run build

Option 3: Kubernetes

See k8s/ directory for Kubernetes manifests (Deployment, Service, ConfigMap, Secret).

---

Configuration Reference

Environment Variables

Create /etc/metricis/.env or use environment variables:

# ===================
# Core Settings
# ===================
ENVIRONMENT=production
DEBUG=false
API_HOST=0.0.0.0
API_PORT=8000

# ===================
# Database
# ===================
DATABASE_URL=postgresql+asyncpg://user:password@localhost:5432/metricis

# ===================
# Security (REQUIRED in production - no defaults!)
# ===================
# Generate with: python -c "import secrets; print(secrets.token_urlsafe(32))"
JWT_SECRET_KEY=your-256-bit-secret-key        # REQUIRED
SESSION_SECRET_KEY=another-256-bit-secret-key  # REQUIRED

JWT_ALGORITHM=HS256
JWT_ACCESS_TOKEN_EXPIRE_MINUTES=30
JWT_REFRESH_TOKEN_EXPIRE_DAYS=7
SESSION_EXPIRY_HOURS=2

# ===================
# Session Storage
# ===================
# Backend: "memory" (default, dev only) or "redis" (recommended for production)
SESSION_STORAGE_BACKEND=redis

# ===================
# Redis / Celery
# ===================
REDIS_URL=redis://localhost:6379/0
CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/0

# ===================
# Reminders
# ===================
REMINDER_DAYS_BEFORE=1
REMINDER_CHECK_INTERVAL_MINUTES=5

# ===================
# Email (SMTP)
# ===================
SMTP_HOST=smtp.sendgrid.net
SMTP_PORT=587
SMTP_USER=apikey
SMTP_PASSWORD=your-sendgrid-api-key
SMTP_FROM=noreply@yourdomain.com
SMTP_TLS=true

# ===================
# SMS (Twilio)
# ===================
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your-auth-token
TWILIO_FROM_NUMBER=+15551234567

# ===================
# Push Notifications (Firebase)
# ===================
FCM_CREDENTIALS_PATH=/etc/metricis/firebase-credentials.json
FCM_PROJECT_ID=your-firebase-project

# ===================
# REDCap Sites
# ===================
REDCAP_TOKEN_SITE_A=your-32-character-token
REDCAP_TOKEN_SITE_B=another-32-character-token

# ===================
# Rate Limiting
# ===================
RATE_LIMIT_PER_MINUTE=60
AUTH_RATE_LIMIT_PER_MINUTE=10   # Stricter limit for login/register endpoints

# ===================
# CORS
# ===================
ALLOWED_ORIGINS=https://yourdomain.com,https://portal.yourdomain.com

Nginx Configuration

# /etc/nginx/sites-available/metricis
upstream api {
    server 127.0.0.1:8000;
}

server {
    listen 80;
    server_name yourdomain.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;

    # Client app
    location / {
        root /var/www/metricis/client;
        try_files $uri $uri/ /index.html;
    }

    # Portal app
    location /portal {
        alias /var/www/metricis/portal;
        try_files $uri $uri/ /portal/index.html;
    }

    # API proxy
    location /api {
        proxy_pass http://api;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # API documentation
    location /docs {
        proxy_pass http://api/docs;
    }
}

Systemd Services

API Server (/etc/systemd/system/metricis-api.service):

[Unit]
Description=Metricis API Server
After=network.target postgresql.service

[Service]
Type=simple
User=metricis
WorkingDirectory=/opt/metricis/server
Environment="PATH=/opt/metricis/server/.venv/bin"
EnvironmentFile=/etc/metricis/.env
ExecStart=/opt/metricis/server/.venv/bin/uvicorn app.main:app --host 0.0.0.0 --port 8000
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Celery Worker (/etc/systemd/system/metricis-worker.service):

[Unit]
Description=Metricis Celery Worker
After=network.target redis.service

[Service]
Type=simple
User=metricis
WorkingDirectory=/opt/metricis/server
Environment="PATH=/opt/metricis/server/.venv/bin"
EnvironmentFile=/etc/metricis/.env
ExecStart=/opt/metricis/server/.venv/bin/celery -A app.celery_app worker -Q reminders -l info
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Celery Beat (/etc/systemd/system/metricis-beat.service):

[Unit]
Description=Metricis Celery Beat Scheduler
After=network.target redis.service

[Service]
Type=simple
User=metricis
WorkingDirectory=/opt/metricis/server
Environment="PATH=/opt/metricis/server/.venv/bin"
EnvironmentFile=/etc/metricis/.env
ExecStart=/opt/metricis/server/.venv/bin/celery -A app.celery_app beat -l info
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Enable and start services:

sudo systemctl daemon-reload
sudo systemctl enable metricis-api metricis-worker metricis-beat
sudo systemctl start metricis-api metricis-worker metricis-beat

---

Database Setup

Initial Setup

# Create database and user
sudo -u postgres psql << EOF
CREATE DATABASE metricis;
CREATE USER metricis WITH ENCRYPTED PASSWORD 'secure-password';
GRANT ALL PRIVILEGES ON DATABASE metricis TO metricis;
\c metricis
CREATE SCHEMA app;
GRANT ALL ON SCHEMA app TO metricis;
EOF

# Apply migrations
cd /opt/metricis/server
source .venv/bin/activate
alembic upgrade head

Connection Pooling (PgBouncer)

For high-traffic deployments, use PgBouncer:

# /etc/pgbouncer/pgbouncer.ini
[databases]
metricis = host=localhost dbname=metricis

[pgbouncer]
listen_addr = 127.0.0.1
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 20

Update DATABASE_URL to use PgBouncer:

DATABASE_URL=postgresql+asyncpg://metricis:password@localhost:6432/metricis

---

Background Workers

Worker Architecture

┌─────────────┐     ┌─────────┐     ┌──────────────┐
│ Celery Beat │────>│  Redis  │<────│ Celery Worker│
│ (Scheduler) │     │ (Broker)│     │ (Processor)  │
└─────────────┘     └─────────┘     └──────────────┘
                          │
                          v
                    ┌───────────┐
                    │ PostgreSQL│
                    │  (Results)│
                    └───────────┘

Scheduled Tasks

Task	Schedule	Description
process_pending_reminders	Every 5 min	Send visit reminders
update_visit_statuses	Every hour	Update visit states
send_daily_summary	Daily 9 AM UTC	Coordinator summary

Monitoring Workers

# Check worker status
celery -A app.celery_app inspect active

# Check scheduled tasks
celery -A app.celery_app inspect scheduled

# View task results
celery -A app.celery_app inspect stats

Scaling Workers

# Run multiple workers with concurrency
celery -A app.celery_app worker -Q reminders -c 4 -l info

# Run workers on specific queues
celery -A app.celery_app worker -Q reminders,default -l info

---

Notification Providers

Email (SMTP)

Works with any SMTP provider (SendGrid, AWS SES, Mailgun, etc.):

SMTP_HOST=smtp.sendgrid.net
SMTP_PORT=587
SMTP_USER=apikey
SMTP_PASSWORD=SG.xxxxxxxxxxxxxxxxx
SMTP_FROM=noreply@yourdomain.com
SMTP_TLS=true

SMS (Twilio)

1. Create a Twilio account at https://www.twilio.com
2. Get Account SID and Auth Token from console
3. Purchase a phone number
4. Configure environment:

TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your-auth-token
TWILIO_FROM_NUMBER=+15551234567

Push Notifications (Firebase)

1. Create a Firebase project at https://console.firebase.google.com
2. Go to Project Settings > Service Accounts
3. Generate new private key (JSON file)
4. Configure:

FCM_CREDENTIALS_PATH=/etc/metricis/firebase-credentials.json
FCM_PROJECT_ID=your-project-id

Testing Notifications

# Test email
curl -X POST http://localhost:8000/api/test/email \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"recipient": "test@example.com", "subject": "Test", "body": "Test message"}'

# Test SMS
curl -X POST http://localhost:8000/api/test/sms \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"recipient": "+15551234567", "body": "Test message"}'

---

REDCap Integration

Adding a Site

1. Get API Token from REDCap admin
2. Configure in config.py:

SITE_CONFIGS["hospital-a"] = SiteConfig(
    site_id="hospital-a",
    name="Hospital A",
    redcap_url="https://redcap.hospital-a.org/api/",
    token_env_var="REDCAP_TOKEN_HOSPITAL_A",
    event_name="baseline_arm_1",
)

1. Set environment variable:

   REDCAP_TOKEN_HOSPITAL_A=your-32-character-token
   

Field Mapping

Configure in the Portal under Study Settings > REDCap Configuration:

{
  "participant_code": "record_id",
  "simple_rt_mean": "cog_simple_rt",
  "cpt_accuracy": "cog_cpt_acc",
  "nback_dprime": "cog_nback_dp"
}

---

Security Configuration

Startup Validation

In production (ENVIRONMENT=production), the server validates critical dependencies at startup:

1. Database connectivity - Verifies PostgreSQL connection
2. Redis connectivity - Verifies Redis connection (if SESSION_STORAGE_BACKEND=redis)
3. Secret key validation - Ensures JWT_SECRET_KEY and SESSION_SECRET_KEY are set (no defaults in production)

If validation fails, the server will not start and will log the specific error.

Session Storage

For production deployments, use Redis-backed session storage:

SESSION_STORAGE_BACKEND=redis
REDIS_URL=redis://localhost:6379/0

Benefits: - Sessions persist across server restarts - Multi-instance deployments share session state - Automatic TTL-based expiration

SSL/TLS

# Install Certbot
sudo apt install certbot python3-certbot-nginx

# Obtain certificate
sudo certbot --nginx -d yourdomain.com

# Auto-renewal (already configured by certbot)
sudo systemctl status certbot.timer

Firewall

sudo ufw allow 22/tcp    # SSH
sudo ufw allow 80/tcp    # HTTP (redirects to HTTPS)
sudo ufw allow 443/tcp   # HTTPS
sudo ufw enable

Database Security

-- Restrict database access
REVOKE ALL ON DATABASE metricis FROM PUBLIC;
GRANT CONNECT ON DATABASE metricis TO metricis;

-- Use strong password
ALTER USER metricis WITH PASSWORD 'very-secure-password';

-- Enable SSL connections (in postgresql.conf)
ssl = on

---

Monitoring & Logging

Application Logs

# API server logs
journalctl -u metricis-api -f

# Worker logs
journalctl -u metricis-worker -f

# All Metricis logs
journalctl -u 'metricis-*' -f

Structured Logging

The server uses structlog for structured JSON logging (with automatic fallback to standard logging if structlog is not installed).

Features: - Correlation IDs: Every request receives a correlation_id (from X-Correlation-ID header or auto-generated) - Request IDs: Short unique request_id for each request - JSON Output: Log entries are JSON-formatted for easy parsing - Context Binding: Participant ID, session ID, and other context automatically included

Example log output:

{
  "timestamp": "2026-01-16T10:30:00Z",
  "level": "info",
  "event": "request_completed",
  "correlation_id": "abc123-def456",
  "request_id": "f8a2b3c4",
  "path": "/api/submit",
  "method": "POST",
  "status_code": 200,
  "duration_ms": 45.2
}

Response Headers

All API responses include tracing headers: - X-Correlation-ID: Request correlation ID (pass this to support for debugging) - X-Request-ID: Short request identifier

Health Checks

# API health
curl http://localhost:8000/api/health

# Database connectivity
curl http://localhost:8000/api/health/db

# Redis connectivity
curl http://localhost:8000/api/health/redis

---

Backup & Recovery

Database Backup

# Daily backup script
#!/bin/bash
BACKUP_DIR=/var/backups/metricis
DATE=$(date +%Y%m%d_%H%M%S)

pg_dump -U postgres metricis | gzip > $BACKUP_DIR/metricis_$DATE.sql.gz

# Keep only last 30 days
find $BACKUP_DIR -name "*.sql.gz" -mtime +30 -delete

Add to crontab:

0 2 * * * /opt/metricis/scripts/backup.sh

Recovery

# Restore from backup
gunzip -c /var/backups/metricis/metricis_20240101_020000.sql.gz | psql -U postgres metricis

Data Export

# Export all study data
curl -X GET "http://localhost:8000/api/exports/studies/STUDY_ID/full" \
  -H "Authorization: Bearer $TOKEN" \
  -o study_export.json

---

Troubleshooting

Common Issues

API won't start:

# Check logs
journalctl -u metricis-api -n 100

# Verify database connection
psql $DATABASE_URL -c "SELECT 1"

# Check port availability
ss -tlnp | grep 8000

Workers not processing tasks:

# Check Redis connection
redis-cli ping

# Check worker status
celery -A app.celery_app inspect active

# Check for stuck tasks
celery -A app.celery_app purge

Notifications not sending:

# Check notification logs in database
psql $DATABASE_URL -c "SELECT * FROM app.notification_logs ORDER BY created_at DESC LIMIT 10"

# Test provider connectivity
curl -X POST https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/Messages.json \
  -u $TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN

Database connection errors:

# Check PostgreSQL status
systemctl status postgresql

# Check connection limits
psql -U postgres -c "SELECT count(*) FROM pg_stat_activity"

# Restart connection pool
systemctl restart metricis-api

Performance Tuning

PostgreSQL:

# /etc/postgresql/15/main/postgresql.conf
shared_buffers = 256MB
effective_cache_size = 768MB
maintenance_work_mem = 64MB
checkpoint_completion_target = 0.7
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 4MB
min_wal_size = 1GB
max_wal_size = 4GB
max_worker_processes = 4
max_parallel_workers_per_gather = 2
max_parallel_workers = 4

Redis:

# /etc/redis/redis.conf
maxmemory 512mb
maxmemory-policy allkeys-lru

Support

For additional support: - Check the GitHub Issues - Review documentation - Contact support@metricis.app