Uptime Monitoring

For whoever gets woken up. Checks that an endpoint is answering — HTTP, TCP or WebSocket — on an interval you choose, and turns a failure into work rather than into a notification nobody owns: a monitor that goes down opens a ticket, and can page the on-call rotation for the team that owns it.

The rest of this document is how it is built and configured.

Features#

• Multi-Protocol Checks: HTTP, TCP, and WebSocket endpoint monitoring
• Configurable Intervals: 1, 5, 15, 30, or 60 minute check intervals
• Automatic Incident Management: Incidents created after consecutive failures
• Ticket Integration: Automatically creates and closes support tickets
• Notifications: Slack, email, and webhook alerts
• SSL Monitoring: Track SSL certificate expiration (HTTP checks)
• Post-Mortem Notes: Root cause and resolution documentation

Check Types#

HTTP Check#

Monitors HTTP/HTTPS endpoints with configurable settings.

Collected Metrics:

• Response time (ms)
• Status code
• SSL certificate expiry (days remaining)
• Error messages

TCP Check#

Monitors TCP port availability.

Collected Metrics:

• Connection time (ms)
• Connection success/failure
• Error type (timeout, connection_refused)

WebSocket Check#

Monitors WebSocket endpoints with optional message validation.

Collected Metrics:

• Connection time (ms)
• Handshake success
• Response validation result
• Error messages

Incident Management#

How It Works#

1. Consecutive Failures: Monitor tracks consecutive failed checks
2. Threshold Reached: When failures >= threshold (default: 3), incident is created
3. Ticket Created: Support ticket automatically created with incident details
4. Notifications Sent: Slack/email/webhook alerts dispatched
5. Recovery Detected: When check succeeds, incident is resolved
6. Ticket Closed: Linked ticket automatically closed with resolution notes

Failure Flow#

Check fails
    ↓
Increment consecutive_failures
    ↓
consecutive_failures >= threshold?
    ├─ No → Schedule next check
    └─ Yes → Create incident
                ↓
            Create support ticket
                ↓
            Send notifications

Recovery Flow#

Check succeeds
    ↓
Has ongoing incident?
    ├─ No → Reset counters
    └─ Yes → Resolve incident
                ↓
            Close linked ticket
                ↓
            Send recovery notifications (if enabled)

Incident States#

Monitor Status#

Notifications#

Supported Channels#

Notification Events#

• Incident Created: When threshold failures reached
• Incident Acknowledged: When team acknowledges incident
• Incident Resolved: When service recovers (if notify_on_recovery enabled)

Webhook Payload#

{
  "event": "incident.created",
  "monitor": {
    "id": "uuid",
    "name": "API Server",
    "check_type": "http",
    "url": "https://api.example.com/health"
  },
  "incident": {
    "id": "uuid",
    "status": "ongoing",
    "started_at": "2024-01-15T10:30:00Z",
    "error_message": "Connection timeout after 30s"
  },
  "workspace_id": "uuid"
}

Ticket Integration#

Automatic Ticket Creation#

When an incident is created, a support ticket is automatically generated:

• Title: [UPTIME] {monitor_name} is down
• Severity: high
• Priority: urgent
• Description: Monitor details, error message, timestamp
• Team: Assigned to configured team (if set)

Automatic Ticket Closure#

When service recovers:

1. Resolution comment added to ticket:

   Service has recovered.
   
   Duration: 15 minutes
   Started: 2024-01-15 10:30:00 UTC
   Resolved: 2024-01-15 10:45:00 UTC
   Total checks during incident: 15
   Failed checks: 15
   

2. Ticket status set to closed
3. Recovery notifications sent (if enabled)

API Endpoints#

Monitors#

GET    /workspaces/{id}/uptime/monitors                    # List monitors
POST   /workspaces/{id}/uptime/monitors                    # Create monitor
GET    /workspaces/{id}/uptime/monitors/{monitor_id}       # Get monitor
PATCH  /workspaces/{id}/uptime/monitors/{monitor_id}       # Update monitor
DELETE /workspaces/{id}/uptime/monitors/{monitor_id}       # Delete monitor
POST   /workspaces/{id}/uptime/monitors/{monitor_id}/pause # Pause monitoring
POST   /workspaces/{id}/uptime/monitors/{monitor_id}/resume # Resume monitoring
POST   /workspaces/{id}/uptime/monitors/{monitor_id}/test  # Run immediate test
GET    /workspaces/{id}/uptime/monitors/{monitor_id}/checks # Get check history
GET    /workspaces/{id}/uptime/monitors/{monitor_id}/stats # Get monitor stats

Incidents#

GET    /workspaces/{id}/uptime/incidents                   # List incidents
GET    /workspaces/{id}/uptime/incidents/{incident_id}     # Get incident
PATCH  /workspaces/{id}/uptime/incidents/{incident_id}     # Update incident
POST   /workspaces/{id}/uptime/incidents/{incident_id}/acknowledge # Acknowledge
POST   /workspaces/{id}/uptime/incidents/{incident_id}/resolve     # Resolve

Statistics#

GET    /workspaces/{id}/uptime/stats                       # Workspace stats

Database Tables#

uptime_monitors#

Stores monitor configurations.

uptime_checks#

Stores individual check results (time-series data).

uptime_incidents#

Stores incident records linked to tickets.

Temporal Schedules#

Defined in backend/src/aexy/temporal/schedules.py:

{
    "id": "uptime-process-due-checks",
    "activity": "process_due_checks",
    "input_module": "aexy.temporal.activities.uptime",
    "interval": timedelta(seconds=60),
    "queue": TaskQueue.OPERATIONS,
},
{
    "id": "uptime-cleanup-old-checks",
    "activity": "cleanup_old_checks",
    "input_module": "aexy.temporal.activities.uptime",
    "interval": timedelta(hours=24),
    "queue": TaskQueue.OPERATIONS,
},

Activity Descriptions#

Inspect runs and history in the Temporal UI at http://localhost:8080.

Frontend Pages#

Configuration#

Environment Variables#

No additional environment variables required. Uses existing:

• REDIS_URL - For caching and LLM rate limiting
• TEMPORAL_ADDRESS - For the Temporal workflow engine that runs checks
• DATABASE_URL - For PostgreSQL storage

Check Intervals#

Default Settings#

File Structure#

frontend/src/
├── app/
│   ├── (app)/uptime/
│   │   ├── page.tsx                    # Dashboard
│   │   ├── monitors/
│   │   │   ├── page.tsx                # Monitors list
│   │   │   └── [monitorId]/page.tsx    # Monitor detail
│   │   ├── incidents/
│   │   │   ├── page.tsx                # Incidents list
│   │   │   └── [incidentId]/page.tsx   # Incident detail
│   │   └── history/page.tsx            # Check history
│   └── products/uptime/page.tsx        # Product landing page
└── lib/
    └── uptime-api.ts                   # Uptime API client

backend/src/aexy/
├── models/uptime.py                    # SQLAlchemy models
├── schemas/uptime.py                   # Pydantic schemas
├── services/
│   ├── uptime_service.py               # Business logic
│   └── uptime_checker.py               # Check executors
├── api/uptime.py                       # REST endpoints
└── temporal/activities/uptime.py       # Temporal activities (process_due_checks, execute_check, cleanup_old_checks)

Troubleshooting#

Monitor Not Running Checks#

1. Verify monitor is active (is_active = true)
2. Check the Temporal worker is running (uptime checks run as Temporal scheduled activities):

   docker compose logs temporal-worker
   

3. Open the Temporal UI at http://localhost:8080 and verify the uptime schedule is active and recent runs are succeeding.

Checks Timing Out#

1. Increase timeout_seconds on the monitor
2. Check if endpoint is accessible from container network
3. Verify any firewall rules allow outbound connections

Ticket Not Created#

1. Verify consecutive_failures_threshold has been reached
2. Check ticket service is accessible
3. Review Temporal worker logs for errors

Notifications Not Sending#

1. Verify notification channel is configured on monitor
2. Check Slack integration is connected (for Slack alerts)
3. Verify webhook URL is reachable
4. Review Temporal worker logs

SSL Errors#

1. If certificate is valid but failing, try setting verify_ssl = false
2. Check certificate chain is complete
3. Verify system trust store includes required root CAs

WebSocket Connection Failures#

1. Verify WS URL uses correct protocol (ws:// or wss://)
2. Check if server requires specific subprotocols
3. Test connection manually with wscat:

   wscat -c wss://your-endpoint.com/ws
   

Access Control#

Uptime monitoring is enabled for:

• Engineering Bundle: Full access
• Full Access Bundle: Full access

Permission required: can_view_uptime

Configure access in Settings > Access for your workspace.