SRE Prompts

role: >
  You are a senior site reliability engineer who has written and maintained
  on-call runbooks for web services, APIs, data pipelines, and background
  workers. You write runbooks for a tired engineer paged at 3 a.m.: short,
  ordered steps, exact places to look, safe mitigations first, and clear
  rules for when to escalate. You never invent dashboard names, commands, or
  contacts that the user did not give you; you leave clearly marked
  placeholders instead.

task: >
  Build an on-call runbook scaffold for the service described below:
  severity levels, detection signals and alerts, a triage flow, mitigation
  playbooks for each common incident, a rollback procedure, communication
  templates, an escalation path, and a post-incident checklist, ready to be
  pasted into the team's docs and filled in.

inputs:
  service_name: "checkout-api"
  what_it_does: "handles cart checkout and payment authorization for the web shop and mobile app"
  architecture: "stateless Node.js service on Kubernetes, 6 pods, PostgreSQL primary with one replica, Redis cache, calls an external payment provider"
  traffic_and_slo: "about 40 requests per second at peak, 99.9 percent availability and p95 latency under 400 ms"
  common_incidents: "payment provider timeouts, database connection pool exhaustion, bad deploys, Redis eviction storms, certificate expiry"
  dashboards_and_tools: "Grafana dashboard 'Checkout Overview', Loki logs, PagerDuty, Argo CD for deploys, status page on Statuspage"
  deploy_and_rollback: "Argo CD sync from main, roughly 10 deploys a week, feature flags in LaunchDarkly"
  escalation_contacts: "primary on-call (payments team), secondary on-call, payments team lead, database on-call from platform team, payment provider support line"
  comms_channels: "#inc-checkout Slack channel, customer support lead, public status page"
  business_hours_and_timezone: "Europe/Berlin, 24x7 on-call, weekly rotation"
  known_gaps: "no alert on certificate expiry yet, replica lag is not graphed"

instructions:
  - State assumptions where inputs are missing instead of asking questions.
  - Define 4 severity levels (SEV1 to SEV4) with customer impact, examples for this service, response time, who is paged, and whether the status page is updated.
  - For detection, list each alert or signal with a metric or log query placeholder, a threshold idea tied to the SLO, the likely causes, and the first dashboard panel to open.
  - Write a generic triage flow of at most 10 ordered steps for the first 15 minutes, starting with acknowledging the page and confirming impact.
  - Write one mitigation playbook per common incident with symptoms, quick checks, safe mitigations ordered from least to most risky, when to escalate, and how to confirm recovery.
  - Write a rollback procedure for a bad deploy, including how to find the last good version, how to roll back, how to disable a feature flag, and what to verify after.
  - Provide short communication templates for the incident channel kickoff, a 30-minute update, the status page (investigating, identified, monitoring, resolved), and a note to customer support.
  - Give an escalation path with conditions and time limits, using the roles I gave.
  - Add a post-incident checklist (timeline, impact numbers, follow-up tickets, alert tuning, runbook updates) and a blameless review agenda.
  - Turn known gaps into follow-up tasks with an owner role.
  - Use <PLACEHOLDER> markers for every link, query, command, or name that the team must fill in; never make them up.

output_format: YAML only, no prose outside the YAML, using exactly this structure

output_schema:
  assumptions: [string]
  service:
    name: string
    purpose: string
    dependencies: [string]
    slo: string
    owners: string
  severity_levels:
    - level: SEV1 | SEV2 | SEV3 | SEV4
      customer_impact: string
      examples: [string]
      response_time: string
      page: [string]
      status_page_update: boolean
  detection_signals:
    - name: string
      signal: string
      query_placeholder: string
      threshold_idea: string
      likely_causes: [string]
      first_panel_to_open: string
  triage_steps:
    - step: number
      action: string
      look_at: string
      decision: string
  playbooks:
    - incident: string
      symptoms: [string]
      quick_checks: [string]
      mitigations:
        - order: number
          action: string
          risk: low | medium | high
          command_placeholder: string
      escalate_when: string
      confirm_recovery: [string]
  rollback:
    find_last_good_version: [string]
    steps: [string]
    feature_flag_kill_switch: [string]
    verify_after: [string]
  comms_templates:
    incident_channel_kickoff: string
    update_every_30_min: string
    status_page:
      investigating: string
      identified: string
      monitoring: string
      resolved: string
    customer_support_note: string
  escalation_path:
    - role: string
      when: string
      after_minutes: number
      how_to_reach: string
  post_incident_checklist: [string]
  blameless_review_agenda: [string]
  follow_up_tasks:
    - task: string
      owner_role: string
      priority: high | medium | low

# Kubernetes taste of a platform team
- Deploy everything through GitOps (Argo CD or Flux); nobody runs kubectl apply against production by hand. Confidence: 0.90
- Package shared services with Helm or Kustomize, but keep per-environment overlays small and readable. Confidence: 0.80
- Set CPU and memory requests on every container, and set a memory limit equal to the request. Confidence: 0.90
- Avoid CPU limits on latency-sensitive services; rely on requests and node sizing to prevent throttling. Confidence: 0.70
- Give every Deployment a readiness probe; add a liveness probe only when the app can truly deadlock. Confidence: 0.85
- Use a startup probe for slow-booting apps instead of a long initialDelaySeconds on liveness. Confidence: 0.80
- Run at least two replicas behind a PodDisruptionBudget for anything users depend on. Confidence: 0.90
- Spread replicas across zones with topologySpreadConstraints. Confidence: 0.80
- Pin images by digest or an immutable version tag, never latest. Confidence: 0.95
- Run containers as non-root with a read-only root filesystem and all capabilities dropped. Confidence: 0.90
- Enforce the restricted Pod Security Standard on application namespaces. Confidence: 0.80
- Start every namespace with a default-deny NetworkPolicy and open only the flows you need. Confidence: 0.80
- Give each workload its own ServiceAccount and disable token automount when it never calls the API server. Confidence: 0.85
- Keep secrets out of Git; sync them from a secret manager with External Secrets or use Sealed Secrets. Confidence: 0.90
- Use one namespace per team or per app, with a ResourceQuota and a LimitRange. Confidence: 0.80
- Apply the standard app.kubernetes.io labels to every resource. Confidence: 0.75
- Handle SIGTERM and add a short preStop sleep so in-flight requests drain before shutdown. Confidence: 0.85
- Scale stateless services with the HorizontalPodAutoscaler on CPU or a real request-rate metric. Confidence: 0.75
- Run databases on a managed cloud service rather than StatefulSets unless there is a strong reason. Confidence: 0.80
- Validate manifests in CI with kubeconform and a policy engine such as Kyverno or OPA Gatekeeper. Confidence: 0.80
- Keep clusters within two minor versions of the latest Kubernetes release and upgrade on a schedule. Confidence: 0.75
- Expose HTTP services through one shared ingress or Gateway API controller, not a LoadBalancer per service. Confidence: 0.75