The Challenge of Manual On-Call Management
Managing on-call rotations is a critical but often tedious task for any team responsible for system reliability. Manually updating PagerDuty schedules, especially in dynamic environments with frequent team changes or project handovers, is prone to errors and can consume valuable engineering time. At SoftCrafter, where we specialize in robust web development and mobile solutions, efficiency and reliability are paramount. We’ve found that automating these processes is key to maintaining high operational standards and ensuring prompt incident response.
This article will guide you through automating PagerDuty on-call rotation management using Infrastructure as Code (IaC) with Terraform and orchestrating these changes with GitHub Actions. This approach not only reduces manual overhead but also brings version control, auditability, and consistency to your on-call infrastructure.
Setting Up PagerDuty for Automation
Before diving into Terraform, ensure your PagerDuty environment is ready. You’ll need:
- A PagerDuty account with appropriate API access.
- An API Token: Generate a new ‘Read/Write’ API Token in PagerDuty under
Integrations > API Access Keys. Keep this token secure, as it grants significant control over your PagerDuty account. - A Team: Ensure you have a team defined in PagerDuty that will own the schedules and services.
For demonstration, let’s assume we have a team named ‘SoftCrafter DevOps’ and we want to manage a primary on-call schedule for them.
Defining PagerDuty Resources with Terraform
Terraform is an excellent tool for managing PagerDuty resources because it allows you to define your schedules, escalation policies, and services as code. This means your on-call setup can be version-controlled, reviewed, and deployed just like any other piece of your infrastructure.
First, set up your Terraform configuration. You’ll need the PagerDuty provider:
terraform {
required_providers {
pagerduty = {
source = "PagerDuty/pagerduty"
version = "~> 1.16"
}
}
}
provider "pagerduty" {
token = var.pagerduty_token
}
variable "pagerduty_token" {
description = "PagerDuty API token"
type = string
sensitive = true
}
variable "team_id" {
description = "ID of the PagerDuty team"
type = string
}
variable "users" {
description = "List of PagerDuty user IDs for the rotation"
type = list(string)
}Next, let’s define a simple on-call schedule. This example creates a daily rotation. For corporate services where uptime is non-negotiable, having a clearly defined and automated schedule is crucial.
resource "pagerduty_schedule" "primary_oncall" {
name = "SoftCrafter Primary On-Call Schedule"
time_zone = "America/New_York"
team_id = var.team_id
layer {
name = "Primary Layer"
start = "2023-01-01T09:00:00-05:00" # Example start date, adjust as needed
rotation_turn_length_seconds = 86400 # Daily rotation (24 hours)
rotation_virtual_start = "2023-01-01T09:00:00-05:00"
users = var.users
}
}
resource "pagerduty_escalation_policy" "devops_policy" {
name = "SoftCrafter DevOps Escalation Policy"
team_id = var.team_id
rule {
escalation_delay_in_minutes = 5
target {
type = "schedule"
id = pagerduty_schedule.primary_oncall.id
}
}
rule {
escalation_delay_in_minutes = 10
target {
type = "user"
id = "P123456" # Example: Fallback user ID for broader team notification
}
}
}
resource "pagerduty_service" "devops_service" {
name = "SoftCrafter Core Services"
auto_resolve_timeout_minutes = 1440 # 24 hours
acknowledgement_timeout_minutes = 60
escalation_policy = pagerduty_escalation_policy.devops_policy.id
team = var.team_id
}Automating Deployments with GitHub Actions
Now that our PagerDuty configuration is defined in Terraform, we can use GitHub Actions to automate its deployment. This ensures that any changes to your main branch automatically update your PagerDuty schedules.
Create a workflow file (e.g., .github/workflows/pagerduty-terraform.yml):
name: PagerDuty Terraform Deployment
on:
push:
branches:
- main
workflow_dispatch:
env:
TF_WORKING_DIR: ./terraform
jobs:
terraform:
name: 'Terraform Apply'
runs-on: ubuntu-latest
environment: production # Use GitHub Environments for secrets management
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup Terraform
uses: hashicorp/setup-terraform@v2
with:
terraform_version: 1.x.x
- name: Terraform Init
id: init
run: terraform init
working-directory: ${{ env.TF_WORKING_DIR }}
- name: Terraform Validate
id: validate
run: terraform validate
working-directory: ${{ env.TF_WORKING_DIR }}
- name: Terraform Plan
id: plan
run: terraform plan -no-color -input=false
working-directory: ${{ env.TF_WORKING_DIR }}
env:
PAGERDUTY_TOKEN: ${{ secrets.PAGERDUTY_API_TOKEN }}
- name: Terraform Apply
id: apply
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
run: terraform apply -auto-approve -input=false
working-directory: ${{ env.TF_WORKING_DIR }}
env:
PAGERDUTY_TOKEN: ${{ secrets.PAGERDUTY_API_TOKEN }}
TF_VAR_pagerduty_token: ${{ secrets.PAGERDUTY_API_TOKEN }}
TF_VAR_team_id: ${{ secrets.PAGERDUTY_TEAM_ID }}
TF_VAR_users: '["PXXXXXXXX", "PYYYYYYYY"]' # Replace with actual PagerDuty User IDsImportant Security Note: Store your PagerDuty API Token and other sensitive variables as GitHub Secrets in your repository or organization settings. Never hardcode them directly into your workflow files or Terraform code. For example, PAGERDUTY_API_TOKEN and PAGERDUTY_TEAM_ID would be GitHub Secrets.
This workflow will:
- Checkout your repository code.
- Initialize Terraform.
- Validate your Terraform configuration.
- Create a Terraform plan (useful for review).
- Automatically apply changes when pushed to the
mainbranch.
Benefits and Best Practices
By adopting this automated approach, SoftCrafter teams can significantly reduce the time spent on administrative tasks related to on-call management. This allows our engineers to focus more on building innovative e-commerce solutions and robust software, rather than wrestling with manual configurations.
Key benefits include:
- Version Control: All PagerDuty configurations are in Git, providing a complete history of changes.
- Auditability: Every change is tied to a commit, making it easy to track who changed what and when.
- Consistency: Ensures that all environments adhere to the same on-call policies.
- Reduced Errors: Eliminates human error associated with manual updates.
- Faster Onboarding: New team members or changes to rotation can be implemented quickly via code.
When implementing this, consider:
- State Management: Use a remote backend for Terraform state (e.g., S3, Azure Blob Storage) to ensure state is shared and secured.
- Branch Protection: Enforce pull request reviews on your
mainbranch to ensure changes to on-call schedules are reviewed by multiple team members. - Dynamic User Lists: For larger organizations, consider dynamically generating the
TF_VAR_userslist from an external source (e.g., an internal directory or a separate configuration file) if user IDs change frequently.
Conclusion
Automating PagerDuty on-call rotation management with Terraform and GitHub Actions is a powerful strategy for any organization looking to enhance operational efficiency and reliability. By treating your on-call infrastructure as code, you gain the benefits of version control, auditability, and consistency, freeing up valuable engineering time. SoftCrafter is committed to leveraging such automation to deliver superior software services to our clients. For more information on how we can help your business with robust and automated solutions, feel free to contact us.
#PagerDuty #Terraform #GitHubActions #DevOps #Automation #OnCall #IaC #SoftCrafter