Skip to main content

Create a Runbook

Runbooks in Harness AI SRE enable you to automate incident response workflows, operational procedures, and remediation actions. This comprehensive guide walks you through creating, configuring, and deploying effective runbooks that can significantly reduce mean time to resolution (MTTR) and improve your team's operational efficiency.

Before You Begin​

Prerequisites​

Ensure you have the following before creating your first runbook:

  • Platform Access: Active Harness AI SRE account with appropriate permissions.
  • User Permissions: Required Account, Organisation and Project level permissions.
  • Integration Access: Configured integrations for the tools you plan to use (Slack, Jira, ServiceNow, etc.).
  • Monitoring Setup: Alert sources configured (Datadog, New Relic, PagerDuty, etc.).

Key Concepts​

Before diving into runbook creation, familiarize yourself with these core concepts:

  • Actions: Individual tasks or operations within a runbook (notifications, API calls, pipeline executions).
  • Triggers: Conditions that automatically initiate runbook execution.
  • Variables: Dynamic values that can be passed between actions and customized per execution.
  • Sequences: The order in which actions are executed within your workflow.

Creating Your Runbook​

Follow this interactive guide to create automated runbooks with actions, workflows, and integrations.

Advanced Configuration​

Trigger Configuration with Key Events​

Once Key Events are configured in your action parameters (as covered in Step 8):

  1. Set Trigger Condition: In the Triggers section, set the condition to "Key Event Created"
  2. Map Event to Input: The input field that was configured with the Key Event text can now be automatically populated when the trigger fires
  3. Dynamic Execution: When the specified Key Event occurs, the runbook will execute with the event data automatically mapped to your configured inputs

This approach allows for seamless integration between event detection and automated response, without requiring manual configuration of input variables for each execution.

Available Actions and Integrations​

Harness AI SRE provides a comprehensive library of pre-built actions across multiple categories. Choose the right combination of actions to build effective automation workflows.

Communication & Collaboration Tools​

Establish immediate communication channels and keep stakeholders informed throughout incident resolution.

Slack Integration​

  • Send Notifications: Broadcast alerts to channels or direct messages.
  • Create Channels: Automatically create incident-specific channels.
  • Start Threads: Organize discussions and updates.
  • Add Members: Add members to the channel.
  • Archive Channels: Clean up after incident resolution.

Microsoft Teams Integration​

  • Send Messages: Send alerts to specific teams or channels.
  • Create Meetings: Automatically create Teams meeting, optionally attaching an AI transcription bot.

Zoom Integration​

  • Create Meetings: Instantly set up incident response calls, optionally attaching an AI transcription bot.
  • End Meetings: End an active Zoom meeting.

Incident Response & Ticketing Systems​

Automate incident tracking, assignment, and resolution workflows across your preferred ticketing platforms.

Jira Integration​

  • Issue Creation: Automatically create tickets with relevant context.
  • Status Updates: Progress incidents through workflow states.
  • Update Tickets: Updates an existing Jira issue's summary, description, issue type, or adds a comment with relevant context.

ServiceNow Integration​

  • Incident Management: Create and manage ServiceNow incidents.
  • Change Requests: Initiate emergency or standard changes.
  • Update Incidents: Updates an existing ServiceNow incident's summary, description, issue type, or adds a comment with relevant context.

Automation & Pipeline Execution​

Execute remediation actions, deploy fixes, and trigger operational workflows.

Harness Pipelines Integration​

  • Pipeline Execution: Trigger deployment or remediation pipeline.
  • Feature Flag Management: Deploy specific versions or rollback changes.
  • Environment Management: Manage infrastructure scaling or configuration.

Configure Triggers​

Triggers determine when and how your runbooks execute automatically. Proper trigger configuration ensures your runbooks respond to the right conditions at the right time.

Setting Up Triggers​

  1. Access Trigger Configuration: Click the Triggers tab in your runbook editor.
  2. Add Trigger: Click + New Trigger to begin the trigger setup process.
  3. Choose Trigger Template: Select the type from available templates.
  4. Define Conditions: Set specific conditions for runbook activation based on the frequency of events or changes to specific resources.
  5. Test Triggers: Validate trigger logic before deployment.
info

Note: A user can add more than one trigger to a runbook based on the use case.

Trigger Configuration Best Practices​

  • Avoid Trigger Overlap: Ensure multiple runbooks don't trigger simultaneously for the same event.
  • Use Appropriate Delays: Add delays between related triggers to prevent conflicts.
  • Test Thoroughly: Validate trigger conditions in non-production environments.
  • Monitor Execution: Track trigger effectiveness and adjust conditions as needed.

Test Your Runbook​

Thorough testing is essential before deploying runbooks to production. A well-tested runbook prevents failures during critical incidents and ensures reliable automation.

Testing Steps​

  1. Select an Alert or Incident: Go to AI SRE → Alerts or Incidents in your Harness platform, then select the alert or incident you want to test.
  2. Select a Runbook: Click the Runbooks tab and select the runbook you want to test.
  3. Execute Runbook: In case of no associated runbooks, click Execute Runbook to begin the testing process.
  4. Test Runbook: Click Execute to begin the testing process.

Pre-Production Testing​

1. Environment Preparation​

  • Test Environment: Set up a dedicated testing environment that mirrors production.
  • Test Data: Prepare realistic test scenarios and data sets.
  • Integration Sandboxes: Use test instances of integrated tools (Slack, Jira, etc.).
  • Mock Services: Create mock endpoints for external dependencies.

2. Functional Testing​

  • Action Validation: Verify each action executes correctly with expected parameters.
  • Sequence Testing: Confirm actions execute in the correct order.
  • Variable Passing: Validate that variables are correctly passed between actions.
  • Error Handling: Test failure scenarios and error recovery mechanisms.

3. Integration Testing​

  • Notification Delivery: Confirm all notifications reach intended recipients.
  • Pipeline Executions: Verify that triggered pipelines complete successfully.
  • API Responses: Check that external API calls return expected results.
  • Authentication: Ensure all integrations authenticate properly.

4. End-to-End Testing​

  • Complete Workflows: Execute full runbook scenarios from trigger to completion.
  • Multiple Scenarios: Test various input combinations and edge cases.
  • Performance Testing: Measure execution times and resource usage.
  • Concurrent Execution: Test behavior when multiple instances run simultaneously.

Testing Checklist​

  • All actions execute without errors.
  • Notifications are delivered to correct channels/recipients.
  • Variables are properly populated and passed.
  • External integrations respond as expected.
  • Error conditions are handled gracefully.
  • Execution logs provide sufficient detail for troubleshooting.
  • Performance meets acceptable thresholds.
  • Security permissions are correctly enforced.

Deploy and Monitor​

Once testing is complete, deploy your runbook to production and establish monitoring to ensure continued effectiveness.

Deployment Process​

  1. Final Review: Conduct a final review of runbook configuration and testing results.
  2. Stakeholder Approval: Obtain necessary approvals from the team.
  3. Production Deployment: Activate the runbook in your production environment.
  4. Documentation Update: Update operational documentation with runbook details.

Best Practices for Runbook Creation​

Design Principles​

  • Start Simple: Begin with basic workflows and gradually add complexity as you gain experience.
  • Modular Design: Create reusable actions and workflows that can be combined for different scenarios.
  • Clear Naming: Use descriptive names for runbooks, actions, and variables that clearly indicate their purpose.

Operational Excellence​

  • Regular Updates: Review and update runbooks regularly to reflect changes in infrastructure and processes.
  • Timeout Configuration: Set appropriate timeouts to prevent runbooks from hanging indefinitely.
  • Conditional Logic: Use conditional statements to avoid unnecessary action execution.

Troubleshooting Common Issues​

Execution Failures​

Problem: Runbook actions fail to execute​

  • Solution: Check integration credentials and network connectivity.
  • Prevention: Implement health checks and credential rotation.

Problem: Variables not passing between actions​

  • Solution: Verify variable names and data types match expectations.
  • Prevention: Use consistent naming conventions and validate variable mappings.

Performance Issues​

Problem: Runbooks execute slowly​

  • Solution: Optimize action sequences and enable parallel execution where possible.
  • Prevention: Regular performance testing and monitoring.

Next Steps​

Advanced Configuration​

Integration Setup Guides​

Communication & Collaboration​

Incident Management​

Automation & Pipelines​

info

Need Help? Contact our support team or visit the Harness Documentation for additional resources and troubleshooting guides.