How to Build Bespoke Operational Workflows with AWS DevOps Agent Custom SRE Agents

Every engineering team has critical, unwritten rules - like ensuring a read replica never lags more than five seconds or verifying a nightly data pipeline scales to zero by dawn - that general-purpose monitoring tools simply cannot track. When these bespoke production standards live only in static runbooks or institutional memory, they are inevitably forgotten during high-pressure deployments. The introduction of AWS DevOps Agent custom SRE agents fundamentally changes this dynamic, allowing teams to define highly specific, workload-aware operational workflows in plain language and enforce them automatically.

Broad, reusable checks are valuable, but they cannot anticipate every rule specific to your environment. The most important operational questions are often intentionally narrow. Teams need to know if a production workload still aligns with policies written after its last security review, or if a new resource appeared without the tags required by data handling processes. These targeted questions keep changing as incidents, new customer requirements, and migrations reshape the infrastructure.

Because these agents are configured and executed directly within the AWS DevOps Agent environment, they bring in enriched context from Production Intelligence. This includes topology discovery across integrated accounts, AWS API access through approved tools, artifact generation for reports, and operations backlog integration. The true differentiator is not just writing the objective in natural language, but executing it where the context, tools, and connections to existing systems already live.

Operational work rarely starts and ends in one place. A team might discover a resource configuration issue in AWS, triage it in an operations backlog, assign remediation in Jira, notify an owning team in Slack, and attach evidence to an operational review. Without a unified integration layer, each of these steps becomes a separate, fragile automation project.

Custom SRE agents provide a consistent way to connect those fragmented steps. A custom agent can inspect AWS resources, generate an artifact, open a ticket, send a notification, read an exception list from Amazon S3, or hand off findings to another specialized agent. This capability serves two distinct roles: automating bespoke work for specific workloads and extending the platform itself, turning operational intent into repeatable action.

Drift detection is a perfect use case because every team defines "drift" differently. For one workload, it might mean missing encryption on a data store; for another, it could be a public path to a database or an observability requirement that no longer matches the production standard. A drift detection agent runs the same four steps an experienced engineer would: load the team's policies, discover the production topology, inspect each relevant resource via the AWS API, and compare the current state to the expected state.

To begin, you must define the standards the agent should evaluate. Consider a production workload team with strict standards for encryption, database network isolation, resource tagging, credential handling, and observability. You can create a core-policies skill that captures these exact requirements.

---
  name: core-policies
  description: Defines infrastructure and data policies, best practices, and sensitive data handling requirements. Use when auditing resources for compliance, reviewing code changes for policy violations, or determining if a resource handles sensitive data.
  ---
  # Core Policies
  This skill defines the organization's infrastructure standards, security policies, and best practices that all cloud resources and applications must follow.
  
  ## Infrastructure & Data Policies
  **No exceptions.** Block code changes that violate these policies and immediately flag infrastructure discovered to have violations.
  
  | Policy | Requirement |
  |--------|-------------|
  | Encryption at rest | All customer data must be encrypted at rest |
  | Encryption in transit | All customer data must be encrypted in transit |
  | Database network isolation | There can be no direct network route to databases from public internet ingress |
  | Resource tagging | All cloud resources must have the `sensitive_data` tag with a `true` or `false` value |
  | No credentials in code | There are no credentials in the code repository |
  
  ## Best Practices
  **Warn but do not block** code changes that violate these standards.
  
  | Practice | Requirement |
  |----------|-------------|
  | Logging | All applications should log to Splunk |
  | Observability | All applications should send traces and metrics to Datadog |
  
  ## Sensitive Data Classification
  Sensitive data can include customer data, intellectual property, or any other data deemed to require higher security measures to protect from exfiltration.
  
  A resource or application requires sensitive data handling if **any** of the following apply:
  1. Classified as `sensitive` in the `DATA_CLASSIFICATION.md` file in the code repository
  2. Tagged with `sensitive_data: true` in AWS or Azure Cloud

In AWS DevOps Agent, you can create this custom agent conversationally through Chat. You simply describe what you need in plain language, such as: "Create an agent that reviews production AWS resources for drift from our infrastructure standards (encryption, network isolation, tagging, credential handling, and observability) and creates recommendations for anything out of compliance."

From that single prompt, the Chat interface guides you through the creation process. It confirms your intent, checks for duplicate agents, proposes necessary tools (like use_aws for API inspection), determines the output type, and drafts the full system prompt for your review. The resulting system prompt acts as the definitive instruction set for the agent.

You are a drift detection agent specializing in identifying deviations from operational standards.
  
  ## Goal
  
  Review provisioned AWS resources in the production environment for drift from defined standards and best practices, and create recommendations for any improvements needed. Additionally, identify opportunities to codify your standards as AWS Config rules for automated enforcement.
  
  ## Approach
  
  1. Load the `core-policies` skill to understand the defined standards and best practices.
  2. Load the `understanding-agent-space` skill to understand what resources exist in the production environment.
  3. Identify resources that should be checked against policies (e.g., Lambda functions, RDS instances, S3 buckets, security groups).
  4. Use `use_aws` to make API calls and inspect the actual configuration state of each relevant resource.
  5. Compare each resource's current configuration against the defined policies.
  6. For each deviation found, create a recommendation in the improvements backlog.
  7. Use `use_aws` to list existing AWS Config rules and their configurations.
  8. Compare your defined policies from `core-policies` against existing Config rules to identify coverage gaps  -  standards that could reasonably be enforced via AWS Config but aren't yet.
  
  ## Constraints
  
  - Read-only access to infrastructure  -  do not make changes directly.
  - Only flag deviations that clearly violate defined policies.
  - Focus on provisioned resources, not code or deployment pipelines.
  
  ## Output
  
  Produce a single artifact titled "Drift Detection Report  -  [date]" containing:
  - A summary of resources scanned and deviations found
  - A table listing each drift item: resource ARN, policy violated, current vs. expected state, and suggested remediation
  - A section titled "Config Rule Coverage Gaps" with:
    - Policies from core-policies that could be codified as AWS Config rules but aren't currently
    - For each gap: the policy, why it's a good fit for Config, and a suggested rule approach (managed rule if available, or custom rule outline)
  
  If a drift detection report artifact already exists, update it with the latest findings instead of creating a new one.
  
  Additionally, for each drift item found, create a recommendation with:
  - A clear title describing the deviation
  - The specific policy being violated
  - The resource ARN and current configuration state vs. expected state
  - Suggested remediation steps
  
  Before creating a new recommendation, list existing recommendations and update an existing one if the same drift is already tracked.

Once executed, the agent produces durable outputs. These include structured reports (Artifacts) containing tables and topology diagrams, actionable items in the Improvements backlog, and Model Context Protocol (MCP) tool actions that can trigger downstream processes like Jira ticket creation or Slack notifications.

In a practical test run, the agent scanned 14 Lambda functions, roughly 20 key Amazon S3 buckets, and a DynamoDB table across the us-east-1 region. The results were highly specific: it found six distinct drift items. All 14 Lambda functions and the DynamoDB state table were missing the mandatory sensitive_data tag. Furthermore, several S3 buckets were untagged, and two Lambda functions were still running on end-of-life Node.js runtimes (nodejs12.x and nodejs14.x).

Crucially, the agent also discovered that zero AWS Config rules were deployed, meaning none of the core policies had automated enforcement at the infrastructure level. However, it did verify that encryption at rest (AES256) was correctly configured on all checked buckets. Each of these findings was automatically converted into a recommendation in the Improvements backlog, preventing duplicate tickets by updating existing entries.

For the initial runs, it is best to trigger the agent on demand. This allows your team to review the findings, tune the policy language, and ensure that recommendations are routed correctly without generating unnecessary noise. Once the output is refined, you can configure the custom agent to run automatically using schedule-based triggers.

AWS DevOps Agent supports EventBridge-compatible cron and rate expressions. For example, setting the trigger to cron(0 6? * MON *) will execute the agent every Monday at 6:00 AM UTC, perfectly timed before a weekly operational review. You can configure this by navigating to the custom agent, selecting the Triggers tab, and entering the schedule expression.

Drift detection is just one application of this technology. The same underlying mechanism can be adapted to support a wide variety of bespoke operational workflows across your infrastructure.

• SLO Agent: Checks whether a specific service is deviating from its latency or availability targets.
• Deployment Risk Agent: Reviews production changes against workload-specific risk signals before they go live.
• Resilience Posture Agent: Verifies whether a critical workflow still meets its original recovery assumptions.
• Operational Handoff Agent: Automatically converts backlog items into Jira tickets or targeted team notifications.
• Cost Optimization Agent: Reviews resource utilization on a daily schedule and recommends rightsizing or cleanup.
• Latency Reporting Agent: Generates a daily p50/p95/p99 summary across all customer-facing endpoints.

When designing these agents, always scope them to a specific workload or environment. Separate mandatory policies from best-practice warnings so the agent knows exactly what to block versus what to simply flag for review.

The introduction of AWS DevOps Agent custom SRE agents marks a fundamental shift in how cloud infrastructure is governed. For years, teams have relied on deterministic tools like AWS Config to enforce broad rules. However, AWS Config is binary - it checks if a resource matches a predefined state. It struggles with heuristic, context-heavy evaluations, such as determining if a specific combination of tags, network routes, and data classifications violates a nuanced business policy.

By bridging the gap between static infrastructure rules and dynamic operational context, these custom agents effectively digitize the "tribal knowledge" of senior engineers. The fact that the agent in the test run identified a lack of AWS Config rules and suggested which policies should be codified as managed rules is a massive leap forward. It transforms the agent from a simple auditing tool into an active consultant that helps mature a team's automation posture.

Ultimately, this reduces alert fatigue. Instead of bombarding developers with generic security warnings, the agent provides highly contextualized, workload-specific recommendations directly into the tools they already use. As infrastructure continues to scale in complexity, relying on human memory and static runbooks is no longer viable; context-aware, AI-driven SRE agents will become the baseline for maintaining production stability.