When the pager wakes you up at 3 AM because the payment gateway is returning intermittent 500 errors, the last thing you want to see is a dashboard with hundreds of red metrics and no context. You want to know exactly where the system is bleeding.

I've spent years fighting with different intelligent observability tools (what marketing insists on calling AIOps). In the end, in distributed architectures with microservices and databases that won't stop spitting out logs, the volume of data exceeds our human capacity for analysis. This is where CloudWatch Anomaly Detection, Amazon DevOps Guru, and Dynatrace Davis AI come in.

Each tool has a radically different approach. Let's tear them apart.

CloudWatch Anomaly Detection: The Statistical Hammer

This is AWS's most purist approach to the problem. You take a specific metric (for example, CPUUtilization or NetworkIn) and apply an algorithm that analyzes its historical behavior.

It works by creating prediction bands. If the actual value falls outside that band, the alarm triggers.

Pros: - Native integration. If you already use CloudWatch, you activate it with a couple of clicks. - Dynamic threshold configuration. You forget about having to readjust static alarms because traffic naturally spikes on Sunday afternoons. - Cheap. You pay per configured anomaly alarm.

Cons: - Zero context. It warns you that CPU usage has abnormally spiked by 40%, but it doesn't tell you why or what impact it has on other services. - Constant false positives during deployments. If consumption rises slightly due to Lambda cold starts, get ready for the noise.

When to use it: Use it on very isolated core metrics where a deviation always means trouble, like the number of database connections or stuck messages in an SQS (Dead Letter Queue).

Amazon DevOps Guru: The AWS Black Box

AWS tried to solve CloudWatch's lack of context with DevOps Guru. Instead of looking at isolated metrics, it analyzes the behavior of your entire stack based on operational patterns. It ingests metrics, logs, CloudTrail events, and even deployment history.

Pros: - Enriched context. When it detects a failure, it groups the metrics. It tells you: "Latency has spiked in API Gateway and at the same time this DynamoDB table is experiencing throttling." - Trivial setup. You tell it "watch this account" or "watch these tags" and forget about it.

Cons: - It's a black box. You have no control over the algorithms. If it misses something, you can't manually tweak the model. - It takes time to learn. In environments with highly irregular traffic or intermittent batch workloads, it takes days to adjust, flooding the alert channel in the meantime. - Restricted to AWS. If you have a hybrid system, DevOps Guru is blind outside of its ecosystem.

When to use it: Ideal for completely serverless architectures hosted on AWS, where AI governance in production relies almost 100% on managed services.

Dynatrace Davis AI: The Deterministic Scalpel

Unlike purely statistical models, Davis AI uses a deterministic approach based on topological dependency graphs. Dynatrace injects its OneAgent at the operating system level and maps in real-time which process talks to which container and which database it connects to.

Pros: - True Root Cause Analysis (RCA). It doesn't tell you "there's a problem in these three services." It tells you: "5% of users can't checkout because the inventory service is queuing requests due to a MySQL query taking 4 seconds after the last commit in the repository." - Zero initial configuration. You inject the agent and it automatically discovers the topology. - Full hybrid observability. It reaches all the way to the user's browser and bare-metal servers.

Cons: - Very high price. It stings quite a bit. In large systems with thousands of hosts, the bill is scary. - Intrusive. The OneAgent gets right into the kitchen at the kernel level. It rarely fails, but when an agent of this level collapses, it takes down the entire host.

When to use it: When the application handles high-value transactions and you need minimum resolution time. Very useful if you are integrating agents for incident management into automated flows.

Landing the Concept: CloudWatch Anomaly Detection with Terraform

So we don't just stick to theory, let's configure an anomaly detection alarm in AWS using Terraform.

Imagine this case: we want to monitor the number of 5XX errors in a load balancer (ALB). Setting a static alarm (e.g., alert if there are > 50 errors) brings problems. If traffic multiplies by 10 on Black Friday, 50 errors might be a ridiculous and acceptable percentage.

Here is how you spin this up in infrastructure as code:

resource "aws_cloudwatch_metric_alarm" "alb_5xx_anomaly" {
  alarm_name          = "ALB-5XX-Anomaly"
  comparison_operator = "GreaterThanUpperThreshold"
  evaluation_periods  = 2
  threshold_metric_id = "e1"

  metric_query {
    id          = "e1"
    expression  = "ANOMALY_DETECTION_BAND(m1, 2)"
    label       = "5XX Anomaly Band (2 Standard Deviations)"
    return_data = true
  }

  metric_query {
    id = "m1"
    metric {
      metric_name = "HTTPCode_Target_5XX_Count"
      namespace   = "AWS/ApplicationELB"
      period      = 300
      stat        = "Sum"
      dimensions = {
        LoadBalancer = aws_lb.produccion.arn_suffix
      }
    }
  }

  alarm_actions = [aws_sns_topic.alertas_pagos.arn]
}

In this block, ANOMALY_DETECTION_BAND(m1, 2) creates the band using 2 standard deviations over AWS's predictive model. The alarm triggers with GreaterThanUpperThreshold when the actual 5XX errors exceed the predicted upper limit for 2 consecutive 5-minute periods.

If you are interested in event-driven architectures that react to these metrics, check out how to integrate these signals with autonomous agents in linux environments.

Each tool serves a specific function depending on your architecture. CloudWatch for isolated critical metrics. DevOps Guru for your entire AWS account without configuring anything. Dynatrace Davis when the economic impact justifies paying for total deterministic visibility.