Centralized AWS Observability: Practical Guide (Part 2)

By [Denislav Tsonev]· March 30, 2026 · 4 min read

In part one we covered the architecture and design behind the centralized observability solution. In this post, we walk through the practical configuration and provide code snippets to build the first part of the solution: metrics centralization.

CloudWatch cross-account observability (OAM) requires you to designate a monitoring account (or create one). AWS recommends a dedicated monitoring account. In our case, we used the existing Shared Services account where we already run shared services, so we did not use a separate account.

AWS configuration

As I promised, I will provide as much as possible code snippets how I configured everything. The first part was to configure AWS to send logs and metrics in that one account.

Monitoring account (Sink) — Policy Sink Examples

# Monitoring account

resource "aws_oam_sink" "this" {
  count = var.source_account_ids != null ? 1 : 0

  name = var.monitoring_account_sink_name
}

resource "aws_oam_sink_policy" "this" {
  count = var.source_account_ids != null ? 1 : 0

  sink_identifier = aws_oam_sink.this[0].id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "oam:CreateLink",
          "oam:UpdateLink"
        ]
        Effect   = "Allow"
        Resource = "*"
        Principal = {
          "AWS" = var.source_account_ids
        }
        Condition = {
          "ForAllValues:StringEquals" = {
            "oam:ResourceTypes" = [
              "AWS::CloudWatch::Metric"
            ]
          }
        }
      }
    ]
  })
}

# Source accounts

resource "aws_oam_link" "this" {
  count = local.monitoring_account_sink_identifier != "" ? 1 : 0

  label_template = "$AccountName"
  resource_types = ["AWS::CloudWatch::Metric"]
  sink_identifier = local.monitoring_account_sink_identifier
}

resource "aws_iam_role" "cloudwatch-cross-account-observability" {
  count = local.monitoring_account_sink_identifier != "" ? 1 : 0

  name = "CloudWatch-CrossAccountSharingRole-${local.aws_region_abbrev}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          AWS = local.monitoring_account_id
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "policy" {
  for_each = local.monitoring_account_sink_identifier != ""
    ? toset(var.cross_account_monitoring_managed_policy)
    : []

  policy_arn = "arn:aws:iam::aws:policy/${each.key}"
  role       = aws_iam_role.cloudwatch-cross-account-observability[0].name
}

Some clarification (related to our Terraform structure and repositories): once you apply this in the monitoring account, you get the Sink and policies that allow the specified source accounts to create and update links. In the source accounts, the aws_oam_link, IAM role, and policy attachments enable cross-account sharing of CloudWatch metrics (and optionally logs/traces when added to the policy).

Note: The Sink and Link resources should be created for each region where you want to centralize CloudWatch metrics. Duplicate the configuration across all target regions in both the monitoring and source accounts to ensure complete observability coverage.

Log Centralization

With metrics now flowing from source accounts, you can centralize logs using AWS CloudWatch Logs centralization rules. This feature (released September 2025) allows you to aggregate logs from multiple accounts and regions into a single monitoring account and region.

# Monitoring account

resource "aws_observabilityadmin_centralization_rule_for_organization" "this" {
  count = local.deploying_default_region && var.monitoring_regions != [] ? 1 : 0

  rule_name = "aggragated-logs-rule"

  rule {
    destination {
      region  = local.region
      account = data.aws_caller_identity.current.account_id
      destination_logs_configuration {
        logs_encryption_configuration {
          encryption_strategy = "AWS_OWNED"
        }
      }
    }

    source {
      regions = var.monitoring_regions
      scope   = "OrganizationId = '${data.aws_organizations_organization.current.id}'"

      source_logs_configuration {
        encrypted_log_group_strategy = "ALLOW"
        log_group_selection_criteria = "*"
      }
    }
  }
  tags = var.tags
}

Note: While you can also use OAM Sink and Link for logs (similar to metrics), centralization rules offer more granular control. You can apply subscription filters to specific log groups, set encryption strategies, and define scope conditions for selective log aggregation across your organization.

Lambda functions

Now that logs are centralized into one account, we need to process them and ship them to Loki. To do this in an automated way we need two Lambda functions.

Log group subscription manager — every time a developer creates a new log group in any source account, we need to automatically attach a CloudWatch subscription filter to it so the logs flow to our processing function. A Lambda that periodically lists all log groups and configures subscription filters for any that are missing one handles this cleanly. Without this you would have to manually add filters every time a new service or log group appears.

Log shipper to Loki and S3 — lambda-promtail (use the bootstrap.zip release artifact) ships CloudWatch logs to Loki, but it only supports access logs, not application logs. For application logs we need a custom Lambda that receives the log events from the subscription filter, parses and transforms them, pushes them to Loki via the push API, and also stores a copy in S3 for compliance retention.

Both Lambdas, along with the IAM roles, subscription filter configuration, and S3 bucket setup, are in the cloudymountains/aws-observability repo. The code is too involved for inline snippets here, but the repo contains working examples you can adapt.

Access logs

AWS access logs (ALB, CloudFront, WAF, S3, etc.) are another very common log type you will want in one place. There are a few approaches, but the most common pattern is:

S3 buckets per workload account with cross-account replication — ALB and CloudFront both support writing access logs directly to an S3 bucket in a different account, so you can point them at a bucket owned by the monitoring account or use replication rules to pull them in. WAF is the notable exception: its access logs must go to a bucket (or Firehose) in the same account as the WAF resource, so you need to replicate from there.
Centralized S3 buckets from the start — alternatively, provision the access log buckets directly in the monitoring account and grant the necessary cross-account s3:PutObject permissions to each workload account. This avoids replication entirely but requires coordinating bucket policy updates whenever a new account or service is onboarded.

Neither approach is universally better — the replication pattern is easier to retrofit onto an existing multi-account setup, while central buckets are cleaner if you are building from scratch.

Summary

This post walked through the Terraform for OAM Sink and Link (metrics centralization), CloudWatch log centralization rules (aggregating logs from all source accounts into the monitoring account), and the two Lambda functions that close the loop — one to automatically manage subscription filters on new log groups, and one to ship application logs to Loki while also storing them in S3 for compliance. Even at this stage, as shown in the AI generated picture below, having all logs land in a single account already makes operations significantly easier — no more jumping between accounts to investigate an incident. In the next parts of the series we will cover YACE configuration and Grafana datasource setup.

“Single pane of glass”