Skip to content

iam-auth.md

Latest commit

 

History

History
83 lines (68 loc) · 3.08 KB

iam-auth.md

File metadata and controls

83 lines (68 loc) · 3.08 KB

AWS MSK IAM authentication for Burrow

Burrow can now connect to Amazon MSK clusters that use IAM (SASL/OAUTHBEARER) + TLS without proxies or sidecars.
The feature is built on:

  • [github.com/aws/aws-msk-iam-sasl-signer-go v1.0.4] – generates the SigV4 bearer token expected by the broker.
  • [github.com/IBM/sarama ≥ v1.45.2] – already exposes the generic OAUTHBEARER mechanism and AccessTokenProvider hook we plug into.

Supported credential sources

Credential chain node Works? Notes
EC2 / EKS instance profile ✅ No extra config needed.
EKS Pod Identity ✅ Preferred on EKS ≥ 1.29.
IRSA (IAM Roles for SA) ✅ Works via the same default chain.
Shared credentials / static keys ✅ Standard ~/.aws/credentials or env vars.
STS AssumeRole w/ session tags ✅ Add role-arn in the TOML (requires sts:AssumeRole + sts:TagSession).

IAM auth requires the broker listener TLS + IAM (port 9098 or 9198) to be enabled.

Quick-start

  1. Enable the SASL/IAM and TLS security settings on your MSK cluster.
  2. Make sure Burrow’s container can reach the IAM listener (security groups, NACLs, etc.).
  3. Add the sections shown in examples/burrow.toml (below) and restart Burrow.

Example configuration

#######################################################################
#  TLS: trust the Amazon root CAs shipped in most Linux distros
#######################################################################
[tls.msk-tls]
# For Debian/Ubuntu/RHEL/… the CA bundle is already present:
cacert = "/etc/ssl/certs/ca-certificates.crt"

#######################################################################
#  IAM: pick one of the credential modes below
#######################################################################

# --- A) Use the pod / instance credentials directly ---------------
[iam.msk-iam]
region = "us-west-1"

# --- B) Re-assume a dedicated read-only role ----------------------
#[iam.msk-iam]
#region   = "us-west-1"
#role-arn = "arn:aws:iam::123456789012:role/burrow-readonly"
#profile  = "burrow"     # optional named profile

#######################################################################
[client-profile.msk-iam]
kafka-version = "2.8.0"
client-id     = "burrow"
tls           = "msk-tls"
iam           = "msk-iam"

#######################################################################
#  Cluster + consumer definitions (use the TLS+IAM listener)
#######################################################################
[cluster.prod]
class-name     = "kafka"
client-profile = "msk-iam"
servers = [
  "b-1.prod.abcd.c2.kafka.us-west-1.amazonaws.com:9098",
  "b-2.prod.abcd.c2.kafka.us-west-1.amazonaws.com:9098",
  "b-3.prod.abcd.c2.kafka.us-west-1.amazonaws.com:9098"
]

[consumer.prod]
class-name     = "kafka"
client-profile = "msk-iam"
cluster        = "prod"
start-latest   = true
group-denylist = '^(console-consumer-|.*\s.*).*$'
servers        = ${cluster.prod.servers}   # reuse same list