← Writing

Goodbye PEM files: moving to RDS IAM database authentication

How I replaced a shared .pem key and a bastion host with AWS RDS IAM auth: short-lived tokens, per-user identity, and nothing left to rotate by hand.

For years, connecting to our production database looked like this: find the shared .pem file (usually in someone’s Slack DMs), SSH into a bastion host, open a tunnel to the private RDS instance, and paste in a database password that hadn’t changed since the instance was created. It worked, which is exactly why nobody fixed it. This is how I finally replaced it with RDS IAM database authentication, and what I’d tell past me before starting.

The old setup, and why it aged badly

The database lived in a private subnet, as it should. The only way in was an SSH tunnel through a bastion:

ssh -i team-bastion.pem -N -L 5432:mydb.abc123.ap-southeast-1.rds.amazonaws.com:5432 ec2-user@bastion.example.com

Humans were only half of it. Our Go services run on Lambda, and the functions needed the same path in, so the key was baked straight into the deployment artifact and the tunnel was opened in code:

//go:embed team-bastion.pem
var bastionKey []byte

func connect(ctx context.Context) (*pgx.Conn, error) {
	signer, err := ssh.ParsePrivateKey(bastionKey)
	if err != nil {
		return nil, err
	}

	bastion, err := ssh.Dial("tcp", "bastion.example.com:22", &ssh.ClientConfig{
		User:            "ec2-user",
		Auth:            []ssh.AuthMethod{ssh.PublicKeys(signer)},
		HostKeyCallback: ssh.InsecureIgnoreHostKey(), // yes, really
	})
	if err != nil {
		return nil, err
	}

	cfg, err := pgx.ParseConfig(os.Getenv("DATABASE_URL")) // static password inside
	if err != nil {
		return nil, err
	}
	cfg.DialFunc = func(ctx context.Context, network, addr string) (net.Conn, error) {
		return bastion.Dial(network, addr)
	}
	return pgx.ConnectConfig(ctx, cfg)
}

Every invocation paid for an SSH handshake before it could run a query, and every function in the account carried a copy of the key to production.

The problems weren’t hypothetical. We hit all of them:

  • One key, many people. The .pem file was shared, so rotating it meant coordinating with everyone at once. So we didn’t rotate it.
  • The key shipped with every build. Because the Lambdas embedded it, rotating the key also meant rebuilding and redeploying every function that carried a copy. One secret, dozens of deployment artifacts.
  • Offboarding was a lie. When someone left, they kept a working copy of the key unless we rotated it (see above).
  • No audit trail. Every connection came from ec2-user through the same tunnel. CloudTrail and the database logs both said the same thing: someone connected. Great, thanks.
  • The bastion itself. One more EC2 instance to patch, monitor, and pay for, whose entire job was being a hole in the network.
  • Static database credentials. The app password and the human password were both long-lived strings sitting in password managers and, let’s be honest, a few shell histories.

How RDS IAM auth works

The idea is simple: instead of a password, the database accepts a short-lived token signed by AWS. Your IAM identity becomes your database identity. There are four pieces.

1. Enable IAM auth on the instance. A checkbox in the console, or:

aws rds modify-db-instance \
  --db-instance-identifier mydb \
  --enable-iam-database-authentication \
  --apply-immediately

2. Create a database user mapped to IAM. For Postgres, you grant the rds_iam role. For MySQL you’d create the user with AWSAuthenticationPlugin instead.

CREATE USER exo_mercado WITH LOGIN;
GRANT rds_iam TO exo_mercado;
GRANT readonly TO exo_mercado; -- normal grants still apply

3. Attach an IAM policy allowing rds-db:connect for that user. Note the resource ARN uses the DbiResourceId (starts with db-), not the instance name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "rds-db:connect",
      "Resource": "arn:aws:rds-db:ap-southeast-1:123456789012:dbuser:db-ABCDEFGHIJKL01234/exo_mercado"
    }
  ]
}

4. Generate a token and use it as the password. The token is a presigned request, valid for 15 minutes:

TOKEN=$(aws rds generate-db-auth-token \
  --hostname mydb.abc123.ap-southeast-1.rds.amazonaws.com \
  --port 5432 \
  --username exo_mercado)

PGPASSWORD=$TOKEN psql \
  "host=mydb.abc123.ap-southeast-1.rds.amazonaws.com port=5432 dbname=app user=exo_mercado sslmode=verify-full sslrootcert=global-bundle.pem"

Yes, there’s still a .pem file in that command. It’s the public RDS CA bundle, not a secret, and I find that funny.

The Go side: Lambda through RDS Proxy

For the Lambdas we didn’t point pgx at the instance directly. Lambda churns connections by design, so we put RDS Proxy in front and enabled IAM auth on the proxy. The proxy holds the warm pool and speaks to the database using credentials it keeps in Secrets Manager; our code only ever authenticates to the proxy with a token. Two details matter here:

  • The rds-db:connect policy for a proxy uses the proxy id (starts with prx-) in the resource ARN, not the instance’s db- id, and it goes on the Lambda’s execution role. The function itself holds no credentials at all.
  • Tokens expire in 15 minutes, so you generate one per new connection, not one per deploy. pgx has the perfect hook for this: BeforeConnect.
const proxyEndpoint = "app-proxy.proxy-abc123.ap-southeast-1.rds.amazonaws.com:5432"

func newPool(ctx context.Context) (*pgxpool.Pool, error) {
	awsCfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		return nil, err
	}

	cfg, err := pgxpool.ParseConfig(
		"postgres://app_service@" + proxyEndpoint + "/app?sslmode=verify-full")
	if err != nil {
		return nil, err
	}
	cfg.BeforeConnect = func(ctx context.Context, cc *pgx.ConnConfig) error {
		token, err := auth.BuildAuthToken(
			ctx, proxyEndpoint, awsCfg.Region, cc.User, awsCfg.Credentials)
		if err != nil {
			return err
		}
		cc.Password = token
		return nil
	}
	return pgxpool.NewWithConfig(ctx, cfg)
}

auth.BuildAuthToken comes from github.com/aws/aws-sdk-go-v2/feature/rds/auth and does the same thing as generate-db-auth-token, using whatever credentials the Lambda role provides. Every fresh connection in the pool gets a fresh token, so expiry never bites. Compare this to the SSH version above: no embedded key, no tunnel, no static password, and about half the code.

One pleasant surprise: the proxy endpoint presents a certificate that public roots already trust, so verify-full works without shipping the RDS CA bundle in the artifact.

The gotchas

Tokens expire in 15 minutes. That’s the point, but it means anything that reconnects needs to generate a fresh token each time, not cache one in an env var. In Go, BeforeConnect handles it (see above). For humans, a small wrapper script around psql does the same job.

SSL is required. Connections without TLS are rejected outright. Set sslmode=verify-full and ship the CA bundle; don’t settle for require.

Auth rate limits are real. IAM auth adds work per connection, and AWS documents a connection-rate ceiling. For humans running psql this is irrelevant. For Lambda it absolutely is not, which is the other reason the functions go through RDS Proxy: the database sees a stable pool instead of a stampede of token validations. If a proxy is overkill for your case, a pooler like PgBouncer covers the same ground.

You still need a network path. IAM auth replaces the password, not the VPC. The database is still in a private subnet, so you need to reach it somehow: client VPN, SSM port forwarding to any instance in the VPC, or just running from workloads already inside the VPC. We went with SSM port forwarding for humans, which killed the bastion too, since SSM needs no open inbound ports and no SSH keys at all.

What actually improved

  • No shared secret. There’s no PEM file for the database and no password to pass around. Access is your IAM identity, full stop.
  • Per-user audit trail. exo_mercado connects as exo_mercado. Database logs and CloudTrail finally agree on who did what.
  • Credentials expire on their own. A leaked token is worthless in 15 minutes. Rotation stopped being a project and became a property.
  • Offboarding is one action. Disable the IAM user and every database they could reach is closed to them, immediately, without touching the database.
  • Deploys stopped carrying secrets. The Lambda artifact is code and nothing else. Rotating anything no longer touches a build pipeline.
  • One less server. The bastion is gone, along with its patch schedule.

The migration itself took an afternoon per database, most of it spent standing up the proxy, swapping the tunnel code for BeforeConnect, and updating runbooks. The hardest part was believing it would be that small. If you’re still passing a .pem file around to reach RDS, this is one of the rare security upgrades that makes daily life easier instead of worse.