Microservices Security: Lessons from a 7-Phase Plan

In regulated fintech environments, security isn’t a feature — it’s a business requirement. Designing a hardening program over a 40+ microservice production platform, with multiple squads and zero downtime tolerance, is one of the most complex challenges I’ve faced as a technical leader.

This article documents the structured approach we applied: a 7-phase program that took six months to execute and transformed the security posture of the entire platform.

Why a Phased Plan

The natural temptation is “fix everything now.” But on a platform with 40+ microservices, 3 squads, and zero downtime tolerance, an abrupt change in credential management can impact dozens of services simultaneously in production.

A phased approach allows you to:

1. Isolate the blast radius of each change
2. Validate in staging before production
3. Give teams time to adapt their workflows
4. Document before executing (critical for regulatory audits)

The 7 Phases

Phase 1 — Secrets Inventory and Classification

Before changing anything, you need to know what exists. The first step was building a complete inventory of all secrets and environment variables across every Kubernetes namespace, classified by type and sensitivity level:

# Basic classification by pattern
kubectl get secrets --all-namespaces -o json | \
  jq '.items[] | {name: .metadata.name, namespace: .metadata.namespace, keys: [.data | keys[]]}'

This inventory is the foundation of the entire program. Without it, any hardening effort is blind.

Phase 2 — Bastion Host and Direct Access Restriction

A fundamental principle: no developer should have direct access to production pods. The solution was a dedicated bastion host accessible only via SSM Session Manager — no open SSH — with mandatory MFA and full session logging in CloudTrail.

# Terraform — Bastion with SSM
resource "aws_instance" "bastion" {
  ami           = data.aws_ami.amazon_linux_2023.id
  instance_type = "t3.micro"
  
  iam_instance_profile = aws_iam_instance_profile.bastion_ssm.name
  
  # No SSH ingress — SSM only
  vpc_security_group_ids = [aws_security_group.bastion_no_ssh.id]
  
  tags = { Name = "bastion-prod", Environment = "prod" }
}

Phase 3 — Migration to AWS Secrets Manager

Migrating secrets from Kubernetes to AWS Secrets Manager was the most technically complex phase. We adopted the External Secrets Operator pattern, which automatically synchronizes AWS SM secrets to Kubernetes Secrets in encrypted form:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: ClusterSecretStore
  target:
    name: db-credentials-k8s
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: payments/prod/db
        property: password

Phase 4 — Granular IAM per Microservice

A shared IAM role per namespace amplifies the blast radius of any compromise. With IRSA (IAM Roles for Service Accounts), each microservice gets its own IAM role with access only to the resources that specific service needs. Policies are reviewed quarterly.

The result: the blast radius of a compromise drops from “access to the entire platform” to “access only to that specific service’s resources.”

Phase 5 — Automatic Credential Rotation

Static credentials are a cumulative risk. AWS Secrets Manager supports automatic rotation with Lambda functions, eliminating the need for manual rotation:

def lambda_handler(event, context):
    secret_id = event['SecretId']
    step = event['Step']
    
    if step == 'createSecret':
        new_password = generate_secure_password(32)
        client.put_secret_value(
            SecretId=secret_id,
            ClientRequestToken=event['ClientRequestToken'],
            SecretString=json.dumps({'password': new_password}),
            VersionStages=['AWSPENDING'],
        )
    elif step == 'setSecret':
        update_database_password(secret_id, event['ClientRequestToken'])
    elif step == 'finishSecret':
        client.update_secret_version_stage(...)

Phase 6 — Network Policies and Segmentation

A notifications microservice shouldn’t be able to talk to the core banking service. Kubernetes NetworkPolicies restrict east-west traffic to only explicitly permitted flows:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-isolation
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: payment-core
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: api-gateway
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: database

Phase 7 — SAST/DAST and Security Gates in CI/CD

The final phase closes the loop: preventing problems from appearing in the first place. We integrated:

• Semgrep in GitHub Actions for SAST on every PR
• OWASP ZAP in the staging pipeline for DAST
• Trivy for Docker image scanning before pushing to ECR
• Checkov for IaC validation (Terraform, Helm charts)

If any of these gates fail, the PR cannot merge to main. No exceptions.

The Most Important Lesson

Security isn’t a project that ends. It’s a continuous process. The 7-phase plan gave us structure, but the real value came from the habits we installed in the teams: reviewing IAM quarterly, doing threat modeling before each major feature, and treating secrets with the same care as user data.

In regulated fintech, the question isn’t whether you’ll be audited. It’s when. And when that moment arrives, you want clear answers for every security architecture decision you made.