TL;DR
I designed and executed the end-to-end provisioning of isolated Kubernetes namespaces on a shared production GKE cluster for a Fortune 10 tech company. The work spanned infrastructure-as-code, GitOps config sync, database provisioning, identity management, message streaming, search services, and certificate management — all delivered through a strict 7-changelist dependency chain with zero direct cluster mutations.
The Problem
A new application team needed fully isolated environments (dev, QA, UAT) on an existing shared GKE cluster. “Isolated” meant:
- Separate database — dedicated Cloud SQL Postgres instance per namespace
- Separate codebase — no shared application state with existing tenants
- Separate users — different developers, testers, and end users
- Shared infrastructure — same cluster, same node pool, same operator stack
The application was a modern UI blueprint app with ~8 weeks of build time. Small app, but the infrastructure provisioning was complex because the platform had accumulated 2+ years of tooling, automation, and undocumented dependencies.
Constraints
- GitOps only — all changes via code review and config sync. No
kubectl apply, no manual cluster mutations. The config sync system continuously reconciles cluster state against the repository. Any direct change gets silently overwritten. - Shared operator stack — Kafka (managed by a streaming operator), Elasticsearch, and clustering services are shared across all namespaces via a base configuration that every namespace inherits.
- Automation platform — an internal infrastructure automation system generates Terraform, IAM bindings, static IPs, DNS, and service accounts from a central configuration file. Some files are auto-generated (“DO NOT EDIT”), others are manual.
- No new service accounts via manual request — the automation platform talks to the internal identity management system on our behalf. But understanding which files trigger which automation took significant research.
Architecture
Users → Auth Proxy → Global Load Balancer → GKE Cluster
│
┌───────────────────────────────────────────┐
│ Shared GKE Cluster (regional, 3 zones) │
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ ns: app │ │ ns: app │ │ ns: app │ │
│ │ dev │ │ qa │ │ uat │ │
│ │ │ │ │ │ │ │
│ │ web │ │ web │ │ web │ │
│ │ utility │ │ utility │ │ utility │ │
│ │ sql-prxy│ │ sql-prxy│ │ sql-prxy│ │
│ │ kafka │ │ kafka │ │ kafka │ │
│ │ ES+SRS │ │ ES+SRS │ │ ES+SRS │ │
│ │ cluster │ │ cluster │ │ cluster │ │
│ └────┬────┘ └────┬────┘ └────┬────┘ │
│ │ │ │ │
│ ┌────┴───────────┴───────────┴────┐ │
│ │ Shared: streaming operator │ │
│ │ (manages Kafka lifecycle) │ │
│ └─────────────────────────────────┘ │
└───────────────────────────────────────────┘
│
┌─────────┴──────────┐
│ Cloud SQL Postgres │
│ (per-namespace) │
└────────────────────┘
Per namespace: web tier, background processing (utility), batch reporting (BIX), Cloud SQL Proxy sidecar, Kafka brokers (3) + ZooKeeper (3), Elasticsearch (3-replica StatefulSet) + Search/Reporting Service, Hazelcast clustering, UI service, debug pod.
Shared: Streaming operator (manages Kafka across all namespaces from a dedicated namespace), base Kustomize configuration (inherited by all namespaces).
The Rollout Plan
Each step is a separate code change. Each depends on the previous completing + config sync (~30-40 min). This ordering is not arbitrary — it’s driven by hard dependencies.
| Step | What | Why This Order |
|---|---|---|
| 1 | Infrastructure config — add namespace definitions + database config to central config | Automation platform provisions: namespace dirs, Cloud SQL Terraform, IAM bindings, static IPs, DNS, service accounts (via internal identity system) |
| 2 | RBAC + Identity + Kustomize init — streaming operator permissions, secret manager access, Kubernetes service account (Workload Identity), initial kustomization file | Config sync needs the kustomization file to process ANY manifests. Identity must exist before database job can authenticate. Streaming operator RBAC must exist before namespace can manage Kafka resources. |
| 3 | Database creation — one-time job creates Postgres database + schemas + grants | Database must exist before the application installer can seed it. Job uses Secret Manager (CSI driver) for admin password — requires RBAC from step 2. |
| 4 | Application installer — seeds database with application rules and schema | Application cannot start without seeded database. Installer uses same Cloud SQL Proxy + Workload Identity path. |
| 5 | Search + reporting services — Elasticsearch cluster + Search/Reporting Service + UI service | Search service must be running before the main application starts (hard dependency). Certificates copied from an environment with recently renewed certs. |
| 6 | Full application deployment — web, utility, batch, clustering, ingress, monitoring config | Everything is ready: database seeded, search running, identity configured. This step also adds the base inheritance (../../base) and all Kustomize patches. |
| 7 | Cleanup + verification — delete one-time jobs, verify health, add monitoring probes | One-time jobs (database creation, installer) are no longer needed. Health verification before adding to monitoring. |
The kustomization file is the engine. Without it, config sync ignores all YAML files in the namespace — they sit as dead text in the repository. It must be created in step 2 and updated incrementally in every subsequent step as new files are added.
Key Technical Decisions
1. GitOps Over Direct Cluster Mutations
The config sync system continuously reconciles. Any kubectl apply gets silently overwritten within minutes. This means:
- Every change must go through code review
- Rollback = revert the code change
- Audit trail is automatic (repository history)
- But iteration speed is slower (~30-40 min per sync cycle)
2. Shared vs Per-Namespace Secrets
| Secret | Shared or Per-Env | Why |
|---|---|---|
| Kafka CA certificates | Shared (from base config) | Same CA for all brokers in the cluster |
| Kafka admin password | Shared (from base config) | Single operator manages all brokers |
| Elasticsearch TLS certs | Shared (same cert all nonprod) | Certificate SAN uses generic internal name, not namespace-specific DNS |
| Cloud SQL credentials | Per-namespace | Each namespace has its own database instance |
| Service account identity | Per-namespace | Workload Identity binds namespace-specific KSA to namespace-specific GCP SA |
The Elasticsearch certificate decision was non-obvious. Initial assumption was to generate fresh certs per namespace for isolation. Research showed the certificate’s Subject Alternative Name uses a generic internal name (DNS:security-master), not a namespace-specific service DNS. It was designed for cross-namespace reuse. Generating new certs would have been unnecessary work with no security benefit.
3. Service Account Three-Layer Model
Layer 1: Cloud Identity (GCP Service Account)
└── Created by: infrastructure automation (talks to identity management system)
└── Has IAM roles: database client, secret accessor
Layer 2: IAM Bindings (Terraform)
└── Created by: infrastructure automation (auto-generated, "DO NOT EDIT")
└── Grants roles to the GCP SA
Layer 3: Kubernetes Service Account
└── Created by: us (manual YAML in namespace)
└── Annotation maps KSA → GCP SA via Workload Identity
└── Pods use this KSA to authenticate as the GCP SA
Key learning: The infrastructure automation handles layers 1 and 2 automatically when the central config is processed. We only create layer 3 manually. Initial research gave contradictory answers about whether a separate manual identity provisioning step was needed — tracing the actual changelist history of the original environment setup proved it was fully automated.
4. Fresh Install vs Database Clone
Chose fresh install (empty database, seed with installer) over clone and clean (copy existing database, remove tenant data). Reasons:
- No risk of leftover data from another tenant
- Clean schema names (standard defaults, not migration-era names)
- The installer job is a well-understood, repeatable process
- Clone approach requires careful data scrubbing — error-prone for isolation
5. Streaming Operator RBAC
The streaming operator (manages Kafka) runs in a shared namespace but needs permissions in every application namespace. Without 3 specific RoleBindings per namespace, the namespace reconciler gets blocked with a permissions error that prevents ALL resources from syncing — not just Kafka resources.
This was discovered from a real incident. A sandbox namespace was completely blocked because it was missing these RoleBindings. The fix was a separate code change to add them. For new namespaces, these go in step 2 (before any application resources).
6. Kustomize Base Inheritance
Every namespace inherits from a shared base configuration containing:
- Cloud SQL Proxy deployment template
- Elasticsearch cluster base config
- Hazelcast clustering service
- Kafka resources (CR, secrets, metrics)
- Application deployment templates
- Monitoring/probing config
Per-namespace overrides via Kustomize patches:
- Cloud SQL connection string (points to namespace-specific database)
- Search service endpoint (namespace-specific Elasticsearch)
- Kafka reconciliation pause (prevent premature broker startup)
- Monitoring config (namespace-specific base URLs)
Failure Modes and Gotchas
-
Config sync rollbacks may not actually roll back. A changelist that was supposedly rolled back was still present in the current repository state. Always verify current file state, never trust changelist history alone.
-
The infrastructure automation creates directories but not all files. The gap between auto-created and manually-needed files is the danger zone. Some files that appear auto-generated (based on headers) are actually manual.
-
Streaming operator RBAC is the #1 blocker. Missing it doesn’t just break Kafka — it blocks ALL resource syncs in the namespace.
-
Certificate choice matters. If some environments have renewed certs and others don’t, copy from the one with the newest cert. An expiring cert is a ticking clock.
-
Base inheritance is powerful but invisible. Kafka, many shared resources come from the base. Don’t recreate what’s already inherited.
-
Kustomization file is NOT auto-created for new namespaces. The managed section tags (wrapping apiVersion/kind/namespace) are auto-managed, but the resources list, patches, and config generators are fully manual.
-
Secret Manager CSI driver needs RBAC first. The database config job uses SecretProviderClass to read the admin password. Without the secret manager rolebinding synced first, the job fails with “permission denied.”
-
Workload Identity must exist before database operations. The Cloud SQL Proxy sidecar uses Workload Identity to authenticate. Without the Kubernetes Service Account annotation mapping to the GCP SA, the proxy can’t connect.
-
The infrastructure automation’s AI assistant gave contradictory answers about service account creation. The first answer said to edit a specific YAML file. The second said a separate manual provisioning step was required. Tracing the actual changelist history of the original environment proved the first answer was closer to correct — the automation handles it end-to-end.
What I Would Do Differently
-
Document the automation platform’s behavior, not just its outputs. The biggest time sink was figuring out what the automation creates vs what’s manual. A decision tree (“if you add X to the config, the automation creates Y”) would have saved days.
-
Build a preflight checklist before the first changelist. Research everything before submitting anything. We did this and it paid off — but formalizing it as a reusable template would help future namespace provisioning.
-
Verify current state, not historical state. Several assumptions from documentation and changelist history were wrong. The source of truth is always the current repository, not the history.
Traffic Flow Through the Platform
Ingress (Users → Application)
User → Auth Proxy (SSO/SAML) → Global Load Balancer (GCLB)
→ Static IP (IPv4 + IPv6, auto-provisioned per namespace)
→ TLS termination at GCLB (self-signed cert)
→ Ingress resource (hostname-based routing)
→ /prweb → web Service (port 80) → web pods (8080)
→ /kibana → Kibana Service (5601) → Kibana pod
→ /stream-service → stream Service (7003) → streaming pods
Pod → Database (Cloud SQL)
Pod → Cloud SQL Proxy sidecar (localhost:5432)
→ Workload Identity auth (KSA → GKE metadata server → GCP SA token)
→ Private IP → Cloud SQL Postgres instance (per-namespace)
→ No JSON keys, no secret rotation — identity IS the credential
Pod → Search (Elasticsearch)
Pod → SRS service (namespace-internal DNS: srs.app-{env}.svc)
→ Elasticsearch cluster (HTTPS, PKCS12 mutual TLS)
→ 3-replica StatefulSet with persistent volumes
Pod → Kafka (Streaming)
Pod → Kafka bootstrap (pega-kafka-kafka-bootstrap:9093)
→ TLS + SCRAM-SHA-512 authentication
→ 3 brokers per namespace (managed by Strimzi operator)
→ Strimzi runs in shared namespace, manages Kafka across all app namespaces
Interview Talking Points
For detailed interview questions with answers, see 17 GKE Whiteboard Questions.
This project can answer these common interview questions:
- “Tell me about a complex infrastructure project you owned.” — End-to-end namespace provisioning with 7-step dependency chain, GitOps, and multiple infrastructure layers.
- “How do you handle multi-tenant isolation on shared infrastructure?” — Namespace isolation with separate databases, service accounts, and identity, on shared compute and operator stack.
- “Describe a time you debugged a production issue.” — Streaming operator RBAC incident that blocked an entire namespace, traced to 3 missing RoleBindings.
- “How do you make infrastructure changes safely?” — GitOps with code review, strict dependency ordering, incremental rollout, and verification gates between each step.
- “How do you handle conflicting information?” — AI assistant gave contradictory answers about service account creation. Resolved by tracing actual changelist history to find ground truth.
- “What’s your approach to documentation?” — Research-first: document every file, every find-replace, every dependency before writing the first changelist. Zero unknowns at execution time.