Ram's Notes

GKE Platform Interview Questions — 17 Whiteboard Topics

20 min read

Source

Generated by Gemini 3.1 Pro based on my production GKE platform experience. Organized into 6 categories with my answers mapped to each.

Category 1: Architecture & Traffic Flow

Q1. The Pod Lifecycle

Question: Walk me through exactly what happens in the cluster from the moment your GitOps tool syncs a new Deployment manifest to the moment the container is running on a node. Mention API Server, etcd, Scheduler, Kubelet, Container Runtime.

My answer from experience:

  • Config Sync detects new/changed manifest in the OCI image → sends to API Server
  • API Server validates, writes to etcd (desired state stored)
  • Scheduler watches for unscheduled pods → evaluates node affinity, taints/tolerations, resource requests → assigns pod to a node
  • Kubelet on that node picks up the assignment → pulls container image (BinAuthz verifies signature) → Container Runtime (containerd) starts the container
  • Startup probe runs (e.g., nc -vz 127.0.0.1 5432 for Cloud SQL proxy sidecar) → once passing, pod is marked Ready
  • Service endpoints update → Ingress/GCLB starts routing traffic

My production evidence: I’ve seen every step of this fail — images failing BinAuthz (wrong registry), pods stuck Pending (resource pressure), sidecars not passing startup probes (Cloud SQL proxy can’t reach database because Workload Identity wasn’t configured).

Q2. The Ingress Flow

Question: Draw the architecture of an external request hitting your application. Trace the path from external DNS → Cloud Load Balancer → Ingress Controller → Service → Pod. Where does TLS termination happen?

My answer from experience:

User browser
  → cims-pega-gke-{env}.corp.goog (DNS — auto-provisioned by infrastructure automation)
  → Auth Proxy (SSO/SAML — authenticates user before traffic reaches cluster)
  → Global Cloud Load Balancer (GCLB)
    → Static IP (IPv4 + IPv6, auto-provisioned per namespace)
    → TLS termination HERE (self-signed cert at GCLB level)
  → GKE Ingress resource (hostname-based routing rules)
    → /prweb      → app-web Service (port 80) → web pods (port 8080)
    → /kibana     → kibana Service (port 5601) → Kibana pod
    → /stream     → stream Service (port 7003) → streaming pods
  → Pod receives unencrypted HTTP internally

Key detail: We use both IPv4 and IPv6 Ingress resources (separate Kubernetes objects, same routing rules). The infrastructure automation creates the static IPs from the central config. The self-signed cert at the GCLB is fine because the Auth Proxy handles real authentication upstream.

Q3. Service Discovery (CoreDNS)

Question: How does internal DNS (CoreDNS) work in Kubernetes? If Pod A needs to talk to Pod B, how does it resolve the IP, and how does kube-proxy route the traffic?

My answer from experience:

  • CoreDNS runs as a Deployment in kube-system — it serves DNS for all cluster-internal names
  • Pod A calls srs.cims-pega-gke-dev-gsn.svc.cluster.local (or short form srs.cims-pega-gke-dev-gsn.svc)
  • CoreDNS resolves this to the ClusterIP of the Service
  • kube-proxy (iptables/IPVS mode) intercepts traffic to that ClusterIP → load-balances across healthy pod endpoints
  • For Headless Services (like Elasticsearch StatefulSet), DNS returns individual pod IPs — no load balancing, direct pod-to-pod

Production examples I use daily:

  • elasticsearch-master.cims-pega-gke-{env}.svc:9200 — SRS → Elasticsearch
  • clusteringservice-service.cims-pega-gke-{env}.svc — Hazelcast discovery
  • pega-kafka-kafka-bootstrap:9093 — app → Kafka (internal, namespace-scoped)
  • srs.cims-pega-gke-{env}.svc — web tier → Search/Reporting Service

Gap to study: kube-proxy modes (iptables vs IPVS), DNS caching behavior, and how DNS resolution differs between pods and external headless services.

Category 2: Stateful Workloads (Kafka & Elasticsearch)

Q4. StatefulSets vs Deployments

Question: You ran Kafka and Elasticsearch. Why use a StatefulSet instead of a Deployment? Explain how network identity and Persistent Volume Claims (PVCs) behave differently during scaling or pod restarts.

My answer from experience:

  • StatefulSet: Ordered creation/deletion, stable network identity (pod-0, pod-1, pod-2), persistent storage per pod. When a pod restarts, it gets the SAME PVC reattached.
  • Deployment: No ordering, random pod names, shared/no persistent storage. Any pod can replace any other.

Why it matters for Kafka:

  • Kafka brokers have partition assignments tied to their broker ID → need stable identity
  • Each broker stores log segments on its PVC → must survive pod restarts
  • Ordered rolling restarts prevent data loss during upgrades

Why it matters for Elasticsearch:

  • ES nodes have stable cluster roles (master/data) → need stable identity
  • Index shards are on persistent volumes → must reattach after restart

My production numbers: Kafka = 3 brokers + 3 ZK (each with 100Gi SSD PVC). Elasticsearch = 3-replica StatefulSet (30Gi SSD PVC per node).

Q5. Storage Failure

Question: A node running an Elasticsearch data pod permanently dies. Walk me through how the cluster recovers. What happens to the underlying GCP Persistent Disk, and how does the new pod attach to it?

My answer from experience:

  • Node dies → pod marked Terminating → eventually evicted
  • GKE Cluster Autoscaler may provision a new node (or scheduler picks existing node with capacity)
  • StatefulSet controller recreates the pod with SAME ordinal (elasticsearch-master-2)
  • The PVC still exists (it’s a separate Kubernetes object, not tied to the node)
  • The PD (Persistent Disk) in GCP is zonal — new pod must be scheduled in the SAME ZONE as the disk
  • Once scheduled in the right zone, the PD detaches from the dead node and attaches to the new node
  • Pod starts, ES rejoins cluster, shard rebalancing begins (if replicas were available, no data loss)

Gap to study: What happens if the PD is corrupt? Regional PD vs zonal PD trade-offs. How GKE’s topology.gke.io/zone label affects scheduling.

Q6. Operator Pattern (Strimzi)

Question: You used the Strimzi operator for Kafka. Explain the Operator pattern. How does a Custom Resource Definition (CRD) work with a custom controller to manage complex stateful upgrades?

My answer from experience:

  • CRD: Extends the Kubernetes API. We define a Kafka CRD — now kubectl get kafka works.
  • Custom Controller (Strimzi): Watches for Kafka CR changes → reconciles actual state to match desired state
  • The lifecycle: We change the Kafka CR (e.g., update version: 3.9.0) → Strimzi operator detects the change → performs a rolling upgrade of all brokers in order (broker-2, broker-1, broker-0) → verifies each broker is ISR before proceeding

From my Kafka upgrade (3.5→3.9):

  1. First upgrade Strimzi operator (0.36→0.45) — this updates the CRDs
  2. Then update inter.broker.protocol.version in the Kafka CR — this triggers rolling broker restart
  3. Strimzi handles the entire rolling restart sequence — we just change YAML
  4. If a broker fails health check during upgrade, Strimzi pauses and waits

The pause-reconciliation pattern: For new namespaces, we annotate the Kafka CR with strimzi.io/pause-reconciliation: "true" so Strimzi doesn’t try to spin up brokers before the namespace is ready. This is a Kustomize patch in our overlay.

3 RBAC RoleBindings required per namespace: Without them, Strimzi can’t manage Kafka resources in that namespace. I learned this from a real incident — SBX namespace was completely blocked (KNV2009) because it was missing these bindings.

Category 3: Security, Identity & Isolation

Q7. Workload Identity (3-Layer Model)

Question: Draw the “3-layer service account model” for GCP Workload Identity. How exactly does a Pod get a token to authenticate to Cloud SQL without you storing a JSON key in a Kubernetes Secret?

My answer from experience:

Layer 1: GCP Service Account
  email: cloudsqlproxy-user-{env}@project.iam.gserviceaccount.com
  Created by: infrastructure automation (triggered by central config)
  Has IAM roles: cloudsql.client, cloudsql.instanceUser, secretmanager.secretAccessor

Layer 2: IAM Policy Binding (Terraform — auto-generated)
  Binding: KSA "cloudsqlproxy-user" in namespace "app-{env}"
           → can impersonate GCP SA above
  Type: roles/iam.workloadIdentityUser

Layer 3: Kubernetes Service Account (manual YAML)
  Name: cloudsqlproxy-user (SAME name across all namespaces)
  Annotation: iam.gke.io/gcp-service-account: <GCP SA email>
  Pods with serviceAccountName: cloudsqlproxy-user → authenticate as GCP SA

The flow (no JSON key anywhere):

  1. Pod starts with serviceAccountName: cloudsqlproxy-user
  2. GKE metadata server intercepts the pod’s token request
  3. Metadata server checks: does this KSA have a GCP SA annotation? → Yes
  4. Metadata server checks: does the IAM binding allow this KSA to impersonate? → Yes
  5. Returns a short-lived OAuth2 token for the GCP SA
  6. Cloud SQL proxy uses this token to authenticate to Cloud SQL
  7. Token auto-refreshes — no key rotation needed

Why this matters: Zero secrets stored in Kubernetes. No JSON key files. No secret rotation. The chain of trust is: Kubernetes identity → GKE metadata server → GCP IAM → Cloud SQL.

Q8. Multi-Tenant Isolation

Question: You provisioned namespaces for multiple apps. How do you ensure one compromised app cannot access another app’s database or consume all the cluster’s CPU? Discuss Network Policies, RBAC, and Resource Quotas/LimitRanges.

My answer from experience:

  • RBAC: Each namespace has its own ServiceAccount. Workload Identity binds it to a namespace-specific GCP SA. Even if a pod in namespace A could reach namespace B’s Cloud SQL, it can’t authenticate — different SA, different IAM bindings.
  • Separate databases: Each namespace gets its own Cloud SQL Postgres instance (not just a separate schema — a separate instance). Full database-level isolation.
  • Resource limits: Every pod spec has CPU/memory requests and limits set. Web tier: 4 CPU / 16Gi. Utility: 4 CPU / 16Gi. BIX: 9 CPU / 36Gi. Prevents one namespace from starving others.
  • Dedicated node pools with taints: ES and Kafka run on dedicated node pools with NoSchedule taints (dedicated=elasticsearch:NO_SCHEDULE, dedicated=kafka:NO_SCHEDULE). Web/utility pods can’t schedule on stateful nodes and vice versa.

Verified gaps in our platform (from codebase audit):

  • Network Policies: None exist. No NetworkPolicy objects anywhere in the repo. Any pod can theoretically reach any other pod’s IP across namespaces. Fix: add default-deny-all NetworkPolicy per namespace + explicit allow rules.
  • Resource Quotas: None exist. No ResourceQuota or LimitRange objects in any namespace. A runaway deployment could consume all cluster resources. Fix: add per-namespace quotas (e.g., requests.cpu: "10", limits.memory: 40Gi).
  • Pod Security Standards: No PSA labels on namespaces (pod-security.kubernetes.io/enforce: restricted not set). Switching to restricted would break pods needing root (e.g., ES init container for vm.max_map_count).

How to talk about this honestly: “We have strong identity isolation (Workload Identity + separate databases) and workload placement isolation (tainted node pools). The gaps are at the network layer — no NetworkPolicies — and resource governance — no namespace-level quotas. I’d prioritize adding both for the new GSN namespaces.”

Q9. Certificate Management

Question: Walk me through the automated certificate renewal process. How does cert-manager interact with your Ingress and Let’s Encrypt (or internal CA) to solve the ACME challenge and rotate the cert?

My answer from experience:

  • We DON’T use cert-manager or ACME/Let’s Encrypt. Our certs are managed differently:
    • Ingress TLS: Self-signed cert at the GCLB level. Auth proxy handles real authentication upstream.
    • Elasticsearch: PKCS12 certs generated manually inside an ES pod, then committed to GitOps.
    • Kafka: CA cert managed by Strimzi operator, shared from base config.

My ES cert renewal process (production-tested):

  1. Generate new CA + node cert inside an ES pod (elasticsearch-certutil ca + cert)
  2. Extract certs from pod (using kubectl exec + base64, NOT kubectl cp)
  3. Base64 encode and update both elasticsearch.yaml AND srs.yaml (SRS truststore must match)
  4. Increment restart-version annotation to trigger rolling restart
  5. Submit via code review → config sync → rolling restart (~30-40 min)
  6. Verify new cert serial on all pods after sync

cert-manager gap: I understand it conceptually (Issuer/ClusterIssuer → Certificate → auto-renewal) but haven’t configured it. This is a Tier 2 learning item — highly relevant for open-source/startup environments.

Category 4: GitOps & Configuration Management

Q10. GitOps Reconciliation

Question: Your ACM/Config Sync is managing the cluster state. What happens if a developer manually runs kubectl edit deployment and changes an image tag? How does the system react?

My answer from experience:

  • Config Sync continuously reconciles cluster state against the repository (OCI image)
  • The manual change goes through — it works immediately
  • But within the next sync cycle (~minutes), Config Sync detects drift: actual state ≠ desired state
  • Config Sync silently overwrites the manual change back to what’s in the repo
  • No alert, no notification — the change just disappears
  • The developer thinks their change stuck, but it’s gone

This is why we have a hard rule: No kubectl apply, no kubectl edit, no kubectl set image. All changes via code review → repo → sync. Read-only commands only (get, describe, logs, exec for inspection).

If we WANT the change: Update the YAML in the repo → submit for code review → merge → sync deploys it. Rollback = revert the code change.

Q11. Kustomize Architecture

Question: Show me how you structure your Kustomize directories for a multi-environment setup (Dev, Staging, Prod). How do you handle environment-specific configurations (like different Cloud SQL connection strings) using bases and overlays?

My answer from experience:

base/                              # Shared across ALL namespaces
├── cloud_sql_proxy/               # Proxy deployment template
├── elasticsearch/                 # ES cluster base config
├── hazelcast/                     # Clustering service
├── kafka/                         # Kafka CR + secrets + metrics
├── pega/                          # Application deployment templates
└── kustomization.yaml             # References all subdirs

namespace_repos/dev-gsn/           # Per-namespace overlay
├── kustomization.yaml             # namespace: app-dev-gsn
│                                  # resources: ../../base + local files
│                                  # patches: Cloud SQL, Kibana, Kafka
│                                  # configMapGenerator: Kibana config
├── pega_base.yaml                 # JDBC_URL, schema names, system name
├── service_accounts.yaml          # Workload Identity annotation
├── cims_pega_gke-ingress.yaml     # Static IPs, hostnames, services
└── ...

Environment-specific overrides via Kustomize patches:

patches:
# Cloud SQL connection string (different DB per env)
- patch: |-
    - op: replace
      path: /spec/template/spec/containers/0/args
      value: ["project:region:postgres-db-dev-gsn"]
  target:
    kind: Deployment
    name: cloudsqlproxy
 
# Elasticsearch endpoint (namespace-scoped)
- patch: |-
    - op: replace
      path: /spec/template/spec/containers/0/env/0
      value:
        name: ELASTICSEARCH_HOSTS
        value: "https://elasticsearch-master.app-dev-gsn.svc:9200"
  target:
    kind: Deployment
    name: kibana-kibana

Key insight: The managed section (apiVersion/kind/namespace) is auto-generated. Everything else — resources list, patches, configMapGenerator — is manual. This was a research finding; the documentation was ambiguous.

Q12. Secret Management in GitOps

Question: If you are using GitOps, you can’t commit plain-text secrets to Git. How did you inject secrets (like DB passwords or API keys) into your pods? (e.g., External Secrets Operator, SOPS, GCP Secret Manager integration).

My answer from experience:

  • GCP Secret Manager + CSI Driver: For high-sensitivity secrets (Cloud SQL admin password). The SecretProviderClass CRD mounts secrets from GCP Secret Manager directly into the pod filesystem. No secret data in Git.
  • Kubernetes Secrets in Git (base64): For shared/low-sensitivity secrets (Kafka CA cert, Hazelcast auth, Elasticsearch certs). These are base64-encoded in YAML and committed to the repo. Not ideal, but the repo is access-controlled and internal.
  • Workload Identity (no secrets at all): For Cloud SQL auth. The proxy uses Workload Identity — no password, no key file, no secret. The SA chain IS the credential.

The hierarchy:

  1. Best: Workload Identity — no secret exists at all
  2. Good: Secret Manager + CSI driver — secret exists but never in Git
  3. Acceptable (internal): Base64 Kubernetes Secret in access-controlled repo
  4. Never: Plain-text secrets in Git

Gap to study: External Secrets Operator (syncs from Secret Manager to K8s Secrets automatically), SOPS (encrypted secrets in Git), sealed-secrets.

Category 5: Scaling & Node Management

Q13. Autoscaling Conflicts

Question: Explain how the Horizontal Pod Autoscaler (HPA) and the GKE Cluster Autoscaler work together. What happens if the HPA scales up pods, but there is no room on the existing nodes?

My answer from experience:

  • HPA: Watches CPU/memory metrics → increases pod replicas when threshold exceeded
  • Cluster Autoscaler: Watches for pods stuck in Pending (unschedulable) → adds nodes to the pool
  • The chain: Load increases → HPA creates new pod replicas → scheduler can’t place them (no capacity) → pods go Pending → Cluster Autoscaler detects Pending pods → provisions new nodes → scheduler places pods → HPA is satisfied

Our cluster config (verified from component_cluster.tf):

  • 3 node pools (not 1!):
    • regular-nodes: min 3, max 5 — n2-custom-32-128000 (32 vCPU, 128 GB RAM) — web, utility, BIX pods
    • elasticsearch: min 1, max 3 — n2-custom-32-128000 — tainted dedicated=elasticsearch:NO_SCHEDULE
    • kafka: min 1, max 3 — n2-custom-32-128000 — tainted dedicated=kafka:NO_SCHEDULE
  • Regional (3 zones) — total max: 33 nodes across all pools
  • Current utilization with 9 namespaces: ~50% of max capacity

What I haven’t done: Configure HPA rules (our workloads use static replica counts). Set up VPA (Vertical Pod Autoscaler). Tune Cluster Autoscaler profiles (balanced vs optimize-utilization).

Q14. Workload Placement (Taints, Tolerations, Affinity)

Question: You have a mixed cluster with standard web apps and GPU-intensive ML workloads. How do you ensure the web apps don’t schedule on the expensive GPU nodes, and how do you ensure the ML pods only schedule on the GPU nodes? Discuss Taints, Tolerations, and Node Affinity.

My answer from ACTUAL production config:

We already do this — not for GPUs, but for stateful workloads (ES + Kafka). Same pattern applies:

Our production setup (from component_cluster.tf):

  • 3 node pools: regular-nodes, elasticsearch, kafka
  • Stateful pools are tainted: 
    elasticsearch pool: dedicated=elasticsearch:NO_SCHEDULE
kafka pool:         dedicated=kafka:NO_SCHEDULE
  • ES/Kafka pods have matching tolerations + node affinity in their YAML
  • Web/utility/BIX pods DON’T have these tolerations → can only schedule on regular-nodes

For GPU workloads (same pattern):

  • Taint the GPU nodes: gpu=true:NoSchedule
  • Add toleration to ML pods: 
    tolerations:
- key: "gpu"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"
  • Add node affinity to ML pods: 
    affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: cloud.google.com/gke-accelerator
          operator: Exists

Key insight: I’ve used taints/tolerations in production for workload isolation. GPU scheduling is the same mechanism — just swap dedicated=elasticsearch for gpu=true and add the accelerator affinity.

Gap: No hands-on GPU node pool provisioning or ML framework scheduling (Kubeflow, Ray, KServe).

Category 6: Failure Modes & Troubleshooting

Q15. CrashLoopBackOff

Question: A pod is stuck in CrashLoopBackOff. Walk me through your exact debugging steps. (Logs, describe pod, previous container logs, exit codes).

My answer from experience:

# Step 1: Check pod status and restart count
kubectl get pod <pod> -n <ns> -o wide
 
# Step 2: Describe for events (scheduling, image pull, OOM, etc.)
kubectl describe pod <pod> -n <ns>
# Look for: Events section, Exit Code, Reason, Last State
 
# Step 3: Current container logs (may be empty if crash is instant)
kubectl logs <pod> -n <ns> -c <container>
 
# Step 4: PREVIOUS container logs (the crash that happened)
kubectl logs <pod> -n <ns> -c <container> --previous
 
# Step 5: Check exit code
# Exit 1 = application error (check logs)
# Exit 137 = OOMKilled (check memory limits)
# Exit 0 = successful exit but restartPolicy: Always keeps restarting

Real incidents I’ve debugged:

  • OOMKilled (exit 137): Dev web pods killed because a custom portal skin was generating ~540 MB CSS. Fix: increase memory limits + fix the skin.
  • Cloud SQL proxy crash: Workload Identity not configured → proxy can’t get token → crash. Fix: create KSA with correct annotation.
  • Init container failure: ES init container needs vm.max_map_count kernel param → fails without it. Fix: add privileged init container to set the param.

Q16. Sidecar Failures

Question: Your Cloud SQL proxy sidecar is failing to connect, but the main application container is running. How do you troubleshoot this? What layers do you check? (IAM permissions, network egress, DB connection limits).

My answer from experience:

Checklist (in order):

1. Is the proxy container running?
   kubectl logs <pod> -n <ns> -c cloud-sql-proxy
   → Connection errors? Auth errors? Timeout?

2. Is Workload Identity configured?
   kubectl get sa cloudsqlproxy-user -n <ns> -o yaml
   → Check annotation: iam.gke.io/gcp-service-account
   → Missing annotation = proxy can't get a token

3. Does the GCP SA exist and have correct roles?
   → cloudsql.client + cloudsql.instanceUser
   → Check IAM bindings in Terraform (auto-generated)

4. Is the Workload Identity binding correct?
   → The IAM policy must allow the KSA to impersonate the GCP SA
   → roles/iam.workloadIdentityUser

5. Does the Cloud SQL instance exist?
   → Connection string: project:region:instance-name
   → Typo in instance name = connection timeout

6. Network — is private IP enabled?
   → Cloud SQL uses private IP only (no public)
   → Pod must be on the VPC with peering to Cloud SQL

7. Database connection limits?
   → Check Cloud SQL connections in GCP console
   → max_connections hit = new connections refused

Real incident: New namespace, Cloud SQL proxy in CrashLoopBackOff. Root cause: service_accounts.yaml hadn’t been synced yet — KSA existed but without the Workload Identity annotation. The proxy got no token, crashed, retried, crashed. Fix: verify CL 1.5 is fully synced before proceeding.

Q17. Control Plane Saturation

Question: What metrics would indicate that your GKE control plane is struggling, and how do you mitigate it?

My answer (conceptual + partial experience):

Indicators:

  • apiserver_request_duration_seconds increasing — API server is slow
  • etcd_request_duration_seconds increasing — etcd is the bottleneck
  • apiserver_current_inflight_requests at limit — request throttling
  • kubectl commands timing out or returning 429 (Too Many Requests)
  • Config Sync reconciliation falling behind (drift growing)

Common causes:

  • Too many watchers (every controller, every namespace reconciler, every operator watches the API server)
  • Large Secrets or ConfigMaps being watched frequently
  • CRDs with many instances (Kafka CRs across many namespaces)
  • Excessive list operations (poorly written controllers listing all pods in all namespaces)

Mitigations:

  • GKE auto-scales the control plane — but there are limits
  • Reduce CRD cardinality (we pause Kafka reconciliation in new namespaces — fewer watchers)
  • Namespace-scoped operators vs cluster-scoped (reduce blast radius)
  • Rate-limit CI/CD tools that hammer the API server

Additional context from codebase audit: Our platform uses Google Cloud Managed Service for Prometheus with PodMonitoring CRDs for Kafka, ES, Hazelcast, and Cloud SQL Proxy exporters (e.g., acm/base/kafka/exporter/podmonitoring.yaml). SLOs are defined in modules/monitoring/infra_slo.tf tied to cloudprober_success/counter metrics. We have a custom Cloudprober (Bazel-built, packages/cloudprober/BUILD) with Go probes: psql_probe (Cloud SQL), mx_probe (mail adapter), bug_probe (IssueTracker). So our monitoring is more sophisticated than I initially thought — I just haven’t operated the dashboards directly.

Bonus: CI/CD Pipeline (Discovered from Codebase)

Question: How does a container image go from code change to running in the cluster?

Answer (from image_workflows/cims_pega_gke_images.blueprint):

  1. Build & Attest: Code changes trigger Rapid workflows (localImageProdWorkflow). These build the container images and create Binary Authorization attestations.
  2. Push: Images pushed to us-central1-docker.pkg.dev/cims-pega-gke-artifacts/
  3. Config Sync: ACM pulls manifests from acm/namespace_repos/ directories (OCI image)
  4. Admission: GKE cluster has BinAuthz enabled (modules/glindagke_cluster/v2/binauthz.tf). It verifies the cims-pega-gke-prod-attestor signature before the Kubelet runs the pod.

Key detail: Every container image in our YAMLs uses SHA256 digests (not tags). BinAuthz enforces that only attested images from our Artifact Registry can run. An unattested image = pod rejected at admission.

Market Positioning (From Gemini)

The Problem in the Market

  • Data Scientists / ML Researchers: Great at Python, PyTorch, building models — but can’t deploy, scale, or secure them on Kubernetes
  • Traditional SREs / Platform Engineers: Know Kubernetes deeply but don’t understand ML workloads (GPU scheduling, distributed training, model serving)

Your Differentiator

You speak both languages. You can talk to an ML engineer about their LangChain agent and model weights, then translate that into a Kustomize overlay, provision GPU node pools, and set up Workload Identity for BigQuery/Cloud SQL access.

3 Roles to Target

  1. MLOps Engineer / ML Platform Engineer — Build the Internal Developer Platform (IDP) for data scientists. Abstract away Kubernetes so ML engineers deploy models easily (Kubeflow, Ray, KServe on GKE).
  2. AI Infrastructure Engineer — Manage GKE clusters designed for ML: GPU scheduling, high-throughput training, low-latency inference, network optimization.
  3. Platform Engineer (AI/Data Focus) — Central platform team onboarding AI initiatives. You’re the Kubernetes expert ensuring AI agents deploy securely using GitOps and IAM.

The Pitch

“I don’t just build models; I build the production-grade, secure, and scalable infrastructure required to run them. I can take an AI agent from a local Python script to a highly available, GitOps-managed deployment on GKE, complete with secure database access and automated scaling.”

Graph View

Backlinks