GKE Quick Reference — Patterns, Commands, Gotchas

Read-Only Inspection Commands

These are safe to run in any environment. They do not mutate cluster state.

# Cluster info
kubectl cluster-info
kubectl get nodes -o wide
 
# Namespace overview
kubectl get pods -n <namespace>
kubectl get pods -n <namespace> -o wide    # includes node assignment
kubectl get all -n <namespace>
 
# Pod details
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> -c <container>    # specific container
kubectl logs <pod> -n <namespace> --previous        # previous crash
 
# Deployments and StatefulSets
kubectl get deployments -n <namespace>
kubectl get statefulsets -n <namespace>
kubectl rollout status deployment/<name> -n <namespace>
 
# Services and Ingress
kubectl get svc -n <namespace>
kubectl get ingress -n <namespace>
kubectl describe ingress <name> -n <namespace>
 
# Config and secrets (metadata only — don't dump secret values in logs)
kubectl get configmaps -n <namespace>
kubectl get secrets -n <namespace>
 
# RBAC
kubectl get rolebindings -n <namespace>
kubectl get clusterrolebindings | grep <namespace>
 
# Events (most useful for debugging)
kubectl get events -n <namespace> --sort-by='.lastTimestamp'
 
# Resource usage
kubectl top pods -n <namespace>
kubectl top nodes

GitOps Config Sync Troubleshooting

Rule: Never kubectl apply in a GitOps-managed cluster. Config sync will overwrite your change silently.

Common Errors

KNV2009 — Permissions error (namespace reconciler blocked)

Symptom: ALL resources in a namespace fail to sync
Cause: Missing RoleBindings for the namespace reconciler (often streaming operator RBAC)
Fix: Add the missing RoleBindings via code review, not kubectl
Key insight: This blocks EVERYTHING, not just the resource that needs the permission

Silent ignore — files exist but nothing happens

Symptom: YAML files in the namespace directory, but no resources created
Cause: Missing kustomization.yaml — config sync uses Kustomize to discover files
Fix: Create kustomization.yaml with the file listed in resources:
Key insight: Without the kustomization file, config sync doesn’t even look at the other files

Missing YAML — resources exist in kustomization but file is missing

Symptom: Sync error referencing a file that doesn’t exist
Cause: File was deleted but still listed in kustomization.yaml resources
Fix: Remove the entry from kustomization.yaml

Config Sync Timing

Changes land ~30-40 minutes after code review merge
Monitor via sync dashboard or kubectl get reposync -n <namespace>
Don’t panic if it’s not immediate — check the dashboard first

Kustomize Patterns

Base + Overlay Structure

base/                          # Shared across all namespaces
├── cloud_sql_proxy/           # Proxy deployment template
├── elasticsearch/             # ES cluster base config
├── hazelcast/                 # Clustering service
├── kafka/                     # Streaming resources
├── pega/                      # Application templates
└── kustomization.yaml

namespace_repos/<env>/         # Per-namespace overrides
├── kustomization.yaml         # References ../../base + local files
├── service_accounts.yaml      # Namespace-specific identity
├── pega_base.yaml             # Namespace-specific app config
└── ...

Kustomize Patch Types

JSON Patch (most common for overrides):

patches:
- patch: |-
    - op: replace
      path: /spec/template/spec/containers/0/args
      value:
        - "project:region:database-instance"
  target:
    kind: Deployment
    name: cloudsqlproxy

ConfigMap Generator (for file-based config):

configMapGenerator:
- name: kibana-config
  behavior: create
  files:
    - kibana.yml=files/kibana.yml.data
  options:
    disableNameSuffixHash: true

Managed Sections

Some automation systems manage specific sections of kustomization.yaml:

## ManagedSectionStart | service | KustomizationManaged
## DO NOT MODIFY/DELETE TAGS
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: app-namespace      # <-- automation manages this block
## ManagedSectionEnd | service | KustomizationManaged
 
resources:                    # <-- YOU manage everything below
- ../../base
- ./service_accounts.yaml
- ./app_config.yaml
 
patches:                      # <-- YOU manage patches
- patch: |-
    ...

Streaming Operator (Strimzi/Kafka) Patterns

RBAC Required Per Namespace

Every namespace that inherits Kafka from the base needs 3 RoleBindings:

strimzi-cluster-operator → ClusterRole strimzi-cluster-operator-namespaced
strimzi-cluster-operator-watched → ClusterRole strimzi-cluster-operator-watched
strimzi-cluster-operator-entity-operator-delegation → ClusterRole strimzi-entity-operator

Each binds two subjects:

The operator’s ServiceAccount (in the operator namespace)
The namespace reconciler’s ServiceAccount (in config management namespace)

Without these: KNV2009 blocks ALL syncs. Not just Kafka — everything.

Pause Reconciliation (for new namespaces)

# In kustomization.yaml patches:
- patch: |-
    - op: add
      path: /metadata/annotations
      value:
        strimzi.io/pause-reconciliation: "true"
  target:
    kind: Kafka
    name: pega-kafka

This prevents the operator from spinning up brokers before the namespace is fully configured.

Cloud SQL Proxy + Workload Identity Pattern

The Three Layers

1. GCP Service Account (cloud identity)
   └── email: cloudsqlproxy-user-<env>@project.iam.gserviceaccount.com
   └── IAM roles: cloudsql.client, secretmanager.secretAccessor

2. IAM Bindings (Terraform — auto-generated)
   └── Grants the roles to the GCP SA

3. Kubernetes Service Account (namespace identity)
   └── Annotation: iam.gke.io/gcp-service-account: <GCP SA email>
   └── Name: cloudsqlproxy-user (same across all namespaces)
   └── Pods using this KSA authenticate as the GCP SA

Cloud SQL Proxy Sidecar

# Runs as init container with restartPolicy: Always (sidecar pattern)
containers:
- name: cloud-sql-proxy
  image: cloud-sql-proxy:2.x-alpine
  args:
    - --auto-iam-authn     # Uses Workload Identity, not keys
    - --port=5432           # PostgreSQL
    - "project:region:instance-name"
  startupProbe:
    exec:
      command: ["nc", "-vz", "127.0.0.1", "5432"]

Certificate Renewal Pattern

Elasticsearch (PKCS12)

Step 1: Generate new CA + node cert inside an ES pod (/tmp only)
Step 2: Extract certs from pod to local machine (kubectl exec cat, not kubectl cp)
Step 3: Base64 encode and update the GitOps-managed YAML
Step 4: Update BOTH elasticsearch.yaml AND srs.yaml (SRS truststore must match)
Step 5: Increment restart-version annotation to trigger rolling restart
Step 6: Submit via code review → config sync → rolling restart
Step 7: Verify new cert serial on all pods after sync

Always update elasticsearch.yaml AND srs.yaml in the same change. SRS uses the same certificate as its truststore. If they don’t match, SRS can’t connect to Elasticsearch.

Dependency Order Checklist (New Namespace)

Use this checklist when provisioning a new namespace:

Production Gotchas (From Real Incidents)

Rolled-back changes may still be live. Always check current file state in the repository, not changelist history.
Auto-generated files have “DO NOT EDIT” headers. But some files that LOOK auto-generated are actually manual. Check the file history, not just the header.
Config sync doesn’t retry intelligently. If step 3 fails, it won’t automatically succeed when step 2 eventually syncs. You may need to re-submit or force a re-sync.
Certificate expiry is a silent killer. If some environments have renewed certs and others don’t, use the one with the newest cert when provisioning new namespaces.
Base inheritance means implicit dependencies. Kafka, Elasticsearch, and other shared resources come from the base config. If you don’t have the RBAC to manage them, they block everything.
The infrastructure automation’s AI tools can give contradictory answers. Always verify with actual changelist history when the answer matters for production.

Ram's Notes

Explorer