Introduction

While Kubernetes has established itself as the de facto standard for container orchestration, its extensibility is one of the core features that sets it apart from other platforms. At the heart of this extensibility are controllers, the control loops that drive almost every Kubernetes function — from managing Pods in a ReplicaSet to handling node failures.

Most Kubernetes users are familiar with writing Custom Resource Definitions (CRDs) paired with custom controllers to manage domain-specific resources. However, the abstraction goes deeper: not all custom controllers need CRDs, and not all controllers follow the canonical “reconcile” model as seen in the kubebuilder-scaffolded projects.

This article dives deep into advanced patterns for writing Kubernetes controllers, especially when you need to go beyond CRDs — for example, when managing native resources in non-standard ways, or coordinating across multiple types of resources simultaneously.

Controllers in Kubernetes: The Basics

A controller is a control loop that watches the state of your cluster and makes changes attempting to move the current state toward the desired state. Each controller typically watches one or more types of Kubernetes resources and reconciles actual state against a desired state.

Controllers are fundamentally structured around these components:

  • Informer: Subscribes to changes on resources.
  • Work Queue: Stores keys of resources that need to be reconciled.
  • Reconciler: Contains business logic to reconcile the desired and actual state.

Most tutorials and tools (like kubebuilder or operator-sdk) assume that you will:

  • Define a CRD (e.g., BackupJob).
  • Scaffold a controller that watches that CRD.
  • Implement reconciliation logic for that CRD.

This pattern works for a wide variety of use cases. But what if you don’t need a CRD? Or what if your controller must operate across several different native resource types? That’s when advanced patterns become essential.

Pattern 1: Writing Controllers Without CRDs

Controllers don’t require CRDs. A controller can watch any Kubernetes resource (like Pod, Service, Node, or even Event) and reconcile accordingly. This is particularly useful for:

  • Policy enforcement controllers
  • Multi-resource coordination
  • Infrastructure orchestration based on native types

Example: Node Annotator Controller

Consider a controller that watches Node objects and ensures that nodes running a certain version of the kernel are annotated for special scheduling.

nodeInformer := informers.Core().V1().Nodes()
nodeInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc: enqueueNode,
    UpdateFunc: func(old, new interface{}) {
        if !reflect.DeepEqual(old, new) {
            enqueueNode(new)
        }
    },
})

In your reconciler, you inspect the Node, detect kernel version, and patch annotations as needed.

The benefit here is performance: no need for CRDs, no need for users to manage another resource. It’s a “zero-footprint” controller in terms of Kubernetes API surface.

Pattern 2: Cross-Resource Reconciliation

Sometimes your desired state spans multiple resource types. For example, you may want to ensure that every Service of type LoadBalancer is backed by a specific NetworkPolicy and monitored by a sidecar Pod.

Here, your controller must:

  1. Watch Service objects.
  2. On reconcile, query or create associated NetworkPolicy and Pod objects.
  3. Handle deletion and ownership to avoid orphaned resources.

This becomes a multi-source controller.

Challenges:

  • Informers for each resource type.
  • Indexing relationships (e.g., Services to NetworkPolicies).
  • Handling fan-out updates.

The controller-runtime package (used by kubebuilder) supports watching secondary resources, but for complex relationships, a hand-written controller may offer more control and performance.

Pattern 3: Informer Factories and Shared Indexers

When writing high-performance controllers, consider using shared informer factories and custom indexers to avoid expensive API calls.

indexer.AddIndexers(cache.Indexers{
    "byOwner": func(obj interface{}) ([]string, error) {
        metaObj, err := meta.Accessor(obj)
        if err != nil {
            return nil, err
        }
        owner := metaObj.GetOwnerReferences()
        if len(owner) > 0 {
            return []string{owner[0].UID}, nil
        }
        return nil, nil
    },
})

This allows your controller to quickly find all dependent resources based on an owner reference or label selector — essential for high-scale clusters with thousands of objects.

Pattern 4: Controllers Outside the API Server

Sometimes, a controller’s source of truth is outside Kubernetes. For example, syncing firewall rules from a cloud provider to Kubernetes NetworkPolicy objects, or vice versa.

This requires:

  • External client watchers (e.g., cloud SDKs).
  • Reconciliation across boundaries.
  • Idempotent mutations, as eventual consistency and rate limits become critical.

These controllers often implement a bi-directional reconciliation pattern, similar to service meshes like Istio synchronizing across control and data planes.

Pattern 5: Event-Driven Controllers With Minimal Polling

Polling is expensive. A more modern design favors event-driven reconcilers using watch APIs and rate-limited work queues.

You can implement a controller that reacts only to actual changes using:

  • cache.SharedInformer
  • workqueue.RateLimitingInterface
  • client-go/tools/record for Event generation

Avoid calling .List() on each reconcile. Use Listers and Indexers backed by informers to read from local caches.

This makes your controller:

  • Faster (no network calls for reads)
  • Lighter (less load on API server)
  • More resilient to large-scale cluster churn

Pattern 6: Reconciliation as a DAG

When reconciling complex objects with multiple sub-resources, model your reconciliation as a directed acyclic graph (DAG) of stages.

For example, a custom resource may require:

  1. Validating spec
  2. Creating ConfigMap
  3. Deploying StatefulSet
  4. Registering metrics

Each step can be a node in a DAG. Errors or delays in one node halt the progression, allowing retries without duplicating prior work.

This architecture provides clarity, modular testing, and better observability (especially when integrated with tracing or logging at the stage level).

Debugging and Observability Tips

Advanced controllers require advanced debugging. Here are best practices:

  • Structured Logging: Use klog or zap with context-aware fields.
  • Prometheus Metrics: Track reconcile counts, durations, and error rates.
  • Event Recording: Emit Kubernetes Events to provide visibility into resource reconciliation.
  • Leader Election: Avoid double writes in HA setups.
  • Health Probes: Expose /healthz and /readyz endpoints with custom logic.

Conclusion

Kubernetes controllers are more than just CRDs and kubebuilder projects. They’re the backbone of every reconciliation loop in Kubernetes, and mastering advanced controller patterns allows you to build scalable, efficient, and powerful systems.

Whether you’re building policy engines, managing infra with native resources, or building multi-resource operators, understanding the deeper mechanics of controllers — informers, caches, work queues, and reconciliation graphs — is essential.

Controllers are where the real Kubernetes logic lives. The better you understand them, the more control you gain over the platform itself.