How I operate

Root cause over workaround

A restart usually clears the symptom, and that is exactly what makes it dangerous — you stop looking at the moment the evidence is still warm. Three from the last year:

A kubelet was crash-looping on a production node. The obvious read was memory pressure. The actual cause was a stale fstab entry re-enabling 6 GB of swap on a 2 GB server after every reboot — the kubelet refuses to run with swap on. It looked random because it only appeared after restarts.
Every upload returned 413. Everyone was looking at the backend's body limit. The requests were never reaching the backend: API routes passed through the frontend's nginx catch-all, which had its own smaller limit.
A developer lost access to the staging database and it was reported as a networking problem. I checked firewall drop counters, then SSH forwarding, then created a dedicated tunnel account. The cause was a password authentication failure on the database role. The network had been fine the whole time.

The pattern is the same each time: the layer everyone is staring at is rarely the layer that is broken. Ruling things out in order is slower on the day and much faster over a quarter.

Secrets live in one place, and leaking is a rotation not an apology

Every platform I run keeps secrets in a managed store rather than in build config, and syncs them into the cluster using a machine identity — not a person's credentials. Nobody's laptop is a dependency and nobody's departure breaks the platform.

Moving a secret out of source control does not un-leak it. Anything committed stays in history and has to be rotated at the provider. I have rotated leaked mail credentials, and rotated a private key after a config search printed it into a working transcript. Exposure is not only about what you commit — it is about everywhere a value can be echoed, logged or scrolled past.

Guardrails beat vigilance

A client's cloud account was suspended and took their document storage down with it. I migrated the storage — and then made the failure mode impossible to repeat: budgets and alarms at a deliberately low threshold, with a policy that denies further spend at 100% of budget.

A hard stop turns the worst case from "the money is gone" into "the service paused". For a non-profit that is the only responsible default. The same instinct is why I set PodDisruptionBudgets before they are needed and why abuse limits went into a marketplace before it had abusers.

The repository is the intent, not a record of it

Every cluster I run uses ArgoCD with automated pruning and self-healing switched on. That setting is what makes GitOps mean anything: edit a live resource by hand and the cluster puts it back. Without it you have a repository that describes what someone hoped was deployed.

The furthest I have taken this is a pipeline where backend configuration — not just secrets — syncs from the secret store into a ConfigMap, and a reloader restarts the deployment when it changes. I verified it by flipping one key and watching the operator sync, the reloader fire and a new ReplicaSet come up with the new value, then reverting it. No redeploy, no pipeline run, nobody editing YAML in a live cluster.

Assume the user is hostile when they are

Building a platform that runs strangers' containers on someone's GPU meant treating the workload as adversarial from the first design decision. The user's container runs no SSH daemon at all; access comes from sidecars that attach to its namespaces and drop privileges before handing over a shell.

Then I attacked it. Two adversarial pentests, roughly a thousand lines of tests running real attacks over real sessions rather than asserting on config — reading other processes' environments, filling the host disk, reaching the proxy admin API, escaping via cgroup release-agent. That produced 17 critical, 25 high, 38 medium and 5 low findings, all fixed, plus a threat model and a repeatable playbook.

Writing the exploit is the only honest way to know whether the defence works. Everything else is a belief.

Infrastructure nobody else can run is a liability

I write the runbooks, threat models and architecture decisions as I go, not afterwards. A cluster bootstrap is captured as code, with the credentials that must exist out-of-band documented as the commands to create them — never as values in a file.

The same reasoning is why I authored an end-to-end DevOps training curriculum and mentor the junior engineers on my team. A platform that depends on one person having memorised it is not a platform, it is a hostage situation. Being the only one who understands the system is not job security — it is an outage waiting for a holiday.