The problem
One client, two products: an NGO platform with a conventional backend and frontend, and a community application built as four backend microservices with its own admin interface. The naive answer is two environments and two sets of everything. That doubles the operational surface for an organisation that has no operations team.
I put both on a single self-hosted Kubernetes cluster with namespace separation, one shared data layer, and one delivery mechanism — so there is exactly one thing to learn, patch and back up.
Cluster and delivery
K3s on a self-hosted server, with ArgoCD pulling from a dedicated GitOps repository. Automated pruning and self-healing are both on, which is the setting that makes GitOps mean something: if someone edits a live resource by hand, the cluster reverts it. The repository is not a record of intent, it is the intent.
- Namespace layout separating shared infrastructure, each application, the secret manager and ArgoCD itself.
- Images built multi-stage from slim base images and published to a container registry, with the frontend served by a minimal nginx layer rather than a Node runtime.
- Health probes, horizontal pod autoscaling and PodDisruptionBudgets on the workloads that matter.
Shared infrastructure
PostgreSQL runs as a StatefulSet with Redis beside it in a shared namespace, exposed into each application namespace through alias services. The applications address it as if it were local to them; I operate exactly one database platform. Backups, upgrades and tuning happen once.
A self-hosted secret manager is the single source of truth for configuration, synchronising into Kubernetes secrets on a short interval. It authenticates with a machine identity, not a person's credentials — so nobody's departure breaks the platform, and nobody's laptop is a dependency.
When the tool fights you: the secret manager's web interface refused to create a machine identity. Rather than stall the deployment, I created the identity, membership and authentication records directly in its own database with a correctly hashed client secret — then documented the path so it was reproducible rather than folklore.
The microservices
Four services — users, chat, forum and notifications — plus an admin interface. I authored every manifest, the ArgoCD application that delivers them, and their secret synchronisation resources.
Authentication uses asymmetric JWT signing: the user service holds the private key and signs; every other service verifies with the public key. There is no shared secret to distribute, and a compromised forum service cannot mint tokens for anyone. That decision costs nothing at deploy time and removes an entire class of lateral movement.
The frontends were originally destined for a managed hosting platform, but the client required on-server deployment — so they were containerised and moved onto the cluster with everything else.
An account suspension, and the guardrails that followed
The client's original cloud account was suspended, which took document storage down with it. I migrated storage to a new account and reconfigured bucket CORS to restore uploads.
Then I made the failure mode impossible to repeat: budgets and alarms at a deliberately low threshold, with a policy that denies further spend at 100% of budget. For a non-profit, a hard stop is strictly better than a surprise invoice — the worst case becomes "the service paused" instead of "the money is gone".
Operational debugging
- Registry pulls returned 403. The CI job's default token didn't carry permissions for cross-repository image pulls; a scoped access token fixed it.
- A seed script failed on module resolution. Running it inside a mounted container with the right runtime and TLS library beat fighting the host's environment.
- A renamed schema field left a partially applied migration. Resolved by aligning the column and marking the migration applied — not by forcing a destructive reset on a database with real data in it.
- An organisation policy disabled deploy keys mid-project, so continuous delivery stayed on the side that retained write access while the client's side kept CI only.
A "networking" outage that wasn't
A developer's connection to the staging database started failing. I worked outward in order: checked firewall drop counters to rule out packet filtering, verified the SSH daemon's forwarding configuration, and created a dedicated no-login account for tunnelling so access didn't depend on a shell user.
The actual cause was a password authentication failure on the database role. The network had been fine the whole time. The walk was still worth it — ruling things out in order is what makes the final answer trustworthy instead of lucky.
Outcome
- Two applications delivered by GitOps onto one cluster, with drift automatically corrected.
- One shared data layer instead of two, and one secret store as the source of truth.
- Autoscaling and disruption budgets so routine maintenance is not an outage.
- Cloud spend capped by policy, not by vigilance.