TumharaJob — GitOps & config pipeline

Where it started

TumharaJob was running in production and shipping, but the delivery path had the problems that accumulate when a platform grows faster than its operations:

• Secrets lived in the build specification — including payment gateway keys, in source control, readable by anyone with repository access.
• No declarative deployment. Cluster state was whatever the last person applied.
• Configuration changes meant a full redeploy, so nobody made them casually — which is its own kind of risk.

Getting secrets out of the repository

First move was the obvious one, in stages so nothing broke mid-flight: hardcoded values out of the build spec into build-time environment variables, then out of the CI provider entirely and into a managed secret store. From there I wrote a CI workflow that applies secrets into the cluster, so nobody needs cluster credentials on their laptop to rotate a key.

GitOps rebuild

With secrets handled, I restructured delivery around ArgoCD so that the repository is the source of truth and the cluster reconciles toward it. Sixteen improvements went in, the ones that mattered most being:

• App-of-Apps structure — one root application managing the rest, so a new service is a file rather than a console session.
• Horizontal pod autoscaling tuned to real traffic rather than defaults.
• PodDisruptionBudgets, so node maintenance and evictions stop taking the service down with them.
• Rolling update strategy configured deliberately instead of inherited.

I also removed a metrics stack that had been installed but never used. Unused observability is not free — it consumes cluster resources, produces alerts nobody reads, and creates a false sense of coverage.

Configuration that deploys itself

The piece I'm most pleased with. Backend configuration — not just secrets — now lives in the secret store, and the cluster picks it up on its own:

• A custom resource syncs a configuration folder from the secret store into a ConfigMap.
• A reloader controller watches that ConfigMap and triggers a rolling restart of the annotated deployment when it changes.

I verified it end to end rather than trusting the diagram: flipped a single test key in the secret store, watched the operator sync it, watched the reloader restart the deployment, confirmed a new ReplicaSet came up carrying the new value — then reverted it and watched the same path run backwards. No redeploy, no pipeline run, no one hand-editing YAML in a cluster.

The cluster's bootstrap is captured as code too, with the two credentials that must exist out-of-band documented as the commands to create them — never as values in a file.

Production incidents

Three that taught me something worth repeating:

• Every upload returned 413. The obvious suspect was the backend's body limit. The actual cause was that API routes were passing through the frontend's nginx catch-all, which had its own smaller limit. Raising it in the frontend config fixed it — the request never reached the service everyone was looking at.
• The kubelet kept crash-looping. A stale fstab entry re-enabled 6 GB of swap on a 2 GB server after every reboot, and the kubelet refuses to run with swap on. Disabling it ended the loop — a reboot-persistent problem that looked random because it only appeared after restarts.
• Intermittent CORS failures traced to a misconfigured base URL rather than any CORS policy.

Frontend cutover

Finally I moved the frontend off a CDN origin onto the Kubernetes cluster: verified DNS, the deployment and its pods, and the actual served bundle over HTTPS, with certificates issued automatically through an ACME challenge routed via the ingress controller. One less external dependency and one less place for configuration to drift.

The application work underneath

I also fixed the platform's real-time chat, which had three reported bugs — users flipping offline on refresh, messages not arriving live after a refresh, and delivery failing after a socket reconnect. All three turned out to be lifecycle and state-management problems rather than messaging logic: a reconnect listener bound to the socket instead of the Manager, an unmemoised callback in a dependency array tearing down listeners every render, and a history fetch that replaced state wholesale and wiped messages arriving mid-request.

Being the person who fixed those is also what made the platform work credible — I knew which services were sensitive to restarts before I wrote the restart policy.

Outcome

• Secrets out of source control and out of build configuration, applied to the cluster by pipeline.
• Declarative GitOps delivery with autoscaling, disruption budgets and a documented bootstrap.
• Configuration changes reaching production with no deploy and no manual step.
• Frontend served from the cluster with automated certificate renewal.