All work

Case study 03

Self-Hosted Platform Operations

Being the entire platform team for a company that runs its own services instead of renting them. Mail, identity, CI, certificates, secrets — and the pager.

Role
Platform / infrastructure operations
Timeline
June 2026 — ongoing · 43 active days
Platform
Docker · Caddy · nginx · Stalwart · Authentik · Novu · Infisical · MinIO

The remit

A company running its own infrastructure rather than paying for managed equivalents: mail server, webmail, single sign-on, notifications, a social scheduler, self-hosted visual-regression CI, calendaring, and the application deployments on top of all of it.

My job was to keep it working and improve it without taking it down. Deployments run through a container orchestration CLI over SSH, with a reverse proxy handling virtual hosts and certificates, and a self-hosted secret manager as the source of truth for configuration.

Mail, in detail

Self-hosted mail is where infrastructure work stops being theoretical. Either messages land in inboxes or they don't, the feedback loop is unforgiving, and the failure mode is silent — mail that vanishes into spam folders looks exactly like mail nobody replied to.

  • Outbound sending was broken. The relay was configured for implicit TLS on the legacy port; switching it to STARTTLS on the standard submission port fixed delivery. Verified by sending real mail end to end, not by reading the config back.
  • Deliverability went from 9/10 toward a clean score by working through SPF, DKIM and DMARC — each one verified by DNS lookup rather than by trusting a control panel's green tick.
  • Leaked credentials for two shared accounts were rotated as soon as they were identified.
  • The deployment was made redeploy-safe by bind-mounting bootstrap configuration, so recreating a container stopped silently wiping settings.
  • A real certificate from the existing ACME setup replaced the self-signed one on the submission and IMAP ports — and I removed a recovery-admin backdoor the default configuration had left enabled. Defaults are not a security posture.

Secrets discipline

The standing rule was simple: secrets live in the secret manager, never in cleartext. Rules like that survive only if the correct path is also the easy path, so I built tooling for it — a command that exports an application's secrets directly into its production environment file. Nobody has to choose between doing it right and doing it quickly.

An incident I'll describe plainly: a search across configuration once printed a private key into a working transcript. The key was rotated immediately. Secret exposure isn't only about what you commit — it's about everywhere a value can be echoed, logged, scrolled past or captured. The fix is rotation, not embarrassment.

Services operated

  • Webmail for a non-technical team — which meant it had to be genuinely reliable, not merely running.
  • A social scheduler with public registration disabled, email-only onboarding, and a rate-limiting sidecar in front of it.
  • Self-hosted visual-regression CI, built from source with patches for object-storage compatibility and wired into a browser test runner.
  • Single sign-on and a notification service, plus calendaring — and the decommissioning of a scheduling tool that was no longer earning its keep.

Problems worth remembering

  • A database kept segfaulting on this particular virtualised CPU. I pinned to the previous stable major version rather than chase it. When the bug is in someone else's interaction with the hypervisor, the cheapest correct move is to step back one version and move on.
  • Reverse-proxy sidecars returned intermittent 502s. Root cause was DNS: the resolver ignored search domains, so short service names didn't resolve. Fully-qualified internal names fixed it. "Intermittent" was really "whichever request hit the unresolvable name".
  • An admin account was locked out of a service with no recovery path, so I reset the password hash directly in its datastore.

Review and coordination

Alongside operations I reviewed and approved pull requests across three infrastructure repositories, merging gated changes on request. My working rule was that approval and merge are different decisions — reviewing autonomously is fine; merging someone else's branch is a conversation.

One recovery worth recording honestly: merging a pull request with branch deletion enabled removed the base branch of another open request and auto-closed it. I recovered the work by cherry-picking it into a fresh pull request. It is a good argument for checking what depends on a branch before deleting it, and a better argument for writing down the mistakes you make on shared infrastructure.

I also built small internal tools to cut recurring toil — standup drafting, the secrets export command, and work-log harvesting — and kept cross-session coordination notes reconciled against the project board.

Outcome

  • Mail sending restored and deliverability hardened across SPF, DKIM and DMARC.
  • Real TLS on mail ports, a vendor backdoor removed, and leaked credentials rotated.
  • Deployments made safe to recreate, and secrets centralised with tooling that makes the right path the easy one.
  • A stack a single engineer can actually operate — documented, not memorised.

Next case study

ServingServers — container isolation