ADR-0003 — Kubernetes as Orchestration Platform
Status: Accepted — with a recorded reservation · Date: 2026-07-20 · Deciders: Amir Haroun (Tech Lead)
Related: CON-15, QAS-OPS-01, QAS-OPS-02, R-09, DevOps deep-dive, ADR-0001
Context#
The workload is: a stateless PHP API tier, queue workers, a singleton scheduler, three stateful or specialised sidecars (document renderer, malware scanner, search engine), and conditionally a media tier with hostNetwork requirements (ADR-0005).
Launch scale is near-zero, growing to ~10⁴ accounts and hundreds of concurrent video sessions over 12–18 months.
This decision was made by the architect as a stated constraint (CON-15). This ADR records it, its rationale, and — honestly — the tension it creates, so that tension is managed deliberately rather than discovered later.
Decision#
Kubernetes, using a managed control plane (OKE on Oracle Cloud per ADR-0001), with vanilla upstream primitives only.
Rationale#
| Reason | Detail |
|---|---|
| The workload genuinely has heterogeneous scaling profiles | API scales on RPS; workers on queue depth; the renderer on job burst; media on session count. Kubernetes expresses per-workload autoscaling natively. |
| Sidecar extraction (ADR-0002) needs an orchestrator | Four separately-deployed components with independent lifecycles is past what a single-host setup handles comfortably. |
| Portability under provider volatility | The KSA provider landscape is in flux — Azure arrives Q4 2026, AWS status unresolved. Vanilla Kubernetes is the most portable target available. |
| Phase-2 headroom | CONF-04's scope will add services. The orchestration decision would be forced eventually; making it once is cheaper than migrating mid-growth. |
| Predictive scaling is expressible | QAS-SCL-02 — video demand is known in advance from the booking calendar. Kubernetes makes calendar-driven pre-warming straightforward. |
⚠ The reservation, recorded honestly#
Kubernetes raises the operational floor, and this directly opposes two constraints:
CON-01— budget sensitivityQAS-OPS-01— operability by a small team at 02:00
A simpler orchestration (Compose/Swarm, or a managed PaaS) would have satisfied the launch workload with materially less operational surface. The decision to use Kubernetes is therefore an investment in the 18-month position, paid for from month one.
The consequence is not "we should be careful." It is that the observability and automation investment Kubernetes forces is now a stated requirement, not an optional extra. Choosing Kubernetes without paying that cost converts an infrastructure choice into a permanent operational tax — which is exactly how small teams end up with a cluster nobody can debug.
This is recorded as R-09.
Consequences#
Mandatory investments — the price of this decision#
These are not "nice to have". Without them the decision is net-negative.
| Requirement | Rationale |
|---|---|
| Correlation IDs end-to-end — API → queue jobs → sidecars → external calls → every log line | Without this, debugging upload→scan→review→approve→notify means correlating timestamps across four components by hand |
| Alerting on user-facing SLOs, not CPU graphs | Time-to-detect < 5 min (QAS-OPS-01) |
| Single-command rollback under 10 minutes, tested | An untested rollback is not a rollback |
| Runbooks for the top 10 predicted failure modes | Replaces tribal knowledge |
| Staged hardening ordered by risk-reduction-per-hour | DevOps §9 — resists the temptation to do the interesting things first |
Portability constraint#
Because the provider landscape is volatile, the deployment must not become Oracle-specific:
- Vanilla upstream primitives; no proprietary control-plane features
- PostgreSQL, Redis, S3-compatible storage — standard interfaces only
- Infrastructure as code with provider specifics isolated in modules
- No dependency on a proprietary queue, function runtime, or serverless container product
This trades some managed-service convenience for the ability to move. Given ADR-0001's verification gaps, that trade is correct.
Laravel-specific traps this decision introduces#
Each is a real failure mode that does not exist without an orchestrator:
| Trap | Mitigation |
|---|---|
Migrations in an initContainer — every pod runs it, so a 3-replica rollout runs migrations 3× concurrently |
Run as a Job, once per release, before rollout |
| Rolling deploys run old and new code against one schema | Migrations must be backward-compatible; destructive changes use expand/contract across releases |
terminationGracePeriodSeconds shorter than the longest job → SIGKILL mid-job on every deploy |
Grace period must exceed the longest job; split worker deployments by job duration so a notification deploy doesn't wait on a PDF job |
| PHP not PID 1 → never receives SIGTERM | exec form in the entrypoint |
| Liveness probe checking the database → a DB blip restarts every pod simultaneously | Readiness may check dependencies; liveness must not |
PSA restricted vs a writable filesystem |
Build-time cache warming + Redis + stderr + object storage (DevOps §1.3) |
Deliberately deferred#
| Deferred | Reconsider when |
|---|---|
| Service mesh | Many services with complex mTLS needs. Network policies cover the security requirement today; a mesh would add significant complexity against QAS-OPS-01. |
| Dedicated API gateway | Modules become real services (Phase 2) |
| Multi-cluster | GCC expansion with per-country residency |
| Runtime security (Falco/Tetragon) | Only when someone will triage the alerts. Deploying it unowned produces false assurance, which is worse than absence. |
Revisit if#
Operational burden demonstrably exceeds the team's capacity — measured as incident MTTR trending up, or cluster maintenance consuming more than roughly a fifth of engineering time. In that case a managed PaaS for the stateless tier, with the cluster retained only for media and stateful sidecars, is the pragmatic retreat.