Qwizin — DevOps, Containerization, Load Balancing & Orchestration

Status: Draft v0.1 · Date: 2026-07-20 Traces to: CON-14 (modular monolith), CON-15 (Kubernetes), ADR-0001 (Oracle OCI in-Kingdom), QAS-OPS-01, QAS-OPS-02, QAS-SCL-01/02/03

Environment anchor (verified 2026-07-20): Kubernetes v1.36.2; supported minors 1.34/1.35/1.36. Laravel 13 (released 2026-03-17); no LTS release exists — the last was Laravel 6 in 2019. All releases now get 18 months bugfix / 24 months security. PHP 8.4 recommended (8.3 is security-only since 2025-12-31; Laravel 13 requires ≥ 8.3).

⚠ Laravel has no LTS. This has a direct DevOps consequence that is easy to miss: there is no "install it and leave it for three years" option. A continuous upgrade cadence must be budgeted from day one — roughly an annual major upgrade. Laravel 12's bugfix window ends 2026-08-13, three weeks from now. Start on Laravel 13.

1. Container Strategy#

1.1 One image, many roles#

The API, queue workers, and scheduler are the same application. Building three images means three chances for version skew — a worker running yesterday's code processing today's job payloads is a genuinely nasty class of bug.

Build one image; vary the entrypoint.

1.2 Multi-stage build#

Stage 1 (composer)  → composer install --no-dev --optimize-autoloader
                                       --classmap-authoritative
Stage 2 (assets)    → node build (admin portal assets)
Stage 3 (runtime)   → php:8.4-fpm-alpine
                      + opcache, pcntl, intl, gd, redis, pdo_pgsql
                      + Arabic fonts (see §1.4)
                      + artisan config:cache route:cache view:cache event:cache
                      + USER 10001, chown app tree

Non-negotiables:

• --no-dev — PHPUnit, debug bars, and faker in a production image are attack surface with no purpose. A leaked debug toolbar is a classic PHP breach vector.
• Pin the base image by digest, not tag. php:8.4-fpm-alpine moves; php@sha256:… does not. Renovate/Dependabot automates bumps so pinning does not mean staleness.
• Explicit allow-plugins in composer.json. Composer plugins execute arbitrary code at install time — this is PHP's equivalent of npm install scripts and a real supply-chain path.
• Disable dangerous PHP functions in php.ini: exec, shell_exec, passthru, system, proc_open, popen. This closes the most common PHP RCE-to-shell path at the source.
• APP_DEBUG=false, APP_ENV=production — verified by an automated check, not by hoping. Laravel's debug error page leaks environment variables including database credentials. It is one of the most common Laravel breaches in the wild.

1.3 Build-time cache warming — required for a read-only root filesystem#

Running config:cache, route:cache, view:cache, and event:cache during the image build turns Laravel's runtime-writable bootstrap files into read-only baked artifacts. This is both a hardening requirement (§5) and a performance win.

⚠ The config:cache footgun. Once config is cached, env() returns null everywhere outside config/*.php. Audit for env() calls in application code before enabling this — it fails silently and in production. This is the single most common Laravel-on-Kubernetes mistake.

To eliminate the remaining writes:

Moving uploads to object storage is not merely hardening — legal documents and certificates need durability, encryption, and lifecycle policy that pod-local disk cannot provide.

1.4 Arabic font stack#

The Document Renderer image must embed Arabic fonts with full glyph coverage and correct shaping tables (QAS-LOC-01). This is a build-time image concern, not an application concern — a renderer without the fonts produces boxes or disconnected letterforms, and the failure only appears in generated certificates, not in tests that check for a non-empty PDF. Golden-image tests over Arabic fixtures belong in CI.

1.5 Runtime: PHP-FPM vs Octane#

Recommendation: start with PHP-FPM + OPcache preload. Do not adopt Octane at launch.

Rationale for deferring: Octane's persistent-worker model means a singleton holding tenant context can leak one company's data into another company's request — a direct violation of QAS-SEC-01, the system's highest-priority scenario, and a bug class that does not exist under FPM. Adopting Octane before the tenant-isolation model is proven and test-covered trades the top quality attribute for performance the launch scale does not require.

Revisit when: p95 latency approaches the QAS-PERF-01 budget under real load, tenant isolation has automated cross-tenant test coverage, and there is capacity to audit every singleton. FrankenPHP is listed first in Laravel 13's Octane documentation and is the likely choice at that point.

2. Kubernetes Topology#

2.1 Namespaces#

qwizin-video is deliberately isolated. Self-hosted media servers require hostNetwork, which is incompatible with PSA restricted. Rather than weakening the whole cluster's baseline, the exception is confined to one namespace with its own network policy. This is the correct pattern whenever one workload needs privilege: isolate the exception, never downgrade the baseline.

2.2 Node pools#

3. Load Balancing#

3.1 Three independent traffic paths#

The critical topology fact: video media never traverses the API path. Treating them as one system is the most common load-balancing error in a platform like this.

graph LR
    subgraph P1["Path 1 — API traffic (L7)"]
        C1["📱 Client"] --> WAF["WAF / DDoS"] --> LB1["L4 Load Balancer"]
        LB1 --> ING["Ingress Controller<br/>TLS 1.3 termination<br/>routing · rate limiting"]
        ING --> SVC["Service → API pods<br/><i>round-robin, readiness-gated</i>"]
    end

    subgraph P2["Path 2 — Static & downloads"]
        C2["📱 Client"] --> CDN["CDN"] --> OBJ["Object Storage<br/><i>signed URLs</i>"]
    end

    subgraph P3["Path 3 — Video media (L4/UDP)"]
        C3["📱 Client"] -.->|"WebRTC SRTP<br/><b>bypasses ingress entirely</b>"| MED["Media servers<br/><i>hostNetwork, UDP</i>"]
        C3 -.->|"TURN fallback<br/>TCP/TLS 443"| TURN["TURN relay"]
    end

    style MED fill:#d98c1f,stroke:#96610f,color:#fff
    style TURN fill:#d98c1f,stroke:#96610f,color:#fff

Path 1 — API. L4 cloud load balancer → ingress controller. TLS 1.3 terminated at ingress. Routing, rate limiting, and request-size limits applied here. Backed by readiness probes so pods leave rotation before they fail.

Path 2 — Static and bulk downloads. Template files and certificates are served from object storage via short-TTL signed URLs, fronted by CDN. Bulk file transfer must never pass through the API tier — it would consume PHP-FPM workers for the duration of each download, which is the fastest way to exhaust the request tier.

Path 3 — Video media. UDP, hostNetwork, no ingress, no service mesh. A parallel network path requiring its own firewall rules and its own security treatment. TURN fallback on TCP/TLS 443 is mandatory in production — users behind carrier-grade NAT (common on Saudi mobile networks) and restrictive corporate firewalls cannot connect without it. For a paid consultation, an unconnectable call is a refund and a churned customer.

3.2 Session affinity#

Not required, and deliberately avoided. The API is stateless: sessions in Redis, cache in Redis, uploads in object storage. Sticky sessions would undermine even load distribution and complicate rolling deploys. The one component with affinity requirements is the media tier, and it handles its own room-to-node routing.

4. Autoscaling#

4.1 Per-workload scaling signals#

Choosing the wrong signal is the most common autoscaling failure. CPU is rarely the right one here.

KEDA is recommended for queue-depth scaling, including scale-to-zero for the document renderer during quiet periods (QAS-OPS-02).

4.2 Predictive scaling for video ⭐#

Qwizin knows its future load. Consultations are booked in advance, so video demand is scheduled, not stochastic. This is architecturally unusual and worth exploiting.

Reactive autoscaling is a poor fit for media because:

• One media pod per node → the scaling unit is a node, with minutes of boot latency
• Scale-down cannot interrupt live calls. Draining means refusing new conferences and waiting for existing ones to end — up to the length of the longest call. For hour-long consultations, that is an hour of paying for a node you are trying to remove.

Recommended: the scheduler reads the appointment calendar and pre-warms media capacity ahead of booked windows, with reactive HPA as a safety net only. Capacity is ready before the appointment rather than arriving three minutes into a call the customer paid for.

4.3 Graceful shutdown — the Laravel-specific trap#

Queue workers must finish their current job before terminating. If terminationGracePeriodSeconds is shorter than the longest-running job, Kubernetes SIGKILLs mid-job on every deploy and every scale-down. For a job that generates a certificate or calls a payment API, that means lost work or an indeterminate external state.

• terminationGracePeriodSeconds must exceed the longest expected job duration
• Workers handle SIGTERM by finishing the current job and exiting — Laravel's queue worker does this correctly if given the time
• Long-running jobs should be chunked so the grace period stays bounded
• preStop hook to drain from service endpoints before shutdown begins

5. Security Baseline#

Full detail in deep-dives/security-architecture.md. The container/orchestration-specific essentials:

5.1 Pod Security Admission#

PodSecurityPolicy was removed in Kubernetes 1.25. The in-tree replacement is Pod Security Admission, GA since 1.25.

pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: v1.36   # pin — never "latest"
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted

Pin enforce-version: without it, a cluster upgrade can silently tighten enforcement and break deploys. Roll out warn + audit first, read the warnings, then enforce.

What restricted breaks for a PHP stack — and the fixes:

readOnlyRootFilesystem is not part of the restricted profile — a common misconception. It must be enforced separately via Kyverno or Validating Admission Policy.

5.2 Network policy#

Default-deny ingress and egress per namespace, with a mandatory DNS carve-out on both UDP and TCP port 53 (TCP is needed for large responses and DNSSEC). Forgetting the DNS carve-out is the number-one cause of "everything broke" after enabling default-deny egress.

⚠ AdminNetworkPolicy and BaselineAdminNetworkPolicy no longer exist under those names. In October 2025 they were merged into a single v1alpha2.ClusterNetworkPolicy. It remains alpha, out-of-tree, and just went through a breaking rename — do not build the security model on it. Use standard NetworkPolicy, and approximate admin-tier rules by simply not granting developers RBAC write access to NetworkPolicy objects.

The egress problem worth planning for: standard NetworkPolicy takes CIDRs, not DNS names. Payment gateways and video providers sit behind rotating anycast IPs, so a CIDR allowlist is fragile and will page someone at 02:00 when a provider re-IPs. Three options:

1. CNI-native FQDN policy — Cilium toFQDNs or Calico domain rules. The clean answer, and a strong reason to choose the CNI deliberately rather than accept a default.
2. Egress forward proxy with a domain allowlist — CNI-agnostic, and yields a single auditable egress log (useful evidence in a payments audit).
3. Broad ipBlock excluding RFC1918 — weak, but blocks lateral movement; acceptable as a week-1 stopgap.

5.3 RBAC#

• Auto-generated ServiceAccount token Secrets no longer exist (removed 1.24, cleaned up 1.29). Workloads receive short-lived, audience-bound, auto-rotated projected tokens. A manually created kubernetes.io/service-account-token Secret in a manifest is a long-lived non-expiring credential — treat any occurrence as a finding.
• automountServiceAccountToken: false — the Laravel pods do not talk to the API server. Set this on default in every namespace and per-workload.
• Per-workload ServiceAccounts, never shared default, so RBAC and audit logs are attributable.
• Watch for these effective-cluster-admin escalation paths: secrets:get/list, pods/exec, pods/attach, pods:create, escalate/bind, serviceaccounts/token, and any wildcard.

5.4 Secrets#

Landscape has shifted materially and both leading options carry caveats:

• External Secrets Operator paused releases in August 2025 when maintainers burned out and the backing company wound down. It recovered — new maintainers, formal governance, stable v1 API, monthly cadence, now at v2.8.0 (July 2026), CNCF-hosted. Healthy again, but this was a near-death event within 12 months. Check the stability tier of the specific provider you use — ESO's core is stable while individual providers range from stable to community-alpha.
• HashiCorp Vault moved to BUSL 1.1 in August 2023 and IBM closed its acquisition in February 2025. The licence has not been reversed. BUSL creates genuine ambiguity for a commercial SaaS.
• OpenBao — Linux Foundation fork under MPL 2.0, now materially diverging rather than merely tracking Vault. v2.6.0 (July 2026), commercial support available.

Recommendation: if OCI's managed KMS and secrets store are available in me-riyadh-1 (⚠ unverified — see ADR-0001 gap #1), use ESO pointed at it plus KMS v2 etcd encryption. If not, run OpenBao rather than Vault — MPL 2.0 removes the licensing ambiguity entirely, and its Kubernetes auth method provides workload identity independently of the platform. Budget honestly: OpenBao is a stateful HA service with an unseal ceremony and a real DR obligation.

Sealed Secrets is a reasonable week-1 stopgap to get plaintext out of Git, cheap to migrate off later.

5.5 Admission control#

Kyverno reached CNCF Graduated status on 2026-03-24 and is the clear 2026 choice — YAML policies rather than Rego, which is decisive for a team without a dedicated platform engineer. It also mutates and generates, and has built-in Cosign verifyImages.

Validating Admission Policy is GA (since 1.30) and MutatingAdmissionPolicy became stable in v1.36. Use both: VAP for simple in-tree invariants (no webhook, no cert rotation, no failure mode where a down webhook blocks all API writes), Kyverno for anything needing cross-object lookups, image verification, or generation.

⚠ Exclude kube-system and Kyverno's own namespace from webhook scope, and choose failurePolicy deliberately. A failurePolicy: Fail webhook that cannot reach its own pods bricks the cluster in a way that is then hard to fix.

6. CI/CD Pipeline#

graph LR
    A["Commit / PR"] --> B["Static analysis<br/>PHPStan · Pint<br/><b>Deptrac — module<br/>boundaries</b>"]
    B --> C["Tests<br/>unit · feature<br/><b>cross-tenant suite</b>"]
    C --> D["composer audit --locked"]
    D --> E["Build image<br/>multi-stage, digest-pinned"]
    E --> F["Trivy scan<br/>fail on HIGH/CRITICAL"]
    F --> G["SBOM<br/>CycloneDX + SPDX"]
    G --> H["Cosign sign<br/>+ attest"]
    H --> I["Push to registry"]
    I --> J["Deploy staging"]
    J --> K["Smoke tests"]
    K --> L["Deploy production<br/><i>manual gate</i>"]
    L --> M["Kyverno verifies<br/>signature + identity"]

    style B fill:#2d8659,stroke:#1c5638,color:#fff
    style C fill:#2d8659,stroke:#1c5638,color:#fff
    style F fill:#c94a4a,stroke:#8b2f2f,color:#fff
    style M fill:#c94a4a,stroke:#8b2f2f,color:#fff

6.1 Two gates that carry unusual weight here#

Deptrac (module boundaries). QAS-MOD-03 requires that Phase-2 extraction stay cheap, which requires boundaries that do not erode. A boundary enforced only by code review is a comment. Deptrac fails the build on cross-module violations — no cross-module foreign keys, no cross-module ORM traversal, no reaching past a published contract.

⚠ Package move: qossmic/deptrac is abandoned; the maintained package is deptrac/deptrac (v4.6.2, 2026-07-01).

Cross-tenant test suite. QAS-SEC-01 requires that every tenant-scoped endpoint reject access with another tenant's IDs. The suite creates two tenants with full object graphs, authenticates as A, requests every route with B's IDs, and asserts 403/404 and that no response body contains B's data. A new endpoint without coverage fails the build. This is the control that actually holds over time.

6.2 Image signing#

Cosign keyless signing via Fulcio is now the default, with signatures logged to the Rekor transparency log; Rekor v2 has GA'd. Keyless works cleanly when building in GitHub Actions or GitLab CI — the CI OIDC token is the identity and there is no key to leak.

Verify the identity, not merely the presence of a signature. Kyverno's verifyImages should assert the specific repository and workflow permitted to sign. Verifying "signed by someone" is nearly worthless.

Keyless requires reaching Fulcio and Rekor at both sign and verify time. If in-Kingdom egress is restricted, either allowlist those endpoints or use KMS-key-based signing.

6.3 Database migrations — the Laravel-on-Kubernetes trap#

Never run migrations in an initContainer. Every pod runs its init container, so a 3-replica rollout runs migrations three times concurrently — a race that corrupts schema state.

Correct approach:

• Migrations run as a Kubernetes Job, once per release, before the rollout
• Laravel's migration table provides advisory locking, but the Job must still be gated to a single execution
• Migrations must be backward-compatible, because a rolling deploy runs old and new code against the same schema simultaneously
• Destructive changes use the expand/contract pattern: add the new column → deploy code writing both → backfill → deploy code reading new → drop old in a later release. A column dropped in the same release as the code that stopped using it will break every old pod still serving traffic.

6.4 Deployment strategy#

Rolling updates with maxSurge: 1, maxUnavailable: 0 for the API. Readiness probes gate traffic; liveness probes restart genuinely stuck pods. Rollback must be a single command completing in under 10 minutes (QAS-OPS-01) — this is a tested procedure, not an aspiration.

7. Observability#

QAS-OPS-01 requires that a single on-call engineer diagnose a 02:00 incident without tribal knowledge. Kubernetes raises the operational floor, so this investment is mandatory rather than optional — it is the price of the CON-15 decision.

Kubernetes audit logging deserves explicit mention because it cannot be retrofitted after an incident — you cannot audit-log the past. Minimum policy: RequestResponse for secrets, configmaps, RBAC objects and pods/exec; Metadata for everything else; drop get/list/watch on endpoints, leases, and events, which would otherwise be 90 % of volume. Ship off-cluster immediately — audit logs stored only on a compromised cluster are worthless.

8. ⚠ Architectural Constraint: ZATCA Serialisation#

If ZATCA e-invoicing applies (OQ-11, unconfirmed), it imposes a constraint that directly conflicts with horizontal scaling and must be designed for rather than discovered.

ZATCA Phase 2 requires invoice hash chaining — each invoice embeds the previous invoice's hash — plus a monotonic counter. That is stateful, strictly ordered, and cannot be produced concurrently by multiple pods. A naive implementation across 3+ API replicas produces a broken chain and rejected invoices.

Required design: a single serialisation point per invoice sequence — a dedicated single-consumer queue with a distributed lock, not a horizontally-scaled code path. Raise this now; retrofitting ordering into a concurrent design is expensive.

Build-vs-buy: certified ZATCA solution providers exist. For a small team, integrating a provider is almost certainly correct over implementing ECDSA stamping, CSID onboarding, and XAdES signing in-house.

9. Staged Hardening Roadmap#

Ordered by risk reduction per hour, not by interest.

Week 1 — highest ratio, lowest breakage risk#

1. APP_DEBUG=false / APP_ENV=production verified in production
2. Plaintext secrets out of Git; gitleaks in CI; rotate anything ever committed
3. automountServiceAccountToken: false on default SA in every namespace
4. etcd encryption at rest
5. composer audit --locked + --no-dev, failing the build
6. Trivy in CI (one week at --exit-code 0 to gauge backlog, then enforce)
7. PSA warn + audit: restricted — collect the breakage list, enforce nothing yet
8. Audit RBAC for cluster-admin bindings and wildcards
9. Confirm the API server is not publicly reachable

Month 1 — structural controls#

10. PSA restricted enforced — the nginx/PHP-FPM work from §5.1
11. readOnlyRootFilesystem — the Laravel work from §1.3
12. Uploads and certificates to object storage with SSE and lifecycle policy
13. Default-deny NetworkPolicy with DNS carve-out, one namespace at a time
14. Per-workload ServiceAccounts with minimal namespaced Roles
15. Kubernetes audit logging shipped off-cluster
16. Kyverno in Audit mode
17. Digest-pinned base images + Renovate; explicit allow-plugins
18. kube-bench (CIS v1.12 profile); document accepted exceptions
19. Decide the secrets architecture — do not drift on Sealed Secrets indefinitely

Quarter 1 — depth#

20. Kyverno to Enforce (excluding kube-system and its own namespace); add VAP policies
21. Cosign signing + Kyverno identity verification
22. SBOM generation, attested and retained
23. Migrate to the chosen secrets backend; test a rotation before needing one
24. FQDN egress control to payment and video providers
25. Workload identity federation, or OpenBao Kubernetes auth
26. Registry-side rescanning so new CVEs against shipped images surface
27. Kubelet hardening; block pod → :10250
28. Runtime security only if someone will triage it — see below

On runtime security (Falco/Tetragon), a deliberate position: this is where hardening programmes usually die. The predictable failure is deploying Falco with default rules, receiving hundreds of daily alerts from normal container behaviour, and having the DaemonSet muted or removed within a month — leaving runtime security on the architecture diagram and none in reality, which is worse than absence because it creates false assurance.

If adopted: start with six high-signal rules, not the defaults. For a PHP workload the highest-value rule is "shell spawned in a container" — in a PHP-FPM pod that is near-certainly RCE via a webshell. Two weeks audit-only, routed to a channel with a named owner. If nobody will triage, spend the effort on items 10–19 instead. That is correct prioritisation, not a cop-out.

10. Open Items#