Skip to content

Consize — Testing Strategy

Testing is layered: pure logic is unit-tested with golden fixtures; behavior against real systems is integration-tested; the full loop is e2e-tested in a sandbox cluster. The safety engine (apply/verify/rollback) gets adversarial tests — if Consize can't be proven safe, it can't auto-apply.

1. Test pyramid

Layer Scope Runs Where
Unit Percentile math, policy, pricing, skip conditions, guardrail logic Every commit CI
Integration Collector ↔ Prometheus/k8s API, engine ↔ Postgres, verifier ↔ Prometheus Every commit CI (minikube/kind + compose)
E2E Full loop against a real cluster + real DB Nightly / pre-release Dedicated sandbox GKE cluster
Adversarial Rollback, concurrency, partial failure, guardrail bypass Pre-release Sandbox cluster
Property/fuzz Percentile + policy functions Every commit (quick) CI

2. Unit tests (Go)

Analysis engine — golden fixtures. Synthetic usage series with known values; assert recommended request/limit to the byte:

// fixture: usage that is exactly 1GB p95, 2.5GB p99
// policy: request = p95×1.2, limit = max(2×request, p99)
wantRequest := resource.MustParse("1228Mi")  // 1024Mi × 1.2, rounded up to MiB
wantLimit   := resource.MustParse("2457Mi")  // 2× request > p99

Cases: empty data, <5 days, single-day spikes, constant usage, zero usage, missing buckets (gaps), negative deltas, multi-container pods (aggregate), init containers (excluded), DaemonSets (excluded by default).

Policy: every configurable knob (headroom %, percentile target, step limit) has table tests for boundary values — 0, negative, absurd (headroom 500%).

Skip conditions: excluded label, protected namespace, data-loss-risk, insufficient data, unstable usage (p99/max ratio > threshold).

Pricing: known price fixtures (AWS price JSON sample, GCP catalog sample) → exact savings to the cent; cache staleness → confidence downgrade, never a wrong number.

Guardrails: pure-function tests for the decision matrix: - excluded workload + auto-apply label → BLOCK - step > 30% → SPLIT into sub-steps - protected namespace → BLOCK - concurrent apply in namespace → REJECT - no approval + mode=auto-not-enabled → WAIT_APPROVAL - dry-run → no write call issued

Fuzz: fuzz.Peek-style fuzzing on percentile bucket parsing and policy math (no NaN, no panic, monotonic request ≤ limit invariant after rounding).

3. Integration tests

Run against a real ephemeral stack (docker compose: engine + Postgres + a scrapeable Prometheus with recorded fixtures; k8s calls against kind/minikube).

  • Collector: pointed at a Prometheus with fixture series, upserts exactly the expected buckets; re-run is idempotent (row counts stable); backfill flag replays history.
  • Store: migrations run clean; upserts under concurrency don't duplicate; foreign keys hold.
  • Verifier: with fixture SLI series (baseline + post), verdicts match hand-computed results for PASS / FAIL / INCONCLUSIVE (missing data).
  • API: contract tests on the OpenAPI spec (request validation, error shapes, authz: read-only vs writer).

4. E2E (sandbox cluster)

Synthetic workloads with known usage, generated by a fixture exporter (a workload that emits CPU/memory at a fixed profile — e.g., 300 MB constant, 2 GB spikes).

Happy path (compute): 1. Deploy 10 fixture workloads with inflated requests (8 GB requested / 300 MB used). 2. Run collector + analyze → recommendations exist with correct targets and savings. 3. Apply in dry-run → diff only, no mutation. 4. Approve → rollout applies; verifier PASS after window; recommendation verified. 5. Savings dashboard shows realized savings = sum of verified applies.

Rollback path: workload with a latency bug deployed; apply rightsizing → error rate +50% → verifier FAIL → automatic rollback to previous values → rolled_back with evidence → alert fired. Assert rollout actually restored requests/limits.

DB surface (staging RDS): oversized instance seeded with low utilization → recommendation = one class down → apply refused outside maintenance window → apply within window with approval → verified; forced CPU saturation during verification → rollback to previous class.

Idempotency: apply twice with same recommendation → second is a no-op (already at target).

5. Adversarial tests

  • Concurrent applies: 20 parallel apply requests across namespaces → global + per-namespace limits hold; no double patch.
  • k8s API failure: kill the patch call mid-rollout (network policy on sandbox) → apply marked failed with partial state; verifier still runs; retry is safe.
  • Store failure: stop Postgres mid-apply → apply blocked before the write (never apply without audit trail).
  • Clock skew: collector with skewed clock writes buckets under the correct window_start (bucketing by source timestamp, not local time).
  • Guardrail bypass attempt: direct API call to apply an excluded workload → 403/409 with reason; UI cannot bypass (server-side enforcement).
  • Rollback storm: verifier flapping (inconclusive data) → no repeated rollback loops; state machine prevents re-apply while verification pending.

6. UI tests

  • Component tests (Vitest + Testing Library): savings chart rendering, recommendation cards, apply modal guardrail messages.
  • One Playwright smoke: login → dashboard loads → open workload detail → chart renders with real fixture API data.

7. Performance

  • Benchmark: analysis of 10,000 workloads completes under the nightly 15-min budget (bench target: < 5 min on CI-class hardware).
  • Load test API: 1,000 reads/s with 100 ms p99 budget (k6, in CI on the staging deploy).
  • Collector backfill: 30 days × 5,000 workloads completes without exhausting memory (bounded batch).

8. Test data hygiene

All fixtures synthetic; no real customer data ever. RDS e2e uses a dedicated test instance, destroyed after the run (Terraform lifecycle: prevent_destroy off, explicit destroy step in the pipeline).