gridpilot.gg/plans/website-testing-gap-closure.md

Concept: Close all testing gaps for apps/website

This plan defines a route-driven test strategy to prove the website is fully working without any visual UI tests (no screenshots/video/trace assertions). It leverages the existing unified Docker E2E environment described in README.docker.md.

1) Definition of done (route-driven)

For every route defined by routes and enumerated by WebsiteRouteManager.getWebsiteRouteInventory(), we prove:

1. SSR behavior (HTTP black-box)

   • Correct status: 200, 302, 404, 500.
   • Correct redirect targets for unauthenticated and wrong-role cases.
   • SSR HTML sanity markers and absence of Next.js error markers.

2. Client-side behavior

   • Client navigation works for representative transitions (no brittle selectors).
   • No unexpected console errors (allowlist only for known benign warnings).

3. RBAC correctness

   • Roles: unauthenticated, authenticated, admin, sponsor.
   • For every route: allowed roles can access, disallowed roles are redirected/forbidden as defined.

4. Negative cases are covered

   • Invalid route params (non-existent IDs).
   • Expired/invalid session.
   • API 5xx/timeouts and other failure injections.
   • Missing/invalid env configuration (fail fast rather than silently falling back).

The single source of truth is the route contract inventory, evolving from getWebsiteRouteContracts().

2) Architecture: one contract, three enforcement layers

flowchart TD
  A[Route inventory
  RouteConfig + WebsiteRouteManager] --> B[Route contracts
  RouteContractSpec]
  B --> C[Unit tests
  invariants]
  B --> D[Integration tests
  SSR HTTP black box]
  B --> E[E2E tests
  Playwright in Docker]

2.1 Single source of truth: Route contracts

Current implementation: getWebsiteRouteContracts()

Conceptual evolution:

• Extend the contract to be scenario-based:
  • unauth
  • auth
  • admin
  • sponsor
  • wrong-role variants
• Make expectations explicit per scenario:
  • expectedStatus: ok, redirect, notFoundAllowed, errorRoute
  • expectedRedirectTo (pathname)
  • SSR assertions: ssrMustContain, ssrMustNotContain, minTextLength
  • runtime assertions: consoleErrorPolicy (allowlist + denylist)

This keeps tests from becoming a pile of one-off scripts; instead, tests are generated from the contract.

2.2 Unit tests: invariants that keep the system honest

Target:

• Routing classification and RBAC rules:

  • routeMatchers.isPublic()
  • routeMatchers.requiresRole()

• API coupling correctness:

  • base URL selection via getWebsiteApiBaseUrl()
  • HTTP client error mapping via BaseApiClient

Desired outcome:

• A refactor of routing or API base URL logic breaks unit tests immediately.

2.3 Integration tests: SSR is an HTTP contract

We validate SSR using a black-box approach:

• Bring up the website server using the existing harness patterns under tests/integration/harness.
• For each route contract:
  • Issue an HTTP request to the website.
  • Assert status/redirect.
  • Assert SSR markers and absence of __NEXT_ERROR__ as already encoded in DEFAULT_SSR_MUST_NOT_CONTAIN.

Why this layer matters:

• It catches failures that E2E might hide (for example, client-side redirecting after a bad SSR response).

2.4 E2E tests: runtime and navigation, in unified Docker

We rely on the unified Docker stack described in README.docker.md and run tests via npm run test:e2e:website.

Key properties:

• Playwright visuals are disabled by configuration: playwright.website.config.ts.
• E2E must validate:
  • route loads and final URL matches expectation
  • RBAC works (unauth redirects, wrong-role redirects)
  • no unexpected console errors using ConsoleErrorCapture
  • representative navigation flows (href-based selectors) like navigation.e2e.test.ts

Additionally:

• Promote “nightly exhaustive” execution by enabling RUN_EXHAUSTIVE_E2E to include heavier suites (like website-pages.e2e.test.ts).

3) Close gaps by building a route coverage matrix

We produce a matrix derived from WebsiteRouteManager.getWebsiteRouteInventory():

• Rows: each route
• Columns: scenarios
  • unauth → expected redirect or ok
  • auth → ok or role redirect
  • admin → ok on admin routes
  • sponsor → ok on sponsor routes
  • invalid param → notFoundAllowed or inline error behavior
  • API failure injection → error boundaries behave, no runtime crash

This matrix becomes a checklist that prevents “coverage by vibe”.

4) Failure mode strategy (no visual tests)

4.1 Invalid IDs and missing resources

Use WebsiteRouteManager.getParamEdgeCases() to ensure invalid IDs lead to:

• 404 where appropriate, or
• 200 with an inline not-found message for CSR-heavy pages

4.2 Session drift and wrong-role

Drive auth contexts using WebsiteAuthManager.createAuthContext() and assert redirect behavior for:

• auth user hitting admin route
• auth user hitting sponsor route
• unauth user hitting protected routes

4.3 API failures and timeouts

Test website behavior when the API returns 5xx or times out:

• SSR path: prove the server renders an error boundary (or specific 500 route) with the expected status.
• CSR path: prove it does not crash and produces deterministic error messaging.

This should be done without mocking UI visuals; we assert status, URL, and text markers.

5) Test isolation: prevent real external calls

We must ensure tests never hit real third-party services (analytics, payments, email):

• For E2E: Playwright network interception with a denylist of external hosts.
• For Node tests: fetch interception in test setup like tests/setup.ts.

Outcome: if a new external dependency is added, tests fail fast with a clear message.

6) CI pipeline shape

One deterministic pipeline (PR gating):

• Lint: website:lint
• Typecheck: typecheck:targets
• Unit tests: test:unit
• Integration tests: test:integration
• E2E unified Docker: test:e2e:website

Nightly:

• Run exhaustive E2E with RUN_EXHAUSTIVE_E2E.

7) Proposed implementation sequence

1. Extend the route contract structure in RouteContractSpec to include per-role scenarios and explicit negative-case expectations.
2. Generate tests from contracts:
   • integration SSR suite
   • e2e runtime and RBAC suite
3. Add failure-mode suites (invalid IDs, expired session, API 5xx/timeout).
4. Add network denylist guards.
5. Wire CI scripts and keep nightly exhaustive separate.