wip
This commit is contained in:
134
docs/TESTS.md
134
docs/TESTS.md
@@ -1168,4 +1168,136 @@ For the iRacing hosted-session automation, confidence is provided by these concr
|
||||
- Add **step E2E tests** when changing DOM/step behavior for a specific wizard step.
|
||||
- Add or extend **workflow E2E tests** when behavior spans multiple steps, touches authentication/session lifecycle, or affects confirmation/checkout behavior end-to-end.
|
||||
|
||||
By following BDD principles and maintaining clear test organization, the team can confidently evolve GridPilot while preserving correctness and stability, with a dedicated, layered confidence story for hosted-session automation.
|
||||
By following BDD principles and maintaining clear test organization, the team can confidently evolve GridPilot while preserving correctness and stability, with a dedicated, layered confidence story for hosted-session automation.
|
||||
|
||||
## Hosted-session automation layers
|
||||
|
||||
The hosted-session automation stack is covered by layered suites that balance real-site confidence with fast, deterministic fixture runs:
|
||||
|
||||
- **Real-site hosted smoke (opt-in)**
|
||||
- [`login-and-wizard-smoke.e2e.test.ts`](tests/e2e/hosted-real/login-and-wizard-smoke.e2e.test.ts:1)
|
||||
- Gated by `HOSTED_REAL_E2E=1` and exercises the real `members.iracing.com` login + Hosted Racing landing page + "Create a Race" wizard entry.
|
||||
- Fails loudly if authentication, Hosted DOM, or wizard entry regress.
|
||||
|
||||
- **Fixture-backed auto-navigation workflows**
|
||||
- [`full-hosted-session.autonav.workflow.e2e.test.ts`](tests/e2e/workflows/full-hosted-session.autonav.workflow.e2e.test.ts:1)
|
||||
- Uses the real Playwright stack (adapter + `WizardStepOrchestrator` + `FixtureServer`) with auto navigation enabled (`__skipFixtureNavigation` forbidden).
|
||||
- Drives a representative subset of steps (e.g., 1 → 3 → 7 → 9 → 13 → 17) and asserts each step lands on the expected wizard container via [`IRACING_SELECTORS`](packages/infrastructure/adapters/automation/dom/IRacingSelectors.ts:1).
|
||||
|
||||
- **Step-level fixture E2Es with explicit mismatch path**
|
||||
- Existing step suites under [`tests/e2e/steps`](tests/e2e/steps:1) now have two execution paths via [`StepHarness`](tests/e2e/support/StepHarness.ts:1):
|
||||
- `executeStepWithFixtureMismatch()` – explicitly sets `__skipFixtureNavigation` for selector/state-mismatch tests (e.g., cars/track validation).
|
||||
- `executeStepWithAutoNavigation()` – uses the adapter’s normal auto-navigation, forbidding `__skipFixtureNavigation`.
|
||||
|
||||
### `__skipFixtureNavigation` guardrails
|
||||
|
||||
To avoid silently masking regressions in auto navigation:
|
||||
|
||||
- **Allowed (`__skipFixtureNavigation` may be set)**
|
||||
- Step-level mismatch tests in [`tests/e2e/steps`](tests/e2e/steps:1) that call `executeStepWithFixtureMismatch()`, such as:
|
||||
- [`step-08-cars.e2e.test.ts`](tests/e2e/steps/step-08-cars.e2e.test.ts:1)
|
||||
- [`step-09-add-car.e2e.test.ts`](tests/e2e/steps/step-09-add-car.e2e.test.ts:1)
|
||||
- [`step-11-track.e2e.test.ts`](tests/e2e/steps/step-11-track.e2e.test.ts:1)
|
||||
|
||||
- **Forbidden (guarded; will throw if set)**
|
||||
- Any suite that must exercise `PlaywrightAutomationAdapter.executeStep()` auto navigation, including:
|
||||
- [`full-hosted-session.autonav.workflow.e2e.test.ts`](tests/e2e/workflows/full-hosted-session.autonav.workflow.e2e.test.ts:1) – uses [`executeStepWithAutoNavigationGuard`](tests/e2e/support/AutoNavGuard.ts:1) and will fail if `__skipFixtureNavigation` is present in the config.
|
||||
- Future workflow / overlay / validator E2Es that assert behavior across multiple steps should either:
|
||||
- Use [`executeStepWithAutoNavigationGuard`](tests/e2e/support/AutoNavGuard.ts:1), or
|
||||
- Call [`StepHarness.executeStepWithAutoNavigation`](tests/e2e/support/StepHarness.ts:1), which rejects configs that attempt to sneak in `__skipFixtureNavigation`.
|
||||
|
||||
### Hosted-session behavior coverage matrix (initial slice)
|
||||
|
||||
| Behavior | Real-site smoke | Fixture step E2Es | Fixture workflows |
|
||||
|----------------------------------------------|------------------------------------------------------------------------------|--------------------------------------------------------------|-----------------------------------------------------------------------------------|
|
||||
| Real login + Hosted landing | ✅ [`login-and-wizard-smoke.e2e.test.ts`](tests/e2e/hosted-real/login-and-wizard-smoke.e2e.test.ts:1) | ⛔ (fixtures only) | ⛔ (fixtures only) |
|
||||
| Step 3 – Race Information DOM/fields | 🔍 via hosted wizard modal in real smoke (presence only) | ✅ [`step-03-race-information.e2e.test.ts`](tests/e2e/steps/step-03-race-information.e2e.test.ts:1) | ✅ via step 3 in [`full-hosted-session.autonav.workflow.e2e.test.ts`](tests/e2e/workflows/full-hosted-session.autonav.workflow.e2e.test.ts:1) |
|
||||
| Cars / Add Car flow (steps 8–9) | 🔍 via Hosted page + Create Race modal only | ✅ [`step-08-cars.e2e.test.ts`](tests/e2e/steps/step-08-cars.e2e.test.ts:1), [`step-09-add-car.e2e.test.ts`](tests/e2e/steps/step-09-add-car.e2e.test.ts:1) | ✅ steps 7–9 in [`steps-07-09-cars-flow.e2e.test.ts`](tests/e2e/workflows/steps-07-09-cars-flow.e2e.test.ts:1) and autonav slice workflow |
|
||||
|
||||
### Real-site hosted and companion workflows (opt-in)
|
||||
|
||||
Real iRacing and companion-hosted workflows are **never** part of the default `npm test` run. They are gated behind explicit environment variables and npm scripts so they can be used in local runs or optional CI jobs without impacting day-to-day feedback loops.
|
||||
|
||||
#### Real-site hosted smoke and focused flows
|
||||
|
||||
- Smoke + wizard entry:
|
||||
- [`login-and-wizard-smoke.e2e.test.ts`](tests/e2e/hosted-real/login-and-wizard-smoke.e2e.test.ts:1)
|
||||
- Focused real-site wizard steps:
|
||||
- [`step-03-race-information.real.e2e.test.ts`](tests/e2e/hosted-real/step-03-race-information.real.e2e.test.ts:1)
|
||||
- [`cars-flow.real.e2e.test.ts`](tests/e2e/hosted-real/cars-flow.real.e2e.test.ts:1)
|
||||
|
||||
Run them locally with:
|
||||
|
||||
```bash
|
||||
HOSTED_REAL_E2E=1 npm run test:hosted-real
|
||||
```
|
||||
|
||||
Intended CI usage:
|
||||
|
||||
- Optional nightly/weekly workflow (not per-commit).
|
||||
- Example job shape:
|
||||
|
||||
- Checkout
|
||||
- `npm ci`
|
||||
- `HOSTED_REAL_E2E=1 npm run test:hosted-real`
|
||||
|
||||
#### Companion fixture-hosted workflow (opt-in)
|
||||
|
||||
- Companion-hosted workflow over fixtures:
|
||||
- [`companion-ui-full-workflow.e2e.test.ts`](tests/e2e/companion/companion-ui-full-workflow.e2e.test.ts:1)
|
||||
|
||||
Run it locally with:
|
||||
|
||||
```bash
|
||||
COMPANION_FIXTURE_HOSTED=1 npm run test:companion-hosted
|
||||
```
|
||||
|
||||
Intended CI usage:
|
||||
|
||||
- Optional companion-centric workflow (nightly or on-demand).
|
||||
- Example job shape:
|
||||
|
||||
- Checkout
|
||||
- `npm ci`
|
||||
- `COMPANION_FIXTURE_HOSTED=1 npm run test:companion-hosted`
|
||||
|
||||
These suites assume the same fixture server and Playwright wiring as the rest of the hosted-session tests and are explicitly **opt-in** so `npm test` remains fast and deterministic.
|
||||
|
||||
#### Selector ↔ fixture ↔ real DOM guardrail
|
||||
|
||||
For hosted-session automation, [`IRACING_SELECTORS`](packages/infrastructure/adapters/automation/dom/IRacingSelectors.ts:1) must match **either**:
|
||||
|
||||
- The current `html-dumps-optimized` fixtures under [`html-dumps-optimized/iracing-hosted-sessions`](html-dumps-optimized/iracing-hosted-sessions:1), or
|
||||
- The real-site DOM as exercised by the hosted-real tests above.
|
||||
|
||||
Manual workflow when the iRacing DOM changes:
|
||||
|
||||
1. Detect failure:
|
||||
|
||||
- A hosted-real test fails because a selector no longer matches, or
|
||||
- A fixture-backed step/workflow test fails in a way that suggests large DOM drift.
|
||||
|
||||
2. Refresh DOM fixtures:
|
||||
|
||||
```bash
|
||||
npm run export-html-dumps
|
||||
```
|
||||
|
||||
This script runs [`exportHtmlDumps.ts`](scripts/dom-export/exportHtmlDumps.ts:1) to regenerate `html-dumps-optimized` from the raw HTML under [`html-dumps`](html-dumps:1).
|
||||
|
||||
3. Re-align selectors and tests:
|
||||
|
||||
- Update [`IRACING_SELECTORS`](packages/infrastructure/adapters/automation/dom/IRacingSelectors.ts:1) to reflect the new DOM shape.
|
||||
- Fix any failing step/workflow E2Es under [`tests/e2e/steps`](tests/e2e/steps:1) and [`tests/e2e/workflows`](tests/e2e/workflows:1) so they again describe the canonical behavior.
|
||||
- Re-run:
|
||||
- `npm test`
|
||||
- `HOSTED_REAL_E2E=1 npm run test:hosted-real` (if access to real iRacing)
|
||||
- `COMPANION_FIXTURE_HOSTED=1 npm run test:companion-hosted` (optional)
|
||||
|
||||
This keeps fixtures, selectors, and real-site behavior aligned without forcing real-site tests into every CI run.
|
||||
|
||||
The intent for new hosted-session work is:
|
||||
|
||||
- Use fixture-backed **step E2Es** to lock DOM and per-step behavior.
|
||||
- Use fixture-backed **auto-navigation workflows** to guard `WizardStepOrchestrator` and `PlaywrightAutomationAdapter.executeStep()` across multiple steps.
|
||||
- Use **opt-in real-site smoke** to catch drift in authentication and Hosted Racing DOM without impacting default CI.
|
||||
|
||||
Reference in New Issue
Block a user