153 lines
7.8 KiB
Markdown
153 lines
7.8 KiB
Markdown
# 🧪 Instagram Bot Testing Standards
|
|
|
|
This project follows a strict **Test-Driven Development (TDD)** philosophy. We do not write features blindly; we ground our development in real-world observations and automated verification.
|
|
|
|
## 🔴🟢🔵 The TDD Workflow (Red-Green-Refactor)
|
|
|
|
Every new feature or bugfix should follow this cycle:
|
|
|
|
1. **RED**: Start by obtaining a **Real XML Dump** (using the Testing Toolkit) of the target UI state. Write a test that fails against this dump or a `--live` device.
|
|
2. **GREEN**: Implement the minimum amount of code (logic in `TelepathicEngine`, `QNavGraph`, etc.) to make the test pass.
|
|
3. **REFACTOR**: Clean up the code. Ensure it adheres to our [Best Practices](#-best-practices--no-gos), is well-documented, and doesn't introduce regressions.
|
|
|
|
> [!TIP]
|
|
> TDD-specific tests, regression fixes, and edge-case hardening should be placed in the `tests/tdd/` directory.
|
|
|
|
---
|
|
|
|
|
|
## 1. Mock Testing (Offline Mode)
|
|
This is the **default mode** used in CI/CD pipelines and for rapid local development iterations.
|
|
|
|
- **Concept**: The `DeviceFacade` is fully mocked. UI states are loaded from static XML fixtures located in `tests/fixtures/`.
|
|
- **Primary Benefit**: Extremely fast (<1s per test) and requires no physical device or emulator.
|
|
- **Command**:
|
|
```bash
|
|
pytest
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Fixture Management (The Toolkit)
|
|
To prevent offline tests from validating against outdated Instagram layouts, XML fixtures must be periodically synchronized. Use the **Testing Toolkit** located in `scripts/`.
|
|
|
|
### Interactive Guide (Full Sync)
|
|
This guide walks you through 13 critical views (Home, Explore, DMs, etc.) and automatically captures the required dumps.
|
|
- **Command**:
|
|
```bash
|
|
python3 scripts/sync_fixtures.py --config test_config.yml --interactive
|
|
```
|
|
|
|
### Single Fixture Update
|
|
If only a specific screen has changed:
|
|
- **Command**:
|
|
```bash
|
|
python3 scripts/sync_fixtures.py --config test_config.yml --fixture explore_feed_dump.xml
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Live Hardware Testing (Real Device Mode)
|
|
Validates that the bot correctly interacts with a real device (emulator or physical phone) by performing actual clicks and swipes.
|
|
|
|
- **Concept**: Disables mocks. `pytest` connects via ADB to the active device.
|
|
- **Primary Benefit**: Detects UI synchronization issues, animation delays, and ADB connection drops.
|
|
- **Command**:
|
|
```bash
|
|
pytest --live
|
|
```
|
|
*(Note: Uses the device ID specified in your config or `conftest.py` defaults).*
|
|
|
|
---
|
|
|
|
## 4. AI & LLM Validation
|
|
Verifies that the bot's "brain" (Telepathic Engine) still understands XML structures and correctly maps elements to actions (e.g., finding the "Like Button").
|
|
|
|
- **Concept**: Sends real prompts to your local LLM server (Ollama/Qwen).
|
|
- **Primary Benefit**: Protects against "prompt drift" or performance regressions after model updates.
|
|
- **Command**:
|
|
```bash
|
|
RUN_LIVE_AI_TESTS=1 pytest tests/integration/test_live_telepathy.py
|
|
```
|
|
|
|
---
|
|
|
|
## 5. E2E Functional Sequences
|
|
These tests simulate full automation loops to ensure that different components (Goap, Navigation, SAE) play together correctly.
|
|
|
|
- **Concept**: Targeted scenario tests that verify a complete user flow from start to finish.
|
|
- **Example Flows**:
|
|
- `tests/e2e/test_e2e_explore_feed.py`: Validates the full "Explore -> Analyze -> Interact" loop.
|
|
- `tests/e2e/test_e2e_dm_sequence.py`: Validates handling of message threads.
|
|
- **Command**:
|
|
```bash
|
|
pytest tests/e2e/
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Cognitive Benchmarking
|
|
Measures the "IQ" and latency of your LLM models to ensure they are suitable for autonomous navigation.
|
|
|
|
- **Concept**: Runs a series of 13+ UI-parsing scenarios against your configured models and scores them on accuracy and speed.
|
|
- **Benefit**: Identifies models that are too slow or "hallucinate" UI coordinates before you let them loose on your real account.
|
|
- **Command (Ollama)**:
|
|
```bash
|
|
python3 benchmarks/run_competitive_benchmark.py --all-ollama
|
|
```
|
|
- **Command (Existing Config)**:
|
|
```bash
|
|
python3 benchmarks/run_competitive_benchmark.py --config test_config.yml
|
|
```
|
|
|
|
---
|
|
|
|
## 7. Latency and Adaptive Snap Validation
|
|
Ensuring the agent handles slow network responses or missing feed markers (e.g., getting trapped in a Story) is critical for Full Self-Driving autonomy.
|
|
|
|
- **Concept**: Simulates UI rendering delays to trigger the `post_load_timeout` and verify the `Adaptive Snap` recovery logic.
|
|
- **Implementation**: When testing `bot_flow.py`, mock `_wait_for_post_loaded` or the underlying `device.dump_hierarchy()` to return an incomplete or missing feed XML (like `reel_viewer_root`) to verify the bot presses `back` or wobbles successfully.
|
|
- **Key Assertions**:
|
|
- Verify that `nav_graph.do('align')` or `device.press("back")` is called when `_wait_for_post_loaded` fails to find `FEED_MARKERS`.
|
|
- Validate that the timeout gracefully escapes loop-locks rather than blindly proceeding with bad UI state.
|
|
|
|
---
|
|
|
|
## 🛠 Troubleshooting
|
|
- **Device offline**: Ensure that `adb devices` lists your device and it is authorized.
|
|
- **LLM Timeout**: Verify that Ollama is running (`ollama list`) and the required model (e.g., `qwen3.5:latest`) is loaded.
|
|
- **Missing Fixture**: If a test fails with `MISSING REAL DUMP`, use the Toolkit (Step 2) to capture the missing screen.
|
|
- **Benchmark Failures**: If a model fails benchmarks, it is automatically marked as `is_unsuitable` and should not be used for critical navigation tasks.
|
|
|
|
---
|
|
|
|
## 💎 Golden Rules of Implementation
|
|
|
|
To maintain 100% reliability and "Tesla-level" autonomy, every developer (and AI agent) MUST follow these rules:
|
|
|
|
1. **Strict Green Light Policy**: All tests (both existing and new) MUST be green before a task is considered finished. No exceptions.
|
|
2. **No Fix Without a Red Test**: Never implement a fix or a feature without first having a failing test that demonstrates the problem or the missing capability.
|
|
3. **Explicit Test Summary**: Every completion summary must explicitly list exactly which tests were added or modified to verify the change.
|
|
4. **Exhaustive Edge-Case Coverage**: Consider and test for failure modes: "What if the DB is down?", "What if the screen is empty?", "What if the user is in a state we've never seen?".
|
|
5. **Efficient, Fail-Fast Testing**:
|
|
* Do not run the entire suite if you know where the failure is.
|
|
* Run targeted tests immediately after a change.
|
|
* Fail fast: fix the first failing test before moving to the next.
|
|
* Maintain a mental (or written) list of remaining failing tests to ensure none are forgotten.
|
|
|
|
---
|
|
|
|
## 💎 Best Practices & No-Gos
|
|
- **Use Golden Fixtures**: Always use real, freshly pulled XML dumps. If the Instagram UI changes, update the fixtures immediately using the Testing Toolkit.
|
|
- **Singleton Isolation**: Ensure all core singletons (`TelepathicEngine`, `GoalExecutor`) are reset between tests in `conftest.py`.
|
|
- **Hermetic Tests**: Each test must be independent. Ensure on-disk caches (JSON files) are wiped before each run.
|
|
- **Layered Validation**: Start with fast mock tests for logic, then verify with `--live` hardware tests for physical interaction.
|
|
- **Relative Pathing**: Use `os.path.join` relative to `__file__` for all fixture loading.
|
|
|
|
### ❌ No-Gos
|
|
- **"Lying" Mocks**: Never create hand-written or "guessed" XML structures. If you don't have a dump, pull a real one.
|
|
- **Hardcoded Absolute Paths**: Never use paths like `/Users/name/...`. These break CI and other developers' environments.
|
|
- **State Leakage**: Never rely on the side effects of a previous test. If a test fails, it should not cause subsequent tests to fail.
|
|
- **Implicit Timing in Mocks**: Do not use `time.sleep()` for UI waiting in offline tests. Rely on the `VirtualClock` or state-based assertions.
|
|
- **Mocking Navigation Logic**: In E2E tests, do not mock the internal decision-making of the `TelepathicEngine` or `GrowthBrain`. Force them to process real (fixture) data.
|