fix(e2e): purge FakeSAENormal and instantiate real Cognitive Stack #5

Open
mmintel wants to merge 0 commits from fix/e2e-deception-purge into main
Owner
No description provided.
mmintel added 19 commits 2026-04-30 10:46:09 +00:00
Introduces the GoalDecomposer class that bridges mission.strategy + plugin
capabilities into concrete, weighted Task objects. Each Task has a target
screen, budget, weight, and human-readable intent.

Key design decisions:
- Pure logic, zero LLM/device dependencies
- Strategy weights (aggressive_growth, community_builder, etc.) drive selection
- Plugins declare which screens they operate on (multi-screen map)
- Screens need BOTH an action route AND active plugin to be viable
- Frozen dataclass ensures Task immutability

12/12 TDD tests passing.
- GrowthBrain.select_task() uses weighted random from concrete Task objects
- Removed self.goals from Config (no longer reads goals: from config.yml)
- Mission + plugins are now the SSOT for bot behavior

The bot no longer receives abstract strings like 'Nurture my community' that
the LLM Brain can't operationalize. Instead, the GoalDecomposer generates
Task(browse_feed, HomeFeed, budget=7) which routes to concrete feed loops.

18/18 TDD tests passing.
Replace the old dual-path orchestrator (abstract goals vs legacy desires)
with unified GoalDecomposer-driven task routing:

1. GoalDecomposer reads mission.strategy + plugins config
2. Generates weighted Task objects (verb, target_screen, budget)
3. GrowthBrain.select_task() picks one probabilistically
4. Selected Task's target_screen routes through existing nav_graph
5. Feed loops + PluginRegistry handle the actual interactions

The abstract goals path (goal_executor.achieve('Nurture community'))
that caused infinite scrolling is now eliminated entirely.

Legacy desire fallback preserved for configs without plugins.

22/22 tests passing.
🔴 RED → 🟢 GREEN for 4 critical bugs found in production run 2026-04-29:

1. ResonanceEvaluator: Add null-guard for evaluate_post_vibe() return.
   When VLM returns truncated JSON, the function returns None. The caller
   now handles this gracefully instead of crashing with AttributeError.

2. ScreenIdentity POST_DETAIL: Replace broken 'and not selected_tab'
   condition with structural differentiator using main_feed_action_bar.
   Posts opened from feed retain feed_tab selected, which was causing
   misclassification as HOME_FEED → LLM fallback → OWN_PROFILE hallucination
   → permanent Qdrant cache poisoning.

3. ActionMemory VLM verification: When VLM returns JSON instead of YES/NO,
   treat as inconclusive (fall through to structural delta) rather than
   hard failure. Only return False when response explicitly contains 'no'.

4 new E2E regression tests, 75/75 pass, zero regressions.
- Resolved Bug #5: Fixed list vs str parsing in ResonanceEvaluator
- Resolved Bug #6: Use 'should_like' key for vibe score
- Resolved Bug #7: Guard TelepathicEngine against 'Follow' nodes for post media
- Resolved Bug #8: Implemented failed_bounds exclusion loop breaker in PerfectSnapping
- Resolved Bug #10: Corrected available_actions string parsing
- Validated with E2E regression suite (100% green)
🔴 RED: Tests proved obstacle_guard silently ignored system permission dialogs
   and foreign app overlays, causing the bot to get stuck on Android modals.
🟢 GREEN: obstacle_guard now dismisses OBSTACLE_SYSTEM and OBSTACLE_FOREIGN_APP
   with immediate back-press, preventing the interaction chain from running
   against non-Instagram UI elements.
🔵 REFACTOR: Fixed broken import in resonance_evaluator (was importing
   humanized_scroll from utils instead of physics.humanized_input).
THE ROOT CAUSE OF LYING TESTS:
81 E2E tests existed but ZERO tested the full plugin chain (execute_all)
against non-Instagram environments. Each plugin was tested in isolation,
but the integration between perception (SAE) and action (obstacle_guard)
was a complete blind spot.

NEW TESTS:
- test_chain_terminates_on_permission_dialog: Proves the full plugin chain
  terminates at obstacle_guard when a system permission dialog is detected.
  No downstream interaction plugins (resonance, like, follow) may fire.
- test_chain_terminates_on_foreign_app: Same for Chrome/browser takeover.
- test_chain_passes_through_on_normal_instagram: Sanity check that the
  chain runs normally on valid Instagram feeds.

These tests would have caught the 23:01 stuck-loop bug BEFORE it shipped.
PROBLEM:
All 81 existing E2E tests were disguised unit tests. They tested individual
components (IntentResolver, SpatialParser, SAE perception, single plugins)
in isolation. NONE ran the actual production workflow.

This meant:
- A broken import in resonance_evaluator? No test caught it.
- obstacle_guard ignoring OBSTACLE_SYSTEM? No test caught it.
- The full plugin chain running against a permission dialog? Never tested.

THE FIX — test_full_workflow_hostile_env.py:
These tests mock ONLY the Android device (XML dumps). Everything else is real:
- Real DopamineEngine
- Real SessionState
- Real Config
- Real PluginRegistry with ALL 18+ production plugins
- Real BehaviorContext built exactly like bot_flow.py:942-954

Tests:
1. test_feed_loop_iteration_recovers_from_permission_dialog
   → Runs the EXACT code path from bot_flow.py against a permission dialog
   → Asserts should_skip=True, zero interactions, BACK pressed

2. test_normal_feed_post_is_processable
   → Sanity check that the full pipeline processes normal posts

3. test_all_plugins_importable_and_instantiable
   → Catches broken imports (humanized_scroll bug) at registration time

4. test_plugin_chain_does_not_swallow_import_errors
   → Explicit import of every plugin module — any ImportError = test failure

RULE CODIFIED: E2E = mock ONLY Instagram. Everything else is REAL.
BEFORE: Giant monolith test files with duplicated device stubs, config
factories, and cognitive stack builders. The infrastructure was copy-pasted
into each file.

AFTER: Clean separation:
- conftest.py: E2EDeviceStub, e2e_device, e2e_cognitive_stack,
  e2e_session, e2e_workflow_ctx fixtures
- test_workflow_permission_dialog.py: Permission dialog recovery
- test_workflow_foreign_app.py: Foreign app (Chrome) recovery
- test_workflow_normal_feed.py: Normal post processing
- test_workflow_plugin_integrity.py: Import + interface sanity

Each workflow file is <100 lines. Zero duplication. Every test uses
the shared e2e_workflow_ctx fixture which runs the EXACT code path
from bot_flow.py:942-971.

DELETED: test_full_workflow_hostile_env.py, test_plugin_chain_hostile_env.py
(monolith files replaced by the above)
NEW WORKFLOW TESTS (all using shared conftest fixtures):

Feed workflows:
- test_workflow_reels_feed.py — Reels feed processing
- test_workflow_explore_feed.py — Explore grid processing
- test_workflow_stories_feed.py — Stories feed + story view
- test_workflow_search_feed.py — Search feed processing
- test_workflow_dm_inbox.py — DM inbox + thread
- test_workflow_post_detail.py — Post detail + carousel
- test_workflow_profiles.py — User profile, scraping, followers, unfollow

Obstacle workflows:
- test_workflow_instagram_modal.py — Survey/rating modals
- test_workflow_locked_screen.py — Locked screen detection

Production regressions from 2026-04-30 session:
- test_workflow_stuck_other_profile.py — Bot stuck cycling through
  plugins on OTHER_PROFILE, all returning 'Cannot X on other_profile'
  (7x same profile, 35x snapping attempts)
- test_workflow_same_post_loop.py — Infinite snapping loop when
  device returns identical XML after each swipe

Total workflow coverage: 24 tests across 13 files, <100 LoC each.
PRODUCTION BUG 2026-04-30 11:50:
VLM returned action_bar_button_back (desc='Back') for 'tap profile tab'
intent on OTHER_PROFILE screen. This caused account_switcher to navigate
AWAY from the profile instead of opening the account selector bottom
sheet, halting the session.

Root cause: _visual_discovery pre-filter did not exclude navigation
buttons (Back, Close) when the intent was for a tab element.

Fix:
- Added filter_navigation_conflicts() method to IntentResolver
- Guards tab intents by excluding nodes with 'back' in resource_id
  or content_desc == 'Back'
- Does NOT apply for 'tap back button' or 'press back' intents

TDD:
- RED: test_intent_resolver_back_guard.py — 2 tests that call
  filter_navigation_conflicts on real OTHER_PROFILE XML
- GREEN: filter_navigation_conflicts method + wired into _visual_discovery
- E2E: test_workflow_account_switch.py, test_workflow_vlm_tab_confusion.py

31/31 tests passing.
PRODUCTION FIX 2026-04-30 12:45:
The entire E2E suite was systematically lying by testing a shell of the bot:
1. FakeSAENormal was masking the actual SituationalAwarenessEngine, meaning structural perception failures (like the 'stuck on Other Profile' bug) were completely invisible to tests.
2. The e2e_cognitive_stack was hardcoded to return None for 80% of engines (NavGraph, ZeroEngine, Telepathic, Darwin, ActiveInference), causing plugins to silently bypass their core logic during tests.
3. get_screenshot_b64 and screenshot() returned None, preventing the VLM stack from even running structural fallback code.

Fix:
- Systematically stripped FakeSAENormal from all 29 E2E workflow tests.
- Built e2e_cognitive_stack_factory to instantiate the REAL Cognitive Stack (GrowthBrain, QNavGraph, ZeroLatencyEngine, etc.) just like in bot_flow.py.
- Patched ONLY the external LLM network requests in conftest.py via mock_llm_network_calls so that structural execution stays 100% real but avoids non-deterministic LLM timeouts.
- Injected a valid 1x1 black image for screenshots.

All 29 E2E tests now execute the true production logic path and pass in 49 seconds.
mmintel added 1 commit 2026-05-01 10:09:48 +00:00
- test_device_connection.py: replace MagicMock with SimpleNamespace to satisfy mock ban
- test_is_ad_substring.py: add feed context markers — is_ad only checks exact labels in feed context
- pre_commit_tests.sh: smart E2E test discovery by module name words, preventing false coverage failures
- conftest.py: fix profile tab visual discovery regex (case-insensitive desc match)
- test_production_bug_regression.py: fix TelepathicEngine singleton poisoning via monkeypatch
mmintel added 1 commit 2026-05-01 14:55:59 +00:00
mmintel added 1 commit 2026-05-01 14:57:10 +00:00
mmintel added 1 commit 2026-05-01 19:58:02 +00:00
Bug 1 — VLM Profile Tab Hallucination:
  - VLM confused 'Profile' nav tab with post author username
  - Added Author Tab Guard to filter_navigation_conflicts()
  - Fixed mock LLM to use structural row_feed_photo_profile matching
    instead of hardcoded username (was the test lie)

Bug 2 — Like Button 1-Byte Delta Kill:
  - Toggle actions (like/save) produce 1-byte XML deltas
  - GOAP's MIN_UI_CHANGE_BYTES=50 threshold killed them as 'no change'
  - Split interaction gate: interactions use ANY xml diff, navigations
    keep the 50-byte threshold

Bug 3 — Empty Username Silently Accepted:
  - PostDataExtraction returned 'Post by @' with no warning
  - Added 'username_missing' reliability flag to result dict
  - Downstream consumers can now detect degraded data quality

Cleanup:
  - Removed all debug print() statements from production code
  - Replaced with structured logger.debug() calls
mmintel added 1 commit 2026-05-01 20:27:40 +00:00
mmintel added 1 commit 2026-05-01 21:41:58 +00:00
mmintel added 1 commit 2026-05-01 22:26:39 +00:00
mmintel added 1 commit 2026-05-02 16:59:39 +00:00
mmintel added 1 commit 2026-05-02 17:57:36 +00:00
Root cause: bot_flow.py injected configs.args.global_start_time = datetime.now()
which polluted the args namespace. SessionStateEncoder blindly serialized
args.__dict__ via json.dump, which crashed mid-write on the datetime object,
leaving sessions.json truncated/corrupt. Every subsequent restart failed.

Fixes:
- Remove global_start_time from configs.args (only lives on engine class vars)
- Harden SessionStateEncoder with _sanitize_value() to convert any
  non-JSON-serializable type (datetime, timedelta, arbitrary objects) to strings
- Add test_system_session_persistence.py with 4 tests covering the exact
  production crash scenario (datetime injection → json.dumps → round-trip)
- Fix test_engine_timeout.py broken GoalExecutor module reference
mmintel added 1 commit 2026-05-02 18:15:37 +00:00
Replaces reactive bug-specific tests with 4 structural CONTRACTS
that make entire categories of bugs impossible:

1. Serialization Contract (13 parametrized hostile payloads):
   Proves SessionStateEncoder NEVER crashes regardless of what
   gets injected onto args (datetime, lambda, bytes, inf, nan, etc.)

2. Persistence Contract:
   Forces the REAL persist() path by removing PYTEST_CURRENT_TEST guard.
   Validates write → read round-trip with poisoned sessions.

3. Production Parity Contract:
   Scans ALL GramAddict source for pytest/test-mode divergence guards.
   Any unaudited divergence point fails the build. New guards require
   explicit justification in KNOWN_DIVERGENCES registry.

4. Import Integrity Contract:
   Imports every single GramAddict module to catch syntax errors,
   circular imports, and missing dependencies at test time.

This addresses the root systemic failure: tests were running in a
parallel universe where serialization was silently skipped, allowing
a datetime injection to corrupt sessions.json and kill production.
mmintel added 1 commit 2026-05-02 19:56:22 +00:00
Two critical test additions:

1. test_system_full_lifecycle.py — Full A-to-Z integration test:
   Exercises the ENTIRE bot pipeline across 4 screen types (HomeFeed →
   Explore → Post Detail → Profile) with a real InstagramEmulator state
   machine. Validates plugin execution, session state accumulation,
   cognitive stack survival, dopamine session timeout, and post-session
   serialization. This is the single most important test in the suite.

2. test_system_coverage_gate.py — Critical Path Coverage Enforcement:
   Reads coverage_e2e.json and enforces minimum coverage thresholds on
   production-critical modules. Each threshold is tied to a real production
   incident. If persistent_list.py drops below 20% coverage, the build
   fails — because that's exactly how the sessions.json corruption
   bug went undetected.

   Usage: pytest tests/e2e --cov=GramAddict.core \
     --cov-report=json:coverage_e2e.json

   Critical paths enforced:
   - session_state.py (>=50%): serialization crash prevention
   - persistent_list.py (>=20%): persist() path must be tested
   - dopamine_engine.py (>=40%): session timeout logic
   - screen_identity.py (>=60%): screen identification
   - spatial_parser.py (>=70%): UI element parsing
   - intent_resolver.py (>=50%): click decision logic
   - q_nav_graph.py (>=25%): navigation integrity
   - device_facade.py (>=40%): device interface
mmintel added 1 commit 2026-05-02 20:07:14 +00:00
Validates the deterministic processing logic around the VLM without
needing a real LLM. Five contracts covering the full perception stack:

1. VLM Response Parsing (20 parametrized formats):
   Tests _parse_yes_no against every response format observed in production:
   clean YES/NO, JSON variants, explanations, edge cases (now/not/nothing),
   empty strings, free-form JSON. Documents the startswith('yes') early-exit
   behavior for ambiguous responses.

2. Box Index Extraction (7 formats):
   Validates JSON parsing of VLM box selections across all observed key
   variants: 'box', 'selected_index', 'box_index', 'index', null, and
   out-of-range values.

3. Structural Guards (11 tests):
   Tests every hallucination prevention guard in IntentResolver:
   - Tab intent excludes Back button (bug 2026-04-30)
   - Back intent preserves Back button
   - Tab height guard (y > 85% screen)
   - Author intent excludes nav tabs (bug 2026-05-01)
   - Structural fast-paths (message input, send, author, comment)
   - Abstract goals return None
   - Semantic quoted-target guard

4. Semantic Match Validation (10 tests):
   Prevents memory poisoning via _intent_matches_node:
   - Correct matches (follow/like/save + German locale)
   - Poisoning attempts: Reel thumbnails, photo content, comments
   - Non-toggle intents always pass through

5. ActionMemory Lifecycle (7 tests):
   Full track → confirm/reject cycle with FakeUIMemoryDB:
   - Correct follow stores in memory
   - Poisoned follow is BLOCKED (production bug reproduction)
   - Rejected click triggers confidence decay
   - No-track confirm is noop
   - Mismatched intent is ignored
   - Follow/view-post structural verification
mmintel added 1 commit 2026-05-02 20:10:59 +00:00
CONTRACT 6: extract_json() — LLM Response Sanitizer (9 tests)
Previously COMPLETELY untested. Every single LLM response in the entire
system passes through this function. Tests:
- Clean JSON, markdown fences (json+plain), thinking block purge
- Text prefix before JSON, truncated JSON fuzzy recovery
- Garbage/empty/None resilience

CONTRACT 7: _visual_discovery Pre-Filters (8 tests)
The guards INSIDE _visual_discovery() that run BEFORE the VLM ever
sees the candidates. If they fail, the VLM gets garbage and hallucinates.
Tests:
- Area filter: tiny nodes (<200px²) and huge containers (>400000px²)
- SystemUI filter: Android system elements, notifications, battery
- Strict Button Guard: long text captions excluded for button intents
- Grid Item Guard: non-grid elements excluded, fallthrough on no match
- Author/Username Guard: nav tabs excluded for author intents

CONTRACT 8: verify_success Structural Delta (8 tests)
The post-click verification logic that decides if a click actually worked.
Tests:
- Toggle massive XML shift = navigation error (False)
- Toggle small diff = success (True)
- Toggle zero diff = inconclusive (None)
- Follow success with 'Requested' marker (private accounts)
- Follow success with German locale ('Abonniert')
- View post with clips_viewer marker (Reels)
- View post still on grid = inconclusive (None)
- Semantic gate blocks wrong toggle element (caption != like button)
mmintel added 1 commit 2026-05-02 20:28:05 +00:00
Production bug 2026-05-02 22:19:
Intent 'post author username text (exclude bottom tabs)' contained the
word 'tab' in its parenthetical qualifier. The old check:
    is_tab_intent = 'tab' in intent_lower
matched this intent as a TAB navigation intent, activating the Tab
Height Guard which excluded ALL nodes with center_y < 85% of screen.

This nuked the actual clips_author_username node at y=2012, leaving
only 4 irrelevant items for the VLM → bot stuck in a loop unable to
find the author username.

Fix: Replace substring match with precise regex patterns that only
match real tab navigation intents:
  - 'tap profile tab', 'tap home tab' (tap + word + tab)
  - 'profile tab', 'explore tab' (word + tab)
  - 'tab ...' at start of intent

TDD: Red test added first, reproducing exact production state.
81 perception tests + 172 total tests pass.
mmintel added 1 commit 2026-05-02 20:34:48 +00:00
Production bug 2026-05-02 22:30: Bot got trapped in a restart loop:
  1. 'tap reels tab' → VLM clicked wrong element → EXPLORE_GRID
  2. After 2 failures → action masked
  3. HD Map: 'REELS_FEED unreachable due to masked edges'
  4. GOAP: 'completely trapped' → force Instagram restart
  5. Repeat forever

ALL TESTS WERE GREEN because NONE of this logic was tested.

NEW: test_system_goap_navigation.py — 63 tests across 8 contracts:

CONTRACT 1: Screen Identification from XML (16 tests)
  - Home, Explore, Reels, Profile, DM, Story, Modal, Foreign App
  - Post detail vs Home feed (action_bar discriminator)
  - Reels full-screen (no tab bar, structural markers only)

CONTRACT 2: Available Actions Extraction (4 tests)
  - All 5 tab actions present on home feed
  - Screen-specific actions (first post on Explore)

CONTRACT 3: HD Map BFS Routing (7 tests)
  - Direct routes, multi-step routes, already-there
  - Masked edges, all-edges-masked → None

CONTRACT 4: Goal → Target Screen Mapping (14 tests)
  - All navigation goals + non-navigation goals return None

CONTRACT 5: Action Failure Masking (4 tests)
  - Mask after MAX_RETRIES, no mask under threshold
  - Full production chain reproduction

CONTRACT 6: Goal Achievement Detection (9 tests)
  - Screen-based, context-based, non-nav goals

CONTRACT 7: Structural Action Protection (4 tests)
  - HD Map actions are structural, random actions are not

CONTRACT 8: Step-Aware Navigation Validation (5 tests)
  - Expected screen for action, mismatch detection
mmintel added 1 commit 2026-05-02 20:37:35 +00:00
mmintel added 1 commit 2026-05-02 20:40:22 +00:00
mmintel added 1 commit 2026-05-02 20:40:55 +00:00
mmintel added 1 commit 2026-05-02 20:44:55 +00:00
mmintel added 1 commit 2026-05-02 20:51:00 +00:00
mmintel added 1 commit 2026-05-02 21:11:34 +00:00
mmintel added 1 commit 2026-05-02 21:17:15 +00:00
mmintel added 1 commit 2026-05-03 08:21:30 +00:00
- device_emulator.py: Replace MagicMock(watcher) with explicit WatcherStub
  that implements exact u2 watcher chain (when/click/start). Any API drift
  now causes AttributeError instead of silent acceptance.
- test_system_sae.py: Inject XML into emulator state instead of bypassing
  dump_hierarchy with lambda
- test_behavior_ad_guard.py: Pass XML to fixture, remove lambda override
- test_behavior_scrape_profile.py: Pass XML to fixture, remove lambda override

Result: ZERO unittest.mock imports, ZERO MagicMock instances,
ZERO dump_hierarchy lambdas across entire E2E suite (50 files).
mmintel added 1 commit 2026-05-03 09:24:33 +00:00
mmintel added 1 commit 2026-05-03 09:31:33 +00:00
mmintel force-pushed fix/e2e-deception-purge from 4bc9183901 to 5fbbe3d273 2026-05-03 12:44:11 +00:00 Compare
mmintel added 2 commits 2026-05-03 14:20:32 +00:00
mmintel added 1 commit 2026-05-03 14:31:40 +00:00
mmintel added 1 commit 2026-05-03 14:36:05 +00:00
mmintel added 1 commit 2026-05-03 14:38:04 +00:00
mmintel added 2 commits 2026-05-03 15:00:32 +00:00
The SAE perceive() had no structural fast-path for Instagram-internal
modal overlays (surveys, rating prompts, interstitials). These modals
live inside com.instagram.android and were invisible to the foreign-app
detector. The LLM fallback frequently misclassified them as NORMAL,
causing the bot to get trapped in survey dialogs.

Added two O(1) structural detection layers:
1. Resource-ID markers: survey_overlay_container, interstitial_container,
   mystery_interstitial, nux_overlay, rating_prompt, feedback_dialog
2. Dismiss-button heuristic: cross-validates dismiss text with overlay
   container structure to prevent caption false positives

Also removed dead code: xml_dump.lower() no-op.

Fixes: test_perceive_instagram_survey_modal
Zero regressions: 464 passed, 6 pre-existing failures
1. ScreenIdentity: Restructured priority cascade to prevent Qdrant semantic
   cache from overriding deterministic structural heuristics. Cached types
   now resolve AFTER structural checks (Priority 3) instead of before.
   Story/Reels hallucinations from both cache and VLM are rejected if
   structural markers are absent.

2. bot_flow: Added error handling for story ring avatar tap and curiosity
   HomeFeed navigation failures — prevents silent continuation into
   undefined states.

3. GOAP: Extended is_navigation detection to include ScreenTopology
   structural actions, ensuring HD Map routes are correctly classified.

4. action_memory: Tightened _parse_yes_no to prevent JSON fall-through
   into ambiguous text matching. Structured responses with 'success'
   boolean are now handled directly.
mmintel added 1 commit 2026-05-03 15:15:19 +00:00
Ensuring 100% code-documentation parity per strict architectural rules.
mmintel added 1 commit 2026-05-03 16:29:34 +00:00
mmintel added 1 commit 2026-05-03 16:38:30 +00:00
mmintel added 1 commit 2026-05-03 18:30:33 +00:00
This branch is already included in the target branch. There is nothing to merge.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/e2e-deception-purge:fix/e2e-deception-purge
git checkout fix/e2e-deception-purge
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mmintel/instagram-bot#5