Marc Mintel
c98e2caaa1
TDD cycle (RED → GREEN): SAE perceive() — Zero Maintenance: - Removed ALL hardcoded marker tuples (instagram_modal_markers, creation_flow_markers, dismiss_button_patterns, blocked_markers) - Removed ALL hardcoded model names (llava:latest, qwen3.5:latest) - Removed ALL hardcoded URLs (localhost:11434) - Removed naked except blocks with model fallback defaults New autonomous flow (zero hardcoded UI identifiers): 1. Package-based foreign app detection (Android-level, not app-specific) 2. Qdrant semantic cache (O(1) recall of learned screen types) 3. ScreenIdentity structural delegation (MODAL/FOREIGN_APP) 4. LLM autonomous classification (first-encounter learning then cached) GOAP smart UI polling: - Replaced static random.uniform(1.6, 2.8) sleep with MAX_POLLS=5 polling loop that detects actual UI transitions Model config centralized via _get_model_config(): - ALL model/URL lookups go through Config() SSOT - Fail Fast - no silent hardcoded fallbacks Tests: 9 new, 119 passing, 0 regressions
55 lines
5.7 KiB
Markdown
55 lines
5.7 KiB
Markdown
# GramPilot Architecture
|
|
|
|
Welcome to the internal workings of **GramPilot** (formerly GramAddict). This document outlines the radical shift from fixed-state deterministic scripting to our current **Vision-Language-Action (VLA)** architecture that powers our "Full Self-Driving" behavior.
|
|
|
|
## Core Design Philosophy
|
|
We treat the Instagram Android App like a dynamic, partially-observable environment. Instead of maintaining thousands of fragile XPaths, the bot relies on a **Cognitive Stack** to infer intent, learn layouts dynamically, and mathematically avoid detection.
|
|
|
|
---
|
|
|
|
## 1. The Telepathic Engine (3-Stage Resolution Cascade)
|
|
At the center of UI interactions is the `TelepathicEngine` which resolves semantic intent ("tap the like button") into precise screen coordinates via a strictly enforced performance cascade:
|
|
- **Stage 1.5: Deterministic Keyword Fast Path**. Over 90% of interactions are handled by a high-performance string matcher that costs 0 API tokens and executes in `<2ms`.
|
|
- **Stage 2: Vector Similarity Engine**. If keywords fail, an Ollama Semantic Embedding of the intent is generated and compared (Cosine Similarity) against cached UI vectors via Qdrant. Highly reliable for semantic synonyms.
|
|
- **Stage 3: Agentic Fallback**. The ultimate safety net. If visual confidence drops `<0.82`, it falls back to an OpenRouter LLM (e.g., `gemini-3.1-flash-lite-preview`) which parses the raw XML to structurally guarantee a hit without hallucination.
|
|
|
|
## 2. Telepathic Memory & Autonomy
|
|
When Stage 3 successfully resolves an unknown interaction, the bot records the semantic signature into its positive memory (`telepathic_memory.json`). The next time the bot requires this action, it is instantly resolved via the local cache, guaranteeing that expensive LLM operations are only ever performed once per UI permutation.
|
|
|
|
## 3. The Cognitive Stack
|
|
|
|
### ⚖️ Active Inference (Shadow Mode)
|
|
Found in `active_inference.py`. Based on the free-energy principle, the bot calculates "Surprise" (prediction errors).
|
|
- **Shadow Mode**: Before transitioning screens, the bot predicts the target UI. If it lands somewhere unexpected (a popup), it registers a prediction error, hits "Back", and averts a crash.
|
|
|
|
### 🛡️ Honeypot Radome & Anti-Trap Sensors
|
|
Found in `sensors/honeypot_radome.py`.
|
|
- **Topological Traps**: Instagram deploys 1x1 pixel or 0x0 traps to detect bots. The Radome strictly strips these nodes prior to processing.
|
|
- **The Interceptor Sentinel**: Detects and purges full-screen invisible `clickable="true"` overlays that act as touch traps (e.g., bounds >= 90% with no content description).
|
|
- **Ghost Engagement Guard**: Strips DOM nodes explicitly tagged with `visible-to-user="false"` to prevent triggering Accessibility Hooks.
|
|
- **VLM Sanity Guard**: Woven into `telepathic_engine.py`, it sends semantic matches for destructive actions (Like/Follow) through a Vision Language Model step to prevent executing semantic "Bait and Switch" tricks.
|
|
|
|
### 🧠 Situational Awareness Engine (SAE)
|
|
Found in `situational_awareness.py`. Handles autonomous obstacle detection, recovery, and learning with **ZERO hardcoded UI element identifiers**.
|
|
- **Autonomous 3-Layer Classification** (zero maintenance — no resource-ids, no button text, no localized strings):
|
|
1. **Package-Based Foreign App Detection**: If our app's package is absent from the XML hierarchy, it's a foreign app. Uses Android package names (not Instagram UI elements).
|
|
2. **Qdrant Semantic Cache**: Previously classified screens are instantly recalled from the vector database (O(1) latency). The bot learns from every first-encounter.
|
|
3. **ScreenIdentity Structural Delegation**: The `ScreenIdentity` module classifies known screen types via its own structural logic. If it identifies a MODAL, the SAE trusts it.
|
|
4. **LLM Autonomous Classification**: Unknown screens are classified by the LLM (OBSTACLE_MODAL, DANGER_ACTION_BLOCKED, or NORMAL). Results are cached in Qdrant — so each screen type is learned exactly once.
|
|
- **Zero-Maintenance Guarantee**: When Instagram updates its UI (changes resource-ids, adds new modals), the bot discovers and learns the new patterns autonomously via the LLM. No code changes required.
|
|
|
|
### 🦾 Biometric Facade (Gaussian Clicks)
|
|
Found in `device_facade.py`.
|
|
- Human touches do not follow a flat mathematical uniform grid. The GramPilot simulates genuine **biometric dispersion** using `random.gauss(mu, sigma)`, strictly centering clicks inside a thumb-bias radius (bottom-left skew for right-handers). In tests, this hits a 68% standard deviation precision.
|
|
|
|
### 💉 Dopamine Engine & Resonance Oracle
|
|
Instead of hardcoding limits like `max_likes = 50`, the bot stops interacting based on **simulated boredom**.
|
|
- The `ResonanceEngine` calculates the aesthetic score of content.
|
|
- The `DopamineEngine` uses this score to modulate pace. High resonance = engagement. Low resonance over multiple posts = early session termination (simulating human fatigue).
|
|
|
|
## 4. The 100% Autonomy Directive (Zero Hardcoding)
|
|
GramPilot is designed as a true agent, not a state-machine script. It operates on **absolute zero hardcoded UI states or edge cases**.
|
|
- **No Manual Guards**: Features like `if "row_feed_button_like" not in xml:` or `if state == "ReelsFeed":` are strictly prohibited. The bot must understand the screen via its Vision-Language-Action (VLA) pipeline.
|
|
- **No Hand-Holding**: If the LLM makes a mistake (e.g., clicking the wrong button in a DM), the solution is to improve the VLM prompt, the system architecture, or the Visual Critic. We never insert `if is_dm_thread:` hacks.
|
|
- **Smart like a human**: The bot navigates by visually confirming targets, detecting obstacles when the UI organically stops responding, and inferring context precisely like a real user scrolling.
|