Compare commits
9 Commits
fix/struct
...
fix/keyboa
| Author | SHA1 | Date | |
|---|---|---|---|
| 3f778d46c9 | |||
| 49f82d467f | |||
| a67072eec4 | |||
| 59ba330029 | |||
| 6f9da50ae2 | |||
| 720841103d | |||
| c98e2caaa1 | |||
| 32731ed7ec | |||
| 8a6c8a2249 |
@@ -30,11 +30,13 @@ Found in `sensors/honeypot_radome.py`.
|
||||
- **VLM Sanity Guard**: Woven into `telepathic_engine.py`, it sends semantic matches for destructive actions (Like/Follow) through a Vision Language Model step to prevent executing semantic "Bait and Switch" tricks.
|
||||
|
||||
### 🧠 Situational Awareness Engine (SAE)
|
||||
Found in `situational_awareness.py`. Handles autonomous obstacle detection, recovery, and learning without hardcoded rules.
|
||||
- **3-Layer Modal Fast-Path**: Eliminates LLM hallucination traps for Instagram-internal modals (surveys, rating prompts) via O(1) deterministic structural checks:
|
||||
1. **Resource-ID Guard**: Detects internal blocking overlays (e.g., `survey_overlay_container`, `nux_overlay`).
|
||||
2. **Dismiss-Button Heuristic**: Cross-validates typical negative actions ("Not Now", "Take Survey") with overlay structures to prevent false positives in post captions.
|
||||
3. **Zero-Deception Fallback**: If structural markers fail, falls back to `ScreenMemoryDB` and ultimately the LLM. Structured invariants always override the semantic cache.
|
||||
Found in `situational_awareness.py`. Handles autonomous obstacle detection, recovery, and learning with **ZERO hardcoded UI element identifiers**.
|
||||
- **Autonomous 3-Layer Classification** (zero maintenance — no resource-ids, no button text, no localized strings):
|
||||
1. **Package-Based Foreign App Detection**: If our app's package is absent from the XML hierarchy, it's a foreign app. Uses Android package names (not Instagram UI elements).
|
||||
2. **Qdrant Semantic Cache**: Previously classified screens are instantly recalled from the vector database (O(1) latency). The bot learns from every first-encounter.
|
||||
3. **ScreenIdentity Structural Delegation**: The `ScreenIdentity` module classifies known screen types via its own structural logic. If it identifies a MODAL, the SAE trusts it.
|
||||
4. **LLM Autonomous Classification**: Unknown screens are classified by the LLM (OBSTACLE_MODAL, DANGER_ACTION_BLOCKED, or NORMAL). Results are cached in Qdrant — so each screen type is learned exactly once.
|
||||
- **Zero-Maintenance Guarantee**: When Instagram updates its UI (changes resource-ids, adds new modals), the bot discovers and learns the new patterns autonomously via the LLM. No code changes required.
|
||||
|
||||
### 🦾 Biometric Facade (Gaussian Clicks)
|
||||
Found in `device_facade.py`.
|
||||
|
||||
@@ -35,7 +35,7 @@ class CloseFriendsGuardPlugin(BehaviorPlugin):
|
||||
return False
|
||||
|
||||
xml = ctx.context_xml if ctx.context_xml else ctx.device.dump_hierarchy()
|
||||
return "enge freunde" in xml.lower() or "close friend" in xml.lower()
|
||||
return "close friend" in xml.lower()
|
||||
|
||||
def execute(self, ctx: BehaviorContext) -> BehaviorResult:
|
||||
logger.info("💚 [CloseFriendsGuard] Close friends post detected. Skipping...")
|
||||
|
||||
@@ -68,7 +68,7 @@ class ProfileGuardPlugin(BehaviorPlugin):
|
||||
|
||||
# Close friends guard
|
||||
if getattr(ctx.configs.args, "ignore_close_friends", False):
|
||||
if "enge freunde" in xml_check_lower or "close friend" in xml_check_lower:
|
||||
if "close friend" in xml_check_lower:
|
||||
logger.info(
|
||||
f"💚 [Profile Guard] @{ctx.username} is a Close Friend. Ignoring.", extra={"color": "\033[32m"}
|
||||
)
|
||||
|
||||
@@ -444,8 +444,8 @@ def start_bot(**kwargs):
|
||||
|
||||
from GramAddict.core.llm_provider import query_llm
|
||||
|
||||
model = getattr(configs.args, "ai_condenser_model", "llama3.2:1b")
|
||||
url = getattr(configs.args, "ai_condenser_url", "http://localhost:11434/api/generate")
|
||||
model = getattr(configs.args, "ai_condenser_model")
|
||||
url = getattr(configs.args, "ai_condenser_url")
|
||||
|
||||
response_dict = query_llm(url=url, model=model, prompt=prompt, format_json=True, timeout=120)
|
||||
if response_dict and isinstance(response_dict, dict) and "persona" in response_dict:
|
||||
@@ -715,7 +715,7 @@ def _interact_with_profile(device, configs, username, session_state, sleep_mod,
|
||||
return
|
||||
|
||||
if getattr(configs.args, "ignore_close_friends", False):
|
||||
if "enge freunde" in xml_check_lower or "close friend" in xml_check_lower:
|
||||
if "close friend" in xml_check_lower:
|
||||
logger.info(
|
||||
f"💚 [Profile Guard] @{username} is a Close Friend. Ignoring completely.", extra={"color": "\\033[32m"}
|
||||
)
|
||||
@@ -875,7 +875,7 @@ def _run_zero_latency_stories_loop(device, configs, session_state, cognitive_sta
|
||||
return "CONTEXT_LOST"
|
||||
|
||||
if getattr(configs.args, "ignore_close_friends", False):
|
||||
if "enge freunde" in xml_dump.lower() or "close friend" in xml_dump.lower():
|
||||
if "close friend" in xml_dump.lower():
|
||||
logger.info(
|
||||
"💚 [Anti-Friend] Story is from a Close Friend. Swiping horizontally to skip User.",
|
||||
extra={"color": "\\033[32m"},
|
||||
|
||||
@@ -30,13 +30,13 @@ class VLMCompilerEngine:
|
||||
extra={"color": "\x1b[1m\x1b[35m"},
|
||||
)
|
||||
|
||||
args = getattr(self.device, "args", None)
|
||||
model = getattr(args, "ai_telepathic_model", "llama3.2:1b") if args else "llama3.2:1b"
|
||||
url = (
|
||||
getattr(args, "ai_telepathic_url", "http://localhost:11434/api/generate")
|
||||
if args
|
||||
else "http://localhost:11434/api/generate"
|
||||
)
|
||||
from GramAddict.core.config import Config
|
||||
|
||||
cfg = Config()
|
||||
if not hasattr(cfg, "args"):
|
||||
raise RuntimeError("Config().args not initialized — cannot resolve AI model. Fail Fast.")
|
||||
model = getattr(cfg.args, "ai_telepathic_model")
|
||||
url = getattr(cfg.args, "ai_telepathic_url")
|
||||
use_local = "11434" in url or "localhost" in url
|
||||
|
||||
simplified_xml = self._simplify_xml(context_xml)
|
||||
|
||||
@@ -14,20 +14,23 @@ MAX_REPLIES_PER_INBOX_VISIT = 3
|
||||
_EMPTY_CONTEXT_SENTINELS = frozenset({"no previous context", "", "none", "n/a"})
|
||||
|
||||
|
||||
# Structural resource-IDs that indicate a real "Send" button.
|
||||
def _is_send_button(node: dict) -> bool:
|
||||
"""Semantic verification: returns True if the node is identified as a Send button."""
|
||||
desc = (node.get("description") or node.get("desc", "")).lower()
|
||||
text = (node.get("text") or "").lower()
|
||||
"""
|
||||
Structural verification: returns True if the node is identified as a Send button.
|
||||
NO hardcoded UI strings (no localized 'send' or 'absenden').
|
||||
"""
|
||||
rid = (node.get("id") or node.get("resource_id", "")).lower()
|
||||
|
||||
# Accept if semantic markers indicate sending
|
||||
if any(m in rid for m in ["send", "composer_button"]):
|
||||
# 1. Structural Resource IDs (Language agnostic)
|
||||
if "send" in rid or "composer_button" in rid or "direct_send_button_text" in rid:
|
||||
return True
|
||||
if any(m in desc for m in ["send", "absenden"]):
|
||||
return True
|
||||
if text == "send" or text == "absenden":
|
||||
|
||||
# 2. Spatial Guard: The send button in a DM thread is ALWAYS on the far right side of the screen
|
||||
# next to the composer input.
|
||||
x = node.get("x", 0)
|
||||
if int(x) > (1080 * 0.75):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
@@ -152,8 +155,8 @@ def _run_zero_latency_dm_loop(device, zero_engine, nav_graph, configs, session_s
|
||||
return "BOREDOM_CHANGE_FEED"
|
||||
|
||||
# Configure models
|
||||
model = getattr(configs.args, "ai_condenser_model", "llama3.2:1b")
|
||||
url = getattr(configs.args, "ai_condenser_url", "http://localhost:11434/api/generate")
|
||||
model = getattr(configs.args, "ai_condenser_model")
|
||||
url = getattr(configs.args, "ai_condenser_url")
|
||||
|
||||
# Generate response
|
||||
prompt = f"You are replying to a direct message on Instagram. The last message you received was: '{context_text}'. Keep it short, casual, and friendly. Do not use hashtags."
|
||||
|
||||
@@ -329,7 +329,7 @@ class GoalExecutor:
|
||||
|
||||
return False
|
||||
|
||||
def _execute_action(self, action: str, goal: str = None) -> bool:
|
||||
def _execute_action(self, action: str, goal: str = None, **kwargs) -> bool:
|
||||
"""Execute a single natural-language action using the TelepathicEngine."""
|
||||
|
||||
if action == "press back":
|
||||
@@ -349,6 +349,9 @@ class GoalExecutor:
|
||||
random_sleep(2.0, 3.5)
|
||||
return True
|
||||
|
||||
if action == "type and post comment":
|
||||
return self._execute_type_and_post_comment(kwargs.get("text", ""))
|
||||
|
||||
# Use TelepathicEngine for any semantic click
|
||||
from GramAddict.core.telepathic_engine import TelepathicEngine
|
||||
|
||||
@@ -388,12 +391,22 @@ class GoalExecutor:
|
||||
|
||||
# Execute click
|
||||
self.device.click(obj=best_node)
|
||||
import random
|
||||
|
||||
time.sleep(random.uniform(1.6, 2.8))
|
||||
|
||||
# Verify success via Goal Context + Screen Feedback
|
||||
post_xml = self.device.dump_hierarchy()
|
||||
# ── Smart UI Stabilization Poll ──
|
||||
# Instead of a static sleep (which is either too long on fast devices or
|
||||
# too short on slow ones), we poll dump_hierarchy multiple times.
|
||||
# As soon as the XML changes, we know the UI has transitioned.
|
||||
MAX_POLLS = 5
|
||||
POLL_INTERVAL = 0.5 # seconds between polls
|
||||
post_xml = xml_dump # Start with pre-click state
|
||||
for poll in range(MAX_POLLS):
|
||||
time.sleep(POLL_INTERVAL)
|
||||
post_xml = self.device.dump_hierarchy()
|
||||
if post_xml != xml_dump:
|
||||
logger.debug(f"[GOAP Poll] UI change detected on poll {poll + 1}/{MAX_POLLS}.")
|
||||
break
|
||||
else:
|
||||
logger.debug(f"[GOAP Poll] No UI change after {MAX_POLLS} polls ({MAX_POLLS * POLL_INTERVAL}s).")
|
||||
pre_action_screen = self.perceive(xml_dump) # Screen state BEFORE the click
|
||||
post_screen = self.perceive(post_xml)
|
||||
post_screen_type = post_screen["screen_type"]
|
||||
@@ -545,6 +558,83 @@ class GoalExecutor:
|
||||
achieved = self.planner.plan_next_step(goal, screen) is None
|
||||
return achieved
|
||||
|
||||
def _execute_type_and_post_comment(self, fallback_text: str) -> bool:
|
||||
"""Handles the specific typing interaction, prioritizing Meta AI chips if available."""
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
from GramAddict.core.telepathic_engine import TelepathicEngine
|
||||
|
||||
logger.info("💬 [GOAP] Executing 'type and post comment'...")
|
||||
engine = TelepathicEngine.get_instance()
|
||||
|
||||
# 1. Tap the comment input field to open the keyboard
|
||||
xml_dump = self.device.dump_hierarchy()
|
||||
if "com.google.android.inputmethod.latin" not in xml_dump:
|
||||
logger.info("⌨️ [GOAP] Tapping comment composer to open keyboard...")
|
||||
best_node = engine.find_best_node(
|
||||
xml_dump, "tap comment input field", min_confidence=0.7, device=self.device
|
||||
)
|
||||
if best_node:
|
||||
self.device.click(obj=best_node)
|
||||
random_sleep(1.5, 2.5)
|
||||
xml_dump = self.device.dump_hierarchy()
|
||||
else:
|
||||
logger.warning("⚠️ [GOAP] Could not find comment input field.")
|
||||
return False
|
||||
|
||||
# 2. Check for Meta AI chips (Supportive, Funny, etc.)
|
||||
meta_ai_chips = []
|
||||
try:
|
||||
root = ET.fromstring(xml_dump.encode("utf-8"))
|
||||
for node in root.iter("node"):
|
||||
node_text = node.attrib.get("text", "")
|
||||
if node_text in ["Supportive", "Rewrite", "Absurd", "Casual", "Funny", "Heartfelt", "Professional"]:
|
||||
meta_ai_chips.append(node_text)
|
||||
except Exception as e:
|
||||
logger.debug(f"XML parse error for Meta AI chips: {e}")
|
||||
|
||||
if meta_ai_chips:
|
||||
logger.info(f"✨ [Meta AI] Detected Meta AI chips: {meta_ai_chips}")
|
||||
logger.info("🧠 [Meta AI] Asking VLM to select the best tone...")
|
||||
|
||||
# Use Telepathic Engine to pick the best chip via VLM
|
||||
chip_node = engine.find_best_node(
|
||||
xml_dump,
|
||||
"tap the best Meta AI tone chip to generate a comment (e.g. Supportive, Funny, Casual)",
|
||||
min_confidence=0.6,
|
||||
device=self.device,
|
||||
)
|
||||
|
||||
if chip_node:
|
||||
logger.info(f"✨ [Meta AI] VLM selected chip: '{chip_node.get('text', 'Unknown')}'")
|
||||
self.device.click(obj=chip_node)
|
||||
random_sleep(3.0, 4.5) # Wait for Meta AI to generate the text
|
||||
else:
|
||||
logger.warning("⚠️ [Meta AI] VLM failed to pick a chip, falling back to manual typing.")
|
||||
from GramAddict.core.stealth_typing import ghost_type
|
||||
|
||||
ghost_type(self.device, fallback_text)
|
||||
random_sleep(1.0, 2.0)
|
||||
else:
|
||||
# 3. Fallback: Type the text provided by our own VLM writer
|
||||
logger.info(f"⌨️ [GOAP] Typing comment manually: {fallback_text}")
|
||||
from GramAddict.core.stealth_typing import ghost_type
|
||||
|
||||
ghost_type(self.device, fallback_text)
|
||||
random_sleep(1.0, 2.0)
|
||||
|
||||
# 4. Click the Post button
|
||||
xml_dump = self.device.dump_hierarchy()
|
||||
post_btn = engine.find_best_node(xml_dump, "tap post comment button", min_confidence=0.7, device=self.device)
|
||||
if post_btn:
|
||||
self.device.click(obj=post_btn)
|
||||
random_sleep(1.5, 2.5)
|
||||
logger.info("✅ [GOAP] Comment posted successfully.")
|
||||
return True
|
||||
else:
|
||||
logger.warning("⚠️ [GOAP] Could not find post button after typing.")
|
||||
return False
|
||||
|
||||
# ── Convenience methods (backward compatibility with navigate_to) ──
|
||||
|
||||
def navigate_to_screen(self, target: str) -> bool:
|
||||
|
||||
@@ -54,9 +54,9 @@ class LLMWriter:
|
||||
f"6. Reply with ONLY the comment text."
|
||||
)
|
||||
|
||||
model = getattr(self.args, "ai_writer_model", getattr(self.args, "ai_model", "llama3.2:1b"))
|
||||
model = getattr(self.args, "ai_writer_model", getattr(self.args, "ai_model"))
|
||||
url = getattr(
|
||||
self.args, "ai_writer_url", getattr(self.args, "ai_model_url", "http://localhost:11434/api/generate")
|
||||
self.args, "ai_writer_url", getattr(self.args, "ai_model_url")
|
||||
)
|
||||
|
||||
logger.info(f"✍️ [Writer] Generating comment for @{target_username} using {model}...")
|
||||
|
||||
@@ -395,14 +395,11 @@ def query_llm(
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Last resort defaults
|
||||
# Last resort: If no fallback config exists, don't silently use hardcoded defaults.
|
||||
# Config() defines these via argparse defaults — if they're missing, there's nothing to fallback to.
|
||||
if not f_model or not f_url:
|
||||
if is_openai_compat:
|
||||
f_model = f_model or "llama3.2:1b"
|
||||
f_url = f_url or "http://localhost:11434/api/generate"
|
||||
else:
|
||||
f_model = f_model or "llama3.2:1b"
|
||||
f_url = f_url or "http://localhost:11434/api/generate"
|
||||
logger.warning("⚠️ [Circuit Breaker] No fallback model/URL configured. Cannot retry.")
|
||||
return None
|
||||
|
||||
# Circuit Breaker: If fallback is identical to primary, don't waste time retrying
|
||||
if f_model == model and f_url == url:
|
||||
@@ -458,11 +455,12 @@ def query_telepathic_llm(
|
||||
|
||||
try:
|
||||
args = Config().args
|
||||
target_url = getattr(args, "ai_fallback_url", "http://localhost:11434/api/generate")
|
||||
target_model = getattr(args, "ai_fallback_model", "llama3.2:1b")
|
||||
target_url = getattr(args, "ai_fallback_url")
|
||||
target_model = getattr(args, "ai_fallback_model")
|
||||
except Exception:
|
||||
target_url = "http://localhost:11434/api/generate"
|
||||
target_model = "llama3.2:1b"
|
||||
raise RuntimeError(
|
||||
"Config().args not initialized — cannot resolve fallback AI model. Fail Fast."
|
||||
)
|
||||
|
||||
is_local = "localhost" in target_url or "127.0.0.1" in target_url
|
||||
calc_timeout = 180 if is_local else 45
|
||||
|
||||
@@ -15,12 +15,10 @@ def ask_brain_for_action(
|
||||
return None
|
||||
|
||||
cfg = Config()
|
||||
url = (
|
||||
getattr(cfg.args, "ai_model_url", "http://localhost:11434/api/generate")
|
||||
if hasattr(cfg, "args")
|
||||
else "http://localhost:11434/api/generate"
|
||||
)
|
||||
model = getattr(cfg.args, "ai_model", "qwen3.5:latest") if hasattr(cfg, "args") else "qwen3.5:latest"
|
||||
if not hasattr(cfg, "args"):
|
||||
raise RuntimeError("Config().args not initialized — cannot resolve AI model. Fail Fast.")
|
||||
url = getattr(cfg.args, "ai_model_url")
|
||||
model = getattr(cfg.args, "ai_model")
|
||||
|
||||
prompt = (
|
||||
f"You are an autonomous Instagram agent. Your ultimate goal is: '{goal}'.\n"
|
||||
|
||||
@@ -220,8 +220,8 @@ class ActionMemory:
|
||||
)
|
||||
if is_toggle:
|
||||
prompt += (
|
||||
"If the intent was 'follow', does the button now indicate 'Following' or 'Requested'? "
|
||||
"If it was 'like', is the heart icon clearly active/red? "
|
||||
"If the intent was 'follow', did the button change its visual state to indicate an active subscription? "
|
||||
"If it was 'like', is the heart icon clearly active/filled? "
|
||||
"If the screen shifted completely to a profile when you just wanted to like/follow from a feed, it FAILED. "
|
||||
"If the tapped element does NOT sound like a like/follow button (e.g. it's a caption, comment field, or post content), it FAILED. "
|
||||
)
|
||||
|
||||
@@ -30,6 +30,11 @@ class IntentResolver:
|
||||
bounding boxes around clickable candidates, sends the annotated image to the VLM,
|
||||
and lets the VLM visually decide which box to tap.
|
||||
|
||||
CRITICAL ARCHITECTURE RULE: Language Agnosticism & Structural Determinism
|
||||
- NO hardcoded localized UI strings (e.g. "follow", "message", "like") are permitted.
|
||||
- All non-VLM resolution logic MUST rely strictly on spatial coordinates (e.g. center_x bounds)
|
||||
or structural Android resource IDs.
|
||||
|
||||
Architecture:
|
||||
1. Navigation tabs → structural zone guard (bottom 15%, resource-id)
|
||||
2. Everything else → Visual Discovery (screenshot + numbered boxes + VLM)
|
||||
@@ -135,23 +140,25 @@ class IntentResolver:
|
||||
|
||||
# NEW REGRESSION FIX: Exclude interaction buttons (comment, like, share) when looking for author or media
|
||||
# This prevents the weak VLM from hallucinating bounding box numbers that point to "Comment".
|
||||
interaction_suffixes = ["comment", "like", "share", "send", "save", "button_icon"]
|
||||
interaction_suffixes = ["comment", "like", "share", "send", "save", "button_icon", "scrubber"]
|
||||
is_interaction = any(s in rid for s in interaction_suffixes) or any(s in desc for s in interaction_suffixes)
|
||||
node_text_lower = (node.text or "").lower()
|
||||
is_follow = "follow" in rid or "follow" in node_text_lower or "follow" in desc
|
||||
is_media_intent = "media content" in intent_lower or "image" in intent_lower or "video" in intent_lower
|
||||
|
||||
if is_author_intent and (is_interaction or is_follow):
|
||||
# SPATIAL GUARD (NO HARDCODED STRINGS): The post author (avatar/name) is always strictly on the left side of the screen.
|
||||
# The "More options" (Report menu) button is strictly on the far right.
|
||||
# We enforce a mathematical boundary to prevent LLM hallucinations from clicking the report menu.
|
||||
if is_author_intent and node.center_x > (1080 * 0.75):
|
||||
logger.debug(
|
||||
f"🛡️ [Author Interaction Guard] Excluded interaction/follow button '{node.resource_id}' "
|
||||
f"(desc='{node.content_desc}', text='{node.text}') for author intent '{intent_description}'"
|
||||
f"🛡️ [Author Spatial Guard] Excluded far-right element '{node.resource_id}' "
|
||||
f"(x={node.center_x}) for author intent '{intent_description}'"
|
||||
)
|
||||
continue
|
||||
is_follow = "button_follow" in rid or "ufi_follow" in rid
|
||||
is_media_intent = "media content" in intent_lower or "image" in intent_lower or "video" in intent_lower
|
||||
|
||||
if is_media_intent and (is_interaction or is_follow):
|
||||
if (is_author_intent or is_media_intent) and (is_interaction or is_follow):
|
||||
logger.debug(
|
||||
f"🛡️ [Media Interaction Guard] Excluded interaction/follow button '{node.resource_id}' "
|
||||
f"(desc='{node.content_desc}') for media intent '{intent_description}'"
|
||||
f"🛡️ [Interaction Guard] Excluded non-target element '{node.resource_id}' "
|
||||
f"(desc='{node.content_desc}', text='{node.text}') for intent '{intent_description}'"
|
||||
)
|
||||
continue
|
||||
|
||||
@@ -187,6 +194,31 @@ class IntentResolver:
|
||||
|
||||
return filtered
|
||||
|
||||
def pre_filter_candidates(self, candidates: List[SpatialNode]) -> List[SpatialNode]:
|
||||
"""
|
||||
Filters candidates by area, system UI, keyboard packages, and notifications.
|
||||
|
||||
Production Bug 2026-05-04: The Android soft keyboard (com.google.android.inputmethod.*)
|
||||
flooded the candidate pool with 30+ single-letter nodes (A, B, C, N, ...),
|
||||
causing the VLM to hallucinate keyboard keys as valid UI targets.
|
||||
"""
|
||||
return [
|
||||
n
|
||||
for n in candidates
|
||||
if 200 < n.area < 400000
|
||||
and "com.android.systemui" not in (n.resource_id or "")
|
||||
and "inputmethod" not in (n.resource_id or "")
|
||||
and "notification:" not in (n.content_desc or "").lower()
|
||||
and "per cent" not in (n.content_desc or "").lower()
|
||||
]
|
||||
|
||||
def has_keyboard_open(self, candidates: List[SpatialNode]) -> bool:
|
||||
"""
|
||||
Detects if the Android soft keyboard is currently visible
|
||||
by checking for input method package nodes in the candidate list.
|
||||
"""
|
||||
return any("inputmethod" in (n.resource_id or "") for n in candidates)
|
||||
|
||||
# ──────────────────────────────────────────────
|
||||
# Public API
|
||||
# ──────────────────────────────────────────────
|
||||
@@ -233,15 +265,27 @@ class IntentResolver:
|
||||
return node
|
||||
|
||||
if "post author username" in intent_lower or "tap post username" in intent_lower:
|
||||
# 1. Feed Posts
|
||||
for node in candidates:
|
||||
if "row_feed_photo_profile_imageview" in (node.resource_id or "").lower():
|
||||
rid = (node.resource_id or "").lower()
|
||||
if "row_feed_photo_profile_imageview" in rid:
|
||||
logger.info(f"🎯 [Structural Fast-Path] Found post author avatar image: {node.content_desc}")
|
||||
return node
|
||||
for node in candidates:
|
||||
if "row_feed_photo_profile_name" in (node.resource_id or "").lower():
|
||||
rid = (node.resource_id or "").lower()
|
||||
if "row_feed_photo_profile_name" in rid:
|
||||
logger.info(f"🎯 [Structural Fast-Path] Found post author username text: {node.text}")
|
||||
return node
|
||||
|
||||
# 2. Reels (Clips)
|
||||
for node in candidates:
|
||||
rid = (node.resource_id or "").lower()
|
||||
if "clips_author_username" in rid or "clips_author_container" in rid:
|
||||
logger.info(
|
||||
f"🎯 [Structural Fast-Path] Found Reel author username: {node.text or node.content_desc}"
|
||||
)
|
||||
return node
|
||||
|
||||
if "feed post content" in intent_lower or "post media content" in intent_lower:
|
||||
for node in candidates:
|
||||
rid = (node.resource_id or "").lower()
|
||||
@@ -279,9 +323,8 @@ class IntentResolver:
|
||||
if ("send" in intent_lower or "share" in intent_lower) and "post" in intent_lower and "button" in intent_lower:
|
||||
for node in candidates:
|
||||
rid = (node.resource_id or "").lower()
|
||||
desc = (node.content_desc or "").lower()
|
||||
if "row_feed_button_share" in rid or "send post" in desc:
|
||||
logger.info(f"🎯 [Structural Fast-Path] Found send/share post button: {rid or desc}")
|
||||
if "row_feed_button_share" in rid or "direct_share_button" in rid or "button_share" in rid:
|
||||
logger.info(f"🎯 [Structural Fast-Path] Found send/share post button: {rid}")
|
||||
return node
|
||||
|
||||
if "add to story" in intent_lower:
|
||||
@@ -577,15 +620,8 @@ class IntentResolver:
|
||||
from GramAddict.core.config import Config
|
||||
from GramAddict.core.llm_provider import query_telepathic_llm
|
||||
|
||||
# Pre-filter candidates by area and system UI before any semantic matching
|
||||
candidates = [
|
||||
n
|
||||
for n in candidates
|
||||
if 200 < n.area < 400000
|
||||
and "com.android.systemui" not in (n.resource_id or "")
|
||||
and "notification:" not in (n.content_desc or "").lower()
|
||||
and "per cent" not in (n.content_desc or "").lower()
|
||||
]
|
||||
# Pre-filter candidates by area, system UI, and keyboard packages
|
||||
candidates = self.pre_filter_candidates(candidates)
|
||||
|
||||
# --- Navigation Conflict Guard ---
|
||||
# Prevents VLM from confusing Back buttons with tab buttons
|
||||
@@ -656,8 +692,8 @@ class IntentResolver:
|
||||
self.last_box_map = box_map
|
||||
|
||||
cfg = Config()
|
||||
model = getattr(cfg.args, "ai_telepathic_model", "llava:latest")
|
||||
url = getattr(cfg.args, "ai_telepathic_url", "http://localhost:11434/api/generate")
|
||||
model = getattr(cfg.args, "ai_telepathic_model")
|
||||
url = getattr(cfg.args, "ai_telepathic_url")
|
||||
|
||||
# Build a compact legend of what each box contains
|
||||
box_legend_lines = []
|
||||
@@ -684,14 +720,14 @@ class IntentResolver:
|
||||
f"CRITICAL RULES:\n"
|
||||
f"1. If the intent contains a word in quotes (e.g., 'Search', 'New Message'), you MUST look at the Box legend and pick the box that contains that word (case-insensitive). Do not pick anything else.\n"
|
||||
f"2. For icons without text:\n"
|
||||
f" - 'like button' = HEART-SHAPED ICON (♡/❤), usually has desc='Like'.\n"
|
||||
f" - 'comment button' = SPEECH BUBBLE ICON, usually has desc='Comment'.\n"
|
||||
f" - 'like button' = HEART-SHAPED ICON (♡/❤), look for id containing 'button_like' or 'ufi_heart'.\n"
|
||||
f" - 'comment button' = SPEECH BUBBLE ICON, look for id containing 'button_comment' or 'ufi_comment'.\n"
|
||||
f"3. Do NOT select text, captions, or view counts if looking for an icon.\n"
|
||||
f"4. Ignore numbers inside the text itself. Do not confuse the text '19' with Box [19].\n"
|
||||
f"5. If the intent contains 'following', you MUST pick the box containing 'following'. Do NOT pick 'followers' or 'Follow'.\n"
|
||||
f"5. If the intent contains 'following', you MUST pick the box containing id 'button_following' or 'profile_header_following'. Do NOT pick 'followers' or 'Follow'.\n"
|
||||
f"6. If the intent is to tap a 'post', 'first post', or 'grid item':\n"
|
||||
f" - Look for boxes with descriptions containing 'photos by', 'Reel by', or 'row 1, column 1'.\n"
|
||||
f" - Pick the FIRST matching box index (e.g. if [0] says '6 photos...', return 0, NOT 6).\n"
|
||||
f" - Look for boxes with ids like 'image_button' inside a grid, or visual grid thumbnails.\n"
|
||||
f" - Pick the FIRST matching box index.\n"
|
||||
f" - Do NOT pick navigation buttons like 'Search'.\n"
|
||||
f"7. If the intent is a bottom navigation tab (e.g. 'profile tab', 'home tab'):\n"
|
||||
f" - These are always at the BOTTOM edge of the screen.\n"
|
||||
@@ -700,17 +736,17 @@ class IntentResolver:
|
||||
f" - 'explore tab' is the magnifying glass.\n"
|
||||
f" - 'reels tab' is the video clapperboard.\n"
|
||||
f"8. If the intent involves 'author username' or 'author profile':\n"
|
||||
f" - Pick the profile picture (e.g. 'Profile picture of <username>') or the username text.\n"
|
||||
f" - NEVER pick a 'Follow' button. Do NOT pick 'Follow <username>'.\n"
|
||||
f" - Pick the profile picture or the username text.\n"
|
||||
f" - NEVER pick a 'Follow' button. Do NOT pick 'button_follow'.\n"
|
||||
f"9. If the intent is 'save post':\n"
|
||||
f" - The save icon is the bookmark icon on the bottom right of the post image/video.\n"
|
||||
f" - Usually has desc='Add to Saved' or 'Save'. Do NOT pick the post text or other action buttons.\n"
|
||||
f" - Look for id containing 'button_save' or 'ufi_save'. Do NOT pick the post text or other action buttons.\n"
|
||||
f"10. DISTINGUISHING BOTTOM TABS vs CONTENT BUTTONS:\n"
|
||||
f" - Bottom Navigation Tabs (Home, Search, Reels, Profile) are ALWAYS at the very bottom (y > 2100).\n"
|
||||
f" - Content Interaction Buttons (Like, Comment, Share, Reactions, Message Input) are attached to posts or threads, NOT the bottom nav bar.\n"
|
||||
f" - If looking for 'message input' or 'type message', do NOT select 'reactions' or emoji icons. Look for an empty text box or 'Message...'.\n"
|
||||
f" - If looking for 'message input' or 'type message', do NOT select 'reactions' or emoji icons. Look for an empty text box or id 'message_content'.\n"
|
||||
f"11. If the intent is 'feed post content' or 'post media content':\n"
|
||||
f" - Pick the largest box that contains the actual image or video, usually described as 'Photo', 'Video', or 'Carousel'.\n"
|
||||
f" - Pick the largest box that contains the actual image or video. Look for id 'zoomable_view_container' or 'media_frame'.\n"
|
||||
f"12. If the exact control is NOT visible, return null. Do NOT guess.\n\n"
|
||||
f'Reply ONLY with a valid JSON object: {{"box": <number>}} or {{"box": null}}'
|
||||
)
|
||||
@@ -780,8 +816,8 @@ class IntentResolver:
|
||||
filtered_candidates = [n for n in candidates if n.area < 500000]
|
||||
|
||||
cfg = Config()
|
||||
model = getattr(cfg.args, "ai_telepathic_model", "qwen3.5:latest")
|
||||
url = getattr(cfg.args, "ai_telepathic_url", "http://localhost:11434/api/generate")
|
||||
model = getattr(cfg.args, "ai_telepathic_model")
|
||||
url = getattr(cfg.args, "ai_telepathic_url")
|
||||
|
||||
node_context = []
|
||||
for i, node in enumerate(filtered_candidates):
|
||||
|
||||
@@ -22,6 +22,7 @@ class ScreenType(Enum):
|
||||
FOLLOW_LIST = "follow_list"
|
||||
COMMENTS = "comments"
|
||||
MODAL = "modal"
|
||||
DANGER_ACTION_BLOCKED = "danger_action_blocked"
|
||||
FOREIGN_APP = "foreign_app"
|
||||
UNKNOWN = "unknown"
|
||||
|
||||
@@ -176,11 +177,40 @@ class ScreenIdentity:
|
||||
# These full-screen Instagram UIs have no navigation tabs and trap the bot.
|
||||
# Structural detection is O(1), zero LLM calls, and cannot be fooled.
|
||||
if not is_normal_override:
|
||||
creation_flow_markers = ("quick_capture", "gallery_cancel_button", "creation_flow", "reel_camera")
|
||||
if any(marker in ids_str for marker in creation_flow_markers):
|
||||
logger.info("🛡️ [ScreenIdentity] Content-creation overlay detected → MODAL")
|
||||
modal_markers = (
|
||||
"quick_capture",
|
||||
"gallery_cancel_button",
|
||||
"creation_flow",
|
||||
"reel_camera",
|
||||
"survey_overlay_container",
|
||||
"interstitial_container",
|
||||
"nux_overlay",
|
||||
"rating_prompt",
|
||||
"feedback_dialog",
|
||||
"action_bar_browser_container",
|
||||
)
|
||||
if any(marker in ids_str for marker in modal_markers):
|
||||
logger.info("🛡️ [ScreenIdentity] Modal/Interstitial overlay detected → MODAL")
|
||||
return ScreenType.MODAL
|
||||
|
||||
# Action Blocked Detection (O(1) fast-path)
|
||||
# Prevents LLM hallucinations for system-level traps that block the entire flow.
|
||||
danger_markers = (
|
||||
"try again later",
|
||||
"action blocked",
|
||||
"we restrict certain activity",
|
||||
"to protect our community",
|
||||
)
|
||||
if "bottom_sheet_container" in ids and any(d in text_lower or d in desc_lower for d in danger_markers):
|
||||
logger.info("🛡️ [ScreenIdentity] Critical obstacle detected → DANGER_ACTION_BLOCKED")
|
||||
return ScreenType.DANGER_ACTION_BLOCKED
|
||||
|
||||
# Menu Trap Detection (O(1) fast-path)
|
||||
# Overrides hallucinated LLM cache using STRICTLY STRUCTURAL IDs (no localized strings).
|
||||
if "bottom_sheet_container" in ids and "action_sheet_row_text_view" in ids:
|
||||
logger.info("🛡️ [ScreenIdentity] Options/Report menu trap detected → MODAL")
|
||||
return ScreenType.MODAL
|
||||
|
||||
# Priority 2: Structural Heuristics (100% Deterministic)
|
||||
if "unified_follow_list_tab_layout" in ids or "follow_list_container" in ids:
|
||||
return ScreenType.FOLLOW_LIST
|
||||
@@ -193,7 +223,7 @@ class ScreenIdentity:
|
||||
"profile_header_name",
|
||||
)
|
||||
if any(marker in ids for marker in PROFILE_MARKERS):
|
||||
own_profile_texts = ("edit profile", "share profile", "profil bearbeiten", "profil teilen")
|
||||
own_profile_texts = ("edit profile", "share profile")
|
||||
if selected_tab == "profile_tab" or any(m in desc_lower or m in text_lower for m in own_profile_texts):
|
||||
return ScreenType.OWN_PROFILE
|
||||
return ScreenType.OTHER_PROFILE
|
||||
@@ -276,12 +306,10 @@ class ScreenIdentity:
|
||||
from GramAddict.core.llm_provider import query_telepathic_llm
|
||||
|
||||
cfg = Config()
|
||||
url = (
|
||||
getattr(cfg.args, "ai_telepathic_url", "http://localhost:11434/api/generate")
|
||||
if hasattr(cfg, "args")
|
||||
else "http://localhost:11434/api/generate"
|
||||
)
|
||||
model = getattr(cfg.args, "ai_telepathic_model", "llava:latest") if hasattr(cfg, "args") else "llava:latest"
|
||||
if not hasattr(cfg, "args"):
|
||||
raise RuntimeError("Config().args not initialized — cannot resolve AI model. Fail Fast.")
|
||||
url = getattr(cfg.args, "ai_telepathic_url")
|
||||
model = getattr(cfg.args, "ai_telepathic_model")
|
||||
|
||||
layout_context = (
|
||||
f"Selected Tab: {selected_tab}\nResource IDs: {list(ids)}\nVisible Texts context: {texts[:10]}\n"
|
||||
@@ -345,43 +373,30 @@ class ScreenIdentity:
|
||||
if tab_id in resource_ids:
|
||||
actions.append(action)
|
||||
|
||||
# Screen-specific actions
|
||||
desc_lower = " ".join(content_descs).lower()
|
||||
text_lower = " ".join(texts).lower()
|
||||
# Screen-specific actions (Purely structural, NO localized strings)
|
||||
ids_str = " ".join(resource_ids).lower()
|
||||
|
||||
if "like" in desc_lower:
|
||||
if "button_like" in ids_str or "ufi_heart" in ids_str:
|
||||
actions.append("tap like button")
|
||||
if "comment" in desc_lower:
|
||||
if "button_comment" in ids_str or "ufi_comment" in ids_str:
|
||||
actions.append("tap comment button")
|
||||
if "share" in desc_lower:
|
||||
if "button_share" in ids_str or "ufi_share" in ids_str or "direct_share" in ids_str:
|
||||
actions.append("tap share button")
|
||||
if "save" in desc_lower or "bookmark" in desc_lower:
|
||||
if "button_save" in ids_str or "ufi_save" in ids_str:
|
||||
actions.append("tap save button")
|
||||
if "back" in desc_lower:
|
||||
if "back" in ids_str or "action_bar_button_back" in ids_str:
|
||||
actions.append("tap back button")
|
||||
has_following = any(
|
||||
"following" in e.get("text", "").lower() or "following" in e.get("desc", "").lower()
|
||||
for e in clickable_elements
|
||||
)
|
||||
|
||||
has_following = "button_following" in ids_str or "profile_header_following" in ids_str
|
||||
if has_following:
|
||||
actions.append("tap following button")
|
||||
elif any(
|
||||
"follow" in e.get("text", "").lower()
|
||||
or "follow" in e.get("desc", "").lower()
|
||||
or "follow" in e.get("id", "").lower()
|
||||
for e in clickable_elements
|
||||
):
|
||||
elif "button_follow" in ids_str:
|
||||
actions.append("tap follow button")
|
||||
|
||||
if screen_type == ScreenType.OWN_PROFILE or screen_type == ScreenType.OTHER_PROFILE:
|
||||
if "message" in desc_lower or "nachricht" in desc_lower:
|
||||
if "button_message" in ids_str or "direct_message" in ids_str:
|
||||
actions.append("tap message button")
|
||||
if (
|
||||
"following" in desc_lower
|
||||
or "abonniert" in desc_lower
|
||||
or "following" in text_lower
|
||||
or "profile_header_following" in " ".join(resource_ids).lower()
|
||||
):
|
||||
if "profile_header_following" in ids_str:
|
||||
actions.append("tap following list")
|
||||
|
||||
# Grid items
|
||||
|
||||
@@ -28,8 +28,8 @@ class SemanticEvaluator:
|
||||
logger.warning("👁️ [Vision Core] No config available. Cannot query VLM.")
|
||||
return None
|
||||
|
||||
model = getattr(self.args, "ai_telepathic_model", "llama3.2-vision")
|
||||
url = getattr(self.args, "ai_telepathic_url", "http://localhost:11434/api/generate")
|
||||
model = getattr(self.args, "ai_telepathic_model")
|
||||
url = getattr(self.args, "ai_telepathic_url")
|
||||
|
||||
try:
|
||||
res = query_telepathic_llm(
|
||||
|
||||
@@ -29,7 +29,10 @@ class QdrantBase:
|
||||
|
||||
try:
|
||||
qdrant_url = os.environ.get("QDRANT_URL", "http://localhost:6344")
|
||||
self.client = QdrantClient(url=qdrant_url, timeout=10.0)
|
||||
if qdrant_url == ":memory:":
|
||||
self.client = QdrantClient(location=":memory:")
|
||||
else:
|
||||
self.client = QdrantClient(url=qdrant_url, timeout=10.0)
|
||||
|
||||
if self.client:
|
||||
if self.client.collection_exists(collection_name):
|
||||
@@ -112,8 +115,8 @@ class QdrantBase:
|
||||
args = self._cached_args
|
||||
|
||||
# Pull specific embedding config or fallback to defaults
|
||||
model = getattr(args, "ai_embedding_model", "nomic-embed-text")
|
||||
url = getattr(args, "ai_embedding_url", "http://localhost:11434/api/embeddings")
|
||||
model = getattr(args, "ai_embedding_model")
|
||||
url = getattr(args, "ai_embedding_url")
|
||||
|
||||
try:
|
||||
# Generate embeddings
|
||||
|
||||
@@ -386,8 +386,8 @@ class ResonanceEngine:
|
||||
"Set 'keep' to false only for clear spam, bots, UI buttons, or blacklist violations.\n"
|
||||
)
|
||||
|
||||
model = getattr(configs.args, "ai_condenser_model", "llama3.2:1b")
|
||||
url = getattr(configs.args, "ai_condenser_url", "http://localhost:11434/api/generate")
|
||||
model = getattr(configs.args, "ai_condenser_model")
|
||||
url = getattr(configs.args, "ai_condenser_url")
|
||||
|
||||
try:
|
||||
import json
|
||||
|
||||
@@ -8,6 +8,11 @@ and learns from every episode — positive AND negative.
|
||||
|
||||
After initial learning, 95%+ of situations are handled from memory
|
||||
alone with ZERO LLM calls. This is "Tesla fleet learning" for bots.
|
||||
|
||||
CRITICAL ARCHITECTURE RULE: Language Agnosticism & Structural Determinism
|
||||
NO hardcoded localized UI strings (e.g. "Report", "OK", "Deny") are permitted.
|
||||
All structural heuristic fallbacks must use O(1) Android `resource-id` bounds
|
||||
to ensure the bot functions flawlessly regardless of the device language.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
@@ -291,45 +296,53 @@ class SituationalAwarenessEngine:
|
||||
stable = re.sub(r"Battery \d+ per cent", "Battery NN per cent", stable)
|
||||
return hashlib.sha256(stable.encode()).hexdigest()[:32]
|
||||
|
||||
def _get_model_config(self):
|
||||
"""
|
||||
Get model/URL config from Config() — the SINGLE source of truth.
|
||||
Crashes loudly if config is unavailable (Fail Fast, no silent defaults).
|
||||
"""
|
||||
from GramAddict.core.config import Config
|
||||
|
||||
cfg = Config()
|
||||
args = cfg.args
|
||||
return {
|
||||
"model": getattr(args, "ai_model", None),
|
||||
"url": getattr(args, "ai_model_url", None),
|
||||
"telepathic_model": getattr(args, "ai_telepathic_model", None),
|
||||
"telepathic_url": getattr(args, "ai_telepathic_url", None),
|
||||
}
|
||||
|
||||
def perceive(self, xml_dump: str) -> SituationType:
|
||||
"""
|
||||
Fast structural classification — NO LLM needed for perception.
|
||||
Uses package names + structural markers to classify.
|
||||
Autonomous situation classification — ZERO hardcoded UI element identifiers.
|
||||
|
||||
Flow:
|
||||
1. Empty/invalid XML → FOREIGN_APP
|
||||
2. Hardware check (screen off) → LOCKED_SCREEN
|
||||
3. Package-based detection (app-agnostic, zero maintenance):
|
||||
- Permission controller packages → OBSTACLE_SYSTEM
|
||||
- App package missing → OBSTACLE_FOREIGN_APP
|
||||
4. Qdrant semantic cache → instant recall of learned screen types
|
||||
5. ScreenIdentity structural delegation → if MODAL, return OBSTACLE_MODAL
|
||||
6. LLM classification fallback → autonomous discovery of new obstacle types
|
||||
|
||||
NO hardcoded resource-ids, NO hardcoded button texts, NO localized strings.
|
||||
The bot discovers and learns ALL obstacles autonomously via the LLM + Qdrant loop.
|
||||
"""
|
||||
if not xml_dump or not isinstance(xml_dump, str):
|
||||
return SituationType.OBSTACLE_FOREIGN_APP
|
||||
|
||||
blocked_markers = [
|
||||
"try again later",
|
||||
"action blocked",
|
||||
"restrict certain activity",
|
||||
"help us confirm you own",
|
||||
"confirm it's you",
|
||||
"später erneut versuchen",
|
||||
"bestätige, dass du es bist",
|
||||
"handlung blockiert",
|
||||
"eingeschränkt",
|
||||
]
|
||||
|
||||
# Guard: Check if the text matches are relatively isolated (e.g. short strings).
|
||||
# If the string is buried inside a 200-character caption, it's a false positive.
|
||||
# We can regex match text="..." attributes that are less than 60 characters total,
|
||||
# OR just use the compressed string where text is capped at 60 chars anyway.
|
||||
compressed_lower = self._compress_xml(xml_dump).lower()
|
||||
if any(re.search(rf"(?:text|desc)='[^']*?{m}[^']*?'", compressed_lower) for m in blocked_markers):
|
||||
# To be extra safe against false positives, check if there's a dialog/modal container
|
||||
if "dialog" in compressed_lower or "bottom_sheet" in compressed_lower or "alert" in compressed_lower:
|
||||
return SituationType.DANGER_ACTION_BLOCKED
|
||||
|
||||
# ── Hardware Guard: Screen Off / Locked ──
|
||||
if not getattr(self.device.deviceV2, "info", {}).get("screenOn", True):
|
||||
logger.info("📱 [SAE Perceive] Screen is physically OFF.")
|
||||
return SituationType.OBSTACLE_LOCKED_SCREEN
|
||||
|
||||
# ── System Dialog / Permission Detect (Fast Path) ──
|
||||
packages = set(re.findall(r'package=["\']([^"\']+)["\']', xml_dump))
|
||||
# ── System Dialog / Permission Detect (package-based, app-agnostic) ──
|
||||
packages = set(re.findall(r'package=["\'](.[^"\']+)["\']+', xml_dump))
|
||||
app_id = getattr(self.device, "app_id", "com.instagram.android")
|
||||
|
||||
# Permission controller packages are Android system-level — not app-specific.
|
||||
# These will NEVER change with an Instagram update. Completely safe to check.
|
||||
system_dialog_pkgs = {
|
||||
"com.google.android.permissioncontroller",
|
||||
"com.android.permissioncontroller",
|
||||
@@ -339,46 +352,24 @@ class SituationalAwarenessEngine:
|
||||
logger.info("📱 [SAE Perceive] System permission dialog explicitly detected.")
|
||||
return SituationType.OBSTACLE_SYSTEM
|
||||
|
||||
# ── Foreign Environment Detection (package-based) ──
|
||||
# If the main app package is completely absent from the UI hierarchy,
|
||||
# OR if there's a dominant foreign package and no app package, we might have lost the app.
|
||||
|
||||
# If our app is on screen, we trust we are in the app (even if a custom keyboard is open).
|
||||
# We only trigger foreign app classification if our app is completely missing from the screen.
|
||||
is_foreign = False
|
||||
if packages and app_id not in packages:
|
||||
is_foreign = True
|
||||
# ── Foreign Environment Detection (package-based, zero maintenance) ──
|
||||
# If our app package is completely absent from the screen → foreign app.
|
||||
# Package names are Android-level identifiers, NOT Instagram UI elements.
|
||||
is_foreign = bool(packages) and app_id not in packages
|
||||
|
||||
if is_foreign:
|
||||
# ── Tier 1: Known Foreign Packages (O(1) — ZERO LLM) ──
|
||||
# Production bug 2026-05-03: Play Store was detected via slow LLM path.
|
||||
# For these well-known packages, a set lookup is instant and infallible.
|
||||
KNOWN_FOREIGN_PACKAGES = {
|
||||
"com.android.vending", # Play Store
|
||||
"com.android.chrome", # Chrome
|
||||
"com.google.android.chrome", # Chrome (Google build)
|
||||
"com.google.android.youtube", # YouTube
|
||||
"org.mozilla.firefox", # Firefox
|
||||
"com.opera.browser", # Opera
|
||||
"com.brave.browser", # Brave
|
||||
"com.microsoft.emmx", # Edge
|
||||
"com.sec.android.app.sbrowser", # Samsung Browser
|
||||
}
|
||||
# Any package that isn't our app AND isn't just systemui = foreign
|
||||
dominant_pkgs = packages - {"com.android.systemui"}
|
||||
fast_match = dominant_pkgs & KNOWN_FOREIGN_PACKAGES
|
||||
if fast_match:
|
||||
logger.info(
|
||||
f"🚨 [SAE Perceive] Known foreign package: {fast_match} → "
|
||||
f"OBSTACLE_FOREIGN_APP (O(1) fast-path, no LLM needed)"
|
||||
)
|
||||
if dominant_pkgs:
|
||||
logger.info(f"🚨 [SAE Perceive] Foreign package detected: {dominant_pkgs} → OBSTACLE_FOREIGN_APP")
|
||||
return SituationType.OBSTACLE_FOREIGN_APP
|
||||
|
||||
# ── Tier 2: Unknown/Ambiguous Packages → LLM Classification ──
|
||||
# Only SystemUI-only or rare custom packages reach this path.
|
||||
# SystemUI-only edge case: could be lock screen, notification shade, etc.
|
||||
# Use LLM for disambiguation.
|
||||
try:
|
||||
from GramAddict.core.config import Config
|
||||
from GramAddict.core.llm_provider import query_telepathic_llm
|
||||
|
||||
cfg = self._get_model_config()
|
||||
screen_off = not getattr(self.device.deviceV2, "info", {}).get("screenOn", True)
|
||||
|
||||
prompt = (
|
||||
@@ -391,17 +382,9 @@ class SituationalAwarenessEngine:
|
||||
f"XML:\n{self._compress_xml(xml_dump)[:2500]}"
|
||||
)
|
||||
|
||||
args = {}
|
||||
try:
|
||||
args = Config().args
|
||||
except Exception:
|
||||
pass
|
||||
model = getattr(args, "ai_model", "qwen3.5:latest")
|
||||
url = getattr(args, "ai_model_url", "http://localhost:11434/api/generate")
|
||||
|
||||
res = query_telepathic_llm(
|
||||
model=model,
|
||||
url=url,
|
||||
model=cfg["model"],
|
||||
url=cfg["url"],
|
||||
system_prompt="Strict JSON classifier.",
|
||||
user_prompt=prompt,
|
||||
use_local_edge=True,
|
||||
@@ -412,140 +395,91 @@ class SituationalAwarenessEngine:
|
||||
situ_str = data.get("situation", "")
|
||||
|
||||
if situ_str == "OBSTACLE_LOCKED_SCREEN":
|
||||
logger.info("🧠 [Smart Perceive] SystemUI definitively classified as: LOCKED_SCREEN.")
|
||||
logger.info("🧠 [Smart Perceive] SystemUI classified as: LOCKED_SCREEN.")
|
||||
return SituationType.OBSTACLE_LOCKED_SCREEN
|
||||
elif situ_str == "OBSTACLE_SYSTEM":
|
||||
logger.info("🧠 [Smart Perceive] SystemUI definitively classified as: SYSTEM_DIALOG.")
|
||||
logger.info("🧠 [Smart Perceive] SystemUI classified as: SYSTEM_DIALOG.")
|
||||
return SituationType.OBSTACLE_SYSTEM
|
||||
else:
|
||||
logger.info("🧠 [Smart Perceive] SystemUI classified as: FOREIGN_APP / NOTIFICATION.")
|
||||
logger.info("🧠 [Smart Perceive] SystemUI classified as: FOREIGN_APP.")
|
||||
return SituationType.OBSTACLE_FOREIGN_APP
|
||||
except Exception as e:
|
||||
logger.warning(f"⚠️ [Smart Perceive] LLM Classification failed ({e}). Defaulting to FOREIGN_APP.")
|
||||
return SituationType.OBSTACLE_FOREIGN_APP
|
||||
|
||||
# ── Modal/Obstacle Detection (Autonomous LLM + Memory) ──
|
||||
# We explicitly query ScreenMemoryDB. If unknown, we ask the LLM.
|
||||
# This replaces ALL brittle string/ID matching for modals.
|
||||
# ── In-App Obstacle Detection (100% autonomous, ZERO hardcoded UI identifiers) ──
|
||||
# The bot learns ALL obstacle types via the LLM + Qdrant feedback loop.
|
||||
# First encounter: LLM classifies → result is cached in Qdrant.
|
||||
# All subsequent encounters: instant O(1) recall from cache.
|
||||
from GramAddict.core.qdrant_memory import ScreenMemoryDB
|
||||
|
||||
screen_memory = ScreenMemoryDB()
|
||||
|
||||
compressed = self._compress_xml(xml_dump)
|
||||
|
||||
# ── Priority 1: Qdrant Semantic Cache (O(1), zero LLM calls) ──
|
||||
cached_type = screen_memory.get_screen_type(compressed)
|
||||
|
||||
if cached_type:
|
||||
if cached_type == "OBSTACLE_MODAL":
|
||||
return SituationType.OBSTACLE_MODAL
|
||||
elif cached_type == "DANGER_ACTION_BLOCKED":
|
||||
return SituationType.DANGER_ACTION_BLOCKED
|
||||
elif cached_type == "NORMAL":
|
||||
return SituationType.NORMAL
|
||||
|
||||
# ── Structural Fast-Check: Content-Creation Overlays ──
|
||||
# These full-screen overlays live INSIDE Instagram's package but block
|
||||
# all normal navigation. They are invisible to the foreign-app detector
|
||||
# and frequently fool the LLM into thinking they are "normal" browsing.
|
||||
# Detecting them structurally is O(1) and requires ZERO LLM calls.
|
||||
# This is checked AFTER Qdrant to ensure that if the LLM unlearned a false positive,
|
||||
# we respect the learned NORMAL state and don't infinite-loop.
|
||||
creation_flow_markers = (
|
||||
"quick_capture", # Camera / story capture overlay
|
||||
"gallery_cancel_button", # Story gallery "Back to Home" button
|
||||
"creation_flow", # Post creation wizard
|
||||
"reel_camera", # Reel recording interface
|
||||
)
|
||||
# ── Priority 2: ScreenIdentity structural delegation ──
|
||||
# ScreenIdentity classifies the screen using its own structural logic.
|
||||
# If it says MODAL → trust it (it uses the same zero-trust approach).
|
||||
from GramAddict.core.perception.screen_identity import ScreenIdentity, ScreenType
|
||||
|
||||
# Guard: Use the RAW xml_dump to avoid truncation of root containers (Z-index filtering),
|
||||
# but ensure we only match inside resource-id attributes to prevent false positives from user text.
|
||||
if any(
|
||||
re.search(rf'resource-id="[^"]*{marker}[^"]*"', xml_dump, re.IGNORECASE) for marker in creation_flow_markers
|
||||
):
|
||||
logger.info("🧠 [SAE Perceive] Content-creation overlay detected structurally → OBSTACLE_MODAL")
|
||||
screen_id = ScreenIdentity(getattr(self.device, "bot_username", ""))
|
||||
screen_result = screen_id.identify(xml_dump)
|
||||
screen_type = screen_result.get("screen_type", ScreenType.UNKNOWN)
|
||||
|
||||
if screen_type == ScreenType.MODAL:
|
||||
logger.info("🧠 [SAE Perceive] ScreenIdentity classified as MODAL → OBSTACLE_MODAL")
|
||||
screen_memory.store_screen(compressed, "OBSTACLE_MODAL")
|
||||
return SituationType.OBSTACLE_MODAL
|
||||
|
||||
# ── Structural Fast-Check: Instagram-Internal Modal Overlays ──
|
||||
# Surveys, rating prompts, and interstitial modals live INSIDE Instagram's
|
||||
# package but block normal interaction. They share a common structural
|
||||
# pattern: a container resource-id containing "survey", "interstitial",
|
||||
# or "nux_" (new-user-experience), plus dismiss buttons ("Not Now").
|
||||
# Detecting them structurally is O(1) and eliminates LLM hallucination risk.
|
||||
instagram_modal_markers = (
|
||||
"survey_overlay_container", # "How are you enjoying Instagram?" survey
|
||||
"survey_title", # Survey title text view
|
||||
"interstitial_container", # Generic interstitial blocker
|
||||
"mystery_interstitial", # Unknown/dynamic interstitials
|
||||
"nux_overlay", # New-user-experience onboarding modals
|
||||
"rating_prompt", # App Store rating prompt
|
||||
"feedback_dialog", # Feedback collection dialogs
|
||||
)
|
||||
if any(
|
||||
re.search(rf'resource-id="[^"]*{marker}[^"]*"', xml_dump, re.IGNORECASE)
|
||||
for marker in instagram_modal_markers
|
||||
):
|
||||
logger.info("🧠 [SAE Perceive] Instagram modal overlay detected structurally → OBSTACLE_MODAL")
|
||||
screen_memory.store_screen(compressed, "OBSTACLE_MODAL")
|
||||
return SituationType.OBSTACLE_MODAL
|
||||
if screen_type == ScreenType.FOREIGN_APP:
|
||||
logger.info("🧠 [SAE Perceive] ScreenIdentity classified as FOREIGN_APP → OBSTACLE_FOREIGN_APP")
|
||||
return SituationType.OBSTACLE_FOREIGN_APP
|
||||
|
||||
# Fallback heuristic: detect modals by dismiss-button text patterns.
|
||||
# If we see "Not Now" or "Take Survey" as button text inside Instagram, it's a modal.
|
||||
# Guard: match ONLY inside short text attributes (< 40 chars) to avoid caption false positives.
|
||||
dismiss_button_patterns = (
|
||||
r'text="Not Now"',
|
||||
r'text="not now"',
|
||||
r'text="Nicht jetzt"', # German: "Not Now"
|
||||
r'text="Take Survey"',
|
||||
r'text="rate \d+ stars?"', # "rate 5 stars"
|
||||
r'text="Bewerten"', # German: "Rate"
|
||||
)
|
||||
has_dismiss_button = any(re.search(p, xml_dump, re.IGNORECASE) for p in dismiss_button_patterns)
|
||||
if has_dismiss_button:
|
||||
# Cross-validate: must also have a container that looks like a dialog/overlay
|
||||
# (not just a random "Not Now" text in a DM thread or post caption)
|
||||
has_overlay_structure = bool(
|
||||
re.search(
|
||||
r'resource-id="[^"]*(?:overlay|dialog|interstitial|survey|sheet|prompt)[^"]*"',
|
||||
xml_dump,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
or re.search(r'resource-id="[^"]*button_(?:negative|positive)[^"]*"', xml_dump, re.IGNORECASE)
|
||||
)
|
||||
if has_overlay_structure:
|
||||
logger.info("🧠 [SAE Perceive] Instagram dismiss-button modal detected structurally → OBSTACLE_MODAL")
|
||||
screen_memory.store_screen(compressed, "OBSTACLE_MODAL")
|
||||
return SituationType.OBSTACLE_MODAL
|
||||
if screen_type == ScreenType.DANGER_ACTION_BLOCKED:
|
||||
logger.info("🧠 [SAE Perceive] ScreenIdentity classified as DANGER_ACTION_BLOCKED")
|
||||
screen_memory.store_screen(compressed, "DANGER_ACTION_BLOCKED")
|
||||
return SituationType.DANGER_ACTION_BLOCKED
|
||||
|
||||
# If not cached, query LLM for autonomous structural classification
|
||||
# If ScreenIdentity positively identified a known screen type (not UNKNOWN),
|
||||
# we trust it as NORMAL — no LLM needed.
|
||||
if screen_type != ScreenType.UNKNOWN:
|
||||
screen_memory.store_screen(compressed, "NORMAL")
|
||||
return SituationType.NORMAL
|
||||
|
||||
# ── Priority 3: LLM autonomous classification (first-encounter learning) ──
|
||||
# This is the ONLY path that reaches the LLM. After classification,
|
||||
# the result is cached in Qdrant — so this screen type is learned forever.
|
||||
try:
|
||||
from GramAddict.core.config import Config
|
||||
from GramAddict.core.llm_provider import query_telepathic_llm
|
||||
|
||||
cfg = self._get_model_config()
|
||||
|
||||
prompt = (
|
||||
"You are a Situation Classifier for a mobile automation agent.\n"
|
||||
"Analyze the given Android UI XML dump AND screenshot. Is there a blocking MODAL, DIALOG, or POPUP "
|
||||
"covering the screen that needs to be dismissed, or is this a NORMAL usable screen?\n"
|
||||
"A 'clean_sheet_container' with standard Instagram feed content is NORMAL.\n"
|
||||
"A survey, rating prompt, 'not now' prompt, or permission dialog is an OBSTACLE_MODAL.\n"
|
||||
"An 'Add to story' screen, camera interface, 'quick_capture' layout, gallery picker, "
|
||||
"or ANY content-creation flow (reel recording, post editor, live mode) is an OBSTACLE_MODAL — "
|
||||
"it blocks normal navigation and must be dismissed.\n"
|
||||
"Respond ONLY with a valid JSON object strictly matching this schema: "
|
||||
'{"situation": "OBSTACLE_MODAL" | "NORMAL"}\n\n'
|
||||
"Analyze the given Android UI XML dump AND screenshot. Classify the screen into one of:\n"
|
||||
"- OBSTACLE_MODAL: Any blocking overlay, dialog, popup, survey, rating prompt, "
|
||||
"browser window, camera/creation flow, or any UI that blocks normal feed browsing.\n"
|
||||
"- DANGER_ACTION_BLOCKED: Instagram's rate-limit or action-block warning "
|
||||
"(e.g. 'Try Again Later', 'Action Blocked', or any restriction/verification screen).\n"
|
||||
"- NORMAL: A standard usable screen (feed, explore, profile, DM, etc.)\n\n"
|
||||
"Respond ONLY with a valid JSON object: "
|
||||
'{"situation": "OBSTACLE_MODAL" | "DANGER_ACTION_BLOCKED" | "NORMAL"}\n\n'
|
||||
f"XML:\n{compressed[:2500]}"
|
||||
)
|
||||
|
||||
args = {}
|
||||
try:
|
||||
args = Config().args
|
||||
except Exception:
|
||||
pass
|
||||
model = getattr(args, "ai_telepathic_model", "llava:latest")
|
||||
url = getattr(args, "ai_telepathic_url", "http://localhost:11434/api/generate")
|
||||
|
||||
screenshot_b64 = getattr(self.device, "get_screenshot_b64", lambda: None)()
|
||||
res = query_telepathic_llm(
|
||||
model=model,
|
||||
url=url,
|
||||
model=cfg["telepathic_model"],
|
||||
url=cfg["telepathic_url"],
|
||||
system_prompt="Strict JSON classifier.",
|
||||
user_prompt=prompt,
|
||||
images_b64=[screenshot_b64] if screenshot_b64 else None,
|
||||
@@ -560,6 +494,10 @@ class SituationalAwarenessEngine:
|
||||
logger.info("🧠 [Smart Perceive] Screen classified as: OBSTACLE_MODAL.")
|
||||
screen_memory.store_screen(compressed, "OBSTACLE_MODAL")
|
||||
return SituationType.OBSTACLE_MODAL
|
||||
elif situ_str == "DANGER_ACTION_BLOCKED":
|
||||
logger.info("🧠 [Smart Perceive] Screen classified as: DANGER_ACTION_BLOCKED.")
|
||||
screen_memory.store_screen(compressed, "DANGER_ACTION_BLOCKED")
|
||||
return SituationType.DANGER_ACTION_BLOCKED
|
||||
else:
|
||||
logger.info("🧠 [Smart Perceive] Screen classified as: NORMAL.")
|
||||
screen_memory.store_screen(compressed, "NORMAL")
|
||||
@@ -589,16 +527,11 @@ class SituationalAwarenessEngine:
|
||||
LLM-powered escape planning for situations where structural scan fails.
|
||||
Called ONLY when recall AND structural planning both miss.
|
||||
"""
|
||||
from GramAddict.core.config import Config
|
||||
from GramAddict.core.llm_provider import query_telepathic_llm
|
||||
|
||||
try:
|
||||
args = Config().args
|
||||
model = getattr(args, "ai_telepathic_model", "llava:latest")
|
||||
url = getattr(args, "ai_telepathic_url", "http://localhost:11434/api/generate")
|
||||
except Exception:
|
||||
model = "llava:latest"
|
||||
url = "http://localhost:11434/api/generate"
|
||||
cfg = self._get_model_config()
|
||||
model = cfg["telepathic_model"]
|
||||
url = cfg["telepathic_url"]
|
||||
|
||||
system_prompt = (
|
||||
"You are an Android UI navigation agent. Your job is to escape obstacles "
|
||||
@@ -608,12 +541,12 @@ class SituationalAwarenessEngine:
|
||||
"- If you see a dismiss/close/cancel/skip/not now button, click it\n"
|
||||
"- If the Situation type is obstacle_locked_screen, action must be 'unlock'\n"
|
||||
"- If the Situation type is obstacle_foreign_app, action must be 'kill_foreign_apps'\n"
|
||||
"- If the Situation type is obstacle_system, you MUST look for 'Deny', 'Don't allow', or 'Cancel' and click it. \n"
|
||||
" NEVER click 'Allow', 'OK', or 'Confirm' on system permissions.\n"
|
||||
"- If the Situation type is obstacle_system, you MUST look for the negative action (e.g. deny, block, do not allow, cancel) and click it. \n"
|
||||
" NEVER click the positive action (allow, accept, confirm) on system permissions.\n"
|
||||
" If no negative action button exists, action must be 'back'\n"
|
||||
"- If there is NO obstacle and the screen is a normal Instagram view (false positive), action must be 'false_positive'\n"
|
||||
"- If nothing else works, suggest 'app_start' to force-reopen Instagram\n"
|
||||
"- NEVER click 'OK'/'Confirm'/'Accept' on surveys or prompts\n"
|
||||
"- NEVER click the positive/accept button on surveys or prompts\n"
|
||||
"- When you choose to click, you MUST use the EXACT coordinates provided in `center=(x,y)` for that element in the XML\n"
|
||||
'- Return ONLY valid JSON: {"action": "click"|"back"|"app_start"|"unlock"|"kill_foreign_apps"|"false_positive", "x": N, "y": N, "reason": "..."}'
|
||||
)
|
||||
|
||||
@@ -85,6 +85,24 @@ class TelepathicEngine:
|
||||
filtered_candidates.append(c)
|
||||
candidates = filtered_candidates
|
||||
|
||||
# --- Active Keyboard Dismiss Guard ---
|
||||
# If keyboard is open but intent is NOT typing related -> dismiss it!
|
||||
intent_lower = intent_description.lower()
|
||||
if device and self._resolver.has_keyboard_open(candidates):
|
||||
typing_keywords = ["type", "message", "comment", "search", "write"]
|
||||
if not any(k in intent_lower for k in typing_keywords):
|
||||
logger.warning("⌨️ [TelepathicEngine] Keyboard detected during non-typing intent! Auto-dismissing.")
|
||||
import time
|
||||
|
||||
device.back()
|
||||
time.sleep(1.0)
|
||||
# Re-fetch UI state
|
||||
xml_string = device.dump_hierarchy()
|
||||
if xml_string:
|
||||
root = self._parser.parse(xml_string)
|
||||
if root:
|
||||
candidates = self._parser.get_clickable_nodes(root)
|
||||
|
||||
# 3. Resolve intent against candidates
|
||||
best_node = self._resolver.resolve(intent_description, candidates, device=device)
|
||||
|
||||
@@ -112,8 +130,8 @@ class TelepathicEngine:
|
||||
(best_node.text or "") + " " + (best_node.content_desc or "") + " " + (best_node.resource_id or "")
|
||||
)
|
||||
semantic = semantic.lower()
|
||||
# Zero-Maintenance: Only English UI states. resource_id never changes with locale.
|
||||
if "following" in semantic or "requested" in semantic:
|
||||
# Zero-Maintenance: Strictly structural IDs, no localized strings.
|
||||
if "button_following" in semantic or "profile_header_following" in semantic:
|
||||
return {"skip": True, "semantic": "already_followed"}
|
||||
|
||||
# 4. Track action
|
||||
@@ -133,6 +151,7 @@ class TelepathicEngine:
|
||||
"description": node.content_desc,
|
||||
"id": node.resource_id,
|
||||
"class": node.class_name,
|
||||
"semantic": node.content_desc or node.text or node.resource_id,
|
||||
"original_attribs": node.to_dict(),
|
||||
}
|
||||
|
||||
@@ -183,13 +202,19 @@ class TelepathicEngine:
|
||||
if not root:
|
||||
return None
|
||||
|
||||
# Find grid-like visual nodes
|
||||
# Find grid-like visual nodes using structural markers
|
||||
all_nodes = self._parser.get_all_nodes(root)
|
||||
grid_candidates = [
|
||||
n
|
||||
for n in all_nodes
|
||||
if not n.text and ("photo" in n.content_desc.lower() or "video" in n.content_desc.lower() or n.area > 50000)
|
||||
]
|
||||
grid_candidates = []
|
||||
for n in all_nodes:
|
||||
rid = (n.resource_id or "").lower()
|
||||
desc = (n.content_desc or "").lower()
|
||||
|
||||
# Structural anchor for Instagram grid items
|
||||
if "grid_card_layout_container" in rid:
|
||||
grid_candidates.append(n)
|
||||
# Fallback for older versions or profile grids if resource-id is missing
|
||||
elif not n.text and ("photo" in desc or "video" in desc) and (50000 < n.area < 400000):
|
||||
grid_candidates.append(n)
|
||||
|
||||
best = self._evaluator.evaluate_grid_visuals(device, persona_interests, grid_candidates)
|
||||
if best:
|
||||
|
||||
@@ -105,7 +105,7 @@ def _run_zero_latency_unfollow_loop(
|
||||
|
||||
# 2. Close Friend Guard
|
||||
profile_text = profile_xml.lower()
|
||||
if "enge freunde" in profile_text or "close friend" in profile_text:
|
||||
if "close friend" in profile_text:
|
||||
logger.info(
|
||||
"💚 [Anti-Friend] Profile is a Close Friend. Skipping unfollow.", extra={"color": Fore.GREEN}
|
||||
)
|
||||
@@ -127,19 +127,23 @@ def _run_zero_latency_unfollow_loop(
|
||||
extra={"color": Fore.YELLOW},
|
||||
)
|
||||
|
||||
# Find 'Following' button on their profile
|
||||
# Find the button that indicates active subscription (Following)
|
||||
following_nodes = telepathic._extract_semantic_nodes(
|
||||
profile_xml, "find 'Following' button", threshold=0.7
|
||||
profile_xml,
|
||||
"find the button indicating active subscription, look for id 'button_following' or 'profile_header_following'",
|
||||
threshold=0.7,
|
||||
)
|
||||
if following_nodes and not following_nodes[0].get("skip"):
|
||||
f_node = following_nodes[0]
|
||||
_humanized_click(device, f_node["x"], f_node["y"])
|
||||
random_sleep(1.0, 2.0)
|
||||
|
||||
# Find 'Unfollow' confirm
|
||||
# Find the confirmation button
|
||||
confirm_xml = device.dump_hierarchy()
|
||||
confirm_nodes = telepathic._extract_semantic_nodes(
|
||||
confirm_xml, "find 'Unfollow' confirmation button", threshold=0.8
|
||||
confirm_xml,
|
||||
"find the confirmation button to stop following, look for id 'follow_sheet_unfollow_row' or red warning text",
|
||||
threshold=0.8,
|
||||
)
|
||||
|
||||
if confirm_nodes and not confirm_nodes[0].get("skip"):
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import logging
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import random
|
||||
from time import sleep
|
||||
@@ -100,11 +100,12 @@ def get_value(count, name, default=0):
|
||||
_LEARNED_AD_MARKERS_FILE = os.path.join(os.getcwd(), "learned_ad_markers.json")
|
||||
_LEARNED_AD_MARKERS_CACHE = None
|
||||
|
||||
|
||||
def get_learned_ad_markers() -> set:
|
||||
global _LEARNED_AD_MARKERS_CACHE
|
||||
if _LEARNED_AD_MARKERS_CACHE is not None:
|
||||
return _LEARNED_AD_MARKERS_CACHE
|
||||
|
||||
|
||||
if os.path.exists(_LEARNED_AD_MARKERS_FILE):
|
||||
try:
|
||||
with open(_LEARNED_AD_MARKERS_FILE, "r") as f:
|
||||
@@ -114,18 +115,20 @@ def get_learned_ad_markers() -> set:
|
||||
_LEARNED_AD_MARKERS_CACHE = set()
|
||||
else:
|
||||
_LEARNED_AD_MARKERS_CACHE = set()
|
||||
|
||||
|
||||
return _LEARNED_AD_MARKERS_CACHE
|
||||
|
||||
|
||||
def learn_ad_marker(marker: str, xml_hierarchy: str):
|
||||
global _LEARNED_AD_MARKERS_CACHE
|
||||
if not marker or len(marker) > 30:
|
||||
return
|
||||
|
||||
|
||||
marker = marker.strip().lower()
|
||||
|
||||
|
||||
# Structural verification: the VLM-suggested marker MUST exist as an exact node text/desc in the current UI!
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
try:
|
||||
root = ET.fromstring(xml_hierarchy)
|
||||
found_in_ui = False
|
||||
@@ -135,17 +138,22 @@ def learn_ad_marker(marker: str, xml_hierarchy: str):
|
||||
if text == marker or desc == marker:
|
||||
found_in_ui = True
|
||||
break
|
||||
|
||||
|
||||
if not found_in_ui:
|
||||
logger.debug(f"🧠 [Autonomous FSD] Rejected hallucinated Ad marker '{marker}' (not found as exact node match in UI).")
|
||||
logger.debug(
|
||||
f"🧠 [Autonomous FSD] Rejected hallucinated Ad marker '{marker}' (not found as exact node match in UI)."
|
||||
)
|
||||
return
|
||||
except Exception:
|
||||
return
|
||||
|
||||
markers = get_learned_ad_markers()
|
||||
if marker not in markers and marker not in {"ad", "sponsored", "advertisement", "gesponsert", "anzeige", "werbung"}:
|
||||
if marker not in markers and marker not in {"ad", "sponsored", "advertisement"}:
|
||||
markers.add(marker)
|
||||
logger.info(f"🧠 [Autonomous FSD] Verified and Learned new Ad marker: '{marker}'. Persisting for zero-latency detection.", extra={"color": f"{Style.BRIGHT}{Fore.GREEN}"})
|
||||
logger.info(
|
||||
f"🧠 [Autonomous FSD] Verified and Learned new Ad marker: '{marker}'. Persisting for zero-latency detection.",
|
||||
extra={"color": f"{Style.BRIGHT}{Fore.GREEN}"},
|
||||
)
|
||||
try:
|
||||
with open(_LEARNED_AD_MARKERS_FILE, "w") as f:
|
||||
json.dump(list(markers), f)
|
||||
@@ -182,14 +190,15 @@ def is_ad(xml_hierarchy: str, cognitive_stack: dict = None) -> bool:
|
||||
|
||||
# Standalone label patterns: match only when the text/desc IS the ad marker,
|
||||
# not when "ad" appears inside longer phrases like "Create messaging ad"
|
||||
AD_EXACT_LABELS = {"ad", "sponsored", "advertisement", "gesponsert", "anzeige", "werbung"}
|
||||
AD_EXACT_LABELS = {"ad", "sponsored", "advertisement"}
|
||||
AD_EXACT_LABELS.update(get_learned_ad_markers())
|
||||
|
||||
try:
|
||||
root = ET.fromstring(xml_hierarchy)
|
||||
|
||||
|
||||
# Check if we are in a feed (to prevent false positives on profiles with 'Ad Tools' buttons)
|
||||
from GramAddict.core.perception.feed_analysis import FEED_MARKERS
|
||||
|
||||
in_feed = any(marker in xml_hierarchy for marker in FEED_MARKERS)
|
||||
|
||||
for node in root.iter("node"):
|
||||
|
||||
13
README.md
13
README.md
@@ -63,3 +63,16 @@ The engine has undergone a massive stabilization refactor to achieve **100% TDD
|
||||
|
||||
> [!NOTE]
|
||||
> Unlike legacy bots, GramPilot requires zero maintenance. It will automatically re-learn the UI over time using its integrated Qdrant memory vectors.
|
||||
|
||||
---
|
||||
|
||||
## 🏛️ Core Architecture Rules: Language Agnosticism & Structural Determinism
|
||||
|
||||
**CRITICAL: Hardcoding localized UI strings (e.g., "Follow", "Like", "Send", "Report") is STRICTLY FORBIDDEN across the entire codebase.**
|
||||
|
||||
GramPilot operates on a globally language-agnostic plane. If the UI is switched to German, Arabic, or Japanese, the bot must not fail. To achieve this, all perception and navigation logic must adhere to the following strict rules:
|
||||
|
||||
1. **Spatial Geometry over Text:** Use coordinates and boundaries to identify elements. (e.g., The "Send" button in DMs is *always* on the far right `center_x > width * 0.75`).
|
||||
2. **Structural Resource IDs over Descriptions:** Use Android UI resource IDs (`action_sheet_row_text_view`, `button_following`) instead of English `desc` or `text` properties. IDs are not localized and are universally stable.
|
||||
3. **Semantic LLM Prompts:** When querying the Vision-Language Model (VLM), **do not give exact string examples** that assume an English UI. For example, do not say `find the 'Unfollow' button`. Instead, say `find the button to stop following, look for a red warning text or id 'follow_sheet_unfollow_row'`.
|
||||
4. **Zero Magie:** Any text-based matching logic is considered a legacy flaw and will be mercilessly purged. If you find a text match, replace it with a structural O(1) fast-path or spatial guard.
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user