fix(navigation): eliminate VLM hallucination on bottom navigation tabs via structural guard

Added a 'Structural Navigation Guard' in IntentResolver to map critical bottom navigation intents (e.g., 'tap profile tab') directly to their stable resource-ids. This bypasses the VLM entirely, guaranteeing 100% deterministic clicks and resolving the issue where the VLM failed to locate the profile tab, causing the edge to become masked and trapping the bot on the home feed.
Included TDD proof.
This commit is contained in:
2026-04-29 01:16:06 +02:00
parent 71310b8e84
commit b9c29a5a2d
3 changed files with 66 additions and 10 deletions

View File

@@ -254,11 +254,27 @@ class IntentResolver:
and "per cent" not in (n.content_desc or "").lower()
]
# --- Structural Navigation Guard ---
# For strictly defined navigation tabs, we bypass the VLM entirely.
tab_mapping = {
"tap home tab": "feed_tab",
"tap explore tab": "search_tab",
"tap reels tab": "clips_tab",
"tap profile tab": "profile_tab",
"tap messages tab": "direct_tab",
}
intent_lower = intent_description.lower()
if intent_lower in tab_mapping:
target_id = tab_mapping[intent_lower]
for node in candidates:
if target_id in (node.resource_id or ""):
logger.info(f"🎯 [Navigation Guard] Exact match found for '{intent_lower}' via resource-id. Skipping VLM.")
return node
# --- Strict Button Guard ---
# If the intent specifically asks for a "button", "icon", or "tab",
# filter out candidates that contain long text (e.g. captions, comments)
# to prevent the VLM from hallucinating text nodes as interactive buttons.
intent_lower = intent_description.lower()
if "button" in intent_lower or "icon" in intent_lower or "tab" in intent_lower:
filtered_candidates = []
for node in candidates: