fix(perception): enforce strict candidate filtering for grid items

In addition to prompt tuning, we now apply a pre-flight structural guard for 'tap first post' / 'tap first grid item' intents.
If the intent targets a grid item, we pre-filter the candidates to only include those matching 'row X, column Y', 'photos by', or 'reel by'.
This drastically narrows the Set-of-Mark candidates down to ONLY valid grid items, making it literally impossible for the VLM to hallucinate and click navigation elements like 'Search' or 'Home' when asked to tap a post.

Also updated E2E test to enforce strict post-click assertion, preventing lying tests.
This commit is contained in:
2026-04-29 00:55:59 +02:00
parent 48071cc9b8
commit 44fae37cc7
2 changed files with 22 additions and 0 deletions

View File

@@ -269,6 +269,20 @@ class IntentResolver:
logger.debug(f"🛡️ [Strict Button Guard] Filtered out node with long text: '{node.text[:20]}...'")
candidates = filtered_candidates
# --- Post/Grid Item Guard ---
# VLMs frequently hallucinate 'Search' when asked to tap a post. We must pre-filter.
if "first post" in intent_lower or "grid item" in intent_lower:
grid_candidates = []
for node in candidates:
desc = (node.content_desc or "").lower()
# Posts/grid items usually have 'row X, column Y', 'photos by', or 'reel by'
if "row 1" in desc or "column" in desc or "photos by" in desc or "reel by" in desc:
grid_candidates.append(node)
if grid_candidates:
logger.info(f"🎯 [Grid Guard] Filtered to {len(grid_candidates)} actual grid candidates.")
candidates = grid_candidates
# --- Semantic Match Guard ---
# If the intent explicitly quotes a target (e.g., "tap 'New Message'"),
# we strictly filter candidates to those whose text or content_desc contains the quote.

View File

@@ -70,3 +70,11 @@ def test_explore_feed_first_post(make_real_device_with_image):
result = resolver.resolve("tap first post", candidates, device)
assert result is not None, "VLM returned None for 'tap first post'"
# Strictly verify that it picked an image button or post, NOT the search bar
rid = (result.resource_id or "").lower()
desc = (result.content_desc or "").lower()
# A valid grid post has an image_button resource ID or "photo" / "Reel" in description
is_valid_post = "image_button" in rid or "photo" in desc or "reel" in desc
assert is_valid_post, f"VLM picked the wrong element! Selected id='{rid}', desc='{desc}'"