fix(perception): enforce strict candidate filtering for grid items

In addition to prompt tuning, we now apply a pre-flight structural guard for 'tap first post' / 'tap first grid item' intents.
If the intent targets a grid item, we pre-filter the candidates to only include those matching 'row X, column Y', 'photos by', or 'reel by'.
This drastically narrows the Set-of-Mark candidates down to ONLY valid grid items, making it literally impossible for the VLM to hallucinate and click navigation elements like 'Search' or 'Home' when asked to tap a post.

Also updated E2E test to enforce strict post-click assertion, preventing lying tests.
This commit is contained in:
2026-04-29 00:55:59 +02:00
parent 48071cc9b8
commit 44fae37cc7
2 changed files with 22 additions and 0 deletions

View File

@@ -70,3 +70,11 @@ def test_explore_feed_first_post(make_real_device_with_image):
result = resolver.resolve("tap first post", candidates, device)
assert result is not None, "VLM returned None for 'tap first post'"
# Strictly verify that it picked an image button or post, NOT the search bar
rid = (result.resource_id or "").lower()
desc = (result.content_desc or "").lower()
# A valid grid post has an image_button resource ID or "photo" / "Reel" in description
is_valid_post = "image_button" in rid or "photo" in desc or "reel" in desc
assert is_valid_post, f"VLM picked the wrong element! Selected id='{rid}', desc='{desc}'"