fix(perception): enforce strict candidate filtering for grid items
In addition to prompt tuning, we now apply a pre-flight structural guard for 'tap first post' / 'tap first grid item' intents. If the intent targets a grid item, we pre-filter the candidates to only include those matching 'row X, column Y', 'photos by', or 'reel by'. This drastically narrows the Set-of-Mark candidates down to ONLY valid grid items, making it literally impossible for the VLM to hallucinate and click navigation elements like 'Search' or 'Home' when asked to tap a post. Also updated E2E test to enforce strict post-click assertion, preventing lying tests.
This commit is contained in:
@@ -70,3 +70,11 @@ def test_explore_feed_first_post(make_real_device_with_image):
|
||||
|
||||
result = resolver.resolve("tap first post", candidates, device)
|
||||
assert result is not None, "VLM returned None for 'tap first post'"
|
||||
|
||||
# Strictly verify that it picked an image button or post, NOT the search bar
|
||||
rid = (result.resource_id or "").lower()
|
||||
desc = (result.content_desc or "").lower()
|
||||
|
||||
# A valid grid post has an image_button resource ID or "photo" / "Reel" in description
|
||||
is_valid_post = "image_button" in rid or "photo" in desc or "reel" in desc
|
||||
assert is_valid_post, f"VLM picked the wrong element! Selected id='{rid}', desc='{desc}'"
|
||||
|
||||
Reference in New Issue
Block a user