fix(perception): resolve grid hallucination by using 'tap first post' and prompt tuning

The bot previously hallucinated when given the intent 'tap first grid item' because the visual layout and description ('photos by...') didn't semantically map to the abstract 'grid item' concept in the VLM's eyes, causing it to click the 'Search' input instead.

Fixed by:
1. Updating  to generate 'tap first post' instead of 'tap first grid item', matching natural language expectations.
2. Hardening the Visual Discovery Set-of-Mark prompt in  to explicitly guide the VLM on how to visually identify posts/grid items (looking for 'photos by' or 'Reel by' instead of navigation elements).
This commit is contained in:
2026-04-29 00:52:17 +02:00
parent 10a85a91f1
commit 48071cc9b8
2 changed files with 3 additions and 2 deletions

View File

@@ -346,7 +346,8 @@ class IntentResolver:
f"3. Do NOT select text, captions, or view counts if looking for an icon.\n"
f"4. Ignore numbers inside the text itself. Do not confuse the text '19' with Box [19].\n"
f"5. If the intent contains 'following', you MUST pick the box containing 'following'. Do NOT pick 'followers' or 'Follow'.\n"
f"6. If the exact control is NOT visible, return null. Do NOT guess.\n\n"
f"6. If the intent is to tap a 'post' or 'grid item', look for boxes with descriptions containing 'photos by', 'Reel by', or 'row 1, column 1' and pick the first matching one. Do NOT pick navigation buttons like 'Search'.\n"
f"7. If the exact control is NOT visible, return null. Do NOT guess.\n\n"
f'Reply ONLY with a valid JSON object: {{"box": <number>}} or {{"box": null}}'
)

View File

@@ -317,7 +317,7 @@ class ScreenIdentity:
# Grid items
if screen_type == ScreenType.EXPLORE_GRID:
actions.append("tap first grid item")
actions.append("tap first post")
# Scroll
actions.append("scroll down")