fix(perception): resolve grid hallucination by using 'tap first post' and prompt tuning
The bot previously hallucinated when given the intent 'tap first grid item' because the visual layout and description ('photos by...') didn't semantically map to the abstract 'grid item' concept in the VLM's eyes, causing it to click the 'Search' input instead.
Fixed by:
1. Updating to generate 'tap first post' instead of 'tap first grid item', matching natural language expectations.
2. Hardening the Visual Discovery Set-of-Mark prompt in to explicitly guide the VLM on how to visually identify posts/grid items (looking for 'photos by' or 'Reel by' instead of navigation elements).
This commit is contained in:
@@ -346,7 +346,8 @@ class IntentResolver:
|
||||
f"3. Do NOT select text, captions, or view counts if looking for an icon.\n"
|
||||
f"4. Ignore numbers inside the text itself. Do not confuse the text '19' with Box [19].\n"
|
||||
f"5. If the intent contains 'following', you MUST pick the box containing 'following'. Do NOT pick 'followers' or 'Follow'.\n"
|
||||
f"6. If the exact control is NOT visible, return null. Do NOT guess.\n\n"
|
||||
f"6. If the intent is to tap a 'post' or 'grid item', look for boxes with descriptions containing 'photos by', 'Reel by', or 'row 1, column 1' and pick the first matching one. Do NOT pick navigation buttons like 'Search'.\n"
|
||||
f"7. If the exact control is NOT visible, return null. Do NOT guess.\n\n"
|
||||
f'Reply ONLY with a valid JSON object: {{"box": <number>}} or {{"box": null}}'
|
||||
)
|
||||
|
||||
|
||||
@@ -317,7 +317,7 @@ class ScreenIdentity:
|
||||
|
||||
# Grid items
|
||||
if screen_type == ScreenType.EXPLORE_GRID:
|
||||
actions.append("tap first grid item")
|
||||
actions.append("tap first post")
|
||||
|
||||
# Scroll
|
||||
actions.append("scroll down")
|
||||
|
||||
Reference in New Issue
Block a user