fix(perception+brain): story view detection + autonomous prompt
Two root causes for 'scroll on story' bug:
1. ScreenIdentity had ZERO structural markers for story views.
reel_viewer_media_layout, reel_viewer_header, reel_viewer_progress_bar
and content-desc 'Like Story'/'Send story' now → STORY_VIEW.
2. Brain prompt was prescriptive ('you MUST scroll down'), overriding
the LLM's intelligence. Rewritten to give context about screen types
and let the AI reason autonomously about which action makes sense.
Philosophy: AI decides navigation, we provide correct perception data.
No hardcoded 'if story → press back' escape hatch.
4 new perception tests (all green), 0 regressions.
This commit is contained in:
@@ -29,11 +29,13 @@ def ask_brain_for_action(
|
||||
prompt += f"Context: {context}\n"
|
||||
|
||||
prompt += (
|
||||
"CRITICAL INSTRUCTIONS:\n"
|
||||
"1. If your goal requires an element (like 'following list') that you previously tried but failed to find, it is highly likely hidden off-screen.\n"
|
||||
"2. If you haven't scrolled yet, you MUST choose to 'scroll down' or 'scroll up' to reveal more of the screen.\n"
|
||||
"3. Only choose 'press back' or 'tap home tab' if you are completely trapped or in the wrong section entirely.\n"
|
||||
"4. DO NOT hallucinate actions. Reply ONLY with the exact string from the available actions list."
|
||||
"INSTRUCTIONS:\n"
|
||||
"1. Reason about where you are. Consider the screen type and what actions make sense on that screen.\n"
|
||||
"2. If the goal requires navigating away from the current screen, choose the action that moves you closest to the goal.\n"
|
||||
"3. 'scroll down' reveals more UI elements on scrollable screens (feeds, profiles, lists). Some screens like stories or modals are NOT scrollable.\n"
|
||||
"4. 'press back' exits the current screen and returns to the previous one. Use it when you are on a screen that doesn't lead to your goal.\n"
|
||||
"5. DO NOT hallucinate actions. Reply ONLY with the exact string from the available actions list.\n"
|
||||
"6. Reply with ONLY the action string, nothing else."
|
||||
)
|
||||
|
||||
try:
|
||||
|
||||
@@ -195,6 +195,16 @@ class ScreenIdentity:
|
||||
if "row_feed_button_like" in ids and "row_feed_photo_profile_name" in ids and not selected_tab:
|
||||
return ScreenType.POST_DETAIL
|
||||
|
||||
# Story view structural markers — present in full-screen story viewer.
|
||||
# Stories hide the navigation tab bar, so selected_tab is always None.
|
||||
# Must be checked BEFORE tab-based fallbacks to prevent UNKNOWN classification.
|
||||
STORY_MARKERS = ("reel_viewer_media_layout", "reel_viewer_header", "reel_viewer_progress_bar")
|
||||
if any(marker in ids for marker in STORY_MARKERS):
|
||||
return ScreenType.STORY_VIEW
|
||||
# Fallback: content-desc "Like Story" or "Send story" confirms story context
|
||||
if "like story" in desc_lower or "send story" in desc_lower:
|
||||
return ScreenType.STORY_VIEW
|
||||
|
||||
if selected_tab == "feed_tab":
|
||||
return ScreenType.HOME_FEED
|
||||
if selected_tab == "clips_tab":
|
||||
|
||||
@@ -363,3 +363,4 @@ class TestStoryViewDetection:
|
||||
assert len(tab_actions) == 0, (
|
||||
f"Story view should have NO tab navigation, but found: {tab_actions}"
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user