I need the page to automatically click the arrows on this shuttle box. Currently, I can select it using the WebUI, but the LLM fails to recognize it even after trying many prompts like 'arrow' or 'greater than symbol'.
I'd like to understand how the framework passes the UI selection to the LLM, and how I can find the most effective natural language description for this UI element to guide the LLM's action. Could you please guide me on this?
Image
Image