feat: add GetFile, GetFileList, ClearVirtualDisplays agent OS commands #277
feat: add GetFile, GetFileList, ClearVirtualDisplays agent OS commands #277mlikasam-askui wants to merge 5 commits into
Conversation
Wire up three new commands on AskUiControllerClient via _send_command: - get_file_list(path): returns the list of file paths at the given directory on the device under automation. - get_file(path): decodes the base64 payload returned by the controller and dispatches to Image.Image when the bytes parse as an image, or to str when they decode cleanly as UTF-8 text. Anything else raises a DesktopAgentOsError, matching the server contract that always base64-encodes the raw file bytes. - clear_virtual_displays(): removes all virtual displays, leaving only real ones active.
|
|
||
| class RemoveVirtualDisplaysCommand(BaseModel): | ||
| name: Literal['RemoveVirtualDisplays'] = 'RemoveVirtualDisplays' | ||
| parameters: List[Any] = [] |
There was a problem hiding this comment.
can we specify the type here instead of using Any?
| self._start_session() | ||
| self._start_execution() | ||
| self.set_display(self._display) | ||
| if self._settings.clean_virtual_displays: |
There was a problem hiding this comment.
shall we add an INFO log message here?
|
|
||
| @staticmethod | ||
| def _decode_file_payload(base64_data: str) -> Image.Image | str: | ||
| try: |
There was a problem hiding this comment.
on one hand this omits the need for a "magic" set of endings that we convert, on the other this might be inefficient. Not sure how much time/compute this consumes for huge text files that are clearly not images
| detail = f"image ({decoded.format}, {decoded.size[0]}x{decoded.size[1]})" | ||
| self._reporter.add_message( | ||
| "AgentOS", f"get_file({path}) -> {detail}", decoded | ||
| ) |
There was a problem hiding this comment.
why not add the PIL image here as well?
There was a problem hiding this comment.
the decoded response is the PIL image.
| "controller is using the real display" | ||
| ) | ||
| ) | ||
| self.set_display(1) |
There was a problem hiding this comment.
will this become a problem if I in the act command I set removing displays to true, but set the display id to 2?
| }, | ||
| agent_os=agent_os, | ||
| required_tags=[ToolTags.SCALED_AGENT_OS.value], | ||
| ) |
There was a problem hiding this comment.
please add the self.is_cacheable=True flag to init
| }, | ||
| agent_os=agent_os, | ||
| ) | ||
|
|
There was a problem hiding this comment.
please add the self.is_cacheable=True flag to init
|
What happens if I have the current controller on mac and the agent tries to call the read_file, I assume it will crash? should we need to update the docs and add a hint, that e.g. the read_file operation is only available on windows currently? |
It won’t crash. The agent will receive an error indicating that the currently used control does not support the requested command. |
2 participants
Jira: SOLENG-363, SOLENG-362