Search code, repositories, users, issues, pull requests...

@mlikasam-askui mlikasam-askui commented May 12, 2026 •

edited

Loading

Copy link

Copy Markdown

Contributor

Summary

MultiComputerTargetAgentOS can now manage multiple Agent OS target
computers (local and/or remote) at once. One is active at a time and receives
the agent's actions; callers can switch at runtime or scope a switch to a with
block.

New API:

ComputerTarget (base), LocalComputerTarget (replaces the old
AskUiControllerServer — owns the local controller subprocess; auto-detects
the Windows AskuiCoreService and switches to its port),
RemoteComputerTarget (already-running remote controller, no process
management). Each target owns its own gRPC connection
(ComputerTargetConnection).
ComputerTargetPool — enforces invariants (at most one local, unique session
GUIDs / computer_ids / remote addresses), tracks the active target, and
orchestrates connect/disconnect. Targets are addressed exclusively by
computer_id (single dict lookup; no secondary indices).
On MultiComputerTargetAgentOS and exposed through ComputerAgentOS /
ComputerAgentOsFacade: add_agent_os_target_computer,
describe_agent_os_target_computers (returns the targets' repr() strings,
not live objects), get_current_computer_target_id (returns the computer_id
string of the active target), switch_agent_os_target_computer,
reset_agent_os_target_computers, temporary_select.
PpadbAgentOs gets a sibling temporary_select(device_sn).
ComputerAgent auto-registers three new act() tools so the LLM can drive
multi-target flows: ComputerListAgentOsTargetComputersTool,
ComputerSwitchAgentOsTargetComputerTool,
ComputerGetCurrentComputerTargetIdTool.
LocalComputerTarget and RemoteComputerTarget are re-exported from the
top-level askui package.

New unit tests cover the target-computer classes, the pool, the multi-target
client, and the new computer tools. The e2e test was updated to the new
constructor.

Example

from askui import ComputerAgent
from askui import LocalComputerTarget, RemoteComputerTarget
with ComputerAgent(
 agent_os_target_computers=[
 LocalComputerTarget(computer_id="research-box"),
 RemoteComputerTarget(
 address="192.168.1.42:26000",
 description="Writer box with a text editor open",
 computer_id="writer-box",
 ),
 ],
) as agent:
 agent.act(
 "On research-box, google 'askui' and read the top results. Then switch "
 "to writer-box and write a Markdown summary in the open text editor."
 )

mlikasam-askui and others added 13 commits

May 8, 2026 12:17


 feat: support multiple target computers in AskUiControllerClient

b654464

Replace single-controller config with a TargetComputer / TargetComputerManager
abstraction. AskUiControllerClient now manages a list of local and remote
controller servers, opens a gRPC connection per target on connect(), and routes
agent-os actions to a single active target. Add tools for the agent to list,
switch, and inspect the active target computer.


 add reporter messgaes

768653c


 fix logger

01e8f4f


 refactor: drop tags from target computers and prefix tool output with...

cfb84f6

... target session GUID


 fixed reporter

de55cf2


 add temporary_select target

c7f9631


 add device_id

aa0aba5


 improve doc string

78523b4


 improve error message

4ca7f80


 add unit tests


 improve tool results

e40c02c


 fix lint problems

3493c4a

feat: multi Agent OS server support in AskUiControllerClient #275


 improve-doc-string

9a475c5

@mlikasam-askui mlikasam-askui mentioned this pull request

May 12, 2026

Closed

@mlikasam-askui mlikasam-askui requested review from philipph-askui and programminx-askui

May 12, 2026 09:01

@mlikasam-askui mlikasam-askui self-assigned this

May 12, 2026

mlikasam-askui and others added 2 commits

May 12, 2026 11:47


 feat: capture unhandled errors in reporter

7bd98be


 Revert "feat: capture unhandled errors in reporter"

24de514

This reverts commit 7bd98be.

philipph-askui

philipph-askui reviewed

May 14, 2026

src/askui/tools/computer/__init__.py Outdated

Comment on lines +18 to +19

from .set_active_display_tool import ComputerSetActiveDisplayTool

from .switch_agent_os_server_tool import ComputerSwitchAgentOsServerTool

@philipph-askui philipph-askui May 14, 2026

Copy link

Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't these 3 new tools also be in experimental?

@mlikasam-askui mlikasam-askui May 15, 2026

Copy link

Copy Markdown

Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see them as part of the default tool list, similar to the Android agent.

src/askui/tools/computer/get_active_agent_os_server_tool.py Outdated

src/askui/tools/computer/get_mouse_position_tool.py Outdated

cursor_position = self.agent_os.get_mouse_position()

return f"Mouse is at position ({cursor_position.x}, {cursor_position.y})."

return (

f"[Server with id '{server.computer_id}']: Mouse is at position "

@philipph-askui philipph-askui May 14, 2026

Copy link

Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not happy with the name "Server in general". So far, I was using the term "machine" when telling the agent that it is operating multiple devices.
The name of one machine could then be server. If we use the term server here, this will introduce ambiguity that might cause the currently working stuff to break

src/askui/tools/computer/get_system_info_tool.py Outdated

return str(self.agent_os.get_system_info().model_dump_json())

server = self.agent_os.get_active_agent_os_server(report=False)

system_info_json = self.agent_os.get_system_info().model_dump_json()

return f"[Server with id '{server.computer_id}']: {system_info_json}"

@philipph-askui philipph-askui May 14, 2026

Copy link

Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here (Server ->machine)

src/askui/tools/computer/switch_agent_os_target_computer_tool.py

agent_os=agent_os,

)

def __call__(self, computer_id: str) -> str:

@philipph-askui philipph-askui May 14, 2026

Copy link

Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is this computer_id determined? By the user, e.g. in the init of the agent? Currently, they can set local_machine_name and remote_machine_name

@mlikasam-askui mlikasam-askui May 15, 2026

Copy link

Copy Markdown

Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can be set by the user, or it defaults to the session ID, which is a UUID4.

@philipph-askui

philipph-askui commented May 14, 2026

Copy link

Copy Markdown

Contributor

tbh: at the moment I really like the concept with 2 dedicated toolsets to operate 2 machines. Still, I see that this here will scale also to even more machines.

Have you checked if the agent understands that it can operate multiple machines and that it calls the switch tool before executing operations if needed?

Further: we need to automatically adjust the system capabilities and device information prompts if multiple agentOS are added.

Also: we should definitely update the docs with this PR to explain these changes here

philipph-askui

philipph-askui requested changes

May 14, 2026

@mlikasam-askui mlikasam-askui requested a review from philipph-askui

May 15, 2026 06:55

programminx-askui requested changes

May 15, 2026

src/askui/models/shared/tools.py Outdated

self, required_tags: list[str]

) -> AgentOs | AndroidAgentOs:

"""

Find the first registered agent OS whose tags are a superset of

@programminx-askui programminx-askui May 15, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Find the first registered agentOS whose tags are a superset of

Find the first registered AgentOS whose tags are a superset of

Please find all occurance. in Strings

src/askui/tools/android/agent_os.py

src/askui/tools/android/agent_os_facade.py

src/askui/tools/android/ppadb_agent_os.py

src/askui/tools/askui/agent_os_server.py Outdated

logger = logging.getLogger(__name__)

def _generate_session_guid() -> str:

@programminx-askui programminx-askui May 15, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have already such a function?

src/askui/tools/askui/agent_os_server_manager.py Outdated

return server

return None

def switch(self, computer_id: str) -> AgentOsServer:

@programminx-askui programminx-askui May 15, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can switch to the same computer? Should this not failing?

src/askui/tools/askui/agent_os_server_manager.py Outdated

Raises:

KeyError: If no server with the given computer id is registered.

"""

server = self.get(computer_id)

@programminx-askui programminx-askui May 15, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please wrap a proper AgentOSServerManager Error around.

src/askui/tools/askui/askui_controller.py Outdated

Comment on lines +143 to +145

def agent_os_server_manager(self) -> AgentOsServerManager:

"""The underlying Agent-OS-server manager."""

return self._manager

@programminx-askui programminx-askui May 15, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have the server_manager inside the client?

@programminx-askui programminx-askui May 15, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

THis should not be here

src/askui/tools/computer/get_mouse_position_tool.py Outdated

self.is_cacheable = True

def __call__(self) -> str:

server = self.agent_os.get_active_agent_os_server(report=False)

@programminx-askui programminx-askui May 15, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intresting.

tests/unit/tools/askui/test_agent_os_target_computer.py

assert _generate_session_guid() != _generate_session_guid()

class TestReplacePort:

@programminx-askui programminx-askui May 15, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@mlikasam-askui mlikasam-askui requested a review from programminx-askui

May 19, 2026 09:54


 implement review remarks

55cdac0

@mlikasam-askui mlikasam-askui force-pushed the SOLENG-360-refactor/askui-controller-multi-target branch from c20d744 to 55cdac0 Compare

May 19, 2026 10:08


 Merge remote-tracking branch 'origin/main' into SOLENG-360-refactor/a...

40873be

...skui-controller-multi-target

programminx-askui requested changes

Jun 3, 2026

@programminx-askui programminx-askui left a comment

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's have a talk.

src/askui/tools/askui/agent_os_target_computer.py Outdated

OS-level operations (screenshot, mouse, keyboard, ...), and is identified by

a unique session GUID. Each server also tracks which display it is currently

operating against.

A target computer runs the server-side counterpart of the `AgentOs` client

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AgentOS

src/askui/tools/askui/agent_os_target_computer.py Outdated

Comment on lines +167 to +169

"sc",

"query",

LocalAgentOsTargetComputer._ASKUI_CORE_SERVICE_NAME,

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which user rights do you need to perfrom the query? Did you tested it without admin rights?

src/askui/tools/askui/agent_os_target_computer_manager.py Outdated

Comment on lines +98 to +99

if self.is_connected:

self.connect(target)

@programminx-askui programminx-askui Jun 3, 2026 •

edited

Loading

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

~~(削除) Why? When you have already on connected target in the list and then you add another unconnected new target. Then it does not connect the new target. (削除ここまで)~~

Why do we automaticaly connect to the target? This is only an add-function. Then I would prefere a param auto_connect = True. And replace the buggy implementation, when one target is connected, that the next one get's not connected.

src/askui/tools/askui/agent_os_target_computer_manager.py Outdated

Comment on lines +102 to +121

def add_remote(

self,

address: str,

description: str,

) -> RemoteAgentOsTargetComputer:

"""

Convenience method to construct and register a remote Agent OS target

computer. Auto-connects when the manager already has at least one open

connection.

Args:

address (str): gRPC address of the remote Agent OS target computer.

description (str): Human-readable description.

Returns:

RemoteAgentOsTargetComputer: The newly registered target.

"""

target = RemoteAgentOsTargetComputer(address=address, description=description)

self.add(target)

return target

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we delete this function? It does not make sense.

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still see no benefit of this funciton. -> please remove

src/askui/tools/askui/agent_os_target_computer_manager.py Outdated

Comment on lines +176 to +177

if self.is_connected and computer_id not in self._connections:

self.connect(target)

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you trying to fix the logic from above.

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What should be the Connection Livecycle of an Target? Is the Manager Responsible to connect to the targets?

src/askui/tools/askui/agent_os_target_computer_manager.py Outdated

for target in self._by_computer_id.values():

self.connect(target)

except Exception:

self.disconnect_all()

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whats happend when an exception is happing on the disconnect function?

src/askui/tools/askui/agent_os_target_computer_manager.py Outdated

Comment on lines +301 to +310

if hasattr(e, "add_note"):

e.add_note(

f"While connecting to Agent OS target computer "

f"{target.description!r} "

f"(computer_id={target.computer_id!r}, "

f"session_guid={target.session_guid}, "

f"display={target.display}, "

f"address={target.address})"

)

raise

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why aer you not creating a new Exception? You should not manipulate a old one.

src/askui/tools/askui/agent_os_target_computer_manager.py Outdated

Comment on lines +268 to +278

target.start()

started_process = True

channel = grpc.insecure_channel(

target.address,

options=[

("grpc.max_send_message_length", 2**30),

("grpc.max_receive_message_length", 2**30),

("grpc.default_deadline", 300000),

)

stub = controller_v1.ControllerAPIStub(channel)

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have inside the Manager GRPC Code? This should be moved to the Client. We need to get rid of this.

src/askui/tools/askui/askui_controller.py Outdated

started_process: bool

class AskUiControllerClient(AgentOs):

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does a Client has a Manager with multiple Targets? This should be another way around: Manager has Multiple Target and the Target should have the Client.

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename to: MultiComputerTargetAgentOS

programminx-askui requested changes

Jun 3, 2026

@programminx-askui programminx-askui left a comment

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @mlikasam-askui ,

I added some changes. But we have to do after this more work.

src/askui/tools/askui/askui_controller.py Outdated

started_process: bool

class AskUiControllerClient(AgentOs):

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename to: MultiComputerTargetAgentOS

src/askui/tools/askui/agent_os_target_computer_manager.py Outdated

started_process: bool

class AgentOsTargetComputerManager:

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename to `ComputerTargetPool``

src/askui/tools/askui/agent_os_target_computer.py Outdated

logger = logging.getLogger(__name__)

class AgentOsTargetComputer:

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

class AgentOsTargetComputer:

class ComputerTarget

src/askui/tools/askui/agent_os_target_computer.py Outdated

)

class LocalAgentOsTargetComputer(AgentOsTargetComputer):

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

class LocalAgentOsTargetComputer(AgentOsTargetComputer):

class LocalComputerTarget(ComputerTarget):

src/askui/tools/askui/agent_os_target_computer.py Outdated

self._process = None

class RemoteAgentOsTargetComputer(AgentOsTargetComputer):

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

class RemoteAgentOsTargetComputer(AgentOsTargetComputer):

class RemoteComputerTarget(ComputerTarget):

src/askui/tools/askui/agent_os_target_computer_manager.py Outdated

started_process=started_process,

)

def disconnect_all(self) -> None:

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

src/askui/tools/askui/askui_controller.py Outdated

@telemetry.record_call()

@override

def add_remote_agent_os_target_computer(

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we not have a add function with accepts

Suggested change

def add_remote_agent_os_target_computer(

def add(computer_target: RemoteComputerTargetOptions|LocalComputerTargetOptions

src/askui/tools/askui/askui_controller.py

Comment on lines +207 to +210

def reset_agent_os_target_computers(

self,

agent_os_target_computers: list[AgentOsTargetComputer] | None = None,

) -> None:

@programminx-askui programminx-askui Jun 3, 2026

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function should not accept a list. It should use the current pool and reset it

Suggested change

def reset_agent_os_target_computers(

self,

agent_os_target_computers: list[AgentOsTargetComputer] | None = None,

) -> None:

def reset_computer_targets(

self,

) -> None:

src/askui/tools/agent_os.py

src/askui/tools/computer/switch_agent_os_target_computer_tool.py Outdated

mlikasam-askui added 4 commits

June 11, 2026 15:02


 refactor: rename AgentOs and computer-target classes for clarity

a8e6bef

Rename the computer-specific Agent OS class and the multi-target
classes to better reflect their scope. `AgentOs` is the computer Agent
OS interface (mouse, keyboard, displays), not a universal abstraction —
so it becomes `ComputerAgentOS` while preserving the "Agent OS" term.
Renames:
- AgentOs -> ComputerAgentOS
- AskUiControllerClient -> MultiComputerTargetAgentOS
- AgentOsTargetComputerManager -> ComputerTargetPool
- AgentOsTargetComputer -> ComputerTarget
- LocalAgentOsTargetComputer -> LocalComputerTarget
- RemoteAgentOsTargetComputer -> RemoteComputerTarget
Add `AgentOs = ComputerAgentOS` alias (still exported from
askui.tools) to avoid breaking existing imports. Update docstrings to
the new vocabulary and fix the reporter-source doc reference.


 refactor: extract ComputerTargetConnection and let targets own their ...

...connection


 rm add_remote_agent_os_target_computer

c4fc57b


 replace list with describe

b9389b1

@mlikasam-askui mlikasam-askui requested a review from programminx-askui

June 12, 2026 10:45

programminx-askui approved these changes

Jun 17, 2026