Thanks for writing this up. My brief comments:

1. I tend to use roles, which are kind-of a superset of places. I haven't tried to use groups.
2. Yes the separate configuration doesn't seem well motivated. In my case I have it on an NFS share, but it isn't ideal. I agree it should be unified, e.g. by allowing the 'environment' to go in the export file, with perhaps some overrides in the client. In my case I use a single environment file for the whole lab, which has required patching Labgrid.
3. That would be nice! What are you suggesting? So sort of relay on the exporter?
4. Agreed

Re your questions (I have various PRs pending):

Q1: How do you manage configurations across many users?
I use an NFS share which all machines can see

Q2: How do you handle SSH access and permissions at scale?
Just copying credentials around...not great

Q3: Do you centralize configurations, or keep them client-side?
The config is centralised - with the export and environment yaml (and a list of places) in the same NFS directory.

Q4: What are your main pain points and best practices?
Getting traction on PRs. For now I am just doing my own thing.

My lab is here:
https://lab.u-boot.org/

Hi Simon,

Thank you very much for your detailed feedback!

1. On roles vs. groups/places
   What I was trying to highlight is that we currently have groups on the exporter side and places on the client side, and in practice these are usually initialized to match each other. Because of that, having a single naming or classification mechanism (ideally defined on the exporter side) could simplify things significantly. It should be possible to configure everything on the exporter's side. I would be interested to hear more about how you use roles in your setup.

2. On configuration and access management
   Yes, I assume you had to patch certain aspects (e.g., around network service username requirements). Are you effectively configuring all machines in the lab for a single user, or how are you structuring access? Did you propagate your own user across all systems, or do you treat each machine as an independent "client"? As you also mentioned, this becomes quite challenging to manage, especially across multiple offices and labs.

3. On client–exporter interaction
   The specific problem I would like to solve is the direct communication path between the client and the exporter. In my view, the client should not need to reach the exporter directly, nor know exporter hostnames, SSH users, ports, or local permission details.

As you mentioned, one possible approach would be a relay/agent model where the client communicates only with the coordinator, and the coordinator forwards the requested operation to the relevant exporter-side agent. The exporter would then perform the local actions on the hardware.

Labgrid already follows a partially similar approach, as both exporters and clients connect to the coordinator. In practice, however, the coordinator is mainly responsible for resource discovery and reservation, while the actual execution still often involves direct client-to-exporter communication.

This model could be extended further by routing all actions through the coordinator, effectively turning exporters into backend agents. This would make labgrid easier to scale and easier to explain to non-expert users, as they would interact with "boards" or "targets" rather than exporter-specific infrastructure details.

Overall, I agree with your observation, many setups right now seem to rely on custom patches to achieve a stable workflow. Without them, it can be quite difficult to run a robust shared environment. That’s why I think it’s very valuable to discuss these use cases and potential improvements together.

Hi Simon,

Thank you very much for your detailed feedback!

Thanks for writing this up, Ozan!

1. On roles vs. groups/places
   What I was trying to highlight is that we currently have groups on the exporter side and places on the client side, and in practice these are usually initialized to match each other. Because of that, having a single naming or classification mechanism (ideally defined on the exporter side) could simplify things significantly. It should be possible to configure everything on the exporter's side. I would be interested to hear more about how you use roles in your setup.

You're right that for single-exporter labs there's redundancy, and I assume most labs are like that. The split makes more sense when a place aggregates resources from multiple exporters (e.g. serial from one host, power switch from another), or when resources are moved between places without restarting the exporter. But for a typical lab where one exporter owns all the resources for a board, defining the same thing twice does feel awkward.

One option might be to have optional, auto-creation of places from exporter groups, with explicit place definitions still supported for the multi-exporter case. That would simplify the common path without losing flexibility.

In my lab I have ~40 boards (although a few are just QEMU), and roles let me use a command like ub-int bbb that doesn't need to know the underlying place name. Roles let me have multiple roles share a place (e.g. different U-Boot builds for the same board, such as ff3399 and vbe).

PR: #1411

lab config:

e3b7760

2. On configuration and access management
   Yes, I assume you had to patch certain aspects (e.g., around network service username requirements). Are you effectively configuring all machines in the lab for a single user, or how are you structuring access? Did you propagate your own user across all systems, or do you treat each machine as an independent "client"? As you also mentioned, this becomes quite challenging to manage, especially across multiple offices and labs.

Currently a single shared user ('gitlab-runner') on the 'runner' machine which can ssh to the exporter machine. It isn't great, but OK for my simple setup so far.

3. On client–exporter interaction
   The specific problem I would like to solve is the direct communication path between the client and the exporter. In my view, the client should not need to reach the exporter directly, nor know exporter hostnames, SSH users, ports, or local permission details.

As you mentioned, one possible approach would be a relay/agent model where the client communicates only with the coordinator, and the coordinator forwards the requested operation to the relevant exporter-side agent. The exporter would then perform the local actions on the hardware.

Labgrid already follows a partially similar approach, as both exporters and clients connect to the coordinator. In practice, however, the coordinator is mainly responsible for resource discovery and reservation, while the actual execution still often involves direct client-to-exporter communication.

This model could be extended further by routing all actions through the coordinator, effectively turning exporters into backend agents. This would make labgrid easier to scale and easier to explain to non-expert users, as they would interact with "boards" or "targets" rather than exporter-specific infrastructure details.

Overall, I agree with your observation, many setups right now seem to rely on custom patches to achieve a stable workflow. Without them, it can be quite difficult to run a robust shared environment. That’s why I think it’s very valuable to discuss these use cases and potential improvements together.

Yes I strongly agree :-) The current model where the client SSHes directly to the exporter is not ideal and one of the main reasons I haven't made my lab available for interactive access. A coordinator-mediated execution model would be a significant improvement from this POV.

The hard part is probably the streaming use cases (console, video) where direct connection is currently used for latency. Those would need a forwarding mechanism through the coordinator. But for the request/response style operations (power, IO, file transfer) it should be easy enough to route through the coordinator.

This would also make it possible to expose labgrid through a higher-level service without users needing direct SSH credentials at all.

One other thing to mention is that I found it painful to have to fully specify each USB device. When something is broken, it it hard to figure out which USB hub/port to replug, etc. So I created this:

#1854

It may be that my USB-heavy lab is unusual, not sure.

• Simon

This model could be extended further by routing all actions through the coordinator,

Isn't this what the labgrid-exporter --isolated option does?

This model could be extended further by routing all actions through the coordinator,

Isn't this what the labgrid-exporter --isolated option does?

I think you are right that labgrid-exporter --isolated is an important part of the solution, and it may already cover many practical cases. My understanding is that --isolated mainly improves the network access model. That is very useful for isolation.

The part I am still trying to separate is whether this also solves the execution model. As far as I understand it, the coordinator still mainly handles discovery, state, release, reservations, etc. Actual resource access or command execution still seems to happen through an SSH or proxy path from the client side.

For example, operations such as bootstrap still appear to run through the client-side driver. With --isolated, the SSH path may go through the proxy or exporter instead of a directly reachable resource host, but it is still not quite the same as the coordinator dispatching the action to an exporter.

This also leaves some user and permission questions. A shared exporter user such as labgrid can make setup simpler, but it can make auditing and per-user isolation weaker. For example, multiple users writing as the same Unix user could also create overwrite, or cleanup problems.

There is also a similar issue with the --isolated when user identity is passed through end to end. In that setup, the client connects to the proxy as their own user and then reaches the exporter as the same user. This is good for traceability, but it still means every exporter needs the relevant users and keys. As the number of users and exporters grows, the key distribution and permission management problem remains. So both approaches have trade-offs.

So I currently see two related but separate layers: network isolation, which --isolated addresses, and coordinator-mediated action execution, which would be a different model. In that second model, clients would send action requests to the coordinator, and the coordinator would dispatch them to the relevant exporter. That could reduce the need for client SSH access to exporters and make the user-facing workflow simpler.

Of course, this would be a larger change, and some use cases may still need a more direct path for latency and throughput. But request-response operations such as power control, file transfer, bootstrap, or some hardware actions might be interesting candidates.

I may also be missing part of the intended model here, so I would be interested in your view. Do you think --isolated is already enough for this kind of multi-user setup, or do you see value in adding some form of coordinator-mediated execution for (selected or every) actions?

I may also be missing part of the intended model here, so I would be interested in your view. Do you think --isolated is already enough for this kind of multi-user setup, or do you see value in adding some form of coordinator-mediated execution for (selected or every) actions?

I don't know. I'm a few steps behind you and Simon. I've just been using local setups so far, so I haven't had to deal with that issue yet.

At BayLibre, we've been looking at using headscale/tailscale to make a virtual network for all exporters/coordinators/clients. This adds another layer of admin and potential problems though.

In general, I've run most of the same issues/pain points listed in the OP.

Points 1 and 2 I don't see as such a big deal since just one knowledgeable person has to set those up in the first place, then anyone can use them without having to set them up again.

For points 3 and 5, the direction I have been going is putting as much as possible in the .yaml config file. For example, you can put the place name and the coordinator hostname in there, so one only has to export LG_ENV=my-config.yaml and then they can use labgrid-client without extra args or other environment variables.

This also includes a Python file for strategy definitions. So for distributing, I've been looking at making a Python project and distributing it with git. This also includes a pyproject.toml file for setting up a venv with labgrid. So steps for letting someone else use it would be:

1. git clone ...
2. cd my-project
3. uv sync (or whatever your Python tool of choice is)
4. . .venv/bin/activate
5. export LG_ENV=my-config.yaml

Point 4 I haven't got to yet, but I am not looking forward to having to set up a user for everyone I want to share my exporters with. My thought here is to just create a labgrid user instead and have everyone use that. And it still requires getting a public key from every user and installing it on the exporter.

Another pain point for me has been setting up the exporters. I would ideally like to have one exporter per system under test (reasoning being to minimize disruptions to other users when tweaking a config), which means lots of exporters. I like the idea of using docker containers for this. One thing I haven't solved there is how best to deal with hot-plugable devices (maybe something like this) and you still have to deal with udev rules on the host system, so it isn't quite a simple as spinning up a container.

Quick update: I've made the lab available for interactive remote access since my earlier comment, and a few things have shaken out that are relevant to this discussion.

Isn't this what the labgrid-exporter --isolated option does?

Agreeing with @ozan956 - --isolated helps with network isolation (proxying through the exporter) but doesn't change the execution model. The client still drives drivers locally and SSHes to the exporter to run individual commands. Per-user identity propagation, audit trails, and exposing labgrid as a service all still need true coordinator-mediated execution.

What I've set up since the previous comment

I now have a working remote-access setup that addresses some (not all) of the pain points without needing protocol changes:

• Network: self-hosted headscale (open-source Tailscale control server) on a public host. Lab + clients all join the tailnet, which gives a stable IP that works through NAT. No port forwarding, no per-client config.
• Auth: a single shared labgrid UNIX account on the lab host, with each user authenticating via their own SSH key. Each authorized_keys entry forces a small wrapper script (via command="...") that refuses interactive shells and logs every command to an audit file, attributed to the user. Revocation = remove one line.
• Serial audit: small labgrid changes I want to discuss separately upstream:
  • New optional user field in ExporterSetAcquiredRequest so the exporter knows who acquired a place
  • LG_SERIAL_TRACE_DIR env var that, when set, makes the exporter pass trace-both to the per-acquire ser2net. Result: per-board, per-user serial trace files capturing every byte both ways

None of this changes the fundamental architecture (it's still SSH/TCP under the hood) but it gives a usable remote workflow today and per-user accountability. It also demonstrates the value of "clients only need one well-known endpoint" - which is essentially what coordinator-mediated execution would give us natively, without needing a VPN.

Lab access for anyone who wants to try

Minimal client setup is now small - no U-Boot tree required for read-only access:

1. Install tailscale and email me your desired username + SSH public key
2. I send back a tailscale preauth key
3. sudo tailscale up --login-server=https://headscale.u-boot.org --auth-key=<key>
4. Install labgrid from U-Boot integration [WIP] #1411 (the U-Boot integration branch - the lab's drivers come from this PR, and it ships an example_env.cfg already configured for the kea coordinator with all ~50 boards as RemotePlace entries):

   git clone https://github.com/labgrid-project/labgrid.git
   cd labgrid && gh pr checkout 1411
   pip install --user .
   

5. Three environment variables:

   export PATH=$HOME/.local/bin:$HOME/labgrid/contrib/u-boot:$PATH
   export LG_COORDINATOR=100.64.0.1:20408
   export LG_ENV=$HOME/labgrid/contrib/u-boot/example_env.cfg
   

6. labgrid-client console -p bbb (or any of ~50 boards), or ub-int -T bbb for the full strategy without rebuilding U-Boot.

Total client footprint is a few MB; no U-Boot tree, no buildman, no per-machine setup. Access is currently manual rather than self-service; could be opened up with OIDC if there's interest. Documented in contrib/u-boot/index.rst of #1411.

--

Now I'm wondering what it would take to keep the environment in the exporter...

Hi David,

Points 1 and 2 I don't see as such a big deal since just one knowledgeable person has to set those up in the first place, then anyone can use them without having to set them up again.

I believe the challenge begins when you need to manage or modify the setup over time. For example, whenever a new board is introduced, all clients need to update their environments accordingly if they want to use it.

Centralizing this responsibility by having more on exporter side would be valuable. It allows a smaller group of "expert" users to handle configurations, while others can focus on using the system. This separation can scale well within a team.

For points 3 and 5, the direction I have been going is putting as much as possible in the .yaml config file...

This approach certainly improves usability, but it still has some limitations. For instance, if your environment file defines multiple targets and you attempt to acquire a single one using:

labgrid-client -c env.yaml -p PLACE1 acquire

Labgrid initially uses -p PLACE1 to select the matching role from the environment. However, during the acquire phase, it switches to operating on all RemotePlace entries defined in the environment, rather than only PLACE1. As a result, it becomes impossible to acquire a single place in isolation without bypassing the environment file and interacting directly with the coordinator.

This behavior suggests that the environment file is fundamentally designed to be client-oriented and tailored to specific use cases, rather than serving as a shared, scalable abstraction. I further think that environment files are designed to exist as separate instances for each use case for each user (multiple envs in every local).

Point 4 I haven't got to yet...

I had similar thoughts initially. Using a shared user account (e.g., a labgrid user) for all clients. However, this introduces another issue. When using commands like:

labgrid-client -c env.yaml -p PLACE1 scp images/u-boot-spl :u-boot-spl

the file is copied to a default location such as ~/u-boot-spl on the remote system. If multiple users share the same account and perform similar operations, they may unintentionally overwrite each other’s files.

To mitigate this, users would need to follow strict conventions, such as using separate directories (e.g., /home/labgrid/user1/). However, this shifts the burden onto users and introduces potential for human error, as there is no built-in isolation mechanism.

Another pain point for me has been setting up the exporters...

That's an interesting idea. Thanks for bringing this up. I will give it more thought.

Hi Simon,

Now I'm wondering what it would take to keep the environment in the exporter...

So we would be moving from:

• exporter: knows local hardware and exports resources
• coordinator: knows places and matching
• client: builds targets, drivers, and strategies

to:

• exporter: knows hardware, target configuration, drivers, and strategies
• coordinator: still manages place ownership and matching
• client: becomes much thinner and mostly consumes what the exporter/coordinator provides

We would need a significant design change.

On the client side, the environment defines the "targets." These targets are bound to "places," which are created by clients and managed by the coordinator.

1 - We would need to move "targets" to the exporter. The client would then need to fetch the target, so the _get_target() logic would require a major redesign.

2 - The tricky part would be the strategies. There will likely be multiple design approaches to discuss.

3 - This change would also remove the ability to use multiple devices from multiple exporters within a single place. However, I am not sure whether there is a real use case where someone needs multiple resources from different exporters in one place, so this may not be a significant issue.

There are likely many other points to consider. In the end, it is doable, but it will be challenging.

Hi Simon,

Now I'm wondering what it would take to keep the environment in the exporter...

So we would be moving from:

• exporter: knows local hardware and exports resources
• coordinator: knows places and matching
• client: builds targets, drivers, and strategies

to:

• exporter: knows hardware, target configuration, drivers, and strategies
• coordinator: still manages place ownership and matching
• client: becomes much thinner and mostly consumes what the exporter/coordinator provides

We would need a significant design change.

On the client side, the environment defines the "targets." These targets are bound to "places," which are created by clients and managed by the coordinator.

1 - We would need to move "targets" to the exporter. The client would then need to fetch the target, so the _get_target() logic would require a major redesign.

2 - The tricky part would be the strategies. There will likely be multiple design approaches to discuss.

3 - This change would also remove the ability to use multiple devices from multiple exporters within a single place. However, I am not sure whether there is a real use case where someone needs multiple resources from different exporters in one place, so this may not be a significant issue.

There are likely many other points to consider. In the end, it is doable, but it will be challenging.

Yes this is not easy. I have been prototyping a way for exporters to send their environment file to the coordinator, which then merges them into a single file, which it then provides to clients. Once I've tested it a bit more I will see if I can create a clean PR.

Disclaimer: I converted this into a discussion, unfortunately the conversion does not take the answer order into account...

Answering the questions:

1. Resource Discovery Confusion

The seperation between resources and places was an intentional choice since Pengutronix operates a lab with removable hardware but fixed resources for places. This means we have to separate the physical connections (resources) from the actual requirement for the hardware (place) and need to be able to assemble them as needed.

If you have a 1:1 dut to exporter configuration, you can take advantage of the matches as well by allocating all resources of a group to a place via a wildcard match, i.e.

labgrid-client -p testplace add-match adi-test/group-1/*

But mostly your users should not care about resources, they should care about places.

2. Redundancy Between Groups and Places

We envisioned the resources to be aggregated by physical physical, i.e. if you have a rackspace with serial connection, ethernet and power, these would all be under the same group. Aggregating all possible USB resources for a USB Hub is also possible, even if it just helps to make the hierarchy clearer.

As explained above, with wildcard matches the distinction between groups and places is required.

3. Environment on Client Side

An environment should only be required if you want to run a set of scripts or a test suite on the device, in that case it should be kept with the respective code.

Most interactive usage via the labgrid client should not require the setup of an environment configuration file.
The environment configuration files also allow a purely local setup by defining all local resources within them, the idea was to allow a seamless move from a remote setup on the local developer desk into the remote infrastructure by employing the RemotePlace resource where all resource information is fetched from the coordinator.

4. SSH and Access Management Complexity

We need an authenticated connection to the exporter to execute commands and SSH was a natural choice for that.
Our expectation is that multi-user and multi-exporter deployments will be managed by a configuration management solution as well, which can handle the roll-out of the SSH keys for users too.

Rolling out the SSH configuration for users depends on your exporter DNS setup, but can ideally be a three line configuration i.e.:

Host *.exporters.mycompanyinternaldomain.com
 User <myusername>
 IdentityFile ~/.ssh/my-company-identity

NetworkService should point to the DUT and not the exporter.

5. Configuration Management Problem

While clients require a working SSH configuration, everything else should not be required for interactive usage of labgrid via labgrid-client.