The following examples describe how to enable I/O virtualization for a PCIe bus, manage direct I/O devices on non-primary root domains, and manage SR-IOV virtual functions on non-primary root domains.
The following example shows how to enable I/O virtualization by using the ldm add-io and ldm set-io commands.
The following SPARC T4-2 I/O configuration shows that bus pci_1 already has been removed from the primary domain.
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ pci_0 BUS pci_0 primary IOV pci_1 BUS pci_1 niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary OCC /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 UNK /SYS/MB/PCIE3 PCIE pci_1 UNK /SYS/MB/PCIE5 PCIE pci_1 UNK /SYS/MB/PCIE7 PCIE pci_1 UNK /SYS/MB/PCIE9 PCIE pci_1 UNK /SYS/MB/NET2 PCIE pci_1 UNK /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary
The following listing shows that the guest domains are in the bound state:
primary# ldm list NAME STATE FLAGS CONS VCPU MEMORY UTIL NORM UPTIME primary active -n-cv- UART 8 8G 0.6% 0.6% 8m rootdom1 bound ------ 5000 8 4G ldg2 bound ------ 5001 8 4G ldg3 bound ------ 5002 8 4G
The following ldm add-io command adds the pci_1 bus to the rootdom1 domain with I/O virtualization enabled for that bus. The ldm start command starts the rootdom1 domain.
primary# ldm add-io iov=on pci_1 rootdom1 primary# ldm start rootdom1 LDom rootdom1 started
If a specified PCIe bus is assigned already to a root domain, use the ldm set-io command to enable I/O virtualization.
primary# ldm start-reconf rootdom1 primary# ldm set-io iov=on pci_1 primary# ldm stop-domain -r rootdom1
The root domain must be running its OS before you can configure the I/O devices. Connect to the console of the rootdom1 guest domain and then boot the OS of the rootdom1 root domain if your guest domains are not already set to autoboot.
primary# telnet localhost 5000 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Connecting to console "rootdom1" in group "rootdom1" .... Press ~? for control options .. ok> boot ... primary#
The following command shows that the pci_1 PCIe bus and its children are now owned by the rootdom1 root domain.
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ pci_0 BUS pci_0 primary IOV pci_1 BUS pci_1 rootdom1 IOV niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary OCC /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 rootdom1 OCC /SYS/MB/PCIE3 PCIE pci_1 rootdom1 OCC /SYS/MB/PCIE5 PCIE pci_1 rootdom1 OCC /SYS/MB/PCIE7 PCIE pci_1 rootdom1 OCC /SYS/MB/PCIE9 PCIE pci_1 rootdom1 EMP /SYS/MB/NET2 PCIE pci_1 rootdom1 OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/PCIE5/IOVNET.PF0 PF pci_1 rootdom1 /SYS/MB/PCIE5/IOVNET.PF1 PF pci_1 rootdom1 /SYS/MB/NET2/IOVNET.PF0 PF pci_1 rootdom1 /SYS/MB/NET2/IOVNET.PF1 PF pci_1 rootdom1
The following example shows how to manage direct I/O devices on non-primary root domains.
The following command produces an error because it attempts to remove a slot from the root domain while it is still active:
primary# ldm remove-io /SYS/MB/PCIE7 ldg1 Dynamic I/O operations on PCIe slots are not supported. Use start-reconf command to trigger delayed reconfiguration and make I/O changes statically.
The following command shows the correct method of removing a slot by first initiating a delayed reconfiguration on the root domain.
primary# ldm start-reconf ldg1 Initiating a delayed reconfiguration operation on the ldg1 domain. All configuration changes for other domains are disabled until the ldg1 domain reboots, at which time the new configuration for the ldg1 domain will also take effect. primary# ldm remove-io /SYS/MB/PCIE7 ldg1 ------------------------------------------------------------------------------ Notice: The ldg1 domain is in the process of a delayed reconfiguration. Any changes made to the ldg1 domain will only take effect after it reboots. ------------------------------------------------------------------------------ primary# ldm stop-domain -r ldg1
The following ldm list-io command verifies that the /SYS/MB/PCIE7 slot is no longer on the root domain.
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ pci_0 BUS pci_0 primary IOV pci_1 BUS pci_1 ldg1 IOV niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary OCC /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 ldg1 OCC /SYS/MB/PCIE3 PCIE pci_1 ldg1 OCC /SYS/MB/PCIE5 PCIE pci_1 ldg1 OCC /SYS/MB/PCIE7 PCIE pci_1 OCC /SYS/MB/PCIE9 PCIE pci_1 ldg1 EMP /SYS/MB/NET2 PCIE pci_1 ldg1 OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/PCIE5/IOVNET.PF0 PF pci_1 ldg1 /SYS/MB/PCIE5/IOVNET.PF1 PF pci_1 ldg1 /SYS/MB/NET2/IOVNET.PF0 PF pci_1 ldg1 /SYS/MB/NET2/IOVNET.PF1 PF pci_1 ldg1
The following commands assign the /SYS/MB/PCIE7 slot to the ldg2 domain. The ldm start command starts the ldg2 domain.
primary# ldm add-io /SYS/MB/PCIE7 ldg2 primary# ldm start ldg2 LDom ldg2 started
These commands create two virtual functions from each of the two physical functions that belong to the non-primary root domain.
primary# ldm create-vf /SYS/MB/PCIE5/IOVNET.PF0 Created new vf: /SYS/MB/PCIE5/IOVNET.PF0.VF0 primary# ldm create-vf /SYS/MB/PCIE5/IOVNET.PF0 Created new vf: /SYS/MB/PCIE5/IOVNET.PF0.VF1 primary# ldm create-vf /SYS/MB/NET2/IOVNET.PF1 Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF0 primary# ldm create-vf /SYS/MB/NET2/IOVNET.PF1 Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF1
You can also use the –n option to create the two virtual functions by using the following two commands:
primary# ldm create-vf -n 2 /SYS/MB/PCIE5/IOVNET.PF0 Created new vf: /SYS/MB/PCIE5/IOVNET.PF0.VF0 Created new vf: /SYS/MB/PCIE5/IOVNET.PF0.VF1 primary# ldm create-vf -n 2 /SYS/MB/NET2/IOVNET.PF1 Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF0 Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF1
If you were unable to dynamically create the virtual functions on a given physical function, initiate a delayed reconfiguration to create them statically.
primary# ldm start-reconf ldg1 primary# ldm create-vf /SYS/MB/PCIE5/IOVNET.PF0 Created new vf: /SYS/MB/PCIE5/IOVNET.PF0.VF0 primary# ldm create-vf /SYS/MB/PCIE5/IOVNET.PF0 Created new vf: /SYS/MB/PCIE5/IOVNET.PF0.VF1 primary# ldm create-vf /SYS/MB/NET2/IOVNET.PF1 Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF0 primary# ldm create-vf /SYS/MB/NET2/IOVNET.PF1 Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF1
Then reboot the root domain, ldg1, to effect the changes.
primary# ldm stop-domain -r ldg1
The following command shows the new virtual functions.
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ pci_0 BUS pci_0 primary IOV pci_1 BUS pci_1 ldg1 IOV niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary OCC /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 ldg1 OCC /SYS/MB/PCIE3 PCIE pci_1 ldg1 OCC /SYS/MB/PCIE5 PCIE pci_1 ldg1 OCC /SYS/MB/PCIE7 PCIE pci_1 ldg2 OCC /SYS/MB/PCIE9 PCIE pci_1 ldg1 EMP /SYS/MB/NET2 PCIE pci_1 ldg1 OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/PCIE5/IOVNET.PF0 PF pci_1 ldg1 /SYS/MB/PCIE5/IOVNET.PF1 PF pci_1 ldg1 /SYS/MB/NET2/IOVNET.PF0 PF pci_1 ldg1 /SYS/MB/NET2/IOVNET.PF1 PF pci_1 ldg1 /SYS/MB/PCIE5/IOVNET.PF0.VF0 VF pci_1 /SYS/MB/PCIE5/IOVNET.PF0.VF1 VF pci_1 /SYS/MB/NET2/IOVNET.PF1.VF0 VF pci_1 /SYS/MB/NET2/IOVNET.PF1.VF1 VF pci_1
The following command dynamically adds the /SYS/MB/PCIE5/IOVNET.PF0.VF1 virtual function to the ldg1 non-primary root domain:
primary# ldm add-io /SYS/MB/PCIE5/IOVNET.PF0.VF1 ldg1
The following command dynamically adds the /SYS/MB/NET2/IOVNET.PF1.VF0 virtual function to the ldg2 domain:
primary# ldm add-io /SYS/MB/NET2/IOVNET.PF1.VF0 ldg2
The following command adds the /SYS/MB/NET2/IOVNET.PF1.VF1 virtual function to the bound ldg3 domain:
primary# ldm add-io /SYS/MB/NET2/IOVNET.PF1.VF1 ldg3 primary# ldm start ldg3 LDom ldg3 started
Connect to the console of the ldg3 domain and then boot its OS.
The following output shows that all the assignments appear as expected. One virtual function is unassigned so it can be assigned dynamically to the ldg1, ldg2, or ldg3 domain.
# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ pci_0 BUS pci_0 primary IOV pci_1 BUS pci_1 ldg1 IOV niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary OCC /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 ldg1 OCC /SYS/MB/PCIE3 PCIE pci_1 ldg1 OCC /SYS/MB/PCIE5 PCIE pci_1 ldg1 OCC /SYS/MB/PCIE7 PCIE pci_1 ldg2 OCC /SYS/MB/PCIE9 PCIE pci_1 ldg1 EMP /SYS/MB/NET2 PCIE pci_1 ldg1 OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/PCIE5/IOVNET.PF0 PF pci_1 ldg1 /SYS/MB/PCIE5/IOVNET.PF1 PF pci_1 ldg1 /SYS/MB/NET2/IOVNET.PF0 PF pci_1 ldg1 /SYS/MB/NET2/IOVNET.PF1 PF pci_1 ldg1 /SYS/MB/PCIE5/IOVNET.PF0.VF0 VF pci_1 /SYS/MB/PCIE5/IOVNET.PF0.VF1 VF pci_1 ldg1 /SYS/MB/NET2/IOVNET.PF1.VF0 VF pci_1 ldg2 /SYS/MB/NET2/IOVNET.PF1.VF1 VF pci_1 ldg3