Node API performance testing for OpenVPN connections #2

fkooman commented 2024年05月25日 09:33:22 +02:00

When an OpenVPN client connects to a VPN node, this generated a HTTP(S) connection to the portal/controller. If you have 1000+ clients and they (for some reason) all (re)connect at the same time, this might overwhelm the portal/controller and result in clients being unable to connect.

We should try to benchmark e.g. 1000 OpenVPN clients connecting at the same time, by for example calling node-api.php on the portal/controller in parallel using e.g. ab to try and establish a connection and see what happens.

Related:

• #1
• eduVPN/vpn-server-node#1

When an OpenVPN client connects to a VPN node, this generated a HTTP(S) connection to the portal/controller. If you have 1000+ clients and they (for some reason) all (re)connect at the same time, this might overwhelm the portal/controller and result in clients being unable to connect. We should try to benchmark e.g. 1000 OpenVPN clients connecting at the same time, by for example calling `node-api.php` on the portal/controller in parallel using e.g. `ab` to try and establish a connection and see what happens. Related: * https://codeberg.org/eduVPN/deploy/issues/1 * https://codeberg.org/eduVPN/vpn-server-node/issues/1

fkooman commented 2024年05月25日 13:07:03 +02:00

Idea for an approach:

• Use the Admin API to create 500 (or more) OpenVPN configurations;
• Make sure it is for a profile that does not use default gateway and does not push any routes;
• Load/enable them all using the systemd target openvpn-client, you may need to increase the limits using systemctl edit openvpn-client (option: LimitNPROC=10), initial testing seems LimitNPROC does not need to be increased?!;
• Run kill -9 openvpn (or use systemctl to restart all of them) which forces all OpenVPN processes to restart and thus talk to the node-api.php;
• Make vpn-server-node talk over https://vpn.example.org/ instead of http://localhost/ to see if TLS overhead causes issues.

The openvpn-client@.service override I'm using:

$ cat /etc/systemd/system/openvpn-client@.service.d/override.conf
[Service]
LimitNPROC=1000
Restart=always

The Restart=always is there because I want to be able to kill all OpenVPN processes and then it should reconnect in parallel instead of one after the other.

Idea for an approach: - Use the [Admin API](https://docs.eduvpn.org/server/v3/admin-api.html) to create 500 (or more) OpenVPN configurations; - Make sure it is for a profile that does not use default gateway and does not push any routes; - Load/enable them all using the `systemd` target `openvpn-client`, you may need to increase the limits using `systemctl edit openvpn-client` (option: `LimitNPROC=10`), initial testing seems `LimitNPROC` does not need to be increased?!; - Run `kill -9 openvpn` (or use `systemctl` to restart all of them) which forces all OpenVPN processes to restart and thus talk to the `node-api.php`; - Make `vpn-server-node` talk over `https://vpn.example.org/` instead of `http://localhost/` to see if TLS overhead causes issues. The `openvpn-client@.service` override I'm using: ```bash $ cat /etc/systemd/system/openvpn-client@.service.d/override.conf [Service] LimitNPROC=1000 Restart=always ``` The `Restart=always` is there because I want to be able to kill all OpenVPN processes and then it should reconnect *in parallel* instead of one after the other.

fkooman commented 2024年05月25日 13:17:02 +02:00

/etc/vpn-user-portal/config.php:

 [
 'profileId' => 'bench',
 'displayName' => 'OpenVPN Benchmark',
 'hostName' => 'vpn.example.org',
 'defaultGateway' => false,
 'oUdpPortList' => [1200],
 'oTcpPortList' => [],
 'oRangeFour' => '10.137.160.0/22',
 'oRangeSix' => 'fdd4:a80f:9fd1:af89::/64',
 ],

`/etc/vpn-user-portal/config.php`: ```php [ 'profileId' => 'bench', 'displayName' => 'OpenVPN Benchmark', 'hostName' => 'vpn.example.org', 'defaultGateway' => false, 'oUdpPortList' => [1200], 'oTcpPortList' => [], 'oRangeFour' => '10.137.160.0/22', 'oRangeSix' => 'fdd4:a80f:9fd1:af89::/64', ], ```

fkooman commented 2024年05月25日 13:22:09 +02:00

Using the Admin API, run this on the portal/controller itself:

#!/bin/sh

for I in $(seq -w 1 500); do
	echo $I
	curl \
		-s \
		-o "bench-${I}.conf" \
		-H "Authorization: Bearer $(sudo cat /etc/vpn-user-portal/keys/admin-api.key)" \
		-d "profile_id=bench" \
		-d "user_id=bench-user" \
		-d "display_name=Bench-${I}" \
		"http://localhost/vpn-user-portal/admin-api.php/v1/create"
	
	sudo cp bench-${I}.conf /etc/openvpn/client
	sudo systemctl enable --now "openvpn-client@bench-${I}"
done

NOTE: we SHOULD also test with having additional users, i.e. not all assign the configs to one user, we can have thus e.g. bench-001 etc. as user names (in case there is a performance difference if we have many users).

Using the Admin API, run this on the portal/controller itself: ```bash #!/bin/sh for I in $(seq -w 1 500); do echo $I curl \ -s \ -o "bench-${I}.conf" \ -H "Authorization: Bearer $(sudo cat /etc/vpn-user-portal/keys/admin-api.key)" \ -d "profile_id=bench" \ -d "user_id=bench-user" \ -d "display_name=Bench-${I}" \ "http://localhost/vpn-user-portal/admin-api.php/v1/create" sudo cp bench-${I}.conf /etc/openvpn/client sudo systemctl enable --now "openvpn-client@bench-${I}" done ``` **NOTE**: we SHOULD also test with having additional users, i.e. not all assign the configs to one user, we can have thus e.g. `bench-001` etc. as user names (in case there is a performance difference if we have many users).