problem

We have an ACS environment that we have been successfully upgrading since version 4.18. We last upgraded from version 4.20 to 4.21. ACS in HA was upgraded fine. However, when upgrading the KVM cloudstack agents, there were problems connecting the agents to ACS. They were connecting and disconnecting all the time. Finally, we did a rollback but we still have problems connecting the agents even though we downgraded the agents to 4.20.1. The VMs are running, but the KVMs cannot connect.

versions

ACS 4.20 on OL9, KVM OL8,

The steps to reproduce the bug

1. Upgrade ACS from 4.20 to 4.21
2. Upgrade CS agent from 4.20 to 4.20.2

What to do about it?

We try dowgrade the agent to version 4.20 but we are not sucessfull. There were 2 different errors on the CS agent. I am attaching the log file.

Error message on agent.log

2025年10月27日 12:49:26,477 INFO [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) iscsi session clean up is disabled
2025年10月27日 12:49:26,481 INFO [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) The [vm.memballoon.stats.period] property is set to '0', this prevents memory statistics from being displayed corre
ctly. Adjust (increase) the value of this parameter to correct this.
2025年10月27日 12:49:31,387 INFO [cloud.agent.Agent] (main:[]) (logid:) Agent [id = 5, uuid = b1991b02-ccc4-4980-a416-0bc962d8d845, name = kvm01.xxx.sk, type = LibvirtComputingResource, zone = 1, pod = 1, workers = 5, port = 8250] with host = 172.xx.xx.xx local id = 1
2025年10月27日 12:49:31,392 INFO [utils.nio.NioClient] (main:[]) (logid:) Connecting to 172.16.x.x:8250
2025年10月27日 12:49:31,398 INFO [utils.nio.NioClient] (main:[]) (logid:) Connected to 172.16.x.x:8250
2025年10月27日 12:49:31,401 INFO [utils.nio.Link] (main:[]) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2025年10月27日 12:49:31,825 INFO [utils.nio.NioClient] (main:[]) (logid:) SSL: Handshake done
2025年10月27日 12:49:31,897 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching CPU speed from command "lscpu".
2025年10月27日 12:49:31,929 ERROR [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Unable to retrieve the CPU speed from lscpu. java.lang.NullPointerException: Cannot invoke "String.trim()" because "in" is null
at java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1838)
at java.base/jdk.internal.math.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
at java.base/java.lang.Float.parseFloat(Float.java:476)
at org.apache.cloudstack.utils.linux.KVMHostInfo.getCpuSpeedFromCommandLscpu(KVMHostInfo.java:148)
at org.apache.cloudstack.utils.linux.KVMHostInfo.getCpuSpeed(KVMHostInfo.java:119)
at org.apache.cloudstack.utils.linux.KVMHostInfo.getHostInfoFromLibvirt(KVMHostInfo.java:214)
at org.apache.cloudstack.utils.linux.KVMHostInfo.(KVMHostInfo.java:68)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.initialize(LibvirtComputingResource.java:4123)
at com.cloud.agent.Agent.sendStartup(Agent.java:556)
at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1314)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

2025年10月27日 12:49:06,970 ERROR [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Failed to parse GPU discovery result: com.google.gson.stream.MalformedJsonException: Expected value at line 1 column 1 path $ com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Expected value at line 1 column 1 path $
at com.google.gson.internal.Streams.parse(Streams.java:61)
at com.google.gson.JsonParser.parseReader(JsonParser.java:102)
at com.google.gson.JsonParser.parseReader(JsonParser.java:70)
at com.google.gson.JsonParser.parseString(JsonParser.java:51)
at com.google.gson.JsonParser.parse(JsonParser.java:115)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.parseGpuDevicesFromResult(LibvirtComputingResource.java:2016)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getGpuDevices(LibvirtComputingResource.java:2009)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.initialize(LibvirtComputingResource.java:4164)
at com.cloud.agent.Agent.sendStartup(Agent.java:556)
at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1314)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.google.gson.stream.MalformedJsonException: Expected value at line 1 column 1 path $
at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1659)
at com.google.gson.stream.JsonReader.doPeek(JsonReader.java:599)
at com.google.gson.stream.JsonReader.peek(JsonReader.java:433)
at com.google.gson.internal.Streams.parse(Streams.java:47)
... 15 more