Hello everyone,

I have a lab with a vCenter / NSX and i am trying to add Cloudstack ontop. I have followed the docs to configure vCenter with NSX and the vCenter cluster has been added to Cloudstack successfully.

Cloudstack version: 4.21.0.0
NSX Version: 4.2.0.1.0.24210154
vCenter Version: 8.0.3

But when i try to create an isolated network the call fails with error:

unable to create NSX network ad00c442-6a64-4772-ab94-43141cc98337 due to: Failed API call to NSX controller

The strange thing is that in NSX the T1 Gateway is being created

Here are the logs from Cloudstack:

2025年10月27日 07:10:55,010 DEBUG [o.a.c.s.NsxApiClient] (DirectAgent-22:[ctx-c6ddeea7]) (logid:ea3ad574) NSX Tier-1 gateway with name: D1-A2-Z1-N224 not found
2025年10月27日 07:10:55,195 ERROR [o.a.c.s.NsxApiClient] (DirectAgent-161:[ctx-385a40c3]) (logid:ea3ad574) Error creating segment D1-A2-Z1-S224: General error has occurred.
2025年10月27日 07:10:55,195 ERROR [o.a.c.r.NsxResource] (DirectAgent-161:[ctx-385a40c3]) (logid:ea3ad574) Failed to create network: adf
2025年10月27日 07:10:55,195 ERROR [o.a.c.u.NsxControllerUtils] (qtp1245065720-24:[ctx-d382e352, ctx-25d4fd50]) (logid:ea3ad574) NSX API Command failed
2025年10月27日 07:10:55,219 ERROR [c.c.a.ApiServer] (qtp1245065720-24:[ctx-d382e352, ctx-25d4fd50]) (logid:ea3ad574) unhandled exception executing api command: [Ljava.lang.String;@2edc4927 com.cloud.utils.exception.CloudRuntimeException: unable to create NSX network 2b833a44-68a5-4d63-b449-8129c4074289due to: Failed API call to NSX controller
at org.apache.cloudstack.service.NsxGuestNetworkGuru.setup(NsxGuestNetworkGuru.java:153)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:838)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator11ドル.doInTransaction(NetworkOrchestrator.java:3098)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator11ドル.doInTransaction(NetworkOrchestrator.java:2995)
at com.cloud.utils.db.Transaction2ドル.doInTransaction(Transaction.java:48)
at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.createGuestNetwork(NetworkOrchestrator.java:2995)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.createGuestNetwork(NetworkOrchestrator.java:2734)
at com.cloud.network.NetworkServiceImpl4ドル.doInTransaction(NetworkServiceImpl.java:2332)
at com.cloud.network.NetworkServiceImpl4ドル.doInTransaction(NetworkServiceImpl.java:2270)
at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
at com.cloud.network.NetworkServiceImpl.commitNetwork(NetworkServiceImpl.java:2270)
at com.cloud.network.NetworkServiceImpl.createGuestNetwork(NetworkServiceImpl.java:1830)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy200.createGuestNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.CreateNetworkCmd.execute(CreateNetworkCmd.java:431)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:854)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:668)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:416)
at com.cloud.api.ApiServlet1ドル.run(ApiServlet.java:193)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext1ドル.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:190)
at com.cloud.api.ApiServlet.doPost(ApiServlet.java:149)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle1ドル(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint1ドル.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:840)

I have also tried to create a VPC which works ok and i can ping the public IP address it allocates but when i try to create a network inside the VPC it fails again with the same error.

So my guess is that there is something going on when Cloudstack tries to create a segment on NSX.

Is this a bug or have i forgotten to configure something?