Continuous integration testing for Windows desktop application

Question

I'm looking for ways to implement continuous integration testing for Windows desktop application we're developing. Basically, we need the following:

• After new build finishes start VM
• Deploy new build
• Run testing scripts
• Report

At first, I tried Vagrant but later came to conclusion it's cumbersome, since VM configuration is static, and I can't use single configuration for multiple VMs as some ports must be forwarded to host and they will collide. Now, I'm considering using OpenStack.

My question is: are there ready-to-use testing platforms?

Answer 1

As your building Windows desktop applications have you looked at what MS has to offer?

I think their Lab Management tools are designed for these challenges. Combined with TFS build server its should be able to bootstrap an test environments on the fly for each build and test run.

And here an article about combining TFS, Lab Management and CodedUI: https://visualstudiomagazine.com/articles/2011/07/01/pfcov_tfs2010-builds.aspx

• Andrey: It does look interesting from description on the site. Still we mainly use Atlassian products, so it's unlikely IT will agree to use another solution for process that is already set up.

Answer 2

Depending on your CI tool, you could just have it kick off a PowerShell or Bash script. That script can be made to handle everything you just mentioned. It is not exactly an out-of-the-box solution, but will only take a few hours to setup if you know a bit about scripting and know what command line commands to call in your script.

I used to have script that I would run after a build. To run it I would hop on the VM and double-click the file on the desktop. It would copy the build over and then fire off automated tests. When the automated tests finished they would handle the reporting (I use Telerik Test Studio).

Since you are using a Windows Desktop Application, your automated tools are limited to, pretty much, just AutoIT. Whose scripts can be called via command line (in your deployment script).

Answer 3

I haven't found ready to use solution capable of what I was looking for. So, I combined some building blocks: KVM/Libvirt transient VM + Salt automation + Robot framework testing. Altogether it allows to run multiple VMs on multiple hosts, running predefined tests. In theory, it also allows for distributed testing, with Salt Orchestrate and Robot framework Remote. But I haven't explored it yet.