Continuous integration and modifying existing UI automation tests

Question

Our team is currently facing a situation where we are changing a feature that is in the product today. This feature has Selenium automation tests associated with it.

We are performing a fairly major overhaul of the system, including its UI. We expect this work to take quite a few sprints. During this time, other teams may do work that would affect the feature as it exists today. For this reason, we want the existing automation tests for the feature to remain intact so that they can detect issues.

Of course, we will need to write automation for the feature as it will be after our changes are done. The most efficient way for us to do this from a Dev/QA perspective would be to modify the existing test cases in a separate branch (testing locally) and then merge those changes in when the feature is ready. That way the existing automation continues to function and we don't have to do extra work duplicating each test.

However, this separate branch doesn't fit well with the principles of continuous integration (which our organization likes to adhere to). Is our only CI-compliant option to do the extra overhead of duplicating the tests for modification?

• Bart van Ingen Schenau: A few questions: 1) Will the product retain its functionality during the time it takes to do the overhaul? Or will the change be disruptive?
• Bart van Ingen Schenau: 2) What are the chances of the existing UI tests flagging a false-positive result on the new design?
• Bart van Ingen Schenau: 3) What does CI mean to you and your company? Could you set-up a second CI system for the branch on which the overhaul gets done?
• Robert Winslow Dalpe: 1) The change will not be disruptive. The modified feature will be developed and then "swapped in" when it's ready. 2) Assuming I understand you correctly, the tests will certainly fail with the new design. 3) Our team isn't sure if this is a possibility or not (we have yet to talk to higher management). Is it "OK" if we do our full build process on this potential branch and then merge it? Is that sufficient integration testing?
• Bart van Ingen Schenau: Given your answers to 1 and 2, without a separate branch for the new design, you are either not testing it in the CI environment until you think its done, or it will break the CI environment while you are working on it due to false-positives from the old tests (and possibly also from the new tests on the old code).

Answer 1

I would try to see it from this point of view:

• you are going to develop a parallel branch for a defined time interval (which is of course oppsoed to the "standard" CI model)

• this will cause a parallel development of your tests as well - branch them accordingly (and merge them back into the trunk when you merge the main source code).

• apply that branch to your CI environment as well - for example, if you have automated builds and tests on a server, let your server build and test both, the trunk as well as the branch, as long as the branch exists

• you can still apply continuos integration techniques to your trunk and your branch separately

• to minimize the effort of merging later, you should try to merge any changes in the trunk back into the development branch as often as possible, once per day would be best. This may not be simple in the area where the UI of the dev branch differs from the trunk, so try to restrict changes to the trunk UI in that area to a minimum during that time.

As you see, when you leave the "CI" path and make a long-term branch and merge, you may have to do it completely, for your whole source code including the tests and the CI environment, causing additional administrative effort. But sometimes pure CI is just not the development model you need and it may be worth it.

EDIT: as an alternative, you could try avoid branching within your VCS and solve your problem with a feature toggle. You could design that toggle in a way so can switch between the old and the new UI at runtime. This may be used within your Selenium tests - modfiy the tests in a way so the old tests run only against the old UI (feature disabled), and the tests adapted to the new UI only against the new UI (feature enabled).

• Robert Winslow Dalpe: We're going to speak with our build team and see if this is an acceptable approach for them. Unfortunately the feature toggle approach won't work ideally for the tests themselves, else we would take that approach. Thank you for your input as well.
• Doc Brown: @RobertWinslowDalpe: "Unfortunately the feature toggle approach won't work ideally for the tests themselves" - sure? To my experience that depends much on how you implement the toggle, there is more than one way to do it.
• Robert Winslow Dalpe: I'll look deeper into the possibility of using a feature toggle; you're right that I shouldn't dismiss it so quickly. That said, without going into the details, I'm not wholly optimistic that our tests will be able to detect the feature toggle without us having to do the extra overhead that we're trying to avoid in the first place.
• Luc Franken: In most cases you have something like a test runner. That point might be enough to decide which tests to run or to ignore based on the feature you want to test.