[openstack-dev] False Positive testing for 3rd party CI
Octavian Ciuhandu
ociuhandu at cloudbasesolutions.com
Fri Feb 21 19:13:18 UTC 2014
Hi,
On 21 Feb 2014, at 20:34, Aaron Rosen <aaronorosen at gmail.com<mailto:aaronorosen at gmail.com>> wrote:
Hi,
Yesterday, I pushed a patch to review and was surprised that several of the third party CI systems reported back that the patch-set worked where it definitely shouldn't have. Anyways, I tested out my theory a little more and it turns out a few of the 3rd party CI systems for neutron are just returning SUCCESS even if the patch set didn't run successfully (https://review.openstack.org/#/c/75304/).
Here's a short summery of what I found.
Hyper-V CI -- This seems like an easy fix as it's posting "build succeeded" but also puts to the side "test run failed". Would probably be a good idea to remove the "build succeeded" message to avoid any confusion.
The Hyper-V CI is non-voting (as required for new third-party CIs) and this is the reason why any post from it will show “build succeeded”. As published in other threads, AFAIK the only way to get rid of this issue is to have the CI as voting.
Brocade CI - From the log files it posts it shows that it tries to apply my patch but fails:
2014年02月20日 20:23:48 + cd /opt/stack/neutron
2014年02月20日 20:23:48 + git fetch https://review.openstack.org/openstack/neutron.git refs/changes/04/75304/1
2014年02月20日 20:24:00 From https://review.openstack.org/openstack/neutron
2014年02月20日<https://review.openstack.org/openstack/neutron2014-02-20> 20:24:00 * branch refs/changes/04/75304/1 -> FETCH_HEAD
2014年02月20日 20:24:00 + git checkout FETCH_HEAD
2014年02月20日 20:24:00 error: Your local changes to the following files would be overwritten by checkout:
2014年02月20日 20:24:00 etc/neutron/plugins/ml2/ml2_conf_brocade.ini
2014年02月20日 20:24:00 neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py
2014年02月20日 20:24:00 Please, commit your changes or stash them before you can switch branches.
2014年02月20日 20:24:00 Aborting
2014年02月20日 20:24:00 + cd /opt/stack/neutron
but still continues running (without my patchset) and reports success. -- This actually looks like a devstack bug (i'll check it out).
PLUMgrid CI - Seems to always vote +1 without a failure (https://review.openstack.org/#/dashboard/10117) though the logs are private so we can't really tell whats going on.
I was thinking it might be worth while or helpful to have a job that tests that CI is actually fails when we expect it to.
Best,
Aaron
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Thanks,
Octavian.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140221/971f6ff4/attachment.html>
More information about the OpenStack-dev
mailing list