[openstack-dev] Quantum in Full Gate Issues (was Re: Gerrit's Jenkins should stop running tests after first failure)
Ala Rezmerita
ala.rezmerita at cloudwatt.com
Wed Jun 19 15:38:13 UTC 2013
I put it here
https://etherpad.openstack.org/gate-tempest-devstack-vm-quantum-full
Ala
2013年6月19日 Miguel Lavalle <miguel at mlavalle.com>
> No, we haven't put this document in etherpad. But it's a great idea. This
> way other folks can jump in with ideas. I'll put it up in eteherpad over
> the next few days
>>> On Wed, Jun 19, 2013 at 10:10 AM, Sean Dague <sean at dague.net> wrote:
>>> This is great stuff! You guys have this up in etherpad somewhere?
>>>> Really great to see people organizing around this. I'd reviewed a bunch
>> of Jordan's code, so I knew he was in the mix, sorry for not realizing
>> there were other folks working on this as well.
>>>> -Sean
>>>>>> On 06/19/2013 10:27 AM, Miguel Lavalle wrote:
>>>>> Hi,
>>>>>> As of last week, we have organized a team to deal with the Quantum in
>>> full gate issues. There are 3 people working on this: Jordan Pittier,
>>> Ala.Rezmerita and myself. Below you will find a document describing all
>>> the identified issues and who is assigned to fix them. Of cource, we can
>>> use more hands / heads. If anyone wants to help or has identified other
>>> issues not listed below, please get in touch with us.
>>>>>> Regards
>>>>>> Fix Jenkins gate-tempest-devstack-vm-**quantum-full
>>>>>>>>> This document lists all the Tempest tests that are failing on a Devstack
>>> + Quantum Trunk setup. The goal is to make these tests pass for the
>>> Havana Milestone 3. Alla, Miguel and Jordan are working on this.
>>>>>>>>> For each test :
>>>>>> *
>>>>>> Provide the full path of the test. If the test has a double
>>> interface (JSON and XML), it’s enough to provide only the JSON path.
>>>>>> *
>>>>>> Provide a small stacktrace or a link to a Paste service such as
>>> http://paste.openstack.org/
>>>>>> *
>>>>>> Would be great to provide the CURL URL that triggers the bug
>>>>>> *
>>>>>> Provide a small analysis of what the bug may be and the components
>>> involved (Quantum, Quantum API on Nova’s side, Nova-network etc.)
>>>>>> *
>>>>>> As far as possible, provide the file and the line number where the
>>> Python exception is raised or translated (“casted”)
>>>>>> *
>>>>>> If already filed, an URL to the bug report in Launchpad
>>>>>>>>> Tests related to the Fixed IPs Compute API extension (Miguel)
>>>>>>>>> 1)
>>> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_list_
>>> **fixed_ip_details
>>>>>> *
>>>>>> Trace: http://paste.openstack.org/**show/38363/<http://paste.openstack.org/show/38363/>
>>>>>> *
>>>>>> curl-H "X-Auth-Token:$TOKEN" -X GET
>>> http://$IP:8774/v2/$TENANT_ID/**os-fixed-ips/10.0.0.3 <
>>> http://10.0.0.3>
>>>>>> *
>>>>>> API: http://api.openstack.org/api-**ref.html#ext-os-fixed-ips<http://api.openstack.org/api-ref.html#ext-os-fixed-ips>
>>>>>>>>> *
>>>>>> Explanation: Call to
>>> nova.api.openstack.compute.**contrib.fixed_ips.**
>>> FixedIPController::show()
>>> This file doesn’t use the Quantum API nor the Nova-Network API. It
>>> interacts directly with the DB, which is bad.
>>>>>> *
>>>>>> A possible fix would be to :
>>>>>> 1.
>>>>>>>>> Change
>>> nova.api.openstack.compute.**contrib.fixed_ips.**
>>> FixedIPController::show()
>>> to use either nova.network.api.API::get_**fixed_ip() (for Nova
>>> Network) or nova.network.quantumv2.api.**API:get_fixed_ip() (for
>>> Nova Network)
>>>>>> 2.
>>>>>>>>> Implement nova.network.quantumv2.api.**API:get_fixed_ip() which
>>> currently raises a NotImplementedError exception
>>>>>> 2)
>>> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_set_*
>>> *reserve
>>>>>> *
>>>>>> Trace: http://paste.openstack.org/**show/38372/<http://paste.openstack.org/show/38372/>
>>>>>> *
>>>>>> curl-H "X-Auth-Token:$TOKEN" -X POST
>>> http://$IP:8774/v2/$TENANT_ID/**os-fixed-ips/10.0.0.3/action
>>> <http://10.0.0.3/action>
>>>>>> *
>>>>>> API: http://api.openstack.org/api-**ref.html#ext-os-fixed-ips<http://api.openstack.org/api-ref.html#ext-os-fixed-ips>
>>>>>>>>> *
>>>>>> Possible Fix: Should call
>>> nova.api.openstack.compute.**contrib.fixed_ips.**
>>> FixedIPController::_set_**reserved()
>>> (once the result of db.fixed_ip_get_by_address() is made through the
>>> API). Here the problem is that neither Nova-Network nor Quatum API
>>> implement an equivalent of db.fixed_ip_update()
>>>>>> 3)
>>> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_set_*
>>> *unreserve
>>>>>> *
>>>>>> Same as 2)
>>>>>>>>>>>> Tests related to Quotas Admin(Miguel)
>>>>>>>>> 4)tempest.api.compute.admin.**test_quotas:**QuotasAdminTestJSON.test_**
>>> security_groups_exceed_limit
>>>>>> 5)tempest.api.compute.admin.**test_quotas:**QuotasAdminTestJSON.test_**
>>> security_groups_rules_exceed_**limit
>>>>>>>>> Tests related to Floating IPs
>>>>>>>>> 6)tempest.api.compute.**floating_ips.test_floating_**ips_actions:**
>>> FloatingIPsTestJSON.test_**associate_ip_to_server_**
>>> without_passing_floating_ip
>>>>>> *
>>>>>> Trace Tempest http://paste.openstack.org/**show/38431/<http://paste.openstack.org/show/38431/>
>>>>>> *
>>>>>> Bug : https://bugs.launchpad.net/**quantum/+bug/1190242<https://bugs.launchpad.net/quantum/+bug/1190242>
>>>>>> *
>>>>>> curl-H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
>>> POST http://$IP:8774/v2/$TENANT_ID/**servers/$SERVER_ID/action
>>> <http://10.1.59.157:8774/v2/**de6c5fbc55a34cbcaa3d79eb6b21a7**
>>> 84/servers/0b2ad3b6-c14a-4d89-**b2a0-c015f0a88a1f/action<http://10.1.59.157:8774/v2/de6c5fbc55a34cbcaa3d79eb6b21a784/servers/0b2ad3b6-c14a-4d89-b2a0-c015f0a88a1f/action>
>>> >-d
>>> ‘{"addFloatingIp": {"address": ""}}’
>>>>>>>>> *
>>>>>> Notice that the address of the FloatingIP is empty.
>>>>>> *
>>>>>> API : http://api.openstack.org/api-**ref.html#ext-floating-ips<http://api.openstack.org/api-ref.html#ext-floating-ips>
>>>>>>>>> *
>>>>>> Explanation : Mismatch in the 2 API. If there is only one IP in the
>>> pool, Quantum allows the floating IP to be empty and returns the one
>>> and only IP in the pool. Nova-network doesn’t allow this and returns
>>> a 404
>>>>>> *
>>>>>> Review https://review.openstack.org/#**/c/32740/<https://review.openstack.org/#/c/32740/>
>>>>>>>>> 7)tempest.api.compute.**floating_ips.test_floating_**ips_actions:**
>>> FloatingIPsTestJSON.test_**delete_floating_ip
>>>>>> *
>>>>>> Trace Tempest : http://paste.openstack.org/**show/38505/<http://paste.openstack.org/show/38505/>
>>>>>> *
>>>>>> Bug : https://bugs.launchpad.net/**tempest/+bug/1160309(see<https://bugs.launchpad.net/tempest/+bug/1160309(see>comment #2)
>>>>>>>>> *
>>>>>> This is related to bug 9)
>>>>>>>>> 8)tempest.api.compute.**floating_ips.test_floating_**ips_actions:**
>>> FloatingIPsTestJSON.test_**delete_nonexistant_floating_ip
>>>>>>>>> *
>>>>>> Related to 9)
>>>>>>>>> 9)tempest.api.compute.**floating_ips.test_list_**floating_ips:**
>>> FloatingIPDetailsTestJSON.**test_get_nonexistant_floating_**ip_details
>>>>>> *
>>>>>> Bug :https://bugs.launchpad.net/**tempest/+bug/1160309<https://bugs.launchpad.net/tempest/+bug/1160309>
>>>>>> *
>>>>>> Trace tempest:http://paste.**openstack.org/show/38430/<http://paste.openstack.org/show/38430/>
>>>>>> *
>>>>>> Trace nova: http://paste.openstack.org/**show/38433/<http://paste.openstack.org/show/38433/>
>>>>>>>>> *
>>>>>> Curl:curl -H "X-Auth-Token:$TOKEN" -X GET
>>> http://$IP:8774/v2/$PROJECT_**ID/os-floating-ips/**99987878787878
>>>>>> Proposed Fix: https://review.openstack.org/**33024<https://review.openstack.org/33024>
>>>>>>>>> Tests Related to Security Groups (Jordan)
>>>>>>>>> 10)tempest.api.compute.**security_groups.test_security_**group_rules:**
>>> SecurityGroupRulesTestJSON.**test_security_group_rules_**
>>> create_with_invalid_id
>>>>>> *
>>>>>> TRACE: http://paste.openstack.org/**show/38373/<http://paste.openstack.org/show/38373/>
>>>>>> *
>>>>>> curl-H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
>>>>>> POST http://$IP:8774/v2/$TENANT_ID/**os-security-group-rules -d
>>> ‘{"security_group_rule": {"from_port": 22, "ip_protocol": "tcp",
>>> "to_port": 22, "parent_group_id": "9991393497170", "cidr": null,
>>> "group_id": null}}’
>>>>>> *
>>>>>> Explanation: Notice that the parent_goup_id is a numerical ID and
>>> not a UUID. Quantum has an additional check to validate that the ID
>>> is an UUID (see
>>> nova/network/security_group/**quantum_driver.py::validate_**id())
>>>>>> *
>>>>>> API: http://api.openstack.org/api-**ref.html#ext-os-security-**
>>> groups <http://api.openstack.org/api-ref.html#ext-os-security-groups>
>>>>>> *
>>>>>> Bug: https://bugs.launchpad.net/**tempest/+bug/1182384<https://bugs.launchpad.net/tempest/+bug/1182384>
>>>>>>>>> *
>>>>>> Review: https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/>
>>>>>>>>> 11)tempest.api.compute.**security_groups.test_security_**group_rules:**
>>> SecurityGroupRulesTestJSON.**test_security_group_rules_**
>>> delete_with_invalid_id
>>>>>> *
>>>>>> http://paste.openstack.org/**show/38424/<http://paste.openstack.org/show/38424/>
>>>>>>>>> *
>>>>>> curl -H "X-Auth-Token:$TOKEN" -X DELETE
>>> http://$IP:8774/v2$TENANT_ID/**os-security-group-rules/**
>>> 9991407551273
>>>>>> *
>>>>>> Explanation : Same bug as 10
>>>>>> *
>>>>>> API: http://api.openstack.org/api-**ref.html#ext-os-security-**
>>> groups <http://api.openstack.org/api-ref.html#ext-os-security-groups>
>>>>>>>>> *
>>>>>> Review: https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/>
>>>>>> 12)tempest.api.compute.**security_groups.test_security_**
>>> groups:SecurityGroupsTestJSON.**test_delete_nonexistant_**security_group
>>>>>> *
>>>>>> Same as bug 10
>>>>>>>>> *
>>>>>> Review: https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/>
>>>>>>>>> 13)tempest.api.compute.**security_groups.test_security_**
>>> groups:SecurityGroupsTestJSON.**test_security_group_get_**
>>> nonexistant_group
>>>>>> *
>>>>>> Same as bug 10
>>>>>>>>> *
>>>>>> Review: https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/>
>>>>>>>>> 14)tempest.api.compute.**security_groups.test_security_**
>>> groups:SecurityGroupsTestJSON.**test_security_group_create_**
>>> with_duplicate_name
>>>>>> Security Group with duplicate name should not be created, but two groups
>>> with the same name can be created in quantum. We have here the same
>>> problem as in 15 and in 16. With Quantum, there is no validation that a
>>> group with given name exists already or if the given SG name is empty
>>> or is composed of white spaces or is more than 255 chars.
>>>>>>>>> In the description of bug
>>> https://bugs.launchpad.net/**nova/+bug/1161411this<https://bugs.launchpad.net/nova/+bug/1161411this>issue is generally
>>>>>> discussed. SecurityGroup API are based on the ID and not the names,
>>> except for adding an instance to a security group. In order to solve
>>> the last problem the bug https://bugs.launchpad.net/**
>>> nova/+bug/1161473was <https://bugs.launchpad.net/nova/+bug/1161473was>
>>> added.
>>>>>>>>> The major question is if these 3 tests (14, 15, 16) : does the name of a
>>> security group is really that important? If so, we must add some
>>> validation methods. If not the test suit concerning this part must be
>>> disable in tempest. What do you think Miguel?
>>>>>>>>>>>> 15)tempest.api.compute.**security_groups.test_security_**
>>> groups:SecurityGroupsTestJSON.**test_security_group_create_**
>>> with_invalid_group_description
>>>>>> 16)tempest.api.compute.**security_groups.test_security_**
>>> groups:SecurityGroupsTestJSON.**test_security_group_create_**
>>> with_invalid_group_name
>>>>>> BUG:https://bugs.launchpad.**net/nova/+bug/1161411+<https://bugs.launchpad.net/nova/+bug/1161411+>
>>> https://bugs.launchpad.net/**nova/+bug/1161473<https://bugs.launchpad.net/nova/+bug/1161473>
>>>>>> Traceback(tempest) :http://paste.openstack.org/**show/38423/<http://paste.openstack.org/show/38423/>
>>>>>> The Security Group should not be created with group name an empty string
>>> or with white spaces/chars more than 255
>>>>>> CURL:curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN"
>>> -X POST http://$IP:8774/v2/$PROJECT_**ID/os-security-groups -d
>>> '{"security_group": {"name": " ", "description":
>>> "description-1554950088"}}'
>>>>>> curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
>>> POST http://$IP:8774/v2/$PROJECT_**ID/os-security-groups -d
>>> '{"security_group": {"name": " ", "description":
>>> "description-1554950088"}}'
>>>>>>>>> 17)tempest.api.compute.**security_groups.test_security_**
>>> groups:SecurityGroupsTestJSON.**test_server_security_groups
>>>>>> *
>>>>>> Tempest Trace : http://paste.openstack.org/**show/38427/<http://paste.openstack.org/show/38427/>
>>>>>> *
>>>>>> Fixed by https://review.openstack.org/#**/c/32288/(merged<https://review.openstack.org/#/c/32288/(merged>on June,12th)
>>>>>>>>>>>>>>> Tests related to servers(Ala)
>>>>>>>>> 18)tempest.api.compute.**servers.test_list_server_**filters:**
>>> ListServerFiltersTestXML.test_**list_servers_filtered_by_ip_**regex
>>>>>> *
>>>>>> BUG: https://bugs.launchpad.net/**quantum/+bug/1182883<https://bugs.launchpad.net/quantum/+bug/1182883>
>>>>>> *
>>>>>> BP: https://blueprints.launchpad.**net/quantum/+spec/like-op-list<https://blueprints.launchpad.net/quantum/+spec/like-op-list>
>>>>>>>>> *
>>>>>> CURL : GET http://$IP:8774/v2/$PROJECT_**ID/servers?ip=10.
>>>>>> *
>>>>>> Explanation: The regex search is not supported by Quantum. Thus
>>> Quantum returns a 404 Not Found (0 server match) where Tempest
>>> expects one server to be found.
>>>>>> *
>>>>>> Possible Fix : For "search port by IP with regex" feature, I think
>>> the best place to hack it would be in file
>>> quantum/db/db_base_plugin_v2.**py::_get_ports_query()
>>>>>>>>> 19)tempest.api.compute.**servers.test_servers_negative:**
>>> ServersNegativeTest.test_**create_with_nonexistent_**security_group
>>>>>> FIXED : https://review.openstack.org/#**/c/30271/<https://review.openstack.org/#/c/30271/>
>>>>>>>>> 20)tempest.api.compute.**servers.test_virtual_**interfaces:**
>>> VirtualInterfacesTestXML.test_**list_virtual_interfaces
>>>>>> *
>>>>>> TRACE (NOVA): http://paste.openstack.org/**show/38371/<http://paste.openstack.org/show/38371/>
>>>>>> *
>>>>>> BUG: https://bugs.launchpad.net/**tempest/+bug/1183436<https://bugs.launchpad.net/tempest/+bug/1183436>
>>>>>>>>> *
>>>>>> CURL: GET
>>> http://$IP:8774/v2/$PROJECT_**ID/servers/$SERVER/os-virtual-**
>>> interfaces
>>>>>> *
>>>>>> Explanation: This HTTP request calls the Quantum API
>>> (nova/nova/network/quantumv2/**api.py) and specifically the
>>> get_vifs_by_* methods which are not implemented (raise
>>> NotImplementedError())
>>>>>> *
>>>>>> Possible Fix:
>>>>>>>>> o
>>>>>> skip this test if Quantum is enabled as set in Tempest
>>> configuration. Or
>>>>>> o
>>>>>> Implement the get_vifs_by_* methods
>>>>>> PATCH: (still not approved) https://review.openstack.org/#**/c/31755/<https://review.openstack.org/#/c/31755/>
>>>>>>>>> On Tue, Jun 18, 2013 at 5:56 PM, Sean Dague <sean at dague.net
>>> <mailto:sean at dague.net>> wrote:
>>>>>> On 06/18/2013 03:32 PM, Monty Taylor wrote:
>>>>>>>>>>>> On 06/18/2013 03:14 PM, David Ripton wrote:
>>>>>> On 06/18/2013 12:43 PM, Martina Kollarova wrote:
>>>>>> Jenkins keeps running all the tests, even if the basic
>>> pep8 test fails,
>>> and runs all of the (very slow) Tempest Quantum tests,
>>> even though
>>> almost all of them are failing.
>>>>>> I propose that it should fail and stop all of the other
>>> tests once there
>>> is a failure in a voting test. For non-voting tests, it
>>> should stop only
>>> itself, not the others.
>>>>>> This would decrease the feedback loop and we wouldn't
>>> have to wait for
>>> the non-voting Quantum tests to see that they failed as
>>> always.
>>>>>>>>> -1
>>>>>> In addition to the other objections, we currently get a lot
>>> of false
>>> positives (fail, retry, fail, retry, succeed), and it would
>>> be harder to
>>> debug these problems if the output was truncated differently
>>> each time.
>>>>>> Is anyone working on fixing the perma-failing Quantum test?
>>> When the
>>> Postgres test was perma-failing, one of the infrastructure
>>> folks gave us
>>> an ultimatum that if nobody fixed it soon, it would be
>>> disabled. (Happy
>>> ending: Mauro fixed it before it got disabled.)
>>>>>>>>> That was brought up a little while ago, but we had already spent
>>> so much
>>> effort to get it working in the first place, none of us had the
>>> heart to
>>> put in such an ultimatum. But seriously- it might be time for an
>>> all-hands-on-deck dogpile to figure out what's up the the
>>> quantum gate.
>>>>>>>>> The biggest cause of the Quantum vs. Full Tempest runs is that a lot
>>> of the network api's in nova currently don't do translation of
>>> errors. So under nova-network certain data validation and error
>>> codes are returned, when quantum is used others are returned.
>>>>>> This is a nova-api, so it needs to be consistent regardless of
>>> backend (i.e. we don't return different API responses on different
>>> databases).
>>>>>> Issues like this one -
>>> https://bugs.launchpad.net/__**nova/+bug/1160309<https://bugs.launchpad.net/__nova/+bug/1160309>
>>>>>> <https://bugs.launchpad.net/**nova/+bug/1160309<https://bugs.launchpad.net/nova/+bug/1160309>
>>> >
>>>>>> Jordan Pittier has been working on some of these issues (he's the
>>> only one I've seen working them from a Tempest / nova side), and got
>>> to the crux of the problem. It could use more hands though to
>>> organize the rest of those and get them banged out.
>>>>>> I'm sure there are other issues once we get past this class. But
>>> that would go a long way.
>>>>>> -Sean
>>>>>> --
>>> Sean Dague
>>> http://dague.net
>>>>>> ______________________________**___________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.**__org
>>> <mailto:OpenStack-dev at lists.**openstack.org<OpenStack-dev at lists.openstack.org>
>>> >
>>> http://lists.openstack.org/__**cgi-bin/mailman/listinfo/__**
>>> openstack-dev<http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev><
>>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>> >
>>>>>>>>>>>>>>>>>> ______________________________**_________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.**org <OpenStack-dev at lists.openstack.org>
>>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>>>>>>>> --
>> Sean Dague
>> http://dague.net
>>>> ______________________________**_________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.**org <OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>>> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
--
Ala Rezmerita
Ingénieur
CloudWatt
Tel : 06 77 43 23 91
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130619/3438a25a/attachment.html>
More information about the OpenStack-dev
mailing list