We had another great PTG in Denver for Stein cycle. It was good time to meet all co-developers and discuss the stein cycle plan. I am writing the QA PTG summary report. Detailed discussion can be found in respective topic Etherpad or in main Etherpad.

QA Help Room

QA team was present in Help Room on Monday. We were happy to help few queries from Octavia multinode job and Kuryr-kubernetes testing part. Other than that, there was not much that day except few other random queries.

Rocky Retrospective

We discussed the Rocky Retrospective as first thing on Tuesday. We went through 1. what went well and 2. what needs to improve and gather some concrete action items.

Patrole has good progress in Rocky cycle with code as well as documentation. Also we were able to fill the compute microversion gap almost all till Rocky.

Stable interfaces from Tempest Plugins

We discussed about having stable interface from Tempest plugins like Tempest so that other plugins can consume those. Service client is good example of those which are required to do cross project testing. For example: congress tempest plugin needs to use mistral service clients for integration testing of congress+mistral. Similarly Patrole need to use neutron tempest plugin service client(for n/n-1/n-2).

Idea here is to have lib or stable interface in Tempest plugins side like Tempest so that other plugins can use them. We will start with some documentation about use case and benefits and then work with neutron-tempest-plugin team to make their service client expose as stable interface. Once that is done then, we can suggest the same to other plugins.

Action Items:

Need some documentation and guidance with use case and example, benefits for plugins. – felipemonteiro

mailing list discussions on making specific plugins stable that are consumed by other plugins – felipemonteiro

check with requirement team to add the tempest plugin in g-r and then those can be added on other plugins requirement.txt – gmann

Tempest Plugins CI setup & Plugins release and tagging clarification

We discussed about how other projects or Plugins can setup the CI to cover the stable branches testing on their master changes. Solution can be simple to define the supported stable branches and run them on master gate (same way Tempest does). QA team will start the guidelines on this.

Other part we need to cover is release and tagging guidelines. There were lot of confusion about release of Tempest plugins in Rocky. To make it better, QA team will write guidelines and document the clear process.

Action Items:

move/update documentation on branchless considerations in tempest to somewhere more global so that it covers plugins documentation too – gmann

Add tagging and release clarification for plugins.

talk with neutron team about moving in-tree tempest plugins of stadium projects to neutron-tempest-plugin or separate tempest-plugins repositories – slaweq

Tempest Cleanup Feature

Current Tempest CLI for cleanup the test resource is not so good. It does cleanup the resources based on saved_state.json file which save the resources difference before and after Tempest run. This can end up cleaning up the other non-test resources which got created during time period of tempest run.

There is a QA spec which proposing the different approach for cleanup. After discussing all those approach, we decided to go with resource_prefix. We will bring back the resource_prefix approach (which got removed after deprecation) and modify the “tempest cleanup” cli to cleanup resource based on resource_prefix. Complete discussion can be found in etherpad. As of felipemonteiro is the owner but he will check with nicholashelgeson or AT&T folks to further work on this.

Tempest conf Plugin Discovery process

This topic is about generating the Tempest conf from plugins config options too. This idea is more for python-tempestconf not for QA as such. But there were python-tempestconf folks present in QA room and discussed that this is doable in python-tempestconf itself.

There is nothing from QA side on this so I would like to drop this item from QA tracking.

Tempest tests not handling the hotplug/unplug events properly. The guest does not check for button press event at early boot time.

The hotplug events sent before the kernel fully initialized can be lost. test_stamp_pattern.py could be unskipped, if we would try to ssh vm before hot plug event (volume attach). Also there are API tests which knows nothing about the guest state, therefore it cannot know when the guest is ready for checking the button press. Details problem can be found here.

Idea is to perform the ssh validation step before we consider the test server ready to use in test.

Action Items:

Proposal for New QA project: Harbinger

OpenStack QA currently lacks a dataplane testing framework that can be easily consumed, configured and maintained. To fill that gap, there is a new project proposal called “Harbinger”. Harbinger allows execution of various OpenStack data plane testing frameworks while maintaining a single standardized interface and input format.

Currently it cover Shaker and Yardstick and Kloudbuster is WIP. This can be useful to consume in Eris (extreme testing idea). There are few points which need more clarification like Standardization of output, can it cover Control plane testing etc. IMO, this is good project to start and can be consumed in Eris and cross community testing. Author or this project was not present in PTG and felipemonteiro proxy this too. We would like to extend this discussion on ML and with Extreme testing stackholders and also start the QA spec.

Action Items:

There are many item we planned as AI but first step will to start the ML and spec.

Clean up the tempest documentation

This is always outstanding item :-). We discussed about more improvement in documentation like better doc structure, CLI doc, consuming the Tempest related docs at central place which is Tempest. We had list of item to cover with different assignee.

Action Items:

Complete all the documentation points written in etherpad. – tosky, gmann, masayukig

Consuming all Tempest CLI in gate

Tempest has many CLI and due to lack of unit tests, there are chance where we can/did break the CLI. Idea is to consume all the CLI on gate job so that we can improve their testing coverage. Few CLI will be covered in main gate job and other as functional testing.