Testing and Quality Assurance

I created a GHOP proposal to lay grounds for a simpletest suite of CCK field storage in http://drupal.org/node/209391.

This Wiki page aims at defining the actual list of tests that would constitute an exhaustive suite when we have an overall structure. The actual scope of the proposed GHOP task is not to implement all these tests, but to provide the overall structure and reusable code (note the 'Repeat the above' items), and actually code, say, the first two 'dark bullets' items below.

So I am writing a new CCK field and because I practice test driven development I find myself testing that I have implemented the CCK api hooks correctly. This basically means testing against the interface. Usually in a pure OO class orientated world for unit testing we would use a mock library (e.g.

Hello, since the QA BOF at DrupalCon Barcelona we have been developing an automated patch and unit testing system for Drupal core. We aren't quite ready to launch it live, but wanted to give QA enthusiasts an early look at what we are doing. Please review http://testing.drupal.org and file issues.

Hi everyone. I've written an article introducing unit testing and the Simpletest module on Lullabot.com. Hope you enjoy it!

There will be a discussion about Drupal testing and how to make it happen at Drupalcon Barcelona BOF room 1.
Sorry for such a short notice.

Hi Folks,

I just created a group as a "meta testing group" at http://groups.drupal.org/quality-assurance

Webchick pointed out this Unit Testing group which has a fair number of "general testing" notes in it beyond just unit testing.

So, either

1) A lot of you are probably interested in joining that other group as well
2) We can just merge them and rename this one to be a general testing group - perhaps "Testing and QA" could be the title so that it's as general as possible, though I'm not big on the importance of titles...

What kinds of things should the QA team work towards?

What kinds of tools might we need to get there?

Goals I can think of are:

1. Ensuring that "active" issues are high quality (non duplicate, full information to reproduce, etc)
2. Providing regression testing prior to major releases
3. Providing a way for "non-coders" to get involved in the project
4. {your goal here}

Tools to investigate to support some or all of these goals:

SimpleTest, CoScripter, Selenium, DAST, SimpleTest Automation Framework

This is a list of manual tests I perform every time I test Drupal 6.x core. I figure this is maybe a good start for things that we should have unit tests for. If nothing else, it provides a checklist for people who make changes to other modules of things to try to ensure they still work. Please feel free to add to this list!

Common things to check for:
- Visible errors (notices, warnings, etc.)
- Broken functionality
- "White screens of death (WSOD)" - Basically a completely blank screen when you go to a page.
- Regressions (when something used to work in a previous version of Drupal and no longer does)
- Broken links
- Incomplete or unclear help text/descriptions
- Typos in text, comments or code.

This is the best bit from my blog post: http://starbowconsulting.com/blog/tao/simpletest-tips
(Also, if anyone want to read me rant about the state of unit testing in drupal: http://starbowconsulting.com/blog/tao/joys-unit-testing)
Validating HTML Fragments
I am currently working on code that generates html fragments. Simple Test out-of-the-box does a great job helping you test entire http pages, but only it if goes and gets them itself. I could not find any easy way to load in a local page/fragment. So I did it the hard way:

I am trying to functional testing on a custom node creation form, and cannot figure out the correct syntax for selecting the taxonomy terms.

A couple of modules exist to aid the Drupal developer in creating functionally correct and robust code. Specifically for testing two types exist : there are the white box testing modules such as simple-test and the black box testing modules such as selenium.

Does anyone out there have a server which they control and is willing to let us install/maintain a simpletest-auto installation there? this will continously check developer patches to see if they break test suite. will also check core drupal itself.

Here you will find all the information you need to setup an environment where you will be able to automatically test patches.

One of the biggest concerns about automatic testing is how to prevent potentially dangerous code to cause harm to the server and other tests.

Here are short instrucions on how to use this module.

After some discussions with my mentor Thomas and other people on devel mailling list I have created sort of a first prototype.

How it works

One part of this module is used to catch the patches and submites them to other part of the module via xml-rpc call. The second part receives this call and tests the patch. Results are stored in a new node on test server node address is returend by xml-rpc call response.

Congrats to student Rok Zlender and mentor Thomas Ilsche! Here's the list of Drupal SOC approved projects, the Drupal Google Summer of Code Page and link to the Drupal Unit Testing Automation page. Please let us know how we can help!

After the jump, here's what I know is happening as of late May 2006. Add a comment and or post your activities related to Unit Testing because i am sure this is not comprehensive:

Ok - I'll get yea olde ball rolling with a question: why are the tests for the image and profile modules part of the main "simpletest" distribution?

Are these modules being moved into core?

These tests (duh) fail when these modules aren't present..which they are not in a default drupal + simpletest distro.

One of the big psychological goals of testing is to "prove" that everything is working as it should...so having the default simpletest suite fail because it expects two modules which are not core to be present is counter to that goal...

Thoughts?

s