Posted by catch on June 4, 2008 at 10:43am
If you're looking for current status, look at the most recent comment
Since we've still got core test failures, and no automated patch testing, I thought it'd be worth doing a snapshot of tests that pass week by week (or more frequently if there's lots of commits). This allows us to keep a historical overview of where bugs have been introduced into either core code, simpletest.module or tests themselves. It should also hopefully help to reduce duplicate issues about broken tests (since a test failure can be down to any one of those conditions).
Comments
5th/9th June
5th June 2008
22 modules pass
9 modules fail
of which:
2 are due to core bugs where the test is finding the bug
2 have failures or exceptions due to testing framework issues
5 have failures due to bugs in the test itself.
Both core bugs have RTBC patches. A couple of tests have RTBC patches too, so the situation could be better after some commits, but equally some issues don't have patches yet (and I had to make an issue for a couple of failures I couldn't find issues for).
Any corrections I'll edit in, although more than a week and it'd be worth doing another snapshot.
Tests that pass:
Block
Blog
Book
Comment
Contact
DBLog
Forum
Help
Locale
Menu
Node
PHP
Poll
Profile
Search (pgsql issue: http://drupal.org/node/264836)
SimpleTest
Syslog
Taxonomy
Tracker
Translation
Trigger
User
Tests that fail due to core bug:
Path: http://drupal.org/node/196862
Filter: http://drupal.org/node/161217
Fails or exceptions due to test framework issues:
Statistics: no patch http://drupal.org/node/255613
System: no patch http://drupal.org/node/256886
XML-RPC - no patch http://drupal.org/node/260507
BlogAPI - patch http://drupal.org/node/259988
Aggregator - no patch. http://drupal.org/node/266521
Fails:
Filter - patch: http://drupal.org/node/266539 (fixed 6th June)
Upload - patch (CNW) http://drupal.org/node/253506 (fixed 6th June)
9th June update
Here's another run through since there's been a lot of activity since last week.
Highlights
All failures are now down to bugs in simpletest.module or core, rather than the tests themselves.
There were a couple of new failures which were due to simpletest checking for invalid permissions - which it'd ignored before **
Passes:
Aggregator (but see RTBC patch here: http://drupal.org/node/260778)
Block
Blog
Book
Comment
DBLog
Forum
Help
Locale
Menu
Node
PHP
Poll
Profile
Search
SimpleTest
Syslog
System
Taxonomy
Tracker
Translation
Trigger
Upload
User
New fails since last snapshot
contact (patch: http://drupal.org/node/253506) - test framework regression
Fails due to bug in core code
Filter (patch: http://drupal.org/node/161217)
Path: http://drupal.org/node/196862
fails/exceptions due to bug in test framework
BlogAPI (RTBC http://drupal.org/node/260778 - also fixes aggregator edge case)
Statistics: no patch http://drupal.org/node/255613
XMLRPC: http://drupal.org/node/260507
--
**
new failures which were fixed while typing this up.
comment (patch: http://drupal.org/node/268449
System (ditto)
23 June
Unpatched:
Passes:
Aggregator
Block
Blog
Book
Comment
DBLog
Forum
Help
Locale
Menu
Node
PHP
Poll
Profile
Search
SimpleTest
Statistics (with exceptions http://drupal.org/node/255613)
Syslog
Taxonomy
Tracker
Translation
Trigger
Upload
New fails:
User ??
Forum (exception)
fails/exceptions due to test framework:
BlogAPI - http://drupal.org/node/260778
Contact - http://drupal.org/node/253506
System http://drupal.org/node/256886
XML-RPC
Fails due to core bug:
Filter: http://drupal.org/node/161217
Path: http://drupal.org/node/196862
Down to three:
edit: XML-RPC doesn't fail, but doesn't run either
So four then:
Contact: http://drupal.org/node/253506
Filter: http://drupal.org/node/161217
Poll (new): http://drupal.org/node/276200
XML-RPC: http://drupal.org/node/268063
today (July 2) I
today (July 2) I get:
Site-wide contact form: 120 passes, 4 fails, 0 exceptions
Core filters: 48 passes, 12 fails, 0 exceptions
Poll create: 27 passes, 7 fails, 0 exceptions
XML-RPC validator 0 passes, 0 fails, 3 exceptions
July 3rd
Getting longer again.
Cache expire - new test which exposes core bug: http://drupal.org/node/276267
Contact form
Filter
Poll
XML-RPC
sigh... :(
sigh... :(
July 9th
Cache expire test: 21 passes, 1 fail, 0 exceptions
Core filters: 68 passes, 12 fails, 0 exceptions
String translate: 85 passes, 2 fails, 0 exceptions - http://drupal.org/node/282340 - new
Poll create: 27 passes, 7 fails, 0 exceptions
18th July
No change:
Cache expire test: 21 passes, 1 fail, 0 exceptions
Core filters: 68 passes, 12 fails, 0 exceptions
String translate: 85 passes, 2 fails, 0 exceptions
Poll create: 27 passes, 7 fails, 0 exceptions
Tests go boom in comments
I pulled down a fresh copy of head this morning, and did a Run all tests. It got about 10% of the way in and went boom, with 775 passes, 1 fail, 0 exceptions. It looks like it was in the comments testing, 188 done, and then:
An error occurred. /head/batch?id=4&op=do <br /> <font size='1'>...(entire page of HTML)
Can you tell me the best place to report this?
I had something like this
I had something like this earlier - does it work the second time? Would probably post it against the simpletest component (as a critical bug).
Max execution time issue
I parsed the HTML, and it turns out to be a Max execution time exceeded error. Which brings up a couple of interesting issues, but there are probably better places for me to be talking about them.
Thanks.
Heh... ever since the
Heh... ever since the introduction of the registry, I've been forced to add "set_time_limit(0)" to my settings.php file for any Drupal instance I run on my slow local machine. ;)
Thanks
That did the trick. Although I am not sure if my test machine qualifies as slow (2 x 3Ghz CPUs, w/2G RAM), so this is probably hitting most people that try and test. Maybe that function should be called whenever the test suite is run?
Anyway, my results mirror catch's, except I also get
Syslog functionality: 15 passes, 4 fails, 0 exceptions
July 23rd
4775 passes, 26 fails, 0 exceptions
Cache expire test: 21 passes, 1 fail, 0 exceptions
Core filters: 68 passes, 12 fails, 0 exceptions
String translate: 85 passes, 2 fails, 0 exceptions
Poll create: 27 passes, 7 fails, 0 exceptions
Syslog functionality: 15 passes, 4 fails, 0 exceptions
August 3rd - postgres
(on postgres)
every test has 4 exceptions
blocks - 1 faily
freezes my firefox on page cache test.
August 5th
Everything passes except:
Core filters: 68 passes, 12 fails, 0 exceptions
We're down to one test failing! :)
1st November
It's been a while, but we're back :(
Module list functionality: 81 passes, 4 fails, 0 exceptions
Schema API
2 passes, 0 fails, 3 exception