In Dries' keynote on Monday, he announced that he was willing to push back the code freeze to mid-October if Drupal has full test coverage. This is going to require a cultural shift towards test-driven development that is supported by a community who writes both functional and unit tests. This will ensure that improvements in one area of Drupal don't cascade and break other areas, which means that more time is spent in the development cycle fixing bugs rather than adding new features.

The unit testing working group met with Dries on Tuesday morning, and they came up with a tentatively-approved plan to start including SimpleTests in core. This will emphasize the importance of writing testable code, but requires active support from the Drupal community to help write functional and unit tests.

If there isn't enough progress by May, then Dries will be forced to shorten the code freeze and focus on fixing bugs. On Monday morning, four members of the testing working group presented the following information on the SimpleTest framework and the future of test-driven development in Drupal.

JIMMY BERRY: We're going to start with an introduction to unit testing and Simple Tests using the presentation that I put together as a GHOP task.

Simple Test helps ensure that modules are tested, and it's an easy way to validate your patches. It can facilitate change because developers are less likely to submit a patch that creates errors.

SimpleTest is a great way to test for edge cases. It's also pretty good for debugging from the error messages that are provided.

Definition of Unit Test: Procedure used to validate that the code is working properly.

What is SimpleTest? It's an open source PHP framework for quickly and easily performing functional and unit tests. Creates an abstract class to separate your code into functions.
Performs common testing, and it has an internal web browser to simulate user requests. No more clicking on forms since it is automated.

Some SimpleTest adaptations: It's set up as a Drupal module, and an admin page to select the tests to be run, and ways to automate tasks like creating a node, and it automates other stuff as well.

The SimpleTest automator module provides a quick way of creating SimpleTests. First select the user permission and then start recording your actions, which is like a macro-builder for SimpleTests. This is made possible by Drupal 6, and not likely to be available for Drupal 5.

The interface that SimpleTest provides to the admin area -- allows you to select tests -- like BlogAPI. Test results are easy to read, and the results show up in Green for pass and Red for fail.

A simple workflow for creating SimpleTest, plan out what you want to test -- find the minimum privileges for the test user, and then choose to automate it or create from scratch. If you automate it to start with, then clean it up. Be sure that the code actually catches the error.

For every IF condition, then you want to have a SimpleTest to test that so that you can assert that you're either expecting either an error or success.

Create modules and the test user, and make sure the testing environment is working. Test the log-in user and assert what the end result from be and then repeat for all tasks, and then clean up whatever was created by the SimpleTest in the database.

"Functional testing" is the user clicking around on the web interface, and the "unit testing" component is testing the PHP functions and passing in and out as many data as possible. You pick a specific function, and turn the other functions into mock functions that return mock data so that you can test the individual functions. So SimpleTest can do both functional testing and unit testing.

All SimpleTests should DrupalTestCase class. The get_info() is called to provide a simple description. A test function, always start off with the word "test"
Enable modules in order to ensure that the testing is configured.

SimpleTest captures the state of the database, and returns the db state after it is finished. It'll clean up the created users as well. The internal browser can emulate a POST request, and pass though an array of data similar to via a form, and you can choose the SAVE button -- or anything that that the user can see in the web interface.

Assertions are made in order to check the results of the actions to be sure that the text changes have been made. You can check to see that test exists, that it doesn't exist or if it's NULL, and if the proper response header comes back.

It will only check to see if the exact words show up on the response page. And so it's checking the correct interface strings, but also the correct database info, and there are other ways for asserting.

SimpleTest lacks support for javascript, and you can't really test the visual interface, which isn't the intention of SimpleTest

References: SimpleTest, SimpleTest Automator, http://www.SimpleTest.org, http://www.drupal.org/SimpleTest

http://www.lullabot.com/articles/introduction-unit-testing
http://www.lullabot.com/articles/drupal-module-developer-guide-SimpleTes...
http://www.starbowconsulting.com/blog/tao/SimpleTest-tips

ANGIE BYRON: Let's show a more extensive example located here: bio.test

bio.test extends the Drupal test case. SimpleTest has assertions to test input and output, and the WebTestCase makes it Drupal Specific.

Another limitation of SimpleTest is that it acts directly on the database you're working on. So Don't use SimpleTest on a production site

You can do table prefixing to prevent that, which is currently broken.

drupalCreateNode is a handy helper function.
drupalCreateContentTypes
drupalGet is how you retrieve a URL
ClickLink will actually click link on a link.
drupalRandomName to create a random username
You can choose your database prefix
Module enable and disable
drupalVariableSet and drupalVariableGet
drupalCreateUserRolePerm creates a role with combo of permissions
SimpleTest has a SetUP and a TearDown.

Let's look at SimpleTest in Action:
You need SimpleTest module & the sourceforge repository of SimpleTest library, which is LGPL.

Let's look at the SimpleTest admin page, and some of the return messages are a bit obtuse, but you can change the text output to be more specific. When you start with working tests, and you change some code which breaks tests, then you have some detailed info for where to look. You can also run a subset of SimpleTests since it takes a while to run the entire suite.

There is a patch in the queue where you can run SimpleTest from the command line.

So let's look at why we use testing for the bio.module, which creates a node per user profile.There are lots of questions like when you delete user, does it delete the bio node? And so these SimpleTests allow for a lot of confidence that code changes doesn't break the functionality.

BioTest extends DrupalTestCase -- there is a DrupalTestCase class, and we're using that and doing something various different alterations to it.
The "parent::setUp()" is used for initialization.
Setting a variable to bio, and a set of permissions to do various different stuff.
We're not doing anything with tear down in this specific case.
Just start the function with test, and now let's take a look at the testBioCreatesBasicUserBio() function...
bioCreateBio() -- automatically creates the node...

You should think through what functionality your module has to go through, and then start writing your tests for each part of that functionality. And then if you make changes, it's easier for you to keep track of what you're module is doing both before and after you make any changes.

Clicking on the browser-based UI interface is what a SimpleTest looks like from a functional perspective. Let's take a look at what a unit test would be for a simple function like check_plain(), which is in bootstrap.inc
You can iterate between passing in HTML, false, text, etc. and then see what comes back. So a simple one-line function can have something like 12 different unit tests associated with it just trying to check all of the various different functionality.

The function url() has lots of inputs, and a good rule of thumb is to cycle through all of the IF or SWITCH statements, and try to figure out what it would take to check the functionality of each condition.

Drupal needs a lot of help in order to get unit test coverage, and we get extra development time and an extended code freeze time. But we need an army of people to write tests to get full test coverage.

QUESTION: So would you need 72 different tests testing all of the permutations?
Yeah, we sort of need to be as comprehensive as possible, but CHX suggests that we start where the bugs are. Where ever Drupal functionality tends to break and have bugs filed against it, then that's a good place to start writing unit tests.

The SimpleTest documentation is really awesome, but for the most part writing a test is a matter of copy and pasting something similar for what you want to test.

The Search.module has some really good unit tests written by Steven Wittens located in the search_match.test file.

He is doing a number of search queries to test the entire search functionality that it does in Drupal. He does a search for all of the crazy edge cases documented by creating the node content in _setup(), then putting the data in search index tables with the search_index function. The basic idea of SimpleTests is to first create the environment, do the search, and then test the search results with the assertEqual function.

For unit testing, SimpleTest is really handy for making sure that your regular expressions are doing what you want them to be.

By doing all of the tests, they've created all sorts of book nodes, story nodes and other stuff in the core tests. So it's a great place to look at for when you're starting to write more tests.

QUESTION: Is there a link where people can look at for a checklist for everything that they can do?
No, but a summary checklist would be helpful, and please help out by creating an issue with all of the things that you found helpful in learning how to create them.

The SimpleTest documentation has a list of what you can do and what you can assert, but the whole SimpleTest tutorial is really helpful.

Tao Starbow suggests creating a URL for a specific test instead of filling out the form and confirming over and over again, which is a lot faster.

QUESTION: Drupal during code thaw, then Drupal goes through many, many different API changes. And any change to the interface with also break the APIs. Then how's the best way to deal with it?
Not sure. It's difficult.
But that's also exactly why we doing testing, because if you break the tests then you should check to see why. If new tests need to be written, then you should write the test first. If you write the test first, and then write out the code until all of the tests all pass, which is quite a culture-shift from the current way Drupal development works more towards test-driven development.

QUESTION: PHPdocs are so well integrated in the IDEs, but if you're making changes to the API, then why not make changes to the PHPdocs at the same time?

QUESTION: In ideal world, then the tests would be running non-stop, and that you'd notified when they break. Also if you write something that's not targeted to specific bugs, then it can take up a lot of time to test everything.

QUESTION: What if the coder.module could help create and stub out SimpleTest code?
It would be awesome if coder.module could help make some stub SimpleTest code.
But there is a tool called PHPcoverage that does something similar. It tells you which lines were checked by the test, and so you know where you might have holes in what you need to unit test.

ROK ŽLENDER: http://testing.drupal.org is an automated patch testing framework to help to patch submitters and committers filter out which patches are worth reviewing. "TDO" was the work of Now Public.

How does it work?
Drupal.org submits patches, and then it's submited to testing.drupal.org, which does a clean install of Drupal, and then applies the patch and runs tests, and then the test server reports back the results. Check out http://testing.drupal.org for more details.

Looking at http://testing.drupal.org/node/137 -- it saves the result from the test, and in this case something failed. It also saves the test site so that you can check it out to see why something failed. There are still some problems with it since a clean Drupal install is failing a number of SimpleTests at the moment.

QUESTION: If you make a change so that you break the current suite of unit tests, but you include new tests as well, then how would that work?
This was one of the main concerns that Dries had. You should submit a patch to fix both things, first the new test and then the changed Drupal code. But this still needs to be worked out.

QUESTION: How long does it take?
It currently takes about 5 minutes to run through the whole cycle of installing a freshing install, clean testing, and then patch the code, test the changes, and then pass back the results. But you can scale out a farm of testing servers as well.

When taking a look at all of the core modules, they currently only have about 43% coverage for core functional test, and only about 1% unit test coverage for core. Jimmy wrote about half of the functional tests, and there are only about 6 people who have written all of the tests. So we need a lot more help to get full test coverage.

KÁROLY NÉGYESI: Let's take a look at how to get SimpleTest into core, and the tentatively approved to get it in.
It is very unlikely that any other framework other than SimpleTest will be able to do it because we need browser-based testing

[Shows a Picture of Darth Vader] Testing: I find your lack of tests disturbing

Testing brings in real corporate money. We have outgrown the size of the community to maintain NOT having tests. Webchick and Catch were doing a lot of testing, and it'll soon be practically impossible to run through function testing for ALL of Drupal core.
We will be bringing SimpleTest to core, but it's HUGE in size-- bigger than drupal itself.
They will start to ship the unit and functional tests with core, and a slim framework enabled by a core SimpleTest module to ship with core. There may need to have additional LGPL code to download to have it fully work.

Functional testing are browser-based tests. There are a lot of tests, there are articles, and now the automator module that are out there. So it's easy to write one, but we just need peple to come out and write them.

But with Unit Testing, Drupal has more of a problem here.
Unit testing works by separating out a unit of code, and replacing all of the other functions with a simple stub with mock data that emulates what should ideally be returned. But we cannot replace FilterXSS with a mock call that returns data, and so we can not do that.

Unit testing can be enabled by a RunKit PHP extension that has to be compiled into your PHP. So this means that only a few people will be able to run these tests because you have to install this special extension.

* SimpleTest is really good at documentation, and so learn to write tests.
* Don't break tests, if you do, then fix them.
* Need to add more tests -- expect to see a pyramid of people who are able to fix the existing tests, and then other people will write more.

We will continue to write browser-based tests, and there is a code sprint on Friday.
Our goals are that:
* Unit tests and functional tests shipped with core
* Slim framework for unit tests shipped with core
* functional (browser ) test runner in contrib
* Runkit?

Go get involved in the Unit Test group at http://groups.drupal.org/unit-testing

Tests are a long-term investment, and if they're hard to write, you may not see the return right away.

QUESTION: Drupal 5.4 shouldn't have existed with SimpleTests, a bug was fixed, but introduced a critical bug, and that would have never happened with proper unit and functional testing.

QUESTION: So is there going to be a SimpleTest junior in Core?
Yes, absolutely the smallest part will be included -- starting with the SimpleTest module.

QUESTION: Does it make sense to include tests in release download, or strip out into directories?
They will be including them in the tarball in order to make people realize that it's part of core. Even though it increases the size, we're all right with it. If they're only in contrib, then it's harder for people to realize that they're actually a part of Drupal. Storage space is cheap. Shipping the entire framework is way too much, but they don't see a problem with the size considering the importance of testing.

We can lengthen the development cycle and shorten the testing cycle with full unit test coverage.

QUESTION: What is RunKit?
RunKit is a PHP extension that lets you replace a function with another function in run-time, so that you can replace functions with very small stubs with known data so that you can isolate individual units of code. And there isn't another way to really do it.

When you start with index.php, then you go into the execute_menu_handler, and then drills down. We'll also have to shift our APIs around so that they're easier to test, which is a big, big win.

QUESTION: What is drush?
Allows you to run tests via the command line without running Drupal in the browser. Accepts some input from the command line, like which tests to include.

QUESTION: What about using the Selenium IDE to run tests that include javascript?
Selenium is a bit of a problem. There is the Selenium in IDE that only runs in Firefox, but our javascript doesn't usually break in FF, but usually in IE, Opera and Safari. So the Selenium IDE isn't really usable for anything.

QUESTION: There is an alternative to Selenium, called Watir -- http://wtr.rubyforge.org
It's promising, but not yet have a large adoption rate. It's worth investigating and looking into more.

QUESTION: Is it possible to package the entire LAMP stack of the testing platform -- complied with RunKit -- so that we can do it locally?
Look, this just came together within the last 3 days. So, the fact that it is getting into core was decided in Dries keynote, and got a tentative yes from Dries on Tuesday morning. So it's still very early.

QUESTION: What's the benefit for contributed module developers to do testing?
One of the secret plans is to put some type of indicator on contributed module pages for how much test coverage you have -- which is an indicator of quality.
You can reuse core tests on your own module tests. More people can also trust your work and modules.