Commit 86a67763a2421d84117c037d459c75fe9f52b824

Waylan Limberg
Mon Feb 15 05:59:37 CET 2010

Tree SHA1: aeb6c5d
Parent SHA1: 67ad9f6 (Added run-tests.py script and added a helpful error message if nose is not installed. Note the run-tests script will be used by simply running 'nosetests' from the commandline, so the hack in 'tests/__init__.py' has been removed.)
raw diff | raw patch

Added documentation for the test suite.

Diff rendering mode:
inline
side by side


		1	# Test Suite
		2
		3	Python-Markdown comes with a test suite which uses the [Nose][] testing
		4	framework.The test suite primarily serves to ensure that new bugs are not
		5	introduced as existing bugs are patched or new features are added. It also
		6	allows Python-Markdown to be tested with the tests from other implementations
		7	such as John Gruber's [Perl][] implementation or Michel Fortin's [PHP][]
		8	implementation.
		9
		10	The test suite can be run by calling the `run_tests.py` command at the root of
		11	the distribution tarball or by calling the `nosetests` command directly. Either
		12	way, Nose will need to be installed on your system first (run `easy_install
		13	nose`). Any standard nosetests config options can be passed in on the command
		14	line (i.e.: verbosity level or use of a plugin like coverage).
		15
		16	Additionally, a nicely formatted HTML report of all output is written to a
		17	temporary file in `tmp/test-output.html`. Open the file in a browser to view
		18	the report.
		19
		20	The test suite contains three kinds of tests: Markdown Syntax Tests, Unit
		21	Tests, and Doc Tests.
		22
		23	# Markdown Syntax Tests
		24
		25	The Syntax Tests are in the various directories contained within the 'tests'
		26	directory of the packaged tarball. Each test consists of a matching pair of txt
		27	and html files. The txt file contains a snippet of Markdown source text
		28	formated for a specific syntax feature and the html file contains the expected
		29	HTML output of that snippet. When the test suite is run, each txt file is run
		30	through Markdown and the output is compared with the html file as a separate
		31	Unit Test.
		32
		33	In fact, this is the primary reason for using Nose, it gives us an easy way to
		34	treat each of these tests as a separate unit test which is reported on
		35	separately. Additionally, with the help of a couple custom Nose plugins which
		36	are included with the Markdown Test Suite, we are able to get back an easy to
		37	read diff of the actual output compared to expected output when a test fails.
		38
		39	Here is some sample output with a test that is failing because of some
		40	insignificant white space differences:
		41
		42	$ ./run-tests.py
		43	..........................................................M...........
		44	............................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
		45	SSSSSSSSSS.................S..........................................
		46	.........
		47	======================================================================
		48	MarkdownSyntaxError: TestSyntax: "misc/lists3"
		49	----------------------------------------------------------------------
		50	MarkdownSyntaxError: Output from "/home/waylan/code/python-markdown/te
		51	sts/misc/lists3.txt" failed to match expected output.
		52
		53	--- /home/waylan/code/python-markdown/tests/misc/lists3.html
		54	+++ actual_output.html
		55	@@ -1,5 +1,5 @@
		56	<ul>
		57	<li>blah blah blah
		58	-sdf asdf asdf asdf asdf
		59	-asda asdf asdfasd</li>
		60	+ sdf asdf asdf asdf asdf
		61	+ asda asdf asdfasd</li>
		62	</ul>
		63
		64	----------------------------------------------------------------------
		65	Ran 219 tests in 7.698s
		66
		67	FAILED (MarkdownSyntaxError=1, SKIP=53)
		68
		69	Note that 219 tests were run, one of which failed with a `MarkdownSyntaxError`.
		70	Only Markdown Syntax Tests should fail with a `MarkdownSyntaxError`. Nose then
		71	formats the error reports for `MarkdownSyntaxError`s so that they only include
		72	useful information. Namely the txt file which failed and a unified diff showing
		73	the failure. Without the plugin, you would also get a useless traceback showing
		74	how the code stepped through the test framework, but nothing about how Markdown
		75	actually ran.
		76
		77	If, on the other hand, a Syntax Test failed because some other exception gets
		78	raised by either Markdown or the test suite, then that would be reported as per
		79	a normal unit test failure with the appropriate traceback for debugging
		80	purposes.
		81
		82	### Syntax Test Config Settings
		83
		84	The other thing to note about the above example is that 53 tests were skipped.
		85	Those tests have been explicitly configured to be skipped as they are primarily
		86	tests from either PHP or Perl which are known to fail for various reasons. In
		87	fact, a number of different configuration settings can be set for any specific
		88	test.
		89
		90	Each Syntax Test directory contains a `test.cfg` file in the ini format. The
		91	file may contain a separate section for each txt file named exactly as the file
		92	is named minus the file extension (i.e.; the section for a test in `foo.txt`
		93	would be `[foo]`). All settings are optional. Default settings for the entire
		94	directory can be set under the `[DEFAULT]` section (must be all caps). Any
		95	settings under a specific file section will override anything in the
		96	`[DEFAULT]` section for that specific test only.
		97
		98	Below are each of the config options available and the defaults used when they
		99	are not explicitly set.
		100
		101	* `extensions`: A comma separated list of extensions to use with the test.
		102	Defaults to an empty list.
		103	* `safe_mode`: Switches safe_mode on and off. Defaults to `False` (off).
		104	* `output_format`: Defines the expected output_format. Defaults to `xhtml1`.
		105	* `normalize`: Switches whitespace normalization of the test output on or off.
		106	Defaults to `0` (off). Note: This requires that [uTidylib] be installed on
		107	the system. Otherwise the test will be skipped, regardless of any other
		108	settings.
		109	* `skip`: Switches skipping of the test on and off. Defaults to `0` (off).
		110	* `input_ext`: Extension of input file. Defaults to `.txt`. Useful for tests
		111	from other implementations.
		112	* `output_ext`: Extension of output file. Defaults to `.html`. Useful for tests
		113	from other implementations.
		114
		115	## Unit Tests
		116
		117	All Unit Tests shipped with Python-Markdown are standard Python Unit Tests and
		118	are currently all contained in `tests/test_apis.py`. Standard discovery methods
		119	are used to find and run the tests. Therefor, when writing new tests, those
		120	standards and naming conventions should be followed.
		121
		122	## Doc Tests
		123
		124	Some Python-Markdown extensions also include standard Python doctests, which
		125	are discovered and run in the standard manner; one Unit Test for each file.
		126
		127
		128	[Nose]: http://somethingaboutorange.com/mrl/projects/nose/
		129	[Perl]: http://daringfireball.net/projects/markdown/
		130	[PHP]: http://michelf.com/projects/php-markdown/
		131	[uTidylib]: http://utidylib.berlios.de/

Commit 86a67763a2421d84117c037d459c75fe9f52b824

Gitorious