DocTest 1.0 For Ruby Released

by Sebastien Auvray on Jun 23, 2008 |
Included in the Python standard library, various DocTest Ruby implementations were made available starting one year ago by Tom Locke, Roger Pack, and more recently Dr Nic.
DocTest allows the easy generation of tests based on output from the standard interpreter shell (IRB for Ruby), cut and pasted into docstrings.

Copy/paste your IRB testing output as comment for your function:
# doctest: Add 5 and 5 to get 10
# >> five_and_five
# => 10
def five_and_five
  5 + 5
end
Later you'll be able to run the test:
$ rubydoctest five.rb
=== Testing 'five.rb'...
 OK  | Add 5 and 5 to get 10
1 comparisons, 1 doctests, 0 failures, 0 errors
This will run with latest sources from the git repository. An updated gem will soon be available

Such docstring-driven testing has its pros:
  • Those who are familiar with IRB live testing will find it useful and natural to use.
  • One place to look at for the test.
  • Simple.
... And cons:
  • Large docstring should be avoided and put into a separate test file.
  • Some complex testing assertion are not available.
  • It can be tedious to test large outputs.
Duane Johnson merged his changes on Tom Locke's repository to make version 1.0 available.

A screencast demonstrating DocTest usage is available.

InfoQ caught up with Duane Johnson to discuss DocTest and docstring-driven tests.

InfoQ: Firstly, what led you to write DocTest? I read you looked at 2 previous implementations.

Duane Johnson: I am currently working on an improvement to a large ruby project (memorypress.com) that has very little testing and documentation.  In order to make this a clean upgrade, we will definitely need some kind of testing in place to make sure we don't break things unintentionally.  The trouble is, most of my time "testing" is spent at the irb console, because it's not always clear what things do and how I should go about testing them.  In other words, testing to me is a kind of exploratory thing--I need to see how things work at the same time that I formally define how things work.  Ruby DocTest is a perfect combination since I can copy and paste my successful experiments into the codebase and check off the tests as complete.

The latest code can be found on github? It's a merge with a previous version of DocTest? What did you add?

Yes, that's the latest code.  Earlier this week, I forked Tom Locke's project at github.com/tablatom/rubydoctest, and then merged my changes into his project once he gave permission.  I found Tom's code to be very readable, and his approach to comparing results by evaling (rather than direct string testing, as in Python's doctests) was compelling.  This allows ruby's unordered hashes to be compared successfully, for example.  I also took Roger Pack's idea of putting doctests into comments right in the ruby source file.  I didn't borrow any of Roger's code (at code.google.com) but I borrowed his ideas.  Now, Ruby DocTest gives the option of both the "inline doctest" style of Roger's package, with the "separate documentation in a markdown file" style of Tom's package.

Aren't you satisfied with all the Ruby test frameworks?

Yes and no.  I still like rspec quite a bit.  But for my own workflow, I always find that I'm trying to catch up with the actual code written because I'm too lazy to switch to another file (or create a file) and write the tests.  Often, I have this little niggling feeling that I should test some edge case of a method I'm working on, but I forego it because so many other things are on my mind at the time.  If I could just put an irb session in there, I thought, that would cover 90% of my needs.  So I did.

DocTest is outside the BDD cycle?

Yes, that's correct.  This is not a test-first approach.  My current needs, as mentioned above, are in documenting an already-existing project that is not well-tested.

Is DocTest supposed to be a replacement for the other unit test framework or just an additional tool?

To be honest, I don't know yet.  It's a new way of writing tests for me, so I'll be playing with it to find the answer soon.  I imagine my 90% target above is close... there will probably be some places where it makes more sense to rely on traditional rspec or Test::Unit cases.

Isn't the way DocTest annotates code polluting the reading?

I imagine the migration path of doctest to go something like this:

1. You write some code, and then add an inline doctest comment or two.
2. You make some other code modifications, and later come back to the original code.
3. You add more test cases, but the number of tests (or length) gets unwieldy, so you move them into a separate *.doctest file.
4. Eventually, you approach a fairly complete set of markdown files with code examples as documentation, with instructions and comments all around them.

One of the interesting side-effects of writing tests side-by-side with code is that it tends to encourage you to make each ruby source file independent--in other words, you want to make it possible to "eval" the file without any dependency errors, since that's what the doctests require.  I find that once I unshackle a file from its complex network of hidden assumptions (as in a Rails file) then it becomes easier to test, and I'm more aware of what each file is relying on.

Related Editorial

Hello stranger!

You need to Register an InfoQ account or to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

DocTests in Zope and Python by Kevin Teague

Doc testing is fun! :)



Doctesting has been a common form of testing in Zope for a long time (around 2001 or so). The merits and good uses of doc testing were recently discussed in this thread. The summary is it's important to seperate the use cases for doc testing:




  • * Executable documentation doctesting: Intended as "testable" documentation. Documentation comes first with this form of doc test. Outdated or wrong documentation can be rather annoying, so this is good. The risk is that it's easy to go of on a tangent and start testing corner cases - this spoils the readability of the documentation. Narrative doc tests are also harder to maintain, since you may need to comprehend the whole narrative to expand upon one.


  • * Edge-case and bug fix doctesting: In this case the narrative between the
    test cases is there primarily to explain what is being tested and why. Some
    developers like to bang out doc tests as they work, writing down thoughts and snippets of code as they go.



This second form can produce files that are also semi-usable as documentation, and are often preferable when approaching software over reading source code. Sometimes the software you are developing doesn't justify the cost of proper documentation, so this type of doc testing is "better than nothing". The flipside is that this may inhibit initiatives at writing proper documentation, "Look at all these files with words in them that this package already has, I guess it doesn't need anymore documentation." And in an open source project, people can find a packages test-centric doc tests and end up complain about the poor quality of documentation ...




Packages such as z3c.testsetup can be directed to more sophisticated automated detetion of doctests. Aside from picking doc tests out of source code, it also can run plain .txt files as doc tests - these files and the testing layer they are associated with (unit, functional, integration) are denoted with a line such as:



:Test-Layer: functional


or



:Test-Layer: unit

Re: DocTests in Zope and Python by Sebastien Auvray

Hi Kevin,
Thank you for this informative comment!
Sébastien.

1.0 Gem Available by Duane Johnson

Thanks for publishing this, Sebastien. The 1.0 gem has been made available since we last spoke:

sudo gem install rubydoctest


Enjoy!

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

3 comments

Educational Content

General Feedback
Bugs
Advertising
Editorial
 InfoQ.com and all content copyright © 2006-2013 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy