@tenderlove asked about the wisdom of teaching RSpec to new Ruby developers. I have some relevant experience. Here it is, for what it's worth.

experience_teaching_ruby_testing.md

Notes on teaching both test/unit and RSpec to new Ruby developers

@tenderlove asked "Is it good to teach RSpec (vs t/u) to people who are totally new to Ruby?" I have experience suggesting that it is a good thing; after a short back and forth, it seemed useful to write it up in detail.

Background

This goes back several years, to when I was the primary Ruby/Rails trainer for Relevance from 2006-2009. I'm guessing that worked out to probably 6-8 classes a year during those years. Since then, RSpec has changed a fair amount (with the addition of expect) and test/unit has changed radically (it has an entirely new implementation, minitest, that avoids some of the inconsistencies that made test/unit a bit confusing during the time I'm writing about here).

I started out as an RSpec skeptic. I've never been afraid of what a lot of people denigrate as "magic" in Ruby libraries … to me, if you take the trouble to understand it, that stuff's just programming—it's just the way Ruby works. But I've also known Ruby and test/unit since 2001, and I definitely had a little bit of old man "it was good enough for me, it should be good enough for you" attitude going on. So I sneered at RSpec quite a bit. And while some at Relevance agreed with me, there were RSpec fans there, ranging from vehement (one thought RSpec was obviously the way everyone would be doing things in the future) to pragmatic (Chad Humphries spent Thanksgiving one year writing micronaut—which went on to become the core of RSpec 2.0—just to get over the "RSpec is too slow" objection so people would let him use RSpec on projects).

As a teacher representing our company, I wanted to emphasize test/unit, but I felt responsible to present both sides, in case a) I was wrong, or b) the students ended up being our clients, having to work with us on a project where we were using RSpec.

Teaching both test/unit and RSpec

So I decided I would teach both. I can't remember the exact details, but IIRC I would teach test/unit on day 2 of a four-day Ruby or Rails class, and then after continuing with some of the main topic, I'd present RSpec as "a popular alternative" on day 3. And I noticed a consistent pattern: they would really start to get it when I got to RSpec. It wasn't dramatic, but it was noticeable … they wrote better tests, and struggled less, with RSpec.

Naturally, I wondered whether this was just because it was the second time around. They'd already had a learning experience with test/unit; maybe that (plus a good night's sleep) was setting them up for success with RSpec the next day.

So I started doing it the other way around. I would teach RSpec on day 2 and test/unit on day 3. To my surprise, RSpec was still easier for people to grasp. They caught on fairly quickly, but still struggled to write good tests with test/unit. Again, it wasn't a dramatic difference: there were still struggles with RSpec, and successes with test/unit. But the difference was noticeable in the students' work, and also in their reactions. I heard things like "RSpec just makes more sense." (Yes, I really heard that a lot!)

But what conclusions can we really draw from this? Perhaps I just had a knack for teaching RSpec, and my long familiarity with test/unit blinded me to some of the problem areas, so that I didn't teach test/unit as well. Maybe the mix of students had something to do with it … or perhaps the big changes in test/unit since then make all of this irrelevant.

How I Teach

The variable I know the most about is the fact that I was the teacher in all of these classes. So it's worth talking a bit about how I teach.

I have always thought it was a mistake to use abstraction to hide the way things work underneath. Abstraction is useful because it means you don't have to pay attention to all the details all of the time. But you get in trouble if you try to pretend those details don't exist, or think you don't need to understand them at all.

When I'm confronted with a "magic", DSL-ish Ruby API, my first impulse is to open the source and learn a little bit about how the magic works. And I teach things that way, too. I'll often show how to write the basic code without the "magic" library, and then show how to gradually abstract the details away until you end up with the fancy, magic interface. That provides the proper foundation for using the interface successfully.

For example: I hate it when APIs (or languages, or whatever) are presented as "it's just English!" That doesn't give anyone anything useful to work with; it's just trying to allay fears, and it replaces a mythical danger (the thing people are afraid of simply because it's unknown) with a real danger: you're telling them they don't need to learn anything, when in fact the opposite is true.

So while I acknowledge that there may be some value in the "english-like-ness" of RSpec, that value doesn't mean you don't need to learn the syntax. And I worked hard to build a solid foundation for that, trying to distill the essentials of RSpec down to the simplest possible core. I explained the difference between expectations (should, should_not) and matchers (==, have, be, etc.). I drilled them on the syntax ("Repeat after me: 'dot should space match …' OK, and now 'dot should underscore not space match …'"). I showed them the fully parenthesized versions of those expressions. I gave a sketchy (but realistic) overview of how RSpec implements its expectations and matchers. And I just can't recall anyone having serious trouble with it, which really surprised me.

And I did a similar thing with test/unit, drilling people on the "expected comma actual" ordering, explaining why that was important, explaining the way the method naming pattern worked, etc.

Caveats / Things to Note

• Many of these students were new to unit testing … it was clear to me that RSpec helped students to get over the hump with learning about testing, but it wasn't clear that someone who already got testing did better with RSpec. (But it wasn't clear that they did worse, either.)

• The original implementation of test/unit had its own quirks and inconsistencies. One was that positive and negative assertions came in pairs, and it was common for library authors to forget the assert_not version; some of the Rails custom assertions only came in the positive variant. By comparison, RSpec's clean separation of expectations and matchers, with should and should_not as the positive and negative expectation methods, eliminated that inconsistency, which was a big help. Today's minitest, with its paired assert and refute methods, is a big improvement (although I personally detest "refute" as a part of the testing vocabulary).

• I know that many people were repulsed by RSpec's insertion of the should method in every object. But it had the advantage of making the expected/actual distinction a bit more intuitive. expected.should == actual doesn't seem quite right; it makes much more sense to say actual.should == expected. That seems like a very subtle point, but in the classes I taught, that ordering was never a problem for students, whereas test/unit's assert_equal expected, actual is arbitrary, and frequently caused confusion (not to mention erroneous, misleading failure messages). RSpec's new expect(actual).to == expected seems slightly worse to me (in that respect), but doesn't seem to cause too many problems; minitest still uses the arbitrary ordering that you simply have to be aware of. To be fair, it quickly becomes second nature, but we're talking here about how easy things are for the newcomer.

• RSpec did start out as an experiment, and the developers tried some very bold ideas before settling down a bit and backing off of some things. There were definitely parts of RSpec that went a bit too far, and many parts of RSpec have been deprecated, removed, or extracted into separate gems to keep the basics a little more reasonable.

• Many objections to RSpec rest on its supposed complexity: it has APIs and methods that you have to learn for things that Ruby just provides naturally! Why not just use classes and methods?

  The problem, as I see it, is that there's a lot of hidden complexity in test/unit that we experienced Ruby developers are just used to, but that often catches new programmers by surprise. The way test/unit uses classes and methods isn't always obvious. What's that method naming convention again? What happens if I lapse back to my Java ways and write a method called "testSomething"? (It's silently ignored.) How does test/unit find those methods? Can I write a module full of test methods and mix that into a test class? (Yes, but it's not a dumb question, because there was a time when that didn't work; the framework must be written so as to make that work, and it's not a foregone conclusion that it was written that way.) And there's more like this. Learning curves can be complex; it's not nearly as clear-cut as "that's simple, this is complex" or "that's easy, this is hard" or even "that's intuitive, this is obscure".

I switched to RSpec on my projects, after we tried it out (in parallel) in a project that already has a lot of test/unit tests. The reason I switched was that the developers wrote BETTER, CLEARER tests in RSpec than they did in test/unit. It was universal.

Like you, I was in the "test unit is all you need" camp for a long time. But I've recently come around, for reasons that you hint towards the end of your piece: building tests and examples explicitly using the same structures as we build applications - classes, modules, methods - ends up being both more confusing for new people ("what happens if I include a module here? Is that a good thing to do?"), and more limiting (speaking from experience, it's much much much harder to implement behaviour that needs to happen once before all the tests, for example).

I came to this conclusion while I was writing a couple of pieces about how MiniTest and Rspec work behind the scenes, in case they are of interest.

http://interblah.net/how-minitest-works
http://interblah.net/how-rspec-works

Thanks, James! Those pieces are great, and I agree with you about the way tests map to classes and instances in t/u-style frameworks. It's just not a very good match.

When I'm confronted with a "magic", DSL-ish Ruby API, my first impulse is to open the source and learn a little bit about how the magic works. And I teach things that way, too.

Makes sense until someone asks "How does actual.should == expected work?" Explaining that to someone who knows OO but is new to Ruby reminds me of the Carl Sagan quote "If you wish to make an apple pie from scratch, you must first invent the universe." How many concepts in Ruby are baked in to just that line?

Off the top of my head:

1. Monkey patching (where does should come from?)
2. Method overloading (how does == work?)
3. Monads

To someone who knows OO, I can explain how assert_equal works in less than a tweet. In fact, I will do it for you now: "assert_equal is a method that checks that a is equal to b and records the result". Can you explain the RSpec line in fewer characters?

For the test unit case, I need to make sure they understand:

1. What is an object
2. What is a method

The number of concepts I need to make sure they understand in order to understand actual.should == expected seems staggering in comparison.

I'm not saying "people shouldn't use RSpec", I'm saying that it's not appropriate to teach to a person new to Ruby. Start them with objects, methods, and test/unit style. Once they grasp blocks, monkey patching, etc, then they're ready to handle RSpec.

The problem, as I see it, is that there's a lot of hidden complexity in test/unit that we experienced Ruby developers are just used to, but that often catches new programmers by surprise.

Links please?

Can I write a module full of test methods and mix that into a test class? (Yes, but it's not a dumb question, because there was a time when that didn't work;

It's worked since at least 2008 (from Ruby 1.8.0):

require 'test/unit'

module MyTest
  def test_zomg
    assert_equal "I'm a module", "yes you are"
  end
end

class MyTestClass < Test::Unit::TestCase
  include MyTest
end

Output:

[aaron@higgins test-unit-1.2.3]$ ruby -I lib omg.rb 
Loaded suite omg
Started
F
Finished in 0.009331 seconds.

  1) Failure:
test_zomg(MyTestClass) [omg.rb:5]:
<"I'm a module"> expected but was
<"yes you are">.

1 tests, 1 assertions, 1 failures, 0 errors

What's that method naming convention again? What happens if I lapse back to my Java ways and write a method called "testSomething"?

When I'm using RSpec, how can I refactor to a method? Is it OK to use def? If so, where does it go? Once I define the method, how do I call it? Can I nest a describe? How is it different from a class? How does it inherit? Can I use tests from outer describes? What is let? What is let!? How are they different from a method?

Learning curves can be complex; it's not nearly as clear-cut as "that's simple, this is complex" or "that's easy, this is hard" or even "that's intuitive, this is obscure".

I agree with this. But the concepts I need to teach someone to use test/unit style are just OO concepts (which they need to learn anyway). Once they're up on OO stuff, then they can do Ruby specific things. This is not true with RSpec.

I'll answer some of these points in detail, but first:

For the most part, I agree with you. It is harder to explain how RSpec works. It is more complex. You have to learn more extra things—and some pretty complex things!—to use RSpec. My analysis of the situation, like yours, suggests that it's not a good idea to teach RSpec to new Ruby programmers.

But my experience suggests, contrary to my expectations, that it is a good idea. As I said, probably somewhere between 15 and 25 four-day classes, spread over a period of three years. People from different companies, with different backgrounds. And consistently, students liked RSpec better, found it easier to get the hang of testing with RSpec, and wrote better tests with RSpec.

To me, that's by far the most interesting question here. We both agree on the analysis, but my experience strongly suggests that the analysis is wrong. So what are we both missing? The parts you quoted are mostly just me fumbling around looking for an answer to that question.

I encourage you to try it and see what your experience is like. We might learn something.

With that said: a few of your points:

When I'm confronted with a "magic", DSL-ish Ruby API, my first impulse is to open the source and learn a little bit about how the magic works. And I teach things that way, too.

Makes sense until someone asks "How does actual.should == expected work?"

Ah … the way I wrote that was a bit misleading. I do explain how the syntax breaks down, and what the syntax rules are (mostly by showing the fully parenthesized version). But if someone asked me that, I'd put them off for a bit, and get back to the question on the last day of the class.

Can you explain the RSpec line in fewer characters?

No, but "fewer characters" is a fairly silly constraint. I would just say something like "The should method uses the == operator to check that actual is equal to expected and record a test failure if not."

Links please?

I don't have links … it's just my observation from the kinds of questions people would sometimes ask in those classes. I sincerely wish I'd written them down.

It's worked since at least 2008

Well, yes. I think it's worked since about 2003. I'm not saying it's bitten anyone recently, except maybe that some people wouldn't even try it because they assume that the lookup is done with instance_methods(false) instead of instance_methods(true). (Not that they're likely to know those particulars, of course.)

I mentioned this just as an example of one of those things you were just asking for links about: things experienced Ruby programmers know about test/unit and minitest that maybe aren't as obvious as we think they are.

What's that method naming convention again? What happens if I lapse back to my Java ways and write a method called "testSomething"?

When I'm using RSpec, how can I refactor to a method? Is it OK to use def? If so, where does it go? Once I define the method, how do I call it? Can I nest a describe? How is it different from a class? How does it inherit? Can I use tests from outer describes? What is let? What is let!? How are they different from a method?

You're responding as if I'd said "test/unit is the complex one; RSpec is easy!" But that's not my point at all. Believe me, I know quite well that RSpec raises all of those questions.

My point is simply that maybe the gap is not as wide as we tend to think; maybe it's narrow enough that some of RSpec's good qualities make up for it. That's one possible explanation for the results I saw.