Numerous TDD experts have made the suggestion that there should only be one assertion per test:
- One Assertion Per Test by Dave Astels
- Write Maintainable Unit Tests That Will Save You Time and Tears by Roy Osherove
The theory goes that each test should only test one thing, and that should be the name of the test.
I don’t do a very good job of following that rule. If I wanted to test a method that returned books for an author I would write tests like so:
1 [Test]
2 public void GetBooksForAuthorSuccessfull()
3 {
4 List<Book> books = AuthorDA.GetBooks(1);
5 Assert.IsNotNull(books, “Null value returned for books”);
6 Assert.IsTrue(books.Count > 0, “No books returned”);
7 Assert.AreEqual(books.Count, 3, “Incorrect number of books returned”);
8 Assert.AreEqual(books[1].ID, 1, “First book returned is incorrect”);
9 Assert.AreEqual(books[2].ID, 2, “Second book returned is incorrect”);
10 Assert.AreEqual(books[3].ID, 3, “Third book returned is incorrect”);
11 }
12
13 [Test]
14 public void GetBooksForAuthorNoBooksReturned()
15 {
16 List<Book> books = AuthorDA.GetBooks(122);
17 Assert.IsNotNull(books, “Null value returned for books”);
18 Assert.AreEqual(books.Count, 0, “Books are returned when they shouldn’t be”);
19 }
20
21 [Test]
22 [ExpectedException(typeof(ArgumentException))]
23 public void GetBooksForAuthorBadAuthorID()
24 {
25 AuthorDA.GetBooks(-1);
26 }
I have three different tests that each test different things, but two of them contain more than one assertion and one of them contains seven assertions. If I wanted to write these tests using the one assertion per test guideline I would have this:
1 [Test]
2 public void GetBooksForAuthorSuccessfullNotNull()
3 {
4 List<Book> books = AuthorDA.GetBooks(1);
5 Assert.IsNotNull(books, “Null value returned for books”);
6 }
7
8 [Test]
9 public void GetBooksForAuthorSuccessfullCountGreaterThanZero()
10 {
11 List<Book> books = AuthorDA.GetBooks(1);
12 Assert.IsTrue(books.Count > 0, “No books returned”);
13 }
14
15 [Test]
16 public void GetBooksForAuthorSuccessfullCountEqualsThree()
17 {
18 List<Book> books = AuthorDA.GetBooks(1);
19 Assert.AreEqual(books.Count, 3, “Incorrect number of books returned”);
20 }
21
22 [Test]
23 public void GetBooksForAuthorSuccessfullFirstBookReturnedCorrect()
24 {
25 List<Book> books = AuthorDA.GetBooks(1);
26 Assert.AreEqual(books[1].ID, 1, “First book returned is incorrect”);
27 }
28
29 [Test]
30 public void GetBooksForAuthorSuccessfullSecondBookReturnedCorrect()
31 {
32 List<Book> books = AuthorDA.GetBooks(1);
33 Assert.AreEqual(books[2].ID, 2, “Second book returned is incorrect”);
34 }
35
36 [Test]
37 public void GetBooksForAuthorSuccessfullThirdBookReturnedCorrect()
38 {
39 List<Book> books = AuthorDA.GetBooks(1);
40 Assert.AreEqual(books[3].ID, 3, “Third book returned is incorrect”);
41 }
42
43 [Test]
44 public void GetBooksForAuthorNoBooksReturnedNotNull()
45 {
46 List<Book> books = AuthorDA.GetBooks(122);
47 Assert.AreEqual(books.Count, 0, “Books are returned when they shouldn’t be”);
48 }
49
50 [Test]
51 public void GetBooksForAuthorNoBooksReturnedCountGreaterThan0()
52 {
53 List<Book> books = AuthorDA.GetBooks(122);
54 Assert.AreEqual(books.Count, 0, “Books are returned when they shouldn’t be”);
55 }
56
57 [Test]
58 [ExpectedException(typeof(ArgumentException))]
59 public void GetBooksForAuthorBadAuthorID()
60 {
61 AuthorDA.GetBooks(-1);
62 }
I have a couple problems with this change:
- I think it’s actually harder to read since the assertions are scattered around in separate methods.
- It would increase the number of tests. On my current project we have 1800 tests, if we followed the one assertion rule we would have over 6,000 I am sure.
- If my method breaks and starts returning null then I have 8 tests failing instead of just 2, this means I have to know the dependency tree of my tests to find the real issue.
- Any code I have to write to setup the data for my test has to be duplicated 8 times. (if I move that setup data to the setup method than I am effectively limiting my fixtures to one fixture per test)
- I now have over double the amount of code. I am constantly trying to reduce the amount of code in my project, whether test or production, and anything that doubles it better add a ton of value.
The main drawback to having multiple asserts in a test is that all of the unit testing frameworks I have used fail the test on the first assert that fails. This means that if my first test fails on the third assert (line 82), the remaining assertions are never run:
76 [Test]
77 public void GetBooksForAuthorSuccessfull()
78 {
79 List<Book> books = AuthorDA.GetBooks(1);
80 Assert.IsNotNull(books, “Null value returned for books”);
81 Assert.IsTrue(books.Count > 0, “No books returned”);
82 Assert.AreEqual(books.Count, 3, “Incorrect number of books returned”); ‘ This Fails
83 Assert.AreEqual(books[1].ID, 1, “First book returned is incorrect”); ‘ Never Run
84 Assert.AreEqual(books[2].ID, 2, “Second book returned is incorrect”); ‘ Never Run
85 Assert.AreEqual(books[3].ID, 3, “Third book returned is incorrect”); ‘ Never Run
86 }
This is the main reason Roy gives in his MSDN article on why you should limit tests to a single assert. My idea is that we should have an attribute we could use to tell the framework that we want it to run all the asserts, something like this:
76 [MultipleAssertTest]
77 public void GetBooksForAuthorSuccessfull()
78 {
79 List<Book> books = AuthorDA.GetBooks(1);
80 Assert.IsNotNull(books, “Null value returned for books”, false);
81 Assert.IsTrue(books.Count > 0, “No books returned”, false);
82 Assert.AreEqual(books.Count, 3, “Incorrect number of books returned”, true);
83 Assert.AreEqual(books[1].ID, 1, “First book returned is incorrect”, true);
84 Assert.AreEqual(books[2].ID, 2, “Second book returned is incorrect”, true);
85 Assert.AreEqual(books[3].ID, 3, “Third book returned is incorrect”, true);
86 }
The new test type would allow me to add an additional parameter to all of my assertions that tells the harness whether or not it should continue with the test. If either of the first two asserts fails I want to abort and not evaluate the other assertions since they will all fail. The last four assertions are not dependent on each other so I want to continue running the rest of the assertions in my test if one of them fails.
You would then need to be able to see each of the failure in the GUI, something like a tree would work:
+ GetBooksForAuthorSuccessfull() Failed
– 2 Failures
* Incorrect number of books returned
* Third book returned is incorrect
This would give me the benefits of one assertion per test without the additional code or huge increase in the number of tests. This would be a great feature for MbUnit.
-James
{ 4 comments }
One thing you might take a look at, is in the first test, building up the expected collection as its own List<Book> and then just assert the two collections are equal. This may or may not make things more readable, but would reduce the number of asserts.
In general, though, I agree that it would be nice if asserts after the fail were still run.
I also tend to not follow the ‘one assert per test’ thing unless it really does make sense for that particular test.
it seems MBUnit is far better suited for what you want
…and commenting in your blog sucks really bad in IE7
Hi James,
Although not quite what you describe here, MbUnit can allow you to pump in a range of selected data to a single assert.
http://www.mertner.com/confluence/display/MbUnit/CombinatorialTestAttribute
Here you can make use of the combintional and factory fixtures to define the data used for testing across an asssert. The row testing that you have shown here before allows you to create a range of tests between values. The combintional test lets you define your test values based on the object your testing. It’s not quite what your looking for but might help.
Andy
RE: My Unit Testing Idea : Facilitating Multiple Assertions
Pingback from One Assert vs. Multiple Asserts « Clackwell’s Weblog
Comments on this entry are closed.