ERT

presentation.org

ERT: Emacs Lisp Regression Testing

Why should you care

“Software is easy to write, hard to debug, impossible to maintain.”

Automatically make sure that you didn’t break stuff by writing tests.

Emacs lisp in particular:

• ancient
• quirky
• no types
• many contributors

Simple example

(defun silly-division (a b)
  "Silly divide A by B."
  (if (equal b 1)
      a
    (/ a b)))

1. divide a by b
2. if b is 1, spare ourselves the computation and return a

Simple test

(require 'ert)

(ert-deftest silly-test-division ()
  (should (equal 4 (silly-division 8 2))))

Check that 8/2 = 4

How to run it

M-x ert and select it, OR

(ert 'silly-test-division)

Then inside the debugging editor you can:

• “TAB” to move around
• ”.” to jump to code
• “b” for backtrace
• “l” to show all should statements

Testing multiple cases

(ert-deftest silly-test-division ()
  (should (equal 4 (silly-division 8 2)))
  (should (equal 2 (silly-division 10 5)))
  (should (equal 10 (silly-division 10 1)))
  (should (equal 2 (silly-division 8 4))))

(ert 'silly-test-division)

Testing for error

(ert-deftest silly-test-division-by-zero ()
  (should-error (silly-division 8 0)
                :type 'arith-error))

(ert 'silly-test-division-by-zero)

Testing for failure

Identify known bugs and help contributors fix things

We have a problem with integer division:

(silly-division 1 2)

Let’s write a test for it:

(ert-deftest silly-test-float-division-bug ()
  :expected-result :failed
  (should (equal .5 (silly-division 1 2))))

(ert 'silly-test-float-division-bug)

Tricky code

(defun silly-temp-writer (str)
  (with-temp-buffer
    (insert (format "*%s*" str))
    (write-file "/tmp/silly-write.txt")))

1. take a string
2. format it
3. write that string to “/tmp/silly-write.txt”
4. doesn’t return anything
5. how do we test this??

What do we actually want to test?

We want to make sure that what ends up being written to the file is what we expect

Naive method

(ert-deftest silly-test-temp-writer ()
  (silly-temp-writer "my-string")
  (should (equal "*my-string*"
                 (with-temp-buffer
                   (insert-file-contents "/tmp/silly-write.txt")
                   (string-trim (buffer-string))))))

(ert 'silly-test-temp-writer)

1. call silly-temp-write-function with string “my-string”
2. read the content of “/tmp/silly-write.txt”
3. remove the new line
4. compare it to expected result ”my-string”

Better approach with mocking

(require 'cl-macs)

(ert-deftest silly-test-temp-writer ()
  (cl-letf (((symbol-function 'write-file)
             (lambda (filename)
               (should (equal "/tmp/silly-write.txt" filename))
               (should (equal "*my-string*" (buffer-string))))))
    (silly-temp-writer "my-string")))

(ert 'silly-test-temp-writer)

1. Define a mock write-file function
   • check that we write in the correct location
   • check that the content is formatted
2. Temporarily replace the real write-file with our mock
3. Call silly-temp-writer

Advantages

• Not actually writing to the system or leaving state
• Test more and closer to the intended behavior
• Not testing something we didn’t intend to (id. the write-file function)

Running all tests at once

Test selector

(ert "silly-test-*")

Visualizing coverage

Interactively see which lines are covered by your tests and how well

1. M-x testcover-start
2. Select silly.el
3. Run tests

   (ert "silly-test-*")
       

4. M-x testcover-mark-all
5. See results
   • red is not evaluated at all
   • brown is always evaluated with the same value

Best Practices

• ask yourself what you want to test
• start with failing tests
• clean test with no side effect
• descriptive in test name
• good tests for good debugging

Resources

https://www.gnu.org/software/emacs/manual/html_mono/ert.html https://www.gnu.org/software/emacs/manual/html_node/elisp/Test-Coverage.html