What if we write the rendering code ourselves? This may be the case if we write our own ray tracer or GPU shaders. In that case, we actually need to verify that the rendered pixels are correct.
Bitmap-based reference testsTDD-style test-first development is not easy to do in this case, and it may not even be possible. The test result itself is much more complex than a trivial number or string result. How can we write a test that validates a complex image without having the rendered image up front?
In this case, it may be more convenient to write the rendering code first and then use a rendered image as a reference image for the test. We will loose some of the benefits of doing proper TDD, but those tests will still act as regression tests that verify that future enhancements do not introduce defects.
The production code and tests will thus be written like this:
- Write production code that renders to a bitmap
- Verify the reference image manually
- Write a test that compares future rendering results with this reference image
- Fail the test if they differ
Allow for slight differences
- Single pixel offset
- Small differences in color or intensity
- Render image
- For each pixel in the rendered image, find the pixel in the 3x3 neighbouring pixels in the reference image with the lowest deviation in RGB pixel value. Add this deviation to a list.
- Create a histogram of all the pixel deviations so that you can calculate the distribution of the errors.
- Decide on an error threshold for acceptable differences. For example, say that
- A maximum of 0.1% of the pixels can have a larger RGB deviation than 50
- A maximum of 2% of the pixels can have a larger RGB deviation than 10
- A maximum of 20% of the pixels can have a larger RGB deviation than 3