Autogenerated tests


Writing tests sucks. You have to painstakingly set up test data, record inputs and outputs, and lay out your expectations for the app’s behavior. It’s absolutely necessary, but it sucks. How can we make this process better?

Selenium has the right idea, in a way – you record behavior in the app, set your expectations, and bingo bango, you’ve got a test. However, it has its flaws:

  1. It’s slow. Spinning up a browser, even a headless browser, takes time, and each test can take upwards of 2 seconds. Multiply this by hundreds or thousands of tests and it becomes untenable.

  2. It’s too high-level sometimes. You can’t test individual low-level modules with Selenium.

What I want is a way to record interactions with an app and record the inputs and outputs of each function in the call stack, filter it down to code the user wrote i.e. exclude external modules, and “canonize” an interaction and a set of its inputs and outputs as a test. If the constraints I define are ever not met by the code, the developer will be alerted (possibly in a CI workflow).

A few questions:

  1. What are the problems with the idea I’ve described?
  2. How would you change this system?
  3. Would you use such a system?