Tests are an important and valuable part of the software development cycle. However, it can be difficult and time consuming to write an extensive unit test suite for an entire library. Furthermore, random testing is sufficient for security testing of an entire system, but is not well suited for unit tests. Unit tests are designed to test a small component or unit such as a single method. Random testing has significant shortcomings in testing methods that take structured data as input. Recall that fuzzers create random sequences of bytes to feed to an input program. The chances of a random sequence producing a valid structured data type as an argument is extremely low. Using random testing to inspect a single unit would be inefficient as most inputs would be immediately thrown away as invalid.

Randoop, the Random Tester for Object-Oriented Programs, is better suited for testing Java units such as methods, classes, and libraries. Randoop is an automated test generation tool that generates sequences of valid method calls and checks for violations in invariants such as the reflexive property of equality. That is, an object should always be equal to itself. Randoop can also check for violations in custom written invariants and pre/post conditions.

In this reading, we will present a walkthrough of running Randoop on a Java Graphing Library. To follow along, you can download the necessary files here.

We will use Randoop to generate error-revealing test cases for the Java library, analyze these test cases to find errors in the source code, fix them, and verify that the errors have been corrected with successive iterations of the tool.

First, use javac to compile the Java files to class files:

$ javac Line.java Point.java 

If the compilation was successful, you should see .class files for each Java program.

Now, let’s make a list of the class files we would like Randoop to target. Add the prefix of each of our Java files to a text file.

$ cat myclasses.txt
Line
Point

Now we are ready to use Randoop! Generate tests using the following command:

$ java -cp .:randoop-all-4.3.3.jar randoop.main.Main gentests --classlist=myclasses.txt --no-regression-tests=true --time-limit=60 --randomseed=42

Specifying the location of resources needed to run Randoop can be tricky. The easiest way to accomplish this is to run the command to execute Randoop in the directory where you downloaded the library source code with the cp option.

You should see ErrorTest*.Java files containing the failing test cases. Let’s open ErrorTest0.Java to see the failing method sequences Randoop has generated.

Of course, Randoop is randomized, so each reader’s ErrorTest file may look different. However, you should see failing test cases regarding the reflexivity and transitivity of equals on various library objects.

Let’s compile and run these tests to see the crashes.

Compile the tests using the following command:

$ javac -cp .:junit-4.12.jar ErrorTest*.java 

In order to run the tests, it makes things easier to create an environment variable to refer to the necessary jar files:

$ export JUNITPATH=junit-4.12.jar:hamcrest-core-1.3.jar

Now, we are ready to run the failing test cases:

$ java -cp .:$JUNITPATH org.junit.runner.JUnitCore ErrorTest 

Let’s inspect the stack traces and fix some of these errors.

Randoop generates sequences of methods using a feedback loop. It starts with a short sequence and continues to expand with method calls from the input class in each iteration. If the sequence is illegal or redundant it is thrown away and not expanded upon. So, it makes sense to start fixing errors with the smallest sequence first.

In this run of Randoop, the first failing test, test001, is:

    @Test
    public void test0001() throws Throwable {
        if (debug)
            System.out.format("%n%s%n", "ErrorTest0.test0001");
        Point point2 = new Point((java.lang.Double) 0.0d, (java.lang.Double) 1.0d);
        java.lang.Double double3 = point2.y;
        point2.y = 1.0d;
        point2.x = 100.0d;
        Point point10 = new Point((java.lang.Double) 0.0d, (java.lang.Double) 1.0d);
        Line line11 = new Line(point2, point10);
        org.junit.Assert.assertTrue("Contract failed: line11.equals(line11)", line11.equals(line11));
    }

Take a look at the Line equals method and see if you can find the bug. Once you find it, fix it in Line.java, recompile, and re-run Randoop.
You might now see a failing test case asserting "Contract failed: equals-hashcode". Remember the contract from lecture. If two objects are equal, their hash codes should also be equal. Try and fix this bug too.

Once you've fixed the bugs let’s execute Randoop with a final long running iteration to confirm there are no additional bugs to be found. You can specify a time limit in seconds using the --time-limit option. Here, we choose five minutes as a conservative limit.

$java -cp .:./randoop-all-4.3.3.jar randoop.main.Main gentests --classlist=myclasses.txt --no-regression-tests=true --timelimit=300

After five minutes, Randoop should report there are “No error-revealing tests to output”.

In summary, Randoop can be used to automatically generate unit test cases to test important properties. These properties can be default intrinsic properties such as the ones we tested in this article, or user specified invariants. Randoop reduces the burden on the developer to create high quality unit tests. Imagine how many test cases we would have had to write to obtain this level of coverage! Furthermore, when testing small units that take structured input as data, Randoop provides a superior technique to randomized testing. Because of this, Randoop is a useful tool used in industry at companies such as ABB, Microsoft, and on many open-source projects. CS383 students: save your fixed Line and Point classes for submission.