| Testing Classes and Libraries with Randoop

Testing Classes and Libraries with Randoop

Tests are an important and valuable part of the software development cycle. However, it can be difficult and time consuming to write an extensive unit test suite for an entire library. Furthermore, random testing is sufficient for security testing of an entire system, but is not well suited for unit tests. Unit tests are designed to test a small component or unit such as a single method. Random testing has significant shortcomings in testing methods that take structured data as input. Recall that fuzzers create random sequences of bytes to feed to an input program. The chances of a random sequence producing a valid structured data type as an argument is extremely low. Using random testing to inspect a single unit would be inefficient as most inputs would be immediately thrown away as invalid.

Randoop, the Random Tester for Object-Oriented Programs, is better suited for testing Java units such as methods, classes, and libraries. Randoop is an automated test generation tool that generates sequences of valid method calls and checks for violations in invariants such as the reflexive property of equality. That is, an object should always be equal to itself. Randoop can also check for violations in custom written invariants and pre/post conditions.

The details of the Randoop algorithm are left to the Automated Test Generation Module. In this reading, we will present a walkthrough of running Randoop on a Java Graphing Library. To follow along, you can download the necessary files here.

We will use Randoop to generate error-revealing test cases for the Java library, analyze these test cases to find errors in the source code, fix them, and verify that the errors have been corrected with successive iterations of the tool.

First, use javac to compile the Java files to class files:

$ javac Line.java Parabola.java Point.java Vector.java

If the compilation was successful, you should see .class files for each Java program.

Now, let’s make a list of the class files we would like Randoop to target. Add the prefix of each of our Java files to a text file.

$ cat myclasses.txt
Line
Parabola
Point
Vector

Now we are ready to use Randoop! Generate tests using the following command:

$ java -classpath {my_dir}:./randoop-all-3.0.8.jar randoop.main.Main gentests --classlist=myclasses.txt --no-regression-tests=true

Specifying the location of resources needed to run Randoop can be tricky. The easiest way to accomplish this is to run the command to execute Randoop in the directory where you downloaded the library source code with the classpath option.

You should see ErrorTest*.Java files containing the failing test cases. Let’s open ErrorTest0.Java to see the failing method sequences Randoop has generated.

Of course, Randoop is randomized, so each reader’s ErrorTest file may look different. However, you should see failing test cases regarding the reflexivity and transitivity of equals on various library objects.

Let’s compile and run these tests to see the crashes.

Compile the tests using the following command:

$ javac -classpath .:{my_dir}/junit-4.12.jar ErrorTest*.java -sourcepath .:{my_dir}

In order to run the tests, it makes things easier to create an environment variable to refer to the necessary jar files:

$ export JUNITPATH=junit-4.12.jar:hamcrest-core-1.3.jar

Now, we are ready to run the failing test cases:

$ java -classpath .:$JUNITPATH:{my_dir} org.junit.runner.JUnitCore ErrorTest

You should see various errors including Class Cast Exceptions, and Null Pointer Exceptions. Each of these failing test cases should have a stack trace specifying the location in the corresponding library files.

Let’s inspect the stack traces and fix some of these errors.

Randoop generates sequences of methods using a feedback loop. It starts with a short sequence and continues to expand with method calls from the input class in each iteration. If the sequence is illegal or redundant it is thrown away and not expanded upon. So, it makes sense to start fixing errors with the smallest sequence first.

In this run of Randoop, the first failing test, test001, is:

  @Test
  public void test001() throws Throwable {
	Point point0 = null;
	Point point1 = null;
	Line line2 = new Line(point0, point1);
    
	// Checks the contract:  line2.equals(line2)
	org.junit.Assert.assertTrue("Contract failed: line2.equals(line2)", line2.equals(line2));
  }

With the crashing stack trace:

java.lang.AssertionError: Contract failed: line2.equals(line2)
    at org.junit.Assert.fail(Assert.java:88)
    at org.junit.Assert.assertTrue(Assert.java:41)
    at ErrorTest0.test001(ErrorTest0.java:21)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...

Because this test case is so small, we can deduce that the error is likely in the Line constructor:

public Line(Point p1, Point p2)
{
   	 if (p1 != null)
   	 {
   		 this.point1 = new Point(p1.x,p1.y);
   	 }
   	 if (p2 != null)
   	 {
   		 this.point2 = new Point(p2.x,p2.y);
   	 }
}

In particular, the crashing test case occurs when the Points p1 and p2 are null. Fix the Line constructor to handle null Points by removing the if statements. We now have:

public Line(Point p1, Point p2)
{
   	this.point1 = new Point(p1.x,p1.y);
   	this.point2 = new Point(p2.x,p2.y);
}

Let’s recompile Line.java and run Randoop again to generate new test cases:

$javac Line.java 
$java -classpath {my_dir}:./randoop-all-3.0.8.jar randoop.main.Main gentests --classlist=myclasses.txt --no-regression-tests=true

Now that we have new ErrorTest*.java files, let’s compile them and run them to inspect the crashing stack traces:

$ javac -classpath .:{my_dir}/$JUNITPATH ErrorTest*.java -sourcepath .:{my_dir}
$ java -classpath .:$JUNITPATH:{my_dir} org.junit.runner.JUnitCore ErrorTest

Note that by fixing the bug in the Line constructor, the number of failing test cases dropped from 340 to 273.

In our second run of Randoop, the first failing test, test001, is:

public void test001() throws Throwable {
	Point point2 = new Point(0.0d, (double)(byte)0);
	Point point5 = new Point(0.0d, (double)(byte)0);
	Line line6 = new Line(point2, point5);
    
	// Checks the contract:  **line6.equals(line6)**
	org.junit.Assert.assertTrue("Contract failed:     line6.equals(line6)", line6.equals(line6));
}

With the crashing stack trace:

 java.lang.AssertionError: Contract failed: line6.equals(line6)
    at org.junit.Assert.fail(Assert.java:88)
    at org.junit.Assert.assertTrue(Assert.java:41)
    at ErrorTest0.test001(ErrorTest0.java:21)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...

Remember that the order of your test cases may not be exactly as shown due to the inherent randomness in Randoop. This error is more subtle and requires some digging. In particular, take a minute to look at the equals method for Line.java.

Did you find the error? Both of our points are non-null. Based on these if statements, equals will return false.

 public boolean equals(Object o) {
   	 if (o == null) return false;
   	 if (!(o instanceof Line)) return false;
   	 Line that = (Line) o;

   	 // Check point 1
   	 if( !(this.point1 == null && that.point1 == null) ){
   		 return false;
   	 }
   	 ....

   	 // Check point 2
   	 if( !(this.point2 == null && that.point2 == null) ){
   		 return false;
   	 }
   	....

   	 // return true if no equality invalid
   	 return true;
     }

We can fix the error by replacing the condition using an “exclusive or” operator. If exactly one of the points we are comparing is null, we know that they cannot be equal and should return false.

if (this.point1 == null ^ that.point1 == null) {

We should also take care to consider the case in which both points are null. In this case, the equals function should not return false. Add a condition to ensure this case is handled properly.

Once you’ve fixed the bug, recompile Line.java, rerun Randoop, and compile and run the crashing test cases.

$javac Line.java 
$java -classpath {my_dir}:./randoop-all-3.0.8.jar randoop.main.Main gentests --classlist=myclasses.txt --no-regression-tests=true
$javac -classpath .:{my_dir}/$JUNITPATH ErrorTest*.java -sourcepath .:{my_dir}
$java -classpath .:$JUNITPATH:{my_dir} org.junit.runner.JUnitCore ErrorTest

Note that by fixing the bug in Line.equals the number of crashing test cases created by Randoop was reduced from 273 to 29.

In our third run of Randoop, the first failing test, test01, is:

  public void test01() throws Throwable {
	Point point2 = new Point(0.0d, (double)(byte)0);
	Point point5 = new Point(0.0d, (double)(byte)0);
	Line line6 = new Line(point2, point5);
	Vector vector8 = new Vector(point2, (java.lang.Integer)(-1));
    
	// This assertion (transitivity of equals) fails
	org.junit.Assert.assertTrue("Contract failed: equals-transitive on vector8, point5, and point5.", !(vector8.equals(point5) && point5.equals(point5)) || vector8.equals(point5));

 }

With the crashing stack trace:

java.lang.ClassCastException: class Point cannot be cast to class Vector (Point and Vector are in unnamed module of loader 'app')
    at Vector.equals(Vector.java:43)
    at ErrorTest0.test01(ErrorTest0.java:22)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Notice that this test case is very similar to the previous test001. In fact, the first three lines are identical. This is expected as Randoop generates new test cases by expanding previously generated test cases with additional method calls.

From the stack trace, we can deduce that the error is on line 43 of Vector.java (marked with a comment):

public boolean equals(Object o) {
   	 if (o == null) return false;
   	 Vector that = (Vector) o; //line 43

   	 if(this.origin == null || that.origin == null)
   		 return this.origin == null && that.origin == null && this.degrees == that.degrees;
   	 else
   		 return this.origin.equals(that.origin) && this.degrees == that.degrees;
}

Let’s fix this by adding a check for if o is an instance of Vector prior to performing the cast:

public boolean equals(Object o) {
   	 if (o == null || !(o instanceof Vector)) return false; //cast here!
   	 Vector that = (Vector) o;

   	 if(this.origin == null || that.origin == null)
   		 return this.origin == null && that.origin == null && this.degrees == that.degrees;
   	 else
   		 return this.origin.equals(that.origin) && this.degrees == that.degrees;
}

Edit both conditions to fix the bug, recompile Vector.java, rerun Randoop, and compile and run the crashing test cases. Now, the number of crashing test cases produced by Randoop is only 6. We are almost done!

In our fourth run of Randoop, the first failing test, test1, is shown here. At this point, the sequence of generated method calls has become quite long. We have shortened it here to show the relevant calls.

public void test1() throws Throwable {
	Point point0 = null;
	Point point1 = null;
	Line line2 = null;
	Parabola parabola3 = new Parabola(point0, point1, line2);
	...
	parabola3.focus = point4;
	...
	Point point35 = new Point(0.0d, (double)(byte)0);
	...
	parabola3.vertex = point35;
	...
    
	// Checks the contract:  equals-hashcode on parabola3 and parabola3
	org.junit.Assert.assertTrue("Contract failed: equals-hashcode on parabola3 and parabola3", parabola3.equals(parabola3) ? parabola3.hashCode() == parabola3.hashCode() : true);

  }

With the crashing stack trace:

java.lang.NullPointerException
    at Parabola.hashCode(Parabola.java:118)
    at ErrorTest0.test1(ErrorTest0.java:47)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

The failing contract here is that if two parabolas are equal, their hash codes must also be equal. So, let’s take a look at the hashCode method to see if anything looks suspicious.

public int hashCode() {
      final int prime = 17;
      int result = 1;
      result = prime * result + ((focus == null) ? 0 : focus.hashCode());
      result = prime * result + ((vertex == null) ? 0 : focus.hashCode());
      result = prime * result + 
                    ((directrix == null) ? 0 : directrix.hashCode());
      return result;
}

Once you’ve found the bug, fix it so the method returns the correct hashCode.

Now, let’s execute Randoop with a final long running iteration to confirm there are no additional bugs to be found. You can specify a time limit in seconds using the --timelimit option. Here, we choose five minutes as a conservative limit.

$java -classpath {my_dir}:./randoop-all-3.0.8.jar randoop.main.Main gentests --classlist=myclasses.txt --no-regression-tests=true --timelimit=300

After five minutes, Randoop should report there are “No error-revealing tests to output”.

In summary, Randoop can be used to automatically generate unit test cases to test important properties. These properties can be default intrinsic properties such as the ones we tested in this article, or user specified invariants. Randoop reduces the burden on the developer to create high quality unit tests. Imagine how many test cases we would have had to write to obtain this level of coverage! Furthermore, when testing small units that take structured input as data, Randoop provides a superior technique to randomized testing. Because of this, Randoop is a useful tool used in industry at companies such as ABB, Microsoft, and on many open-source projects.

Elizabeth Dinella

Testing Classes and Libraries with Randoop