Skip to content

FAT tests

Jump to bottom Edit New page
Scott Kurz edited this page Feb 2, 2018 · 28 revisions

Table of Contents

Overview

The majority of OpenLiberty tests are Functional Acceptance Tests (FAT). A FAT test project ends in _fat and executes tests with a running OpenLiberty image. This differs from Unit tests because Unit tests do not execute on a running OpenLiberty image.

The example FAT project is build.example_fat and can be used as a minimalistic reference point for writing new FATs.

TCK projects

Note that TCK projects follow a slightly different layout. A TCK project ends in _fat_tck and require mvn to be installed when run locally. See here for more information on creating and running a TCK project.

FAT Project Layout

- com.ibm.ws.my_fat/
  - fat/src/            # Root folder for test client side
    - com/ibm/ws/my/
      - FATSuite.java   # Entry point of the FAT.  Lists one or more test classes via '@SuiteClasses'
      - MyTest.java     # Test class containing a group of tests.  Server and app lifecycle is controlled here.
  - test-applications/  # Root folder for all test applications
    - myApp/            # Folder name indicates name of application (by convention)
      - resources/      # Optional: Static resources such as web.xml are included here for 'myApp'
      - src/            # Java code for 'myApp' goes here
  - publish/            # Root folder for non-java resources
    - files/            # Files needed by test client go here.  Such as alternate server.xml configurations
    - servers/          # Root folder for OpenLiberty servers
      - myServer/       # Folder representing a server.  This folder gets copied to ${wlp.install.dir} at runtime
        - server.xml    # Server configuration for 'myServer'
  - build/libs/autoFVT/ # Root folder for a single execution of FAT output
    - output/           # Root folder for server output
    - results/          # Root folder for test client output
      - junit.html/     # Contains JUnit-style test results which can be viewed in a web browser
      - output.txt      # Log file for test client (i.e. fat/src/ classes)
  - bnd.bnd             # Describes source folders and compile-time build dependencies
  - build.gradle        # Build file.  Must include "apply from: '../cnf/gradle/scripts/fat.gradle'"

How to run an existing FAT

To run a FAT, you must have ant on your $PATH.
To test this, you can run the command which ant. If ant is not on your path, you can download ant here and then add export PATH=$ANT_HOME/bin:$PATH to your ~/.bashrc. After adding ant to your path, run ./gradlew --stop to stop the Gradle daemon so that it can be started with the updated $PATH.

To run an entire FAT, use the command:

./gradlew <fat_project_name>:buildandrun

For example:

./gradlew build.example_fat:buildandrun

The buildandrun task will build and run the FAT. If you are not making changes to the FAT and just want to re-run the FAT without rebuilding it, you can use the runfat task instead.

To run only a single test class in a FAT, use the command:

./gradlew <fat_project_name>:buildandrun -Dfat.test.class.name=<fully_qualified_java_test_class>

For example:

./gradlew build.example_fat:buildandrun -Dfat.test.class.name=com.ibm.ws.example.SimpleTest

Overview of how a FAT test runs

When a FAT runs there are two or more JVMs in use:

  • A test-client side JVM which manages server lifecycle, deploys apps, and drives individual test cases on the server side
  • An OpenLiberty server with zero or more test apps running on it.

Most FATs use a test client to drive HTTP GET requests on a test servlet within a test application. The steps for this style of FAT test is the following.

  1. Build packages everything needed to run the FAT into an autoFVT.zip archive
  2. On an automated build, the autoFVT.zip for each FAT is built on the parent build machine. Then the autoFVT.zip archives are distributed evenly among about a dozen child build engines where they will be extracted and run. This allows us to pack about 36 hours of FAT tests into about 3 hours.
  3. Build unzips autoFVT.zip and starts a JUnit process with the class named FATSuite as the entry point. Each test class in the @SuiteClasses will be run in sequence.
  4. Test class runs
    1. Test class @BeforeClass runs. Typically applications are deployed to server and then server is started here.
    2. Test client runs each test case in class in sequence. Typically each test case is a single HTTP GET. For example GET http://localhost:8010/myApp/MyTestServlet?testMethod=myTest
      1. Test application receives HTTP GET request via a Servlet. For example, inside MyTestServlet.doGet() the testMethod parameter is checked, and reflection is used to invoke a method indicated by the parameter value, so MyTestSevlet.myTest() is invoked.
      2. If the test case passes, the test servlet prints a well-known success message string to the HTTPServletResponse. If the test case fails, the test servlet does not print the success message, and instead dumps the stack trace of the failure to the HTTPServletResponse.
      3. Test client reads HTTPServletResponse. If response contains well-known success message, the test passes. If the success message is missing, fail the test case with the HTTPServletResponse content as the failure message.
    3. Test class @AfterClass runs. Typically server is stopped here. When server is stopped, the server logs are copied from ${wlp.install.dir} into build/libs/autoFVT/output/servers/<server_name>-<stop_timestamp>

Using the fattest.simplicity infrastructure

The FAT test infrastructure is housed in the fattest.simplicity project. This contains all of the junit extensions and extra bells and whistles available to FAT projects.

Enabling the simplicity infrastructure

To get most of the custom value-adds of the simplicity infrastructure, test classes must indicate @RunWith(FATRunner.class) on the test class. For example:

@RunWith(FATRunner.class)
public class SimpleTest { /* ... */ }

Controlling server lifecycle with LibertyServer

The LibertyServer class gives the test client a POJO representation of an OpenLiberty server. The easiest way to obtain an object instance is with the @Server("<server_name>") annotation. With this object, you can do things like start and stop a server.

An example usage typically looks like:

@RunWith(FATRunner.class)
public class MyFATTest {

  @Server("MyServer")
  public static LibertyServer server;

  @BeforeClass
  public static void setup() throws Exception {
    server.startServer();
  }

  @AfterClass
  public static void tearDown() throws Exception {
    server.stopServer();
  }

}

Building and deploying test applications with ShrinkWrap

Typically FATs build application artifacts at FAT runtime using an open source library called ShrinkWrap, and then export the artifacts to disk. The ShrinkWrap API is quite flexible and can build many different types of archives including: .zip .jar .war .ear .rar

The official ShrinkWrap javadoc can be found here: ShrinkWrap API 1.2.3 API

The ShrinkWrap API can get a little verbose, so we have a ShrinkHelper class to make conventional operations more concise. A few useful ones are:

  • ShrinkHelper.defaultApp(LibertyServer s, String appName, String... packages)
    Builds an artifact named <appName>.war which includes resources from test-applications/<appName>/resources/ and the specified java packages. Also exports the .war to the indicated LibertyServer's ${server.config.dir}/apps/ directory.
  • ShrinkHelper.defaultDropinApp(LibertyServer s, String appName, String... packages)
    The same thing as ShrinkHelper#defaultApp except it exports the .war to the ${server.config.dir}/dropins/, which means no server.xml configuration is needed for the application.
  • ShrinkHelper.buildDefaultApp(String appName, String... packages)
    The same thing as ShrinkHelper#defaultApp except the archive is not exported to disk anywhere.
  • ShrinkHelper.exportAppToServer(LibertyServer s, Archive<?> a)
    Exports the archive to disk in the ${server.config.dir}/apps/ directory. The name of the exported file will be a.getName().
  • ShrinkHelper.exportToServer(LibertyServer s, String path, Archive<?> a)
    Same thing as ShrinkHelper#exportAppToServer except you can indicate a path to export the archive at (relative to ${server.config.dir} of the indicated server).
  • ShrinkHelper.addDirectory(Archive<?> a, String path)
    Adds a directory (and all sub-dirs) to the indicated archive.

Server-side JUnit and Servlet template with componentTest-1.0

There is a test-only feature in Liberty called componentTest-1.0 which will provide some common utilities for FAT test apps.

  • If the componentTest-1.0 feature is enabled in server.xml, then JUnit will be made available to test applications. This is useful for being able to do server-side assertions.
  • If the componentTest-1.0 feature is enabled in server.xml, the FATServlet class is made available to test applications. This class contains boiler-plate HttpServelt code for invoking test methods.

Declaring test methods on the server side using @TestServlet

Without @TestServlet
With standard JUnit a "test stub" would be needed on the client side for each server-side test. For example:

// Client side
public class MyTest {

  // get and start server in @BeforeClass

  @Test
  public void myTest() throws Exception {
    // invoke GET http://loclahost:8010/myApp/MyTestServlet?testMethod=myTest and assert response is successful
    FATServletClient.runTest(server, "/myApp/MyTestServlet", "myTest");
  }
}

// Server side
@WebServlet("/MyTestServlet")
public class MyTestServlet extends FATServlet {
  public void myTest() throws Exception {
    // run some server-side test
  }
}

This is inconvenient because the client side code is completely boiler plate and if the server side test moves or is renamed it requires a corresponding update on the client side class.

With @TestServlet
To eliminate this, we have the @TestServlet annotation which will scan the referenced test servlet class for @Test methods and add synthetic client-side test stubs at runtime. So the example above becomes:

// Client side
@RunWith(FATRunner.class)
public class MyTest {
  @Server("MyServer")
  @TestServlet(servlet = MyTestServlet.class, contextRoot = "myApp") // References class gets scanned for @Test methods
  public static LibertyServer server;

  // start server in @BeforeClass
}

// Server side
@WebServlet("/MyTestServlet")
public class MyTestServlet extends FATServlet {
  @Test // Able to use standard JUnit @Test annotation in test servlet
  public void myTest() throws Exception {
    // run some server-side test
  }
}

Debugging a FAT

In a typical FAT there are two or more JVMs (client driver, and a server).

To debug the test client, use the -Ddebug.framework option and attach a debugger on port 6666. For example:

./gradlew build.example_fat:buildandrun -Ddebug.framework

If you are using the Eclipse IDE, you can attach a debugger by selecting the debug configuration Liberty-FAT remote debug from the debug menu dropdown. If you've never used this debug configuration, it will not appear in the dropdown until you've visited Debug Configurations... from the Debug dropdown. Look for 'Liberty-FAT remote debug' under 'Remote Java Application'.

To debug a liberty server, use the -Ddebug.server option and attach a debugger on port 7777. For example:

./gradlew build.example_fat:buildandrun -Ddebug.server

If you are using the Eclipse IDE, you can attach the debugger by selecting Liberty- server remote debug from the debug menu dropdown. See above if this item doesn't appear in your debug dropdown.

FAT test modes

A FAT project should run in 5 minutes or less, with each test case running in under 15 seconds. However, sometimes it is necessary to write tests that are important to run, but also time intensive.

The simplicity test framework has an @Mode annotation for this purpose. There are two primary modes: LITE and FULL, with LITE being the default mode. If a test method does not define a @Mode annotation, it will be considered a LITE mode test. For example:

public class MyTest {
  @Test
  public void regularTest() { } // No @Mode annotation, so LITE mode is assumed

  @Test
  @Mode(LITE)
  public void liteTest() { } // functionally equivalent to 'regularTest()'

  @Test
  @Mode(FULL)
  public void testThatTakesAWhile() { } // Will only be run the FAT is run in FULL mode
}

To run a FAT in FULL mode, specify -Dfat.test.mode=FULL in the launch command, for example:

./gradlew build.example_fat:buildandrun -Dfat.test.mode=FULL

Skipping tests based on java level

Currently OpenLiberty supports Java 7 and 8, with all new features requiring a minimum of Java 8 to run. As a result, any FATs for new features must not run on Java 7.

There are 3 ways to skip FAT tests based on Java level:

  • Setting javac.source: 1.8 in the bnd.bnd file (i.e. compiling the FAT with Java 8) will indicate that the entire FAT should skip on Java 7.
  • Setting runtime.minimum.java.level: 1.8 in the bnd.bnd file will indicate that the entire FAT should skip on Java 7.
  • Using the @MinimumJavaLevel(javaLevel = 1.8) annotation on a test class or method will cause the annotated element to skip on Java 7.

Error/Warning/FFDC scanning

Sometimes things can go wrong during a FAT test case that doesn't cause the JUnit test case to fail, such as an error or warning message in server logs. To catch this, whenever a LibertyServer is stopped, the messages.log file will be scanned for errors and warnings. If any errors or warnings are found, an exception is thrown which will trigger a class-level JUnit failure.

If a test generates errors or warnings that are expected to occur, then a list of expected errors can be passed into LibertyServer#stopServer. For example:

  @AfterClass
  public void tearDown() throws Exception {
    // Some test generated the *expected* error:  CWWK1234E: The server blew up because blah blah blah
    server.stopServer("CWWK1234E"); // Stop the server and indicate the 'CWWK1234E' error message was expected
  }

If a test generates an FFDC that is expected to occur, use the @ExpectedFFDC annotation. For example:

  @Test
  @ExpectedFFDC("java.lang.IllegalStateException")
  public void someErrorPathTest() {
    // Do something that causes a java.lang.IllegalStateException
  }

If a test generates an FFDC sometimes and we decide it's OK to allow, use the @AllowedFFDC annotation. Obviously, this annotation should be used sparingly, and @ExpectedFFDC should be used instead whenever possible. An example usage would be:

  @Test
  @AllowedFFDC // allows any FFDCs to occur
  public void someNastyErrorPathTest() {
    // Do something that could cause a variety of error paths to occur
  }
  
  @Test
  @AllowedFFDC("java.lang.IllegalStateException")
  public void someErrorPathTest() {
    // Do something that *sometimes* causes a java.lang.IllegalStateException
  }

Java 2 Security testing

OpenLiberty allows running with Java 2 Security (j2sec) enabled, and so the buildandrun task runs with j2sec enabled by default to test this scenario.

This can be disabled by using -Dglobal.java2.sec=false, for example:

./gradlew build.example_fat:buildandrun -Dglobal.java2.sec=false

Even if there are only a few distinct j2sec issues, the log files can quickly become overwhelming due to the volume of logs per issue multiplied by the potential frequency of issues. It also can be a slow, iterative process to fix the issues one at a time.

To speed up this process, you can use the following -D property combinations:

./gradlew build.example_fat:buildandrun -Dglobal.java2.sec=false -Dglobal.debug.java2.sec=true

This will log j2sec issues, but "catch" the exception and continue execution of the test bucket (to flush out the issues more quickly).

In this case, a report will be generated for each server instance in build/libs/autoFVT/ACEReport-<timestamp>.log, and even more detailed information about the full list of AccessControlException(s) will be contained within each of these server's console logs.

j2sec bootstrap.properties

The -D properties above cause the test framework to generate properties within each server's ${server.config.dir}/bootstrap.properties file, and so the properties in this file can be edited directly by the experienced user (as in the example below). Note because of the test framework writing to these properties, simply editing the bootstrap.properties in the test source will not necessarily help (since the framework will add its own values). See this Knowledge Center article for more info.

If a server should NOT run with java 2 security ever
Add this to the ${server.config.dir}/bootstrap.properties file for the server you want to permanently disable java 2 security for:

websphere.java.security.exempt=true

Repeating FAT tests for different feature/JEE levels

When we have multiple versions of a feature (e.g. cdi-1.2 and cdi-2.0), it is useful to re-run the same tests on the newer feature levels (e.g. re-run the CDI 1.2 tests for both cdi-1.2 and cdi-2.0) in order to prove that functionality that worked in previous feature versions continues to work in newer feature versions.

For example, if a FAT was written with Java EE 7 features, and you want to re-run the FAT with Java EE 8 equivalent features, you can do the following:

@RunWith(Suite.class)
@SuiteClasses({ ... })
public class FATSuite {
    // Using the RepeatTests @ClassRule in FATSuite will cause all tests in the FAT to be run twice.
    // First without any modifications, then again with all features in all server.xml's upgraded to their EE8 equivalents.
    @ClassRule
    public static RepeatTests r = RepeatTests.withoutModification()
                    .andWith(FeatureReplacementAction.EE8_FEATURES());
}

Note that when using the FeatureReplacementAction.EE8_FEATURES action, the additional test iteration will be skipped automatically if the java level is less than Java 8.

For a more fine-grained approach, the RepeatTests class rule can also be applied to individual test classes, for example:

@RunWith(FATRunner.class)
public class SomeTest extends FATServletClient {

    // Runs all tests in 'SomeTest' first without modification, and then again after upgrading
    // publish/servers/someServer/*.xml to use EE8 features
    @ClassRule
    public static RepeatTests r = RepeatTests.withoutModification()
                    .andWith(FeatureReplacementAction.EE8_FEATURES()
                             .forServers("someServer"));

Additionally, it is possible to exclude individual test methods or classes from a given repeat phase using the @SkipForRepeat annotation. For example:

    @Test
    @SkipForRepeat(SkipForRepeat.NO_MODIFICATION)
    public void testEE8Only() throws Exception {
        // This test will skip for the EE7 feature (i.e. NO_MODIFICATION) iteration
    }

    @Test
    @SkipForRepeat(SkipForRepeat.EE8_FEATURES)
    public void testEE7Only() throws Exception {
        // This test will skip for the EE8 feature iteration
    }

And finally, for maximum level of flexibility, it is possible to write custom actions by implementing the RepeatTestAction interface. For example, a FAT could be repeated with multiple database types in the following way:

public class RepeatTestsWithDB2 implements RepeatTestAction {

  public RepeatTestsWithDB2(/* DB2 database info */) { }

  @Override
  public boolean isEnabled() {  /* check if a DB2 database is available for use */ }

  @Override
  public void setup() { /* get and modify server.xml's to use DB2 database config */ }
}

And a custom action could be used as:

  @ClassRule 
  public static RepeatTests r = RepeatTests.withoutModification()
                                           .andWith(new RepeatTestsWithDB2(/* db2 database info */));

If helpful here is the componenttest.rules.repeater package source

Add a custom footer
Add a custom sidebar
Clone this wiki locally