Skip to content

Test Plan

Jump to bottom
Sheng Chen edited this page Aug 12, 2019 · 46 revisions

Project

The following projects can be used (but not limited to):

Version

  1. Check the version is equal to the current milestone.
  2. Check the changelog has the right version and no typo.

Basic

  1. The following events will startup the extension:
  • Workspace contains pom.xml
  • Workspace contains build.gradle
  • Test explorer gets focus
  • Trigger the extension commands
  1. The startup activation time of the extension should no more than 1000ms.

Test Explorer - UI

  1. The explorer should perform as lazy-load behavior: only load the children nodes if the parent nodes are expanded.
  2. The nodes should all be sorted alphabetically.
  3. If the method has annotated with @DisplayName (a JUnit 5 annotation), the node will use the annotation value as it's display name.
  4. The hierarchy should be correct - Both for Single-Root or Multi-Root scenarios.
  5. Click on the class/method level node, the editor should jump to the correct position of the source file.
  6. Update/Delete/Create test code, the explorer should be updated accordingly.
  7. The explorer can be refreshed if the user clicks refresh in either navigation bar or context menu.

Test Explorer - Run/Debug Test

  1. There are actions called run test/debug tests in both navigation bar or context menu. In other words, the tests can be run/debug in the following level:
    1. Workspace Root
    2. Workspace Folder
    3. Package
    4. Class/Inner Class
    5. Method
  2. After executing run tests/debug tests, all the tests under the target node will be run.
  3. After executing run tests/debug tests, in the notification window at the right-bottom side, it will show the status of the current test tasks, such as: searching test items..., then running tests...

Code Lens

  1. Code Lens should be shown at the following places:
    1. Test Methods (Annotated with @Test).
    2. Classes which contains test methods.
  2. By default, the Code Lens Contains following actions:
    1. Run Test
    2. Debug Test
  3. After tests are run, a new status icon should be added as the third Code Lens action. For those tests that have not been run, the status icon should not be shown for it.
  4. Click Run Test will run the tests
  5. Click Debug Test will debug tests
  6. Click the status icon should show the Test Report
  7. User can set the visibility of the Code Lens by using the setting java.test.editor.enableShortcuts

Test Report

  1. Only show the filter button when necessary. For example, the Failed button will appear only when there are failed tests. All of them can filter the results correctly.
  2. Test report theme updates when the user changes the color theme.
  3. The report can auto-update if the user reruns the tests and get different results.
  4. The test report is read-only. (Not prompt saving when closing it).
  5. The methods are categorized by classes.
  6. Can navigate to the method source from the report.
  7. Can config the report position through java.test.report.position.
    • When the report is not opened in the editor, reveal it as the setting specified.
    • When the report is opened in the editor, just update the report and do not change its column.
  8. Can config whether to automatically show the report after execution, default is onFailure. Other available values are: always and never.

Test Status Bar

  1. Should show Running Tests... during running.
  2. When the status bar is showing Running Tests..., click it should open the output channel.
  3. After running/debugging tests, the status bar should show the result.
  4. Click the result, will open the Test Report.

Cancel the running job

  1. start a test run that lasts for a long period, for example, add Thread.sleep in your test.
  2. click Cancel button on the notification message, the running job should be stopped.

Logs

  1. Can trigger command Java: Show Test Output to open the output channel.
  2. Can trigger command Java: Open Test Runner Log to show the log file.
  3. Can set log level: verbose, info, error
  4. Can print the Chinese character correctly (providing that the user has correctly setup the project).

Run With Configuration

  1. The extension will only honor the java.test.config if it exists.
  2. java.test.config could be either an array or an object
  3. If java.test.config is not empty, user will see a list to select the configuration
  4. After the user selects the configuration, the extension will pop up a message box. The user can: Yes: Set the selected config as default, No: Do nothing or Never show: Do not show this message any more
  5. The ${workspaceFolder} is available in workdingDirectory, args, vmargs and the value of entries in env

Migrate deprecated launch.test.json

  • Should be able to migrate the content in launch.test.json into the workspace setting, using Java: Migrate Deprecated 'launch.test.json'.
  • After migration, user can:
    • open setting
    • remove deprecated files
    • view more details

Test Discovery

  • For a Spring Boot project, if both src/main/resource and src/test/resource contain application.properties file, the test runner should pick the one in src/test/resource.
  • Star import should also be supported. (For example, import org.junit.*)
  • JUnit 4:
    • org.junit.Test
    • org.junit.experimental.theories.Theory
    • org.junit.runner.RunWith (class level)
  • JUnit 5:
    • org.junit.jupiter.api.Test
    • org.junit.jupiter.params.ParameterizedTest
    • org.junit.jupiter.api.RepeatedTest
    • org.junit.jupiter.api.TestFactory
    • org.junit.jupiter.api.Nested
    • org.junit.jupiter.api.TestTemplate
    • Meta Annotation
  • TestNG:
    • org.testng.annotations.Test
Clone this wiki locally