Skip to content

Latest commit

 

History

History
318 lines (224 loc) · 16.5 KB

staticAnalysis.md

File metadata and controls

318 lines (224 loc) · 16.5 KB
Raw

Static Analysis

TEAMMATES uses a number of static analysis tools in order to maintain code quality and measure code coverage. This document will cover an overview of these tools and how to run them in local environment.

Version numbers

The version number of all the tool stacks are declared in build.gradle or package.json.

When downloading the plugin for Eclipse, find the plugin version that uses the correct version of the tool, e.g if CheckStyle 6.19 is used find an Eclipse plugin that uses CheckStyle 6.19 as well. If the exact version of the plugin cannot be found, using the latest version is allowed, however there is no guarantee that there will be no backward-incompatible changes.

Conversely, when updating any tool, ensure that the tool version is supported by the Eclipse plugin, e.g when upgrading CheckStyle to 6.19 ensure that there is an Eclipse plugin which supports that version as well.

Tool stack

CheckStyle

CheckStyle helps to enforce coding standard in Java source code. The rules to be used are configured in a ruleset file; in TEAMMATES the file can be found here.

Configuring Checkstyle Eclipse plugin

The plugin for Eclipse can be found here.

  1. In Project > Properties, go to the Checkstyle tab.
  2. In the Local Check Configurations tab, create a new Check Configuration. Select Project Relative Configuration for its Type, enter any Name you wish and set the Location to the teammates-checkstyle.xml file in the Project Folder. Click OK.
  3. In the Main tab, uncheck Use simple configuration.
  4. Add a new File Set. It should include only the .java$ file. Enter any name you wish for the File Set Name, and select the Check Configuration that you created earlier for Check Configuration. Click OK.
  5. Ensure that only the newly created File Set is enabled. Disable all other File Sets if they are enabled. Click OK.
Configuring Checkstyle in IntelliJ IDEA

The plugin for IntelliJ can be found here.

You can configure all the static analysis tools automatically or follow the manual instructions.

  1. Go to File → Settings → Other Settings → Checkstyle.
  2. Set Scan Scope to Only Java sources (including tests).
  3. Click the + to add a new configuration file. Click the Browse button, navigate to the static-analysis folder, and choose the teammates-checkstyle.xml file.
  4. Fill in the Description field with the name of your project (e.g. teammates).
  5. Click Next. Set the value of basedir to the path of your project folder.
  6. Click Finish.
  7. Check the box next to the newly added rule to activate it.
Suppressing Checkstyle warnings

To introduce code that violates Checkstyle rules, wrap the violating code with // CHECKSTYLE.OFF:RuleName and re-enable it afterwards with // CHECKSTYLE.ON:RuleName (note the absence of space around . and :). Checkstyle also provides several other methods of suppressing rule violations, which can be found in the documentation here. The suppression should be as specific as possible, and the reason for violating the rule should be explained.

An example for suppressing the Avoid star imports rule is as follows:

// CHECKSTYLE.OFF:AvoidStarImports as there would be many (>100) import lines added if we were to import all of the ActionURIs
import static teammates.common.util.Const.ActionURIs.*;
// CHECKSTYLE.ON:AvoidStarImports

To suppress multiple violations at once, separate the rules with vertical bar |:

// CHECKSTYLE.OFF:AbbreviationAsWordInName|MemberName the database uses ID
private String ID;
// CHECKSTYLE.ON:AbbreviationAsWordInName|MemberName

PMD

PMD analyses the Java source code for common programming flaws (e.g unused variables, empty catch block). The rules to be used are configured in a ruleset file; in TEAMMATES the file can be found here.

Configuring PMD Eclipse plugin

The plugin for Eclipse can be found here.

  1. In Project > Properties, go to the PMD tab.
  2. Check Enable PMD.
  3. Under Rule Source, check Use the ruleset configured in a project file. Click Browse, navigate to the static-analysis directory of the project and select teammates-pmd.xml. Click OK.
Configuring PMD for IntelliJ

The plugin for IntelliJ can be found here.

You can configure all the static analysis tools automatically or follow the manual instructions.

  1. Go to File → Settings → Other Settings → PMD.
  2. Click the + to add a new rule set. Browse for teammates-pmd.xml. Click OK.
  3. In the Options tab, set Target JDK to 1.7.
  4. Click OK.
Suppressing PMD warnings

To introduce code that violates PMD rules, use @SuppressWarnings("PMD.RuleName") annotation at the narrowest possible scope. PMD also provides several other methods of suppressing rule violations, which can be found in the documentation here. The suppression should be as specific as possible, and the reason for violating the rule should be explained.

FindBugs

FindBugs analyses Java source code for potential bugs at bytecode level, thus able to find potential bugs that PMD cannot find. In Gradle build, the rules are configured by specifying the classes in the visitors variable. The plugin for Eclipse can be found here. The plugin for IntelliJ can be found here.

You can also configure all the static analysis tools automatically for IntelliJ IDEA.

Macker

Macker checks the architectural integrity of Java source code. The rules to be used are configured in a ruleset file; in TEAMMATES the file can be found here.

EclEmma/JaCoCo

EclEmma/JaCoCo measures code coverage for Java test run. Normally, the coverage will be run against all classes specified as the source code, but it can be configured to exclude classes matching certain name patterns. The plugin for Eclipse can be found here.

NodeJS

NodeJS integration is supported in IntelliJ. You can use it to manage your dependencies (optional). The plugin can be found here.

ESLint

ESLint functions both to enforce coding standard and also to find potential bugs in JavaScript source code. The rules to be used are configured in a ruleset file; in TEAMMATES the file can be found here. ESLint integration is currently not supported for Eclipse.

NOTE

You can configure all the static analysis tools automatically for IntelliJ IDEA or follow the manual instructions.

Configuring ESLint for IntelliJ

  1. If you have not installed Node.js and ESLint, please refer to install necessary tools and languages and set up project specific settings and dependencies.
  2. Open File → Settings or IntelliJ IDEA → Preferences.
  3. Go to Languages & Frameworks → JavaScript → Code Quality Tools → ESLint.
  4. Check the box next to Enable.
  5. The Node interpreter and Stylelint package should have been auto-filled to your locally installed NodeJS and $PROJECT_DIR$/node_modules/stylelint respectively. Point them to the right locations if they are not.
  6. Point Configuration file: to the location of teammates-eslint.yml.
  7. Under 'Extra eslint options:', add --ext .es6 --ignore-pattern '**/*.js'.
  8. Click OK.
Suppressing ESLint warnings

To introduce code that violates ESLint rules, wrap the violating code with /* eslint-disable rule-name */ and re-enable it afterwards with /* eslint-enable rule-name */. The suppression should be as specific as possible, and the reason for violating the rule should be explained.

An example to suppress the camelcase rule is as follows:

/* eslint-disable camelcase */ // The variable name is provided by an external library, which does not follow camelcase.
// violating codes go here
/* eslint-enable camelcase */

Stylelint

Stylelint functions both to enforce coding standard and also to find potential bugs and sub-optimal practices in stylesheets (CSS, SCSS). The rules to be used are configured in a ruleset file; in TEAMMATES the file can be found here. Stylelint integration is currently not supported for Eclipse.

You can configure all the static analysis tools automatically for IntelliJ IDEA or follow the manual instructions.

Configuring Stylelint for IntelliJ

  1. If you have not installed Node.js and ESLint, please refer to install necessary tools and languages and set up project specific settings and dependencies.
  2. Open File → Settings or IntelliJ IDEA → Preferences.
  3. Go to Languages & Frameworks → Stylesheets → Stylelint.
  4. Check the box next to Enable.
  5. The Node interpreter and Stylelint package should have been auto-filled to your locally installed NodeJS and $PROJECT_DIR$/node_modules/stylelint respectively. Point them to the right locations if they are not.
  6. Click OK.
  7. Copy $PROJECT_DIR$/static-analysis/teammates-stylelint.yml to $PROJECT_DIR$/.stylelintrc.yml.

blanket.js

blanket.js measures code coverage for JavaScript test run. It is immediately enabled for all scripts with the data-cover attribute (configured via HTML) in a QUnit test run.

IntelliJ automatic setup

  1. Ensure the following plugins are installed. CheckStyle-IDEA, PMDPlugin, FindBugs-IDEA, NodeJS (Optional)

  2. Run the command to setup the settings for the various plugins:

    ./gradlew setupIntellijStaticAnalysis

NOTE

The behavior of the automated setup is described here.

Running static analysis

Note the following:

  • Change ./gradlew to gradlew.bat in Windows.
  • All the commands are assumed to be run from the root project folder, unless otherwise specified.

Travis CI

Travis CI will run static analysis before testing. If violations are found, the build will be terminated with an error before any testing is done in order to save time and resources.

CLI

You can run the static analysis tools via Gradle or NPM. The violations caught, if any, will be printed to the console itself.

./gradlew {toolType}{sourceCodeType}

where {toolType} = checkstyle, pmd, findbugs (lowercase), and {sourceCodeType} = Main, Test (Pascal Case).

To run Macker analysis on all Java source files, run the following command:

./gradlew macker

To run ESLint and Stylelint analysis on all JavaScript, JSON, and CSS source files, run the following command:

npm run lint

To run all static analysis tasks in one sitting, run the following two commands:

./gradlew lint --continue
npm run lint

Eclipse

Eclipse allows CheckStyle, PMD, and FindBugs analysis on the spot; just right-click on the source class or folder and select the appropriate commands. Remember to configure the tools to use the ruleset provided. The analysis results are immediately reported in Eclipse and you can traverse to the violating lines with just a click.

To run Checkstyle analysis on all Java source files with the Eclipse Checkstyle plugin, right click on the Project Folder in the Project Explorer window in Eclipse and select Checkstyle > Check Code with Checkstyle. The report can be found in the Markers window in Eclipse.

To run PMD analysis using the Eclipse PMD plugin, right click on the project under Project Explorer and select PMD > Check Code. The report can be viewed in the PMD Perspective view under Violations Overview.

IntelliJ IDEA

CheckStyle, ESLint and Stylelint are configured as inspection tools in IntelliJ IDEA.
This means they automatically run on every open file in the editor. Any code issues will show up on the right of the editor and you can also see the analysis status of the whole file from by hovering over the icon on the top-right of the editor. You may also be able to see some code inspection status on the line itself (e.g. wriggly red lines).

FindBugs is also an inspection tool but it does not run automatically. You can run it by going to Analyze → FindBugs and choosing the option you want. Alternatively, you can select a number of files and right click, select FindBugs and the option you want.

If you wish to run inspections for the whole project, you can do Analyze → Inspect Code... → Whole project. You may also wish to learn more about code inspections by referring to IntelliJ IDEA's documentation.

NOTE

FindBugs will only appear in the inspection results if you ran it manually before running Analyze → Inspect Code... → Whole project.

PMD is provided as a plugin and does not run automatically. You can run it by selecting a number of files, right clicking, selecting Run PMD and then choosing the option you want.

Macker is not available in IntelliJ IDEA and you have to run it on the command line.

Running code coverage session

Travis CI

For Java tests, if your build and run is successful, Codecov will pull the test coverage data and generate a report on their server. The link to the report will be displayed in each PR, or by clicking the badge on the repository homepage.

For JavaScript unit tests, coverage is done concurrently with the tests themselves. A coverage lower bound is enforced via AllJsTests.java, lower than which the build will fail.

CLI

You can use Gradle to run tests and obtain the coverage data with jacocoReport task, i.e:

./gradlew appengineRun ciTests
./gradlew jacocoReport appengineStop

The report can be found in the build/reports/jacoco/jacocoReport/ directory.

For JavaScript unit tests, coverage is done concurrently with the tests themselves. A coverage lower bound is enforced via AllJsTests.java, lower than which the build will fail.

Eclipse

For Java tests, choose Coverage as TestNG Test instead of the usual Run as TestNG Test to run the specified test or test suite. The coverage will be reported in Eclipse after the test run is over.

For JavaScript unit tests, simply open allJsUnitTests.html and tick Enable coverage, or run AllJsTests.java. The coverage will be reported immediately in the test page.

IntelliJ IDEA

For Java tests, you can measure code coverage for the project using Run → Run... → CI Tests → ▶ → Cover .

NOTE

There are some packages and classes that are excluded when using Jacoco on the comamnd line which are not excluded when you run them in IntelliJ, thus the coverage statistics you see in IntelliJ may not match what you see on Codecov.

Alternatively, you can right click a class with TestNG test(s) and click Run '$FileClass$' with Coverage, this will use IntelliJ IDEA's code coverage runner. You can further configure your code coverage settings by referring to IntelliJ IDEA's documentation.

For JavaScript unit tests, simply open allJsUnitTests.html and tick Enable coverage, or run AllJsTests.java. The coverage will be reported immediately in the test page.