Skip to main content

ItDepends

An (incomplete) list of static analysis tools for Java

This post describes some of the static analysis tools I’ve used in various Java projects over the years. In the last couple of days I’ve been setting up new projects from scratch and decided it would be nice to organize some of the knowledge in a single place. Apart from discussing the tools themselves I also describe why it makes sense to include them in your next (or current) projects.

Defining static analysis

Per the Wikipedia entry on Static Program Analysis:

Static program analysis is the analysis of computer software that is performed without actually executing programs, in contrast with dynamic analysis, which is analysis performed on programs while they are executing.In most cases the analysis is performed on some version of the source code, and in the other cases, some form of the object code.

The term is usually applied to the analysis performed by an automated tool, with human analysis being called program understanding, program comprehension, or code review. Software inspections and software walkthroughs are also used in the latter case.

In more practical terms, we’d like to use a tool that understands the code (syntactically or semantically), looks at it from a certain angle (formatting, code “quality”), and either “improves” the code automatically or generates a report with suggestions. I’m using words like “quality” and “improves” vaguely, there are some practices generally agreed upon (e.g. we should avoid single-letter variables or magical numbers) but those are not rules written in stone. We may decide, for whatever reason, that in the context of a specific project we would like to not follow some of the “good practices”.

Static analysis tools for Java

Over the years multiple static analysis tools have been developed for the Java language. I briefly describe each tool I’m familiar with below and what class of “things” it’s looking for. You can find a Maven project on my GitHub with all of them configured.

SpotBugs, previously FindBugs

https://spotbugs.github.io/

http://findbugs.sourceforge.net/

According to Wikipedia article on FindBugs it was first released on 10.06.2006. For whatever reason, the development on the project stopped and the last stable release is 3.0.1 released on 06.03.2015. The spiritual successor to FindBugs is SpotBugs, according to their website

SpotBugs is a program which uses static analysis to look for bugs in Java code. It is free software, distributed under the terms of the GNU Lesser General Public License.

SpotBugs is the spiritual successor of FindBugs, carrying on from the point where it left off with the support of its community. Please check the official manual for details.

SpotBugs has an extensive list of over 400 “bugs” they look for in your code. The “bugs” recognized by the tool come from different categories, like bad practice, code correctness, or issues with multi-threaded code. Some “bug” examples include: dropping exceptions, defining only hashCode or equals, comparing strings using ==/!= instead of equals, and incrementing a value in a volatile field.

SpotBugs checks are executed against the .class files so theoretically it can be used in any programming language available on JVM (e.g. Scala, Kotlin, Clojure) but I’ve only seen it used in Java projects.

Configuring SpotBugs

SpotBugs executes all the checks by default, we don’t need to provide an additional file which enables specific checks. There is also an option to provide a “filter” file where we can disable specific checks for a given piece of code (e.g. a class or a method), you can find an example “filter” file at the end of this section.

SpotBugs documentation shows how to integrate it with various build systems, e.g. Gradle, Maven, or Ant. In the case of Maven you only need to declare a SpotBugs plugin and you are good to go.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

<build>
    <plugins>
        <!-- Based on https://spotbugs.readthedocs.io/en/latest/maven.html -->
        <plugin>
            <groupId>com.github.spotbugs</groupId>
            <artifactId>spotbugs-maven-plugin</artifactId>
            <version>4.2.0</version>
            <dependencies>
                <!-- overwrite dependency on spotbugs if you want to specify the version of spotbugs -->
                <dependency>
                    <groupId>com.github.spotbugs</groupId>
                    <artifactId>spotbugs</artifactId>
                    <version>4.2.3</version>
                </dependency>
            </dependencies>
            <executions>
                <execution>
                    <goals>
                        <goal>check</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

The above configuration adds a plugin to your project pom.xml and will automatically execute the check goal as defined by the plugin during the verify phase of Maven build. In practice, it means the SpotBugs check will be performed automatically if you execute mvn verify, mvn install, or mvn deploy.

Below you can see an example mvn verify output in a project that defines a class which overrides equals without overriding hashCode.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

adebski@adebski-VirtualBox:~/IdeaProjects/JavaStaticAnalysis$ mvn verify
...
[INFO] >>> spotbugs-maven-plugin:4.2.0:check (default) > :spotbugs @ JavaStaticAnalysis >>>
[INFO] 
[INFO] --- spotbugs-maven-plugin:4.2.0:spotbugs (spotbugs) @ JavaStaticAnalysis ---
[INFO] Fork Value is true
[INFO] Done SpotBugs Analysis....
[INFO] 
[INFO] <<< spotbugs-maven-plugin:4.2.0:check (default) < :spotbugs @ JavaStaticAnalysis <<<
[INFO] 
[INFO] 
[INFO] --- spotbugs-maven-plugin:4.2.0:check (default) @ JavaStaticAnalysis ---
[INFO] BugInstance size is 1
[INFO] Error size is 0
[INFO] Total bugs: 1
[ERROR] High: com.adebski.HelloWorld$ProblematicClass defines equals and uses Object.hashCode() [com.adebski.HelloWorld$ProblematicClass] At HelloWorld.java:[lines 21-24] HE_EQUALS_USE_HASHCODE
[INFO] 


To see bug detail using the Spotbugs GUI, use the following command "mvn spotbugs:gui"

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  7.389 s
[INFO] Finished at: 2021-05-06T20:45:34+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.spotbugs:spotbugs-maven-plugin:4.2.0:check (default) on project JavaStaticAnalysis: failed with 1 bugs and 0 errors  -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
adebski@adebski-VirtualBox:~/IdeaProjects/JavaStaticAnalysis$

In the real world there are cases where we’d like to inform the static analysis tool that some of the patterns should not apply to specific fragments of code (e.g. methods, classes). In SpotBugs we can configure an XML filter file where we specify patterns (e.g. class name) and a range of “bugs” that should be ignored for all places that match a given pattern. You can read more about the filters and see some examples in the official documentation. The filter file below disables all checks for the class com.adebski.HelloWorld$Bar and “bad practice” bugs for method method in com.adebski.HelloWorld$ProblematicClass.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

<?xml version="1.0" encoding="UTF-8"?>
<FindBugsFilter
        xmlns="https://github.com/spotbugs/filter/3.0.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="https://github.com/spotbugs/filter/3.0.0 https://raw.githubusercontent.com/spotbugs/spotbugs/3.1.0/spotbugs/etc/findbugsfilter.xsd">
    <Match>
        <Class name="com.adebski.HelloWorld$Bar" />
    </Match>

    <Match>
        <Class name="com.adebski.HelloWorld$ProblematicClass"/>
        <Method name="method" />
        <Bug code="BC" />
    </Match>
</FindBugsFilter>

Error Prone

https://github.com/google/error-prone

Error Prone is a static code analyzer from Google. It’s implemented as a javac compiler plugin so the checks are performed during the code compilation.

Error Prone can detect over 400 bug patterns but only some of them are enabled as errors by default, the rest of them emit only warnings or they are marked as experimental and won’t be executed unless an Error Prone plugin is configured to consider them. There is some overlap between patterns detected by SpotBugs and Error Prone but Error Prone offers additional checks that verify we are using some of the Google libraries (e.g. Guice or AutoValue) in a correct way.

Configuring Error Prone

Because Error Prone is implemented as a javac plugin all we need to do is to modify the configuration of maven-compiler-plugin as per the plugin documentation. In the snippet below you can also see how to instruct the plugin to emit an error for a check that, by default, is treated only as a warning, e.g. class should not implement both Comparable and Comparator. You can read more about the Maven integration here, all of the command line flags are documented here.

From Java 16 we also need to make sure that Maven forks a separate JVM to run the javac compiler and we also need to add the add-exports and add-opens since Error Prone is accessing some of the private APIs to do its job, you can read more about JEP 396: Strongly Encapsulate JDK Internals by Default if you are curious. If you are using one of the previous Java versions you may see a warning emitted by the javac compiler that private APIs are being accessed and that access will be prohibited in the future.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

<build>
    <!-- Based on https://errorprone.info/docs/installation -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.1</version>
        <configuration>
            <source>11</source>
            <target>11</target>
            <fork>true</fork>
            <encoding>UTF-8</encoding>
                <compilerArgs>
                    <arg>-XDcompilePolicy=simple</arg>
                    <arg>-Xplugin:ErrorProne -Xep:ComparableAndComparator:ERROR</arg>
                    <arg>-J--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED</arg>
                    <arg>-J--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED</arg>
                    <arg>-J--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED</arg>
                    <arg>-J--add-exports=jdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED</arg>
                    <arg>-J--add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED</arg>
                    <arg>-J--add-exports=jdk.compiler/com.sun.tools.javac.processing=ALL-UNNAMED</arg>
                    <arg>-J--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED</arg>
                    <arg>-J--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED</arg>
                    <arg>-J--add-opens=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED</arg>
                    <arg>-J--add-opens=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED</arg>
                </compilerArgs>
            <annotationProcessorPaths>
                <path>
                    <groupId>com.google.errorprone</groupId>
                    <artifactId>error_prone_core</artifactId>
                    <version>2.6.0</version>
                </path>
                <!-- Other annotation processors go here.

                If 'annotationProcessorPaths' is set, processors will no longer be
                discovered on the regular -classpath; see also 'Using Error Prone
                together with other annotation processors' below. -->
            </annotationProcessorPaths>
        </configuration>
    </plugin>
</build>

PMD

https://pmd.github.io/

PMD is an extensible cross-language static code analyzer, it “understands” languages like Java but it also has rules for things like Maven pom files, generic XMLs, or JSP files. It has over 300 rules for Java alone.

Configuring PMD

PMD handles rules differently than Error Prone or SpotBugs. By default, no rules are enabled and we need to construct an XML ruleset file ourselves. The good thing is that we don’t need to reference each of the rules in our ruleset file directly, we can enable all rules from a specific category, e.g. best practices. We can also exclude specific files from analysis, but we are not able to disable only a subset of rules for a specific file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

<?xml version="1.0"?>

<ruleset name="Custom Rules"
         xmlns="http://pmd.sourceforge.net/ruleset/2.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://pmd.sourceforge.net/ruleset/2.0.0 https://pmd.sourceforge.io/ruleset_2_0_0.xsd">

    <description>
        Description of rules.
    </description>

    <!-- Adding specific PMD rule -->
    <rule ref="category/java/errorprone.xml/EmptyCatchBlock" />

    <!-- Adding all rules from a single category -->
    <rule ref="category/java/multithreading.xml" />

    <exclude-pattern>.*/com/adebski/HelloWorld.*</exclude-pattern>
</ruleset>

Once we have our rule set file we need to add the PMD to our pom.xml file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

<build>
    <plugins>
        <!-- Based on https://pmd.github.io/pmd-6.34.0/pmd_userdocs_tools_maven.html -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-pmd-plugin</artifactId>
            <version>3.13.0</version>
            <configuration>
                <!-- failOnViolation is actually true by default, but can be disabled -->
                <failOnViolation>true</failOnViolation>
                <!-- printFailingErrors is pretty useful -->
                <printFailingErrors>true</printFailingErrors>
                <rulesets>
                    <ruleset>/pmd-ruleset.xml</ruleset>
                </rulesets>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>check</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Now, the PMD will be executed as part of the verify goal, the same as SpotBugs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

adebski@adebski-VirtualBox:~/IdeaProjects/JavaStaticAnalysis$ mvn verify
...
[INFO] >>> maven-pmd-plugin:3.13.0:check (default) > :pmd @ JavaStaticAnalysis >>>
[INFO] 
[INFO] --- maven-pmd-plugin:3.13.0:pmd (pmd) @ JavaStaticAnalysis ---
[WARNING] Unable to locate Source XRef to link to - DISABLED
[WARNING] Unable to locate Source XRef to link to - DISABLED
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] 
[INFO] <<< maven-pmd-plugin:3.13.0:check (default) < :pmd @ JavaStaticAnalysis <<<
[INFO] 
[INFO] 
[INFO] --- maven-pmd-plugin:3.13.0:check (default) @ JavaStaticAnalysis ---
[INFO] PMD version: 6.21.0
[INFO] PMD Failure: com.adebski.HelloWorld:13 Rule:EmptyCatchBlock Priority:3 Avoid empty catch blocks.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10.770 s
[INFO] Finished at: 2021-05-08T15:47:48+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-pmd-plugin:3.13.0:check (default) on project JavaStaticAnalysis: You have 1 PMD violation. For more details see: /home/adebski/IdeaProjects/JavaStaticAnalysis/target/pmd.xml -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
adebski@adebski-VirtualBox:~/IdeaProjects/JavaStaticAnalysis$

Checkstyle

https://checkstyle.sourceforge.io/

Checkstyle is a tool we can use to make sure our source code adheres to a coding standard, this covers things like making sure the code is formatted consistently (e.g. placement of { characters, enforcing a line length) but also that it follows some of the “best practices”, e.g. creating only immutable Exceptions or that Strings are compared using equals instead of ==.

Checkstyle offers more than 100 built-in style checks and it’s also possible to write your own checks.

Configuring Checkstyle

Integrating Checkstyle with your Maven build is just a matter of adding Checkstyle Maven plugin to your pom.xml file and specifying a Checkstyle xml configuration file. Checkstyle user guide has a whole section on how to write your configuration file but you can also use one of the existing configurations as a starting point, e.g. Checkstyle configuration that follows recommendations from the Google’s Java style guide.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

<build>
    <plugins>
    <!-- based on http://maven.apache.org/plugins/maven-checkstyle-plugin/usage.html -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-checkstyle-plugin</artifactId>
            <version>3.1.2</version>
            <configuration>
                <configLocation>checkstyle.xml</configLocation>
                <encoding>UTF-8</encoding>
                <consoleOutput>true</consoleOutput>
                <failsOnError>true</failsOnError>
                <linkXRef>false</linkXRef>
            </configuration>
            <!-- explicitly specifying the newest (at the time) Checkstyle version -->
            <dependencies>
                <dependency>
                    <groupId>com.puppycrawl.tools</groupId>
                    <artifactId>checkstyle</artifactId>
                    <version>8.42</version>
                </dependency>
            </dependencies>
            <executions>
                <execution>
                    <id>validate</id>
                    <phase>validate</phase>
                    <goals>
                        <goal>check</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

To suppress a specific check for a given section of our code we need to use one of the Checkstyle suppression filters. For example we can configure SuppressionCommentFilter to mark a block of code with special comments to disable specific checks, there is also an option to specify a suppression file using the SuppressionFilter. The suppression file is an XML document where we configure which checks should be disabled for what classes or their specific parts.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

<?xml version="1.0"?>

<!DOCTYPE suppressions PUBLIC
  "-//Checkstyle//DTD SuppressionFilter Configuration 1.2//EN"
  "https://checkstyle.org/dtds/suppressions_1_2.dtd">
<!-- taken from https://checkstyle.sourceforge.io/config_filters.html#SuppressionFilter -->

<suppressions>
  <suppress checks="JavadocStyleCheck"
    files="AbstractComplexityCheck.java"
    lines="82,108-122"/>
  <suppress checks="MagicNumberCheck"
    files="JavadocStyleCheck.java"
    lines="221"/>
  <suppress message="Missing a Javadoc comment"/>
</suppressions>

Additionally there are a lot of Checkstyle plugins available for various IDEs, e.g. jshiell/checkstyle-idea for IntelliJ. One nice feature of the IntelliJ Checkstyle plugin is the ability to configure the IntelliJ code formatter based on your Checkstyle configuration. You can find instructions on how to do that in this Stack Overflow post.

Advantages and disadvantages of using static analysis

Intuitively it “makes sense” to use tools and automation for:

  • detecting as early as possible, a class of bugs/problems that apply to “most” software engineering projects,
  • keeping the code style/formatting consistent in a single project or all the projects maintained by the team.

Without automating those tasks we rely on humans to catch all problems in the code review process and, from my experience, some things always slip through the cracks. Not all team members may be aware of every convention, especially if a team is compromised of people who joined after the project was started (which is a very common situation) or are new to the company. When reviewing “large” code reviews it may be exhausting and discouraging to point out every little discrepancy with the style guides and conventions and it wastes the mental cycles of the reviewer who should be looking for bugs and issues with the business logic or at larger architectural decisions.

When we use a programming language or a technology we are not very familiar with it’s easy to use it in a non-idiomatic way or introduce bugs to our system. We don’t always have the luxury of having a person in our team (or organization) who we can consult on the way we use a technology. Static analysis tools help us avoid at least some of the potential issues that happen when we try to use something new. It’s also a learning opportunity, when a static analysis flags a part of code due to an error we were not even aware of. In the future there is a higher chance of us avoiding writing a code like that altogether. We may also start to be aware of classes of bugs we did not consider before, e.g. SQL Injection.

It’s important to keep the signal to noise ratio at the acceptable level. If we introduce a static analysis to an existing project and we are presented with hundreds or thousands of “issues” each time we build our software we will most likely just ignore them, what’s the difference between 1004 and 1005 potential errors? Static analysis also increases the time it takes to build/compile the project. If the new tool we try to integrate increases the build time to an unacceptable levels (and this depends on a project and our baseline) we should weigh in the value that the tool brings vs the decrease in developer productivity.

If we integrate static analysis in our project we should incorporate the checks with our build scripts and systems so that the analysis is always performed as part of the build; otherwise, it will be easy to forget (or not even know) that you need to run an additional command to verify that everything meets the quality bar agreed by the team. Additionally, those checks should be executed again in our CI/CD pipelines to catch guard against (hopefully rare) situations where someone ignores the errors and just pushes the code hopping for the best. The checks should be treated as errors and fail the build, once we have a couple of violations in the project it’s hard to distinguish if the umpteenth warning is an actual issue or not. That’s also why tools offer a way to suppress or disable specific checks or rules for a given piece of code, from time to time there are genuine situations where the code needs to violate one rule, but those situations should be rare. If we need to disable a specific check for most of our code we should think about removing the rule altogether.

Out of curiosity I’ve gooled for papers about the usefulness of static analysis tools for languages like Java and C++:

Here are some of the takeaways from the papers above:

  • No static analysis tools are perfect and can catch all the issues, for better coverage we should try and use more than one, even if there is some overlap between them.
  • Static analysis needs to be integrated into the developer workflow and the software build process. When Google initially tried to integrate FindBugs into their process, by scanning the code base nightly and posting a bug dashboard, it saw little use because the dashboard was not part of the developer’s usual workflow.
  • PITEST can be used to perform mutation testing on our code, basically checking if our unit tests fail if the source code is modified according to some rules (e.g. changing the if statement form a >= b to a > b). The assumption is that when the application code changes, it should produce different results and cause the unit tests to fail. If a unit test does not fail in this situation, it may indicate an issue with the test suite.

Conclusion

I feel that static analysis tools are useful tools that should be integrated into the developer workflow and the software build process. They help to catch bugs and errors at the language level and they make sure that our/team’s code meets a specific quality bar. When reviewing a code review we may safely assume that the configured class of bugs is absent from the code and we can focus on finding more subtle bugs, reviewing architectural decisions and making sure that the code fulfills business requirements.

2021-05-22
comments powered by Disqus