Trying to solve Java's Gordian Knot

Fri, 01 Jun 2018 00:00:00 +0000

Java’s dependency management suffers from the dreaded diamond dependency issue. And while this issue is not unique to Java, it is, perhaps, more acute due to the precompiled nature of Java’s Jar files.

A Quick Refresher

If you are unfamiliar with this issue, here is a quick refresher:

Suppose there exists a library which contains a useful method that takes a string as its parameter:

class Common {
    void helpful(String a) {
        ...
    }
}

This library is published to an artifactory as

    com.hrakaroo : common : 1.0

Now, suppose there exists two other libraries which both use this helpful method

class Dog {
    void method1() {
        Common c = new Common();
        c.helpful("dog");
    }
}

and

class Cat {
    void method2() {
        Common c = new Common();
        c.helpful("cat");
    }
}

Each of these are also published to artifactory as:

    com.hrakaroo : dog : 1.0
    com.hrakaroo : cat : 1.0

And each of these have a transitive dependency on Common. Okay, now you decide to build your service which uses Dog and Cat so your dependency tree looks like

    com.hrakaroo : dog : 1.0
        com.hrakaroo : common : 1.0
    com.hrakaroo : cat : 1.0
        com.hrakaroo : common : 1.0

All is good. But now the folks who created Common come out with a new version which changes the signature of helpful and adds a boolean flag. So the new version looks like

class Common {
    void helpful(String a, boolean flag) {
        ...
    }
}

And knowing they are going to break some things publish this under a new version in artifactory

    com.hrakaroo : common : 2.0

The folks who made the Cat library realize this new flag will fix a bug they have had so they update to it as

class Cat {
    void method2() {
        Common c = new Common();
        c.helpful("cat", true);
    }
}

And publish it under

    com.hrakaroo : cat : 2.0

But the Dog library makers don’t need the new functionality so they don’t bother to update.

And finally, you decide to update your service to use the newest version of Cat which has the bug fix you need. This changes your dependency tree to:

    com.hrakaroo : dog : 1.0
        com.hrakaroo : common : 1.0
    com.hrakaroo : cat : 2.0
        com.hrakaroo : common : 2.0

In Java 8 you can not bring in the same dependency more than once with different versions. There are two common ways to deal with this, and they are both wrong.

First, you can not do anything. (This is probably the most popular solution.) In this situation gradle will pick one (usually the latest version) and use that as its version. In this case it will select com.hrakaroo : common : 2.0 which means that when your service calls Dog.method1 it will give you a runtime exception as the JVM will be unable to find the definition for helpful(String).

Or, if you are using gradle, you can use it’s force tag and force the version down to com.hrakaroo : common : 2.0 which means that when your service calls Cat.method2 it will give you a runtime exception as the JVM will be unable to find the definition for helpful(String, boolean)

The only “correct” solution here is to use gradle’s failOnVersionConfict() which will fail to compile your project unless both your dog and cat dependencies use the same version. This means you will be forced to fix the issue before your project can compile, but this may not be practical as a large project has lots of moving parts and compatible versions may not be available. Additionally, failOnVersionConflict() doesn’t understand semantic versioning so it will fail on PATCH level differences which often makes this a very painful and non-practical solution.

Most people just choose to go with the plug-and-pray approach where they just hope they never call a code path which encounters a definition which doesn’t exist.

Cutting the knot

As the two easy solutions are wrong and the one correct solution is impratical, the only real answer here is to avoid the problem altogether. When building a library, limit your dependencies.

Building a service requires a different approach from building a shared library. From the technologies you use to the way you version and test it are substantially different. Services are often just thin wiring together of different frameworks and libraries while libraries are more single tasked. And yet, they too often I don’t see people appreciate this difference. Instead they hack together libraries like they do services.

Stick to vanilla Java

I really like the Kotlin language, but I don’t think it has a place (yet) in shared libraries. Part of what makes Kotlin fun are all of the extension and infix libraries which are all packaged in the kotlin stdlib dependency. Any time you have more than two kotlin library dependencies you are just about assured to have a version conflict on the kotlin stdlib. Libraries should be written in Java to remove as many dependencies as possible.

Avoid huge common or utility libraries

Apache’s commons-lang3 is a fantastic library, but too often I’ve seen brought in so that the developer can use the StringUtils.join() method. Not only is this method trivial to write, but with Java 8 this can be done directly off the stream using .collect(Collectors.joining(...)).

The same can be said for Google’s Guava library, which is an enormous library that often makes non-compatible changes. In one library I reviewed the author had brought in a dependency on Guava so that they could use the Preconditions checks. While I think this type of defensive programming is good, the precondition checks can easily be re-created in your own library.

Copy and attribute

Providing the license allows it, it is also okay to simply copy sections of your dependent library directly into your own shared library and remove the dependency. Be sure to attribute where you got it from, but otherwise copying small to medium sized dependencies is okay.

Relocate and attribute

Finally, if all of the above are failing for you and, again if the licensing allows it, you can use a tool like shadow/shade to relocate a dependent library directly into you own library. These tools can will rebuild your resulting jar so that your dependencies are no longer transitive and all references to their old location have been changed to somewhere in your package. So, for example, you could relocate com.apache.commons to com.hrakaroo.apache.commons.

This will increase your jar size so it should really be used as the last resort, but it will guarantee that no one can later change the depenency version to something which is incompatable.

Whatever your approach, you should take the time when creating a shared library to minimize it’s transitive dependencies as much as possible. By doing so you will help minimize the risk to developers which use your library of creating their own diamon dependency nightmare.