It’s more than disappointing that as I write this in 2022, more than a generation since I first worked as a software developer, nearly a century since the first programmable electronic computers, we are still wrestling with things as archane and crude as build files and what the Java community not-so-affectionately calls “dependency hell.”
For the uninitiated, computer programs are written in relatively “high-level” human-readable languages like Java and C, but execute in a low level machine language understandable by a microprocessor. In simple terms, the process of getting from high-level code to executable code is the “build,” and build files are instructions to help govern that process. Dependency hell arises because Java programs are built up from libraries of pre-built components that do everything from manipulating strings to writing the contents of memory out to an application file (like the .docx file produced by Microsoft Word). No developer or team could possibly afford the time or money to implement all these supporting functions from scratch, and libraries provide standardized mechanisms for various entities to develop and support them and make them available to application developers. So far, so good.
The trouble is, over time, code gets written against specific versions of these libraries, and sometimes becomes unintentionally dependent on version-specific bugs or peculiarities. Library ownership changes hands and the new owners decide to clean house, reorganize, or kill existing libraries. Open-source libraries branch into new flavors and those that were once a defacto standard become tomorrow’s orphan, while the new alternatives may be as similar as two sects of the Greek Orthodox Church or as different as Chinese and English. Meanwhile, applications are always being written and maintained, building up every growing and changing rat’s nests of dependency on these various libraries.
Dependency hell.
This is not unique to Java. .Net developers face “DLL hell,” but .Net developers are far more dependent on components written and maintained by Microsoft. Java is somewhat more vulnerable to it because its greatest strength–an open, platform-independent architecture–is inherently dynamic and fungible. Unfortunately, the language’s original developers didn’t foresee and/or adequately address this issue, so today we face the unenviable situation in which many existing applications cannot, for all practical purposes, be updated to the latest greatest (and more secure and supportable) libraries and language core because the change would simply be too expensive. The best solution would have been to build right into the language some mechanism for guaranteeing backward compatibility to all libraries, forever. Then the compiler could simply choose the latest version of every library it could find and it would be guaranteed to work so long as the application didn’t require functionality newer than the available version. Even that could be addressed by a centralized repository and automated system to permit the compiler to go find what it needs. Alas, that seems reasonable today (though it would be more problematic than it sounds) but it was impossible when Java first appeared.
Still, changes have been gradually made to improve the situation. One of these is that at a certain point, a rule was instituted that dependencies had to be unambiguously specified. Throughout the first 8 major releases of Java, this was not the case. You could reference two libraries, one of which had been compiled with the other, and the two different versions of that other would be equally available to the compiler and runtime. Not only would this make your application code larger, it meant that which library was used at runtime was a bit of a crap shoot. Mechanisms were in place to let you take control, but in the vast majority of cases, developers ignored them except when testing found a specific problem.
Do you know what programs that appear correct because of testing are called? Miracle programs. In theory at least, programs should be provably correct, like mathematical proofs. That’s not often the case in practice, but at the very least, we should avoid practices that necessarily increase the odds of hidden defects. Allowing random resolution of multiple library versions is one such practice, so starting in Java 9, it’s no longer permitted.
This brings me here today. I’m working on a novel design application derived from an open-source Java project originally developed in Java version 7. It contains lots and lots and lots of these “transitive dependencies” as they are called. When I try to compile it under Java 9 or higher (16 is the current version) I get almost 300 errors stating “Package such-and-such is available from more than one module…” This wouldn’t be so bad if the IDE (Eclipse) or build tool (Gradle) contained a nice little hook to let you right-click, resolve the problem and move on–you know, the way normal people would do it. But this is Java, so normal people need not apply. In fairness, it’s not JUST that the Java and open-source communities are arrogant twits who see no problem in requiring others to master arcane technical skills to accomplish simple, obvious tasks that should be baked into the interface. It’s also that very often, they also lack the resources or authority needed to bake-in such features, and that’s why we are stuck in a world dominated by Microsoft. But I digress…
MAKE A COMPLETE BACKUP OF YOUR ENTIRE PROJECT DIRECTORY BEFORE BEGINNING.
KEEP A RECORD OF EACH CHANGE YOU MAKE. YOU WON’T BE ABLE TO COMPILE AND TEST UNTIL ALL ERRORS HAVE BEEN REMOVED, SO THERE’S A POSSIBILITY OF RUNTIME ERRORS CAUSED BY CHANGES MADE TO REMOVE COMPILER ERRORS.
DO NOT TRUST SUDDEN LARGE REDUCTIONS IN ERROR COUNT. THIS MOST LIKELY MEANS ECLIPSE HAS REVERTED TO A LOWER COMPLIANCE LEVEL AND MAY BE LYING ABOUT IT. CHECK THE COMPLIANCE LEVEL. PERFORM GRADLE AND ECLIPSE CLEANS. REVERT LAST CHANGE. RESTART IF NECESSARY.
REMEMBER TO MANUALLY SAVE CHANGES TO THE BUILD FILE. FORGETTING WILL LEAD YOU ON A MERRY CHASE WHEN ERRORS DON’T DISAPPEAR WHEN YOU EXPECT THEM TO.
So here’s how to resolve the problem in Eclipse and Gradle:
- Go to the command line and change to the folder containing your application. You can grab this in Eclipse by opening Project/Properties/Resource/Location.
- Run the command “gradle -q dependencies”
- Copy the resulting report and paste it into a searchable editor (like Notepad++)
- In Eclipse, go to Project/Properties/Java Compiler and change the compliance level to Java 9. It’s probably best to leave the workspace settings alone for now, as you’ll want to switch back and forth a lot to make sure you haven’t broken anything as you proceed. At least in my version of Eclipse, the compliance level gets set back to Java 1.7 (version 7) after almost every build, for which I thus far have no explanation. Change it to 9 (or higher), apply and close, and the project will build with Java 9 compliance. You’ll see a large number of “such-and-such cannot be resolved to a type” error caused by the failure of library import statements that no longer work due to transitive (or other ambiguous) dependencies.
- Double-click on any of these errors, and a related source code (Java) file will open. Go to the top, then scroll down until you see imports that have been flagged with the message “Package such-and-such is available from more than one module…”
- If you see an import ending with a wildcard, skip it for now, find one that’s a specific type.
- Highlight the type and press control-shif-T to open the type viewer (in Eclipse). You should expect to see multiple .jar files listed. The type viewer tells you exactly where the file is, but that’s not actually helpful. Instead, you need the name of the library (group or module), usually indicated in the filename of the jar. We are going to try to eliminate from this list all but one of the dependencies, preferably one with (jre) beside it because that’s a supported part of the Java language itself.
- First, search your gradle.build file–the one written in Groovy script and containing at least a “dependencies” section and multiple “implementation” lines.
- If one of the multiple depencies is referenced here, you can try just deleting it (comment it out though, so you can bring it back if needed).
- Save the build file and use Project/Clean to clean and rebuild. If all errors dissappear, it probably reverted the compliance level to pre Java 9. Use Project/properties/Java Compiler to change it back to 9 or higher, apply and save, and see what that does.
- If the total number of errors has gone down, that’s a good sign, but even if it hasn’t, you can go back and look at the same type in the type viewer and you should see one fewer dependencies. Repeat until there is only one and the error count is down by at least one. Fortunately, fixing this issue anywhere fixes it everywhere, so the error count can go down by tens or more all at once.
- IF THE ERROR COUNT GOES UP, comment the dependency back into the build file and try something else. The idea is to find a library that contains a redundant reference to another, usually more generic library, and either eliminate it (the library containing the reference) or add an exclusion removing either the entire redundant dependency or better yet, the specific redundant dependency added by that library. Put another way, if your application requires library “A” and library “B” but is failing because “A” contains “B” and Java already contains “B”, you need to tell the runtime and build tools to ignore the version of “B” contained inside “A”.
- Search the dependency report you copied earlier for references to the problematic library. This will give you clues as to which listed dependency may be including a redundant (transitive) reference.
- If you can omit a dependency that includes the problematic transitive reference and get away with it, great.
- If you can’t, you might be able to add an exclusion at the bottom of the build file something like this one for xml-apis (though I haven’t worked out the details of how best to use this, I found it following an online thread):
configurations.all {
all*.exclude group: ‘xml-apis’
} - Or better yet, you can specifically exclude the transitive dependency brought along by some other needed library. That library will still use the code it brought with it, but the compiler and runtime won’t see it on the classpath and get confused.
// Apache Batik
// implementation ‘org.apache.xmlgraphics:batik-dom:1.8’ // CSH exclude added for Java 1.8 to 9 update
implementation(‘org.apache.xmlgraphics:batik-dom:1.8’)
{
exclude group: ‘org.apache.xmlgraphics’, module:’batik-ext’
}
And the mouse is easily swapped out for the Wacom tablet when it’s signature time:
the ErgoDox controller and instead soldered each of his two half keyboards to wireless keyboard controllers hacked from a pair of cheap Logitech wireless keyboards and synced to a Logitech Universal USB dongle. Others have mounted their ErgoDox into a homemade housing with a trackball in the middle. To each his own, literally.
