How to find which dependency is pulling in a particular class file?

Question

My project consists of some dependencies which are pulling the same common dependency.

The common dependency storm-kafka has a new version 1.0.2 and an old version 0.10.0

On building a shaded jar, I see classes from both the versions in my fat jar and somehow during execution, the older one is getting picked up which gives a ClassNotFoundError because other dependencies related to the older version are not there.

jar -xvf my_shaded_fat_jar.jar
find . -name KeyValueSchemeAsMultiScheme.class
./org/apache/storm/kafka/KeyValueSchemeAsMultiScheme.class
./storm/kafka/KeyValueSchemeAsMultiScheme.class

storm/kafka is older one and org/apache/storm/kafka is the new one I want.

Surprising part is that I do not see 0.10.0 in my ~/.m2 repo:

ls ~/.m2/repository/org/apache/storm/storm-kafka/
1.0.2
# no 0.10.0 here !

How do I debug maven to find out from where the older dependency is creeping into?

EDIT:

On running mvn dependency:tree -Dverbose -Dincludes=storm.kafka, I get:

[WARNING] The POM for org.apache.storm:flux-core:jar:1.0.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] The POM for org.apache.storm:storm-kafka:jar:1.0.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli)  ---
[WARNING] Failed to build parent project for org.apache.storm:flux-core:jar:1.0.2
[WARNING] Invalid POM for org.apache.storm:flux-core:jar:1.0.2, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] Invalid POM for org.apache.storm:storm-kafka:jar:1.0.2, transitive dependencies (if any) will not be available, enable debug logging for more details
[WARNING] Failed to build parent project for org.apache.storm:flux-core:jar:1.0.2
[WARNING] Invalid POM for org.apache.storm:flux-core:jar:1.0.2, transitive dependencies (if any) will not be available, enable debug logging for more details

Why the poms are invalid? I just cleaned my .m2 repo by removing everything in ~/.m2/repository/org/apache/storm

Answer 1

mvn dependency:build-classpath -Dmdep.outputFile=f

gives a list of all dependend jars.

now iterate of them and add their content into a file:

for f in `cat f | tr ';:' '  '`
do
    echo $f >> out
    unzip -v $f >> out
done

Answer 2

You can use the Maven tree goal to show all dependencies that are used. This prints out the full dep tree by default but you can also get it to find a particular dependency you're interested in - for example try:

mvn dependency:tree -Dverbose -Dincludes=storm.kafka

to see what's pulling in Kafka. More info can be found here:

http://maven.apache.org/plugins/maven-dependency-plugin/examples/resolving-conflicts-using-the-dependency-tree.html

Answer 3

This seems to work if you're into ammonite:

#!/usr/bin/env amm

import $ivy.`com.lihaoyi::os-lib:0.7.0`

// Run in the root of a maven project to find which jars contain the class
// Example: all_classes.sc org.redangus.core.func.Ex
// amm, mvn and jar must be in PATH

@main
def main(fullClassName: String): Unit = {
  val clazz = fullClassName.replace('.', '/') + ".class"
  println("Searching for file " + clazz)
  val file = os.root / "tmp" / s"all_classes_${System.getProperty("user.name")}"
  if (os.exists(file)) os.remove(file)
  if (os.proc("mvn", "dependency:build-classpath",
    s"-Dmdep.outputFile=$file").call().exitCode != 0) {
    println("Something is amiss")
    sys.exit(1)
  }
  var found = false
  os.read(file).split(":").foreach(f => {
    os.proc("jar", "-tf", f).call().toString().split("\n").foreach(l => {
      if (l == clazz) {
        println(s"$f")
        found = true
      }
    })
  })
  os.remove(file)
  if (! found) println("No matches found.")
}

Answer 4

Here a way to find the JAR (and thereby the Maven dependency), where a specific class files comes from for Windows10/11 "DOS" command prompt:

1. dependency:build-classpath > cp.txt (in case -Dmdep.outputFile=cp.txt does not work, like for me)
2. modify cp.txt to cp_sorted.txt (e.g. by using Notepad++):
   1. removing all non-classpath-lines (esp. those lines starting with [INFO])
   2. substituting all Classpath-Separators (;) with newlines
   3. order the one-jar-per-line-list
   4. de-duplicate the one-jar-per-line-list
3. @FOR /F %f IN (cp_sorted_test.txt) DO 7z.exe e -y -o"CLASSPATH_FULL_EXPANDED_TEST_FINAL\%~nf" "%f"
4. make a DIR /B /S for the desired class, e.g. DIR /B /S HttpServlet.class