Dependency conflicts on commons-httpclient:commons-httpclient:jar, leading to inconsistent program behaviors

[HelloCoCooo]

Hi, there are multiple versions of commons-httpclient:commons-httpclient:jar in MiA-mahout-0.5. As shown in the following dependency tree, according to Maven's dependency management strategy, only commons-httpclient:commons-httpclient:jar:3.0.1 can be loaded, and commons-httpclient:commons-httpclient:jar:3.1 will be shadowed.

Your project references the method <org.apache.commons.httpclient.ContentLengthInputStream: int available()> via the following invocation path, which is included in the shadowed version commons-httpclient:commons-httpclient:jar:3.1. However, this method is missing in the actual loaded version commons-httpclient:commons-httpclient:jar:3.0.1. Surprisingly, it will not cause NoSuchMethodError at rumtime.

<org.apache.mahout.clustering.syntheticcontrol.kmeans.Job: void run(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Path,org.apache.mahout.common.distance.DistanceMeasure,int,double,int)> C:\Users\Flipped\.m2\repository\org\apache\mahout\mahout-examples\0.5\mahout-examples-0.5.jar
<org.apache.mahout.utils.clustering.ClusterDumper: void printClusters(java.lang.String[])> C:\Users\Flipped\.m2\repository\org\apache\mahout\mahout-utils\0.5\mahout-utils-0.5.jar
<com.google.common.io.AppendableWriter: void close()> C:\Users\Flipped\.m2\repository\com\google\guava\guava\r03\guava-r03.jar
<org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream: void close()> C:\Users\Flipped\.m2\repository\org\apache\hadoop\hadoop-core\0.20.2\hadoop-core-0.20.2.jar
<org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore: void storeFile(java.lang.String,java.io.File,byte[])> C:\Users\Flipped\.m2\repository\org\apache\hadoop\hadoop-core\0.20.2\hadoop-core-0.20.2.jar
<org.jets3t.service.S3Service: org.jets3t.service.model.S3Object putObject(org.jets3t.service.model.S3Bucket,org.jets3t.service.model.S3Object)> C:\Users\Flipped\.m2\repository\net\java\dev\jets3t\jets3t\0.7.1\jets3t-0.7.1.jar
<org.jets3t.service.S3Service: org.jets3t.service.model.S3Object putObject(java.lang.String,org.jets3t.service.model.S3Object)> C:\Users\Flipped\.m2\repository\net\java\dev\jets3t\jets3t\0.7.1\jets3t-0.7.1.jar
<org.jets3t.service.impl.soap.axis.SoapS3Service: org.jets3t.service.model.S3Object putObjectImpl(java.lang.String,org.jets3t.service.model.S3Object)> C:\Users\Flipped\.m2\repository\net\java\dev\jets3t\jets3t\0.7.1\jets3t-0.7.1.jar
<org.apache.commons.httpclient.ContentLengthInputStream: int available()>

By further analyzing, I found that the caller org.apache.mahout.clustering.syntheticcontrol.kmeans.Job: void run(Configuration, Path, Path, DistanceMeasure, int, double, int) would invoke the method InputStream.available() defined in the superclass of ContentLengthInputStream (ContentLengthInputStream extends InputStream) with the same signature of the expected callee, due to dynamic binding mechanism.

Although the actual invoked method belonging to InputStream has the same method name, same parameter types and return type as the expected method defined in its subclass ContentLengthInputStream, but it has different control flows and different behaviors. Maybe it is buggy behavior.

Solution:

Use the newer version commons-httpclient:commons-httpclient:jar:3.1 to keep the version consistency.

Dependency Tree----
[INFO] com.manning:mia:jar:0.1
[INFO] +- org.apache.mahout:mahout-core:jar:0.5:compile
[INFO] | +- (org.apache.mahout:mahout-math:jar:0.5:compile - omitted for duplicate)
[INFO] | +- org.apache.hadoop:hadoop-core:jar:0.20.2:compile
[INFO] | | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | | +- commons-httpclient:commons-httpclient:jar:3.0.1:compile
[INFO] | | | +- (commons-logging:commons-logging:jar:1.0.3:compile - omitted for conflict with 1.1.1)
[INFO] | | | - (commons-codec:commons-codec:jar:1.2:compile - omitted for conflict with 1.3)
[INFO] | | - commons-codec:commons-codec:jar:1.3:compile
[INFO] | +- commons-dbcp:commons-dbcp:jar:1.2.2:compile
[INFO] | | - (commons-pool:commons-pool:jar:1.3:compile - omitted for conflict with 1.4)
[INFO] | +- commons-pool:commons-pool:jar:1.4:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.6.0:compile
[INFO] | +- commons-lang:commons-lang:jar:2.4:compile
[INFO] | +- org.uncommons.watchmaker:watchmaker-framework:jar:0.6.2:compile
[INFO] | | +- (org.uncommons.maths:uncommons-maths:jar:1.2:compile - omitted for duplicate)
[INFO] | | - com.google.collections:google-collections:jar:1.0-rc2:compile
[INFO] | +- com.thoughtworks.xstream:xstream:jar:1.3.1:compile
[INFO] | | - xpp3:xpp3_min:jar:1.1.4c:compile
[INFO] | +- org.apache.lucene:lucene-core:jar:3.1.0:compile
[INFO] | +- org.apache.lucene:lucene-analyzers:jar:3.1.0:compile
[INFO] | | - (org.apache.lucene:lucene-core:jar:3.1.0:compile - omitted for duplicate)
[INFO] | +- org.apache.mahout.commons:commons-cli:jar:2.0-mahout:compile
[INFO] | +- org.apache.commons:commons-math:jar:2.1:compile
[INFO] | - commons-collections:commons-collections:jar:3.2.1:compile
[INFO] +- org.apache.mahout:mahout-core:test-jar:tests:0.5:test
[INFO] | +- (org.apache.mahout:mahout-math:jar:0.5:test - omitted for duplicate)
[INFO] | +- (org.apache.hadoop:hadoop-core:jar:0.20.2:test - omitted for duplicate)
[INFO] | +- (commons-dbcp:commons-dbcp:jar:1.2.2:test - omitted for duplicate)
[INFO] | +- (commons-pool:commons-pool:jar:1.4:test - omitted for duplicate)
[INFO] | +- (org.slf4j:slf4j-api:jar:1.6.0:test - omitted for duplicate)
[INFO] | +- (commons-lang:commons-lang:jar:2.4:test - omitted for duplicate)
[INFO] | +- (org.uncommons.watchmaker:watchmaker-framework:jar:0.6.2:test - omitted for duplicate)
[INFO] | +- (com.thoughtworks.xstream:xstream:jar:1.3.1:test - omitted for duplicate)
[INFO] | +- (org.apache.lucene:lucene-core:jar:3.1.0:test - omitted for duplicate)
[INFO] | +- (org.apache.lucene:lucene-analyzers:jar:3.1.0:test - omitted for duplicate)
[INFO] | +- (org.apache.mahout.commons:commons-cli:jar:2.0-mahout:test - omitted for duplicate)
[INFO] | +- (org.apache.commons:commons-math:jar:2.1:test - omitted for duplicate)
[INFO] | - (commons-collections:commons-collections:jar:3.2.1:test - omitted for duplicate)
[INFO] +- org.apache.mahout:mahout-math:jar:0.5:compile
[INFO] | +- (org.apache.commons:commons-math:jar:2.1:compile - omitted for duplicate)
[INFO] | +- org.uncommons.maths:uncommons-maths:jar:1.2:compile
[INFO] | +- (com.google.guava:guava:jar:r03:compile - omitted for duplicate)
[INFO] | +- org.apache.mahout:mahout-collections:jar:1.0:compile
[INFO] | - (org.slf4j:slf4j-api:jar:1.6.0:compile - omitted for duplicate)
[INFO] +- org.apache.mahout:mahout-math:test-jar:tests:0.5:test
[INFO] | +- (org.apache.commons:commons-math:jar:2.1:test - omitted for duplicate)
[INFO] | +- (org.uncommons.maths:uncommons-maths:jar:1.2:test - omitted for duplicate)
[INFO] | +- (com.google.guava:guava:jar:r03:test - omitted for duplicate)
[INFO] | +- (org.apache.mahout:mahout-collections:jar:1.0:test - omitted for duplicate)
[INFO] | - (org.slf4j:slf4j-api:jar:1.6.0:test - omitted for duplicate)
[INFO] +- org.apache.mahout:mahout-utils:jar:0.5:compile
[INFO] | +- (org.apache.mahout:mahout-core:jar:0.5:compile - omitted for duplicate)
[INFO] | +- (org.apache.mahout:mahout-math:jar:0.5:compile - omitted for duplicate)
[INFO] | +- org.apache.solr:solr-commons-csv:jar:3.1.0:compile
[INFO] | +- (org.slf4j:slf4j-api:jar:1.6.0:compile - omitted for duplicate)
[INFO] | +- org.slf4j:slf4j-jcl:jar:1.6.0:compile
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.6.0:compile - omitted for duplicate)
[INFO] | | - commons-logging:commons-logging:jar:1.1.1:compile
[INFO] | - (org.apache.lucene:lucene-core:jar:3.1.0:compile - omitted for duplicate)
[INFO] +- org.apache.mahout:mahout-examples:jar:0.5:compile
[INFO] | +- (org.apache.mahout:mahout-core:jar:0.5:compile - omitted for duplicate)
[INFO] | +- (org.apache.mahout:mahout-math:jar:0.5:compile - omitted for duplicate)
[INFO] | +- (org.apache.mahout:mahout-utils:jar:0.5:compile - omitted for duplicate)
[INFO] | +- org.apache.lucene:lucene-benchmark:jar:3.1.0:compile
[INFO] | | +- (org.apache.lucene:lucene-core:jar:3.1.0:compile - omitted for duplicate)
[INFO] | | +- (org.apache.lucene:lucene-analyzers:jar:3.1.0:compile - omitted for duplicate)
[INFO] | | +- org.apache.lucene:lucene-highlighter:jar:3.1.0:compile
[INFO] | | | +- (org.apache.lucene:lucene-core:jar:3.1.0:compile - omitted for duplicate)
[INFO] | | | +- (org.apache.lucene:lucene-memory:jar:3.1.0:compile - omitted for duplicate)
[INFO] | | | - org.apache.lucene:lucene-queries:jar:3.1.0:compile
[INFO] | | | +- (org.apache.lucene:lucene-core:jar:3.1.0:compile - omitted for duplicate)
[INFO] | | | - jakarta-regexp:jakarta-regexp:jar:1.4:compile
[INFO] | | +- org.apache.lucene:lucene-memory:jar:3.1.0:compile
[INFO] | | | - (org.apache.lucene:lucene-core:jar:3.1.0:compile - omitted for duplicate)
[INFO] | | +- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] | | | - (commons-logging:commons-logging:jar:1.0.3:compile - omitted for conflict with 1.1.1)
[INFO] | | +- (commons-collections:commons-collections:jar:3.2.1:compile - omitted for duplicate)
[INFO] | | +- org.apache.commons:commons-compress:jar:1.1:compile
[INFO] | | +- commons-digester:commons-digester:jar:1.7:compile
[INFO] | | | +- (commons-beanutils:commons-beanutils:jar:1.6:compile - omitted for conflict with 1.7.0)
[INFO] | | | +- (commons-logging:commons-logging:jar:1.0:compile - omitted for conflict with 1.1.1)
[INFO] | | | +- (commons-collections:commons-collections:jar:2.1:compile - omitted for conflict with 3.2.1)
[INFO] | | | - xml-apis:xml-apis:jar:1.0.b2:compile
[INFO] | | +- (commons-logging:commons-logging:jar:1.1.1:compile - omitted for duplicate)
[INFO] | | - org.apache.lucene:lucene-xercesImpl:jar:3.1.0:compile
[INFO] | +- (org.apache.lucene:lucene-analyzers:jar:3.1.0:compile - omitted for duplicate)
[INFO] | +- (org.uncommons.watchmaker:watchmaker-framework:jar:0.6.2:compile - omitted for duplicate)
[INFO] | +- org.uncommons.watchmaker:watchmaker-swing:jar:0.6.2:compile
[INFO] | | +- (org.uncommons.watchmaker:watchmaker-framework:jar:0.6.2:compile - omitted for duplicate)
[INFO] | | - (org.uncommons.maths:uncommons-maths:jar:1.2:compile - omitted for duplicate)
[INFO] | +- (org.slf4j:slf4j-api:jar:1.6.0:compile - omitted for duplicate)
[INFO] | +- (org.slf4j:slf4j-jcl:jar:1.6.0:compile - omitted for duplicate)
[INFO] | - net.java.dev.jets3t:jets3t:jar:0.7.1:compile
[INFO] | +- (commons-codec:commons-codec:jar:1.3:compile - omitted for duplicate)
[INFO] | +- (commons-logging:commons-logging:jar:1.1.1:compile - omitted for duplicate)
[INFO] | - (commons-httpclient:commons-httpclient:jar:3.1:compile - omitted for conflict with 3.0.1)
[INFO] +- com.google.guava:guava:jar:r03:compile
[INFO] +- org.apache.thrift:libthrift:jar:0.6.1:compile
[INFO] | +- (org.slf4j:slf4j-api:jar:1.5.8:compile - omitted for conflict with 1.6.0)
[INFO] | +- (org.slf4j:slf4j-log4j12:jar:1.5.8:compile - omitted for conflict with 1.5.11)
[INFO] | +- (commons-lang:commons-lang:jar:2.5:compile - omitted for conflict with 2.4)
[INFO] | +- junit:junit:jar:4.4:compile
[INFO] | +- javax.servlet:servlet-api:jar:2.5:compile
[INFO] | - org.apache.httpcomponents:httpclient:jar:4.0.1:compile
[INFO] | +- org.apache.httpcomponents:httpcore:jar:4.0.1:compile
[INFO] | +- (commons-logging:commons-logging:jar:1.1.1:compile - omitted for duplicate)
[INFO] | - (commons-codec:commons-codec:jar:1.3:compile - omitted for duplicate)
[INFO] +- org.slf4j:slf4j-log4j12:jar:1.5.11:compile
[INFO] | +- (org.slf4j:slf4j-api:jar:1.5.11:compile - omitted for conflict with 1.6.0)
[INFO] | - log4j:log4j:jar:1.2.14:compile
[INFO] +- org.apache.hadoop:zookeeper:jar:3.3.1:compile
[INFO] | +- (log4j:log4j:jar:1.2.15:compile - omitted for conflict with 1.2.14)
[INFO] | - jline:jline:jar:0.9.94:compile
[INFO] | - (junit:junit:jar:3.8.1:compile - omitted for conflict with 4.4)
[INFO] +- org.twitter4j:twitter4j-stream:jar:2.2.3:compile
[INFO] | - org.twitter4j:twitter4j-core:jar:2.2.3:compile
[INFO] - commons-io:commons-io:jar:2.0.1:compile

Thanks!
Best regards,
Coco

[HelloCoCooo]

The code snippet of <org.apache.commons.httpclient.ContentLengthInputStream: int available()> in verison commons-httpclient:commons-httpclient:jar:3.1 ----

commons-httpclient:commons-httpclient:jar:3.0.1 does not contain <org.apache.commons.httpclient.ContentLengthInputStream: int available()>.

The code snippet of <org.apache.commons.httpclient.InputStream: int available()> in version commons-httpclient:commons-httpclient:jar:3.0.1----

The method <org.apache.commons.httpclient.ContentLengthInputStream: int available()> in verison commons-httpclient:commons-httpclient:jar:3.1 deals with more cases and adds more control branches, which changes the control flows and data flows. So referencing <org.apache.commons.httpclient.InputStream: int available()> in version commons-httpclient:commons-httpclient:jar:3.0.1 by dynamic binding, may lead to inconsisitent semantic behaviors.

Using the following test case to run on these two versions of methods separately starting from the entry method javax.swing.ProgressMonitorInputStream.ProgressMonitorInputStream(component, object, InputStream) in your project, then we can find that variable javax.swing.ProgressMonitorInputStream.monitor.Max is assigned different values.
Test (2).txt

Please check whether the changes of this variable value will affect your semantic behaviors.

Thanks again.

[HelloCoCooo]

@tdunning May I pull a request to fix it?

[tdunning]

I am no longer maintaining this code. It depends on an ancient version of mahout that is no longer supported by the community.

If you would like to propose a pull request, feel free. I will look at it and merge it if it looks reasonable.

[HelloCoCooo]

@tdunning Thank you very much.

[q494165244]

@HelloCoCooo I have encountered the same problem when using MiA.
I just noticed something strange happened in the [min, max] range of operation progresses.
Thanks for your provided test case. It helped me reproduce this issue.
By amazing coincidence, I got the similar outputs as your test.