I found exactly the same problem when running Presto(an open source SQL engine built on top of Hadoop) against NativeS3FileSystem. Could not get it fixed now. Currently I am using jets3t 0.6, which is using httpclient 3.1.

Just wonder, would update to jets3t 0.9 get the problem fixed?

Thanks,Zhenxiao

> Deadlock with FSInput and Hadoop NativeS3FileSystem.> ---------------------------------------------------->> Key: AVRO-1144> URL: https://issues.apache.org/jira/browse/AVRO-1144> Project: Avro> Issue Type: Bug> Components: java> Affects Versions: 1.7.0> Environment: Hadoop 1.0.3> Reporter: Shawn Smith> Assignee: Scott Carey> Fix For: 1.7.5>> Attachments: AVRO-1144.patch>>> Deadlock can occur when using org.apache.avro.mapred.FsInput to read files from S3 using the Hadoop NativeS3FileSystem and multiple threads.> There are a lot of components involved, but the basic cause is pretty simple: Apache Commons HttpClient can deadlock waiting for a free HTTP connection when the number of threads downloading from S3 is greater than or equal to the maximum allowed HTTP connections per host.> I've filed this bug against Avro because the bug is easiest to fix in Avro. Swap the order of the FileSystem.open() and FileSystem.getFileStatus() calls in the FSInput constructor:> {noformat}> /** Construct given a path and a configuration. */> public FsInput(Path path, Configuration conf) throws IOException {> this.stream = path.getFileSystem(conf).open(path);> this.len = path.getFileSystem(conf).getFileStatus(path).getLen();> }> {noformat}> to> {noformat}> /** Construct given a path and a configuration. */> public FsInput(Path path, Configuration conf) throws IOException {> this.len = path.getFileSystem(conf).getFileStatus(path).getLen();> this.stream = path.getFileSystem(conf).open(path);> }> {noformat}> Here's what triggers the deadlock:> * FSInput calls FileSystem.open() which calls Jets3t to connect to S3 and open an HTTP connection for downloading content. This acquires an HTTP connection but does not release it.> * FSInput calls FileSystem.getFileStatus() which calls Jets3t to connect to S3 and perform a HEAD request to get object metadata. This attempts to acquire a second HTTP connection.> * Jets3t uses Apache Commons HTTP Client which limits the number of simultaneous HTTP connections to a given host. Lets say this maximum is 4 (the default)... If 4 threads all call the FSInput constructor concurrently, the 4 FileSystem.open() calls can acquire all 4 available connections and the FileSystem.getFileStatus() calls block forever waiting for a thread to release an HTTP connection back to the connection pool.> A simple way to reproduce the problem this problem is to create "jets3t.properties" in your classpath with "httpclient.max-connections=1". Then try to open a file using FSInput and the Native S3 file system (new Path("s3n://<bucket>/<path>")). It will hang indefinitely inside the FSInput constructor.> Swapping the order of the open() and getFileStatus() calls ensures that a given thread using FSInput has at most one outstanding connection S3 at a time. As a result, one thread should always be able to make progress, avoiding deadlock.> Here's a sample stack trace of a deadlocked thread:> {noformat}> "pool-10-thread-3" prio=5 tid=11026f800 nid=0x116a04000 in Object.wait() [116a02000]> java.lang.Thread.State: WAITING (on object monitor)> at java.lang.Object.wait(Native Method)> - waiting on <785892cc0> (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)> at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518)

This message was sent by Atlassian JIRA(v6.1.5#6160)

NEW: Monitor These Apps!

All projects made searchable here are trademarks of the Apache Software Foundation.
Service operated by Sematext