Friday, March 02, 2007

Apache Axis and Commons HTTPClient

Someone asked me how they could turn on NT Authentication for web service using Apache Axis (the web service used by ColdFusion). By default Axis uses its own HTTP client code, org.apache.axis.transport.http.HTTPSender, to send the XML/SOAP POST requests to a web service. This uses HTTP 1.0 and generally works file.

Axis also supports the Jakarta Commons HTTPClient library, and has since 1.0. To configure Axis to use this instead of its own library you must edit the client-config.wsdd file used by Axis. It gets found on the classpath and generally you don't actually have one and the one built in to axis.jar gets used.The interesting line is the http transport. To switch Axis to use the HTTPClient jar, you would change this:

Once you have this code configured, Axis will use the HTTPClient library for it HTTP needs. Since the HTTPClient library supports NT Authentication, you just set the username/password on the Stub object as you would normally do for (say) Basic Authentication and it will just work. If you are talking to a .NET web service, you are done.

BUT switching the line in the client-config.wsdd file alone doesn't do the trick if you are talking to a ColdFusion web service. Talking to a ColdFusion web service via Apache, you will get an "411 - length required" error back that looks like this:

The Apache JRun connector doesn't allow chunked encoding without a content length (generally true for all general pupose connectors, including mod_python) and the CommonsHTTPSender class in Axis does not provide a content-length. Go figure.

Using the built-in JRun web server you get a a "content not allowed in prolog" error because it appears the JRun web service doesn't understand chunked transfer encoding. Go figure again.

I didn't try it with IIS. My guess it that it might work.

To solution? Turn off the chunked encoding, which you can do by setting a property on the web service object in CFML like this:

Wondering if you could shed any light on getting a web service to work in CFMX 7.0.2 multi instance it seems whatever we do we run into package conflicts. Ive notice some technotes on log4j and webservices.jar but they have been no help.

see stack trace below:

04/11 09:24:20 error [1]org.apache.commons.logging.LogConfigurationException: java.lang.ClassCastException at org.apache.commons.logging.LogFactory.newFactory(LogFactory.java:558) at org.apache.commons.logging.LogFactory.getFactory(LogFactory.java:345) at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:409) at org.apache.axis.components.logger.LogFactory.getLog(LogFactory.java:76) at org.apache.axis.client.AxisClient.clinit(AxisClient.java:84) at org.apache.axis.client.Service.getAxisClient(Service.java:143) at org.apache.axis.client.Service.init(Service.java:152) at com.amadeus.xml.AmadeusWebServicesLocator.init(AmadeusWebServicesLocator.java:12)

using the createObject("webservices", method we acutally run into issues compiling since the path to the stub files being created is longer than 256 characters!

so attempting to compile manually at a different location and call via createObject("java", ... causes the stack trace mentioned above.

Id love to see a post on how you would recommend you manually compile a web service?

it appears jrun has axis 1.1 installed and CF is running axis 1.2

im able to compile the stubs using the axis 1.2 libs but it appears when running the code its using the 1.1 libraries ?

This information was very helpful. Worked like a charm. My next question is, we're using WSDL2Java to generate our BindingStub file, where this code snippet belongs. Is there some way to tell WSDL2Java to generate this code (for example, to tell it that we'll be using CommonsHTTPSender), or do we have to add this code manually each time we do WSDL2Java? Thanks for a great post with a very targeted solution!

Thanks for that... I guess that is not my issue. Shoot. I'm getting a weird error when I use CFINVOKE/CreateObject

(505)HTTP Version not supported

I added the 1.1 code you suggested... The strange thing is that I have tested the service with SoapUI and I've created my own soap *message packet* and posted it via CFHTTP and both seem to get a response that looks right.

But when I use the CreateObject method or CFINVOKE tag, I get that bizarre response.

I used Charles (great little proxy) to observe the traffic, and what I see is that CF gets the WSDL happily then either doesn't auth correctly again or potentially CF doesn't understand the WSDL...

I tried switching to CommonsHTTPSender, but when I do a CFDump I get a java factory looking thing...

Would you mind explaining the 505 error to me a little more? I'm obviously forcing HTTP/1.1 with your suggested header...

So I figured out how to get a result set finally. I revamped the rather kludgy convertDotNetDataset function from Joe Rinehart.

Your blog is dope. (that's a good thing). I do have one question though. Does AXIS or CF ever plan to build in dataset conversion to accomodate M$ ever-so-idiotic format? The method I'm using is a dog. Slow slow slow.

Good thing I just sent off an email message before checking for all the comments. :-)

Anyway, yes we have talked about automatically converting the MS proprietary DataSet to our Query object. Its enhancement 64065 and its being evaluated for the next major CF release. This is no guarantee that it will get done. We are looking at enhancing web service support (updating to a newer Axis engine maybe)so I hope that we can do this also.

//without this, it can throw a "content not allowed in prolog" message when using the CommonsHTTPSender; //I had this happen to me when running against the built-in CF webserver; it did not behave this way with IIS. //found the fix here: http://tjordahl.blogspot.com/2007/03/apache-axis-and-commons-httpclient.html Hashtable headers = new Hashtable(); headers.put("chunked", "false"); headers.put("Connection","keep-alive"); _call.setProperty("HTTP-Request-Headers", headers);

Still, though, no matter what I try I get a 401 Not authenticated error when using NTLM. Basic works just fine. But noone I know can get the plugin to work using NTLM.

Do you see anything in the code that I might be missing?

Thanks a lot for taking a few minutes to look!

Also, I don't know if you'll remember this or not, but remember at Max this past year, during the "Meet the CF team" session, when Tim Buntel started fielding questions and a dude piped up and said "Thanks Tom Jordahl for all your work on Axis!".... that dude is me!

@Marc - I am not sure why you would need to hack the WSDL2Java generated code in the manner you describe.

Simple replacing the transport in the Axis client-config.wsdd should take care of most of that for you, including the NTLM negotiation with IIS. Once you use the Commons HTTP library, just setting the username and password should be enough. Perhaps you are not include the NT Domain in the username (i.e. ADOBE\tjordahl)?

I can tell you that the built=in web service in JRun/CF is not going to support this.

IN CF 7.1 I am trying to connect to sharepoint webservices. I have edited the client-config.wsdd file and restarted. I cannot create the webservice (error:cannot generate stub objects), so I cannot set ws.username and ws.password. I cannot pass wsargs (not an option in 7.1) so I am uncertain if it is even possible to accomplish this in CF 7.1. I feel that authentication is the issue here.

@Steve - My guess is that there is something about the WSDL that the sharepoint web service publishes that the Apache Axis engine in ColdFusion MX 7 (which is identical to CF8) can't generate Java code for.

I would try running the wsdl2java.exe on this WSDL by going to the runtime/bin directory of you CF install and running the command:

wsdl2java.exe -v http://url/of/wsdl

This will create a bunch of Java files, or it will generate an error. If it generates an error, then I am uncertain what I can do for you. You could try upgrading the Axis in CF (which is version 1.2.1) to the latest 1.4 release. We haven't tested this at all but it might be worth a try. You will have to update the jar files in the ColdFusion lib directory with the latest versions.

I have used Axis 1.4 as Web service Client for consuming some external webservices. I have to use a proxy host(in our network) to connect to externalservices which is currently handled by setting it in the System Properties.

Things were working fine with the defualt HttpSender which handles stuffbetween Client and the Soap Server.

I came across CommonHttpSender which can be used to to get control of somesettings like setting connections pools and Connection timeout(which i aminterested in) etc.,

When I replaced the HttpSender to CommonsHttpSender in theclilent-config.wsdd, I am facing problems to get it working..*Problem 1:*When a proxy is enabled(System Properties) and session enabled(_locator.setMaintainSession(true);) since I need to maintain session

I get NullPointerException on Line 190 in CommonsHttpSender ( booleansecure =* hostConfiguration.getProtocol()*.isSecure();) .. I don't why..:-(

I fixed(temp work around) it by extending this Sender and overrided thegetHostConfiguration() method by adding the below line explicitly config.setHost(targetURL.getHost(), port, targetURL.getProtocol()); inwhat ever the case may be.

*Problem 2:*The above work around fixes the NullPointerException but now getorg.apache.commons.httpclient.NoHttpResponseException

When I run the same program with out proxy setting (out of our network) Iget(404)Object Not Found at org.apache.axis.transport.http.CommonsHTTPSender.invoke(CommonsHTTPSender.java:218)

I am not sure why the same Service works with defualt HttpSender but notwith the CommonsHttpSender.

Is there any thing like configuration etc., which I missed out thats needsto be taken care to plug in the CommonsHttpSender.

I'm back after two months. I finally upgrade to CF 8. I am trying desperately to connect to the simple lists.asmx webservice on a sharepoint site. When I access the URL from my browser I retrieve the WSDL fine. I do not have to authenticate (I think it is using windows integrated authentication).

I found a post about the CF command line debugger causing this, so I disabled it in CF Admin.

Tried wsdl2java.exe -v http://site/lists.asmx and I got:java.net.ProtocolException: Server redirected too many times (20) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1253) at java.net.URL.openStream(URL.java:1009) at org.apache.crimson.parser.InputEntity.init(InputEntity.java:209) at org.apache.crimson.parser.Parser2.parseInternal(Parser2.java:471) at org.apache.crimson.parser.Parser2.parse(Parser2.java:305) at org.apache.crimson.parser.XMLReaderImpl.parse(XMLReaderImpl.java:442)

at org.apache.crimson.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:185) at org.apache.axis.utils.XMLUtils.newDocument(XMLUtils.java:322) at org.apache.axis.utils.XMLUtils.newDocument(XMLUtils.java:367) at org.apache.axis.wsdl.symbolTable.SymbolTable.populate(SymbolTable.jav

This is where I stopped with CF 7.1.

Some research on this leads me to believe I am still having authentication issues. One post I read said that WSDL tries once without authentication and then tries again with credentials and this causes it to bounce between an error screen and login box or something.

Just out of curiosity I did a cfhttp get to the same url with the same credentials. It returns the HTML for a HTTP Error 401.2 - Unauthorized: Access is denied due to server configuration.Any advice would be appreciated.

@Steve - First this is probably something that you want to work with Adobe support on. Failing that, you can use the Adobe forums to get help as well. Posting stack traces to my blog comments isn't the greatest. :-)

The simplest way to check that ColdFusion can get the WSDL is to use the CFHTTP tag to retrieve it. The CFHTTP code is the same code that cfobject uses to get the WSDL.

If the web service in fact *does* require NT Authentication to get the WSDL (check with a non-MS browser), this is something that ColdFusion doesn't support. I suggest saving a copy of the WSDL for the service in a local directory that is web accessible (i.e. localhost) and use that to consume the service.

If I manually remove the xsi:type="xsd:int" string and perform a manual CFHTTP post of the raw SOAP message (that I captured using getSoapRequest) , it works fine. However, this requires use of CFHTTP and not the native use of the wsdl object, which then requires me to parse the XML response myself, which is another time-consuming and error-prone task.

Things I have tried: See: http://ws.apache.org/axis/java/reference.html#WSDL2JavaReference

Modify server-config.wsdd, add the following to the <globalConfiguration><parameter name="sendXsiTypes" value="false"/>This setting seems to make no difference. The SOAP request generated by AXIS is identical regardless if the above is set to true or false. Yes, I was sure to restart the CFApp server each time I changed the above.

Things I haven’t figured out yet:Perform a _setProperty on the object before calling the method, like: optimalPayments.creditCard._setProperty("java?????.axis.????.sendXsiTypes", false); Although, I’m not sure the above will even work, see http://osdir.com/ml/text.xml.axis.user/2002-12/msg00513.html

Interesting blog :)I've tried what you have written there. But I'm still having some trouble with authorisation (Error:401) trying to get the webservices. (WSDL)It has something to do with SP giving authorization to the CF-server. But how?I can't understand what the problem exactly is.

Hey Tom, I realize this blog entry is quite old, and you've moved on to the Flex team, but if you can spare a minute, you said two things in the content/comments that seem contradictory. I wonder if you can clarify. Or I could be misconstruing things.

In the blog content, you had said, "Once you have this code configured, Axis will use the HTTPClient library for it HTTP needs. Since the HTTPClient library supports NT Authentication, you just set the username/password on the Stub object as you would normally do for (say) Basic Authentication and it will just work."

But then in the comments you say, "If the web service in fact *does* require NT Authentication to get the WSDL (check with a non-MS browser), this is something that ColdFusion doesn't support."

Oh, I think I may now realize the distinction. More in a moment, but I do have another question on all this.

You say we can "just set the username/password on the Stub object as you would normally do for (say) Basic Authentication and it will just work"

But what if we don't want to prompt the user for the password? Is there any way we can get the credentials from CF (via IIS, perhaps) to pass along on their behalf to the web service?

As for the previous comment, was your latter comment just about accessing the WSDL, not the rest of the web service? If so, I see suggestions (http://cfsilence.com/blog/client/index.cfm/2008/3/17/ColdFusionSharepoint-Integration--Part-1--Authenticating) that the solution there is to save the WSDL locally and point to that.

But even if that's solved, the key challenge is how to proceed with the windows authentication without prompting the user again for the password. It would be awesome if that's solved, whether in CF 7,8, or 9. Thanks for any thoughts.

@Charlie - You always have to have a username and password, no way around it if you are authenticating. Windows NT authentication just uses (virtually) the username and password typed in when the user logged in to Windows.

Since CF is a server, you have to provide the username and password no matter what the auth method. The only solution is to hard code them in the CFML template or prompt someone for the info.

I’d argue, though, that it would seem we wouldn't quite want to say it’s because "CF is a server", per se, since (from what I understand) an ASP.NET app can indeed use the user’s authentication (as provided to get past IIS. I realize a .NET app runs inside the IIS process, which likely explains why it can get to it. Sadly, it just seems CF has no way to get that info. I was just hoping maybe you knew something, as others have asked.

Prompting the user when they have already logged in (assuming the CF App is also behind its own IIS setup requiring windows auth/NTLM) is just something they're wanting to avoid.

People have wanted this for other things, like database queries, etc. Do you think there's ever a chance that CF would be able to get/pass along the authentication? Just seems like it shouldn't be impossible, and that JSP, PHP, Ruby, and other developers would be in the same boat so as to drive/identify a solution.

I have been searching for help on resolving this dreaded 401 error when trying to invoke a webservice. This blog so far seems to be the closest to finding a solution.

Following are some facts regarding this issue:-The webservice use basic authentication-I can use cfhttp with id and password to access the wsdl-I can use wsdl2java.exe -v -U [domain]\[username] -P [password] http://company.net:8080/app/ws/SeviceWS?wsdl to create the java files-I know I can send a request and receive a response from webservice because the target site was also set up with a non-secure webservice and we were able to communicate (it is on our intranet)

Following are what is not working:-I can't get to register WS using the CF8 admin. I get error "Error creating web service. Please ensure that you have entered a correct Web Service name or URL"-When running it from within CF using a cfinvoke and passing the username and password, even passing the username and password in the wsdl2javaarguments parameter, I still get an error. The error I get is:

I have been trying different things, even doing a cfhttp request to get the wsdl and then saving it. This then resulted in the "Parsing error: Fatal Error: URI=null Line=3: White spaces are required between publicId and systemId.". I can probably fix this by hand, but that defeats the object of automation.

After reading a lot of blogs and looking at my errors, it seems that the issue is not the http authentication to the original http://company.net:8080/app/ws/SeviceWS?wsdl, but the second request to get the schema http://company.net:8080/app/ws/SeviceWS?xsd=1 is the one that fails. Based on the error it shows that the wsdl was returned but the schema could not be located due to the same basic authentication. This sure sounds to me like ColdFusion is passing the username/password on the wsdl request but not the xsd request.

I would appreciate it if anybody can add some insight to this problem.

@vaalpens - it sounds like there is a problem retrieving the Schema that is part of the WSDL but is NOT inlined in the WSDL.

Is the WSDL using an include for this? If you can invoke WSDL2Java from the command line and it is able to read the Schema, the ColdFusion code is using the Axis API to do exactly the same thing. There could be a problem with the way CF passes the username/password down in to the stub generation code library that it uses to parse the WSDL - I believe that XML parser will have to execute an HTTP call to get the included Schema. This may be where the credentials are not reaching. Hard to say. But as I said, if you can use the Axis command line to get the WSDL, this should match what CF is doing too.

My recommendation is to get all parts of the WSDL and bring them local to the CF server. You can then register this web service in the CF administrator and use it from your applications. If and when the WSDL changes, you can refresh the service in the admin (after downloading the new WSDL/Schema) and your apps will see the new stuff.

This will remove the "getting WSDL" step from the whole process, which simplifies the whole thing.

Based on your suggestion I have decided to pull down the wsdl and xsd files and put it on my webserver. Following are the steps that I followed:-Use cfhttp to get wsdl and xsd files and cffile to save it on the webserver-Manually updated the wsdl file to xsd:import the local xsd file. (I'll probably do the replace during the import/save process at some time)-I then tried to register the web service through admin or use it with cfinvoke. Both these failed again.-This is when I realized that my webserver did not know how to handle .wsdl and .xsd files-On IIS I added the .wsdl and .xsd extensions with MIME type "text/xml". This did the trick.-I was then able to send my request and receive the response from the web service

Sorry for the above detail but hopefully it will help some other developer searching for solutions.

I probably need to contact Adobe support to find out if they know about this issue.

Hey Tom, is there any understandable reason why a webservice trough HTTPSender works and not when switching to CommonsHTTPSender?With CommonsHTTPSender I get {http://xml.apache.org/axis/}stackTrace:java.net.ConnectException: Connection refused: connect. This happens just by switching the transport value in the client-config.wsdd. I wanted to switch to commons since I'm getting 'faultString: java.net.ConnectException: Connection timed out: connect' when I need to invoke multiple times the same webservice.

I tried the solution to resolve issues with the NTLM handshake using AXIS 1.4 and it worked perfectly for me. I edited the client-config.wsdd in the axis.jar file (rather than creating another one somewhere in the classpath) and added a couple of jar files that were required to the classpath (commons-httpclient-3.0.jar and commons-codec-1.3.jar). Then I set the username and password using stub.setUsername and stub.setPassword. Remembering to add the domain at the start of the username separated with a slash. The result - Success :o) Thank You!

Hi Tom, I have to call a web service, but the program will be behind a proxy. I know that whith HttClient I can call any website outside the lan. But with axis I can't make it. The solution that you proppose I can't use it because now commons-httpclient doesn't exists anymore. How can I resolve this problema? Thank a lot, Patricio Otamendi

Well, I found the link to commons-httpclient, but it doesn't work as I spected (proxies issues). So, what I have done was to write my own org.apache.axis.handlers.BasicHandlerI extended this class, and implemented a solution using the HttClient (version 4.1.1, wich is the version that I need). Now I want to rebuild my class that extends BasicHandler to do it better. I need to call to two diferent web services, with different URLs. But my solution was to use a static HttpClient, since the class must have a constructor without parameters. Thanks for your advises, Patricio Otamendi

Hi,I have the same issue with CommonsHTTPSender as João Fernandes had:Hey Tom, is there any understandable reason why a webservice trough HTTPSender works and not when switching to CommonsHTTPSender?With CommonsHTTPSender I get {http://xml.apache.org/axis/}stackTrace:java.net.ConnectException: Connection refused: connect.I'm interested to know how he solved his issue.Thanks in advance,Iris.

// get a modifiable configuration and instantiate it with a basic client configuration config = new SimpleProvider(new BasicClientConfig()); // create a new simple targeted chain SimpleTargetedChain chain = new SimpleTargetedChain(new CommonsHTTPSender()); // and add it to the configuration config.deployTransport("http", chain); // Obtain a service object from service locators MetadataService mdService = new MetadataServiceLocator(config); DataIntegrationService diService = new DataIntegrationServiceLocator(config);