WS-Deathstar for the REST of Us: A Story of Ruby, WSDL, and Salesforce

May 5, 2008

UPDATE: (Jan. 13, 2011) As so often happens in the Ruby world, this blog entry is out of date. I’ve heard reports about various SOAP libraries no longer working with Ruby 1.9, but I haven’t had the need to test them out. If you’re working with Salesforce and Ruby, development dm-salesforce seems to have carried on, meaning that you won’t have to do most of these steps.

I’m going to leave this up for posterity and to help anyone who may need to dig around in the undocumented parts of any of these libraries. If you are one of those unlucky people, please understand that you have my condolences and best wishes for success.

You say, “I use Ruby! We’re all RESTful around here, man! My webapp consumes and produces YAML and JSON with no angle-bracket residue (it’s better for the environment). Our Kool-Aid’s delicious!”

I know — it really, really is.

A long time ago (yesterday), I thought I was safe. Then I had to integrate an internal webapp with Salesforce.

Me: “Do they have a way to do feeds? JSON? Anything?”

Internet: “SOAP.”

Me: “That’s for those Java and C# people! Not hip cats like me.”

Internet: “Try our SOAP. It’s delicious.”

It really, really wasn’t.

Someone asked me, “Why not use Activesalesforce?” Sure, some of these problems could have been avoided that way. However, my app doesn’t use Rails or ActiveRecord. I figured I was going enough against the grain that I should just strike out on my own. And WSDL makes it easy (or just possible), right?

So, as half cautionary tale / half survival guide, I give to you my tale of SOA-WOE. Presented below are the steps I went through to get Ruby and Salesforce’s WSDL to play along… well, not nicely, but at least… well, let’s just leave it at play.

…And before I get into it, let me preface by saying that a lot of my problems look silly now. The answer are, of course, in the documentation. But separating the wheat from the chaff requires knowledge that most people don’t have when they’re starting out tabula ras(o)a.

By the way, if you’re reading this for generic Ruby/WSDL and not Salesforce information, you can skim the first three steps.

1. Signup

As all hip web developers have, I’ve read Getting Real. So I learned that you’re supposed to lower the amount of effort it takes for a user to get on board. Having to sign up just to see the API docs and wiki is not in that vein. It’s sort of the first signpost on the road that we’re treading into unfamiliar territory. Well, I guess them’s the rules.

2. Get your Security Token

So yeah, here’s another thing that is not necessarily straightforward, but I guess there’s a reason in this case: you have to have a Security Token to append to your password when logging in through SOAP. Overall, it’s probably a good idea.

Didn’t notice that? Neither did I. But don’t feel so bad, neither did this guy.

To get your Security Token, go to the Setup link at the top of the page, find the “My Personal Information” section and click on the “Reset your security token” link. It’s around there somewhere.

3. Get your WSDL

Now look on the left column under “App Setup”. Click “Develop” and find the “Download your organization-specific WSDL” link.

Look at all the choices! Well, which one should you use?

You could do what I did:

Stare at the options for a little while

Say “Awh, fuck it.” at a volume that may or may not have been heard by my fellow cube-sters (it probably was).

Download “Enterprise WSDL”. Because I’m feeling enterprisey.

4. Install the SOAP4R gem

From what I’ve gleaned, SOAP4R is the same as the SOAP standard library. By why not get up to date with the latest and greatest:

gem install soap4r

5. Run wsdl2ruby

The example linked above builds the SOAP driver at runtime, which can take awhile. His method is good for playing around, but you should probably just go ahead an build them.

wsdl2ruby.rb --wsdl ~/Desktop/enterprise.wsdl.xml --type client

Change your paths accordingly. This will produce 4 files in your current working directory: default.rb (which does something), defaultDriver.rb (which does something else), defaultMappingRegistry.rb (which also does something, but not the same thing as the other somethings), and SforceServiceClient.rb (which does nothing).

So you’ve got these files. Don’t edit them! Someday you will run that code generator again and you’ll come to rue it. Some even say don’t version control them. Instead they’d put the wsdl under version control and add a Rake task to run wsdl2ruby (this assumes that wsdl2ruby won’t break in the future and it installed on any future machines). Choose for yourself.

7. Maintain the Session ID

Most of the search results in the old Google say this has something to do with the “Lock sessions to the IP address from which they originate” box being checked (which is located under “Administration Setup” -> “Session Settings”). This might be your problem, but in my case it was just that I didn’t know how to set headers in SOAP4R.

SOAP4R uses “header handlers” to modify outgoing and read incoming headers (and note that these are SOAP headers, not HTTP headers), so you’ll have to write one of your own (syntax highlighted version):

Conclusion

Ok, so a lot of these problems are because I was impatient to get started. There were API documents with Java examples I should have been reading! Sorry, I thought we were working on the Web here, where HTTP is king. I know HTTP. I know a couple of serialization formats. I should be good to go, right?

If you’re a seasoned SOA-type then a lot of what I’m complaining about probably seems like old hat to you. Sure, I’m whiny. Kids like me don’t know how good they have it!

Still, I can’t help thinking that there’s no reason for the extra overhead that comes with WSDL and SOAP. It might be that I work with dynamic languages most of the time. I miss my REST. I miss my YAML and JSON. I wanna go home.

I’ve spent this week looking deep into the bowels of SOAP for integration with Amazon’s Merchant@ program. Since discovering that the endpoint in the documentation (ending in .com) isn’t the right one (should have used .co.uk) I’ve been wading through the SOAP4R source to try and get SOAP with Attachements (SwA) working with Amazon’s particular take on it.

I ended up creating a SOAP::Filter::Handler to deal with adding and extracting attachments from requests and responses. (And editing those generated files to boot.)

The problems really boil down to a lack of documentation on both the service provider and the library. Luckily the source to SOAP4R is available with (somewhat simplistic) examples include to get you decent search terms to key into the library proper.

SOAP itself is a fine protocol and in many ways, it’s SUPERIOR to RESTful JSON/XML. With a nicely defined WSDL, you’ll know that property “foo” exists in the message, that it’s a a String, and that’s 40 characters long. There is no Schema-Definition equivalent in REST, and you need a pre-arranged agreement of what will be exchanged across a RESTful endpoint. (JSON, XML, image-file data, etc.)

Your frustrations come from the fact that WSDL2Ruby is not capable of gleaning a WSDL for these schema cues, and is the root cause of all your pain.

For other platforms like .NET and Java, there are fantastic tools (WSDL.exe, WSDL2Java) that generate complete classes off of the WSDL; and it works beautifully.

Well, yeah. That’s true. But it wouldn’t in Java or .Net either with their “better” tools.

Look at my two biggest “gripes”: 1) resetting endpoint URL and 2) session ID. How does WSDL generate code to handle this? It doesn’t. If it did, Salesforce’s “Quick Start” wouldn’t have us manually changing the endpoint and setting up session handlers.

How would a RESTful client handle this? By using HTTP. I’d be redirected to the new endpoint (if that was even necessary). The session id would be a standard cookie (although there are RESTfuller ways of doing this). Hundreds of millions of user agents do exactly this everyday when they check their Gmail, buy things from Amazon, or do pretty much with authenticating a user. That’s more users than Salesforce’s WS API is ever going to have.

So, after I’ve set up the appropriate credentials, I can GET /AndrewO/contacts.json. I can do this from a commandline or from pretty much any programming language with minimal library support.

… and with no code generation. It’s not necessary.

Because we have a simple interface. Four methods for interface is all we need (some say two).

… and documentation. But less of it. Because we don’t have to explain the interface — the HTTP libraries do that for us.

Ok, so with interface issues dealt with, what about data schema? No one says that you can’t use XML Schema, Relax NG or any other validation on a RESTful message. But here’s why you don’t see that as often in RESTful services: valid data starts with the developer. You have to _tell_ the programmer what everything means. There’s no strong AI that’s going to figure out what the symbols mean on their own.

Clients that adapt to API changes are both undesirable and a pipedream. Data schema changes have repercussions for both the client and the server and the developers are going to have to be involved whenever that happens.

… and that point even ignores the argument that you can’t completely represent all of the data rules using any of the popular SOA schema. What about data constraints that have nothing to do with the message, but with the environment? “Field X must be unique across the application.” “Return Y if some other external service is inaccessible.” “Only change this between 12am and 3am (the witching hour).”

But somehow these constraints continue to be enforce. How? Because they can all be programmed into the application.

Now, here’s the kicker: *The only valid description of an application is the application itself.* Everything else is best-efforts. Some applications are easier to read thanks to their host languages, libraries, and architecture (but that’s another discussion altogether).

If we focus on making our (flawed) descriptions human focused (simple interface, easy serialization formats), we’re going to get better software than if it were written by machines (code generation).

@Laurence: BTW, I meant to start off with this, but forgot to add it: thanks for commenting. This entire article stems from the fact that the Ruby community is pretty monolithic when it comes to ROA vs. SOA. Although I disagree adamantly, I’m glad the contrary point of view here.

THANKyou! I’ve been struggling with this for hours!
I hate SOAP so much you can call me a hippy, but this has made the unbearable workable.
BTW — ActiveSalesforce is currently broken, so that isn’t an option. It returns an error for the first call to each kind of object that you have to swallow with a rescue, thus requiring additional calls to the slow API.
Thanks again,
Doug.

@Doug: Thanks! Glad to see that after this long, there’s still something useful to be had. Just, FYI: I’ve been using DM-Salesforce (available on Github) in a recent project and it’s going reasonably well. I’ve had to pull some hair to get a couple of the API calls to work, but the standard CRUD stuff works fine and is probably worth checking out if you’re able to use DataMapper.

[…] the Betfair API. Problem is, given the general disdain of the open source community for the whole WS-clusterfuck, there aren’t really any good SOAP libraries for dynamic languages around. I had a bad […]

Uh-oh… Looks like this is going out of date. I guess progress is great until it leaves you behind.

The last SF integration project I did, I used dm-salesforce which seemed to use most of the same strategies as I did in this article. However, I believe it was based on the SOAP4R library as well, so it looks like we’re out of luck there. I’ll keep you posted if I come up with anything, but for now I’m sorry to say I don’t have a good answer for you.

I had problems getting inbound headers to work as expected with the above code.

As a workaround, I used the following client_auth_header_handler.rb for reading/storing/sending my auth “Ticket” (or could be sessionid) in the headers of each soap call.. note the different method name + args used on the inbound callback:

While doing some generic Ruby WS research (needed to quickly test if a certain WS is delivering what the company needs), I stumbled upon this article and loved it, both the details and the style. Since then I have been trying soap4r, savon, handsoap. None of them impressed me, which kinda sucks because I love Ruby. :/

My concrete problem was the WS-level security — but I won’t go into details. The WSDL itself hat to be downloaded from an SSL location, which I did successfully, but then the WS-level security basically killed my desire to move on.

In short: have any of you guys have any update on a decent Ruby gem (or native library even, since one can always pack a gem which depends on one) which does support both Ruby 1.8 and 1.9 and delivers not only the basic SOAP stuff, but also has some coverage for the WS security?

True indeed. Sorry if I caused you to waste some time. I added a block to the top of the article so people know how old it is.

IIRC, I had moved over to dm-salesforce at some point after writing this. I had some problems with making API calls that didn’t map to Datamapper’s adapter… I think getting the changed records, but the specifics escape me. I ended up having to add some hacky code that dug down into the SOAP interface and called other methods. I tell you, it’s inescapable sometimes…

But you’re right: dm-salesforce is a valid option and probably fits the majority of use-cases best.

No apology needed, this article is still totally relevant. I walked through it and was able to get a good soap connection to salesforce, with a couple modifications to the files that were generated by wsdl2ruby.

I didn’t notice your references to DM until a day later. DM is not that it’s a complete necessity for salesforce api, but it is definitely very nice to have such a well developed solution that we can just add in. It makes querying SF much easier and rails-like.

Thanks very much for the article. I was about to give up on salesforce until I found it.

[…] Hi All, I am not a Ruby guy, but last few days i had to look into this language, moreover in the context of calling apex web service method using Ruby. I love doing this using java and php, even i did in python also, but this time its a challenge for me to achieve this. Its a challenge because lack of documentation and sample codes on net and as a newbie in Ruby world. As every new developer i also searched many times on net but was struggling sometimes setting up Ruby, sometimes rubygems, sometimes things were not working. I left many times work in middle long before also. But searching and searching and i found this blogpost by Andrew very helpful. https://hideoustriumph.wordpress.com/2008/05/05/ws-deathstar-for-the-rest-of-us/. […]