Posted
by
kdawson
on Tuesday December 09, 2008 @03:23PM
from the neither-snow-nor-rain-nor-storm-of-net dept.

m2pc writes "The US Postal Service is experiencing major server issues for its shipping API web services. After spending about an hour debugging my own eCommerce software for a client, I found the problem was with the USPS shipping servers being unavailable. Further research showed that message boards for OS Commerce and other e-Commerce packages are filling with posts from angry users who are experiencing crashing Web store applications and frustrated customers. Developers are scrambling to find interim solutions, from hard-coding fixed price shipping, to 'rolling their own' shipping calculation APIs based on the USPS Fixed Rate Zone Tables, to disabling the USPS option altogether. One user reported yesterday that a call to USPS yielded the response 'we expect it to be down all day.' As of 9:20 AM PST the service is still unavailable."

If it's important enough they should have a backup. I know it's not always within budget, but their website doesn't have to crash because the USPS service is unavailable, it could simply fall back to an acceptable solution, such as a fixed price, an estimation table, or an error message that says the USPS shipping estimator is down and shipping will be calculated at the time of shipment, or they can try again later. Yeah, they'll lose some business, but they can lose less business with a decent backup.

Part of the issue is the USPS servers are not failing consistently. Sometimes they allow a connection and respond so slowly to requests that the ecommerce app. times out. Sometimes they allow no connections and return nothing. Sometimes they are simply returning shipping rates of 0.00. So any live "backup" would have to allow for all that, at least.

"While in principle I agree with that, what are they supposed to do? They are quoting you a price for a service they don't provide themselves."

Fail gracefully. For example, state that they are unable to price that shipping option at that time. Offer you to accept the option without knowing the price, select another type of shipping, or get an email when they know.

In any third party connection for an application I've built, I typically consider the implications of such web service/outside API going down, even if I label it "non vital, don't worry about it" to "have a backup plan". Oh well, no sympathies here

To be fair, what sort of "backup" calculation would you have done here, short of reverse-engineering the USPS algorithm for calculating shipping rates?

I'm not usually a rabid free-market libertarian, but this here can be seen as a result of the fact that the USPS isn't really beholden to its customers. Can you imagine FedEx or UPS being afflicted by such an issue? And, if they were, would they blow off inquiries with a glib, "We expect the servers to be down for the rest of the day?" Of course not, because, for FedEx, UPS, DHL, et. al. such an outage directly affects the health of the organization. If people can't calculate shipping rates, they can't ship, and if they don't ship, the company doesn't make money. The close linkage between revenue and working services tends to put more impetus behind keeping things working and making sure that they get fixed quickly if they do happen to go down.

Why don't more companies just use the old "Shipping & Handling" charge to cover the costs of shipping without having to calculate the rate the the USPS is going ot charge them. Charge a fee that is close to what you expect to pay. Over on a few and under on a few, but should even out at the end.

Because it's no longer reliable enough. The S&H charges change based on too many values for the few factors (carrier, size, weight, destination, type of delivery, timing of delivery, etc.), and things change. So it's not a necessarily easy calculation to perform.

The zones are clearly marked but determining which zone you are shipping to is not. They are based on distance from the originating ZIP Code so the application must use Postal Explorer to generate the zone chart ahead of time. (For now, it happens to be working.) And in the case of non-flat-rate Priority Mail, there's a huge price difference between zone 4 (zone and weight only) and zone 5 (zone and weight *OR* zone and size).

and if the livelihood of my business depended on it you better be damn well sure I'll have my application ready to crunch each of these factors manually in determining shipping cost...it's terminally stupid to do otherwise...

To be fair, what sort of "backup" calculation would you have done here, short of reverse-engineering the USPS algorithm for calculating shipping rates?

A store can charge whatever they want for shipping. So if they know how much it costs on average to ship their products, they can simply charge everyone that flat rate. Or if they sell something simple, they can charge one simple rate per quantity. Or they could cache the last rates they charged and reuse those. Or they could offer everyone free shipping and eat it as the cost of doing business with the USPS.

If TFSummary is to be believed, the least they could do is make sure the app didn't crash on bad data returned from an API. There is NO excuse, ever, for a crashing bug in an online store app like that. There's no problem with the USPS servers being funky, there's a problem when people programming dependent apps somehow didn't at least plan on it ever happening.

... [F]or FedEx, UPS, DHL, et. al. such an outage directly affects the health of the organization. If people can't calculate shipping rates, they can't ship, and if they don't ship, the company doesn't make money.

The same has been true with the USPS since 1970.
Their entire budget is financed by people buying stamps and other services.
They don't get a dime of taxpayer money.

The same has been true with the USPS since 1970. Their entire budget is financed by people buying stamps and other services. They don't get a dime of taxpayer money.

Except, they get to charge an exorbitant amount for their stamps and services by prohibiting competition - a luxury that Fedex and UPS do not have. (If you think their costs are reasonable, then why is there a law the prohibiting Fedex/UPS from charging below a certain amount?)

"Americans are pathetic sometimes -- they expect their government services to do as well as private industry, yet they don't give them the ability to charge what private industry charges. Amtrak is a similar situation, Amtrak is expected to be cash flow positive, yet they are not allowed to own their own tracks, those are owned by the freight companies, whereas their main competitors run on highways that are paid for completely by the taxpayer and gas taxes, or operate out of airports also funded by taxpaye

And if the processor is offline, they deal with it. Store the info until it can be charged, whatever. Crashing or otherwise malfunctioning in the face of 3rd party inaccessibility is definitely an app problem, and never excusable.

And how, pray tell, are you to calculate USPS' shipping rate without relying on the USPS' server?

I suppose you could construct a database of USPS rates for every possible box size and weight you might ship, but that would be a maintenance nightmare as you'd have to update it whenever they change their rates and every time you get a new item in stock.

The building prominently bears the inscription: Neither snow nor rain nor heat nor gloom of night stays these couriers from the swift completion of their appointed rounds, which is frequently mistaken as an official motto of the United States Postal Service. It was actually supplied by William Mitchell Kendall of the firm of McKim, Mead & White, the architects who designed the Farley Building and the original Pennsylvania Station in the same Beaux-Arts style. The sentence is taken from Herodotus' Histories (Book 8, Ch. 98) and describes the faithful service of the Persian system of mounted postal messengers under Xerxes I of Persia. The USPS does not actually have an official motto or creed, but nonetheless the inscription on the building is often recognized as such. The inscription was carved by Ira Schnapp, who later designed the Action Comics logo and many other iconic logos for DC Comics.

and the USPS even used it in their own television spot [wikipedia.org] (albeit in an altered form):

The commercial, which ran after the September 11, 2001, attacks and the anthrax mailings, featured no voiceover, only the following text interspersed on title cards:We are mothers and fathers. And sons and daughters. Who every day go about our lives with duty, honor and pride. And neither snow, nor rain, nor heat, nor gloom of night, nor the winds of change, nor a nation challenged, will stay us from the swift completion of our appointed rounds. Ever.[2]

Note that the [2] link in the above quoted Wikipedia article takes you to this page [usps.com] on the USPS site quoting the exact text of the tvspot.

Yes, they have no Motto, but considering how much they use the meme, they might as well have one.

They never matched the quotes from their desktop app which prints the shipping labels.

Also, at the end of the month, they append a "fuel charge" that is completely unpredictable and not represented in the cost at the label printing time in any way, shape or form. You can't derive an estimate for it from the price of oil, from the price of fuel, from last month... no way to do it that I know of, and UPS agrees. Best thing you can do is take last month's charge and distribute it across your best estimate

Only a really terrible developer would hit a web services API and not code for it to fail. No one should expect a third party service to be up 100%. The apps should fail gracefully. Anyone finding their e-commerce software handling this situation poorly should find another package.

If a store offers only the USPS delivery method and the web service is down, the user could be directed to call the sales number to place their order. If the store offers other deliver methods the store front could instruct the user that USPS isn't currently available and they must choose another method.

osCommerce is well know to be a bowl of spaghetti code and in general a P.O.S. Zen Cart forked a long time ago and did a complete code re-write. osCommerce continues to keep their heads up their asses, and only a fool doesn't look for alternatives at the earliest opertunity.

There is almost no end to how true parent is.
I was tasked by a friend to find a solid, open source webshop for him (a startup clothing business), and while (admittedly lazy) browsing around, I found that osCommerce was well used so I assumed this was a good idea, and recommended it to him. Not a day later he called me up again to help him out with adding a few plugins (very simple things as well, and also quite popular). I assumed this was because he didn't know php very well, so I said sure, but when I s

osC is insidious. You install it because it's open source and a zillion other sites are using it, so it can't be THAT bad, right? Then you have to install a few contributions to add the basic functions you need - and with each contribution, adding the next gets harder and harder.

But you get up and running. Then you discover all of the little annoying usability quirks - US state names don't use proper abbreviations, customers in Ireland can't place an order without a post code even if you set the minimum length to zero, the payment option selection looks like it has a default selected when it doesn't, users don't know that 'authorize.net' is actually the credit card payment option, and so on.

So you spend hours fixing those. Meanwhile, you've got the database populated with all of your products, and you're using it to track sales figures. Maybe you add some back-end order processing stuff for your own convenience.

Eventually this mass of haphazardly patched spaghetti code becomes an absolute nightmare to do ANYTHING with, but by this point you've spent so many hours working on it that it's easier to just keep slogging away at it than to just abandon the whole thing, install something new, and try to move your catalog and your sales records over to the new system.

Only a really terrible developer would hit a web services API and not code for it to fail. No one should expect a third party service to be up 100%. The apps should fail gracefully. Anyone finding their e-commerce software handling this situation poorly should find another package.

I have developed a fulfillment system using UPS so I have some experience with this. This was a few years ago, but at the time, UPS's system was designed on the assumption that a shopping cart system would contact UPS during the order placement process (or checkout). USPS appears to be very similar. This is incredibly moronic, because it means that if their server doesn't respond or any reason, you simply lose the sale. Problems AFTER the order has been accepted are far less serious - that just means fulfillment is interrupted - you don't lose orders. It would be even better if you could print the labels without having to reach their server, but they don't provide this (UPS's own software, however, can.)

The only reasonable way to do it (obviously?) is to have your own shopping cart software calculate the shipping rate. At the time I did this in 2001, UPS did not offer reasonable data to let you do this. Their rating system is extremely complicated, and the spreadsheets they gave me had clearly been maintained by hand and needed a lot of massage to get them into a reasonably parseable format. Not only that but it was very difficult to get these tables in the first place, requiring multiple escalations through tech reps and ultimately a "god damn it just give him the tables" from our sales rep.

I imagine things might have improved a little since then and I know there are shopping carts that can now do the rate calculations internally. But if any shipping company forces its customers to contact a server to calculate shipping costs, then they are just as retarded as the merchants who use them (for a web checkout).

Most e-commerce sites, including the ones I've built, have a multi-step process to place an order. If the step for shipping (or tax, etc.) failed, then the system would re-load the page with any appropriate messages and option changes. So, for example, if the third-party payment processing service didn't respond before a timeout the user would be asked to call customer service to complete their order. The sales rep could see the order in its incomplete state and finish it over the phone.

Most e-commerce sites, including the ones I've built, have a multi-step process to place an order. If the step for shipping (or tax, etc.) failed, then the system would re-load the page with any appropriate messages and option changes. So, for example, if the third-party payment processing service didn't respond before a timeout the user would be asked to call customer service to complete their order. The sales rep could see the order in its incomplete state and finish it over the phone.

Well, yes. All of the e-commerce sites I've worked on had in-house or outsourced phone staff. But my point is that a site must have some sort of failure mode. Whether it's to call customer service, give free or flat shipping, or whatever, it's better that the customer can complete the sale than see an arcane error message that the system has failed.

Retail customers expect to check out knowing the full price they will be paying, including shipping. This means that you HAVE TO calculate shipping during order placement (checkout). You're asking for customer service headaches if you have to manually charge shipping after the customer has already (they thought) completed the order.

So your only option, if you don't want to rely on a third-party server for shipping costs, is to have your shopping cart do its own calculations.

OR you could attempt to contact the remote services servers. If that fails you calculate as best as possible. If after the calculation you find the shipping was actually more then you eat the cost. If it was less then you charge the customer less when you run their card through the batch commit at the end of the night.

And what happens when they change those tables? Sounds like you're asking for a world of business hurt down the road with the costs to update the code and cover the changed price.

The tables don't change very often. When they need to do a quick rate bump it is usually in the form a simple "fuel surcharge". Even when the tables are updated, it can usually be very closely approximated by applying a simple percentage increase.

Also, most sites put a significant markup on shipping (that's the "handling" part), so

This is a great example of why SAS (Software as a Service) in its current form will eventually fail. The very nature of the Internet is to be disconnected and stateless. If there is no guaranteed delivery at the 5 - 9's level (99.999%), then how can business expect to depend on the service? Mind you, I don't have a better model, but we had better come up with one if we intend to continue using the Internet for commerce!

There is a better way. Code your own backup solution, revert to it when the SaS you are counting on fails. In this case, that would be a static shipping 'estimate' process that substitutes data when the USPS service fails. Writing code what depends on a third party is dangerous. If your business depends on that software you are fooling yourself if you think it won't fail. It will. This is where coding disaster recovery functions into your system is important.

Our website has been suffering from time outs and dropped connections to the UPS rate calculator web service as well since yesterday. Seems to be intermittent, refreshing the page seems to help and it will eventually connect. Luckily none of our customers see the problem as our sales tools are all internal. Happy Holidays!

I came in yesterday morning to find the USPS module non-functional. Worse, the only working option was DHL overnight - and in case you've missed the news, DHL is now about an order of magnitude worse than the post office for domestic delivery. Even for places they say they can do next-day delivery to, actual delivery can take more than a week.

Why? Because they hand it off to the post office rather than deliver it themselves. Why it takes the post office a week to deliver it when I can get it there in two days by sending it by priority mail myself is a mystery. In any case, DHL's out of the (US) domestic game entirely next month.

My site was up last time I checked, but if the USPS option goes down again, I think it's time for a 'free economy shipping' promotion. No messy rate tables to deal with!

It's the same for USPS package services (anything above First-Class Mail). You have to generate a zone chart from your originating ZIP Code before determining shipping rates. Priority is based on zone and weight or, in some cases, zone and size. Parcel Post is based on zone and weight, but some ZIP codes that are major hubs offer cheaper "Intra-BMC" rates.

posting anonymously to cover my ass... We upgraded to MySQL 5.1 last week. We had some major table corruption and no backups. Sadly, no one will be fired or even reprimanded for any of this major cluster fuck.

The company I work for provides a hosted e-commerce shopping cart solution, SEO-Cart, which supports the USPS Web Tools. Of course the first call coming in for the day was from a client using USPS and having incorrect shipping prices being calculated for their store.

I went ahead and called USPS and the lady who answered was quite rude and explained to me that they had a Worldwide outage which affected other applications than just their Webtools API, and also that they hire a 3rd party company to handle their Webtools API software. She couldn't provide any other information at all and I told her a company of that size should have some sort of fail over plan in place to prevent them from being down as long as they have been. I was really disappointed in the fact she didn't even ask me for my name, phone number, or company by time the conversation was over, but she was probably being bombarded with phone calls all day.

After figuring out that USPS was completely down, I looked through our fail over code and found the following equations seem to come close to the USPS pricing:

National shipping: [cart-weight]*1.6+3.00
International shipping: [cart-weight]*1.6+15.00

These also include pricing for insurance.

After tweaking the fail over pricing code to this, it seemed that everyone using USPS were happy with the results. We also had to decrease the connection timeout set for the request to the USPS Webtools API which was also slowing things down.

The Webtools API seems to be both up and down today, with some orders having shipping prices directly from USPS and others having the fallback pricing. Either way, hopefully their IT department learns from this and also provide us information as to what exactly went wrong.

On that note, this is a prime example that I use when speaking to prospects about the advantages of using a hosted shopping cart solution rather than a licensed/free download solution. Besides the obvious IT benefits that you get with a good hosted shopping cart solution, hosted shopping cart software is typically a centralized application that can provide quick updates to problems like these. Of course this is assuming that the prospect is serious about their online store and doesn't want to handle technical support themselves.

Apart from the trouble reported in this/. article which I found occuring on one of the existing sites I wrote yesterday (simply because there were no USPS prices being returned, no error, but took about 30 seconds to work out what happened), USPS simply sucks ass.

Here's why:

Some time ago, they had an API to get rates, it was called RateV2.Then they "updated", and now have RateV3.RateV3 is the only specification published.To get access to the Rate API servers, you must first test your implementation against thier testing servers successfully, when they see that they let you on the production server....

Thier testing servers only work with a limited version of RateV2.

So, in order to use the USPS API, you must:

Write to the now unpublished RateV2.
Test that RateV2 on the test servers.
Ask USPS to allow you to use Production (and get the keys etc) because you have successfully tested.
Write completely new code against RateV3.
Test that RateV3 on the production servers.

And if you try and show the USPS staff the logical problem in this process, they will reply "I can not put you on production servers, until you have done three successful tests on the test servers".

The call from the User to the parent poster m2pc could not have been a fun one. But some interesting Ah-Ha's can be gleaned from this event. "Try-Catch-Finish" can be a troubling paradigm, but when your user employs the administrative technique of MBH, ( Management By Hysteria ), having your program tell you what its inputs were, and what the response was from the "other guy's" server in a nice log file can be; satisfying. This problem could have been reduced further by having the program email any nonst

Um... The post office is private, but protected. Even the best hardware and software fails on occasion. That was my point earlier - we need a better service model if we expect SAS to work.
If the small Government crowd had their way the post office would still be the Pony Express.
Let's see you get a letter acorss the country in a week for less than 50 cents. UPS and FedX can't - they get $5 for the smallest item. So much for off loading services to business!

Let's see you get a letter acorss the country in a week for less than 50 cents. UPS and FedX can't - they get $5 for the smallest item. So much for off loading services to business!

First of all, Fedex and UPS are legally prevented from setting a rate below a certain number ( I don't remember what that is).
Secondly, Fedex and UPS are not allowed to deliver normal mail - only 'urgent packages'. That is why you see all the fancy mailers and boxes from them and not a simple envelope that will keep costs d

The government can't even manage to keep a simple web service online, and people still believe that it would be wise to let them control health care.

Get real. On several occasions, I've had to manually intervene to fix idiotic billing f*ckups between my PRIVATE insurer and a PRIVATE hospital, who had entered into mutual contracts to be in the same "network". For some reason, they couldn't get their own computers to talk to each other and I had to fix their bugs by going deciphering cryptic paper printouts myself and wasting hours calling customer service. This kind of stupid private healthcare IT problem happens routinely to millions of people every year. Therefore, using your reasoning I conclude that due to a clear history of incompetence, it is unwise to let private parties handle health care, and such practice should be banned.

Did you get the security guy who came in and told you not to surf porn on company time, then ramble about how the USPS computer network was more important than the DOD? I got that, and they tried to do good cop/bad cop on us - it was comical.