This probably should be critical as it would OOPS today; it also has nothing to do with bzr's xmlrpc usage - note the *internal* name: this is the codehosting use of the internal xmlrpc service (part of our existing SOA infrastructure).

You can grep on carob to find an oops; if there isn't one that should also be a bug (critical too) as thats crucial signal for us.

I think the ssh server probably wants to fail fast here and let the client try again (or not). However, the 503 may be due to asking a server that is being deployed to at the time, in which case, until we have the graceful-deploy tickets completed, this will be a rare, but expected situation.

One possible oops for this is OOPS-2022I35, where it's timing out after 9s updating the branch last-mirror time. It doesn't seem like it ought to be slow so perhaps the database was heavily loaded or locked at the time it happened...

I don't at the moment see any failures from the ssh front end but I may just have not looked.

I got possibly this error using "Bazaar (bzr) 2.5.1" pushing existing source to a new branch. It then left the target directory in a broken state. Trying the same push again to a different title then suceedeed (so the error appears to be ephemeral).

edA-qa@silent-red:/opt/leaf/primordial_type$ bzr push lp:~eda-qa/leaflang/primordial_type
bzr: ERROR: Server sent an unexpected error: ('error', 'xmlrpclib.ProtocolError', '<ProtocolError for xmlrpc.lp.internal:8097/codehosting: 502 Bad Gateway>')
edA-qa@silent-red:/opt/leaf/primordial_type$ bzr push lp:~eda-qa/leaflang/primordial_type
bzr: ERROR: At lp:~eda-qa/leaflang/primordial_type you have a valid .bzr control directory, but not a branch or repository. This is an unsupported configuration. Please move the target directory out of the way and try again.
edA-qa@silent-red:/opt/leaf/primordial_type$ bzr push lp:~eda-qa/leaflang/primordial_types
Using default stacking branch /+branch-id/725500 at chroot-87084816:///~eda-qa/leaflang/
Created new stacked branch referring to /+branch-id/725500.