Running Robots off the Cuff

Earlier in my career, I worked on a project that integrated a robotic body-welding line for Ford Motor Company. In those days, Ford used Kawasaki robots almost exclusively. They had a fixed hardware and software interface standard for each type of robot with a long and storied history.

My project began with an attempt to sell Ford on using a different, better (but more expensive) brand of robot, and a better (but again, more expensive) pinch-welding "gun." This process proceeded apace until about the 11th hour, when suddenly some intransigence from one particular individual at Ford suddenly forced everyone to drop the original plan, cancel the original hardware orders, and order Kawasaki robots. (The better weld guns, however, were kept.)

As the designated shop-floor "robot guy" for this project, I became involved after all the hardware components had arrived, had been uncrated, and were being assembled. Innocent of the consequences of this project's sordid history, I sailed blithely into the mess and set about putting together the various bits and pieces, as I had done many times already in my short career.

It didn't take very long to discover a serious problem.

Back then, the very old, set-in-stone, Ford/Kawasaki standard used a single multi-conductor cable to carry discrete I/O signals (24V DC and even, in some cases 120V AC) from the robot cabinet, through the weld controller, and out to the weld gun. For those of you not familiar with the term, a weld "gun" is the "pinching" bit that you see in movies like Minority Report, or any PBS special about the automotive industry. We encountered our problem when we attempted to wire up the new weld guns to the old cable.

The old cable provided only two signal wires for closing the pneumatic valves that operated the weld gun. The new weld gun required four wires. And the kicker was that although the Ford/Kawasaki standard had provided for several spare wires in its youth, feature creep over the ensuing years had whittled the number of spare wires down to one.

Before calling upstairs to the design engineers responsible for this strange turn of events, I pulled all the electrical drawings: the gun, the weld controller, and the Ford/Kawasaki interface that the Kawasaki robots had been delivered with. I discovered that they didn't match. It wasn't that I was overlooking something. No, this oversight had happened in the design phase, long before I ever got involved.

It wasn't hard to figure out what had happened: When Ford had demanded the last-minute switch to their standard Kawasaki robot, the controls engineers had simply swapped the electrical drawings without thinking about the fact that the weld guns were not the same units that the standard had been designed around.

Have you noticed that it's nearly always the electrical/electronic engineer that rides to the rescue of the management or even the mechanical engineers? How many times do the mechanics solve a problem in our electronics? It's been the story of my life in engineering :-)

We can start talking trash about all EEs who inccurred emergent software hacks by failing to understand the complete system that their assigned module fit into.

2. Reminds me of a similar costly scenario I was involved in years ago, where the production mgt refused to buy the right sized (power) screwdriver for the line. Result, the screws didn't get torqued in enough, and the company paid dearly for material, labor and more while pursuing field replacements. And, yes, here too, an urgent SW hack was required to keep misbuilt hardware from doing bad things -- while still operating correctly -- until the equipment could be attended to.

3. I'm intrigued, hopefully without being nosy, about where and how the certain intransigent individual fit in things. A bean counter imported into production control? An design/production engineer who'd done great things back in the day, but having been promoted out of that scene persisted in "his way or the highway" control over his old venue? Some firebrand climber recruited from outside who wasn't going to risk slipping budget or deadlines, or crossing lines of normative behavior in order to keep his/her career on track?

Gee, I just started a commotion here last week with an email that was about the same kind'a assumptions of compatibility being made. The Ford robot replacement screw-up example is on a much grander scale, and immensely more expensive to resolve I'm sure. Still... this stuff happens at all levels.

Remember the Mars Lander where metric and standard were switched, or not switched as the case may be. When I talked to our Managing Quality Engineer Thursday about some non-updated duel use prints that I had been asked to make a fixture to match, and a separate inaccurate mass callout for the fixture that could have created havoc with our test equipment... My input was simple.

Get everybody on the same page.

And hey TunaFish... We had a situation arise here a couple years ago where someone pulled the torque rating for a standard 1.5D thread engagement for a steel screw and nut but applied that to a thin tapped copper bus bar with about 3 threads.

They didn't catch it until a whole day's worth of those units was messed up... because not every one of them was stripping outright. They thought it was the tapped bus bars' fault until someone checked the recommended torque for 3 threads in copper...

Well, I'm a firm believer in hardware/software co-design. I spent 15 years or so doing logic (chip or card) design and can remember probably half-a-dozen instances where the software/firmware folks bailed the hardware guys out. Even a couple where The hardware folks told the software folks exactly what needed to be done.

I am not an Engineer. I am an Electronics Technician. What this generally means its that I get stuck with the stuff an engineer screwed up and have to make it work anyways, after engineering has used up all the time for the job. It has happened many times. The engineers have been told how important they are. They are rather well paid. But it comes down to some poor sap who has to get the product working when the delivery date has already been missed because the engineers didn't get their part done on time.

Delivery date is 12 weeks out. Assembly and testing will take 3 weeks. Any special parts will need to be orderred no later than 6 weeks out. So, Engineering comes out at 10 weeks and decides all this is rush because they waited until 8 weeks to figure out what was going to change to meet the customers requirements (which had been decided 6 MONTHS ago) and so the special parts won't be in until week 13. But it is the shops fault that the machine is late even though the engineers didn't have ANYTHING ready until week 10 of 12.

The person with the least control over the project is often the one with the most responsibility dumped on them. And could have probably had the new requirements designed at week 2 because he actually has to work with the finished product and the customer.

Yeah, I tend to (well HATE may be a bit strong...) dislike many engineers because they are so far removed from the final product that they don't seem to have much of an idea what is really involved in executing the idea. It works on paper...

Better treat the Engineering TECHNOLOGISTS with a lot of respect as they are generally the ones who find and solve the problem while engineering rides off on their white horses to joust with another windmill...

You paint "engineers" with a very broad brush. I, for one, reject your argument & your thesis. I have been practicing the art & science of electro-mechanical design engineering for the better part of 50 years, and I can assure you that in every company that I've been employed, ALL the (degreed) engineers were VERY intimately knowledge of EVERY piece of equipment or product that left their cubicle & went into production. And, EVERY engineer was directly responsible for EVERY shred of paper relevant to the project. Now, this was true in my experience whether the company had 2 engineers or 22 engineers! And, I can further assure you that EVERY engineer in every environment also had a toolbox w/ all sorts of "goodies" packed into them. Not one fellow engineer was an "Ivory Tower" fellow, who had only two items.... his slide rule & a pencil sharpener!!!!

There's enough blame to spread around, no doubt. As a field engineer the bane of my existance used to be the sales reps. They would sell a system and promise the customer that their new equipment would not only perform flawlessly, but it would also fly around the room, deliver the daily paper, lay an egg, and fry it up for their breakfast. Fast forward past the contract signing to the day the equipment arrives onsite and yours truly is now tasked with the responsibility of getting it up and operational. As the aforementioned system is being brought up and configured the customer's rep will inevitably walk into the room and want to see some flying and egg laying, whereupon it became my job to A.) Keep the stunned, bemused and disgusted "they've done it to me again" look off my face, and B.) Try to carefully explain to the customer what the equipment will and will not do in the real world, not "Salesman Nirvana". I've noticed a consistent reaction amongst customer reps: They inevitably try to explain to you how you are wrong, and what their 'special' package will allow them to do (along with dark rumblings of how they have hinged their company's future on that specific non-existant feature), and, once you have gotten finished explaining the facts to them, they get on the phone with your upper management and demand that your company send out another, more experienced tech who knows how to configure the aviation and egg production features... in short, it's MY fault and couldn't possibly be that the sales rep might have stretched the truth just a little bit....

Vyper3000; As a former electro-mechanical technologist, and now a journeyman electrician and millwright, I have had my share of run-ins with electrical engineers, mechanical engineers, salesmen, saleswomen, managers, and even other electricians and millwrights. The bottom line that I go by is: If I fixed the problem, I get to say what the problem was. And in Automotive especially, I have seen a lot of incompetence in engineers promoted to senior positions.

Being jaded and cynical is not always a bad thing. It had taught you to have an answer before the question was asked. Curious that your hack was adopted in the company with the new weld guns instead of requesting the extra wires on the Kawasaki robots to use the new weld guns with all of their features.

I've had to deal with both types in my long career over DECADES. I WILL say it: The MAJORITY of the new, freshly minted so-called " engineers " barely make it out of the 90 day evaluation period. The ones so arrogant that they think their **** doesn't stink usually gets the door before they screw up a profitable, working company.

ThereWAS NOT a formal title of TEST ENGINEER when I started work in Silicon Valley; there was NOT a real career path at the SJSU School of Engineering, just Power Transmission and Electronic Controls ( like the old E/M stuff ). Many people come out of those types of programs with the same ARROGANCE that they had to endure as students.

I was very naif about how much POLITICS was part of the unofficial coursework. It wasn't about " Learning How to Learn " which was my reason to carry coursework along with my full time job at AMD. I was learning FAR more from AMD than the stuff I was exposed to at SJSU.

This finally came to a head when I took a course in MatSci as part of the Engineering Common Core. I listened to the prof for about half the period when the subjet of creating semiconductor materials was being LECTURED to all of the unknowning students. I made the mistake of CONTRADICTING the IGNORANT prof by saying " We don't do that anymore ". The upshot was that I only got a " D " handed down and I said " enogh of this ****, dropped out and went to Foothill College where learning was far more enjoyable and more well rounded.

Meanwhile, I was offered a management job and a start up the corporate ladder. I never wanted a desk job, but that was where the money was at that time.

The gotcha in all this: Many Engineers BROKE the factory floor test systems and it was our duty to repair and re-calibrate by the next morning's 1st shift. Myself and several others ( still with the title and semi-non-exexempt status of Senior Test Tech ) were responsible for keeping the Test Floor WEEKLY goals no matter how much equipment got broken.

I'll skip over the 30% raise job hopping or the " training " of a DeVry grad that fit the above description. and get to the reasons why the Cray research job was the prize many people wanted but never got.

Many of the skills that Cray Research needed were not taught in any formal Education system at that time. I was told that " We dont have time to retrain someone out of Grad Schol and have to dealing with an " unlearn " environment. After I put together a homemade Test System out of spare parts and a visit to Kiethley Instuments, I was gven my first goal. Repeatable numbers critical to Cray Hardware Design. ( I still refuse to make those numbers public ). THAT got me my first NDE title, I have many others. I've taken two management jobs and regretted both of them.

When you take any job as a contractor, your best information about a person is their " ME WALL " . The most important people in a company usually do not have one. They don't need a reminder that they are GOOD at what they do...

I can agree with this one wholeheartedly. It's amazing to me how engineering "types", myself included, get to solve and / or fix a great number of problems created by 1.) Budgetary restraints, 2.) Management, generally non-technical management, 3.) Extremely short time frames, 4.) "The boys in engineering will handle that" mentality, 5.) That's not my responsibility, etc etc. Luckily, this problem could be addressed and solved. Some cannot and with this being the case, the poor engineer and / or technician usually takes it on the chin. My career spans about 40 years; 1966 to present day and this seems to never change. Some how, "technical types" are expected to have all of the answers. In a perfect world, this would not happen.

I can not recall any story that has brought forth so much finger pointing and downright braggadocio. In the old days we called it, "Doing our job". The whole story that started this string was about rigging incorrect items to perform a function at a level below what they should perform. That should have triggered a response to purchasing to get the right equipment: end of story. Instead the company has been left with items that are probably unrepairable and ill preforming. Who cares if it was a degreed engineer, a technician or a passing janitor. Enough of the name calling.

Every company is set up with different levels of responsibility and we are all over worked and under paid. This whole discussion turned into who could tell the biggest tale first. We all have jobs to do and probably do them to our best ability and with pride. And I would be willing to bet everyone reading this has commited at least one error, who somebody else had to fix and could be the subject of a new story.

There certainly is a wide spread of experiences that different folks have had with different engineer attitudes. Unfortunately the field has produced a few "Prima Donnas", but they are not representative of the whole herd, trust me on that. Just like preachers and teachers, there are all kinds.

I have always held that the way to be a good engineer is to have some level of competence in all of the areas that your overall product includes. That is the way that the software designer can understand the molding issues when the code winds up needing another memory chip, and the PCB thus needs to grow a bit. I also believe that engineers must be able to repair the products they design, at least a few of them. Not only does that provide insights toward improving reliability, but it also serves to provide an understanding about serviceability. And as for those organizations that design products that should be repairable but are not: I Hope you go broke! OUt of business! Closed down for good!

About those who neglected to verify that the previous design would support the new hardware, it would not hurt them at all to spend a few shifts putting new cables on those robots. Generally there is lots of room to work on the third maintenance shift. And it is a great education.

Focus on Fundamentals consists of 45-minute on-line classes that cover a host of technologies. You learn without leaving the comfort of your desk. All classes are taught by subject-matter experts and all are archived. So if you can't attend live, attend at your convenience.