Doctor completes liver surgery with aid of iPad augmented reality app

A German doctor surgeon has leveraged the power of Apple's iPad and an accompanying augmented reality app to assist him in the completion of a surgery, marking one of the first times a procedure has been performed in such a manner.

Using an iPad, surgeons in Bremen, Germany, were able to create a virtual 3D pre-op plan for one patient's procedure, reportsReuters. [Warning: Some graphic images.] The surgeons took a picture of the patient's liver with the iPad's camera. The app then constructs an augmented reality overlay of the liver, showing the physicians where essential structures such as tumors and blood vessels lie.

The augmented reality overlay is built off of scans of the organ that were performed prior to the surgery. Having the actual layout of the organ visible before beginning allows surgeons to avoid potential complications and could result in more efficient operations.

The team behind the app sees it potentially having further applications beyond just the liver. It could also be used to help in excising tumors from the pancreas and other organs.

While this is one of the first times an iPad has played such a role in the actual process of a surgery, Apple's popular tablets and smartphones have become quite popular among doctors. Apple devices are the top choice among physicians, and those doctors are continually finding ways to integrate them into their daily practices.

When I go to the doctor and see them using a large screen tv in the exam room I just want to scream at the chunkiness of it. I have several gripes about medical record keeping in the USA including: why do they make it nearly impossible for the patient to have copies of their own records - we paid for the test, give us the results; why aren't records portable - keeping the records to themselves (again which we paid for) results in needless repeated tests or missed diagnosis; why do we have to wait for the doctor no matter what time the appointment is - if we make them wait, we are charged.

When I go to the doctor and see them using a large screen tv in the exam room I just want to scream at the chunkiness of it. I have several gripes about medical record keeping in the USA including: why do they make it nearly impossible for the patient to have copies of their own records - we paid for the test, give us the results; why aren't records portable - keeping the records to themselves (again which we paid for) results in needless repeated tests or missed diagnosis; why do we have to wait for the doctor no matter what time the appointment is - if we make them wait, we are charged.

The problem is the system the hospital uses, like EpicCare, that is difficult for the hospital personnel to use, especially with Macs and iPads. Yes, they supposedly have the ability to work with iPads but through a Citrix-type interface, which doesn't give them everything they can do on a stupid PC. We should be able to have instant access to our records but as long as the hospital system providers bas everything on an insane system built around Windows, we won't. For those who think I hate Windows, listen to my adult child complain about their documentation system crashing all the time and having to wait for a PC tech to fix them.

When I go to the doctor and see them using a large screen tv in the exam room I just want to scream at the chunkiness of it. I have several gripes about medical record keeping in the USA including: why do they make it nearly impossible for the patient to have copies of their own records - we paid for the test, give us the results; why aren't records portable - keeping the records to themselves (again which we paid for) results in needless repeated tests or missed diagnosis; why do we have to wait for the doctor no matter what time the appointment is - if we make them wait, we are charged.

Dave, did you read the March 4th Time Magazine Cover story? The longest story (36 pages) and the only edition to ever sell out. It's called "Bitter Pill!"

When I go to the doctor and see them using a large screen tv in the exam room I just want to scream at the chunkiness of it. I have several gripes about medical record keeping in the USA including: why do they make it nearly impossible for the patient to have copies of their own records - we paid for the test, give us the results; why aren't records portable - keeping the records to themselves (again which we paid for) results in needless repeated tests or missed diagnosis; why do we have to wait for the doctor no matter what time the appointment is - if we make them wait, we are charged.

Depending on the state, you may have the right to see and copy your medical records. Check with your doctor or a lawyer to see if that is the case where you live.

If I understand correctly... They took those scans previous to the operation and then during the operation when they cut them open they used AR to tie into those scans to better see whats going on.

This is very likely correct. My wife had deep sinus surgery in which her head was imaged with CT first to map her sinuses using a tight-fitting headpiece that allowed the coordinates to be reproduced later. The doctor used an AR computer screen during the surgery next to the surgical table. It was undoubtedly more expensive than an iPad, but somehow cheaper non-bespoke medical technology rarely translates into lower medical bills.

The person who mentioned a stand above: crikey, I hope this picture was just posed to show off the tablet. You'd think an assistant could at least hold the thing. Having a tablet you don't want to drop wrapped in slippery plastic in one hand and a knife inside the patient in the other seems daft.

I wonder why they would not use traditional CT and MRI for their work.

If I understand correctly... They took those scans previous to the operation and then during the operation when they cut them open they used AR to tie into those scans to better see whats going on.

Oh, so in other words the iPad now has access to the PAX network that always existed in the OR. Instead of viewing the imaging on a monitor, the surgeon can now view it on an iPad. What is the advantage with the iPad in this situation?

This is incredible stuff. If Apple come come up with an iOS-driven heads-up display (like Google Glass), that would really take off in areas like this.

Even just working wi google glass would be tons useful for this kind of thing. Split second response is vital at those kinds of times.

What they would likely need is a combo though. Many times surgeons doing that kind of work use magnifying lenses so the 'google glass' part would be perhaps just on top of that. But I feel sure that someone would make a model of that type if the support was there. Show vital or even just a red/yellow/green. Diagrams even if they are just there as a safety blanket and so on

Imagine firefighters going into a building being able to see a diagram of the layout and where they are, tracking locations of team mates and talking to them in real time. Both groups might find being able to record what's happening for later review useful even if done via streaming to a pocket device or nearby computer. For learning or legal

It was undoubtedly more expensive than an iPad, but somehow cheaper non-bespoke medical technology rarely translates into lower medical bills.

A lot of families have run into a similar game. For years parents have fought with insurance companies over high priced, devoted devices for assisted communication. With companies refusing to cover the costs of basic computers that would do the job for cheaper because they could be used for games etc. Came back up again with the iPad when AAC apps like proloquo came out. Insurance companies would rather spend 10k on a big bulky thing than $1k for an iPad, case/mount, apps.

Even just working wi google glass would be tons useful for this kind of thing. Split second response is vital at those kinds of times.

Surgeons are already wearing protective glasses often equipped with microscopes. Google glass would not work in that situation. Besides the current vitals monitors are visible to everyone on the team most often monitored by others not the surgeons themselves.

When I go to the doctor and see them using a large screen tv in the exam room I just want to scream at the chunkiness of it. I have several gripes about medical record keeping in the USA including: why do they make it nearly impossible for the patient to have copies of their own records - we paid for the test, give us the results; why aren't records portable - keeping the records to themselves (again which we paid for) results in needless repeated tests or missed diagnosis; why do we have to wait for the doctor no matter what time the appointment is - if we make them wait, we are charged.

Just insist on the data. I live on both sides of the pond, and in Europe I get the results without asking, in the US they treat patients like imbeciles, so I have to put up a fight to get the data, but I refuse to leave before getting what I want and paid for.
Similarly, in Europe I get meds in factory original packaging, sealed and with documentation on side effects, ingredients, etc. at a fraction of the cost of the same stuff that I get in the US out of bulk containers in generic packaging without any documentation at much higher cost. US healthcare is way overrated.

Oh, so in other words the iPad now has access to the PAX network that always existed in the OR. Instead of viewing the imaging on a monitor, the surgeon can now view it on an iPad. What is the advantage with the iPad in this situation?

Obviously you can use the built in camera to provide a real time overlay.

Better than my Bose, better than my Skullcandy's, listening to Mozart through my LeBron James limited edition PowerBeats by Dre is almost as good as my Sennheisers.