Virtual Fossils

CGSociety talks to three artists who have a different take on modelling creatures. Working for the Idaho Virtualisation Lab they scan and model the most ancient of creatures and reproduce them physically and digitally to share knowledge and aid research about our distant past.

What is the The Idaho Virtualization Laboratory and what do you guys actually do?

Robert, Nick, and Jesse: The Idaho Virtualization Laboratory (IVL) is an integrative, multidisciplinary research laboratory at Idaho State University (ISU) and is a research unit of the Idaho Museum of Natural History. The laboratory houses technology for the digitization of museum and research specimens using surface scanners and virtualization software.

Our main goals are to democratize science, facilitate scientific research and collaboration, improve education and outreach, and better preserve natural and cultural history collections stored in museums and other scientific repositories. We do this through the use of modern informatics and 3D visualization techniques.

The IVL is funded by private foundations, the Idaho Museum of Natural History, and the National Science Foundation. We are proud to be a world-class facility dedicated to developing three-dimensional technologies to solve scientific and investigative questions. We also aim to make specimen data and entire museum repositories available to the world.

The IVL produces extremely high-quality virtual files of objects or specimens and we are renown in the field of heritage digitization for the advances we have made in high accuracy texture-mapping and for developing scanning protocols that optimize protection for the specimens being scanned from discovery through analysis and curation. The primary focus of the laboratory efforts are in the fields of archaeology, palaeontology, and broader areas of natural history.

Tell us about your career path. What drew you to scientific imaging?

Nick: I began 3D modeling with the Idaho Virtualization Laboratory as a graduate student studying Anthropology in 2005. I had a fairly extensive background in art and a professor of mine learned that the lab was looking for another technician and thought I would be a good fit. Once I learned the ropes I fell in love with it. Working within a museum environment and dealing almost exclusively with museum collections is fascinating. The ‘talent’ that we place in front of the lasers is always changing so there was always something new and amazing to work with, each object provides unique challenges requiring different approaches. I started almost nine years ago and the work has not lost any of its luster.

Jesse: Before I started working for the lab, I was an intern in the vertebrate palaeontology department at the Idaho Museum of Natural History working on the early phase of the Helicoprion research. We had received funding to CT scan the fossil that later revealed the jaws of the shark, but not enough money to process those images. Luckily the lab had the software necessary to process CT data so I jumped into the deep end and extracted the fossil from the rock with what I like to call “digital prep”. Once I worked out that I could take a fairly unremarkable chunk of rock and turn it into a paradigm shifting discovery using nothing but computers it was pretty hard to go back!

Robert: I got involved with the IVL because I was working as the IMNH’s webmaster when Ralph Chapman was putting together the original funding proposal. Since he thought of me as “the computer guy” he asked me to put together the computer specifications, and find the current “state of the art” laser scanners. Then Ralph hired me to be one of the scan techs who received the training on how to use the hardware and software, and I’ve been working here ever since.

How has 3D scanning of fossils aided in advancing the knowledge we have of prehistoric creatures? Can you give us an example of how the work you've done at the Idaho Virtualization Lab has led to new knowledge?

Nick: Over the years we have worked on a range of projects that have aided in the development of new and innovative ways of analyzing and distributing scientific data. Our online comparative osteological databases were the first of their kind in the fields of archaeology and vertebrate osteology and during their construction we built the first two fully digitized and articulated Orca whales, the first Humpback whale and many others.

Then we began our first NSF funded heritage digitization project of capturing every bird, fish and mammal in the arctic The Virtual Zooarchaeology of the Arctic Project (or VZAP for short). There were only a handful of scanning labs doing this kind of research in the world, so there was a great deal of room for first offs and we were very fortunate to be a part of that. One notable project that I worked on was a paleo-forensic reconstruction of an archaeologically recovered animal attack victim where we used 3D modeling to aid in understanding the trauma that was present on the skeletal remains. I cannot say too much about that as it is still in the publication stages but, to my knowledge, it was also the first of its kind in using 3D modeling for this kind of archaeological analysis.

Robert: We have scanned a number of fossils in my time here. The majority of the advances that have resulted from examining of those models come in a way that seems counter intuitive at first. When we finish scanning a specimen, we have a colorless model. No surface color at all other than a basic color that the software generates, grey in some applications, blue in others, white in some, etc. By not having color on the surface of the models the only thing visible is the actual morphology of the object. Your eye isn’t distracted by the subtle variation of. That allows a researcher to see more of the fine features, like sutures in crania, which are hidden by the color of the item.

Jesse: One of the newest advancements of 3D scanning fossils is CT scanning fossils still in their plaster jackets then 3D printing the data that is extracted. Fossil preparation is expensive, labor intensive, and runs the risk of damaging the specimen. The combination of CT scanning and 3D printing allows researchers to handle the fossil and preserve the original in a stable state at the same time. And the technology is so advanced that any differences in the original and the print are at the micron level. The surface scanners we use give researchers the ability to manipulate digital surrogates of often fragile specimens without the risk of damaging the irreplaceable fossil. Having a virtual model also allows researchers from foreign countries, who would never have the funding available to travel to wherever the fossils are being kept, to perform analyses for their research without ever having to board a plane. This means that more people can evaluate the fossils, which leads to much more solid results and data.

How did the Idaho Virtualization Laboratory come into existence? Whose "brain-child" was it?

Robert, Nick, and Jesse: The Lab came about through a grant to Idaho State University. The IVL was the brainchild of Ralph Chapman, a paleontologist who, after looking at the condition of the IMNH’s paleontology collection, decided that something needed to be done to keep the very fragile collection from being further damaged through handling by researchers and students. The global expansion of the IVL came when Dr. Herbert Maschner took over the leadership of the lab in 2006, and it became a permanent unit of the Idaho Museum of Natural History when Dr. Maschner became the IMNH Director in 2010. The mission of the IVL is founded in what we call the “Democratization of Science” model: making specimens and even entire collections, available to the public and scholars around the world. Since 2006, the IVL has received over $2 million in funding from the National Science Foundation and $1.2 million is awards from private foundations. We also have partnerships with a number of private corporations including Geomagic and Faro.

Our readers tend to approach 3Dimaging from an artistic, rather than a scientific, point of view. What advice would you give to people wanting to work in your field? Would they need a science based degree, or are their other paths in? Do you see career opportunities in this field increasing, or expanding?

Nick: Career opportunities in heritage digitization are expanding so rapidly that is hard for us to keep up. Factor in the advances in 3D printing and the potential for non-destructive distribution and replication, 3D capture and archiving is the next big thing for museum sciences (and just about everything else from food to car parts). The scientific training helps tremendously. Aside from experience in handling precious and fragile objects, which is an acquired skill in itself, the knowledge accumulated over years of working with collections gives a much greater idea of what things are and how they are supposed to look.

What this means is that there are diagnostic elements of both bones and objects that a trained researcher knows to be on the lookout for and will therefore need to be treated with special attention when digitally archiving an object. In this respect, the scientist cannot be removed from the process; for what would appear a minor detail to an untrained observer can be the one thing which sets an object apart from an assemblage. What should be understood here for someone looking into this type of work is that most of the heritage and museum professionals that begin working in digitization are formally trained in other aspects of science and academia, 3D modeling is a secondary skill set and often at the beginning, poorly understood. As any 3D modeler knows, it is easy to make a model of something, what is not easy is doing it the right way. There are nuances to this work that are hard earned skills to someone learning on the fly as opposed to someone whose training is in 3D modeling and animation first. That said, anyone with a background in modeling and animation can find themselves a thrice welcomed and invaluable asset to any heritage digitization facility.

Robert: The only advice I can really give is simply to have an eye for detail. We have a saying: if it’s worth doing it’s worth overdoing. We capture more information in our models than is required to just capture the shape. It’s similar to adding pores on a character’s face when it will never be seen in the renders.

As far as having a science degree, it helps but isn’t required. We have had artists work with us in the past, whose work scanning and editing models was just as good as that done by this of us with science based degrees. We have had pure science students (biologists, chemists, geologists, even engineers) who couldn’t wrap their heads around how, or why we were doing what we do. In this case what is required is an ability to “see” things in 3d, and to have an interest in producing the best final deliverable possible.

Are career opportunities increasing or expanding? Well, the short answer is yes. As more museums see the benefits of digitizing collections the number of positions for people doing 3D scanning will increase. Regardless of that, there is always a need for scientific illustration and more institutions are looking to create interactive exhibits that have a visual component to them. Those exhibits require artists who can create the environments, actors, props, etc. and produce the animations to entice the public to learn more about the subject. That is only going to grow as time goes by.

Can you talk a bit about your workflow and how you are tackling this enormous project?

Jesse: Between the five of us we generate, on average, 30-50 high resolution 3D models per week. Once we have the models finished, scanned and surfaced in Geomagic studio, we take them into Zbrush for remeshing, cleanup, subdividing, and finally photo textured. Our end result is a 3D model that fully replicates the original item down to the tiniest detail. In the past we have researched the accuracy of our models versus the real thing and they are within 2-3 microns of the original.

What about your process from the scan to the finished models and the software you use? How much cleanup work do you do on the mesh?

The scanning process begins with the object or assemblage that we will be digitizing; factoring in size, complexity, and the nature of the object. Prehistoric basketry versus stone fossil for example, will dictate the workflow that we follow through to the completed mesh.

We use a number of different scanners, or non-contact surface digitizers, to capture our initial data. In house currently we have two Faro Edge Arm non-contact surface digitizing laser systems, a FARO Focus wide range Lidar, two Konica Minolta 9i’s, a Next Engine, two much older and rarely used Cyberware scanners, a desktop M15 and the Model Shop MS. We are also currently working on building our own structured light rig though we would love to have a Steinbichler Comet.

The software that we use for data capture and editing depends a great deal on which scanners are in use, primarily we use Geomagic Studio and InnovMetric PolyWorks for a majority of our scanning and editing and Scene and Recap-Pro for Lidar (large area scans, caves, dig sites, etc.).

We use many other software suites for other projects; Mimics for processing CT-data, Daz-Studio for file conversions in making the U3D files used in 3D pdf’s, Z-Brush, Photoshop, Headus, Maya and 3D-Studio (rarely). We also try to stay in the open source community so we always have a current trunk version of Blender installed to play with from time to time.

There is often a great deal of editing that goes into the build after the data has been captured. More simple objects require less editing than those that are more complex. However, as we are primarily creating heritage archives for science and analysis we save files in a number of different stages for archive integrity and future research. For example, directly after data capture the raw scans gets registered and saved essentially unedited, we then merged the scan files together and save that as the digital heritage archive. We then edit the scan to produce the model that we then distribute and use in the rest of our editing pipeline. Once a scan file has been merged into a single mesh we can go all the way to high resolution full color and export any stage in between. Any researchers upon request can gain access to any stage of this data for their own work if needed and as we are a federally funded open access scanning and research lab we are happy to provide it.

What about texturing? Obviously the scans give you the model but how do you have colour and texture? Is it done manually or do you repurpose photographs or is it a mixture of both?

It is usually a mixture of both. Our Vivid 9i and NextEngine scanners do capture color during the scanning process, but that color information is too low resolution to be useable for what we want. We take a minimum of six high resolution photographs with our 25 megapixel Cannon cameras, one photo for each face of a theoretical cube surrounding the object. If we need to we take more just to ensure the entire surface has been captured photographically.

Unlike most photography, we try to eliminate shadows from our photos and we use macro lenses with a very high f-stop to get as much of the surface in focus as possible with the least amount of optical distortion. This allows us to use the texturing capabilities of software like ZBrush or Mudbox to insert the photographic data into our models which creates as close to an exact color copy of the original object as is currently possible. There will inevitably be seams created at the edges of individual photographs that have been applied to the mesh. These will need to be poly-painted out or smoothed so that the texture is a single cohesive image and not a patchwork of multiple images, experienced texture artists can keep these seams to a minimum if not eliminate them altogether without spending a great deal of time with intensive cleaning.

The manual texturing process is not as exact as relying on color per vertex data capture from a 3D digitizer but the resultant texture map is far cleaner and of much higher resolution than native scan color and can be used to generate even higher levels of detail in the mesh.This is a very time intensive process, both in the setting up and taking the photos and projecting the images onto the model. With our current technology this is unavoidable our goal is to do our best to produce a finished model that matches the original object . We hope to add structured light scanners to our arsenal of scanning equipment soon as they can capture high resolution color data alongside the high resolution 3d data which would eliminate the need for manual texturing. There are times when, for various reasons, we can’t get the highly detailed photos, in these cases we will take a reference photo then use polypaint to manually paint the model to get it as close as possible.

Workflow: Using Alpha Mapping to Overcome Low Resolution Scan Data.

Did the Idaho Virtualization Lab have a hand in discovering the true positioning of the Helicoprion's "whorled-jaw?" Can you tell us how this came about?

Jesse: Yes we did. I took a few of the specimens to a local hospital and had them CT scanned in the regular machine that is used for patients in got some promising results. With the proof of concept we wrote a grant application and got the funding for me to take three fossil sharks to Texas to use the industrial CT machine at UT Austin. I worked with the scan files and with some mentoring from Robert was able to produce the first ever 3D model of the jaws of Helicoprion.

I sent this early model to an expert in the field of fossil CT work, Dr. Alan Pradel, who was able to extract more detail from the scans. With his framework back in hand I went through the scans slice by slice, all 365 of them, from 3 different angles to squeeze every last bit of sharky goodness out of the rock. This is the model that was later published. I then worked with a functional morphologist who works with modern sharks to rebuild the crushed model and fill in missing data. The model was then 3D printed and I built an animatronic set of jaws for the exhibit we opened here.

Helicoprion: The 265 million year old fossil represents the largest specimen found to date, 41 cm in diameter! The jaws are the digitally reconstructed pieces that solved a 100 year old mystery.

What's next for the Lab? Is there any project you're working on specifically at the moment you'd like to tell us about?

Robert: Well, at the moment we have a number of projects we are working on. The Virtual Zooarchaeology of the Arctic, and online osteological comparative collection, has been our central project for a number of years, but that is winding down. We are ramping up on the Virtual Museum of Idaho, which is our attempt to digitize as many of the collections here at the IMNH as possible for distribution online.

We are also working on a pilot project for a Virtual Museum of the Arctic, which will eventually include collections from across the globe for northern latitudes, but at the current time is being limited to a few archaeological sites our director, Dr. Herbert Maschner has excavated over the last decade.

Jesse: Nick and I have taken on a side project to repair a horrible cast of a dire wolf skeleton. The cast has been sitting in the basement for a long time because of its poor condition. We are going to digitally repair the cast and 3D print a higher quality and more realistic version for display. We are also hoping to be able to add some new whale remains to our Whales of the World project. Finally there is my ongoing Virtual Paleontology of Idaho project, which is an online database of the more exceptional fossils in our collection. We just went out and scanned a really cool Native American cave site to capture a large collection of rock art that we will be putting online, hopefully soon.

Dire Wolf Pelvis - before and after

Anything you'd like to add?

Nick: I would like to add a big thank you to CGSociety for giving us the opportunity to talk with you and reach a much broader audience of 3D professionals. Although a majority of the work we have done is available online, the scope has been relatively specific in the sense that it has been directed towards educators and researchers in the natural sciences. This has of course been very rewarding for us as peers within this community, but we would like to reach out to other professionals in the 3D community as well; to let others know what is happening in these institutions with this technology and to introduce ourselves from the other side of the playing field, for both the exchange of ideas and techniques and to strengthen the broader community of 3D artists and professionals.