Pages

Tuesday, December 8, 2009

Simulation-based Engineering Science

gmcrews has a couple interesting posts on model verification and validation (V&V) and his commment on scientific software has links to a couple ’state of the practice’ reports [1][2]. The reports are about something called Simulation-based Engineering Science (SBES), which is the (common?) jargon they use to describe doing research and development with computational modelling and simulation.

Below are some excerpts from the executive summary along with a little commentary.

Simulation has today reached a level of predictive capability that it now firmly complements the traditional pillars of theory and experimentation/observation. Many critical technologies are on the horizon that cannot be understood, developed, or utilized without simulation. At the same time, computers are now affordable and accessible to researchers in every country around the world. The near-zero entry-level cost to perform a computer simulation means that anyone can practice SBE&S, and from anywhere.

1.1 Major Trends Identified

Data-intensive applications, including integration of (real-time) experimental and observational data with modelling and simulation to expedite discovery and engineering solutions, were evident in many countries, particularly Switzerland and Japan.

Achieving millisecond time-scales with molecular resolution for proteins and other complex matter is now within reach using graphics processors, multicore CPUs, and new algorithms.

The panel noted a new and robust trend towards increasing the fidelity of engineering simulations through inclusion of physics and chemistry.

The panel sensed excitement about the opportunities that petascale speeds and data capabilities would afford.

1.2 Threats to U.S. Leadership

The world of computing is flat, and anyone can do it. What will distinguish us from the rest of the world is our ability to do it better and to exploit new architectures we develop before those architectures become ubiquitous.

Furthermore, already there are more than 100 million NVIDIA graphics processing units with CUDA compilers distributed worldwide in desktops and laptops, with potential code speedups of up to a thousand-fold in virtually every sector to whomever rewrites their codes to take advantage of these new general programmable GPUs.

Inadequate education and training of the next generation of computational scientists threatens global as well as U.S. growth of SBE&S. This is particularly urgent for the United States; unless we prepare researchers to develop and use the next generation of algorithms and computer architectures, we will not be able to exploit their game-changing capabilities.

Students receive no real training in software engineering for sustainable codes, and little training if any in uncertainty quantification,validation and verification, risk assessment or decision making, which is critical for multiscale simulations that bridge the gap from atoms to enterprise.

A persistent pattern of subcritical funding overall for SBE&S threatens U.S. leadership and continued needed advances amidst a recent surge of strategic investments in SBE&S abroad that reflects recognition by those countries of the role of simulations in advancing national competitiveness and its effectiveness as a mechanism for economic stimulus.

I don’t know of any engineering curriculums that have a good program for training people in all of the areas (the physics, numerical methods, design of experiments, statistics and software carpentry) to be competent high-performance simulation developers (in scientific computing the users and developers tend to be the same people). It’s requires multi-disciplinary, and deeply technical knowledge at the same time. The groups that try to go broad with the curriculum tend to treat the simulations as a black box. Those sorts of programs tend to produce people who can turn the crank on a code, but don’t have the deeper technical understanding needed to add the next increment of physics, or apply the newer more efficient solver, or adapt the current code to take advantage of new hardware. Right now that sort of expertise is achieved in an ad-hoc or apprenticeship kind of manner (see for example MIT’s program). That works for producing a few experts at a time (after a lot of time), but it doesn’t scale well.

1.3 Opportunities for Investment

There are clear and urgent opportunities for industry-driven partnerships with universities and national laboratories to hardwire scientific discovery and engineering innovation through SBE&S.

There is a clear and urgent need for new mechanisms for supporting R&D in SBE&S.

investment in algorithm, middleware, and software development lags behindinvestment in hardware, preventing us from fully exploiting and leveragingnew and even current architectures. This disparity threatens critical growthin SBE&S capabilities needed to solve important worldwide problems as wellas many problems of particular importance to the U.S. economy and nationalsecurity.

There is a clear and urgent need for a new, modern approach to educating and training the next generation of researchers in high performance computing specifically, and in modeling and simulation generally, for scientific discovery and engineering innovation.

Particular attention must be paid to teaching fundamentals, tools, programming for performance, verification and validation, uncertainty quantification, risk analysis and decision making, and programming the next generation of massively multicore architectures. At the same time, students must gain deep knowledge of their core discipline.

The third finding is interesting, but it is a tall order. So we need to train subject matter experts who are also experts in V&V, decision theory, software development and exploiting unique (and rapidly evolving) hardware. Show me the curiculum that accomplishes that, and I’d be quite impressed (really, post a link in the comments if you know of one).

More on validation:

Experimental validation of models remains difficult and costly, and uncertainty quantification is not being addressed adequately in many of the applications. Models are often constructed with insufficient data or physical measurements, leading to large uncertainty in the input parameters. The economics of parameter estimation and model refinement are rarely considered, and most engineering analyses are conducted under deterministic settings. Current modeling and simulation methods work well for existing products and are mostly used to understand/explain experimental observations. However, they are not ideally suited for developing new products that are not derivatives of current ones.

One of the mistakes that the scientific computing community made early on was in letting the capabilities of our simulations be over-sold without stressing the importance of concurrent, supporting efforts in theory and experiment. There are a significant number of consultants who make outrageous claims about replacing testing with modeling and simulation. It is far to easy for our decision makers to be impressed by the really awesome movies we can make from our simulations, and the claims from the consultants begin to get traction. It is our job to make sure the decision makers understand that the simulation is only as real as our empirical validation of it.

Below are some exerpts from the executive summary along with a little commentary.

Major Findings:

SBES is a discipline indispensable to the nations continued leadership in science and engineering. […] There is ample evidence that developments in these new disciplines could significantly impact virtually every aspect of human experience.

Formidable challenges stand in the way of progress in SBES research. These challenges involve resolving open problems associated with multiscale and multi-physics modeling, real-time integration of simulation methods with measurement systems, model validation and verification, handling large data, and visualization. Significantly, one of those challenges is education of the next generation of engineers and scientists in the theory and practices of SBES.

There is strong evidence that our nations leadership in computational engineering and science, particularly in areas key to Simulation-Based Engineering Science, is rapidly eroding. Because competing nations worldwide have increased their investments in research, the U.S. has seen a steady reduction in its proportion of scientific advances relative to that of Europe and Asia. Any reversal of those trends will require changes in our educational system as well as changes in how basic research is funded in the U.S.

The ’Principle Recommendations’ in the report amount to ’Give the NSF more money’, which is not surprising if you consider the source. It is interesting that their finding about education is largely the same as the other report.

3 comments:

You cover a lot here. Let me focus on: "I don’t know of any engineering curriculums that have a good program for training people in all of the areas (the physics, numerical methods, design of experiments, statistics and software carpentry) to be competent high-performance simulation developers (in scientific computing the users and developers tend to be the same people). It’s requires multi-disciplinary, and deeply technical knowledge at the same time. The groups that try to go broad with the curriculum tend to treat the simulations as a black box. Those sorts of programs tend to produce people who can turn the crank on a code, but don’t have the deeper technical understanding needed to add the next increment of physics, or apply the newer more efficient solver, or adapt the current code to take advantage of new hardware. Right now that sort of expertise is achieved in an ad-hoc or apprenticeship kind of manner (see for example MIT’s program). That works for producing a few experts at a time (after a lot of time), but it doesn’t scale well. "

I agree there do not seem to be many general engineering curricula. One ad-hoc approach would be the University of Tennessee's MS in Engineering Science. (Twenty years ago they had a great faculty that let me take everything from artificial intelligence to computational fluid dynamics.) However, they don't offer a BS in Engineering Science! (The BS I got in General Engineering was from the University of Nevada and is no longer available.)

I tend to look at this from a different angle. I have always felt it is easier for a generalist to learn specific things than it is for a specialist to learn general things. I find that acquiring a "deeper technical understanding" usually involves a paradigm shift in a person's thinking. Generalists know lots of paradigms, and since specialists don't practice paradigm shifts much that tends to favor a generalist too.

But I could be wrong because I find that many employers tend not to agree with me when hiring for all but entry level positions. They highly value specialized knowledge. I see this even in my own field of software development, where the technology is constantly evolving at a rapid rate -- punctuated evolution at that! And I see evidence for their view. For example, even Michael Jordan couldn't make it in pro baseball.

One final thing that is slightly tongue-in-cheek but that IMHO has a noticeable effect. Scientists tend to think they are general engineers. That is, they basically feel they understand how to do engineering if they wanted to really get into it. Engineers, in turn, tend to think they are general programmers. And programmers feel that, in general, they know all there is to really know about V&V.

Whereas generalists think they know everything.

I am not mentioning mathematicians because 1) they think numbers exist and it is the universe that is an illusion, and therefore 2) they will assume anything except responsibility. :-)

Slides on a presentation for a study that builds on the NSF report cited above. From slide 11:Study designed to:- Gather information on the worldwide status and trends of SBE&S research- State of the art, regional levels of activitiesUS leadership status- Opportunities for US leadership- Disseminate this information to government decision makers and the research community- Findings, not recommendations

From Slide 20:Top Four Major Trends in SBE&S Research1. Data-intensive applications (esp Switzerland and Japan) Integration of (real-time) experimental and observational data with modeling and simulation to expedite discovery and engineering solutions2. Millisecond timescales for proteins and other complex matter with molecular resolution 3. Science-based engineering simulations (US slight lead) Increased fidelity through inclusion of physics and chemistry4. Multicore for petascale and beyond: not just faster time to solution - increased problem complexity Cheap GPUs today give up to 200x speed up on hundreds of apps!