3
To provide a basis for design. To assist in human decision making. To explain complex system behaviour. For use within fault detection systems etc.. For simulator development (e.g. for operator training or for engineering development applications). The purpose of models in engineering Keynote Tutorial: Model quality, testing and validation 2

4
1.Conceptual models allow investigation of performance limitations at an early stage of the design process for normal and abnormal operating conditions. 2.Fully developed and proven models can provide information about key parameter sensitivities and inter-dependencies – useful for design decisions and optimisation. 3.Full models allow virtual prototypes to be created before any hardware prototype is available so identifying necessary design alterations at an early stage, avoiding expensive changes later on. Design benefits with fit-for-purpose models Keynote Tutorial: Model quality, testing and validation 3

6
 Level of model quality necessary is of critical importance for any given application........ an inappropriate model is less than useless as it may delay the project and lead to cost escalation  Balance needs to be found between model accuracy and the cost of developing the model.  Rigorous consideration of model quality is most common in applications involving safety critical issues (e.g. aeronautical engineering, automotive engineering etc.) Levels of model quality Keynote Tutorial: Model quality, testing and validation 5

7
Current Problems with ModelsCurrent Problems with Models  Models are often used with very little systematic testing.  Model documentation is often minimal and is not recognised as a vital part of the model development process.  In many organisations models are passed from project to project and end up being used in ways that were never intended by the original developer of the model. Current problems with models Keynote Tutorial: Model quality, testing and validation 6

18
 Tests only deal with a small number of cases.  General statements about validity are impossible.  All testing must be carried out in the context of the application and especially the precise range of operating conditions for that application.  Should start from a well-understood case, even if much simplified; then move incrementally to testing for less certain situations for that application.  The more complex the model the harder the problem of quality assessment becomes: measures of model performance become harder to define and visualisation becomes more difficult. Testing of models: fitness-for-purpose Keynote Tutorial: Model quality, testing and validation 17

19
 Establishing the useful range of the dynamic model for a specific application.  Estimating the limits of accuracy of the model (usually both for steady-state and transient conditions) in terms the magnitude of expected errors in model predictions. Aspects of model quality Keynote Tutorial: Model quality, testing and validation 18

21
Internal verification and external validation  Internal Verification – establishing that a computer–based model is consistent with the underlying mathematical model. i.e. “Is the simulation model right?”  External Validation – demonstrating that a final (nonlinear)) model is adequate for the intended application. i.e. “Is it the right simulation model?” Note that checks of identified linear models are sometimes referred to as “external verification”. Keynote Tutorial: Model quality, testing and validation 20

22
External validation Need to distinguish between:  Functional validation: where model is assessed in terms of how well it mimics input-output behaviour of the real system.  Physical/Theoretical validation where the model is based on theory and intermediate variables in model and system are compared. Approximations and assumptions are investigated within this process. Keynote Tutorial: Model quality, testing and validation 21

23
Approaches to external validation  Holistic approaches (e.g. subjective opinion of an expert on the real system such as an operator).  Model component approaches (e.g. each sub- system tested independently and compared with corresponding components of the real system). Keynote Tutorial: Model quality, testing and validation 22

26
 Plots of simulated and measured responses against an independent variable (often time).  Plots of simulated values against the corresponding measured values (should be 45 degree line).  Different graphical methods may emphasise different aspects of the simulation model performance so there are possible benefits from combining different approaches. Methods of system/model comparison: some examples Keynote Tutorial: Model quality, testing and validation 25

28
Another time history comparison: a multi-input multi-output case The original version of this figure was published by the Advisory Group for Aerospace Research and Development, North Atlantic Treaty Organisation (AGARD/NATO) in AGARD Advisory Report 280 ‘Rotorcraft System Identification’, September 1991 BO-105 helicopter flight test data, DLR SIMH simulation model Keynote Tutorial: Model quality, testing and validation 27

33
Issues of identifiability in external validation Test input design is important since inputs must excite the system and model over an appropriate range of frequencies and amplitudes. The concept of identifiability is central to issues of test input design external validation and is thus very important for external validation. Structural identifiability relates to situations where a model may have an excess of parameters so that some specific parameters cannot be estimated uniquely for any possible experimental design (e.g. Bellman, R. and Åström, K.J., Mathematical Biosciences, 7, 329-339, 1970). Structural identifiability is also important for external validation. Keynote Tutorial: Model quality, testing and validation 32

34
Numerical unidentifiability arises when a structurally identifiable model is being used with data that is inappropriate for the application. Numerical identifiability investigated from parameter information matrix M, the related dispersion matrix D and the parameter correlation matrix P. All depend on the sensitivity matrix X where: Inputs may maximise the overall accuracy of all parameter estimates or may be chosen to maximise accuracy of specific parameter estimates. Issues of numerical identifiability and test input design in model validation Keynote Tutorial: Model quality, testing and validation 33

35
Upgrading of simulation models  Following comparison of model and system behaviour usually need to analyse discrepancies and propose upgrades for model.  Changes must be evaluated systematically on a physical basis – with further iterations in the development cycle.  Parametric changes usually considered first, before structure.  Sometimes possible to associate model deficiencies with specific state variables model (e.g. correlation of output error with a state variable may help identify problem source).  Correlation of model errors with derivatives of state variables may suggest that a higher-order description would be more useful.  Optimisation tools (including evolutionary computing methods such as GA and GP) may be useful but should be used along with physical knowledge and understanding of the model. Keynote Tutorial: Model quality, testing and validation 34

37
Generic and re-usable sub-models  Generally accepted that system design should be based on use of generic descriptions and re-usable sub-models.  Examples of the generic approach may be found in automotive engineering, gas turbines etc. Issues inevitably arise in the external validation of generic models – one approach is discussed in Smith, M.I., Murray-Smith, D.J. and Hickman, D., ‘Verification and validation issues in a generic model of electro-optic sensor systems’ J. Defense Modeling and Simulation, 4(1), 17-27, 2007 Keynote Tutorial: Model quality, testing and validation 36

38
Model documentation and version control  Extra costs of creating good documentation should be more than balanced by the resulting re-usability of models. Version control processes should ensure that changes are fully documented.  Documentation and version control well developed in software engineering field. Same principles should be applied to the model development process. Keynote Tutorial: Model quality, testing and validation 37

39
Items for documentation  Purpose of model and the intended application.  Full description of model and corresponding computer code.  List of all assumptions and approximations used.  Details of all tests carried out on the real system to provide information for model development.  Details of the internal verification process.  Details of the external validation process, with statements about why model was accepted or rejected and information about usable range for model. Keynote Tutorial: Model quality, testing and validation 38

41
Modelling and Simulation in Engineering EducationModelling and Simulation in Engineering Education  Engineering students encounter mathematical and computer- based modelling repeatedly in their university education.  Emphasis is most often on development of models from physical principles and on using models/simulations in place of experiments on real systems. Modelling and simulation in engineering education Keynote Tutorial: Model quality, testing and validation 40

42
Modelling and Simulation in Engineering EducationModelling and Simulation in Engineering Education  Issues of model quality and fitness-for-purpose are seldom emphasised in the teaching of modelling and simulation.  Model validation is neglected in education. The teaching of system modelling and simulation should include much more on model validation methods.  Model testing should become second nature for students.  Documentation, model re-use and libraries of models must be given much more emphasis (especially in more advanced teaching). Model quality and testing issues in engineering education Keynote Tutorial: Model quality, testing and validation 41

43
Part 5: Examples Drawn from external validation and upgrading of helicopter models for flight control system design. Keynote Tutorial: Model quality, testing and validation 42

44
 Plant model limitations often impose serious performance limitations within control system, especially in systems with high-performance requirements.  Particularly important to have highly accurate plant models for the part of the frequency range close to the “cross-over” region in the frequency domain. Model limitations in control Keynote Tutorial: Model quality, testing and validation 43

45
Examples from aircraft and helicopter flight control system design There are many well documented aircraft flight control examples illustrating problems of model quality and model limitations in integrated system design. Problems are often identified during initial testing of hardware. These lead to development of improved models and corresponding control design changes. The later in the design cycle these changes have to be made the more costly they are and the greater the delays to the project. Keynote Tutorial: Model quality, testing and validation 44

48
SS Common test inputs used for system identification and “external verification” of the identified model (BO-105 flight data) The original versions of these figures were published by the Advisory Group for Aerospace Research and Development, North Atlantic Treaty Organisation (AGARD/NATO) in AGARD Advisory Report 280 ‘Rotorcraft System Identification’, September 1991

50
Simulation, identification and “external verification” results (BO-105 flight test data, DLR SIMH model) The original version of these figures were published by the Advisory Group for Aerospace Research and Development, North Atlantic Treaty Organisation (AGARD/NATO) in AGARD Advisory Report 280 ‘Rotorcraft System Identification’, September 1991

52
Assessment of a theoretical nonlinear model for a Puma helicopter Parameter values for two different flight conditions, showing parametric trends from a physically-based nonlinear simulation model (HELISTAB) and the trends in estimates from flight tests involving system identification of separate linearised models for each flight condition. From Bradley, R., Padfield, G.D., Murray-Smith, D.J. and Thomson, D.G., ‘Validation of helicopter mathematical models’, Transactions of the Institute of Measurement and Control’, 12(4), 186-196, 1990. Keynote Tutorial: Model quality, testing and validation 53

53
 Good vehicle models are essential for design of high-bandwidth full-authority active flight control systems.  Published examples show that the achievable performance of flight control systems have, in some cases, been overestimated in initial design studies, usually because of limitations of the flight mechanics model of the vehicle (see e.g. Tischler, M. B. Advances in Aircraft Flight Control Systems, Taylor & Francis, London 1996).  Although control systems can be made robust to compensate for poor accuracy this is usually at the expense of performance. Improved modelling procedures and improved models can offer significant benefits. Otherwise, problems may not be apparent until the flight testing stage, leading to costly redesign, extended flight test programmes and delays in certification. Summary of model quality issues for helicopter flight control system design Keynote Tutorial: Model quality, testing and validation 55

55
Discussion: Model quality in design  The more demanding the design specification the more important is the fitness-for-purpose of models used in design.  More attention needs to be given to the external validation of models for the specific application in question. a) Establishing the useful range of the model. b) Estimating the accuracy of the model within that range.  Model validation is part of the model building process and external validation techniques need to be applied repeatedly.  Models should be retained, maintained and updated throughout the whole life-cycle of the system that they represent. Keynote Tutorial: Model quality, testing and validation 57

56
More attention should be given to the fitness-for-purpose of models used in design, especially for demanding applications. Current methods for external validation are time-consuming and difficult to apply in many situations. More effort should be devoted to improving validation methods. Techniques of version control and rigorous documentation should borrowed from software engineering and applied to the model development process. Re-use of proven models should be made easier and more comprehensive model documentation should be available within model libraries. Issues of model quality should be given far more attention within engineering education. Recommendations Keynote Tutorial: Model quality, testing and validation 58

57
Conclusions  With suitable structure and parameter values and rigorous external validation models that are fit-for-the-purpose of a given application can be developed (iteratively of course).  Good model management can reduce the cost of design and development.  There are no quick answers: a systematic approach is essential, moving incrementally from well-understood cases to less well known situations.  Educational and cultural changes are needed as well as improved management of the modelling, simulation and design processes within most organisations. Keynote Tutorial: Model quality, testing and validation 59

About project

Feedback

To ensure the functioning of the site, we use cookies. We share information about your activities on the site with our partners and Google partners: social networks and companies engaged in advertising and web analytics. For more information, see the Privacy Policy and Google Privacy &amp Terms.
Your consent to our cookies if you continue to use this website.