Conference: Forecasting, monitoring, controlling

People often have to make judgments and decisions in response to information that rolls out over time. Typically, people have to deal with such data streams in one of three ways. First, people may have to use the pattern in the data to forecast what will happen in the future. Second, they may need to monitor the data to decide whether the process producing it has changed in some way. (If it has, this may either indicate a need for action or else provide evidence that some previous action has had an effect.) Third, they may need to control some parameter in the system producing the data to ensure that output is brought into and maintained within a target range.

These three tasks have been investigated by different communities of researchers but they are clearly interconnected.

The â€œForecasting, monitoring, controllingâ€ workshop will bring together separate groups of researchers working on these three different but related tasks, fostering a more coherent approach to the study of how people deal with changes in their environment. Anyone working on dynamic decision making is likely to find attending the workshop worthwhile.

The workshop will be a 2-day event held at University College London, from Thursday September 19 â€“ Friday September 20, 2013. The registration fee is Â£25, with free attendance for students with appropriate ID.

The number of attendees is limited, so to register for the workshop please email Matt Twyman: This email address is being protected from spambots. You need JavaScript enabled to view it.This email address is being protected from spambots. You need JavaScript enabled to view it.

16.45 â€“ 17.15 Paul Goodwin Judgmental forecasting in supply chains when special events are due to occur: A comparison of field and laboratory results

17.15 â€“ Discussion

Abstracts

Thursday 19 September: Morning

Rolling training for improving judgmental time series forecasting performance

Fotios Petropoulos

Lancaster University

Paul Goodwin

Bath University

Robert Fildes

Lancaster University

Unaided judgment is the simplest approach of judgmental forecasting. Participants are requested to give their estimates without any information or prior knowledge of the data. In the current study, we examine the efficiency of a rolling training approach applied on unaided judgment. Focusing on time series data, participants are asked to submit estimates for each series multiple times, from different time origins. Upon submission of each set of forecasts, the true outcomes and performance feedback is provided. By this, we provide a live training scheme, enabling participants to better understand the underlying pattern of the data by learning directly from their forecast errors. A symmetric experimental design allowed us to maximize the use of participants, where each one of them is included in both the control and test group. A total of 165 participants (a mix of students, researchers, practitioners and novices) submitted forecasts for each of the 16 time series considered. Given that forecasts were requested for multiple lead times, the analysis includes more than 10,000 out-of-sample point forecasts. The forecasting performance is analysed in terms of series characteristics (stationary, trended, seasonal) and type of feedback (bias, accuracy). Moreover, we measure any reflections on the confidence levels, the dedication of the participants with the forecasting process and the required time. Results yield that the proposed rolling approach should be used as a tool to enhance judgmental forecasting.

Graph format effects on judgmental forecasting

Stian Reimers and Nigel Harvey

City University and University College London

When people extrapolate trends to forecast future outcomes, they often use time series data that is presented in graphical format: generally bar charts, line graphs or scatterplots. Although objectively, forecasts should be independent of the way in which information is presented, in reality there are good reasons to predict that people might make different judgments with different graphical formats. For example, the graph comprehension literature describes ways in which the cognitive representation of information varies with display format. In a series of experiments, we examine how format affects not just comprehension but forecasting. Across a series of online experiments, using incentivised naÃ¯ve forecasters, we show that forecasts are closer to the x-axis when time series data are presented as bar charts than line graphs, when final vertical position is controlled. In more ecological experiments, using real financial time series data, we show a more complex pattern of results, which still reveals a clear effect of graph format, but an interaction between series data and format on forecasting. The influence of format on forecasting is potentially an important one, and we discuss potential explanations of these effects.

Forecasting tools are considered as essential in a variety of fields. As well as being a traditional tool for weather and economic forecasting, these techniques are increasingly used in other, less traditional settings. Here, we propose the use of forecasting tools in the domain of affective forecasting. In the present study, we employ a novel forecasting experimental approach, the â€œtime-sliceâ€ approach, to study the nature of the optimism bias and more specifically, its temporal dimension. Participants consisted of Traders and Personal Assistants (PAs) recruited from two investment banks. They were asked to provide likelihood judgments of future outcomes in response to positive and negative self-related events. One group provided multiple likelihood judgments (e.g. forecasts) for a range of future horizons while the other group provided likelihood judgments only for two horizons of interest (e.g. a proximal and a distant one). When comparing forecasts between the two groups, we find significant differences in the way participants forecast proximal events suggesting that task characteristics play a role in the way predictions are produced; the same is not true for distant horizons. Explanations for this phenomenon are provided and implications are discussed.

An examination of judgment in currency forecasting

Mary Thomson

Newcastle Business School

Human judgment plays a critical role in most forecasting situations. Currency forecasting is no exception. In some situations, statistical models are used to provide initial estimates which are then updated, judgmentally, by the forecaster, yielding final predictions which are a mixture of both quantitative and subjective analyses. In other situations, the forecaster uses judgment alone to predict future prices. But despite this substantial use of judgment in currency forecasting, very little academic research to date has been directed towards its quality. A considerable amount of research has examined the adequacy of statistical models; but human judgment, as a major component of currency forecasting practice, has virtually been ignored.

However, one exception to this lack of research attention regarding the quality of judgment in currency forecasting is a series of studies conducted by Thomson and colleagues. These studies, which are the focus of this presentation, have utilised both actual currency data and simulated currency data, and have controlled various important aspects of the task (such as the strength, direction and structure of the trend) to examine their impact on the quality of judgment in forecasting. These studies are described and the practical implications of their findings for currency forecasting practice are emphasised.

Thursday 19 September: Afternoon

Forecasting advice in the form of optimistic/pessimistic scenarios: Is it the content or the frame?

Dilek Ã–nkal

Bilkent University

M. Sinan GÃ¶nÃ¼l

Middle East Technical University

K.Zeynep SayÄ±m

Bilkent University

In managing a dynamic world, scenarios may constitute effective tools for communication and information sharing by depicting alternative (optimistic and/or pessimistic) storylines about possible futures. In this way, they may stimulate foresight, enhance visions of future, and may further facilitate the decision making process by serving as forecast advice. Recent research suggests that decision makers effectively utilize optimistic and pessimistic scenarios as channels of forecasting advice to help them construct both individual and group predictions. The findings also indicate that decision makers respond differently to the presence of optimistic versus pessimistic scenarios. However, the scenarios used in these studies are not only optimistic or pessimistic in tone and content, but they are also specifically labeled as such (optimistic scenario carries a label declaring it as the â€œbest-case scenarioâ€, while the pessimistic scenario carries a title of â€œworst-case scenarioâ€).

In the current study, our primary aim is to explore whether the effects of optimistic and/or pessimistic scenarios originate from their content or from framing effects. To test this question, a similar experimental setup with the previous work is employed. Participants are provided with time-series plots showing past demand for mobile phones and these are accompanied by model-based point forecasts. Afterwards, participants are requested to make their predictions in different formats (point, best-case/worst-case forecasts, and surprise index) first as individuals, and later, as part of two-person teams (dyads). In the individual forecasting stage, there are four categories with respect to the content of scenarios and the presence of labels: i) optimistic content with labels, ii) optimistic content without labels, iii) pessimistic content with labels, iv) pessimistic content without labels. For the group forecasting stage, each member receives a different content (i.e., one member of the dyad receives an optimistic scenario, while the other is given a pessimistic scenario), and this content is actively shared to arrive at group forecasts. Hence, two sets of dyads are used whereby dyads in the first set receive their scenarios with labels, while those in the second set receive them without labels. Findings from individual and the group forecasting stages are presented and the implications for the effects of scenario framing are discussed.

Improving judgmental forecasting from series with structural breaks

Nigel Harvey, Matt Twyman, & Maarten Speekenbrink University College London

Though quality of judgmental forecasting for near horizons is often comparable to that of statistical methods (Lawrence et al, 1985), judgmental forecasts are noticeably worse when time series contain structural breaks (Oâ€™Connor et al, 1993). We examined forecast accuracy when people forecast from series with and without structural breaks. In some conditions, forecasters explicitly made a decision about whether there was a structural break in the series before making their forecasts; in others, they were aware that structural breaks in the series could occur but made no explicit decision about whether they were present before making their forecasts. As expected from previous work, accuracy was poorer when structural breaks were present and autocorrelation in series improved forecasting but impaired detection of structural breaks. In addition, however, forecasting from series with structural breaks was better when forecasters made an explicit decision about whether the series contained a structural break before they made their forecasts. Explicitly detecting structural breaks may lead to forecasters to take more account of their effects by changing the way that forecasts are produced. Instead of using an intuitive approach based on use of heuristics, forecasters may switch to a more analytic deliberative mode of processing.

Forecasting and classification using absolute and rate information

Bradley C. Love

University College London

When a stock falls 50 points one day and rebounds 50 points the next day the resulting value is unchanged from the initial value. In contrast, When a stock falls 50% one day and rebounds 50% the next day the resulting value is only 75% of the initial value. The first case involves absolute quantities, therefore the arithmetic mean should be used; whereas second case involves rate information and therefore the geometric mean is appropriate. When classifying observed returns as involving an overall increase or decrease in value, participants relied on the arithmetic mean, even when it was inappropriate to do so. A similar pattern was found in forecasting the next return after observing a sequence of previous returns. Irrespective of whether information was presented in absolute or rate formats, participants linearly extrapolated from previous results using arithmetic operations. In other words, forecasting was most accurate for linear functions when information was presented in absolute terms and for exponential functions when information was presented in a rate format.

Assumptions underlying financial models are highly controversial: over the past half of a century, the debate between supporters of the random walk assumption and the fractal model never ceased. In particular, as fractals may assign rare events with higher probabilities than the random walk model, a new interest in the debate emerged after the financial crisis of 2007-2008. However, the vast majority of studies about this topic concentrated on the question whether these models provide an accurate description of the market, and did not deal with the question why they were correct.

In this study we suggest a possible psychological account for the validity of the fractal model. More precisely, fractal time series are characterised by Hurst exponent, a measure of a seriesâ€™ memory, or fractal dimension. We propose that the way people perceive fractal time series and make forecasts from them have a role in shaping the marketâ€™s statistics. In a laboratory experiment, participants were presented with graphs of fractal time series with different Hurst exponents. Graphs could be smoothed by application of a moving average filter. Participants could control the number of points included in the averaging window. Including more points increased graphsâ€™ smoothness but excluded high frequency detail. Participants were asked to determine the window size most appropriate for making financial decisions. In addition, they were asked to make forecasts from each of the presented graphs. Density of points that had to be forecast was manipulated. For each of the resultant smoothened graphs and participantsâ€™ forecasts we calculated a steepness measure (the mean of the absolute value of the graphsâ€™ gradients)

Our results show that the mean steepness of the smoothened graphs was significantly correlated with the Hurst exponents of the original graphs, as well as with the local gradients of participantsâ€™ forecasts. Furthermore, we found that the Hurst exponent of the data graphs and the required forecast density affected the chosen smoothening levels significantly. It is well known that forecasts guide decisions and actions, and, in particular, financial transaction prices. We, therefore, concluded that participantsâ€™ perception and forecasts may serve as the mechanisms, which help preserving the Hurst exponents of financial time series.

Forecasting financial markets: Some light from the Dark Side

Roy Batchelor

Cass Business School, London

Success in financial forecasting is elusive, but promises rich rewards. The field continues to attract a wide range of talent, from highâ€tech academic â€œquantsâ€, â€œfundamentalistâ€ analysts, and a long tail of practitioners of â€œtechnical analysisâ€, a term that covers everything from pattern recognition in charts to wave theory, magic numbers, and astrology. The mainstream view in finance textbooks, is that in efficient markets any but the most advanced forecasting methods are doomed to failure. It is therefore an uncomfortable fact, supported by much survey evidence, that most traders most of the time claim to use technical analysis to support their trading decisions.

Over the past decade I have spent time these analysts, collecting data on their track records, and investigating their methods. This paper reports findings from these travels on the dark side of the financial forecasting profession. To summarise:

â€ There is a gap between academics caricature of technical analysis, and the behaviour of serious professional technical traders

â€ There is also a gap between what technical traders say they do, and what they actually do when their own or client money is at risk

Â­ There may be a core of valuable insight to be gained from some of the pattern-based methods used by technical analysts

Â­ This is consistent with developments over the past two decades in heterogeneous agents models of markets, in which traders with different forecasting methods adapt their behaviour according to their own experiences, and the reported experiences of others.

Friday 20 September: Morning

Models of judgmental change detection

Maarten Speekenbrink, Nigel Harvey, & Matt Twyman

University College London

Judgmental detection of changes in time series is an ubiquitous task. Previous research has shown that human observers are often relatively poor at detecting change, especially when the series are serially dependent (autocorrelated). We present data from experiments in which participants were asked to judge the occurrence of changes in time series with varying levels of autocorrelation. Results show that autocorrelation increases the difficulty of discriminating change from no change, and that observers respond to this increased difficulty by biasing their decisions towards change. This results in increased false alarm rates, while leaving hit rates relatively intact. We present a rational (Bayesian) model of change detection and compare it to two heuristic models that ignore autocorrelation in the series. Participants appeared to rely on a simple heuristic, where they first visually match a change function to a series, and then determine whether the putative change exceeds the variability in the data.

Knowledge encapsulation: The influence of experiential knowledge when monitoring to detect hostile reconnaissance

This multidisciplinary experimental study brings the notion of knowledge encapsulation from medical diagnosis to bear on intelligence analysis. Knowledge encapsulation argues that knowledge gained by doctors through their experience is encapsulated with theoretical / biomedical knowledge, leading doctors to base their diagnosis, to patients, on the encapsulated knowledge.

The study assesses the influence of experiential knowledge on the judgment of Intelligence Analysts. Addressing the intelligence analysis pitfalls from the perspective of experiential knowledge is novel and opens the doors for further research on analysts training in intelligence and in other similar analytical fields. The similarity between the task of intelligence analysis and medical diagnosis justifies drawing the lesson.

Participants in the experiment played the role of law enforcement analysts making decisions about suspicious behaviour in visitors to an airport, in order to detect hostile reconnaissance. The narratives describing the visitorsâ€™ behaviour were designed to bias the participantsâ€™ decision in a direction opposite to what they should have taken if they had considered only their theoretical knowledge. Participants in the control group went through a training phase that provided experience that did not have this conflict with theoretical knowledge. The study was concerned with monitoring and forecasting the participantsâ€™ behaviour with aim of controlling behaviour. The results of the preliminary experiment â€“ using non parametric statistical analysis â€“ showed an influence of experiential knowledge that supports the notion of knowledge encapsulation.

Detecting different types of change in time series

Matt Twyman, Nigel Harvey, & Maarten Speekenbrink

University College London

Studies of change detection in time series usually focus on changes in the series' mean, where those changes may be obscured by other characteristics such as noise, trends, or sequential dependence (autocorrelation). We present four studies in which participants' ability to detect other types of change was tested. Study 1 focussed on changes in the trends in the time series. These changes occurred at a random point mid-series. We examined whether participants were likely to mistake changes in trend for changes in mean level of the series. Study 2 examined detection of changes in the noise or volatility value of the time series. Each series began with either high or low volatility values, and this value changed at a random point mid-way through the series in 50% of trials. Studies 3 & 4 investigated similar switching in the level of autocorrelation between neighbouring data points, between low and high positive values in Study 3, and between positive and negative values in Study 4. Manipulations across all five studies are assessed in terms of their effects on overall judgment performance, hit and false alarm rates in change detection, and confidence ratings associated with judgments. Together, the results create a picture of peopleâ€™s sensitivity to different types of change in time series.

Learning to control from rewards/feedback rather than outcomes

Magda Osman

Queen Mary, University of London

Often we face situations in which we have to manage outcomes when the main frequent performance indicators are either reward-based or feedback-based information; this is because outcome-based information is harder to acquire regularly. For instance, a manager may have to wait until the end of the month for a full sales report and the breakdown of deviation from sales targets, but in the interim receives information each week as to whether the company hit their targets (or framed alternatively missed their targets). What impact does this kind of informational structure have on performance in the long term? What kinds of strategies do people develop?

To examine these questions we used a control task paradigm, but with some modifications. Typically in the control task literature participants are required to reach and maintain a specific outcome in a system (e.g. maintaining a specific level of sugar production in a sugar factory). The difficulty comes from the fact that the system is dynamic, and so the outcome can vary from trial to trial as a result of the participantsâ€™ decisions (e.g. decide the number of works in the factory to employ) as well as from the other external factors (e.g., bad weather conditions reduce sugar crops), or a combination of both. Crucially, in a common control task set up outcome feedback is always presented on each trial, and additional reward/feedback information is rarely, if ever, introduced. Therefore, available research cannot answer critical questions concerning the role of augmented feedback or rewards as cues to accuracy of control-based decisions, which are often relied upon in real world domains.

In our adapted paradigm, though participants are required to control the value to a target on each trial, participants see the outcome-value intermittently (every 5 trials). Instead, they are presented with either reward-based or feedback-based information on every trial, which they rely on as a guide to performance. There are two other important innovations, which are the manipulation of the reliability of the reward/feedback information, and framing of the information in terms of either losses or gains. We compare the impact on decision making behaviour (e.g., early vs. late strategy development, control performance) when the reliably frequent performance indicator is reward-based vs. feedback-based information. We show that control performance is robust even when the outcome-value is not frequently available, and there is clear evidence of learning across all variant of the task. However, the framing of reward/feedback information in terms of gains/losses has a substantial impact on the learning strategies participants develop and their later success in controlling the system to a specific criterion. The implications of these findings are discussed within the context of explorative and exploitative choice behaviour in dynamic decision making contexts.

Dynamic decision making: Learning processes and cognitive challenges

Cleotilde Gonzalez

Carnegie Mellon University

Many daily life situations involve a series of decisions which are interdependent over time. This is an adaptive process in environments that accumulate, evolve, and develop over time as decisions are being made. This process requires recognizing situations, evaluating opportunities often based on past experience, making predictions about future states, and re-evaluating and adapting according to feedback from the environment. Dynamic decision making is a learning process, where experience plays a key role. I will present my current view of the learning process and the cognitive challenges that hinder and may improve decision making in dynamic tasks. I will also present results from many different paradigms founded on repeated binary choice to demonstrate the fact that experience-based choice relies upon common cognitive processes and mechanisms proposed by the Instance-Based Learning Theory (IBLT). I will introduce IBLT and a recent cognitive model based on it: the IBL model of repeated binary choice. I will also discuss the learning and choice phenomena that the model can explain.

Friday 20 September: Afternoon

On the use and abuse (and rational interpretation) of probability forecasts

Leonard A. Smith

London School of Economics

Von Neumann prophesied famously that we would learn to predict stable systems and to control those too unstable to predict. The Earth's weather and its climate provide the classic showcases for attempts to forecast, monitor, and indeed control complicated dynamical systems. Consideration of the climate is more complicated than that of the weather, of course, as the climate is evolving in an interesting and somewhat controllable way, while weather can often be considered as merely dynamic. Prediction, control and indeed monitoring each hinge on the quantification of uncertainty, usually in terms of probability distributions.

This paper discusses the various kinds of probability, their role in prediction and control, and the extent to which we have access to those kinds that are of most value. While focusing on actual weather forecasts, seasonal forecasts and climate forecasts, financial probability forecasts of the Bank of England will also be touched on to provide a more complete picture of the challenges of probability forecasting. Von Neumann also remarked that there is no sense in being precise when you don't even know what you're talking about. There are different kinds of probability, and it is important to know which kind we are talking about. IJ Good noted several kinds of probability. There are also at least two kinds of dealings with a dynamic world: weatherâ€like and climateâ€like. Rational use of probability forecasts requires access to the right kind of probability for the particular dealing at hand. Absent this, there is no persuasive argument for using "a probability forecast" at all: no rational interpretation of "a probability forecast" need exist. The implications this conclusion holds for applying Bayesian methods, amongst others, both in the context of weather and in the context of climate will be discussed, and two alternatives to traditional probabilityâ€based decision making will be suggested.

The Butterfly Effect is a wellâ€known constraint on the predictability of certain types of nonlinear systems, describing a sensitive dependence on initial conditions such that the "flap of a butterfly's wing" can have profound long term effects on the evolution of the system. Less well known but at least as pernicious is the Hawkmoth Effect, which also constrains predictability of dynamical systems. In this instance, the problem is our understanding of the system itself, and our necessarily imperfect numerical representation of the physical processes involved in a complex physical system. We will present some simple demonstrations which show the key features, and then give some examples of the potential importance of the effect for modelling realâ€world systems such as climate, ecosystems, and economics.

The importance of understanding the limits to predictability in these areas cannot be overstated, since the realâ€world impact of overâ€reliance on inadequate models is poor or maladaptive decisionâ€making: consider the effects of reliance on mathematical models in precipitating the ongoing financial crises, or the magnitude of decisions about mitigation and adaptation to climate change. An understanding of the Hawkmoth Effect motivates greater humility in the construction and interpretation of mathematical models, and a more open epistemic attitude towards uncertainty in model output. Some aspects of uncertainty in forecasting can be quantified; others require a qualitative understanding of the system itself and ultimately a subjective judgement about the likelihood of model failure, which cannot be quantified within the realm of the model itself, even given a large amount of calibration data.

Finally, we will discuss more general implications of the Hawkmoth Effect for forecasting, monitoring and controlling complex systems, suggesting heuristics for interpretation of model results and for working with any system where nonlinearities and feedback effects are known or thought likely to be significant.

Scientific uncertainty and climate change: Constraints on policy choices provided by an ordinal analysis of uncertainty

Stephan Lewandowsky (University of Bristol and University of Western Australia)

Uncertainty forms an integral part of science and uncertainty is intrinsic to many global risks that dynamically unfold over time, from â€œpeak oilâ€ to genetically modified foods to climate change. Uncertainty is often cited in connection with political arguments against mitigative or corrective action. Using climate change as a case study, we apply an ordinal analysis (i.e., ordinal statements are of the form â€œgreater thanâ€ or â€œnot enoughâ€) of uncertainty within the climate system. This analysis yields three mathematical constraints that are robust to a broad range of assumptions and not sensitive to peopleâ€™s cultural cognition or subjective risk perceptions. The constraints imply that greater uncertainty (i.e., â€œgreater than expectedâ€ or â€œtoo great to actâ€) actually provides greater impetus for mitigative action. The constraints involve (a) the inevitable positive skew of estimates of climate sensitivity; (b) the convex damage function, and (c) the bounded aspect of the carbon budget. Those normative constraints are related to human behaviour and the nature of scientific endeavours.

Understanding and researching the organizational forecasting process

Robert Fildes

Lancaster University Centre for Forecasting

In this presentation I first review what we know about the forecasting process in two fields where it might be expected that judgment plays little role: climate modelling and macroeconomic modelling. In business forecasting judgment is known to be a key contributor. The limited behavioural research will be reviewed leading to an abstracted view of the forecasting process which highlights key areas where research should prove productive. Some of them, such as the heuristics used in adjusting statistical forecasts have seen much recent research and the somewhat conflicting findings suggest we do not yet understand the circumstances when judgmental adjustment is valuable. What information is valued in making adjustments and how do peopleâ€™s prior understanding and motivation affect the interpretation of the information they receive?

Judgmental forecasting in supply chains when special events are due to occur: A comparison of field and laboratory results.

Paul Goodwin

University of Bath Management School

Statistical forecasting methods excel at identifying patterns in large sets of data. However, when special events, such as product sales promotions, are forthcoming, judgmental adjustments to these forecasts are often advisable because of the scarcity of data. Field studies of these adjustments suggest that, while they often improve accuracy, they tend to be over-optimistic. However, three experiments that have attempted to replicate supply chain forecasting in the laboratory, have found that the judgmental adjustments suffered from a pessimism bias. This has persisted despite providing incentives for accurate forecasts, rewards for high sales (in an attempt to create a desirability bias), reminders about base rates, a reduction in mean sales uplifts and graphs with scales designed to encourage large upwards adjustments. The latest experiments have focused on understanding the role of information in the adjustment process. New results will be presented on how information is valued but these again pose questions as to how closely the laboratory study matches what we know of forecasting in the field. The possible reasons for the apparent differences between the field and laboratory results will be discussed, together with the implications for the design of laboratory studies in this area.

The Forecasting Principles site summarizes all useful knowledge about forecasting so that it can be used by researchers, practitioners, and educators. The site is devoted to improving decision making by furthering scientific forecasting.