01/01/03 puf

MODEL RISK MANAGEMENT:odel risk is no longer the purview of quants and
analysis groups hidden deep within the organiza-tion. With mark-to-market (MTM) accounting,
M the quants’ MTM pricing models now drive earn-
ings, and, as we all know, earnings drive CEOs.
Model risk was dragged into the spotlight with the 2002 and
2003 restatement of earnings by many energy-trading concerns,due in large part to over-optimistic model valuations, which
removed billions of dollars from the industry’s balance sheet.These losses were far in excess of any value-at-risk (VAR) limits,
causing stock prices and investor confidence to plummet. Giventhat many utilities as a matter of necessity must trade in energy
markets and new financial traders also are playing this market,the issue of model risk is again a cause for concern.
Furthermore, the triple-whammy of MTM accounting,
market illiquidity, and substantial legal liability for auditorsmassively increases model risk in the electricity industry. Manyparticipants focus on market and credit risk, which is a per-fectly reasonable approach, but in some cases it is to the exclu-sion of model risk management. This exclusion may causeundue risk to future earnings.Model Risk, MTM, and Mark-to-Model
What is model risk? A model can be defined as a mathemati-cal representation of price relationships in the marketplace.Model risk occurs when either the model does not representthe market accurately or when it may not appear to do so;both present big risks. If the model does not represent themarket accurately, the contract will be misvalued and earningswill be misstated. If the model appears to be inaccurate, audi-tors and investors may apply a higher uncertainty premium tothose claimed earnings.By John Bampfylde and David Shimko 52 PUBLIC UTILITIES FORTNIGHTLY DECEMBER 2004
IMPACT OF NEGATIVE EARNINGS RESTATEMENTS ON SHARE PRICE:
indexes include exchanges (e.g., NYMEX
ly into the index categories, however.
Experience shows that “plain vanilla”
* change is expressed for Net Income, not in EPS
indexes, often are relatively ineffective
** %change in EPS is an average of the % change from 2000-2002
hedges for a generator or retailer (in par-
*** this table includes all relevant negative earnings restatements, not only those due to pricing models
ticular). Thus, utilities seek out non-stan-dard, or exotic, products that may be more effective hedges and
Examples abound from the days of energy marketing in
perhaps better value. Examples include monthly contracts,
the late 1990s and early 2000s, when valuations were less rig-
structured transactions, options, non-firm contracts, and com-
orously audited. A marketer would do a deal and mark it to
market immediately to create instant “earnings” of $10 mil-
Energy traders model these exotic contracts as part of their
lion, $50 million, or even $100 million (yes, really); in some
core business and buy and sell based on the model’s values;
instances the counterparty on the other side of the deal
this is the business of trading. The problem arises when MTM
reported a large profit too, if that’s what their model said. This
accounting is applied. Most companies keep the exotic con-
tendency was exacerbated by the traders’ performance-related
tract either until delivery or for a long period of time, since
pay, with the “gamers” creating high unrealized “earnings” and
they are non-standard and therefore difficult to on-sell. Under
collecting an employee performance bonus of cold hard cash
MTM though, these contracts have to be frequently valued to
get the current liquidation value. The previously applied valu-
Companies have, and had, two approaches to this problem.
ation model is the most obvious way to calculate this MTM.
First, they apply operational risk management and effort to
This is “mark-to-market-through-a-model,” or mark-to-
ensure the model is a reasonable representation of real life (i.e.,
model, where the model’s output is taken almost directly to
the output values are as close as possible to actual market val-
the profit and loss (P&L). This is a risk because even if the
ues). This is an ongoing process. As more price data is observed,
model is a perfect representation of the market, others may
models can be modified to incorporate these observations.
not see it that way. Auditors will require proof that the model
For example, after a few years of observing out-of-the-
is perfect before they approve the accounts. Aggrieved share-
money option values, a volatility “smile” can be incorporated
holders could sue the company claiming the imperfect model
into the valuation process to capture the observed market
distorted earnings and hence share price. Not only does the
value above the theoretical value. The first guiding principle
model have to be perfect, it has to be seen as perfect.
is a simple check: At the time of the deal the most likely
But anyone who has traded in illiquid markets knows there
MTM value is slightly negative, a reflection of the bid-offer
is no such thing as a perfect model. An element of “black art”
or common sense is applied to all valuations. Model risk man-
Second, traders have well-developed procedures to track
agement is where these assumptions and the model’s logic are
and document how the model works, what changes have been
documented and tested so that the auditor and the aggrieved
done to it, and how these changes have altered the output val-
shareholder’s legal team can clearly see that the company has
ues. This documentation improves transparency of operation
been reasonable and conservative with regard to calculating
but, more importantly, is a fundamental part of the auditing
MTM earnings through internal models.
process and any legal defense. A lack of documentation andprocedure plays right into the hands of plaintiffs and creates a
Learning From History
The 2001/2002 crash of many energy traders’ stock prices,the many restatements of earnings, and the subsequent law-
Model Risk Management
suits provide good examples for how not to do MTM, as well
Model risk management, therefore, is the process of applying
as some shining examples of probity.
operational risk management to modelling and valuation
DECEMBER 2004 PUBLIC UTILITIES FORTNIGHTLY 53
by-line checking to ensure thatthe model uses the defined algo-
Anyone who has traded in illiquid markets
knows there is no such thing as a perfect
Black box testing is a bench-marking/calibration process, not
common sense is applied to all valuations.
looking so much at the formu-lae but how the model outputcompares to what is expected.
models to demonstrate that the models are robust and the
Given the complexity of many portfolio and risk models, black
assumptions reasonable and explicit. Good model risk man-
box testing often is the easier process. It also limits the poten-
agement requires the same discipline and attention that com-
tial for losing intellectual property in the process.
panies give to market risk management.
External review is a plug for specialist consulting services.
First, companies need a policy to dictate the minimum
Many companies don’t recognize that external review is also a
acceptable standards for all models. Models that allocate capi-
fixed cost. Since auditors have been trampled by not spotting
tal and directly impact earnings require the highest standards,
earnings manipulation, and they now spend a lot of time mak-
but almost all models in an organization require a minimum
ing sure they understand exactly what they are auditing. The
standard since, by definition, the models help with business
client pays for this time, either directly through the auditor or
decisions. The policy should describe the conventions of model
by more proactively managing the audit through specialist
development, ownership and management of models, model
structure conventions, acceptance testing, the sign-off process,and change management. The same policy standards and grav-
Corporate Action
itas that are applied to a company’s VAR calculation should be
The Financial Accounting Standards Board’s statement 133
applied to all valuation process models.
and its international counterpart, International Accounting
Second, each model should be thoroughly documented,
Standards statement 39, have pushed the energy industry
ideally during the building process. Documentation should
down the MTM route, which creates significant opportuni-
describe the model’s objective, the algorithms, and the inputs
ties for earnings swings and distortions. To avoid this distor-
and their respective units. Documentation also should pro-
tion or the appearance of distortion, companies need to apply
vide a user manual. The intention is that anyone provided
model risk management where a model creates a value that
with this documentation can use the model, a key part of per-
Energy marketers’ experience shows that common sense
Third, the model should be tested to ensure it functions as
can go a long way to limit model risk (i.e., “fast” MTM earn-
required and as defined in the documentation. By this stage
ings are a rare occurrence and are more likely a reflection of
the model should have been tidied up to remove unnecessary
mis-marking a deal). Common sense needs to be linked to a
code and to rationalize its operation and features to make it
structured model risk management program to ensure models
more company-standardized. The model should be accepted
maintain integrity and to leave an audit trail of assumptions
as fit for use only after successful testing. Then the model
should be frozen, preventing any changes from inadvertently
Model risk grows exponentially where MTM accounting,
altering the model’s output. User passwords and password pro-
illiquid markets, and complex products combine. These three
tection of code are two common tools employed.
features combine in power markets to create the perfect storm
Fourth, each model should include as an integral part of
for model risk. When auditor liability is thrown into this
the code a written commentary describing its version number
storm, it generates a large, unavoidable cost that can be mini-
and date, the developer/manager, and any changes or bugs
mized only through dedicated corporate model risk-manage-
ment action. F
Fifth, given the serious financial consequences of a flawed
model, key models should be reviewed or audited by external
John Bampfylde and David Shimko are partners at Risk Capital,
parties. There are two broad types of review (and acceptance
an independent risk management consultancy based in New
testing): white box and black box. White box testing is line-
York. Contact Shimko at [email protected]54 PUBLIC UTILITIES FORTNIGHTLY DECEMBER 2004