2
2 Introduction An analysis whose results cannot be understood is as good as one that is never performed. General techniques –Line charts, bar charts, pie charts, histograms Some specific techniques –Gantt charts, Kiviat graphs … A picture is worth a thousand words –Plus, easier to look at, more interesting It’s not what you say, but how you say it. – A. Putt

3
3 Outline Types of Variables Guidelines Common Mistakes Pictorial Games Special Purpose Charts Decision Maker’s Games Ratio Games

9
9 Guidelines for Good Graphs (4 of 5) Use commonly accepted practices –Present what people expect –Ex: origin at (0,0) –Ex: independent (cause) on x-axis, dependent (effect) on y-axis –Ex: x-axis scale is linear –Ex: increase left to right, bottom to top –Ex: scale divisions equal Departures are permitted, but require extra effort from reader so use sparingly

11
11 Guidelines for Good Graphs (Summary) Checklist in Jain, Box 10.1, p. 143 The more “yes” answers, the better –But, again, may consciously decide not to follow these guidelines if better without them In practice, takes several trials before arriving at “best” graph Want to present the message the most: accurately, simply, concisely, logically

12
12 Outline Types of Variables Guidelines Common Mistakes Pictorial Games Special Purpose Charts Decision Maker’s Games Ratio Games

13
13 Common Mistakes (1 of 6) Presenting too many alternatives on one chart Guidelines –More than 5 to 7 messages is too many (Maybe related to the limit of human short- term memory?) –Line chart with 6 curves or less –Column chart with 10 bars –Pie chart with 8 components –Each cell in histogram should have 5+ values

14
14 Common Mistakes (2 of 6) Presenting many y-variables on a single chart –Better to make separate graphs –Plotting many y-variables saves space, but better to requires reader to figure out relationship Space constraints for journal/conf! throughpututilization Response time

15
15 Common Mistakes (3 of 6) Using symbols in place of text More difficult to read symbols than text Reader must flip through report to see symbol mapping to text –Even if “save” writers time, really “wastes” it since reader is likely to skip! Y=1 Y=3 Y=5   1 job/sec 3 jobs/sec 5 jobs/sec Arrival rate Service rate

16
16 Common Mistakes (4 of 6) Placing extraneous information on the chart –Goal is to convey particular message, so extra information is distracting –Ex: using gridlines only when exact values are expected to be read –Ex: “per-system” data when average data is only part of message required

18
18 Common Mistakes (6 of 6) Using a Line Chart instead of Column Chart –Lines joining successive points signify that they can be approximately interpolated –If don’t have meaning, should not use line chart MIPS - No linear relationship between processor types! - Instead, use column chart

19
19 Outline Types of Variables Guidelines Common Mistakes Pictorial Games Special Purpose Charts Decision Maker’s Games Ratio Games

20
20 Pictorial Games Can deceive as easily as can convey meaning Note, not always a question of bad practice but should be aware of techniques when reading performance evaluation

22
22 Non-Zero Origins to Emphasize (2 of 2) Choose scale so that vertical height of highest point is at least ¾ of the horizontal offset of right-most point –Three-quarters rule (And represent origin as 0,0) MINE YOURS

23
23 Using Double-Whammy Graph Two curves can have twice as much impact –But if two metrics are related, knowing one predicts other … so use one! Response Time Goodput Number of Users

25
25 Pictograms Scaled by Height If scaling pictograms, do by area not height since eye drawn to area –Ex: twice as good  doubling height quadruples area MINEYOURSMINEYOURS (Worse) (Better)

26
26 Using Inappropriate Cell Size in Histogram Getting cell size “right” always takes more than one attempt –If too large, all points in same cell –If too small, lacks smoothness Frequency Frequency Same data. Left is “normal” and right is “exponential”

27
27 Using Broken Scales in Column Charts By breaking scale in middle, can exaggerate differences –May be trivial, but then looks significant –Similar to “zero origin” problem System A-F

28
28 Outline Types of Variables Guidelines Common Mistakes Pictorial Games Special Purpose Charts Decision Maker’s Games Ratio Games

35
35 Kiviat Graphs (1 of 2) Also called “star charts” or “radar plots” ½ are HB, ½ are LB Note, don’t have to have all at 100% can be “10% busy”, say Useful for looking at balance between HB and LB metrics (“Star” is best) (Geoff Kuenning, 1998) HB LB HB LB

37
37 Outline Types of Variables Guidelines Common Mistakes Pictorial Games Special Purpose Charts Decision Maker’s Games Ratio Games

38
38 Decision Maker’s Games Even if perf analysis is correctly done, may not convince decision makers (boss, conference referees, thesis advisor…) –Box 10.2, p. 162 has list of reasons Most common: 1) “More analysis.” This is always true. Does not mean analysis done is not valuable. 2) “Alternate workload”. Since based on past, can always be questioned as good future workload Lead to endless discussion (“rat holes”). Can “head off” criticism by stating this.

39
39 Outline Types of Variable Guidelines Common Mistakes Pictorial Games Special Purpose Charts Decision Maker’s Games Ratio Games

40
40 Ratio Games (Ch 11) A common way to play games with competitors Two ratios with different bases cannot be compared or averaged –Doing so is called “ratio game” Knowledge of “ratio games” will help protect ourselves, avoid doing If you can’t convince them, confuse them. – Truman’s Law

41
41 Games with Base System Beware! –Normalize each system’s performance for each workload by system A and average ratios –Normalize each system’s performance for each workload by system B and average ratiosWork- Systemload 1load 2Average A B102015Work- Systemload 1load 2Average A B111

42
42 Games with Ratio Metrics Choose a metric that is ratio of two other metrics. Power = thrput/respTime NetworkThrputRespTimePower A1025 B414 Suggests that A is better. But maybe it should be: power = thrput/respTime 2  Power A = 2.5, Power B = 4

43
43 Games with Relative Performance Metric may be specified but can still get ratio game if two are on different machines MFLOPS, System X-Y, accelerators A-B AlternativeWithoutWithRatio A on X B on Y (Base systems are different)

44
44 Games with Percentages (1 of 2) Percentages are really ratios, but disguised –So can play games A is worse under both tests  but it looks better in Total!

45
45 Games with Percentages (2 of 2) Percentages –Have bigger psychological impact 1000% sounds bigger than 10-fold –Are great when both original and final performance are lousy Ex: payment was $40 per week, is now $80 When used, base should be initial, not final value –Ex: Price was $400, now $100 Drop of 400%! But that makes no sense

46
46 Strategies for Winning Ratio Game (1 of 2) (Again, don’t do these, just be aware of them so no-one does them to you) If one system is better by all measures, a ratio game won’t (usually) work –Although, remember percent-passes example! –And selecting the base also lets you change the magnitude of the difference If each system wins on some measures, ratio games might be possible –May have to try all bases

47
47 Strategies for Winning Ratio Game (2 of 2) Work- Systemload 1 load 2Base BBase A A B For LB metrics, use your system as the base –Ex: response time For HB metrics, use the other system as a base –Ex: throughput If possible, adjust lengths of benchmarks –Run longer when your system performs best –Run short when your system is worst –This gives greater weight to your strengths

48
48 Extra Credit for Next Class Bring in one either notoriously bad or exceptionally good example of data presentation –The bad ones may be more fun From proceedings, technical documentation, newspaper … Make copies before class or send to me and I’ll make copies We’ll discuss why good/bad