5
5 Example: Frequent Itemsets uA common marketing problem: examine what people buy together to discover patterns. 1.What pairs of items are unusually often found together at Safeway checkout? Answer: diapers and beer. 2.What books are likely to be bought by the same Amazon customer?

8
8 Models vs. Analytic Processing uTo a database person, data-mining is a powerful form of analytic processing --- queries that examine large amounts of data. wResult is the data that answers the query. uTo a statistician, data-mining is the inference of models. wResult is the parameters of the model.

10
10 Meaningfulness of Answers uA big risk when data mining is that you will “discover” patterns that are meaningless. uStatisticians call it Bonferroni’s principle: (roughly) if you look in more places for interesting patterns than your amount of data will support, you are bound to find crap.

11
11 Examples uA big objection to TIA was that it was looking for so many vague connections that it was sure to find things that were bogus and thus violate innocents’ privacy. uThe Rhine Paradox: a great example of how not to conduct scientific research.

12
12 Rhine Paradox --- (1) uDavid Rhine was a parapsychologist in the 1950’s who hypothesized that some people had Extra-Sensory Perception. uHe devised an experiment where subjects were asked to guess 10 hidden cards --- red or blue. uHe discovered that almost 1 in 1000 had ESP --- they were able to get all 10 right!

13
13 Rhine Paradox --- (2) uHe told these people they had ESP and called them in for another test of the same type. uAlas, he discovered that almost all of them had lost their ESP. uWhat did he conclude? wAnswer on next slide.

14
14 Rhine Paradox --- (3) uHe concluded that you shouldn’t tell people they have ESP; it causes them to lose it.

15
15 A Concrete Example uThis example illustrates a problem with intelligence-gathering. uSuppose we believe that certain groups of evil-doers are meeting occasionally in hotels to plot doing evil. uWe want to find people who at least twice have stayed at the same hotel on the same day.

16
16 The Details u10 9 people being tracked. u1000 days. uEach person stays in a hotel 1% of the time (10 days out of 1000). uHotels hold 100 people (so 10 5 hotels). uIf everyone behaves randomly (I.e., no evil-doers) will the data mining detect anything suspicious?

17
17 Calculations --- (1) uProbability that persons p and q will be at the same hotel on day d : w1/100 * 1/100 * = uProbability that p and q will be at the same hotel on two given days: w10 -9 * = uPairs of days: w5*10 5.

18
18 Calculations --- (2) uProbability that p and q will be at the same hotel on some two days: w5*10 5 * = 5* uPairs of people: w5* uExpected number of suspicious pairs of people: w5*10 17 * 5* = 250,000.

19
19 Conclusion uSuppose there are (say) 10 pairs of evil-doers who definitely stayed at the same hotel twice. uAnalysts have to sift through 250,010 candidates to find the 10 real cases. wNot gonna happen. wBut how can we improve the scheme?