Two of the biggest buzzwords in our industry is “big data” and “data science”. Big Data seems to have a lot of interest right now, but Data Science is fast becoming a very hot topic.

I think there’s room to really define the science of data science – what are those fundamentals that are needed to make data science truly a science we can build upon?

Below are my thoughts for an outline for such a set of fundamentals:

Fundamentals of Data Science

Introduction

The easiest thing for people within the big data / analytics / data science disciplines is to say “I do data science”. However, when it comes to data science fundamentals, we need to ask the following critical questions: What really is “data”, what are we trying to do with data, and how do we apply scientific principles to achieve our goals with data?

– What is Data?
– The Goal of Data Science
– The Scientific Method

Probability and Statistics

The world is a probabilistic one, so we work with data that is probabilistic – meaning that, given a certain set of preconditions, data will appear to you in a specific way only part of the time. To apply data science properly, one must become familiar and comfortable with probability and statistics.

This section is one of the key fundamentals of data science. Whether applied in scientific, engineering, or business fields, we are trying to make decisions using data. Data itself isn’t useful unless it’s telling us something, which we’re making a decision about what it is telling us. How do we come up with those decisions? What are the factors that go into this decision making process? What is the best method for making decisions with data? This section tell us…

To bring various data elements together into a common decision making framework, we need to know how to align the data. Knowledge of coordinate systems and how they are used becomes important to lay a solid foundation for bringing disparate data together.

Once we understand coordinate systems, we can learn why to transform the data to get at the underlying information. This section describe how we can transform our data into other useful data products through various types of transformations, including the popular Fourier transform.

An often overlooked aspect of data science is the impact the algorithms we apply have on the information we are seeking to find. Merely applying algorithms and computations to create analytics and other data products has an impact on the effectiveness data-driven decision making ability. This section take us on a journey of advanced aspects of data science.

One of the key elements to data science is the willingness of practitioners to “get their hands dirty” with data. This means being able to write programs that access, process, and visualize data in important languages in science and industry. This section takes us on a tour of these important elements.

No data science fundamentals course would be complete without exposure to machine learning. However, it’s important to know that these techniques build upon the fundamentals described in previous sections. This section gives practitioners an understanding of useful and popular machine learning techniques and why they are applied.

Richard Feynman is one of the greatest scientific minds, and what I love about him, aside from his brilliance, is his perspective on why we perform science. I’ve been reading the compilation of short works of Feynman titled The Pleasure of Finding Things Out, and I recently came across a section that really hit home with me.

In the world of data science, much is made about the algorithms used to work with data, such as random forests or k-mean clustering. However, I believe there is a missing component – one that deals the fundamentals underlying data science, and that is the real science of data science.

The following paragraphs are taken from The Pleasure of Finding Things Out, which I would encourage you all to read. Feynman’s way of cutting through the scientific and mathematical gobbledygook to get to the essence of what all that stuff represents is remarkable, which in my mind just demonstrates his brilliance since he’s so able to communicate what he knows to other people. I’ve written on the importance of effective communication, especially in science – the most effective scientific communicators were Albert Einstein and Stephen Hawking; I would definitely put Richard Feynman in that class.

One way, that’s kind of a fun analogy in trying to get some idea of what we’re doing in trying to understand nature, is to imagine that the gods are playing some great game like chess, let’s say, and you don’t know the rules of the game, but you’re allowed to look at the board, at least from time to time, in a little corner, perhaps, and from these observations you try to figure out what the rules of the game are, what the rules of the pieces moving are. You might discover after a bit, for example, that when there’s only one bishop around on the board that the bishop maintains its color. Later on you might discover the law for the bishop as it moves on the diagonal which would explain the law that you understood before – that it maintained its color – and that would be analogous to discovering one law and then later finding a deeper understanding of it. Then things can happen, everything’s going good, you’ve got all the laws, it looks very good, and then all of a sudden some strange phenomenon occurs in some corner, so you being to investigate that – it’s castling, some thing you didn’t expect. We’re always, by the way in fundamental physics, always trying to investigate those things in which we don’t understand the conclusions. After we’ve checked them enough, we’re okay.

The thing that doesn’t fit is the thing that’s the most interesting, the part that doesn’t go according to what you expected. Also, we could have revolutions in physics: after you’ve noticed that the bishops maintain their color and they go along the diagonal and so on for such a long time and everybody knows that that’s true, then you suddenly discover one day in some chess game that the bishop doesn’t maintain its color, it changes its color. Only later do you discover a new possibility, that a bishop is captured and that a pawn went all the way down to the queen’s end to produce a new bishop – that can happen but you didn’t know it, and so it’s very analogous to the way our laws are: They sometimes look positive, they keep on working and all of a sudden some little gimmick shows that they’re wrong and then we have to investigate the conditions under which this bishop change of color happened and so forth, and gradually learn the new rule that explains it more deeply. Unlike the chess game, though, in [which] the rules become more complicated as you go along, in physics, when you discover new things, it look more simple. It appears on the whole to be more complicated because we learn about a greater experience – that is, we learn about more particles and new things – and so the laws look complicated again. But if you realize all the time what’s kind of wonderful – that is, if we expand our experience into wilder and wilder regions of experience – every once in a while we have these integrations when everything’s pulled together into a unification, in which it turns out to be simpler than it looked before.

If you are interested in the ultimate character of the physical world, or the complete world, and at the present time our only way to understand that is through a mathematical type of reasoning, then I don’t think a person can fully appreciate, or in fact can appreciate much of, these particular aspects of the world, the great depth of character of the universally of the laws, the relationships of things, without an understanding of mathematics. I don’t know any other way to do it, we don’t know any other way to describe it accurately… or to see the interrelationships without it. So I don’t think a person who hasn’t developed some mathematical sense is capable of fully appreciating this aspect of the world – don’t misunderstand me, there are many, many aspects of the world that mathematics is unnecessary for, such as love, which are very delightful and wonderful to appreciate and to feel awed and mysterious about; and I don’t mean to say that the only thing in the world is physics, but you were talking about physics and if that’s what you’re talking about, then to not know mathematics is a server limitation in understanding the world.

The connection here to data science is the search for understanding. Research and engineering teams use data science to explain things about the data, so that we can use that information later – maybe to make predictions, maybe for better explanations, maybe to make better products. However, the key part is the understanding and without that, data science is merely a collection of tools and techniques used to fit observations. Unless we seek to understand – trying the find “the why” – then we won’t really know whether our data science models, tools, or techniques are actually working.

If you are interested, these passages are from a television interview Feynman conducted as part of a BBC documentary Richard Feynman: No Ordinary Genius.

Question: Do you have any thoughts on the fundamental science of data science or on Richard Feynman? You can leave a comment below.

Related Posts:

]]>I’ve been performing data science before there was a field called “data science“, so I’ve had the opportunity to work with and hire a lot of great people. But if you’re trying to hire a data scientist, how do you know what to look for, and what should you consider in the interview process?

I’ve been doing what is now called “data science” since the early 1990s and have helped to hire numerous scientists and engineers over the years. The teams I’ve had the opportunity to work with are some of the best in the world, tackling some of the most challenging problems facing our country. These folks are also some of the smartest people I’ve ever had the opportunity to work with.

That said, not everyone is a good fit, and the discipline of data science requires important key elements. Hiring someone into your team is incredibly important to your business, especially if you’re a small startup or building a critical internal data science team; mistakes can be expensive in both time and money. This can be even more intimidating if you don’t have the background or experience in hiring scientists, especially someone responsible for this new discipline of working with data.

Through conducting many interviews with potential candidates, not everyone has met our standards. Also, we’ve made some mistakes in hiring that eventually got corrected over time, but might have been avoided if we followed our own lessons earlier. Hopefully I can give you a few helpful things to look for when trying to fill out your data science team.

A note of caution: I may say some things here that seem like heresy. I’ve viewed some job descriptions that make data science seem like it’s all about coding. Be careful! Data science is notall about coding; it’s about understanding what data represent and how to convert it into reliable information. Coding is necessary to make this happen, but there are true fundamentals – the real science of data science – that aren’t being taught well right now (more on that in another post…)

Given that, here are 10 key things you should consider when filling out your data science teams:

Look for strong math backgrounds. Data science requires a background in mathematics – this is really not negotiable. We made some mistakes in the past finding candidates that had strong software backgrounds, but didn’t really have the necessary math fundamentals. What happened? The software team didn’t have a strong appreciation for the data-crunching algorithms being developed, so there was a divide within our team; it became harder for the scientists and the software engineers to work together to achieve common goals. Knowledge of statistics, linear algebra, calculus, geometry, and trigonometry are baseline requirements. I’ve even heard stories of a company (not ours, thank goodness…) that had a programmer implement algorithms incorrectly, and didn’t really appreciate what was being done. An algorithm that used the sum of squared values was implemented by squaring the sum of the values, because it ran faster (by the way, these two algorithms are not the same!) This is a simple example, but if your implementation team doesn’t have a strong math background, they might not know the difference, causing you real problems down the road.

Seek the willingness to program, not necessarily specific languages. Here’s the heretical statement – don’t pay as much mind to the actual programming languages someone has on their resume. They need to have experience programming and show that they can get their hands dirty with coding. However, data science is about learning and discovering; you need to be flexible. So, look beyond the recitation of R, Python, Pig/Hive, C/C++, Perl, MATLAB, IDL, SQL, SAS, Java, Unix shell, Ruby, Scala… Any good scientist who is willing and able to program can pick up a new language. It’s much harder to get someone who knows the ins-and-outs of a particular language to be a good scientist. Keyword searches of résumés targeting specific languages may exclude some strong candidates, while also including others that are weak yet know what to put on their résumé. Manual review can lead to the same result, since if you’re looking for Hadoop experience, you may hire that and onboard a less-than-stellar data scientist in the process.

Make sure candidates have a probabilistic/statistical view of the world. In data science, nothing is black and white. Data has two driving elements – the behavioral (characterized by what we know – our model for how the data is observed) and the statistical (everything else that we currently don’t understand). The job of the quality data science team is to characterize the behavioral and deal with the statistical. As your team gets to know the data better, they will find even more subtle drivers and nuggets of information, turning what used to be statistical into a more accurate behavioral model (this is the cool part of data science – amazing predictive ability!). Candidates must have an appreciation for a probabilistic view of the world, meaning that when a certain condition occurs, you expect the data to appear a given way with some probability (or only some of the time). A background in statistics is an absolute must in data science, so look for that on the résumé and test for it in your interviewing. With that said…

Look for people who are detail-oriented and want to get to the root cause. Statistics come from the lack of information about what drives the data we observe, which you can get at when you have more data. Sometimes there is a real root cause to what we see, and good data scientists try to figure out why. Technical staff members that aren’t detail-oriented tend to make more mistakes than others who are, leading to inaccurate results and incorrect conclusions. I’ve seen really smart people find some very confusing results in their data analysis and be stumped as to what it was. When we looked into it further, it was merely a bug in their algorithm (not necessarily in their implementation) which led to some subtle errors. A probabilistic view of the world is important, but having a taste for getting to the bottom of things is equally as valuable.

Find people who can communicate effectively. An often overlooked quality for data science candidates is top communication skills. Even if someone is working alone on their data analysis, they have to communicate with someone, whether that is his boss or her colleague; no one works alone. I’ve written several articles about the importance of communicating (such as What We Can Learn From Stephen Hawking, Why Scientists Are Lousy Communicators, and tips on Job Interview Presentations), and it becomes especially important for those in the sciences. Math and science geeks think presenting is merely for marketers and sales people… Not so! If you want others to believe the results of your data science teams, your team has an obligation to communicate effectively.

Include your current scientific staff in interviews. We know that hiring is the job for managers. However, including your current staff in the interview process can yield real benefits. It can ensure that new candidates will be good fits for the organization and can even improve the company. In his 1998 letter to shareholders, Jeff Bezos, CEO of Amazon, detailed three questions that were asked of his hiring teams when evaluating candidates. Here’s what Bezos wrote about these questions:

Will you admire this person? If you think about the people you’ve admired in your life, they are probably people you’ve been able to learn from or take an example from. For myself, I’ve always tried hard to work only with people I admire, and I encourage folks here to be just as demanding. Life is definitely too short to do otherwise.

Will this person raise the average level of effectiveness of the group they’re entering? We want to fight entropy. The bar has to continuously go up. I ask people to visualize the company 5 years from now. At that point, each of us should look around and say, “The standards are so high now — boy, I’m glad I got in when I did!”

Along what dimension might this person be a superstar? Many people have unique skills, interests, and perspectives that enrich the work environment for all of us. It’s often something that’s not even related to their jobs. One person here is a National Spelling Bee champion (1978, I believe). I suspect it doesn’t help her in her everyday work, but it does make working here more fun if you can occasionally snag her in the hall with a quick challenge: “onomatopeoeia!”

Bring members of your current team in with the understanding that you’re looking for people who will make their team better, and the help from your current staff will be valuable in assessing talent.

Don’t get so hung up on brainteasers – whether they can or can’t answer them. I know that some companies like to put candidates on the spot and get them to solve brainteasers during their interview. Personally, I find this to be a waste of time and an inaccurate way to tell whether someone will work well as a data scientist on your team. Some people need a little time to work through a problem, but if they have that time, they nail it. Others get to the right answer by trying out many things, learn from their mistakes, and hone in on what works. Brainteasers would make these candidates look like they can’t do the job, so they’d get weeded out. Plus, if someone happened to solve a brainteaser quickly, it may mean that they’ve been exposed to that particular before, which is why they know it so easily (for example, here’s one: For any prime number p > 5, show me why p2-1 is divisible by 24…). You aren’t hiring someone who can solve the problem – you are hiring someone who can find the solution to the problem. They may solve it themselves (which can be especially important when the problem has never been solved before), but if it has been solved, why would you want someone who is predisposed to solving it over again? Instead…

Ask open-ended questions that provoke how people approach problems. There is a great book, Are You Smart Enough To Work at Google?, which details how Google evaluates candidates for their teams. There is even an insightful question they have asked: You are shrunk to theheight of a nickel and thrown into a blender. Your mass is reduced so that your density is the same as usual. The blades start moving in 60 seconds. What do you do? (How would you answer this?…) For their interview process, Google posts on their website how they approach it and what they look for. They generally look at four elements:

Leadership. We’ll want to know how you’ve flexed different muscles in different situations in order to mobilize a team. This might be by asserting a leadership role at work or with an organization, or by helping a team succeed when you weren’t officially appointed as the leader.

Role-Related Knowledge. We’re looking for people who have a variety of strengths and passions, not just isolated skill sets. We also want to make sure that you have the experience and the background that will set you up for success in your role. For engineering candidates in particular, we’ll be looking to check out your coding skills and technical areas of expertise.

How You Think. We’re less concerned about grades and transcripts and more interested in how you think. We’re likely to ask you some role-related questions that provide insight into how you solve problems. Show us how you would tackle the problem presented–don’t get hung up on nailing the “right” answer.

Googlyness. We want to get a feel for what makes you, well, you. We also want to make sure this is a place you’ll thrive, so we’ll be looking for signs around your comfort with ambiguity, your bias to action and your collaborative nature.

Use a group interview process when possible. Having an interview process that is back-to-back-to-back-to-back one-on-one interviews leads to repeat questions, making it tiring for the candidate. Additionally, when the interviewing team gets together to discuss the candidate (if they do at all), each member has a different perspective on the candidate because different questions may have been asked and different answers might have been given for the same repeat question. When you have multiple people (four to six) hearing the same thing as part of a group interview, you can get a better feel for the person coming on board. The information is the same, but different people pick up on different things, so it gives the team a more well-rounded perspective on the candidate. Something to keep in mind: These types of interviews can be intimidating for someone being interviewed, so it’s important to establish an environment of trust from the start. Make them comfortable so that you can get the best out of them.

Look for people that can tell you what they’ve learned, not just telling you what they did. Machine learning algorithms are great at exploiting separations in data. But, why are we looking for separations in data? To make better decisions with that data. The tools of data science are important to know, but if we don’t look for the “why” in the data science we are performing, then we are just using tools for the sake of using tools. Just because someone is an expert in hammers and nails doesn’t make him a carpenter. Extracting information out of data is all about context – what question are you asking of the data, and what drives what you see? This is about understanding, forming hypotheses, drawing conclusions. If a data scientist starts down the path of “we used this algorithm and the metrics came out like this…” without giving you some context or understanding of what it means, then you and your team could run into problems down the road (overfitting, building models that aren’t robust, etc.). It’s the difference between hearing about what you did on your summer vacation and what you learned on your summer vacation. In looking for what makes a good data scientist, DJ Patil talks about storytelling – the ability to use data to tell a story and to be able to communicate it effectively. Data scientists need to understand what they are trying to communicate and let their data science help them tell that story. No one really wants to hear what you did on your summer vacation, but they may want to know what you’ve learned and how you learned it.

There is so much data being created on a daily basis, now is a wonderful opportunity for companies that can leverage key data science disciplines. Knowing what to look for when hiring can let you take a solid step forward in building a successful data science team.

Question: Have you ever hired people to fill out your data science team? Interested in sharing your experiences in “what works” and “what doesn’t”? You can leave a comment below.

]]>Science and business seem like two very different disciplines, but is the best approach to learning any different in these two fields? These areas of life seem so unique, and the people in them can be quite varying (one with the nerdy pocket protector and the other dressed in the well-tailored suit). However, both science and business require learning, and the best approach to learning in each is really the same.

The best approach to learning is generally through failure. For example, Thomas Edison failed an astounding number of times before he invented a working lightbulb, and there are likely thousand of stories about how successes came as a result of many tries and many failures.

NKS, a discipline focused on how simple computational programs can create amazingly complex things, is really the scientific method applied by asking many, many questions and testing each question. The nature of the problems being solved requires that more questions be asked and more tests be conducted.

Science progresses by asking questions and performing tests, and some answers can’t be found unless we ask the question. NKS urges us to ask many questions and perform many tests, and in some cases, ask all possible questions and test each one. Magical and amazing things can be learned as a result.

In business, Eric Ries has written up his approach to building strong companies in his book, The Lean Startup. Ries’ approach is really the scientific method applied to business. Ries (and his mentor Steve Blank) notes that a new business venture is really a temporary organization in search of a repeatable and scalable business model. You can plan your new business all you want, but that plan usually falls apart as soon as it interacts with customers.

Ries describes how you have to pose a hypothesis (for example, customers are more focused on cost) and then perform a test to see if that is really true. It becomes so important to let real data tell you whether you’re on the right track or not. This may require performing many different types of tests and failing often. Keeping track of these failures will guide you toward the best way to succeed.

I’ve launched books that’ve failed. I did a book called “E-mails Addresses of the Rich and Famous” – Roger Ebert got really mad at me. I’ve made videotapes that didn’t work; I’ve made books that didn’t work.

My lesson was: If I fail more than you do, I win, because built into that lesson is this notion that you get to keep playing. If you get to keep playing, you get to keep failing, and sooner or later, you’re going to make it succeed.

The people who lose are either the ones who don’t fail at all and get stuck, or ones who fail so big they don’t get to play again…

If you’re talking to a pacemaker assemblyman or an airline pilot, they don’t try stuff; they don’t say, “I wonder what happens if I do this,” and we’re really glad they don’t do that, because the cost of failing is greater than the cost of discovering what works and what doesn’t.

But almost no one I know builds pacemakers and I don’t know airline pilots. Most of us now live in a world where the kind of failure I’m talking about isn’t fatal at all. If you post a blog post and it doesn’t resonate with people, post another one tomorrow. If you tweet something and no one retweets it, tweet again in an hour. If you’re obsessed with doing what everyone else is doing, because of someone saying “you failed,” then you’re in really big trouble.

Here’s a quick overview of what we need to do – in science and in business – to help learn more and succeed:

Set a goal. Decide what you want to do or what you want to learn. In science, this might be to find an algorithm that performs a certain task or to develop a model that describes how something in the world works. In business, this might be to come up with a product that serves a specific customer need. This is the same as asking a question, such as “Will this model predict what happens next?” or “Will this product serve my customer’s need?” In either case, you need to know what you’re trying to do first or what question you are asking.

Form a hypothesis. A hypothesis is a statement that tries to explain behavior; it’s your belief about why something happens. At this point, it’s only a guess, although an educated one, based upon your previous knowledge. An example hypothesis might be “I believe customers will buy my product because they really want to protect the environment.” How do we know if this is really true? We’ll test it out.

Predict the outcome. You need to test our hypothesis, which means understanding how feedback would come to you under two situations – (1) if your hypothesis is true and (2) if your hypothesis is false. This is a critically important step, and fundamental to being a good scientist or a solid businessperson. I could probably go into a whole other post about how critical this is (and how some really smart people aren’t as careful as they should be with this step…). Get really clear in your mind about these two things – what would the feedback look like if you are right and what would the feedback be if you are not right.

Try it out. Create an experiment that will collect the feedback. Ideally your test will give different answers if your hypothesis was correct or incorrect, and this way you’d be able to tell whether or not you’ve confirmed what you know.

Compare your results to your expectations. Once you have your feedback – data collected from your scientific or business experiment – you need to analyze it to see if it confirms your hypothesis or not. Is the feedback most consistent with your hypothesis or not? If the data is unclear, try a different experiment. If the feedback tells you that your hypothesis is wrong, great! You’ve learned something you didn’t know before, and you can ask another question and carry out another test to learn even more.

Testing your hypothesis and having be incorrect might be considered by some as a failure. However, without performing the test, you would never know that your insights was misplaced. This “failure” is really one more step toward success, and as Seth Godin says, if you keep failing, “sooner or later, you’re going to make it succeed.”

This is really how we learn – ask a question, form a hypothesis, test it out, and look at the results – and fail often to get to the successes.

Question: Besides science and business, are there other areas where you think the scientific method can be helpful? You can leave a comment below.

]]>I read a couple of items in this month’s Fortune magazine that I thought it was worth passing along.

The first was a small article by Brian Dumaine about the work being done at Applied Proteomics to identify cancer before it develops. At Applied Proteomics, they use mass spectroscopy to capture and catalog 360,000 different pieces of protein found in blood plasma, and then let supercomputers crunch on the data to identify anomalies associated with cancer. The company has raised $57 million in venture capital and is backed by Microsoft co-founder Paul Allen. You can read the first bit of the article here.

The second is from the Word Check callout, showing how access to information is making the word a better place:

wasa: Pronounced [wah-SUH]

(noun) Arabic slang: A display of partiality toward a favored person or group without regard for their qualifications. A system that drives much of life in the Middle East — from getting into a good school to landing a good job.

]]>I found this set of business wisdoms in the August 2013 issue of Entrepreneur magazine. While not perfect mantras by which to guide a business, I thought there were pretty fun.

=================

Chris Hardwick didn’t rely on just his nerdy instincts in founding his company; he also took inspiration from his heroes. Super-power your business with these lessons from some epic nerd properties.

SUPERMAN: In this summer’s film Man of Steel, Superman’s dad tells his son, “You will give the people of Earth an ideal to strive torwards.” Hardwick believes businesses can achieve that, too. “Altruism,” he says. “That’s what businesses should learn from Superman.”

LORD OF THE RINGS: With all the adventures of Frodo and the fellowship, J.R.R. Tolkien’s tale has plenty of lessons for startups. “Always do what you believe is right,” Hardwick advises. “No matter how much you think you can handle it, don’t pick up the ring… Don’t toy with darkness.”

X-MEN: As diverse as Wolverine and the gang were, they all had a unified vision, organized behind the infrastructure built by Professor X. “He gave them tools; he gave them a home base; he organized the community,” Hardwick says. “He’s the CEO of X-Men, basically.”

SUPER MARIO: Hardwick’s favorite lesson is that of perserverance. “No matter how many times the princess gets kidnapped, you’re going to rescue her,” he says.

STAR TREK: With species ranging from Vulcans to humans, the voyages of the starship Enterprise are all about diversity, Hardwick says. “Race or species was irrelevant,” he points out. “It was all about working together as a team.”

THE WALKING DEAD: While zombies might be out for themselves (and brains!), Hardwick believes the popular comic book and TV show is all about community. “Essentially they assemble a party of experts in their world who are all good at something and contribute to the group,” he says.

BATMAN: Remember when Bruce Wayne’s parents were killed? Well, Batman never forgot it. And by studying the Dark Knight, Hardwick notes, entrepreneurs can learn about “turning adversity into something constructive.”

STAR WARS: The sprawling epic of Darth Vader and son Luke Skywalker has a simple message. “It’s the David and Goliath story,” Hardwick says. “Fight for what you believe in, don’t stop at any cost, and you’ll triumph.”

Related Posts:

]]>Ever wonder what your own personal network looks like? You are likely connected to many different groups (family, friends, community, work), but do you know how they are connected? Or are they connected at all? Are you the glue that connects these various groups?

This is a great age we’re living in, and I’m glad to be involved with developing lots of really advanced technologies. One of the technology areas that I’m really fascinated with has been pushed forward by Stephen Wolfram. He created the industry standard computing environment Mathematica, which now serves as the engine behind his company’s newest creation, Wolfram|Alpha. (I’ve written a few posts on Wolfram|Alpha in the past, and you can read them here and here).

One feature that they’ve recently added to Wolfram|Alpha is the ability to analyze your Facebook data. Usually, if you use Facebook, you only focus on the posts your friends make – pictures from their great vacations, LOLcats, or sharing articles for other websites (like this one!) However, here are three reasons why it might be worth it for you to unlock these insights from Facebook:

It gives you insight into your connections and their connections. For example, I happen to have a number of groups that I’m connected with. Some are work-related (Areté and Mentor Graphics), some are community-related (Thousand Oaks), some are from where I grew up (Brillion and Virden), and others are politically-related (Ross Perot). With this view of what’s called your social graph, you can see a view of who you are, based on looking at who you’re connected with.

You learn about yourself. Getting your Facebook report through Wolfram|Alpha is kind of like looking in a different type of mirror. You get to see yourself through your own data; it can help you improve in areas where you want to see improvement – I even wrote a post about why it can be good to collect data on yourself.

It’s fun. Viewing yourself in different ways can be interesting and fun! Sometimes it takes these different views to really understand who you are and how you got here.

If you’re interested in unlocking your Facebook data using Wolfram|Alpha, here are some simple steps:

Go to www.wolframalpha.com. It looks very much like the Google search page with a single bar for entering text

Type in “facebook report” or you can click on the stylized Facebook icon.

Wolfram|Alpha will then ask you to click “Analyze My Facebook Data”

Once you’ve done this, Wolfram|Alpha will generate a long report, giving you many views on your data and yourself. If you’re interested, there is a post from the Wolfram|Alpha blog that explains these new features and another good article to read from NBCNews.com.

New technology is allowing us to see more views of ourselves for self-improvement and for entertainment. Take some time and use Wolfram|Alpha to learn a little more about yourself.

Question: Have you ever used Wolfram|Alpha? Are there any other tools you find interesting in looking at your own social network? You can leave a comment below.

]]>This is a technical post about what I’ve discovered in creating my own custom URL shortener. Hopefully, you can learn to do the same things I did, and my experience will save you some headaches if it’s something you’re interesting in trying.

On my website, I focus a lot about decisions and discovery. I love finding out how the world works and then applying what I’ve learned to make better decisions, and I also try to share what I can along the way. I hope that it helps others.

When it came to URL shortening, I was interested by a post from Michael Hyatt, who has a tutorial on how he created his own custom URL shortener. I was intrigued, since he has a great how-to guide on his website. However, I didn’t want to pay for an additional service, so I tried to find out how to create one myself and do it inexpensively. Luckily, I discovered how to do it, and I’m happy with the results, so I thought I’d write this post to share my experience.

Next question, what is a URL shortener and why would I want to use one? It’s a service that allows you to create a link to one of these pages, maybe one to somewhere on your website or anywhere else on the Internet, using a much shorter link name. Bit.ly is such a service, and it’s the one that I use. Here’s an example of why I like to shorten my URLs.

I have a post on my website about Stephen Hawking and his amazing ability to communicate, even though he suffers from ALS. The link itself, as you can see below, is very long:

This link itself is 109 characters, so it takes up a lot of room. Also, if I wanted to share this link on Twitter, I’d use up 109 of the 140 characters that Twitter allows. If the link were longer, I might not be able to pass along the link at all! It would be nice to shorten this link and let people reach this article while still providing a message to interest them in following the link.

I can do this by going to bit.ly and entering the URL itself. Bit.ly will then create a shorter URL that will take me to exactly the same place:

There are a number of reasons why creating a custom short URL can be a benefit for you:

Using a URL shortener allows you to share more with others on Twitter and Facebook. When the links you share don’t take up so much space, you can focus on your message to your readers. And it becomes especially important when using Twitter, since you are only limited to 140 characters for your tweet.

You can keep track of the number of clicks your shared posts get, allowing you to better understand your readership. Bit.ly keeps track of the number of times people click your shortened links, so you can get a sense of which links are more popular and when people chose to click on them.

Making a custom URL shortener allows for more consistent branding. It’s great to use a URL shortener for sharing links with, say, your Twitter followers. However, if you’re able to do this while continuing to promote your website or business, then sharing this information with your followers becomes even more effective.

So, now that you know the benefits, are you ready to learn how to create one for yourself? Great!

Here are the steps that I took in getting and setting up my custom short URL:

Buy the URL you’d like to use for URL shortening. You will have one for your blog or website, but you’ll want to use a different URL for URL shortening. Select one that brands yourself, your website, or your business well, but keep it short (otherwise it defeats the purpose!).

Twitter allows for 140 characters, so keep your shortening URL to 12 characters or less. I use micfarris.us for my URL shortening, but the New York Times uses nyti.ms. This way, you can tweet a shortened URL link (such as micfarris.us/WGtCik) and still have enough room to tweet a helpful message.

Check out available web addresses. You can go to http://domai.nr/ to check out available URLs. I eventually chose micfarris.us for two reasons:

It contains “micfarris” to further the branding for myself and my website

The .us domains are a lot cheaper to purchase. GoDaddy sells .us domains for $3.99 for the first year, so it’s an inexpensive way to get started. I looked at getting micfarr.is (which is a domain from Iceland), but it cost $99/year, so I decided against it (for now!)

Connect with a URL shortening service to let them know your new domain name. As I mentioned before, I use bit.ly – it’s free, and they do a great job with their URL shortening service. Here are the steps for performing this step using bit.ly:

Sign in to your bit.ly account (or if you don’t have one, just create one)

Go to “Settings” from the upper right pull-down menu (or click here), click the “Advanced” tab, and then click the “Add a custom short domain” link.

Enter your new domain to assign it to your account

Connect your new domain to the website for your URL shortening service. Here are the steps for performing this step using GoDaddy.com (the place that keeps my web address) and bit.ly (that performs the URL shortening):

Log into GoDaddy.com, click on “My Account”, then click on “Domains”

Click on the new domain you want to connect with bit.ly

Under the “DNS Manager” heading, click on the Launch link

The “A (Host)” record should be the first on the page, so we’ll want to change this to point to the bit.ly website. You’ll want to change the IP address (the four number sequence in this record) to 69.58.188.49. You should double check with bit.ly to make sure this is the right IP address (don’t just take my word for it!) To double check the IP address and for more specific information from bit.ly, you can go here.

Wait patiently. It can take up to 48 hours for the new information to propagate the right settings through the servers. When I set up my custom short URL, it took less than an hour or so, but sometime it can take longer.

Setting up your own custom short URL for branding yourself, your website, and your business is easy. It was surprisingly painless for me to set this up, and I’m sure that you can follow these simple steps to create your own, just like I did.

Question: Have you ever thought about creating your own custom URL for Twitter? Are there other short URL services that you like? You can leave a comment below.

]]>It’s a complex world, and we are constantly making decisions. Just imagine the number of decisions we make about breakfast: How big a breakfast should I have? Should I have coffee? If so, how much? Should I have toast? Should I use butter? Should I have one piece or two? Should I cut the toast? If so, should they be cut into rectangles or triangles? Should I keep the crust? Should I have juice? Should it be apple juice or orange juice? How about milk? I haven’t even gotten to the pancakes, waffles, syrup, sausage, cereal, bacon… (mmm, bacon…)

And these aren’t the really important ones! How do we know we’re making good decisions, and can we make better ones?

In my professional life, I’ve spent decades understanding and applying the theory of making decisions. Our teams have worked to teach computers to make decisions automatically from tons and tons of data. In fact, these disciplines are now incredibly important for new technology development.

But understanding how decisions are made doesn’t only apply to technology. There are definitely things we can learn from this understanding to help us make better decisions ourselves.

Here are three things that are important to recognize about making decisions:

We don’t know everything. We may not have all the information we might like in order to make our decisions. For example, if you’re playing a card game like poker or bridge, you don’t know what cards the other players have. This lack of knowledge is called uncertainty. Recognizing uncertainty is the first key to making better decisions, since uncertainty is all around us.

We can’t know everything. The sheer number of possibilities for what we see in life makes it impossible to know things with certainty. (In fact, if you can believe it, quantum physics tells us we aren’t able to know everything, at least through our observations, but that’s another story…). There are things that we can get to the bottom of, but don’t sweat trying to get to the bottom of everything; you actually can’t.

There are likely many possible explanations to what we see. Since we don’t (and can’t) know everything, there might be multiple reasons why the information we have came to us. This doesn’t mean that we should get overwhelmed and be afraid of making a wrong decision. Our job is to figure out the most likely explanation and then make our decision with the knowledge.

Making better decisions means first recognizing that life is filled with uncertainty and we’re never getting rid of it. However, we can take steps to reduce this uncertainty and learn how to make better decisions as a result.

P.S. If you’re interested in a good book on uncertainty and how to make better predictions in light of this uncertainty, I have a review of Nate Silver’s book here.

Question: Have you ever been uncomfortable making decisions because you felt you didn’t know enough? You can leave a comment below.

]]>You might think that it’s a bit odd, treating yourself like a science experiement. However, the best way to achieve your goals may be to do just that – be committed to collecting data on yourself.

In science, we’re always collecting data and analyzing it to find out more about the world. However, collecting data isn’t only for people with pocket protectors (although we don’t all wear those!). It is something that any of us can use to help us achieve any goal we set for ourselves.

Several years ago, I used to weigh a lot more than I do now. At some point, I just decided that I wanted to get to a healthier weight. I was concerned about my long term health staying at this higher weight, and I knew if I didn’t take this seriously, I wouldn’t be able to enjoy much of life later on.

I decided to collect data on myself so that I could see how I was doing over time. I weighed myself every morning and recorded it in an iPhone app. I even kept track of how many calories I ate each day. This forced me to see what every handful of snacks and bowl of ice cream was costing me toward my goal of a lower target weight. Eventually I lost 40 pounds from my peak weight, and I’ve kept (most of) it off ever since.

Here are five reasons why you should consider collecting data on yourself to achieve your goals:

Looking at your data shows how you’re trending. If you have a goal in mind, such as losing twenty pounds, you need to know how you’re doing. This can only happen if you are committed to collecting data every day, and watching how the data changes. If you’re getting closer to your goal, you’ll see your weight drop over time.

Not taking data can trick you into thinking we’re on track. It’s far easier to convince yourself you are on track if there is nothing to counter you. However, in science, data is king. If you’re serious about achieving your goal, then you’ll be happy to collect data on yourself to know you’ll get there.

It works for anything. Keeping track of your weight is an easy example, but it truly helps with any goal you set for yourself. Collecting data on yourself is good for your personal development and growth. It can also work for your business (keeping track of new customer contacts and new sales) and even for your community (funds raised for local charities or scholarships for worthy students). It even works for gaining a general understanding of how the world works, which is the ultimate goal of science.

It keeps you honest. You can’t fool the data. If your goal is to lose twenty pounds and you haven’t lost a single pound for an entire week, you know that you haven’t made progress. The data will tell you that something needs to change, and you can make that change to keep you moving forward. Keeping on track requires you to be honest with yourself, and collecting data on yourself helps you do just that.

You learn more about yourself. As you collect your own data and take a look at how you’re doing, you’ll learn new things about yourself. Am I focused enough on my goals? Is it getting any easier? What can I do to acheive my goals faster? Can I even set a new goal, surpassing what I first thought I could achieve?

We can always do more to help ourselves keep us on track. While the first thing we need is the goal itself, we also need to collect the information that keeps us honest about achieving that goal. Be committed to collecting data on yourself and your achievements will start piling up before you know it.

Question: Have you ever tried collecting data on yourself? If so, what did you learn? If not, do you know where to start? You can leave a comment below.