The open source HPCC Systems platform is a proven, easy to use solution for managing data at scale. Visit our Easy Guide to learn more about this completely free platform, test drive some code in the online Playground, and get started today.

Fighting crime isn’t easy, and no matter how hard we work, there will always be tragedies and missed opportunities. But what if we could work to anticipate crimes? What if we could stop them before they happen? No, this isn’t Minority Report, but just another instance of using big data to predict the future.

Predicting the future may be a little too bold, but it’s not far off. Using the ever-increasing amounts of data we create everyday, big data toolshelp us recognize patterns, important variables and other key insights we can use to create predictive models. Already, we have seen big data potential in fighting and preventing disease. Could law enforcement not use the same techniques to fight and anticipate criminal activity? The LAPD is just one example of many using crime data, and other pieces of information, in an effort to improve crime prevention. Big data analytics have helped officers determine criminal hotspots, and have effectively lowered crime across Los Angeles. Already the LAPD has seen a reduction of 33 percent in burglaries, 21 percent in violent crime, and 12 percent in property crime in areas where these algorithms are being applied.

However, it isn’t just local law enforcement looking to benefit from big data. In an effort to take things further, the Federal Government is teaming up with big data and social media in the hopes of cracking down on hate speech and preventing hate crimes. Unfortunately, hate crimes continue to be a major issue in the United States, with nearly 6,000 incidents reported in 2013. The Government is hoping to develop algorithms that could identify regions that tend to have higher instances of hate speech. From there, authorities could focus their efforts on improving awareness and outreach programs to combat the problem. Also, like in the case of the LAPD, officials could target potential hotspots or aggressors.

This may seem like a great solution. After all, we could stop crime before it strikes, which could mean saving lives. However, this approach could come at a cost some aren’t willing to pay. At what point does constant monitoring of our conversations start to trample our right to free speech. As deplorable as some of these comments may be, this is still the United States, and people are entitled to speak their mind. In addition, all of this begs an interesting question; what would someone have to say, or post online, in order to become a person of interest in the federal database? If someone makes one simple comment, or shares an image for fun, are they now subject to greater oversight by the federal program? Will they suddenly be a suspect in every hate crime that occurs in their area? And finally, who decides what is considered a threat, hate speech or simply a joke?

The issues don’t stop at interpreting online conversations. Whenever we look to anticipate behavior or predict crime, there is a serious concern for potential discrimination and profiling. When we start using data to profile possible criminals, we walk into a grey area. Someone may exhibit questionable behavior, but can we really condemn them if they haven't committed a crime? If analysis determines those with brown hair are 75 percent more likely to commit a hate crime, do we stop and question everyone with brown hair when we see them in a criminal hotspot? Will we start to follow people around, or label them as racists, simply because they live in areas our analyses have shown to have higher occurrences of hate speech?

It’s certainly possible that big data, and big data analytics, could hold great potential in promoting civil rights and combating hate speech. Already we have seen the success of the LAPD and other crime units using data as a crime fighting strategy. The hope is it will serve as a tool to protect and empower those who feel underserved and voiceless, while helping us overcome prejudice and preventing crime. Still, authorities need to be very careful to not overstep their rights, and ignore civil freedoms. Overcoming one problem simply to create another won’t solve anything.

Managing data at scale doesn’t have to be hard. Find out how the completely free, open source HPCC Systems platform makes it easier to update, easier to program, easier to integrate data, and easier to manage clusters. Download and get started today.