The path to an intelligent assistent

Machine Learning, Artificial Intelligence and Cognitive Search

Using machine learning methods trained systems can be used to independently index and categorize vast amounts of data. The machine recognizes the content of the document and classifies it according to previous assumptions and experiences. For the search the underlying vector mathematics offers new scenarios e.g. a comprehensive search in foreign language texts without translating them.

Since the advent of of speech based daily helper like Amazon’s Alexa, Apple’s Siri or Google Home, machines take over routine tasks which in the past cost us time, power and efforts. “Alexa, put dog food onto my grocery list!”. “Siri, look for a restaurant near my current position!”. “Ok, Google. Play some classical music!”. With every entry and every command, these devices become more familiar and after a short while Google, Facebook and Co. may know us better than we do ourselves.

Even if we should view some developments rather critical, in most cases they offer extra value and the advantages of machine learning will have a similar impact on our life like the invention of the fridge or the telephone. Although a survival is possible in principle, to use it makes our lives much more comfortable.

Terminology and Technology

Before we start talking about digital assistants, we should straight out some terms we might run into. Fanned by the media and the presence of buzzwords like “artificial intelligence”, “machine learning” and “neural networks” in various publications, there is a noticeable insecurity about the meaning of the key termini.

Artificial Intelligence is the generic term and branch of informatics. The aim of AI is the imitation of human behavior. Machines try to mimic human decision patterns and with the help of advanced programming will be able to decide interdependently.

Machine Learning is part of the artificial intelligence branch and uses methods to implement the AI ideas. main feature of ML is the training of machines (or computers) with the help of examples. Out of these, the machine develops some sort of patterns to solve task independently at a certain stage.

Cognitive Search is a use case or a specialty of machine learning and mainly deals with the search and information retrieval based on neural networks.

Be smart – Information Management using AI-Methods and Machine Learning

The foundation for every machine learning is a broad and sometimes growing data pool. The better the information, the more detailed the later results. Out of a huge amount of data some training data will be extracted and the machine trained by a data scientist. At a certain stage the developing model will start to make its own decisions. Base for this are insights and assumptions on previously data used. At the end, the model rarely or never requires human interference, except the result deviates too far.

The process of machine learning

While processing texts with methods of machine learning e.g. word-embeddings, terms will be transfered into a multi-dimensional vector space. Every terms holds various parameters (frequency, density, etc.), has a unique position within the space and this way is mathematical describable. Through methods of vector analysis each position can be calculated and put into relation to others offering new insights. For example, the word “shoe” and the word “skid” may result in “ice skate” or “inline skates” since they are located between the two in a vector space.

Vectorized German terms in a 2D vector space

While working with texts and language, some bumps my occur on the road ahead, that in previous days could only be resolved with higher effort. The variety of speech on matters of expression or meaning cause problems when for instance words have similar meanings (synonyms) or in the opposite a search term has multiple ones (homonyms). In the past, one had to edit lists or Thesauri to develop connections. Take the word “light”: this could describe a visible electromagnetic radiation and at the same time a form of weight. Search engines previously were not able to “understand” these nuances in language.

Using so called word-embeddings or word co-occurrences – two methods of machine learning – this context can easily be made and developed by an associative network. The important question however is, in which relation the words stand to each other. The following example illustrates the idea with the word “bank”. In English a bank may have various meanings revealing the intended only from a context. If the document is about finance, money and capital the financial institute is probably meant. But if expressions like river, sediment or stream are found, this bank refers to a sandbank.

The word bank as an example

The extra value for search

Now, what is the actual extra value of machine learning for a search? To use these methods while searching through documents and texts will enhance the overall information yield. Especially the classification of content of larger amounts of data can now be entrusted to a trained machine, exploring either existing but also new relations. Additionally, the content could be used to extract relevant keywords and adding it to the document itself, enriching the informational value.

While vectorizing words and transferring them into the world of mathematics, another advantage arises. Terms from different languages usually create the same patterns in a vector space.

related vectorized terms in a vector space

related vectorized terms in German

Instead of translating content the search looks for similar patterns in various language spaces.

The example above shows another aspect of cognitive search. Terms close around a search word may be used for new search in similar contents. If you are looking e.g. for car, the system may extend its results to tires or train as well.

The methods of machine learning and especially cognitive search offers new fields of use:

text classification: as part of a sorting or classification system. emails sent to a central address could be scanned and evaluated to forward them to the recipient in charge. Organization or companies with a continuous inflow of large quantities of messages (insurance companies, banks, etc.) benefit from an automated sorting mechanism.

image recognition/ face detection: larger image archives or publishing houses who use a digital asset management (DAM) still need to add keywords or categorize them manually. With methods of machine learning content of pictures can be recognized automatically. This enriches the search itself making images searchable.

conceptual search: this approach uses huge piles of data which need to be put into relation. Patent and contract management are key industries for this idea. Here, new documents need to be evaluated against already existing ones. to do this, the recognition of content is key whereas the cognitive search has a clear advantage.