This Article is Not…

It is…

to bring an awareness to the masses about corporate decisions to omit the advanced capabilities of cataloguing photos, object recognition, and advanced metadata tagging.

Backstory: The Asks / Needs

Every day my family takes tons of pictures, and the pictures are bulk loaded up to The Cloud using Cloud Storage Services, such as DropBox, OneDrive, Google Photos, or iCloud. A selected set of photos are uploaded to our favourite Social Networking platform (e.g. Facebook, Instagram, Snapchat, and/or Twitter).

Every so often, I will take pause, and create either a Photobook or print out pictures from the last several months. The kids may have a project for school to print out e.g. Family Portrait or just a picture of Mom and the kids. In order to find these photos, I have to manually go through our collection of photographs from our Cloud Storage Services, or identify the photos from our Social Network libraries.

Social Networking Platform Facebook

As far as I can remember the Social Networking platform Facebook has had the ability to tag faces in photos uploaded to the platform. There are restrictions, such as whom you can tag from the privacy side, but the capability still exists. The Facebook platform also automatically identifies faces within photos, i.e. places a box around faces in a photo to make the person tagging capability easier. So, in essence, there is an “intelligent capability” to identify faces in a photo. It seems like the Facebook platform allows you to see “Photos of You”, but what seems to be missing is to search for all photos of Fred Smith, a friend of yours, even if all his photos are public. By design, it sounds fit for the purpose of the networking platform.

Auto Curation

Automatically upload new images in bulk or one at a time to a Cloud Storage Service ( with or without Online Printing Capabilities, e.g. Photobooks) and an automated curation process begins.

The Auto Curation process scans photos for:

“Commonly Identifiable Objects”, such as #Car, #Clock, #Fireworks, and #People

Auto Curation of new photos, based on previously tagged objects and faces in newly uploaded photos will be automatically tagged.

Once auto curation runs several times, and people are manually #taged, the auto curation process will “Learn” faces. Any new auto curation process executed should be able to recognize tagged people in new pictures.

Manual Curation

After upload, and auto curation process, optionally, it’s time to manually tag people’s faces, and any ‘objects’ which you would like to track, e.g. Car aficionado, #tag vehicle make/model with additional descriptive tags. Using the photo curator function on the Cloud Storage Service can tag any “objects” in the photo using Rectangle or Lasso Select.

Curation to Take Action

Once photo libraries are curated, the library owner(s) can:

Automatically build albums based one or more #tags

Smart Albums automatically update, e.g. after ingestion and Auto Curation. Albums are tag sensitive and update with new pics that contain certain people or objects. The user/ librarian may dictate logic for tags.

Where is this Functionality??

Why are may major companies not implementing facial (and object) recognition? Google and Microsoft seem to have the capability/size of the company to be able to produce the technology.

Is it possible Google and Microsoft are subject to more scrutiny than a Shutterfly? Do privacy concerns at the moment, leave others to become trailblazers in this area?

Protecting the Data Warehouse with Artificial Intelligence

Teleran is a middleware company who’s software monitors and governs OLAP activity between the Data Warehouse and Business Intelligence tools, like Business Objects and Cognos. Teleran’s suite of tools encompass a comprehensive analytical and monitoring solution called iSight. In addition, Teleran has a product that leverages artificial intelligence and machine learning to impose real-time query and data access controls. Architecture also allows for Teleran’s agent not to be on the same host as the database, for additional security and prevention of utilizing resources from the database host.

Data Access Policies

Connection Policies

Blocks connections to the database

White list or black list by

DB User Logins

OS User Logins

Applications (BI, Query Apps)

IP addresses

Rule Templates Contain Customizable Messages

Each of the “Policy Templates” has the ability to send the user querying the database a customized message based on the defined policy. The message back to the user from Teleran should be seamless to the application user’s experience.

iGuard Rules Messaging

Machine Learning: Curbing Inappropriate, or Long Running Queries

iGuard has the ability to analyze all of the historical SQL passed through to the Data Warehouse, and suggest new, customized policies to cancel queries with certain SQL characteristics. The Teleran administrator sets parameters such as rows or bytes returned, and then runs the induction process. New rules will be suggested which exceed these defined parameters. The induction engine is “smart” enough to look at the repository of queries holistically and not make determinations based on a single query.

The ultimate goal, in my mind, is to have the capability within a Search Engine to be able to upload an image, then the search engine analyzes the image, and finds comparable images within some degree of variation, as dictated in the search properties. The search engine may also derive metadata from the uploaded image such as attributes specific to the image object(s) types. For example, determine if a person [object] is “Joyful” or “Angry”.

As of the writing of this article, search engines Yahoo and Microsoft Bing do not have the capability to upload an image and perform image/pattern recognition, and return results. Behold, Google’s search engine has the ability to use some type of pattern matching, and find instances of your image across the world wide web. From the Google Search “home page”, select “Images”, or after a text search, select the “Images” menu item. From there, an additional icon appears, a camera with the hint text “Search by Image”. Select the Camera icon, and you are presented with options on how Google can acquire your image, e.g. upload, or an image URL.

Google Search Upload Images

Select the “Upload an Image” tab, choose a file, and upload. I used a fictional character, Max Headroom. The search results were very good (see below). I also attempted an uncommon shape, and it did not meet my expectations. The poor performance of matching this possibly “unique” shape is mostly likely due to how the Google Image Classifier Model was defined, and correlating training data that tested the classifier model. If the shape is “Unique” the Google Search Image Engine did it’s job.

Google Image Search Results – Max Headroom

Max Headroom Google Search Results

Google Image Search Results – Odd Shaped Metal Object

Google Search Results – Odd Shaped Metal Object

The Google Search Image Engine was able to “Classify” the image as “metal”, so that’s good. However I would have liked to see better matches under the “Visually Similar Image” section. Again, this is probably due to the image classification process, and potentially the diversity of image samples.

A Few Questions for Google

How often is the Classifier Modeling process executed (i.e. training the classifier), and the model tested? How are new images incorporated into the Classifier model? Are the user uploaded images now included in the Model (after model training is run again)? Is Google Search Image incorporating ALL Internet images into Classifier Model(s)? Is an alternate AI Image Recognition process used beyond Classifier Models?

Behind the Scenes

I’m not sure if the Cloud Vision API uses the same technology as Google’s Search Image Engine, but it’s worth noting. After reaching the Cloud Vision API starting page, go to the “Try the API” section, and upload your image. I tried a number of samples, including my odd shaped metal, and I uploaded the image. I think it performed fairly well on the “labels” (i.e. image attributes)

Odd Shaped Metal Sample Image

Using the Google Cloud Vision API, to determine if there were any WEB matches with my odd shaped metal object, the search came up with no results. In contrast, using Google’s Search Image Engine produced some “similar” web results.

Odd Shaped Metal Sample Image Web Results

Finally, I tested the Google Cloud Vision API with a self portrait image. THIS was so cool.

Google Vision API – Face Attributes

The API brought back several image attributes specific to “Faces”. It attempts to identify certain complex facial attributes, things like emotions, e.g. Joy, and Sorrow.

Google Vision API – Labels

The API brought back the “Standard” set of Labels which show how the Classifier identified this image as a “Person”, such as Forehead and Chin.

Google Vision API – Web

Finally, the Google Cloud Vision API brought back the Web references, things like it identified me as a Project Manager, and an obscure reference to Zurg in my Twitter Bio.

The Google Cloud Vision API, and their own baked in Google Search Image Engine are extremely enticing, but yet have a ways to go in terms of accuracy %. Of course, I tried using my face in the Google Search Image Engine, and looking at the “Visually Similar Images” didn’t retrieve any images of me, or even a distant cousin (maybe?)

Businesses already exist which have developed and sell Virtual Receptionist , that handle many caller needs (e.g. call routing).

However, AI Digital Assistants such as Alexa, Cortana, Google Now, and Siri have an opportunity to stretch their capabilities even further. Leveraging technologies such as Natural language processing (NLP) and Speech recognition (SR), as well as APIs into the Smartphone’s OS answer/calling capabilities, functionality can be expanded to include:

Call Screening – The digital assistant asks for the name of the caller, purpose of the call, and if the matter is “Urgent”

A generic “purpose” response, or a list of caller purpose items can be supplied to the caller, e.g. 1) Schedule an Appointment

The smartphone’s user would receive the caller’s name, and the purpose as a message back to the UI from the call, currently in a ‘hold’ state,

The smartphone user may decide to accept the call, or reject the call and send the caller to voice mail.

Call / Digital Assistant Capabilities

The digital assistant may schedule a ‘tentative’ appointment within the user’s calendar. The caller may ask to schedule a meeting, the digital assistant would access the user’s calendar to determine availability. If calendar indicates availability, a ‘tentative’ meeting will be entered. The smartphone user would have a list of tasks from the assistant, and one of the tasks is to ‘affirm’ availability of the meetings scheduled.

Allow recall of ‘generally available’ information. If a caller would like to know the address of the smartphone user’s office, the Digital Assistant may access a database of generally available information, and provide it. The Smartphone user may use applications like Google Keep, and any note tagged with a label “Open Access” may be accessible to any caller.

Join the smartphone user’s social network, such as LinkedIn. If the caller knows the phone number of the person, but is unable to find the user through the social network directory, an invite may be requested by the caller.

Custom business workflows may also be triggered through the smartphone, such as “Pay by Phone”.

Entrepreneurs / Science Guys (and Gals),

Are you ready for a challenge, and 150,000 USD to begin to pursue your challenge?

That’s just SBIR Phase I, Concept Development (~6 months). The second phase, Prototype Development, may be funded up to 1 MM USD, and last 24 months.

The Small Business Innovation Research (SBIR) program is a highly competitive program that encourages domestic small businesses to engage in Federal Research/Research and Development (R/R&D) that has the potential for commercialization. Through a competitive awards-based program, SBIR enables small businesses to explore their technological potential and provides the incentive to profit from its commercialization. By including qualified small businesses in the nation’s R&D arena, high-tech innovation is stimulated and the United States gains entrepreneurial spirit as it meets its specific research and development needs.

The program’s goals are four-fold:

Stimulate technological innovation.

Meet Federal research and development needs.

Foster and encourage participation in innovation and entrepreneurship by socially and economically disadvantaged persons.

Increase private-sector commercialization of innovations derived from Federal research and development funding.

It seems that car manufacturers, among others, are building autonomous hardware (i.e. vehicle and other sensors) as well as the software to govern their usage. Few companies are separating the hardware and software layers to explicitly carve out the autonomous software, for example.

Yes, there are benefits to tightly couple the autonomous hardware and software:

2. Synergies with two (or more) teams working in unison to implement functional goals. However, this may also be accomplished through two organizations with tightly coupled teams. Engaged, strong team leadership to help eliminate corp to corp BLOCKERS, must be in place to ensure deliverables.

There are also advantages with two separate organizations, one the software layer, and the other, the vehicle hardware implementation, i.e. sensors

2. The AI Software for Autonomous vehicles, if contractually allowed, may work with multiple brand vehicles, implementing similar capabilities. Vehicles now have capabilities / innovations shared across the car industry. The AI Software may even become a standard in implementing Autonomous vehicles across the industry.

3. Working with multiple hardware / vehicle manufactures may allow the enablement of Software APIs, layer of implementation abstraction. These APIs may enable similar approaches to implementation, and reduce redundancy and work can be used as ‘the gold standard’ in the industry.

4. We see commercial adoption of autonomous vehicle features such as “Auto Lane Change”, and “Automatic Emergency Braking.” so it makes sense to adopt standards through 3rd Party AI software Integrators / Vendors

5. Incorporating Checks and Balances to instill quality into the product and the process that governs it.

In summation, Car parts are typically not built in one geographic location, but through a global collaboration. Autonomous software for vehicles should be externalized in order to overcome unbiased safety and security requirements. A standards organization “with teeth” could orchestrate input from the industry, and collectively devise “best practices” for autonomous vehicles.

Do AI Rules Engines “deliberate” any differently between rules with moral weight over none at all. Rhetorical..?

The ethics that will explicitly and implicitly be built into implementations of autonomous vehicles involves a full stack of technology, and “business” input. In addition, implementations may vary between manufacturers and countries.

In the world of Kosher Certification, there are several authorities that provide oversight into the process of food preparation and delivery. These authorities have their own seal of approval. In lieu of Kosher authorities, who will be playing the morality, seal of approval, role? Vehicle Insurance companies? Car insurance will be rewritten when it comes to autonomous cars. Some cars may have a higher deductible or the cost of the policy may rise based upon the autonomous implementation.

Conditions Under Consideration:

1. If the autonomous vehicle is in a position of saving a single life in the vehicle, and killing one or more people outside the vehicle, what will the autonomous vehicle do?

1.1 What happens if the passenger in the autonomous vehicle is a child/minor. Does the rule execution change?

1.2 what if the outside party is a procession, a condensed population of people. Will the decision change?

As a business, Digital Agent subscriber, Microsoft Bing search results will contain the business’ AI Digital Assistant created using Visio. The ‘Chat’ link will invoke the business’ custom Digital Agent. The Agent has the ability to answer business questions, or lead the user through “complex”, workflows. For example, the user may ask if a particular store has an item in stock, and then place the order from the search results, with a ‘small’ transaction fee to the business. The Digital Assistant may be hosted with MSFT / Bing or an external server. Applying the Digital Assistant to search results pushes the transaction to the surface of the stack.

Bing Digital Chat Agent

Leveraging their existing technologies, Microsoft will leap into the custom AI digital assistant business using Visio to design business process workflows, and Bing for promotion placement, and visibility. Microsoft can charge the business for the Digital Agent implementation and/or usage licensing.

The SDK for Visio that empowers the business user to build business process workflows with ease may have a low to no cost monthly licensing as a part of MSFT’s cloud pricing model.

Microsoft may charge the business a “per chat interaction” fee model, either per chat, or bundles with discounts based on volume.

In addition, any revenue generated from the AI Digital Assistant, may be subject to transactional fees by Microsoft.

Why not use Microsoft’s Cortana, or Google’s AI Assistant? Using a ‘white label’ version of an AI Assistant enables the user to interact with an agent of the search listed business, and that agent has business specific knowledge. The ‘white label’ AI digital agent is also empowered to perform any automation processes integrated into the user defined, business workflows. Examples include:

basic knowledge such as store hours of operation

more complex assistance, such as walking a [perspective] client through a process such as “How to Sweat Copper Pipes”. Many “how to” articles and videos do exist on the Internet already through blogs or youtube. The AI digital assistant “curator of knowledge” may ‘recommended’ existing content, or provide their own content.

Proprietary information can be disclosed in a narrative using the AI digital agent, e.g. My order number is 123456B. What is the status of my order?

Actions, such as employee referrals, e.g. I spoke with Kate Smith in the store, and she was a huge help finding what I needed. I would like to recommend her. E.g.2. I would like to re-order my ‘favorite’ shampoo with my details on file. Frequent patrons may reorder a ‘named’ shopping cart.

Escalation to a human agent is also a feature. When the business process workflow dictates, the user may escalate to a human in ‘real-time’, e.g. to a person’s smartphone.

Note: As of yet, Microsoft representatives have made no comment relating to this article.

Challenges of the Digital Agent

Just like with humans trying to interpret human utterances, many possibilities exist, and misinterpretation occurs.

Meeting someone for the first time, without historical context places additional burden on the interpreter of the intent.

There are innumerable opportunities to ask the same question, to request information, all achieving a similar, or the same goal.

Opportunities for Goal / Intent Accuracy

Business Process Workflows may enable a very broad ‘category’ of subject matter to be disambiguated as the user traverses the workflow. The intended goal may be derived from asking ‘narrowing’ questions, until the ‘goal’ is reached, or the user ‘falls out’ of the workflow.

Methodologies such as leveraging Regex to interpret utterances are difficult to create and maintain.

Utterances are still a necessity, their structure, and correlation to Business Process Workflows. However, as the knowledge base grows, so does the complexity of curation of the content. A librarian, or Content Curator may be required to integrate new information, deprecate stale content, and update workflows.

Ongoing, Partnership between Digital Agent and Human

Business Process Workflows may be initially designed and implemented by Subject Matter Experts (SMEs). However, the SMEs might not have predicted all possible valid variations of the workflow, and achieve a different outcome for the triggered goal.

As the user traverses a workflow, they may encounter a limiting boundary, such as a Boolean question, which should have more than two options. Some digital assistants may enable a user to walk on an alternate path by leveraging ‘human assisted’ goal achievement, such as escalation of a chat. The ‘human assisted’ path may now have a third option, and this new option may be added to the Business Process Workflow for future use.

Microsoft Outlook has had an AI Email Rules Engine for years and years. From using a simple Wizard to an advanced construction rules user interface. Oh the things you can do. Based on a wide away of ‘out of the box’ identifiers to highly customizable conditions, MS Outlook may take action on the client side of the email transaction or on the server side. What types of actions? All kinds of transactions ranging from ‘out of the box’ to a high degree of customization. And yes, Outlook (in conjunction with MS Exchange) may be identified as a digital asset management (DAM) tool.

Email comes into an inbox, based on “from”, “subject”, contents of email, and a long list of attributes, MS Outlook [optionally with MS Exchange], for example, may push the Email and any attached content, to a server folder, perhaps to Amazon AWS S3, or as simple as an MS Exchange folder.

Then, optionally a ‘backend’ workflow may be triggered, for example, with the use of Microsoft Flow. Where you go from there has almost infinite potential.

Analogously, Google Gmail’s new Inbox UI uses categorization based on ‘some set’ of rules is not something new to the industry, but now Google has the ability. For example, “Group By” through Google’s new Inbox, could be a huge timesaver. Enabling the user to perform actions across predefined email categories, such as delete all “promotional” emails, could be extremely successful. However, I’ve not yet seen the AI rules that identify particular emails as “promotional” verses “financial”. Google is implying these ‘out of the box’ email categories, and the way users interact, take action, are extremely similar per category.

Google may continue to follow in the footsteps of Microsoft, possibly adding the initiation of workflows based on predetermined criteria. Maybe Google will expose its AI (Email) Rules Engine for users to customize their workflows, just as Microsoft did so many years ago.

Although Microsoft’s Outlook (and Exchange) may have been seen as a Digital Asset Management (DAM) tool in the past, the user’s email Inbox folder size could have been identified as one of the few sole inhibitors. Workaround, of course, using service accounts with vastly higher folder quota / size.