An internal concept video emerged today that imagines Google and technology taking more of an active role in your life in order to shape and — possibly — improve it.

Nintendo Switch

Published by The Verge, this internal video was created by the head of design at X, Alphabet’s moonshot division responsible for Google Brain, Glass, and Waymo. Nick Foster created it as part of what the company calls a “thought-experiment” that involves “speculative design” to “explore uncomfortable ideas and concepts in order to provoke discussion and debate.”

With that in mind, the core concept in the video is a “constantly evolving representation of who we are” called a “ledger.” It includes information collected from our devices and could be analyzed to determine user “actions, decisions, preferences, movements, and relationships.”

Google imagines a future where this corpus of data is more than historical reference, but “goal-oriented” and given a “volitional purpose.” Rather than being passive, it could work to help users meet pre-defined goals like staying healthy or reducing one’s environmental impact.

This video from 2016 created a mockup of an app called “Resolutions from Google” where goals include “Eat more healthy,” “Protect the environment,” and “Support local business.” After granting Google permission to access data, the service would hypothetically offer suggestions within third-party apps as visual overlays.

“suggestions may be converted not by the user, but by the ledger itself”

For example, it could recommend in Uber that you select the Uber Pool option or when buying groceries choosing locally grown produce. The system works to move users closer to the goal, modifying suggestions along the way. At this initial stage of this concept, the user selects the goal.

However, Google imagines the next step where it — controversially — is the one to offer goals and suggestions for how user’s should live their lives. In the example given, the ledger notices that it is missing a user’s weight for creating a recovery plan.

It would then work to find the user a scale to record and upload this data. If it does not find one, it could create a “bespoke” one through 3D printing that matches a user’s “historical data, taste, and aesthetic sensibility” so they’d be more likely to use it.

This frightening step enforces how Google and the ledger would be actively working to modify user behavior by trying to find a way that usage of the scale sticks.

However, the next step is even more ambitious, or frightening, where Google would compare past ledgers to “emerging” ones from new users to better guide future goals. The end result is enough data “to make increasingly accurate predictions about decisions and future behaviors.”

Google speculates that this mass collection of data could “develop a species level understanding of complex issues such as depression, health, and poverty.”

Even the ambitious video acknowledges that this technology and society is at “the very beginning,” but ends noting how it could benefit future generations and the human species as a whole.

Even calling this video “early days” is an understatement. It’s a concept, and according to Google, it’s noting more than that. However, at the end of the day, it is a natural extension of what Google wants to do with things like Assistant to plan your day or Maps to suggests places you might like and how you could get there with the help of augmented reality.

I think most people will immediately disagree with the idea of the user losing control to determine or approve the goals suggested by a computer and artificial intelligence. Today’s assistants are inherently built with the user having the final say given the assistive nature.

As a society, we are only beginning to accept technology’s suggestions. For example, Waze might recommend we deviate from a known route for a new one that might be faster. With the new Google Maps features announced at I/O, users will begin seeing how well an AI could really discern preferences and offer similar suggestions.

However, at the end of the day, these recommendations are just suggestions that can be easily ignored. A system, like the concept proposed, where we have no direct control or apparent oversight is terrifying.