Imbellus Assessments: Out of the Gate with $23M and McKinsey & Co. as Client

January 14, 2019

On January 11, 2019, Mark Bauerlein of the James B. Martin Center posted a piece entitled, “Be Wary of This Test,” about a testing startup, Imbellus, which has an impressive website deficient in any substance. Even so, Imbellus already has $23M in venture capital funding behind it.

What Kantar purports to do is create a cognitive assessment to rival– and apparently replace– the likes of the SAT and ACT. From Kantar’s Imbellus site:

Imbellus is reinventing how we measure human potential.

We build simulation-based assessments to evaluate how people think, not just what they know.

For nearly 100 years, college admissions tests have shaped our entire education system.

By rewriting this one line of code, we change the way the whole system operates.

Of course, Kantar cannot “rewrite the code” for ditching college admissions testing altogether; such rewriting would kill the market for her proposed product. No, she must keep the 100-years-of-college-admissions-testing market (for she hopes to make it her market) and simply offer a new and improved product.

But that product has yet to appear, even in fledgling form, for public inspection on the Imbellus site. Nevertheless, Imbellus does offer a two-minute video featuring Sonia, Learning Scientist and Designer, who discusses designing an assessment for a “first client.” But forget interacting with any prototype.

It seems that Imbellus the Reinvented is indeed in the works– incubated elsewhere.

Imbellus has been working with management consulting firm McKinsey & Co. since last year on a pilot of digital, scenario-based assessments as part of the recruiting and hiring process.

“Imbellus’ technology is helping us to expand the principles of our case study interviews to a much wider range of talent, providing engaging experiences that expose them to the kinds of problem-solving we do, while giving us accurate and detailed information on how they think about problems,” said Keith McNulty, McKinsey’s director of digital and people analytics.

If one is seeking info on specific Imbellus employees via the Imbellus site, there is no substantive word yet on that front. On the Imbellus page, “our work,” one is invited to “explore the design and engineering behind Imbellus Assessments and meet the minds leading our work,” except there are no “leading minds” listed. Not one. Not even Sonia (whose last name and credentials remain a mystery).

Not even Kantar herself.

But there is this five-page, 2018 educationaldatamining.org paper on Imbellus’ “partnership with McKinsey and Company… to build a simulation-based assessment that evaluates incoming applicants’ cognitive skills and abilities.” That paper lists the following five individuals as authors (Linkedin links added):

Our firm has deep experience using in-person, scenario-based assessments (such as case interviews) in recruiting. Yet we miss more nuanced perspectives on the skills of some candidates and miss out on other candidates completely because we can’t meet everyone in person. McKinsey began investigating this problem in collaboration with a start-up called Imbellus, which has a team of data scientists, engineers, and psychometricians who aspire to replace traditional standardized tests and are committed, above all, to science. We also enlisted the help of UCLA’s National Center for Research on Evaluation, Standards, and Student Testing (CRESST) to help identify and measure the skills we need.

The result of this collaboration is a new scenario-based assessment designed to help our firm attract new and different talent profiles from all parts of the world. It not only evaluates candidates in a lower-stress, more engaging environment than a traditional test does but levels the playing field by minimizing the influence of a candidate’s background.

Imbellus’ “deep experience” seems to be this sole McKinsey and Co. project, in which 20-year McKinsey and Co. veteran, Keith McNulty, serves as second author.

According to McKinsey and Co., Imbellus already has its “team” in place. However, the lack of any specifics regarding the “leading minds” on the Imbellus web site is strange. Furthermore, if any readers are into game and software design, there’s room for you, according to Imbellus’ “career” page:

Current Openings

Platform Engineer (Full-Stack)

Associate Game Designer

Senior Software Engineer (Game)

Principal Game Engineer

Senior Learning Scientist

Fullstack Software Engineer (Game)

Strategic Finance Manager

Learning Science Researcher

3D Artist

From the looks of Imbellus’ employment opportunities and McKinsey and Co.’s plug, Imbellus assessment will involve testing the thinking one uses while playing games. An excerpt from the 2018 McKinsey paper indicates as much and includes a word on Imbellus’ assessment aspirations:

We designed each scenario in the assessment based on a set of problem-solving constructs and workplace activities wrapped in a natural world setting. For example, in one scenario, users may be researching and evaluating an infected species in desert terrain. As users play through a scenario, we test them on both their cognitive process and product by capturing their telemetry data. These
hovers and clicks are captured as evidence to make inferences about their cognitive processing. …

Conclusions and Future Work

Results from the pilot are promising and show that the Imbellus scores can be used to build out predictive cognitive profiles of candidates. Indeed, these results showed that the cognitive profiles of users were predictive of their success through the McKinsey & Company hiring pipeline. Beyond predictability, these results also
show that cognitive processing skills can be captured and quantified using telemetry data within a complex problem-solving task.

To examine the generalizability of these results, we are currently conducting playtests with McKinsey & Company employees and candidates, globally. This extra testing will be used to help us iterate on the design of the assessment and refine our Imbellus scores. In the fall of 2018, we will run a large-scale field test with
an expected sample size of over 1000 of McKinsey & Company candidates.

The current version of the simulation is deployed in a secure, proctored environment. In the future, our assessments will be deployed remotely. As such, our assessment will aim to account for performance effects across demographic factors. At its core, Imbellus will leverage a data-driven, artificial intelligence (AI) architecture to prevent cheating. Every user who takes the Imbellus assessment will receive a unique task instance that, on the surface, is varied by its individual properties, complexity, and visual design, while structurally every task version remains consistent in its assessment. Through this approach, Imbellus assessments will prove robust against cheating, hacking, and gaming challenges that face many existing intelligence tests. Our assessments are designed for scale, enabling our team to reach a variety of domains and populations.

Looking beyond this work, we are exploring capabilities beyond problem-solving, including affective skills that are essential for success in the 21st Century workforce. At Imbellus, we aim to provide insightful data points on incoming applicants and current employees that will help companies build successful and sustainable teams in the future.

Ambitious– and as of yet, decidedly fledgling. Even so– despite an amazing lack of information on its web site– and despite evidence that Imbellus’ only client is McKinsey and Co. (not even mentioned on the Imbellus web site)– and with a McKinsey and Co. employee serving as second author on this single Imbellus study– despite all of this– according to Imbellus– there is an Imbellus waiting list.

Imbellus is eager to work with firms and schools whose employees or students use a variety of skills and abilities relevant to the future of work. We are inundated with interest in our approach to assessment. Please feel free to email us with interest at assessments@imbellus.com.

Okay. This Imbellus site offers no credentials for a single individual, not even creator Kanter. Interested parties can view no prototype, and though Sonia speaks of a “first client,” and though that client seems to be McKinsey and Co., specifics of that connection are not disclosed on the Imbellus web site.

But Imbellus does have money– $23M– and it seems that McKinsey and Co. is the chief org tapping into that cash for the moment as the Imbellus incubator.

Even as I was writing this post, January 14, 2019, PR Newswireindicated that “seasoned testing executive” and “veteran of College Board and AIR,” Jack Buckley, was just hired as Imbellus’ president and chief scientist:

“Jack has an extensive track record of leading interdisciplinary psychometrics, learning science, data science, and engineering teams to help the world measure student achievement and growth,” said Rebecca Kantar, Founder and CEO of Imbellus. “His move demonstrates a commitment to the next generation of high-stakes assessments, and we are thrilled to welcome him as a champion of our novel approach.”

In his role as chief scientist at Imbellus, Dr. Buckley will lead the continued development and deployment of these next-generation tools and build partnerships that will facilitate their application for college admissions, workforce, and lifelong learning. …

Dr. Buckley has held numerous positions in government, academia, and the private sector. In addition to his work at AIR, College Board, and the U.S. Department of Education, he has served as a tenured associate professor of applied statistics at New York University and an assistant professor at Boston College. He began his career serving in the United StatesNavy as a surface warfare officer and nuclear reactor engineer. He holds degrees from Harvard and the State University of New York at Stony Brook.

This is an interesting development: With Imbellus, Kantar states her ambitions in replacing the likes of the SAT, which is produced by College Board. And yet, the freshly-hired president and chief researcher of Imbellus is a former College Board executive.

Hm.

We’ll see where this goes.

According to Kantar’s 2017 Upfront Ventures video (see below– Upfront Ventures is one of Imbellus’ funders), Kantar’s plan appears to be to reproduce the outcomes of Imbellus’ work with McKinsey and Co. 300,000 times and then approach Harvard University (the final word in assessment according to Kantar) to talk Harvard into altering its assessment preference away from tests like ACT and SAT and toward Imbellus-demonstrated cognitive assessment.

This kind of psychometric research takes time. Academics are willing to devote such time to establishing an assessment, but the business world wants to turn a profit. And Imbellus is a business.

Like this:

Related

Richard Phelps-
Your investigative work exposing Fordham and College Board kicks a_s. Thanks for letting the public know what’s happening behind the scenes of the worst non-profits. Have you considered sending an article to the Stanford Social Innovation Review? That group posts a slew of articles that put lipstick on the pig of philanthropy. It would great to see an article published at the Review that tells the truth.

HMMMM…and funny that it has been reported that David Coleman has stepped back from the President position at the College Board (he still retains his position as CEO). Could it be that since the SAT and AP has taken such a big hit lately that a Zombie SAT and AP has to rise from the dead to start feeding off the flesh of our youth?

So…the problem Kantar maintains is that students are learning too much content? Where have we heard this before? Maybe in the Common Core’s misguided premise that generic skills and standards can replace substantive knowledge of content. Try demonstrating insight and understanding of anything without sufficient mastery of content and subject matter. The two go hand in hand.