Mercedes-Benz Intelligent Interior

Tools: C++, Qt, QML, JavaScript, OpenGL, openFrameworks, machine-learning

Officially released as MBUX Interior Assistant in 2019, it is a set of experiences enabled by an interior-mounted camera that take a step towards an intelligent interior. As opposed to forcing the user to learn yet another set of commands and behaviors, the vehicle anticipates and understands natural user’s movements and accomplishes the actions for them.

I was leading the concept development with a small team from start to finish. We did the ux research, idea generation, prototyping them, testing the technology and communicating within the company to get the project funded.

Design goals

focus on known paint points (e.g. too many buttons, too many steps in the UI)
design a solution that requires zero or minimal learning
design interactions that don’t require visual attention and can be imprecise
design a solution that is technically feasible given the constraints
design an experience that is a bit magical and delightful… a responsive intelligent interior

Prototyping

To test these experiences, I developed a custom machine learning-based hand tracking algorithm using a depth camera in a real vehicle. This allowed us to quickly build functional interaction prototypes which were instrumental in refining the user experience through user studies, as well as for convincing the organization to fund the project and make it a reality.

Product

Ultimately, a small subset of all the prototyped ideas ended up in the first product as the 2019 Mercedes CLA MBUX Interior Assistant.

Turn on the reading lights by casually tapping them

Automatically illuminate unoccupied passenger seat

The vehicle understands who is interacting with the touch-screen or a touchpad and customizes the interface to them and to the input device being used. For example, pre-selecting the climate or seat controls, adapting the keyboard for text input based on device, allowing text input for passenger while disallowing for the driver etc.

Adapting the UI to the input device in use.

A shortcut to a favorite function.

Awards

Car HMI 2019 - Most Innovative HMI Feature
AutoSens 2019 - Most Innovative In-Cabin Application