Hello, Bentley: Using Sensory speech technology to create a natural user experience

By Bernard Brafman, Sensory Inc., and Justin Moon, QNX Software Systems

In-vehicle infotainment systems are becoming more and more complex, and integral to the overall driving experience. As this trend continues, it will become increasingly important to create systems that support multiple forms of user interaction. If you’re driving, the last thing you want to do is enter a destination manually, or search for your favorite artist in a playlist by using a touch screen, jog wheel, or other manual input method. Drivers want and require a user experience that is both simple and natural; integration of speech recognition technology goes a long way toward achieving that goal.

In fact, speech recognition is a key component of the latest QNX technology concept car, a modified Bentley Continental GT. The speech rec system lets you plot a route or select your favorite artist using natural speech, but it goes even further by letting you simply ask the car to perform an action. Leveraging Sensory’s FluentSoft SDK, more specifically the TrulyHandsfree^TM Voice Control technology, the QNX concept development team implemented keyword spotting techniques to interact with the vehicle.

So how does this work? Well, let’s say you’re in Vegas and need directions to the Wynn Casino. To engage the cloud-based Watson speech system, you simply say “Hello Bentley” — no need to push a button. You then complete the request by saying “Take me to the Wynn Casino.” FluentSoft, along with the architecture of the advanced speech recognition system included in the QNX CAR platform, allowed the team to create this seamless, easily implemented, and well-executed voice interaction experience.


When you say “Hello Bentley,” the QNX concept car displays a visual prompt at the top of the screen, indicating that the speech rec system is now listening for natural speech or directed commands.

Multiple triggers
The team plans to further utilize Sensory technology in future concept car releases. The current implementation has the single “Hello Bentley” trigger, which engages the speech system. But TrulyHandsfree Voice Control supports multiple active triggers as well as a robust recognition vocabulary to create a rich command-and-control user experience that doesn’t require prompts or pauses. Thus, it’s possible to create a hybrid system that is seamless and transparent to the user. For instance, “Hello Bentley air 68 degrees” or “Hello Bentley what time is it in Tokyo?” can both be executed flawlessly, regardless of which speech rec system is engaged.

A matter of choice
For an even more personalized experience, this technology can allow drivers to create their own custom trigger with a simple one-time enrollment process that verifies their identity as a voice password or identifies one of several previously enrolled. This creates a custom experience not only by letting you choose your own trigger phrases (come on now, who hasn’t named their car at some point?), but also by recalling individual preferences such as seat position, steering wheel position, and multimedia presets.

Look for these enhanced features in concept cars to come!

---

Bernard Brafman is vice president of business development for Sensory, Inc., responsible for strategic business partnerships. He received his MSEE from Stanford University. Contact Bernard at bbrafman@sensoryinc.com

Justin Moon is a global technical evangelist for the automotive business development team at QNX Software Systems.

It's a Bentley! A guided tour of the new QNX technology concept car

"Bend it, shape it, any way you want it"
— Headline from a QNX advertisement, circa 1987

I’m about to show you some pictures of a car. Not just any car, but a powerful, luxurious, and stunningly beautiful car. A car that has undergone a technological transformation.

If you’re like me, you'll be fascinated by the car’s features, some of which have never been seen in a vehicle — until now. But if you can, remember that it isn’t just about the cool features. It’s also about the platform that enabled them.

I’m speaking, of course, of the QNX CAR application platform.

We created the new QNX technology concept car — a modified Bentley Continental GT — to demonstrate that flexibility and customization form the very DNA of the QNX CAR platform. If you’ve seen the QNX reference vehicle, you already know that the platform provides an extremely rich environment for in-car infotainment, complete with HMI frameworks, smartphone integration, an HTML5 engine, a mobile device gateway, and a host of pre-integrated partner technologies — everything to kickstart our customers' projects. But in the automotive world, differentiation is everything. So it’s just as important that the platform enables customers to add their own branding, features, and sizzle. And to do it quickly.

Ease of branding and
personalization is just one

of the capabilities of the
QNX CAR platform.

Which is where the new concept car comes in. To create it, we used the same base QNX CAR platform that we offer our customers. But when you compare the Bentley to the Jeep, which uses a stock version of QNX CAR, the differences are dramatic: different features, different branding, and a different look-and-feel. In effect, the Jeep shows what QNX CAR can do out of the box, while the Bentley shows what QNX CAR lets you do once you start bending it to your imagination. One platform, many possibilities.

Which brings me to the slogan at the top of this post. It's amazing to think that a core value of QNX technology in the 1980s — giving customers the flexibility to achieve what they want to do — remains just as true today. Some values, it seems, are worth keeping.

And now, the car…
I know that you’re anxious to peek inside the car and see what we’ve done. But before we go any further, take a moment to savor the car’s beautifully sculpted exterior. This is one classy set of wheels. In fact, if you ask me, the wheels alone are worth the price of admission:



The awesome (and full HD) center stack
Okay, time to hop in — but get ready to prop up your jaw. Because the first thing you’ll notice is the jaw-droppingly beautiful center stack. This immense touchscreen features a gracefully curved surface, full HD graphics, and TI’s optical touch input technology, which allows a physical control knob to be mounted directly on the screen — a feature that’s cool and useful. (In the photo below, the clock display appears within the knob.)

The center stack supports a host of applications, including a 3D navigation system from Elektrobit that makes full use of the display. Just check out this bird’s-eye view of the Las Vegas Strip:



So how big is the display? Big enough to provide access to other functions, such as the car’s media player or virtual mechanic, and still have plenty room for navigation. Check it out:



The awesome (and very polite) voice rec system
Time to talk to the car. Just say “Hello Bentley,” and the car’s voice recognition system immediately comes to life and begins to interact with you — in a British accent, no less. You can now tell the media player what you’d like to hear and the navigation system where you’d like to go.

To provide natural language speech recognition, the system uses the cloud-based AT&T Watson speech engine, as well as an “intent framework” from QNX. It also uses keyword spotting technology from Sensory so you can start the system simply by talking to it.

The awesome (and nicely integrated) smartphone support
The Bentley also showcases how the QNX CAR platform enables automakers to offer advanced integration with popular smartphones. For instance, the car can communicate with a smartphone to stream music, or to provide notifications of incoming email, news feeds, and other real-time information — all displayed in a manner appropriate to the automotive context. Here's an example:



The awesome (and just plain fun) web app
I know, I know: the car looks cool, but you’re not at CES this week to see it first-hand. But how about the next best thing? Just connect to the web app and keep tabs on the Bentley in real time. (Note: The car will go online later this morning.) The app lets you view a variety of data that the car publishes to the cloud, such as what song the infotainment system is playing and whether someone has just opened a door. It also displays information that would be extremely helpful if this were your personal car, such as fluid levels and tire pressure. (This is a preliminary screen for the app, so I'm not sure if the tire pressures are realistic.)



UPDATE: The web app is now live, and the desktop version features a live camera feed of the Bentley and Jeep. Check it out!



The awesome (and very configurable) digital instrument cluster
The instrument cluster is implemented entirely in software, though you would hardly know it — the virtual gauges are impressively realistic. But more impressive still is the cluster’s ability to morph itself on the fly. Put the car in Drive, and the cluster will display a tach, gas gauge, temperature gauge, and turn-by-turn directions — the cluster pulls these directions from the center stack’s navigation system (cool, that). Put the car in Reverse, and the cluster will display a video feed from the car’s backup camera.



There are other options as well. For instance, the cluster can display information from the media player or display the current weather:



The awesome everything else
I’ve only scratched the surface of what the car can do. For instance, it also provides:

• Advanced multimedia system — Offers direct support for Pandora radio and the first embedded in-car implementation of the Shazam music discovery service.
   
• Video conferencing with realistic telepresence — Separate cameras for the driver and passenger provide independent video streams, while high-definition voice technology from QNX offers expanded bandwidth for greater realism, as well as stereo telepresence for making the remote caller sound as if they’re sitting right next to you.
   
• LTE connectivity — The car features an LTE radio modem, as well as a Wi-Fi hotspot for devices you bring into the car.

Super size those images
Want to see the center stack and instrument cluster in all their high-resolution glory? Just check out our QNX Flickr account.

That's all for now, but stay tuned: We’ll have plenty more news for you today and all through this week.