Hello, Bentley: Using Sensory speech technology to create a natural user experience

By Bernard Brafman, Sensory Inc., and Justin Moon, QNX Software Systems

In-vehicle infotainment systems are becoming more and more complex, and integral to the overall driving experience. As this trend continues, it will become increasingly important to create systems that support multiple forms of user interaction. If you’re driving, the last thing you want to do is enter a destination manually, or search for your favorite artist in a playlist by using a touch screen, jog wheel, or other manual input method. Drivers want and require a user experience that is both simple and natural; integration of speech recognition technology goes a long way toward achieving that goal.

In fact, speech recognition is a key component of the latest QNX technology concept car, a modified Bentley Continental GT. The speech rec system lets you plot a route or select your favorite artist using natural speech, but it goes even further by letting you simply ask the car to perform an action. Leveraging Sensory’s FluentSoft SDK, more specifically the TrulyHandsfree^TM Voice Control technology, the QNX concept development team implemented keyword spotting techniques to interact with the vehicle.

So how does this work? Well, let’s say you’re in Vegas and need directions to the Wynn Casino. To engage the cloud-based Watson speech system, you simply say “Hello Bentley” — no need to push a button. You then complete the request by saying “Take me to the Wynn Casino.” FluentSoft, along with the architecture of the advanced speech recognition system included in the QNX CAR platform, allowed the team to create this seamless, easily implemented, and well-executed voice interaction experience.


When you say “Hello Bentley,” the QNX concept car displays a visual prompt at the top of the screen, indicating that the speech rec system is now listening for natural speech or directed commands.

Multiple triggers
The team plans to further utilize Sensory technology in future concept car releases. The current implementation has the single “Hello Bentley” trigger, which engages the speech system. But TrulyHandsfree Voice Control supports multiple active triggers as well as a robust recognition vocabulary to create a rich command-and-control user experience that doesn’t require prompts or pauses. Thus, it’s possible to create a hybrid system that is seamless and transparent to the user. For instance, “Hello Bentley air 68 degrees” or “Hello Bentley what time is it in Tokyo?” can both be executed flawlessly, regardless of which speech rec system is engaged.

A matter of choice
For an even more personalized experience, this technology can allow drivers to create their own custom trigger with a simple one-time enrollment process that verifies their identity as a voice password or identifies one of several previously enrolled. This creates a custom experience not only by letting you choose your own trigger phrases (come on now, who hasn’t named their car at some point?), but also by recalling individual preferences such as seat position, steering wheel position, and multimedia presets.

Look for these enhanced features in concept cars to come!

---

Bernard Brafman is vice president of business development for Sensory, Inc., responsible for strategic business partnerships. He received his MSEE from Stanford University. Contact Bernard at bbrafman@sensoryinc.com

Justin Moon is a global technical evangelist for the automotive business development team at QNX Software Systems.

Our best CES yet

Anecdotes and observations from the QNX booth at 2013 CES

As a wrap-up to last week’s Consumer Electronics Show, I would love to regale you with all the cool technologies and nifty gadgets that I saw. But over the course of the entire four days, I rarely left the 20’x40’ patch of white carpet that was the QNX booth — with brief exceptions, of course, for bodily maintenance. The booth was just too busy for me to get away. If you checked out the QNX booth webcam, you know what I'm talking about.

Paul Leroux and Nancy Young have already posted a lot of information and photos about the show and the new QNX concept car, which is based on a Bentley Continental GT. So let me provide my personal view of CES through assorted anecdotes or observations collected at the booth.

• As you’d expect, the Bentley got a lot of attention. But our reference vehicle, based on a Jeep Wrangler, got more attention than I thought it would, even though this is the third time we’ve shown it in public. Many of the people interested in the Jeep just wanted to see what our QNX CAR application platform looked like “out of the box” without customization. And some were confessed Jeep or truck aficionados, without the “luxury brand lust” experienced by most.
   
• People in the auto industry knew who we were without introduction. Non-automotive people didn’t know who we were until I mentioned that “we are a wholly owned subsidiary of Research In Motion,” at which point most of them said “Oh, you’re that QNX.” Seems that your average person has heard quite a bit about QNX in the context of BlackBerry, but has no idea that the same company is doing things in automotive — or in anything else, for that matter. I usually then spoke about our 30+ year legacy in life- and mission-critical systems. When people learned that an OS used for mission-critical systems will also power their next potential phone, their reaction was “wow—that’s really cool.”
   
• Tanner Foust is a really nice young kid. (Actually, he’s not that much younger than me, but he sure looks young!) I didn’t know who he was when he was being filmed in the booth, surrounded by a throng of admirers. But since then, I’ve watched a lot of his YouTube videos and boy, can he drive! He's an accomplished race car driver, TV personality, and stuntman for lots of famous movies, but it’s nice to see he hasn’t let it go to his head.
   
• We wanted to make sure that our concept car respected the Bentley brand. To do that, we ran our design sketches by the folks at Bentley and they occasionally suggested some tweaks. It was all our own work, however, and the Bentley folks never saw it before it hit the show floor. When they came to the booth, they were very happy with what they saw — enough so that they said “it looked like we did it.” That, to me, was the ultimate compliment.
   
• Most frequent question: “Are you giving this away?” As it turns out, it’s something that people have said for every concept car we’ve done to date. Second most frequent question: “Can I drive it?” Unfortunately, but unsurprisingly, the answer to both is “No.”
   
• I was a little surprised by the enthusiastic response to the car's video conferencing. Of course, it works only while the car is parked, and you only get audio while the car is in Drive. But the part that seemed to impress people the most is the audio: two channel stereo and a full 20Hz to 22KHz means that the call sounds so much better than your typical hands-free call. You could see the reaction when the our director of acoustics Phil Hetherington started talking — you don’t know what you’ve been missing until you hear it.
   
• Bentley wanted us to add our video conferencing solution to the technology concept car. Because many Bentley vehicle owners aren’t necessarily the drivers, this feature makes a whole lot more sense for rear-seat systems than you might initially imagine.
   
• I was really impressed by two members of the media: Brian Cooley of CNET and Craig Peterson of Clear Channel. Both could receive a five minute technology core dump, quickly digest it, and talk intelligently about it on video or live radio (respectively) with no stumbles, questions, or missteps. I’ve had the pleasure of seeing both in action before, but their consummate professionalism is really quite amazing.
   
• I and every other QNX’er was delighted that we didn’t win the CNET Best of CES award! Instead, our customer, Chevrolet, won it for their MyLink system, and we couldn’t have been happier. Two out of the three nominees were QNX-based systems (the Garmin K2 was the other), so our odds were good. I’d rather that we never win another Best of CES award if it meant that one of our customers could always win instead.
   
• A number of people asked about the RIM booth and its absence. I explained that RIM was focusing on their launch at the end of January, and that since they wouldn’t have a new product to show the public, it didn’t make sense to be there. (It’s notable that Microsoft wasn’t there either, and Apple never is.) RIM was in Las Vegas in a hotel outside the convention center, giving media private previews of the upcoming phones that seemed to be extremely well received. And we had a few of our RIM compatriots helping us out at the QNX booth as well.
   

That’s all I’ve got to say about CES 2013 — our best show yet. See you next year!

Okay, time to get technical

Have glossy photos of the QNX concept car left you hungry for more? Dig into a technical whitepaper with our friends from Texas Instruments.

By now, many of you have seen photos and videos of the new QNX technology concept car, a specially modded Bentley Continental GT. Now, I'd like to say that the car was completed in record time by a small team of highly creative QNX engineers. And in many ways, that's absolutely true. But it's just as true that the work started more than 10 years ago, when QNX Software Systems started to build deep partnerships with leading players in the auto industry.

Because the truth is, you don't create this kind of magic overnight. And you don't do it on your lonesome. QNX has become successful in automotive for many reasons, but one of the most important is our ability to work closely, and productively, with A-list partners like Texas Instruments.

Inside the concept car

Take a look at the amazing displays in the Bentley, and the speed at which the screens redraw, and you get a taste just for how well QNX software and TI silicon work together under the covers.

Which brings me to a new white paper co-authored by Andy Gryc of QNX, and Matt Watson and Scott Linke of TI. It's titled "In-Vehicle Connectivity is So Retro," and among other things, it tells the story of how technologies from QNX and TI have co-evolved to help automotive developers build high-performance systems in less time and at less cost.

If your working vocabulary includes terms like OMAP 5, 1080p video decode/encode, floating-point DSP, MOST MLB, Ethernet AVB, PCIe, SATA, WiLink, Bluetooth, GPS, and NFC, this paper is for you.

QNX at 2013 CES: The media's take

The show ain't over yet, but already, media coverage of the QNX concept car at 2013 CES is pouring in faster than my modest brain can handle. I'm still catching up, but here, in no particular order, are my favorite stories so far.

I'd love to hear what you think of what the media is saying. So before you go, let me know!

New York Times — A Tricked-Out Bentley at CES Shows Off Tech’s Capabilities

Washington Post — CES 2013: Car tech focuses on making the most of your time on the road — safely

Driving.ca — QNX's Bentley gives glimpse of future car tech

Engadget — QNX unveils concept Bentley Continental GT alongside Car Platform 2.0

CNET — QNX concept Bentley showcases future dashboard technologies

Uber Gizmo — QNX Shows Off Bentley Continental GT Concept Car

Car Design News — QNX Car 2 at CES 2013 (video)

CrackBerry —  Dan Dodge shows off the QNX Bentley concept car at CES (video)

BerryReview — Monitor RIM’s New Bentley QNX Concept Car @CES from Your Browser

Auto Guide — QNX Previews Tesla-Sized Touch Screen in a Bentley

Engadget — QNX builds in-car speech framework with AT&T's Watson, knows our true intentions

Itbusiness.ca — RIM subsidiary brings in-car speech recognition (and a Bentley) to CES

LAPTOP — QNX Bringing HTML 5, 4G LTE, Video Conferencing to Future Cars

TechnoBuffalo — Chevy, Ford, and QNX at CES 2013 (video)

Financial Post — CES 2013 kicks off with RIM’s QNX showing off car, a step closer to digitized homes

Ottawa Citizen — Ottawa companies use CES to pitch latest products to the world

Tech 2 — CES 2013: QNX shows off Car Platform 2.0 with concept Bentley Continental GT

Blast Magazine — At 2013’s Consumer Electronics Show, car safety has become a hot topic

N4BB — QNX CAR Application Platform 2.0 Chosen by Delphi for Next-Generation Infotainment

That's it for now. I aim to post some more stories and videos early next week. Stay tuned.

Day two of CES - QNX, Harman, and Vipers, oh my!

Day two came and went very fast. It felt as frantic as day one. I did manage to leave the north hall though... only to find the center hall to be just as intense. Maybe even more so. It had booths the size of city blocks. Maybe even bigger; certainly louder.

I somehow found Harman International. And I was not disappointed. There were at least four demos sporting QNX technology.

QNX CAR 2 was on display and in full form in the Harman booth. Nice to see the QNX logo so prominently displayed.


Closer look at different display in the same demo system. This 3D map from Elektrobit just never gets old.


Harman again. QNX again. This is almost embarrassing! ;-)


Harman and QNX and Vipers, oh my!


I managed to get permission to poke my head into the car. The digital instrument cluster was gorgeous.


The Viper's infotainment system was just as beautiful. Note to my boyfriend: I do believe this car is currently shipping.


The QNX booth staff, including biz dev manager Emil Dautovic, were hard at work once again. Although how hard can it be to talk to nice people all day? Maybe if there were more chairs...


Of course we knew our latest concept car would be a hit but we weren't exactly expecting the mob scene that it was again on day two.


Joe Cusumano, automotive field applications engineer, in conversation.


Linda Campbell, director of QNX strategic alliances, was her usual tireless self.


Our inimitable automotive field applications engineer Dan Baergen.


And our very own Dr. Phil, mastermind of the QNX acoustics processing and noise cancellation products.

QNX-powered Chevy Mylink drives home with 2013 Best of CES award

Congratulations to the infotainment team at Chevrolet! Their next-generation Chevy MyLink system has just won a Best of CES award, in the car tech category. The competition judges were particularly impressed with MyLink's user interface and integration with the car's instrument cluster.

The MyLink system was one of two QNX-powered finalists in this year's competition; the other was Garmin's K2 infotainment platform.

Chevy plans to roll out the new version of MyLink later this year.

Now on YouTube! First video of QNX technology concept car

Yesterday, some friends from Texas Instruments dropped by our CES booth for a demo of the new QNX concept car. The cameras were rolling, and here's what they caught.

Mark Rigley, head of the QNX concept team, did a fantastic job of guiding TI's Michael Guillory through the car's many features, including the gorgeous HD display powered by TI DLP technology and by a TI OMAP 5 processor. Check it out:



My favorite part? The exceedingly cool video conferencing. What's yours?

Day one of CES — so far so great!

The QNX booth was like a mob scene. I think Phil Hetherington, senior director of the QNX acoustics group, hit the nail on the head when he said, "it's like a mosh pit in here." (He meant it in a good way. ;-)

I'm so proud and happy for the guys who spent their Christmas vacation working on the car. People are absolutely clamoring over each other to see it. I think there were more pictures taken of the Bentley than of the attractive talents in all other booths combined.

CES is a show like no other. Even as I write this I know I am missing something exciting on the show floor.

So with this in mind, I bid everyone adieu. I'll get some more shots today.



One of three demos in the Elektrobit booth (3220 North), showcasing the QNX CAR application platform.


Another example of what happens when you put two world-class suppliers together — Elektrobit and QNX, of course.


This demo replicates an impressive infotainment unit created for Audi using the QNX CAR application platform and EB street director.


EB street director and QNX CAR application platform — up close and personal.


Such an honor to be recognized again at this year's CES show!


Solowheel was a big hit! It's a gyro-stabilized electric unicycle that can be used as you would an electric bicycle. www.solowheel.com


Shooting our own video was really challenging as it was so hard to get 5 minutes in the Bentley. When we did, we still couldn't keep people away from the car!


Alexandre James, from the QNX concept design team, moved smoothly from software developer to company spokesperson.


Mark Rigley, concept design team lead, looked a lot more refreshed than he did last week when the car was still in a work in progress.


It seemed almost everyone wanted to capture the excitement of the new QNX technology concept car. Here's Andy Gryc, the perennial spokesperson, giving one of many demos throughout the day.


While the Bentley Continental was the bell of the CES ball, the Jeep Wrangler (2012 QNX reference vehicle) still did the job of capturing people's interest.


Sheridan Ethier, the hard-working manager of the QNX CAR engineering team, talked nonstop about the technology that he and his engineering group create.

Works for me: The making of the QNX technology concept car

If I were asked to create a recruitment video for QNX Software Systems, this is the video I'd want to shoot. Because this is the QNX that I know and feel proud to represent.

A place where people work hard. A place where people work together. A place where people work on technology that is changing how we drive and communicate. A place where I'd rather work than anywhere else.

Enough said. Roll the tape:

Delphi taps QNX CAR platform for next-generation infotainment

This just in: Delphi Automotive, a global supplier of automotive electronics with operations in 30 countries, has chosen the QNX CAR application platform 2.0 for use in next-generation infotainment systems.

This week, at 2013 CES, Delphi will demonstrate how it has leveraged the HTML5 framework in the QNX CAR platform to create a powerful graphics system for automotive use.

According to Jugal Vijayvargiya, senior vice president of Delphi and president of Delphi Electronics & Safety, “Our more than ten-year collaboration with QNX Software Systems has resulted in highly successful infotainment solutions that we’ve been able to bring to market quickly, thanks to the efficient, modular architecture of the QNX platform and its rich associated toolset. We’re very pleased to now extend our work on advanced in-vehicle technology with the latest version of the platform, which we were able to experience through QNX Software Systems' early access program.”

Read the full press release.

If you're 2013 CES this week, be sure to visit the Delphi booth in the North Hall, #730. You can also follow Delphi on Twitter: @DelphiAuto

It's a Bentley! A guided tour of the new QNX technology concept car

"Bend it, shape it, any way you want it"
— Headline from a QNX advertisement, circa 1987

I’m about to show you some pictures of a car. Not just any car, but a powerful, luxurious, and stunningly beautiful car. A car that has undergone a technological transformation.

If you’re like me, you'll be fascinated by the car’s features, some of which have never been seen in a vehicle — until now. But if you can, remember that it isn’t just about the cool features. It’s also about the platform that enabled them.

I’m speaking, of course, of the QNX CAR application platform.

We created the new QNX technology concept car — a modified Bentley Continental GT — to demonstrate that flexibility and customization form the very DNA of the QNX CAR platform. If you’ve seen the QNX reference vehicle, you already know that the platform provides an extremely rich environment for in-car infotainment, complete with HMI frameworks, smartphone integration, an HTML5 engine, a mobile device gateway, and a host of pre-integrated partner technologies — everything to kickstart our customers' projects. But in the automotive world, differentiation is everything. So it’s just as important that the platform enables customers to add their own branding, features, and sizzle. And to do it quickly.

Ease of branding and
personalization is just one

of the capabilities of the
QNX CAR platform.

Which is where the new concept car comes in. To create it, we used the same base QNX CAR platform that we offer our customers. But when you compare the Bentley to the Jeep, which uses a stock version of QNX CAR, the differences are dramatic: different features, different branding, and a different look-and-feel. In effect, the Jeep shows what QNX CAR can do out of the box, while the Bentley shows what QNX CAR lets you do once you start bending it to your imagination. One platform, many possibilities.

Which brings me to the slogan at the top of this post. It's amazing to think that a core value of QNX technology in the 1980s — giving customers the flexibility to achieve what they want to do — remains just as true today. Some values, it seems, are worth keeping.

And now, the car…
I know that you’re anxious to peek inside the car and see what we’ve done. But before we go any further, take a moment to savor the car’s beautifully sculpted exterior. This is one classy set of wheels. In fact, if you ask me, the wheels alone are worth the price of admission:



The awesome (and full HD) center stack
Okay, time to hop in — but get ready to prop up your jaw. Because the first thing you’ll notice is the jaw-droppingly beautiful center stack. This immense touchscreen features a gracefully curved surface, full HD graphics, and TI’s optical touch input technology, which allows a physical control knob to be mounted directly on the screen — a feature that’s cool and useful. (In the photo below, the clock display appears within the knob.)

The center stack supports a host of applications, including a 3D navigation system from Elektrobit that makes full use of the display. Just check out this bird’s-eye view of the Las Vegas Strip:



So how big is the display? Big enough to provide access to other functions, such as the car’s media player or virtual mechanic, and still have plenty room for navigation. Check it out:



The awesome (and very polite) voice rec system
Time to talk to the car. Just say “Hello Bentley,” and the car’s voice recognition system immediately comes to life and begins to interact with you — in a British accent, no less. You can now tell the media player what you’d like to hear and the navigation system where you’d like to go.

To provide natural language speech recognition, the system uses the cloud-based AT&T Watson speech engine, as well as an “intent framework” from QNX. It also uses keyword spotting technology from Sensory so you can start the system simply by talking to it.

The awesome (and nicely integrated) smartphone support
The Bentley also showcases how the QNX CAR platform enables automakers to offer advanced integration with popular smartphones. For instance, the car can communicate with a smartphone to stream music, or to provide notifications of incoming email, news feeds, and other real-time information — all displayed in a manner appropriate to the automotive context. Here's an example:



The awesome (and just plain fun) web app
I know, I know: the car looks cool, but you’re not at CES this week to see it first-hand. But how about the next best thing? Just connect to the web app and keep tabs on the Bentley in real time. (Note: The car will go online later this morning.) The app lets you view a variety of data that the car publishes to the cloud, such as what song the infotainment system is playing and whether someone has just opened a door. It also displays information that would be extremely helpful if this were your personal car, such as fluid levels and tire pressure. (This is a preliminary screen for the app, so I'm not sure if the tire pressures are realistic.)



UPDATE: The web app is now live, and the desktop version features a live camera feed of the Bentley and Jeep. Check it out!



The awesome (and very configurable) digital instrument cluster
The instrument cluster is implemented entirely in software, though you would hardly know it — the virtual gauges are impressively realistic. But more impressive still is the cluster’s ability to morph itself on the fly. Put the car in Drive, and the cluster will display a tach, gas gauge, temperature gauge, and turn-by-turn directions — the cluster pulls these directions from the center stack’s navigation system (cool, that). Put the car in Reverse, and the cluster will display a video feed from the car’s backup camera.



There are other options as well. For instance, the cluster can display information from the media player or display the current weather:



The awesome everything else
I’ve only scratched the surface of what the car can do. For instance, it also provides:

• Advanced multimedia system — Offers direct support for Pandora radio and the first embedded in-car implementation of the Shazam music discovery service.
   
• Video conferencing with realistic telepresence — Separate cameras for the driver and passenger provide independent video streams, while high-definition voice technology from QNX offers expanded bandwidth for greater realism, as well as stereo telepresence for making the remote caller sound as if they’re sitting right next to you.
   
• LTE connectivity — The car features an LTE radio modem, as well as a Wi-Fi hotspot for devices you bring into the car.

Super size those images
Want to see the center stack and instrument cluster in all their high-resolution glory? Just check out our QNX Flickr account.

That's all for now, but stay tuned: We’ll have plenty more news for you today and all through this week.