Hello, Bentley: Using Sensory speech technology to create a natural user experience

By Bernard Brafman, Sensory Inc., and Justin Moon, QNX Software Systems

In-vehicle infotainment systems are becoming more and more complex, and integral to the overall driving experience. As this trend continues, it will become increasingly important to create systems that support multiple forms of user interaction. If you’re driving, the last thing you want to do is enter a destination manually, or search for your favorite artist in a playlist by using a touch screen, jog wheel, or other manual input method. Drivers want and require a user experience that is both simple and natural; integration of speech recognition technology goes a long way toward achieving that goal.

In fact, speech recognition is a key component of the latest QNX technology concept car, a modified Bentley Continental GT. The speech rec system lets you plot a route or select your favorite artist using natural speech, but it goes even further by letting you simply ask the car to perform an action. Leveraging Sensory’s FluentSoft SDK, more specifically the TrulyHandsfree^TM Voice Control technology, the QNX concept development team implemented keyword spotting techniques to interact with the vehicle.

So how does this work? Well, let’s say you’re in Vegas and need directions to the Wynn Casino. To engage the cloud-based Watson speech system, you simply say “Hello Bentley” — no need to push a button. You then complete the request by saying “Take me to the Wynn Casino.” FluentSoft, along with the architecture of the advanced speech recognition system included in the QNX CAR platform, allowed the team to create this seamless, easily implemented, and well-executed voice interaction experience.


When you say “Hello Bentley,” the QNX concept car displays a visual prompt at the top of the screen, indicating that the speech rec system is now listening for natural speech or directed commands.

Multiple triggers
The team plans to further utilize Sensory technology in future concept car releases. The current implementation has the single “Hello Bentley” trigger, which engages the speech system. But TrulyHandsfree Voice Control supports multiple active triggers as well as a robust recognition vocabulary to create a rich command-and-control user experience that doesn’t require prompts or pauses. Thus, it’s possible to create a hybrid system that is seamless and transparent to the user. For instance, “Hello Bentley air 68 degrees” or “Hello Bentley what time is it in Tokyo?” can both be executed flawlessly, regardless of which speech rec system is engaged.

A matter of choice
For an even more personalized experience, this technology can allow drivers to create their own custom trigger with a simple one-time enrollment process that verifies their identity as a voice password or identifies one of several previously enrolled. This creates a custom experience not only by letting you choose your own trigger phrases (come on now, who hasn’t named their car at some point?), but also by recalling individual preferences such as seat position, steering wheel position, and multimedia presets.

Look for these enhanced features in concept cars to come!

---

Bernard Brafman is vice president of business development for Sensory, Inc., responsible for strategic business partnerships. He received his MSEE from Stanford University. Contact Bernard at bbrafman@sensoryinc.com

Justin Moon is a global technical evangelist for the automotive business development team at QNX Software Systems.

Designing interfaces from the outside in

User interfaces are a pet peeve of mine.

I’m one of those people whose VCR always blinked 12:00. Not because I couldn’t figure it out but because I resented that I had to.

Basically, I have neither the time nor the inclination to read manuals. If I’m paying good money for a consumer-facing product then it better not require an engineering degree to use it.

Not surprisingly, then, I think UI design is every bit as important as product; maybe even more so. Because if your user experience sucks, make no mistake; I will be walking and talking to your competitors.

It wasn’t until I entered the glamorous world of software development that I came to the following conclusion: Interfaces are complicated because development tools require an engineer (or similarly brilliant individual) to use them.

Of course this is a sweeping statement and I’ll gladly debate it but the point is this: Someone with unique skills and knowledge about user-centric design should be creating interfaces. Not someone who knows the product from the inside out.

I know in a traditional model this can create a lot of churn but companies like Crank Software have come up with a way to decouple the roles of embedded engineer and UI designer, allowing them to work in parallel while focusing on their individual core competencies.

I spoke to several members of the QNX concept development team when they were heavily embroiled in creating the latest technology concept car. It was obvious when talking to the engineers and the UI designers that Crank’s Storyboard made both jobs that much easier and the process a whole lot quicker. The end result, achieved in a very short time frame, speaks for itself.

This is great news for people like me who curse like sailors whenever using a remote, microwave, GPS, treadmill, camera, and so on. Indeed, I'm counting on teams like QNX and Crank to ensure the digital car is an enjoyable and intuitive  experience. If not, I do know who I'm gonna call.

Meet the QNX concept team: True Nguyen, UX designer

We continue our spotlight on the QNX concept development team with True Nguyen, the team's user experience designer.

We interviewed True just prior to CES 2013, and she was hoping that people's impressions of the latest QNX technology concept car would be as fantastic as hers. True's love of cars stems back to her childhood, and that really comes out in the interview.

If you haven't had a chance to meet the other team members, you can read their stories here.

Next up, we'll interview Alexandre James to get his impressions of the Bentley and the buzz from CES 2013.

Our best CES yet

Anecdotes and observations from the QNX booth at 2013 CES

As a wrap-up to last week’s Consumer Electronics Show, I would love to regale you with all the cool technologies and nifty gadgets that I saw. But over the course of the entire four days, I rarely left the 20’x40’ patch of white carpet that was the QNX booth — with brief exceptions, of course, for bodily maintenance. The booth was just too busy for me to get away. If you checked out the QNX booth webcam, you know what I'm talking about.

Paul Leroux and Nancy Young have already posted a lot of information and photos about the show and the new QNX concept car, which is based on a Bentley Continental GT. So let me provide my personal view of CES through assorted anecdotes or observations collected at the booth.

• As you’d expect, the Bentley got a lot of attention. But our reference vehicle, based on a Jeep Wrangler, got more attention than I thought it would, even though this is the third time we’ve shown it in public. Many of the people interested in the Jeep just wanted to see what our QNX CAR application platform looked like “out of the box” without customization. And some were confessed Jeep or truck aficionados, without the “luxury brand lust” experienced by most.
   
• People in the auto industry knew who we were without introduction. Non-automotive people didn’t know who we were until I mentioned that “we are a wholly owned subsidiary of Research In Motion,” at which point most of them said “Oh, you’re that QNX.” Seems that your average person has heard quite a bit about QNX in the context of BlackBerry, but has no idea that the same company is doing things in automotive — or in anything else, for that matter. I usually then spoke about our 30+ year legacy in life- and mission-critical systems. When people learned that an OS used for mission-critical systems will also power their next potential phone, their reaction was “wow—that’s really cool.”
   
• Tanner Foust is a really nice young kid. (Actually, he’s not that much younger than me, but he sure looks young!) I didn’t know who he was when he was being filmed in the booth, surrounded by a throng of admirers. But since then, I’ve watched a lot of his YouTube videos and boy, can he drive! He's an accomplished race car driver, TV personality, and stuntman for lots of famous movies, but it’s nice to see he hasn’t let it go to his head.
   
• We wanted to make sure that our concept car respected the Bentley brand. To do that, we ran our design sketches by the folks at Bentley and they occasionally suggested some tweaks. It was all our own work, however, and the Bentley folks never saw it before it hit the show floor. When they came to the booth, they were very happy with what they saw — enough so that they said “it looked like we did it.” That, to me, was the ultimate compliment.
   
• Most frequent question: “Are you giving this away?” As it turns out, it’s something that people have said for every concept car we’ve done to date. Second most frequent question: “Can I drive it?” Unfortunately, but unsurprisingly, the answer to both is “No.”
   
• I was a little surprised by the enthusiastic response to the car's video conferencing. Of course, it works only while the car is parked, and you only get audio while the car is in Drive. But the part that seemed to impress people the most is the audio: two channel stereo and a full 20Hz to 22KHz means that the call sounds so much better than your typical hands-free call. You could see the reaction when the our director of acoustics Phil Hetherington started talking — you don’t know what you’ve been missing until you hear it.
   
• Bentley wanted us to add our video conferencing solution to the technology concept car. Because many Bentley vehicle owners aren’t necessarily the drivers, this feature makes a whole lot more sense for rear-seat systems than you might initially imagine.
   
• I was really impressed by two members of the media: Brian Cooley of CNET and Craig Peterson of Clear Channel. Both could receive a five minute technology core dump, quickly digest it, and talk intelligently about it on video or live radio (respectively) with no stumbles, questions, or missteps. I’ve had the pleasure of seeing both in action before, but their consummate professionalism is really quite amazing.
   
• I and every other QNX’er was delighted that we didn’t win the CNET Best of CES award! Instead, our customer, Chevrolet, won it for their MyLink system, and we couldn’t have been happier. Two out of the three nominees were QNX-based systems (the Garmin K2 was the other), so our odds were good. I’d rather that we never win another Best of CES award if it meant that one of our customers could always win instead.
   
• A number of people asked about the RIM booth and its absence. I explained that RIM was focusing on their launch at the end of January, and that since they wouldn’t have a new product to show the public, it didn’t make sense to be there. (It’s notable that Microsoft wasn’t there either, and Apple never is.) RIM was in Las Vegas in a hotel outside the convention center, giving media private previews of the upcoming phones that seemed to be extremely well received. And we had a few of our RIM compatriots helping us out at the QNX booth as well.
   

That’s all I’ve got to say about CES 2013 — our best show yet. See you next year!

Okay, time to get technical

Have glossy photos of the QNX concept car left you hungry for more? Dig into a technical whitepaper with our friends from Texas Instruments.

By now, many of you have seen photos and videos of the new QNX technology concept car, a specially modded Bentley Continental GT. Now, I'd like to say that the car was completed in record time by a small team of highly creative QNX engineers. And in many ways, that's absolutely true. But it's just as true that the work started more than 10 years ago, when QNX Software Systems started to build deep partnerships with leading players in the auto industry.

Because the truth is, you don't create this kind of magic overnight. And you don't do it on your lonesome. QNX has become successful in automotive for many reasons, but one of the most important is our ability to work closely, and productively, with A-list partners like Texas Instruments.

Inside the concept car

Take a look at the amazing displays in the Bentley, and the speed at which the screens redraw, and you get a taste just for how well QNX software and TI silicon work together under the covers.

Which brings me to a new white paper co-authored by Andy Gryc of QNX, and Matt Watson and Scott Linke of TI. It's titled "In-Vehicle Connectivity is So Retro," and among other things, it tells the story of how technologies from QNX and TI have co-evolved to help automotive developers build high-performance systems in less time and at less cost.

If your working vocabulary includes terms like OMAP 5, 1080p video decode/encode, floating-point DSP, MOST MLB, Ethernet AVB, PCIe, SATA, WiLink, Bluetooth, GPS, and NFC, this paper is for you.

Day four of CES: The home stretch

We all thought that day four would be a lot less busy and it was, however, we still had people waiting for a demo up until the dying minutes of the show.

The day opened with a video shoot by Brian Cooley - one of my favorite interviewers. This man can distill a complex technology down to the basics after a 10 minute briefing. And he can shoot a video in one take. Impressive.

Brian Cooley dropped by to do a spot for CNET. I really admire his ability to distill complicated subjects into very digestible bits.

Dr. Phil once again. Only this time he is in the soundproof booth, ready to wow everyone with a the first-ever HD stereo video call in the car.

Delphi had a beautiful QNX-based system in an Audio A7. Everything was gorgeous, responsive, and flawless.

One of the really cool things about the Delphi system was the theater seating in the back seat. Why watch a movie anywhere else?

Andrew Poliak, director of business development, did booth duty whenever not in meetings.

Day three of CES: Almost too exciting

Day three was interesting in that we had too many VIPs to see the Bentley at the same time: Tanner Foust, Thorsten Heins, and high-level execs from both Mercedes and Porsche. All had a good long look and chat inside the Bentley. I won't tell you which ones had to wait. ;-)

Nice problem to have, I know... but a little stressful nonetheless. (You never want to turn people away.) Of course everything worked out just fine. Better than fine, really.

Thorsten was very gracious and took the time to shake hands with all QNX employees. Tanner Foust (whom I did not know before the show) patiently posed for pictures with those QNX employees who were more informed than me about his star status.

Thorsten Heins (RIM CEO) had a good long chat with his buddy, Dan Dodge (QNX CEO).

Race car driver, Tanner Foust, drifted into the booth to see what QNX was all about. Here he is with Justin Moon from the concept development team.

Kerry Johnson, manager of automotive product management, pinch hit between meetings with customers, press, and analysts.


Victor Marques took his job on the show floor as seriously as he does when in the office.

QNX at 2013 CES: The media's take

The show ain't over yet, but already, media coverage of the QNX concept car at 2013 CES is pouring in faster than my modest brain can handle. I'm still catching up, but here, in no particular order, are my favorite stories so far.

I'd love to hear what you think of what the media is saying. So before you go, let me know!

New York Times — A Tricked-Out Bentley at CES Shows Off Tech’s Capabilities

Washington Post — CES 2013: Car tech focuses on making the most of your time on the road — safely

Driving.ca — QNX's Bentley gives glimpse of future car tech

Engadget — QNX unveils concept Bentley Continental GT alongside Car Platform 2.0

CNET — QNX concept Bentley showcases future dashboard technologies

Uber Gizmo — QNX Shows Off Bentley Continental GT Concept Car

Car Design News — QNX Car 2 at CES 2013 (video)

CrackBerry —  Dan Dodge shows off the QNX Bentley concept car at CES (video)

BerryReview — Monitor RIM’s New Bentley QNX Concept Car @CES from Your Browser

Auto Guide — QNX Previews Tesla-Sized Touch Screen in a Bentley

Engadget — QNX builds in-car speech framework with AT&T's Watson, knows our true intentions

Itbusiness.ca — RIM subsidiary brings in-car speech recognition (and a Bentley) to CES

LAPTOP — QNX Bringing HTML 5, 4G LTE, Video Conferencing to Future Cars

TechnoBuffalo — Chevy, Ford, and QNX at CES 2013 (video)

Financial Post — CES 2013 kicks off with RIM’s QNX showing off car, a step closer to digitized homes

Ottawa Citizen — Ottawa companies use CES to pitch latest products to the world

Tech 2 — CES 2013: QNX shows off Car Platform 2.0 with concept Bentley Continental GT

Blast Magazine — At 2013’s Consumer Electronics Show, car safety has become a hot topic

N4BB — QNX CAR Application Platform 2.0 Chosen by Delphi for Next-Generation Infotainment

That's it for now. I aim to post some more stories and videos early next week. Stay tuned.

Full-duplex in the car? Who even knows what this means?

I know lots of people don't understand full-duplex. Hell, I think most people have never even heard of it. Unfortunately, these same people have repeatedly experienced its poor cousin — half-duplex — without really understanding either.

Please don't take me wrong. I'm not patronizing. I spent a few years in telecoms and barely understand it myself. What I do know is that half-duplex = bad. And that full-duplex = good.

So when I talked to my colleagues at the office today, I knew we were using half-duplex. How did I know? I started to say something and so did someone else at the other end of the line. I couldn't hear them talking while I was talking (a certain amount of latency adds to the circus) so I stopped talking... and so did they. Then there were lots of simultaneous barely-understood apologies and a long uncomfortable silence. Then we both tried to break the silence at the same time; more uncomfortable silence. Very awkward and distracting. I know you know what I mean.

So... um... could someone please fix this? I mean, we send people to the moon after all.

I hate to blow our own (proverbial) horn (well sometimes) but believe me, I have to. QNX has THE best audio technology solution in or out of the car. And this technology, just so happens to be in the new QNX technology concept car at CES. Yes, really!

Today I witnessed a conversation in the QNX booth between someone in the Bentley and someone in a sound-proof booth. Well, to my (sheer) delight, one person talked while the other person talked over him... and both heard the other! Just like a real face-to-face conversation with overlapping dialogue. It was so natural, it almost slipped by as if it were expected:



In a world where communication is more often than not at the root of all successes and failures, I think this is nothing short of a long-overdue miracle.

Two QNX customers, Chevrolet and Garmin, shortlisted for 2013 Best of CES awards

Who doesn't love to win an award? Last year, for example, we were absolutely thrilled that our QNX CAR application platform drove home with a Best of CES award. Heck, I'm still excited!

But here's the thing. All of the products and services that QNX offers are designed with one goal in mind: to make our customers successful. The more our customers succeed, the more QNX succeeds. Which is why I am doubly excited today. Because not one, but two customers have nabbed finalist spots at this year's Best of CES awards for their QNX-based products: Chevrolet for its second-generation MyLink system, and Garmin for its K2 infotainment platform.

Congratulations to our friends at Chevrolet and Garmin — I'll be rooting for all of you!

The winners of this year's Best of CES awards will be announced today, at 11 am PT.

Now on YouTube! First video of QNX technology concept car

Yesterday, some friends from Texas Instruments dropped by our CES booth for a demo of the new QNX concept car. The cameras were rolling, and here's what they caught.

Mark Rigley, head of the QNX concept team, did a fantastic job of guiding TI's Michael Guillory through the car's many features, including the gorgeous HD display powered by TI DLP technology and by a TI OMAP 5 processor. Check it out:



My favorite part? The exceedingly cool video conferencing. What's yours?

Day one of CES — so far so great!

The QNX booth was like a mob scene. I think Phil Hetherington, senior director of the QNX acoustics group, hit the nail on the head when he said, "it's like a mosh pit in here." (He meant it in a good way. ;-)

I'm so proud and happy for the guys who spent their Christmas vacation working on the car. People are absolutely clamoring over each other to see it. I think there were more pictures taken of the Bentley than of the attractive talents in all other booths combined.

CES is a show like no other. Even as I write this I know I am missing something exciting on the show floor.

So with this in mind, I bid everyone adieu. I'll get some more shots today.



One of three demos in the Elektrobit booth (3220 North), showcasing the QNX CAR application platform.


Another example of what happens when you put two world-class suppliers together — Elektrobit and QNX, of course.


This demo replicates an impressive infotainment unit created for Audi using the QNX CAR application platform and EB street director.


EB street director and QNX CAR application platform — up close and personal.


Such an honor to be recognized again at this year's CES show!


Solowheel was a big hit! It's a gyro-stabilized electric unicycle that can be used as you would an electric bicycle. www.solowheel.com


Shooting our own video was really challenging as it was so hard to get 5 minutes in the Bentley. When we did, we still couldn't keep people away from the car!


Alexandre James, from the QNX concept design team, moved smoothly from software developer to company spokesperson.


Mark Rigley, concept design team lead, looked a lot more refreshed than he did last week when the car was still in a work in progress.


It seemed almost everyone wanted to capture the excitement of the new QNX technology concept car. Here's Andy Gryc, the perennial spokesperson, giving one of many demos throughout the day.


While the Bentley Continental was the bell of the CES ball, the Jeep Wrangler (2012 QNX reference vehicle) still did the job of capturing people's interest.


Sheridan Ethier, the hard-working manager of the QNX CAR engineering team, talked nonstop about the technology that he and his engineering group create.

Works for me: The making of the QNX technology concept car

If I were asked to create a recruitment video for QNX Software Systems, this is the video I'd want to shoot. Because this is the QNX that I know and feel proud to represent.

A place where people work hard. A place where people work together. A place where people work on technology that is changing how we drive and communicate. A place where I'd rather work than anywhere else.

Enough said. Roll the tape:

It's a Bentley! A guided tour of the new QNX technology concept car

"Bend it, shape it, any way you want it"
— Headline from a QNX advertisement, circa 1987

I’m about to show you some pictures of a car. Not just any car, but a powerful, luxurious, and stunningly beautiful car. A car that has undergone a technological transformation.

If you’re like me, you'll be fascinated by the car’s features, some of which have never been seen in a vehicle — until now. But if you can, remember that it isn’t just about the cool features. It’s also about the platform that enabled them.

I’m speaking, of course, of the QNX CAR application platform.

We created the new QNX technology concept car — a modified Bentley Continental GT — to demonstrate that flexibility and customization form the very DNA of the QNX CAR platform. If you’ve seen the QNX reference vehicle, you already know that the platform provides an extremely rich environment for in-car infotainment, complete with HMI frameworks, smartphone integration, an HTML5 engine, a mobile device gateway, and a host of pre-integrated partner technologies — everything to kickstart our customers' projects. But in the automotive world, differentiation is everything. So it’s just as important that the platform enables customers to add their own branding, features, and sizzle. And to do it quickly.

Ease of branding and
personalization is just one

of the capabilities of the
QNX CAR platform.

Which is where the new concept car comes in. To create it, we used the same base QNX CAR platform that we offer our customers. But when you compare the Bentley to the Jeep, which uses a stock version of QNX CAR, the differences are dramatic: different features, different branding, and a different look-and-feel. In effect, the Jeep shows what QNX CAR can do out of the box, while the Bentley shows what QNX CAR lets you do once you start bending it to your imagination. One platform, many possibilities.

Which brings me to the slogan at the top of this post. It's amazing to think that a core value of QNX technology in the 1980s — giving customers the flexibility to achieve what they want to do — remains just as true today. Some values, it seems, are worth keeping.

And now, the car…
I know that you’re anxious to peek inside the car and see what we’ve done. But before we go any further, take a moment to savor the car’s beautifully sculpted exterior. This is one classy set of wheels. In fact, if you ask me, the wheels alone are worth the price of admission:



The awesome (and full HD) center stack
Okay, time to hop in — but get ready to prop up your jaw. Because the first thing you’ll notice is the jaw-droppingly beautiful center stack. This immense touchscreen features a gracefully curved surface, full HD graphics, and TI’s optical touch input technology, which allows a physical control knob to be mounted directly on the screen — a feature that’s cool and useful. (In the photo below, the clock display appears within the knob.)

The center stack supports a host of applications, including a 3D navigation system from Elektrobit that makes full use of the display. Just check out this bird’s-eye view of the Las Vegas Strip:



So how big is the display? Big enough to provide access to other functions, such as the car’s media player or virtual mechanic, and still have plenty room for navigation. Check it out:



The awesome (and very polite) voice rec system
Time to talk to the car. Just say “Hello Bentley,” and the car’s voice recognition system immediately comes to life and begins to interact with you — in a British accent, no less. You can now tell the media player what you’d like to hear and the navigation system where you’d like to go.

To provide natural language speech recognition, the system uses the cloud-based AT&T Watson speech engine, as well as an “intent framework” from QNX. It also uses keyword spotting technology from Sensory so you can start the system simply by talking to it.

The awesome (and nicely integrated) smartphone support
The Bentley also showcases how the QNX CAR platform enables automakers to offer advanced integration with popular smartphones. For instance, the car can communicate with a smartphone to stream music, or to provide notifications of incoming email, news feeds, and other real-time information — all displayed in a manner appropriate to the automotive context. Here's an example:



The awesome (and just plain fun) web app
I know, I know: the car looks cool, but you’re not at CES this week to see it first-hand. But how about the next best thing? Just connect to the web app and keep tabs on the Bentley in real time. (Note: The car will go online later this morning.) The app lets you view a variety of data that the car publishes to the cloud, such as what song the infotainment system is playing and whether someone has just opened a door. It also displays information that would be extremely helpful if this were your personal car, such as fluid levels and tire pressure. (This is a preliminary screen for the app, so I'm not sure if the tire pressures are realistic.)



UPDATE: The web app is now live, and the desktop version features a live camera feed of the Bentley and Jeep. Check it out!



The awesome (and very configurable) digital instrument cluster
The instrument cluster is implemented entirely in software, though you would hardly know it — the virtual gauges are impressively realistic. But more impressive still is the cluster’s ability to morph itself on the fly. Put the car in Drive, and the cluster will display a tach, gas gauge, temperature gauge, and turn-by-turn directions — the cluster pulls these directions from the center stack’s navigation system (cool, that). Put the car in Reverse, and the cluster will display a video feed from the car’s backup camera.



There are other options as well. For instance, the cluster can display information from the media player or display the current weather:



The awesome everything else
I’ve only scratched the surface of what the car can do. For instance, it also provides:

• Advanced multimedia system — Offers direct support for Pandora radio and the first embedded in-car implementation of the Shazam music discovery service.
   
• Video conferencing with realistic telepresence — Separate cameras for the driver and passenger provide independent video streams, while high-definition voice technology from QNX offers expanded bandwidth for greater realism, as well as stereo telepresence for making the remote caller sound as if they’re sitting right next to you.
   
• LTE connectivity — The car features an LTE radio modem, as well as a Wi-Fi hotspot for devices you bring into the car.

Super size those images
Want to see the center stack and instrument cluster in all their high-resolution glory? Just check out our QNX Flickr account.

That's all for now, but stay tuned: We’ll have plenty more news for you today and all through this week.