Alexa's Questionable AI Upgrade
/cdn.vox-cdn.com/uploads/chorus_asset/file/25469547/DSC07295.jpeg)
A couple days ago, I linked to a post questioning the state of AI within Amazon. Well, we have a new answer today. And it's not great, Alexa. Here's Jennifer Pattison Tuohy:
So, when Amazon announced its new AI-powered voice search function for Fire TVs at its fall event last year, I was intrigued. With its promise to make searching for content easier and smarter, I hoped it would be the solution to my problems. I’ve now had some hands-on time with the new feature, and while it shows promise, like a lot of AI-powered search right now, it’s just not reliable enough to be all that useful.
And this isn't your good (well, mildly good) old Alexa, but shiny new Alexa. Or at least good old Alexa powered by a shiny new LLM:
This is all being powered by a new Amazon-built large language model (LLM) designed to surface movie and TV show content using natural language inputs. It’s starting to roll out to eligible Fire TV devices running Fire OS 6 or higher today. At launch, it’s capable of finding content based on things like topic, genre, plot points, actors, and quotes thanks to being trained on data from services like IMDb.
Sounds promising. But...
Park tells me that kind of in-depth, conversational back and forth is planned for future updates. In my time trying out the current capabilities, I wasn’t able to get it to go beyond two queries before it started to fall apart. It also struggled to offer up more than a couple correct answers for broader queries like “Show me Oscar-winning movies from the 1970s.”
“It’s certainly day one for us,” explained Park when I asked about these limits. “We definitely have a view of what we need to do to improve it, so that no matter what the customer asks, we’re able to find the right content for them.”
That sounds more like day zero to me.
Next, I tried something much more specific. We like to find series we can binge-watch together, so I asked, “Show me TV series with more than six episodes that are highly rated.” It suggested two shows, both anime. One was rated a nine out of 10 but the other was a five out of 10. Even for an avid anime fan, that isn’t a great result.
At this point, I decided to go from what I thought would be a softball question. The kind of thing I might have asked that video store clerk: “Show me something good to watch.” The results were… bizarre. Its first suggestion was Miss Marple (a classic British detective show that I do actually love, but is very old), but its second and third options were The Curious Female and Super Vixens, which not only appear to be ’70s soft-core porn but have very poor ratings on IMDB.
That sounds like more day negative one to me.
Amazon does add helpful context to the results, including showing you which apps you have that can stream the show and whether it’s free to you. But what I want from a more intelligent search service is not something to jog my memory but something smart enough to find me something good to watch. I want it to use its vast dataset to sift through the cruft and find me the quality. I want it to be that old-school video store clerk from my youth.
Yes, and this is related to the idea that AI can be used to sift through all sorts of stuff, like pictures and articles to find you the right one, right now. I know it's a complicated problem, but I also know it's one we have all the basic building blocks for with regard to AI right now. This isn't science fiction AI. This is AI as it stands today AI. And it really should be obvious given the corpus of data on which Amazon sits and what they're trying to do here. I continue to not quite understand why these companies ship this stuff so half baked. I guess to help train the data, but it leads to such a poor user experience that it could backfire. Hopefully help is on the way now. And not in the form of Rufus.