"Prepare for AI-Powered Super Breaches"
Kevin Beaumont
From a pure product perspective, I think Microsoft's new 'Recall' feature – the signature AI feature for Windows which they showed off during the Copilot+ PC unveiling last month – sounds great. The ability to instantly call up anything you've done on your computer – every email you've written, every website you've visited, every program you've used, every message you've sent no matter via which service, etc – is exactly the type of computing superpower that AI can help unlock. But this is a pretty damning look at the downsides:
Q. The data is processed entirely locally on your laptop, right?
A. Yes! They made some smart decisions here, there’s a whole subsystem of Azure AI etc code that process on the edge.
Q. Cool, so hackers and malware can’t access it, right?
A. No, they can.
Q. But it’s encrypted.
A. When you’re logged into a PC and run software, things are decrypted for you. Encryption at rest only helps if somebody comes to your house and physically steals your laptop — that isn’t what criminal hackers do.
For example, InfoStealer trojans, which automatically steal usernames and passwords, are a major problem for well over a decade — now these can just be easily modified to support Recall.
Q. But the BBC said data cannot be accessed remotely by hackers.
A. They were quoting Microsoft, but this is wrong. Data can be accessed remotely.
Okay, but can't hackers already do this via their own malicious software they install on your machine remotely? Sure, but this has the potential to make such infiltrations more scalable as it creates a common target where the data is on every PC running Recall. But personally, I'm less worried about that than this:
Q. Does this enable mass data breaches of website?
A. Yes. The next time you see a major data breach where customer data is clearly visible in the breach, you’re going to presume company who processes the data are at fault, right?
But if people have used a Windows device with Recall to access the service/app/whatever, hackers can see everything and assemble data dumps without the company who runs the service even being aware. The data is already consistently structured in the Recall database for attackers.
So prepare for AI powered super breaches. Currently credential marketplaces exist where you can buy stolen passwords — soon, you will be able to buy stolen customer data from insurance companies etc as the entire code to do this has been preinstalled and enabled on Windows by Microsoft.
It feels like Microsoft, at the very least, may have to limit Recall to individual users only, at least to start. If corporations start installing and running the feature – even if they're not actually using it – it's easy to see the catastrophic outcomes. Yes, it sounds like there will be settings for companies to disable the feature on their fleets, but that's not good enough. If some IT manager at a company that handles credit card information doesn't turn it off, and there's a breach... yikes.
Beaumont concludes:
Q. What should Microsoft do?
A. In my opinion — they should recall Recall and rework it to be the feature it deserves to be, delivered at a later date. They also need to review the internal decision making that led to this situation, as this kind of thing should not happen.
Earlier this month, Microsoft’s CEO emailed all their staff saying “If you’re faced with the tradeoff between security and another priority, your answer is clear: Do security.”
We will find out if he was serious about that email.
They need to eat some humble pie and just take the hit now, or risk customer trust in their Copilot and security brands.
It is sort of wild that Microsoft has just been through the ringer with regard to years worth of privacy issues coming to a head and having to issue the above statement. And then they announce this feature. I'm reminded of the time when Meta (then Facebook) was in the middle of the Cambridge Analytica scandal and then they release a camera for your living room. Optics guys, quite literally!
Again, I think this feature sounds great on paper. It sounds so good that it sounds like something Apple should be doing as well with macOS. But these companies don't just need to think about security here, they probably need to invent a whole new form of security to protect such data collected this way. If Apple does indeed go down this path, you can bet that will be talking point number one. Microsoft, it seems, simply relied on the gullibility of the press to think that "all data stays on your device" is enough. Security checkbox checked!
As I wrote in May, when the feature was still just rumored:
I think this is exactly the type of thing that Microsoft, as an OS-maker should be doing. As Warren notes, the startup Rewind (which is now actually Limitless, makers of that interesting new AI pendant) has been doing this. But a third-party is never going to be able to do this to extent that the OS itself can. Not to mention the security concerns – which maybe Microsoft won't actually be better to address anyway! It's why I suspect Apple may be cooking up something similar as a part of its AI story.
Are Beaumont's concerns overblown? Some are quick to point to his history in criticizing Microsoft, having worked there for a time (which he addresses in his post), but he's far from the only one with such concerns. And even if these are extreme examples of what could go wrong, this is such a potentially dangerous new attack vector that at the very least, this needs to be fully addressed more in-depth than just the simple talking points.
Naturally, I'm reminded of two superhero movies:
And:
Update June 8, 2024: Microsoft has now reversed their general roll-out strategy and changed many of the details for 'Recall':
M.G. Siegler