Reasoning About the Reasoning of Reasoning Models

Twas the night before WWDC and all through the house, not a creature was stirring, but a group of researchers studying AI reasoning models were!

Look, it is a bit curious that on the cusp of this year's WWDC – an event where everyone is expecting Apple to underwhelm when it comes to AI announcements – a team of Apple researchers put out a paper suggesting that AI itself is underwhelming. More specifically, that perhaps the hottest sector of AI advances this past year, so-called "reasoning", breaks down at a certain scale. And, more damning, what such systems are doing isn't really reasoning at all. I'm not suggesting their work isn't legitimate or even correct – I'll leave such assessment to others, that's above my pay grade – I'm simply pointing out that it's fairly convenient for Apple to have any element of the AI revolution called into question just as they're about to be dragged through the mud on the topic once again.

How does the saying go? Timing of your research papers is everything...