Apple pushing faster on agentic Siri would carry three big risks

Yesterday saw two reports about Apple software development. The first said that the company was aiming to give Siri more agentic capabilities “as soon as next year,” while the second said that CEO John Ternus is pushing Craig Federighi to speed up software development.

If accurate, it seems near-certain the two reports are connected. But if Ternus really is pushing for an acceleration in Siri’s ability to act as an AI agent, that comes with three potentially huge risks …

Both reports were by Bloomberg’s Mark Gurman, though he didn’t directly connect them. On the Siri side, his newsletter contrasted the agentic abilities of some competitor models, suggesting that Apple ultimately wants to achieve the same things.

The report about Ternus pushing Federighi for faster software development mentions “moving beyond necessary improvements to Siri,” of which agentic capabilities would be the obvious candidate.

But agentic capabilities are among the most dangerous forms of AI given the multitude of things that could go very, very wrong.

Failing to carry out the task

The most obvious is simply failing to carry out the task. Depending on the importance of it, the consequences could range from annoyance through to significant harm.

Technology has long failed to live up to one of its biggest promises: relieving us of mundane work in order to devote our time to more fulfilling things. For example, obtaining tickets to a popular gig or other event, which currently involves scheduling time in order to repeatedly refresh a page, perhaps wait through a queuing system, and finally carry out the transaction. It would be great to be able to hand this off to an AI.

But even on something relatively unimportant like this, if we delegated the task and the AI failed to obtain the tickets, especially if this was due to a silly error, this could prove enormously frustrating. If the task were more important, the results could obviously be far worse.

Acting inappropriately

Potentially even worse is achieving the task but doing so in inappropriate ways. We’ve already seen high-profile examples of this, like the Hugging Face hack caused by an AI model seeking so hard to carry out instructions that it used underhand methods to bypass crucial safeguards in the process.

Putting privacy at risk

Another huge risk is to privacy. Just today we learned that Claude Code exposed over 2,000 real secrets to GitHub, as Cybernews reports.

This was just a few days after Anthropic started encouraging Claude users to connect their bank and finance card accounts to the AI chatbot. (My own answer is “Hell, no.”)

When we delegate a task to an AI system that requires access to any of our personal data, and we’re not expected to check on it again until the task is complete, the potential for disaster here is enormous.

The bottom line

It’s worth prefixing this with a note that the reliability of the paired reports is far from certain. In particular, in a closed-door meeting between Ternus and Federighi, it’s hard to see how the content could have leaked unless one of the two men chose to do this. If Ternus leaked it, that suggests a very high level of frustration on his part. But leaving that aside …

We’re currently witnessing an AI arms race, with each company aiming to beat its competitors to next-generation capabilities, even while acknowledging a non-trivial possibility that the pace of development could put humanity’s future at risk.

In my view, the very last thing we need is Apple joining this race. I know it can be frustrating when Apple lags behind other companies in the introduction of some types of new tech, but this is one field in which I would be very happy to see Apple proceed very much more cautiously than others.

Right now, AI needs at least one player that prioritizes safety and privacy over speed to market – in that way, we’d get a choice. If we want agentic capabilities sooner rather than later, and are prepared to run the risks involved in this, then we can turn to companies like Anthropic and Meta. If we are prepared to wait for safer systems, then Apple should provide that alternative.

What’s your view on this? Please share your thoughts in the comments.

...