Apple is currently levelling up Siri’s ‘onscreen awareness,’ enabling it to interact with your screen
Smarter, screen-aware Siri could finally read your mind — sort of
When you purchase through links on our site, we may earn an affiliate commission.Here’s how it works.
Among the digital assistants,Sirihas fared fairly well (certainly compared to Cortana, the ill-fated assistant from rivalMicrosoft), and nowAppleis working on making Siri even smarter by giving it a better sense of what you’re looking at on your screen, calling it ‘onscreen awareness.’
Applehas gone into detail aboutthe development of this feature in an Apple Developer Documentation page, which also notes that it’s due to be included in various upcoming Apple operating system (OS) beta releases for testing.
Apple originally showcased onscreen intelligence in June 2024 and this is a pretty solid indication that it’s still in development.
The core idea of onscreen awareness is pretty straightforward – if you’re looking at items on your screen, say a document or a browser with a page open, and you have a question about something you’re looking at, you can ask Siri (equipped withApple Intelligence). Siri should then be able to respond to your question with relevant information or perform an action asked of it, like sending content to a supported third-party app.
If it works as intended (and that’s a big ‘if’), it will result in a smarter Siri that won’t need you to describe what you want it to do as extensively as you probably need to right now. For example, you can have a document open and ask for a summary without entering the contents of the document yourself.
Apple’s plans for Siri’s foreseeable future
MacRumors reportsthat Apple has provided APIs for developers, enabling them to make the content in their apps available to Siri and Apple Intelligence. The idea is to provide developers with this API many months in advance of its release so that you can use onscreen awareness with third-party apps when it’s officially rolled out.
Currently, we know Apple is trialing outChatGPTin the latestiOS 18.2beta version (among other Apple OSs), andChatGPTintegration coupled with Siri will enable you to ask questions about items (like images, PDFs, and videos) on screen. Then, Siri will take a screenshot and pass it on to ChatGPT to reply to your query. This means that, for now, this functionality is limited to screenshots of your screen.
Get the best Black Friday deals direct to your inbox, plus news, reviews, and more.
Sign up to be the first to know about unmissable Black Friday deals on top tech, plus get all your favorite TechRadar content.
However, onscreen awareness is a little bit different, as pointed out by MacRumors, as onscreen awareness is intended to be integrated in a more direct way.
Siri’s onscreen awareness is supposed so be a sort of capacity to directly analyze, interpret, and interact with the content on your screen. If someone messages you their number, and you’d like to save it, you can tell Siri to create a new contact without having to add additional instructions or Siri having to perform many intermediate steps.
Will Siri survive and even thrive in the AI age?
Apparently, onscreen awareness isn’t actually in the iOS 18.2 developer beta just yet, and MacRumors speculates that it’s one of multiple Siri features that we won’t see for a while, but this news is still promising. One prediction has onscreen awareness to possibly be included in iOS 18.4, expected to be released in the first half of 2025.
If this pans out, Siri could become a more helpful digital assistant, and with Apple’s flair for design, it might become the choice digital assistant for many. This development reminds me of whatMicrosoftis aiming for withCopilot. That hasn’t been received very well from what we’ve seen so far, leaving the goal open for Apple.
YOU MIGHT ALSO LIKE…
Kristina is a UK-based Computing Writer, and is interested in all things computing, software, tech, mathematics and science. Previously, she has written articles about popular culture, economics, and miscellaneous other topics.
She has a personal interest in the history of mathematics, science, and technology; in particular, she closely follows AI and philosophically-motivated discussions.
How to delete a character from Character AI
How to turn off Meta AI
Red One isn’t perfect but it proves we need more action-packed Christmas movies
