Apple Intelligence Will Live in All Your New Gear

Tim Cook took to the virtual stage Monday morning to introduce the company's foray into "AI," the term collectively applied to Large Language Models and generative image and text technologies.

Called Apple Intelligence, the new Apple take on AI promises a deeply integrated approach that will show up on your Mac, iPad, and iPhone in very Apple-like ways.

As Cook said onscreen, before segueing to Craig Federighi's deeper dive into the tech, Apple wants its AI to be powerful enough to be useful, intuitive and easy to use, personal, and private.

Deep macOS, iOS, and iPadOS Integration

Federighi talked about how new iOS, iPadOS, and macOS versions will include generative language, generative image technology, and new ways to take action on your behalf.

For example, he said, the new AI will be able to understand your natural language, prioritize notifications, and provide new writing tools system-wide that let you re-write, proof, and summarize text in the OS and in Apple and third-party apps (once developers start to include the new AI APIs into their apps).

You'll be able to create images as you go, personalizing things on the fly, like creating a cute little version of a friend having a birthday in Messages, or a cartoony image of your mom in a superhero cape. There will be three styles to choose from, too, named sketch, illustration, or animation.

Siri Smarts and Privacy

You'll be able to ask Siri (which gets new AI smarts here, soon, and a new look) things like "pull up the files my co-worker shared with me last week," naming the co-worker, and the OS will find what you need. Or, maybe even better, "play the podcast my wife sent me the other day."

Apple Intelligence will use its own on-device generative models to answer your questions and create your images and writing. It can pull information from all your apps, emails, documents, and even whatever's on the screen at the moment.

Federighi noted that when a query goes beyond what the on-device tech can handle, it can be passed with encryption to Private Cloud Compute servers that Apple will maintain with their own silicon chips. Data that's passed off your device will never be stored or made accessible to Apple, and the company has ensured that this can be transparently and independently verified.

Speaking of Siri, Kelsey Peterson, Director of Machine Learning and AI, demoed some cool new features of the 13-year-old personal digital assistant. First of all, Siri can now retain context when you're talking to it and can handle verbal corrections when you're asking for things. In addition, you can type to Siri, and switch between voice and text prompts on the fly. It also inherits on-screen awareness so it can deal with what you're looking at, and gets "hundreds" of new actions that it can manage across apps, while also being able to generate writing and images.

The integration with all your on-device data also helps Siri understand your requests more easily, like grabbing a photo from your driver's license, scanning the number, and putting it into an online form. It also now has access to real-time flight tracking and other data to help you manage plans you never even put into your calendar. Spooky? Maybe. Useful? Probably.

Apple showed off the writing tech in Mail, with examples of writing and re-writing emails with different phrasings and such, along with the ability to summarize your Mail in real-time.

Image and Text Generation

Image generation will take the form of Genmoji, letting you create your own emoji stickers from text prompts. Want a lizard on a skateboard? Make your own Genmoji. There's also a new Image Playground app that will let you mess around with prompts and little images, each created in one of the three styles as above: sketch, illustration, or animation. This playground also refers to the tech, which will be available to app developers as an API.

Need an image to populate a Keynote deck or school presentation? Circle any rough sketch you do on your iPad and Apple Intelligence will create an image that makes it look way better. This new feature is called Image Wand.

Further, Notes gets a summary feature to go along with the new recording and transcription technologies, which also applies to the phone app. If you're on a call and start recording, it will let everyone know that the call is being recorded.

Finally, for those things that Apple Intelligence can't do yet, Federighi said, Apple is adding integration to ChatGPT 4o, with more models to come later.