What’s Really Happening with Siri AI and Apps

Implementing Siri AI in third party apps

0:00Welcome to a special episode of Primary Technology. I'm here with indie developer Ryan Wesley, the founder and maker of FoodNoms, which is an amazing food tracking app, the first one I've stuck with long term, been using it for over a year, and it is incredible and had a huge update for iOS 26. And now I want to ask him about Siri AI and apps. So Ryan, thanks so much for joining me today. Yeah, thanks, Stephen. So I posted on Mastodon that as I was making a video about third party apps and Siri AI, you had a lot of thoughts and explanation about what's happening there.

And even as I started making my video, I was like, man, all these apps say they work with Siri AI, they've been updated. But trying to get Siri AI to use them can is a very mixed bag. And a lot of times, I don't know what's happening. Sometimes it'll ask, do you want to enable these shortcuts actions, but it's not really a shortcut.

0:49So if we can start from the top, maybe you can explain when an app developer right now is wanting to say their app works with Siri AI. What does that mean from an implementation standpoint? Like what are you doing to an app so that it works with Siri AI today? For most developers, what they would say is that they are using App Intense. There's kind of App Intense is like this big umbrella thing. And there's a lot of things that fall underneath it. And some of the newer stuff is what they call like schema domains.

1:22And so Apple has like published these new APIs that are tailored for certain like verticals, certain like specific kinds of apps. So like calendar apps, to do apps, reminder apps, photo editing apps. There's kind of like, I don't know how many, like a dozen different categories. And those are, if your app like fits one of those verticals, then you can basically just like put in the code that like, you know, this is my data model. This is what, like I'm a journaling app. Right. And so this is a journal is corresponds to this data model I have.

2:02And like, if you want to create the journal, it's like basically like a template. And then like the developer, all they have to do is kind of glue in the, the code for actually like executing that template. From the user perspective, like let's say a developer implements those, that, those data points, that schema from one of those categories. Yeah. The ideal is that then the user can ask the voice assistant, the new one, Siri AI, what did, what was my calorie goal yesterday? And like in a perfect world, it would know that I use food noms and it should look at that schema and pull the number and just tell me, but it's maybe not that simple in actual, the implementation.

2:45Right. Is that right?

Missing schemas and shoehorning app features

2:47I mean, I mean, part of the, part of the issue is that like, there is no food tracking schema. So in order for me to do anything, I have to say like, oh, I'm actually like a journal app or like a calendar app, something that I'm not really. And then like, I have to like basically shoehorn my actions into like whatever actions are available. And there's not really many. It's like basically crud, create, read, update, delete. Those are like what you have, what you have at your disposal. And even beyond that, like say there's something specific, like, well, for food noms, it's like, there's a meal type, right?

3:24Breakfast, lunch, dinner. That's like additional metadata that goes on to like, if let's say I use the journal entry, like that's not, Apple doesn't have meal types for journals. So like that sort of metadata just gets left out. So for me, there's not really a great option, unfortunately. And I think Apple's like answer to that today is like, well, we're going to be expanding and adding more schema domains in the future. And so I just have to have kind of hope for that.

3:57There is one other kind of solution that they have told me in the past is like the way to go. If you have an app that doesn't work with a specific domain, there is something called app shortcuts. Curious, do you know what app shortcuts are? Well, so my mind goes to like the shortcuts app, but I assume that's not it. And I think that's what like in my video, when I was trying to get information from the flighty app, I got this weird prompt from Siri AI that said, do you want to enable shortcuts?

4:30And even as a pretty, I would say, power user of shortcuts, I didn't really know what that meant. And I assumed like, is this describe a shortcut? Is this about to vibe code a shortcut for me? But it seemed like that is not the case. Instead, it was enabling some mechanism and then immediately showed me flight information. So I guess I'm wrong that it is not like the shortcuts we all know and love. What is that? Well, it is sort of related. Okay. App shortcuts, I think actually are really cool. And you've probably used them before.

5:01I mean, I know you know the shortcuts app pretty well. If you, if you go like to the shortcuts app and you kind of like, you scroll down to the app section, right? And you just select an app. There's like those defaults that just come like kind of pre-installed, right? And you can just like one tap. You don't have to make a custom shortcut for them to work. They just work kind of out of the box, right? Those are the app shortcuts. So that's like another API. And they've actually existed for quite a while. And they're actually, I love the concept.

5:32It's actually in my mind, like kind of a very elegant design. And it's pretty simple to set up too as a developer. So you have like your, your app intents, like your shortcuts. You know, if you support shortcuts as a developer, you have like all those actions that you can offer to your users. An app intent, I mean, an app shortcut basically wraps an app intent and kind of like simplifies it. And then you add in like some, some phrases and those phrases, you just list out like a series of sort of like things someone could say to Siri and it would invoke that app intent.

6:09So for food noms that, that opens up, I have like, I think 10 or so different app shortcuts. There is a limit on how many you can offer, but you know, there's things like check my food noms protein goal or like log water in food noms. Because if you have water as like a favorite, then you can do that. Or you can say, copy my lunch from yesterday to today or open the food noms scanner or like there's, these are all these like really rich capabilities that. And I do think that some of the developers who are claiming serious AI support, they are kind of talking about app shortcuts because they do work, but not reliably.

6:53And I think I saw that in your video that like, it kind of depends on a series of factors and it, from my investigation, it comes down to just like a handful or more of like, just honestly bugs in the operating system that's causing these to not work reliably. So that brings me to my next question. So I think I understand that. And for like flighty, when I said, yes, use the app shortcuts, now it can show me my flight. But there were a number of instances where I would ask it something that I know, sometimes it'll show me the app result.

7:28And like one is like Lumi, L-U-M-Y, which will do things like tell you about golden hour, what time that is, the moon phase. And if I ask Siri AI, tell me what golden hour is from Lumi, or I've seen other guidance where you say the words, ask Lumi, like I'm telling Siri AI that, and it should reach into the Lumi app and tell me that. It's like 50-50. Sometimes I'll see the little Lumi app logo in the Siri AI result, and it will tell me what the Lumi AI is saying, or Lumi app is saying.

7:58And the other half of the time, it'll give me an answer, but the sources are clearly like web search, or like the general world, broad world knowledge that Siri AI has. In your estimation, like what is happening there?

Operating system layers and reliability failures

8:11Is it that Siri AI is trying to interpret the request? And even if I say the app name, it's like doing something different? Like what do you think is happening there? From what I can tell, and it is kind of hard, I'm having to look at like the logs of the operating system. So I connect my phone, and like I stream all the logs of the whole OS, and I'm doing this with Claude is helping me, because it's just like so many logs. It helps like to filter it out and kind of make sense of it all. Honestly, like my kind of general take on it is that Siri AI, and like when you interact with Siri, like you're still interfacing with like, there are so many layers in the OS.

8:50And these, like some of these layers, I think are really, really old. Like it's not like they like completely started fresh with Siri, with Siri AI. There are still so many like frameworks and like demons and things going on that are like trying to process that request. And it's just like there's so many failure points along that way. So depending on like, yeah, like I've noticed differences from if you like type it versus if you say it, because like there's like that's like one, you know, like part of the pipeline.

9:23It's just like there's different failure points, in my opinion, like bugs along the way to processing those requests. And it's just, yeah, unfortunately, unreliable. So sometimes it it doesn't it like I don't know why. Like if you explicitly say the name, I don't understand why it's not like, like, that's pretty clear cut, like we should handle this with the the associated app shortcut, like it should work that way. For some reason, it doesn't.

9:54There's just too many failure points. And that's something where I've I've noticed with notes, which is clearly something that Siri AI can use third party apps with Jason Aten, my coast on this show contextually, a lot of times I can ask it to do something contextually, it'll do it with bear notes. Again, it's like 75% of the time, if I say, create a bear note with this information, it'll do it. If I ask, tell me about John Ternus, Apple CEO, it gives me a Siri result with world knowledge. And then I say, add this to a bear note or create a new bear note with this information.

10:27It'll just create an Apple note. And like there's not like I can't I can't get it to like shove it into bear instead of notes. And then sometimes it'll ask me. And so sometimes I will say, like, add a new reminder to take out the trash at 8 p.m. And sometimes it'll just create a reminder. Sometimes it will give me a list of 10 apps that I have on my phone, like tasks, can do things, the stock reminders app. And I have to pick one. And I guess and you tell me if Apple has instructed developers in this way. Is Siri AI supposed to learn my habits over time so that if I always choose can do for my tasks that a month from now, after I've done dozens of requests, it'll stop asking me and it'll just default to that third party app?

11:13Or is that is that an unknown process? I don't know. I don't know. OK. So there's not any guidance from Apple of like tell your users do this or why? No, I mean, I haven't heard anything. But, you know, I may be missing something, to be honest. I actually haven't like so with the with the Siri domain stuff that I was talking about. And that is like I think the the apps like Bear and like CanDo, right, like that you're able to take advantage of those.

11:43Like I wasn't able to. So I didn't spend as much time like investigating and like learning about all the intricacies of that. And and so for that, like that example, you know, that might be a gap in my knowledge. Or I don't know. I wouldn't be surprised if it's just an omission or like Apple hasn't been clear about like exactly how that should work. Like I think generally they try to avoid committing to certain behaviors. Right. And yeah, to the to like to their credit, I imagine like because Siri is doing so much now, like it takes a week to index your phone and it might it's reaching your personal context and world knowledge and third party apps.

12:22Like it's complex, you know, what it's all trying to do. Do you does it feel like maybe some apps like you're saying didn't fit nicely into one of those 12 schemas for Siri AI, like journaling, tasks, notes that some are trying to use those app shortcuts to kind of say they have Siri AI support. But that is why some users might get a mixed experience because it's actually not using the proper schema and Siri AI then can't really parse that this request is for this kind of app.

12:53Is that fair to say? Yeah, likely. Yeah. Okay. All right. I think, yeah, there's there's other issues. Like I think you've pointed out, like even when you have a schema, there's like maybe some bugs that they need to work out or some limitations that are causing like potential user confusion or bad experiences. And I think I did see some of those in your video. But yeah, I think there is general confusion when like an app says they have Siri AI support. And instead, it's really app shortcuts that happen to be working well for them.

13:27You know, there's some app shortcuts for me that just like happen to work better than others. And I I don't exactly know why it's something about I think the natural language processing and like the way the phrase is constructed, like it just trips up Siri. And again, like there's like multiple layers of processing and like maybe like flighty, you know, I think you could probably engineer like and try to figure out the perfect phrase that has like a high success rate. Like just like the magic incantation. And yeah, I guess the risk there is that like a user then like has to remember that incantation for it to work reliably.

14:04And if they start like to like say it slightly differently or ask for a different like feature in Siri AI, they may get something like totally unexpected. Right. Like a web general knowledge response. And I actually use those words exactly. My video was like, I guess I got to find the magic words because sometimes like if I find a phrase like sometimes saying ask flighty about my next trip or ask Mercury weather about my next trip, it would work consistently. But if I asked what's my next trip in Mercury weather, it'd be like, no, sorry, you know, and it's one of those things where the promise is supposed to be natural language.

14:38And that's always where Apple has tried to err on where it will take kind of general broad phrases and still try to get up where you're going. But when you replied to my video on Macedon, I mentioned I asked just to see if Food Noms was in Siri AI because I assumed like, oh, maybe it'll it'll have a feature. And of course, you did not advertise that you have Siri AI support. But you said in our thread, here's what you would hope to happen. What do you what has to happen or change for you to feel comfortable to say Food Noms has Siri AI?

15:09Is it those schema that actually works with food tracking type apps specifically? Is it more predictable commands and controls? What are you looking for as a developer?

Open ended MCP support and visual intelligence

15:19What would be completely ideal would be something completely open ended, like true MCP style support. That would be best case scenario. And because like I offer like Food Noms has an MCP beta right now. And so like you can connect Claude to your Food Noms data. And, you know, people are able to do really cool stuff with it, like stuff I hadn't even thought of, even though like it's the same sort of primitives that I offer with shortcuts, you know, like being able to like find your food logs and like process them.

15:50And then like, but like because Quad, you know, is not restricted to certain domains, right? Like that would be ridiculous. Quad MCP is like open ended. It's able to like just do whatever the user like dreams of and like there's no restrictions on how the user requests it. Like the LLM knows how to handle the request properly. That would be like best case scenario. And I don't know, maybe someday that that comes to fruition. And I think in the short term, what I would love to see is, you know, I have, like I said, I've built out these app shortcuts, which I really do think are like genuinely valuable.

16:29I'd love to be able to just ask Siri anywhere, like how am I doing on my protein goal? You know, how many, how many grams of protein do I need to eat today to hit my goal? You know, like also just being able to log, you know, stuff like really easily, at least my favorites, right? If I, if I set up, you know, my snacks that I tend to eat or like my, my favorite meals to just be able to ask Siri to log them. You know, that's basically already there, just waiting to be used. And it's, you know, I built that like a year and a half ago, two years ago, and I've been like waiting for Apple to like kind of fix these issues.

17:04And I was like, just frankly, kind of disappointed that like, I was hoping with this big Siri AI relaunch that they would like make that more reliable. And two is such an opportunity, right? Like, cause I write out those phrases about like the incantations, you know, you can imagine an LLM can just like basically learn from those phrases and be able to like turn it into like any sort of like related query gets automatically funneled to that app shortcut.

17:34And it would just work. And I really think that app shortcuts are just a great solution for them in terms of like the long tail too. Like they're never going to be able to provide like domains for every single kind of app. So like, but every app can create custom intents and then app shortcuts on top of it. And then, you know, that handles again, the long tail of any sort of feature from any sort of app. That would be, yeah, my, that's my hope is that, you know, they, they try to prioritize that.

18:05But yeah, I don't know. So like ideal, ideal world is like MCP for Siri. That's like kind of ideal. And then, you know, some other stuff's underneath. Have you played around with visual intelligence and third-party apps? Because over the last year, I've been seeing more and more apps show up like when I take a screenshot or when I do visual intelligence, like Amazon's in there now. Apps like Bezel have taken advantage of that. Etsy, where it'll basically do like a reverse image search. And you can search for a product and do obviously the reverse Google image search.

18:38But as a third-party app, is there even an option right now where if I go to straight to visual intelligence, could I go to then food noms and send the picture I just took to food noms rather than Etsy or Amazon? Or is that, those aren't really tools that are built in yet for you guys? That's, that's another thing that would be cool. That would be cool. Okay. I think you kind of like described it well in that like I feel like Apple has designed that feature to be intended for apps to do something like reverse image search. And the UX and the UI is like designed that way in that usually there's like more than one result.

19:14So they have like kind of the grid view. And I think there's kind of the expectation that the results are processed fairly quickly, like probably within like half a second or so. And for me with AI processing, you know, queries can take generally like five to 10 seconds. And I don't know, it's just kind of seems like Apple hasn't really quite intended like certain use cases like, you know, food noms AI with visual intelligence. But some like small tweaks there would definitely like, I think, make that a lot more, you know, like appropriate sort of use case.

19:48That would be cool. Well, Ryan, thanks so much for your time. Food noms, everyone should go download it. My wife loves it. I've been using, we've both been using it for the last year, pay for premium. It is a beautifully designed app. You do incredible work. And the AI stuff is amazing. Like I took a picture of three street tacos and it just, it gets it, you know what I mean? And the results over the past year, it's been very accurate. And so thanks for your time on this and be curious to follow, we'll be following your journey if you can implement more Siri AI in the future and what Apple does.

20:18So thanks for being here, Ryan. Yeah, of course. Thanks, Stephen.