[BoF] What Is Local AI Actually For?

Everyone is saying "AI" right now. Fewer people are saying whether it actually made their work better. I build this stuff for a living — an on-device AI appliance around a Raspberry Pi 5 + AI HAT+, and mobile apps running speech and language models locally: sherpa-onnx, whisper.cpp, llama.cpp, Gemma, Style-Bert-VITS2. Not in the cloud, on the device, for people whose data shouldn't leave it. Mostly what I've learned is how it breaks. Small models forget what you just told them. Latency you can feel. Demos that impress someone once and never get opened again. So this is not a talk with answers. It's a room for people actually shipping this stuff to compare notes and be honest: Where has local AI genuinely made something better — and where did we just bolt a chatbot onto a thing that was fine?

What's still missing? Models, tooling, hardware, UX patterns. What would you build if someone else built the piece you keep rewriting?

Real constraints: what did you give up to run on-device, and what turned out not to matter? We'll run it Lean Coffee style — everyone writes down the question they came with, we vote, and we work down the list. No conclusions required. "It'd be nice if this existed" is a perfectly good outcome. A Raspberry Pi 5 will be running fully offline on the table the whole hour, so when the conversation drifts too abstract we can go poke at something real. Anyone welcome: edge AI, on-device LLM / ASR / TTS, low-resource languages, hardware — or you just have a strong opinion about whether any of this is working yet. Newcomers especially welcome. The questions are the point.

大家都在講 AI，但很少人講它到底有沒有讓工作變好。 我平常做的是端上的 AI： Raspberry Pi 5 + AI HAT+ 的裝置，以及把語音與語言模型跑在手機本機的 app （sherpa-onnx、whisper.cpp、llama.cpp、Gemma、Style-Bert-VITS2）。做下來學到最多的 其實是它怎麼壞——小模型記不住剛講過的話、延遲感覺得出來、demo 驚艷一次之後再也 沒人打開。 這場不是有結論的演講，而是想找同樣在做的人誠實聊聊：哪些地方地端 AI 真的讓事情 變好了？哪些只是硬加了一個 chatbot？還缺什麼模型、工具、硬體或 UX pattern？我們 用 Lean Coffee 的方式進行，各自把帶來的問題寫下來、投票、依序討論。不需要結論， 「有這個就好了」也是很好的收穫。桌上會一直放一台完全離線運作的 Raspberry Pi 5， 聊到太抽象的時候可以回來戳戳實體。歡迎做邊緣 AI、端上 LLM/ASR/TTS、低資源語言、 硬體的朋友，或單純對「這些到底有沒有用」有意見的人。新手非常歡迎。 Language: English (Mandarin / Japanese also fine)