Yuino: OSS kana-kanji conversion system that combines LLM and a dictionary-based approach
- Time
- 2026-08-09 13:00 ~ 13:30
- Speaker
- Masahiko Hashimoto
- Room
- TR410
- Co-write
Abstract
Yuino is an OSS Kana-Kanji conversion system that integrates LLMs with a dictionary-based approach.
The LLM-based Kana-Kanji conversion systems have been an actively developing topic in the OSS community in recent years. That representative example is azooKey. It achieves high-quality conversion accelerated by an LLM. On the other hand, it has a clearly weak point. This weak point is 'How to handle new words from context'. The existing method doesn’t do not use dictionaries tied to the LLM. They use a tokenizer before LLM inference, and their behavior is entirely dependent on the training data. One of the example of new words is "Reiwa". "Reiwa" is an era name in Japan. It was created in 2018. The tokenizer splits dictionary words into multiple tokens. It loses the original meaning and part-of-speech (POS) information. As a result, these cases negatively impact the inference process for Kana-Kanji conversion.
To solve this issue, I propose an LLM extension method. This method bypasses the tokenizer and modified the LLM to accept inputs while preserving both the word's form and its POS information. After this modification, we can input feeding word data with attached POS tags directly into the LLM. When we use the new words in input, the system can derive conversion results that accurately reflect the intended usage. Yuino is the world first implementation of my proposed method. Attendees will learn about my proposed method deeply. The implementation of Yuino is a Japanese-specific implementation. On the other hand, the method is possible to reuse other languages in an ideogram user. Because my proposed method is common sense.
Speaker
Masahiko Hashimoto
Masahiko Hashimoto is an open-source software (OSS) enthusiast with a deep passion for Linux and programming. An avid user of openSUSE, he writes articles exploring "slightly unusual ways" to use it. He self-publishes these articles as doujinshi (fanzines) and distributes them at events such as Comic Market. He is also an active member of the "Tokaido LUG" (Linux User Group) community. He frequently attends offline meetups held all over Japan, where he enjoys engaging in lively discussions about Linux with fellow members.
Professionally, He works in AI development, focusing specifically on "Federated Learning"—a machine learning technique that trains AI models across networks. As a result of his research, he has developed a federated learning framework and released it as OSS. In the future, he aims to build a system where an open-source Kana-Kanji conversion engine (IME) continuously evolves every day through federated learning.