Yuino: OSS kana-kanji conversion system that combines LLM and a dictionary-based approach

Yuino is an OSS Kana-Kanji conversion system that integrates LLMs with a dictionary-based approach.

The LLM-based Kana-Kanji conversion systems have been an actively developing topic in the OSS community in recent years. That representative example is azooKey. It achieves high-quality conversion accelerated by an LLM. On the other hand, it has a clearly weak point. This weak point is 'How to handle new words from context'. The existing method doesn’t do not use dictionaries tied to the LLM. They use a tokenizer before LLM inference, and their behavior is entirely dependent on the training data. One of the example of new words is "Reiwa". "Reiwa" is an era name in Japan. It was created in 2018. The tokenizer splits dictionary words into multiple tokens. It loses the original meaning and part-of-speech (POS) information. As a result, these cases negatively impact the inference process for Kana-Kanji conversion.

To solve this issue, I propose an LLM extension method. This method bypasses the tokenizer and modified the LLM to accept inputs while preserving both the word's form and its POS information. After this modification, we can input feeding word data with attached POS tags directly into the LLM. When we use the new words in input, the system can derive conversion results that accurately reflect the intended usage. Yuino is the world first implementation of my proposed method. Attendees will learn about my proposed method deeply. The implementation of Yuino is a Japanese-specific implementation. On the other hand, the method is possible to reuse other languages in an ideogram user. Because my proposed method is common sense.