Dictionary pipeline

Every completion and autocorrect suggestion the keyboard makes ultimately reads from a .wmdict file: a memory-mapped, frequency-ranked trie compiled from a plain-text wordlist. This page covers the compiler that builds them, the format itself, where the wordlists come from, and how to add or improve one.

If you’re looking for the user-facing side of this (the per-language download screen, size tiers, and custom-dictionary import), see Downloadable dictionaries. This page is about the pipeline behind it.

From text file to binary trie

A .wmdict starts life as a plain-text wordlist: one entry per line, word<space>frequency. Lines starting with # and blank lines are skipped; anything else still becomes an entry, and a line with no space (or whose trailing token isn’t a number) defaults to frequency 1 rather than being dropped. This is DictionaryLoader’s format, and both the bundled dictionaries and user-imported custom dictionaries parse a file exactly this way. The downloadable catalog’s own streaming parser (below) follows the same word<space>frequency convention but is a separate implementation with its own, stricter filtering rather than a call into DictionaryLoader.

tools/dictc is the compiler: a small JVM command-line tool, dictc <srcDir> <outDir>, that reads every *.txt in srcDir and writes a matching <name>.wmdict to outDir. Its own code is just that Main.kt entry point. The trie-building logic isn’t a separate reimplementation: build.gradle.kts adds core/prediction’s source directory as an extra Kotlin source directory (Gradle’s srcDir, not a symlink) and filters it down to the four files it actually needs (Trie.kt, PackedTrie.kt, PackedTrieCodec.kt, DictionaryLoader.kt) so the writer and the runtime reader are literally the same code. The build’s own comment on this: “one implementation, so the emitted bytes can never drift from what MappedTrie/PackedTrieCodec read.”

You never invoke dictc by hand for a normal build. app/build.gradle.kts registers it as the compileBundledDictionaries task, which runs dictc over app/dictionaries-src/*.txt via javaexec and wires its output into every build variant’s generated assets: both full and lite, since the dictionary source doesn’t change between flavors. The result lands under the APK’s assets at dictionaries/<name>.wmdict, which is what DictionaryStore opens by name at runtime. See building from source for the rest of the build wiring.

Only two languages are bundled this way today: app/dictionaries-src/en.txt (17,217 word entries) and bn.txt (20,645 word entries). To add a word or adjust a frequency in either one, edit the .txt file directly and rebuild: no code changes needed, no manual compile step.

The .wmdict format, briefly

PackedTrieCodec defines the on-disk layout: an uncompressed, big-endian image of a compressed-sparse-row (CSR) trie, magic "WMDC", version 1. The header carries word/node/edge counts and six section offsets (childStart, edgeLabel, edgeChild, freq, maxSubtree, isWord), each section 4-byte aligned. Every node’s outgoing edges are a contiguous, label-sorted slice of two parallel arrays, so a lookup is a binary search rather than a hash lookup, and maxSubtree (the highest word frequency anywhere below a node) is precomputed at write time so completion can prune the search instead of walking every candidate.

MappedTrie is the reader: it mmaps the file read-only and does every lookup as a raw ByteBuffer read against the mapped pages: nothing is copied to the Java heap, so “loading” a dictionary is a single mmap call, and pages your typing never touches are never read at all. MappedTrie.open() returns null silently on a missing, truncated, or wrong-version file rather than throwing, so a corrupt or half-written download just falls back to whatever other word sources are available instead of crashing the keyboard.

The format itself carries no compression: that happens one layer up. The APK deflates bundled .wmdict assets the ordinary way, and the download pipeline inflates a .gz wordlist before writing it into this format, so a .wmdict on disk is always raw.

Full byte-level layout: file formats reference.

The downloadable catalog

Beyond the two bundled languages, DictionaryCatalog lists every wordlist available from the companion wmkeyboard-data repository: currently 333 entries covering 332 languages (English and Bengali also have downloadable, larger replacements for their bundled lists; Portuguese has two entries, European and Brazilian, that share one download slot). Each DictionaryEntry carries a language id, a repo code, the full list’s word count and compressed size (both display hints, not checksums), and an optional rom suffix for one of the 14 romanized/Latin-script variants (Bengali, Hindi, Arabic, Russian, and others). A handful of language codes are remapped between the data repo’s naming and the app’s own registry (roa_rup→rup, mhr→chm, bxr→bua, nrm→nrf), and both Portuguese entries feed the single pt registry id.

Users pick a size tier before downloading: Small (50,000 words), Medium (150,000, the default), or Large (300,000), and because the source files are pre-sorted by descending frequency, WordlistDownloadManager can stop reading the stream as soon as it’s kept enough words rather than transferring the whole file. That’s what makes a 41 MB compressed Thai list a fast Medium download: it stops well short of the end. While parsing, it also drops any line with frequency below 2 (sorted-descending, so hitting one means only noise follows), skips words over 48 characters or containing a space, and does a pre-flight free-space check with an 8 MiB margin before starting. There’s no HTTP range/resume support by design: a capped download is cheap enough to just restart. The pipeline itself is stream → inflate → parse the first N frequency-sorted lines → build a PackedTrie → write main.wmdict.part → atomic rename to main.wmdict, so a file only exists on disk once it’s completely valid.

On disk, bundled and downloaded dictionaries live under filesDir/dict/ (dict/bundled/<name>.wmdict for the two inflated-from-APK lists, dict/<langId>/main.wmdict for a downloaded one), deliberately separate from filesDir/dictionaries/, the user-imported custom-dictionary tree covered below. At query time, CompositeWordSource merges whichever of these are present for a language; if a word appears in more than one source, it keeps its highest frequency across all of them.

Adding or improving a wordlist

There are three distinct paths, depending on which dictionary you mean:

1. The two bundled lists (English, Bengali). Edit app/dictionaries-src/en.txt or bn.txt directly (same word<space>frequency format as everywhere else) and rebuild. compileBundledDictionaries regenerates the .wmdict automatically; there’s no separate compile step to remember.
2. The 332-language downloadable catalog. This data lives in the separate wmkeyboard-data repository, not in this codebase: DictionaryCatalog.kt only holds the metadata table (id, repo code, sizes) describing it. Regenerating that table against a fresh repo checkout is a manual/scripted process outside this repo.
3. Any other language, without touching either repo. A user can import their own word list (a Hunspell .dic, a frequency list, or a plain word column) from WM Keyboard›Typing›Custom dictionaries (also reachable from a language’s own Dictionary screen). Files land at filesDir/dictionaries/<langId>/<name>.txt, capped at 32 MiB each, and stack additively: several imported lists can sit under one language, and for English and Bengali they layer on top of the bundled list rather than replacing it. This is the fastest way to get real completions for a language with no bundled or downloadable dictionary; see Downloadable dictionaries → Options for the user-facing details.

There’s no CONTRIBUTING.md or issue template in this repo yet. See contributing for what does exist around proposing a change.

Emoji keyword pack generation

Emoji search and suggestions have their own, separate per-language dictionary pipeline: same shape, different codec (EmojiDictCodec, not PackedTrieCodec; see Emoji customization for the user-facing download UI). Two scripts in tools/emoji/ generate that data:

• export_keyword_pack.py converts a CLDR language’s hand-translated emoji annotations into the app’s importable keyword-pack TSV format (emoji<TAB>keyword,keyword,...<TAB>name), cross-referencing the app’s own emoji catalog so it only emits entries for emoji the app actually carries. Because CLDR already has annotations for roughly 100 languages, producing a pack for any of them is a format conversion, not a translation job.
• generate_dict_catalog.py regenerates EmojiDictCatalog.kt from the wmkeyboard-data repo tree, walking it for data/<code>/<code>_emoji.json.gz files and writing the Kotlin table between two markers (--check verifies without writing; --tree reads a saved API tree to dodge GitHub rate limits).

The generated catalog currently holds 125 entries (down from the repo’s 141 language packs), after applying three filters: codes the repo spells differently are remapped (no→nb); exact duplicates of a plainer code already present are dropped (fil==tl, pt_br==pt, zh_cn==zh); codes with no matching app language are dropped (blo, bs, ccp, quc, rhg, zh_tw); and seven near-empty upstream stubs carrying 1–16 emoji each are dropped (bgn, ceb, ckb, mni, su, syr, vec).

Two other scripts in the same directory, generate_catalog.py and generate_gemoji.py, build the base emoji catalog and shortcode tables (the emoji palette itself, and its :shortcode: triggers) rather than per-language dictionaries: a different job from anything on this page.

Details & edge cases

• The compiler is shared code, not a reimplementation. Beyond its own Main.kt entry point, tools/dictc has no trie-building code of its own: its build points straight at core/prediction’s source directory (an extra Gradle source directory, not a symlink) for the exact Trie/PackedTrie/PackedTrieCodec/DictionaryLoader classes the app runs, so a bug fix or format change to the runtime reader applies to the compiler automatically, and the two can never silently drift apart.
• One compile task feeds both flavors. compileBundledDictionaries isn’t per-variant: it runs once and its output is wired into every build variant’s generated assets, since full and lite read identical dictionary data.
• A corrupt or partial .wmdict degrades silently. MappedTrie.open() returns null on a bad magic number, wrong version, or truncated file instead of throwing, so a failed or interrupted download just leaves that word source empty rather than crashing the input method.
• Downloaded and custom dictionaries are additive, never exclusive. They live in separate on-disk trees (filesDir/dict/ vs filesDir/dictionaries/) and CompositeWordSource reads from all of them at once for a language, keeping each word’s highest frequency across sources.
• Direct boot only ever sees the bundled pair. The dict/bundled/ copies are extracted from the APK into device-protected storage (nothing user-specific is exposed by that), so they’re available before the user unlocks the device. Downloaded and custom dictionaries live behind the credential and simply aren’t there yet in that window. See architecture → direct boot for the fuller picture.
• Catalog sizes and counts are hints, not checksums. DictionaryCatalog’s word counts and compressed sizes are display/progress values generated against a point-in-time snapshot of the data repo; drift against a newer repo state is expected and harmless.