# Dictionary pipeline > How wordlists become .wmdict binaries, and how to build your own. URL: https://wmkeyboard.pages.dev/development/dictionaries/ Section: Development The dictionary behind completion and autocorrect is a `.wmdict` file: a memory-mapped, frequency-ranked trie compiled from a plain-text wordlist. It ships in the APK, every download produces one, and imported custom lists become one too: the keyboard compiles a language's switched-on lists into a single `.wmdict` under `dictionary_cache/`, maps it like a download, and compiles again only when a list's name, size or modification time changes. The learned lexicon is the one word source that stays JSON. This page covers the compiler that builds them, the format itself, where the wordlists come from, and how to add or improve one. For the user-facing side of this (the per-language download screen, size tiers, and custom-dictionary import), see [Downloadable dictionaries](https://wmkeyboard.pages.dev/languages/dictionaries/). This page is about the pipeline behind it. ## From text file to binary trie A `.wmdict` starts life as a plain-text wordlist, one entry per line, `wordfrequency`. Lines starting with `#` are skipped, and so are blank lines. Anything else becomes an entry. A line with no space, or whose trailing token isn't a number, defaults to frequency `1` rather than being dropped. That's `DictionaryLoader`'s format, and both the bundled dictionaries and user-imported custom dictionaries parse a file exactly this way. `DictionaryLoader` reads a second format as well, decided once on the file's first meaningful line: the AOSP `.combined` wordlist, whose `dictionary=` header is followed by indented `word=the,f=222,flags=` records. Its 0-255 frequencies are scaled onto this app's 0-10000 range by `scaleAospFrequency`, and `bigram=`/`shortcut=` records are skipped there. They are read at import instead, by `AospCombined`. Before that sniff existed a combined list wasn't rejected, it was *accepted wrongly*: a record line holds no space, so `word=the,f=222,flags=` became a single word at frequency 1. The compiled side of that ecosystem is `AospDictionary` (version 2, one file, with its `codePointTable` applied) and `AospDictionaryV4` (the `.header` plus `.body` pair a keyboard writes on the device, such as a user history). All three readers return the same `Contents`: words, word pairs, shortcuts. `CustomDictionaries.inspect` reports what a pick holds and `write` stores the parts the user chose: the words as this text format, so nothing downstream needs a second reader; the pairs as a `.wmng` pack beside the list, which `NgramPack` reads together with the downloaded pack (the stronger count wins, counts are never summed); and the shortcuts as a `.shortcuts` file of `triggerexpansion` lines, which the IME offers the way it offers the personal dictionary's. Both side files are read only while their list is switched on. `AospScores` holds every conversion from AOSP's 0-255 log scale and from a history's raw counts. The downloadable catalog's own streaming parser (below) follows the same `wordfrequency` convention, but it's a separate implementation with its own stricter filtering rather than a call into `DictionaryLoader`. `tools/dictc` is the compiler: a small JVM command-line tool, `dictc `, that reads every `*.txt` in `srcDir` and writes a matching `.wmdict` to `outDir`. Its own code is only that `Main.kt` entry point. The trie-building logic isn't a separate reimplementation. `build.gradle.kts` adds `core/prediction`'s source directory as an extra Kotlin source directory (Gradle's `srcDir`, not a symlink) and filters it down to the seven files it actually needs: `Trie.kt`, `TrieWalker.kt`, `TrieCompleter.kt`, `PackedTrie.kt`, `PackedTrieCodec.kt`, `RankFloorCache.kt` and `DictionaryLoader.kt`. So the compiler and the app write files with literally the same code. The build's own comment on this: "one implementation, so the emitted bytes can never drift from what `MappedTrie`/`PackedTrieCodec` read." The reader is a separate implementation, in Rust (below), held to the writer by a parity test. You never invoke `dictc` by hand for a normal build. `app/build.gradle.kts` registers it as the `compileBundledDictionaries` task, which runs `dictc` over `app/dictionaries-src/*.txt` through `javaexec` and wires its output into every build variant's generated assets. That includes both language builds, since the dictionary source doesn't change between them. The result lands under the APK's assets at `dictionaries/.wmdict`, which is what `DictionaryStore` opens by name at runtime. See [building from source](https://wmkeyboard.pages.dev/development/building/#details--edge-cases) for the rest of the build wiring. Only two languages are bundled this way today: `app/dictionaries-src/en.txt` (17,306 word entries) and `bn.txt` (20,645 word entries). To add a word or adjust a frequency in either one, edit the `.txt` file and rebuild. No code changes, no manual compile step. Every number in `en.txt` sits on one scale: `973.83 × ln(count / 1000) + 1`, on the OpenSubtitles 2018 counts in `data/en/en_full.txt.gz` of the data repository, so a word seen 1,000 times is 1 and the commonest word is 10,000, the top the engine's weights were tuned against. It wasn't always. The list grew in three bands that each had their own numbering: a seed head ranked 10,000 down to 1 by position, with its last few hundred words in alphabetical order; a handful of hand additions slotted between them; and the corpus band. `meet` ended up at 300 under `met` at 1,757, and `wait` at 8. `scripts/rescale_en_dictionary.py` rebuilt the numbers from the pinned list before that change and the pinned reference, both checked by hash, and `--check` confirms the file still matches. Contractions are carried across from their old place instead, because the reference splits words at the apostrophe. A word you add by hand should be numbered on the same line: look its count up in the reference and apply the formula. ## The .wmdict format, briefly `PackedTrieCodec` defines the on-disk layout: an uncompressed, big-endian image of a compressed-sparse-row (CSR) trie, magic `"WMDC"`, version `3`. The header carries a flags word, word, node and edge counts and a symbol count, plus seven section offsets (`symbols`, `childStart`, `checkpoint`, `edgeLabel`, `freq`, `maxSubtree`, `isWord`), each section 4-byte aligned. Every node's outgoing edges are a contiguous, label-sorted slice of the `edgeLabel` array, so a lookup is a binary search rather than a hash lookup. There is no `edgeChild` array: version 3 numbers nodes breadth-first so edge `e` always leads to node `e + 1`, which is what version 1 was storing an identity function for. Two flag bits say how a given file spelled its labels and its child counts, since a Hangul or Japanese list blows past the one-byte encodings an alphabetic script fits inside. `maxSubtree`, the highest word frequency anywhere below a node, is precomputed at write time so completion can prune the search instead of walking every candidate. `MappedTrie` is the reader. Word lookups, completion and the full word list run in Rust, in `native/wm-engine`'s `wmdict` module, which WMK Desktop uses to read the same files: the library maps the file read-only and reads the bytes where they are. The node-by-node walks of the correction search and glide typing still read a second, Kotlin mapping of the same file through `ByteBuffer` gets, until those walks move to Rust as well. Either way the trie never reaches the Java heap, so "loading" a dictionary costs a `mmap` call, and pages your typing never touches are never read at all. `NativeReaderParityTest` holds the Rust reader to `PackedTrie`, the in-memory trie the writer serialises, on the bundled lists and on downloaded ones: the same words, frequencies and completion order, ties included. The `.wmng` n-gram packs work the same way, through `MappedNgramPack` and the crate's `wmng` module. `MappedTrie.open()` returns `null` silently on a missing, truncated or wrong-version file rather than throwing, so a corrupt or half-written download falls back to whatever other word sources are available instead of crashing the keyboard. The Rust engine crate carries the same format, builder, writer and reader (`native/wm-engine/src/wmdict.rs`), and its tests compare the writer's bytes with the dictionaries the Android build compiles, so WMK Desktop reads a phone's `.wmdict` unchanged. The n-gram pack has its twin in `wmng.rs`. The format itself carries no compression. That happens one layer up: the APK deflates bundled `.wmdict` assets the ordinary way, and the download pipeline inflates a `.gz` wordlist before writing it into this format. A `.wmdict` on disk is always raw. Full byte-level layout: [file formats reference](https://wmkeyboard.pages.dev/reference/file-formats/). ## The downloadable catalog Beyond the two bundled languages, `DictionaryCatalog` lists every wordlist available from the companion [wmkeyboard-data](https://github.com/wasi-master/wmkeyboard-data) repository: currently 431 entries covering 430 languages. English and Bangla also have downloadable, larger replacements for their bundled lists, and Portuguese has two entries, European and Brazilian, that share one download slot. Each `DictionaryEntry` carries a language id, a repo code, the full list's word count and compressed size (both display hints, not checksums), and an optional `rom` suffix for one of the 14 romanized/Latin-script variants (Bangla, Hindi, Arabic, Russian, and others). A handful of language codes are remapped between the data repo's naming and the app's own registry (`roa_rup`→`rup`, `mhr`→`chm`, `bxr`→`bua`, `nrm`→`nrf`), and both Portuguese entries feed the single `pt` registry id. Users pick a size tier before downloading: Small (50,000 words), Medium (150,000), Large (300,000, the default) or Everything, which takes the list whole however long it is. Tiers past the end of a short list all keep the same words, so the picker only offers the first one that reaches the whole list. The source files are pre-sorted by descending frequency, so `WordlistDownloadManager` can stop reading the stream as soon as it has kept enough words rather than transferring the whole file. That's what makes a 41 MB compressed Thai list a fast Medium download. It stops well short of the end. While parsing, it also drops any line with frequency below `2` (the file is sorted descending, so hitting one means only noise follows), skips words over 48 characters or containing a space, and does a pre-flight free-space check with an 8 MiB margin before starting. There's no HTTP range or resume support, by design: a capped download is cheap enough to restart. The pipeline is stream, inflate, parse the first N frequency-sorted lines, build a `PackedTrie`, write `main.wmdict.part`, then atomically rename to `main.wmdict`. A file only exists on disk once it's completely valid. On disk, bundled and downloaded dictionaries live under `filesDir/dict/`: `dict/bundled/.wmdict` for the two inflated-from-APK lists, `dict//main.wmdict` for a downloaded one. That tree is deliberately separate from `filesDir/dictionaries/`, the user-imported custom-dictionary tree covered below. At query time, `CompositeWordSource` merges whichever of these are present for a language. If a word appears in more than one source, it keeps the highest frequency across all of them. ## Adding or improving a wordlist There are three distinct paths, depending on which dictionary you mean: 1. **The two bundled lists (English, Bangla).** Edit `app/dictionaries-src/en.txt` or `bn.txt` directly, in the same `wordfrequency` format as everywhere else. Then rebuild. `compileBundledDictionaries` regenerates the `.wmdict` automatically, so there's no separate compile step to remember. 2. **The 332-language downloadable catalog.** This data lives in the separate `wmkeyboard-data` repository, not in this codebase. `DictionaryCatalog.kt` only holds the metadata table (id, repo code, sizes) describing it. Regenerating that table against a fresh repo checkout is a manual or scripted process outside this repo. 3. **Any other language, without touching either repo.** A user can import their own word list, whether a Hunspell `.aff` and `.dic` pair (unfolded into every word it spells by `Hunspell.kt`), a frequency list or a plain word column, from Typing / Suggestions / Custom dictionaries (also reachable from a language's own Dictionary screen). Files land at `filesDir/dictionaries//.txt`, capped at 32 MiB each, and they stack additively. Several imported lists can sit under one language, and for English and Bangla they layer on top of the bundled list rather than replacing it. This is the fastest way to get real completions for a language with no bundled or downloadable dictionary. See [Downloadable dictionaries → Options](https://wmkeyboard.pages.dev/languages/dictionaries/#options) for the user-facing details. There's no `CONTRIBUTING.md` or issue template in this repo yet. See [contributing](https://wmkeyboard.pages.dev/development/contributing/) for what does exist around proposing a change. ## Emoji keyword pack generation Emoji search and suggestions have their own per-language dictionary pipeline: same shape, different codec (`EmojiDictCodec`, not `PackedTrieCodec`). See [Emoji customization](https://wmkeyboard.pages.dev/emoji/customization/) for the user-facing download UI. Two scripts in `tools/emoji/` generate that data: - **`export_keyword_pack.py`** converts a CLDR language's hand-translated emoji annotations into the app's importable keyword-pack TSV format (`emojikeyword,keyword,...name`). It cross-references the app's own emoji catalog, so it only emits entries for emoji the app actually carries. CLDR already has annotations for roughly 100 languages, so producing a pack for any of them is a format conversion rather than a translation job. - **`generate_dict_catalog.py`** regenerates `EmojiDictCatalog.kt` from the `wmkeyboard-data` repo tree. It walks the tree for `data//_emoji.json.gz` files, taking the code from the file name (the folder is the code too, except for Banglish, filed as `data/bn/bn_rom_emoji.json.gz`), and writes the Kotlin table between two markers. (`--check` verifies without writing; `--tree` reads a saved API tree to dodge GitHub rate limits.) The generated catalog currently holds 126 entries, down from the repo's 142 language packs, after three filters. Codes the repo spells differently are remapped (`no`→`nb`). Exact duplicates of a plainer code already present are dropped (`fil`==`tl`, `pt_br`==`pt`, `zh_cn`==`zh`), as are codes with no matching app language (`blo`, `bs`, `ccp`, `quc`, `rhg`, `zh_tw`). Finally, seven near-empty upstream stubs carrying 1–16 emoji each are dropped (`bgn`, `ceb`, `ckb`, `mni`, `su`, `syr`, `vec`). The rest of the directory does a different job from anything else on this page: `generate_catalog.py` and `generate_gemoji.py` build the base emoji catalog and its `:shortcode:` triggers, `add_names.py` fills in emoji names, and `generate_animated.py` regenerates the list of emoji Noto ships an animated version of. ## Details & edge cases - **The compiler is shared code, not a reimplementation.** Beyond its own `Main.kt` entry point, `tools/dictc` has no trie-building code of its own. Its build points straight at `core/prediction`'s source directory (an extra Gradle source directory, not a symlink) for the exact `Trie`/`TrieWalker`/`TrieCompleter`/`PackedTrie`/`PackedTrieCodec`/`RankFloorCache`/`DictionaryLoader` classes the app runs. A bug fix or format change to the runtime reader applies to the compiler automatically, so the two can never silently drift apart. - **One compile task feeds both flavors.** `compileBundledDictionaries` isn't per-variant. It runs once and its output is wired into every build variant's generated assets, since every variant reads identical dictionary data. - **A corrupt or partial `.wmdict` degrades silently.** `MappedTrie.open()` returns `null` on a bad magic number, wrong version or truncated file instead of throwing, so a failed or interrupted download leaves that word source empty rather than crashing the input method. - **Downloaded and custom dictionaries are additive, never exclusive.** They live in separate on-disk trees (`filesDir/dict/` and `filesDir/dictionaries/`), and `CompositeWordSource` reads from all of them at once for a language, keeping each word's highest frequency across sources. - **Direct boot only ever sees the bundled pair.** The `dict/bundled/` copies are extracted from the APK into device-protected storage, which exposes nothing user-specific, so they're available before the user unlocks the device. Downloaded and custom dictionaries live behind the credential, so they aren't there yet in that window. See [architecture → direct boot](https://wmkeyboard.pages.dev/development/architecture/#direct-boot) for the fuller picture. - **Catalog sizes and counts are hints, not checksums.** `DictionaryCatalog`'s word counts and compressed sizes are display and progress values, generated against a point-in-time snapshot of the data repo. Drift against a newer repo state is expected and harmless.