Skip to content
WM KeyboardWM Keyboard
Accessibility

Help pictures

How the pictures, clips and off/on comparisons in a setting's ? dialog are rendered on the JVM, cut, compressed, packed and published, and how to add one.

A setting’s ? dialog can show a picture above its words. This page is for whoever adds one. The reader’s side is in Seeing what a setting does.

PieceWhereWhat it does
Shot listapp/src/docShots/java/.../docshots/HelpShotList.ktOne line per setting: what to show
RendererHelpShots.kt, beside itBoots the real keyboard on the JVM and saves whole screens
Settings screensHelpScreens.kt, beside itOpens the settings app on a route, for the settings that change its own screens
Buildertools/helpvisuals/build.pyCuts, scales, encodes, packs, writes the index
Indexapp/src/main/assets/help/visuals.tsvShipped in the APK: which setting has what, sizes, checksums
Files and packshelp/files/ and help/packs/ in the data repositoryWhat the app downloads
Readercore/common/.../core/helpvisuals/ and app/.../app/HelpVisualUi.ktStore, downloads, and the dialog

A picture is filed under the resource name of the setting’s title, such as layout_number_row_title. That is the same key the row’s icon and its search entry use, so a row needs no change to gain a picture: the dialog looks its own row up.

Add one line to HELP_SHOTS, then render and build.

// A switch that changes how the keyboard looks: no steps needed.
toggle("layout_number_row_title") { setNumberRow(it) },
// A switch that changes what typing does: steps that make it show.
toggle("typing_autocorrect_title", typed { type("wprld ") }, crop = HelpCrop.TEXT) { setAutocorrect(it) },
// Two values of a slider, under literal labels.
compare(
"layout_key_height_title",
off = { repo.setKeyHeightDp(40) },
on = { repo.setKeyHeightDp(64) },
crop = HelpCrop.KEYBOARD,
labels = "=40 dp" to "=64 dp",
),
// A finger resting on a key, with room kept above the keyboard for the bubble.
toggle("keypress_popup_title", typed { press("g") }, aboveDp = 72, crop = HelpCrop.KEYBOARD) { setKeyPopup(it) },
// Something that moves.
clip("typing_glide_typing_title", typed { record { glide("hello"); hold(900) } }, aboveDp = 96),

toggle and compare boot the keyboard twice, once per half, on seeds that differ in the one setting. Neither half is a flipped copy of the other, so each shows what a user with that value really sees. gallery takes one labelled state per option. A label is a string resource’s name, or the words themselves behind = for a value that reads the same in every language.

A tool’s picture usually starts from the docs screenshot of that tool, docs("tools/app-launcher-grid"), which already fills the panel with fake apps, notes, search results or sensor readings, so nothing leaves the machine. seededWith { } adds one option on top of such a shot for a gallery. A theme editor option is a custom theme put on before the keyboard opens.

Tools that keep their state outside the settings store are seeded in their own files, listed in HelpShotList2.kt: the clock’s preferences, the notes and to-do JSON, a clipboard history with a pinned clip. The gallery tool reads FakeMedia.kt, a MediaStore provider that lists fourteen made-up photos and draws each one as a small landscape when its thumbnail is asked for.

typed { } runs its steps on the keyboard under a plain field. press("g") puts a finger on a key and stops before the long press, so the key’s bubble is up; hold("e") waits for the alternates; glide("hello", lift = false) stops a stroke with the finger still down. Pass crop = HelpCrop.TEXT when the difference lands in the text: the picture is then the changed line of the field set right over the part of the keyboard that changed, with the empty screen between them left out.

Terminal window
./gradlew :app:testEnDebugUnitTest -Pwmkb.docShots=true --tests "*HelpShots*" -Pwmkb.helpShots.only='layout_number_row'
Terminal window
python3 tools/helpvisuals/build.py --sheet /tmp/help-sheet

Add --tests "*HelpScreens*" to render the settings-app pictures as well; their list, HELP_SCREENS, is at the end of HelpScreens.kt. The Gradle run renders only the keys the regex matches, in light and dark, into app/build/helpshots/. Each frame is a test, about eleven seconds, most of it the keyboard loading its dictionaries. build.py then does the rest and prints what it made. --sheet writes one PNG per visual to look over before anything is published. A Gradle test run stops after 90 minutes, so a long list goes in parts of about 40 keys, each its own run.

  • The cut. HelpCrop.DIFF, the default for a comparison, keeps the part of the keyboard that differs between the frames. Its top and bottom move out to the gaps between rows of keys and its sides to the gaps between keys, so no key is cut through, and a cut wider than about two fifths of the screen becomes the full width. KEYBOARD keeps the keyboard from its first drawn row down, plus aboveDp of screen over it for a popup. SCREEN keeps everything below the band at the top that is empty in every frame. Every frame of a visual gets the same cut, so they line up.
  • Nothing to show. A comparison whose frames come out identical is dropped and reported, whatever its cut. That makes it cheap to try a setting: a candidate that shows no difference costs a render and nothing else.
  • Size. Stills are scaled to 720 px wide at most and written as lossy WebP, about 15 kB each. A clip is an animated WebP at 540 px; runs of identical frames become one frame held longer, and the last frame rests before the loop.
  • Names. Every file is named by the SHA-256 of its bytes. A picture that did not change keeps its name across versions, and the app refuses a file that does not hash to its name.
  • Packs. One zip per look, stored not compressed, with fixed timestamps, so the same pictures always make the same pack.

tools/helpvisuals/built.json remembers every visual built so far. A run that renders ten settings adds ten and leaves the rest alone, so the whole set never has to be rendered in one go. --forget <key> takes one out.

The renderer has no other apps, no system dialogs and no window blur, and its field asks for no capitals, so auto-capitalise has nothing to change there. The comment at the top of HelpShotList.kt lists the settings that were tried and drew the same frame twice. For those, record the phone and file the recording:

Terminal window
python3 tools/helpvisuals/build.py --take theme_glass_window_blur_title light.mp4 --dark dark.mp4

Any video ffmpeg reads works, scrcpy --record included. It is sampled at twelve frames a second, scaled, and encoded as the same kind of clip. --span 100.9 103 keeps those seconds of it, and --top and --bottom take the status bar, the app above the keyboard and the gesture bar off, in the recording’s own pixels.

A comparison or a gallery can come from a phone too, out of screenshots or frames pulled from a recording with ffmpeg:

Terminal window
python3 tools/helpvisuals/build.py --look dark --top 1250 --bottom 330 \
--stills typing_trail_width_title compare "=2 dp=thin.png" "=24 dp=thick.png"

Each picture is given as label=file, the label being a string resource’s name or words behind =, as in the shot list. --look dark files a recording made in the dark look; the app shows it to readers in the light look too until a light one is filed. A visual filed from a phone is marked as such in built.json, so building the JVM’s renders again leaves it alone. --forget it to go back to a render.

Glide typing is filed this way. A stroke injected on the JVM stops after a letter or two, so the glide clip, the trail’s width and opacity and the word preview come from a recording of a real glide.

build.py writes into a checkout of the data repository, ../wmkeyboard-data unless --out says otherwise, and stops there. Commit and push that checkout when the pictures are right, then ship the app with the new visuals.tsv. Do it in that order: an app whose index names a file the repository does not have shows the dialog’s “did not download” line.

An older app keeps working after a newer pack replaces the one it knows. Its pack download fails once, and from then on it fetches single files, which are kept in the repository for as long as any released version names them.