NagaYu/halfword-bench

Halfword benchmark: conversational text under access-method cost models

This dataset pairs public conversational sentences with timing cost models for AAC access methods, so that a prediction system can be scored in seconds to utterance rather than in keystrokes saved.

It contains no data from AAC users. It is public conversational text plus simulation.

Configurations

• —utterances (12565 rows) -- normalised sentences with history, pseudo-speaker, source and that source's licence on every row.
• —cost_model (38 rows) -- every timing parameter with its citation, confidence label and published range.
• —traces (14400 rows) -- simulated entry per access method, rate and condition.

What the traces show

Two things in that table are worth reading carefully rather than skimming:

• —The gap tracks positional cost, not speed. Gaze is slow and shows nothing; scanning is faster per utterance at 0.5 s and shows 15%. What matters is whether reaching the tenth candidate costs more than reaching the first, and whether candidate rows push the keyboard further down the sweep. On gaze and touch, neither is true.
• —*On touch the time-optimal policy is slightly worse (about 1%).* Where position is nearly free there is nothing to optimise, and the decoder's one-step approximation costs a little. This is the honest boundary of the method: it is for scanning-like access.

Note also that the time is not free. On these corpora the time-optimal policy gives up a few points of keystroke savings to buy its seconds. That is the argument, not an inconvenience to it: if you report keystroke savings, this will look slightly worse, and the person will still have finished their sentence sooner.

Licences, per source

• —halfword-seed -- CC0-1.0. Hand-authored everyday sentences bundled so the repository runs offline. NOT data from AAC users. <halfword/data/seed_corpus.json>
• —allenai/soda -- cc-by-4.0. SODA (Kim et al., 2023), CC BY 4.0. <https://huggingface.co/datasets/allenai/soda>
• —google/Synthetic-Persona-Chat -- cc-by-4.0. Synthetic-Persona-Chat (Jandaghi et al., 2023), CC BY 4.0. <https://huggingface.co/datasets/google/Synthetic-Persona-Chat>
• —pfb30/multi_woz_v22 -- apache-2.0. MultiWOZ 2.2 (Zang et al., 2020), Apache-2.0. <https://huggingface.co/datasets/pfb30/multiwozv22>
• —Helsinki-NLP/tatoeba -- cc-by-2.0. Tatoeba, CC BY 2.0 FR. Individual sentences are contributed by Tatoeba members. <https://huggingface.co/datasets/Helsinki-NLP/tatoeba>

Limitations, stated plainly

• —The seconds are simulated, not measured with people. They price mechanical scan, dwell and pointing cost only. Visual search and decision cost are not modelled, which the AAC literature identifies as a major contributor -- so these figures are a lower bound on the cost of long candidate lists, not an estimate of it.
• —Pseudo-speakers are a construction from public corpora. They are not people, and the personalisation results should be read as 'the machinery works', not as an effect size anyone should expect.
• —Every duration here is a property of an interface configuration. None of them is a property of a person.

We would rather be corrected than cited

This benchmark was built without AAC users in the room, which is a real limitation and not a small one. If you use AAC, or support someone who does, and the defaults here look wrong -- please open an issue. See the project README for what we most need help with.