Datasetpublic

backtothehuman/anatomy-generative-prototyping

The Anatomy of a Generative Prototyping Session, Study 1 Anonymous authors, CHI 2027 submission. The dataset accompanies a preregistered study in which 100 non-experts completed three ten-minute prototyping tasks with Mark, a generative prototyping environment: two from a blank page, in an unfamiliar and in a familiar domain, and one on an existing prototype. Every generation is dissected into the formulation of the prompt (abstraction and specificity), the extent of the… See the full description on the dataset page: https://huggingface.co/datasets/backtothehuman/anatomy-generative-prototyping.

sourceHugging Facecc-by-4.0updated 14d agoView on Hugging Face

0likes93downloads

Dataset Card

The Anatomy of a Generative Prototyping Session, Study 1

Anonymous authors, CHI 2027 submission. The dataset accompanies a preregistered study in which 100 non-experts completed three ten-minute prototyping tasks with Mark, a generative prototyping environment: two from a blank page, in an unfamiliar and in a familiar domain, and one on an existing prototype. Every generation is dissected into the formulation of the prompt (abstraction and specificity), the extent of the generated change (changed nodes), the verification time, and the move that followed (progression or repair).

Preregistration (anonymized view-only): https://osf.io/8wh4x/?view_only=84f9f96334d44652b9f5ddee4bbf3230

Files

File	Rows	Contents
`participants.csv`	100	one row per session: task order, age band, gender, AI-tool use, the T1 instance and its familiarity rating, the T3 rating, the two domain-knowledge probes, the familiarity differential, whether the session qualifies for the H5 test, and the session length
`generations.csv`	1646	one row per committed generation of the three tasks: prompt text, attached references, abstraction, specificity, changed nodes, generation time, verification time, and the following move
`attempts.csv`	2140	every prompt sent, including the tutorial and the attempts that were aborted or failed validation ({'committed': 1973, 'aborted': 96, 'failed': 70})
`briefs.csv`	100	the T1 instance and its brief, and the T3 brief generated from the participant's own area, with its slots
`events/<session>.jsonl.gz`	100 files	the complete research log of each session
`snapshots/<session>.tar.gz`	100 files	the source snapshots of the prototype before and after each generation, as the tool stored them
`coding/`		the prompts of the three coding passes and the code definitions

The research log

One JSON object per line. type names the event (prompt_sent, generation_started, generation_terminal, preview_click, composition_started, history_rolled_back, window_blur, task_started, task_timeout, and so on), ts_server and ts_client are milliseconds from the session start, task_key is T1, T2 or T3, and payload carries the event's data. A generation is committed when its change reached the preview, aborted when the participant interrupted it, and failed when the generated source did not pass validation; after an aborted or failed attempt the preceding snapshot was restored.

Measures

—abstraction: the level of the prototype at which the prompt operates, product, design_system, feature or component, assigned by the coding pipeline.
—specificity: the extent to which a request constrains its realization, from 1 (the request states an aim and leaves the change open) to 5 (the request determines the change).
—changed_nodes: the number of nodes added, removed or modified in the structural diff of the source snapshots before and after the generation.
—verification_s: seconds from the completion of the generation to the participant's first composition-related interaction, with the time the window was out of focus removed.
—repair and repair_type: whether the next move returned to the request that produced the generation, and how (restate, refine, revert).

Anonymization

Session identifiers are the study's own pseudonyms; Prolific identifiers never entered the logs. Participant and run identifiers and absolute timestamps were removed. Age is given in five-year bands; occupation and the free-text area of expertise were dropped, and the generated T3 brief carries the domain in the words the generator chose. Prompts are the participants' own words; a scan found no e-mail addresses or URLs in them.

License and citation

CC BY 4.0. Cite the CHI 2027 submission "The Anatomy of a Generative Prototyping Session"; the reference will be completed after the review.