bigscience/promptsource

1# Contributing2 3The best way to contribute growing P3 is by writing prompts for new datasets!4 5### What are Prompts?6 7A prompt consists of a template: input template and target template, along with collection of associated metadata. A template is a piece of code written in a templating language called8[Jinja](https://jinja.palletsprojects.com/en/3.0.x/). A template defines9a function that maps an example from a dataset in the10[Hugging Face datasets library](https://huggingface.co/datasets) to two strings of11text. The first is called the _input_ which provides all information that12will be available to solve a task, such as the instruction and the context.13The second piece is called the _target_, which is the desired response to the14prompt.15 16### Quick-Start Guide to Writing Prompts17 181. **Set up the app.** Fork the app and set up using the19[README](https://github.com/bigscience-workshop/promptsource/blob/main/README.md).201. **Examine the dataset.** In the "Sourcing" mode, select or type the dataset into the dropdown.21If the dataset has subsets (subsets are not the same as splits), you can select22which one to work on. Note that prompts are subset-specific. You can find23out background information on the dataset by reading the information in the24app. The dataset is a collection of examples, and each example is a Python25dictionary. The sidebar will tell you the schema that each example has.261. **Start a new prompt**. Enter a name for your first prompt and hit "Create."27You can always update the name later. If you want to cancel the prompt, select28"Delete Prompt."291. **Write the prompt**. In the box labeled "Template," enter a Jinja expression.30See the [getting started guide](#getting-started-using-jinja-to-write-prompts)31and [cookbook](#jinja-cookbook) for details on how to write templates.321. **Fill in metadata**. Fill in the metadata for the current prompt: reference, original task, choices in templates, metrics, languages, and answer choices.33See [Metadata](#metadata) for more details about these fields.341. **Save the prompt**. Hit the "Save" button. The output of the prompt35applied to the current example will appear in the right sidebar.361. **Verify the prompt**. Check that you didn't miss any case by scrolling37through a handful of examples of the prompted dataset using the38"Prompted dataset viewer" mode.391. **Write between 5 and 10 prompts**. Repeat the steps 4 to 9 to create between 540and 10 (more if you want!) prompts per dataset/subset. Feel free to introduce41a mix of formats, some that follow the templates listed in the [best practices](#best-practices)42and some that are more diverse in the format and the formulation.431. **Duplicate the prompts(s).** If the dataset you have chosen bear the same44format as other datasets (for instance, `MNLI` and `SNLI` have identical formats),45you can simply duplicate the prompts you have written to these additional datasets.461. **Upload the template(s).** Submit a PR using the instructions47[here](#uploading-prompts).48 49## Getting Started Using Jinja to Write Prompts50 51Here is a quick crash course on using [Jinja](https://jinja.palletsprojects.com/en/3.0.x/)52to write templates. More advanced usage is in the [cookbook](#jinja-cookbook).53 54Generally, in a template, you'll want to use a mix of hard-coded data that is55task-specific and stays the same across examples, and commands that tailor the56input and target to a specific example.57 58To write text that should be rendered as written, just write it normally. The59following "template" will produce the same text every time:60```jinja261This is just literal text that will be printed the same way every time.62```63 64To make your template do something more interesting, you'll need to use Jinja65expressions. Jinja expressions are surrounded by curly braces `{` and `}`.66One common thing you'll want to do is access information in the dataset example.67When applied to an example, you can access any value in the example dictionary68via its key. If you just want to print that value surround it in double curly69braces. For example, if you want to print a value with the key `text`, use this:70```jinja271The text in this example is {{ text }}.72```73 74You can also use information from the example to control behavior. For example,75suppose we have a label with the key `label` in our example, which either has a76value of 0 or 1. That's not very "natural" language, so maybe we want to decide77which label name to use based on the example. We can do this by creating a list78and indexing it with the example key:79```jinja280The label for this example is {{ ["Label A", "Label B"][label] }}.81```82We can also use dictionaries for the same thing:83```jinja284The label for this example is {{85{"a": "Label A",86 "b": "Label B"87}[label]88}}.89```90 91Note that some things in a template are particular to the task, and should not be92modified by downstream steps that try to increase the diversity of the prompts.93A common example is listing label names in the prompt to provide choices. Anything94that should not be modified by data augmentation should be surrounded by double95curly braces and quoted. For example:96```jinja297The choices are {{"a"}}, {{"b"}}, and {{"c"}}.98```99You can leave binary options like yes/no, true/false, etc. unprotected.100 101Finally, remember that a template must produce two strings: an input and a target.102To separate these two pieces, use three vertical bars `|||`.103So, a complete template for Squad could be:104```jinja2105I'm working on the final exam for my class and am trying to figure out the answer106to the question "{{question}}" I found the following info on Wikipedia and I think107it has the answer. Can you tell me the answer?108{{context}}109|||110{{answers["text"][0]}}'111```112 113## Metadata114In addition to the template itself, you need to fill out several other fields.115These metadata facilitate finding and using the prompts.116* **Prompt Reference.** If your template was inspired by a paper, note the117reference in the "Prompt Reference" section. You can also add a description of118what your template does.119* **Original Task?** The checkbox should be checked if the template requires solving a120task that the underlying dataset is used to study. For example, a template that asks a121question from a question answering dataset would be an original task template, but one that asks122to generate a question for a given answer would not.123* **Choices in Template?** The checkbox should be checked if the input explicitly indicates124the options for the possible outputs (regardless of whether `answer_choices` is used).125* **Metrics.** Use the multiselect widget to select all metrics commonly used to evaluate126this task. Choose “Other” if there is one that is not included in the list.127* **Languages.** Use the multiselect widget to select all languages used in the prompt. This is independent of what languages are used in the underlying dataset. For example, you could have an English prompt for a Spanish dataset.128* **Answer Choices.**  If the prompt has a small set of possible outputs (e.g., Yes/No,129class labels, entailment judgements, etc.), then the prompt should define and use answer130choices as follows. This allows evaluation to consider just the possible targets for131scoring model outputs. The answer choices field is a Jinja expression that should produce132a `|||` separated list of all possible targets. If the choices don't change from example133to example, then you can just list them. For example, AG News is134```jinja2135World News ||| Sports ||| Business ||| Science and Technology136```137Note that whitespace is stripped from the ends of the choices. If answer choices are set,138then they are also available to Jinja in the prompt itself in the form of a list called139`answer_choices`. You should use this list in both input and target templates so that the140resulting inputs and targets match the answer choices field exactly. For example, a prompt141for AG News could use `answer_choices` like this:142```jinja2143{{text}} Which of the following sections of a newspaper would144this article likely appear in? {{answer_choices[0]}}, {{answer_choices[1]}},145{{answer_choices[2]}}, or {{answer_choices[3]}}?146|||147{{ answer_choices[label] }}148```149Since Answer Choices is a Jinja expression that has access to the example, it can also be used150to extract example-specific choices from the underlying data. For example, in AI2 ARC, we could151use152```jinja2153{{choices.text | join("|||")}}154```155 156## Best Practices157 158* **Writing target templates.** The target template should only contain the answer to the task.159It should not contain any extra text such as “The answer is…” (unless that extra text is also in160`answer_choices`). If `answer_choices` is populated, the output should only contain the values161in `answer_choices`.162* **Formatting multple-choice questions.** If the target should match the name of the choice163(e.g., “World News”), then it should list the choices either as part of a grammatical question164or a list with the marker for each (e.g, dashes). If the target should indicate the choice from165the list (e.g., “A,” “Explanation 1,” etc.), then it should list the choices with the indicator166before each one.167* **Choosing input and target pairs.** Lots of datasets have multiple columns that can be168combined to form different (input, target) pairs i.e. different "tasks". Don't hesitate to169introduce some diversity by prompting a given dataset into multiple tasks and provide some170description in the "Template Reference" text box. An example is given171in the already prompted `movie_rationales`.172* **Filtering prompts.** If a prompt is applied to an example and produces an173empty string, that prompt/example pair will be skipped.174You can therefore create prompts that only apply to a subset of the examples by175wrapping them in Jinja if statements. For example, in the `TREC` dataset, there176are fine-grained categories that are only applicable to certain coarse-grained categories.177We can capture this with the following prompt:178```jinja2179{% if label_coarse == 0 %}180Is this question asking for a {{"definition"}}, a {{"description"}}, a {{"manner of action"}}, or a {{"reason"}}?181{{text}}182|||183{{ {0: "Manner", 7: "Defintion", 9: "Reason", 12: "Description"}[label_fine] }}184{% endif %}185```186For datasets that have splits with no labels (for instance test split without ground truth labels), you can wrap the conditional statement on the target side.187For instance for `super_glue/boolq`, the following prompt would return an empty target on the test split, but not an empty prompted example:188```jinja2189{{ passage }}190Question: {{ question }}191Answer:192|||193{% if label != -1 %}194{{ answer_choices[label] }}195{% endif %}196```197* **Conditional generation format.** Always specify the target and separate it from the prompt198by indicating the vertical bars `|||`. The target will be generated by a generative model199conditioned on the input you wrote. You can always transform an "infix" prompt format200```jinja2201Given that {{premise}}, it {{ ["must be true", "might be true", "must be false"][label] }} that {{hypothesis}}202```203into a conditional generation format204```jinja2205Given that {{premise}}, it {{ "must be true, might be true, or must be false" }} that {{hypothesis}}?|||206{{ ["must be true", "might be true", "must be false"][label] }}207```208* **Pre-defined formats.** The goal is to collect a diverse set of prompts with diverse formats, but209we also want to include a few less diverse prompts that follow the following two structures:2101) A question-answer pair with optional multiple choices like:211```212[Context]                         # optional depending on the task213[Question]214[Label1], [Label2], [Label3]      # optional depending on the task215```216So for SNLI it will look like:217```jinja2218{{premise}}219Is it the case that {{hypothesis}}?220{{ "Yes" }}, {{ "No" }}, {{ "Maybe" }} ||| {{ ["Yes", "No", "Maybe"][label] }}221```222 2232) Task description followed by the input. So for SNLI it will look like:224```jinja2225Determine the relation between the following two sentences. The relations are entailment, contradiction, or neutral.226{{premise}}227{{hypothesis}} ||| {{label}}228```229* **Setting variables.** You might want to use the Jinja expression `{% set %}` to define a variable. If you do,230do it at the beginning of the prompt, outside any conditional statements, so that the automatic prompt checks231recognize that the variable is defined.232 233## More Examples234 235Here are a few interesting examples of prompts with explanations.236 237Here's one for `hellaswag`:238```jinja2239First, {{ ctx_a.lower() }} Then, {{ ctx_b.lower() }}...240 241Complete the above description with a chosen ending:242 243(a) {{ answer_choices[0] }}244 245(b) {{ answer_choices[1] }}246 247(c) {{ answer_choices[2] }}248 249(d) {{ answer_choices[3] }}250 251||| {{ answer_choices[label | int()] }}252```253Notice how it uses functions to consistently capitalize the information and provides lots254of context (referring explicitly to "description" and "chosen ending.")255 256Here's one for `head_qa`:257```jinja2258Given this list of statements about {{category}}: {{ answers | map(attribute="atext")259| map("lower") | map("trim", ".") | join(", ") }}.260Which one is the most appropriate answer/completion for the paragraph that follows?261{{qtext}}262|||263{% for answer in answers if answer["aid"]==ra -%}264{{answer["atext"]}}265{%- endfor %}266```267Like above, it uses functions to present the choices in a readable way. Also, it268uses a for loop with conditions to handle the more intricate dataset schema.269 270Here's one for `paws`:271```jinja2272Sentence 1: {{sentence1}}273Sentence 2: {{sentence2}}274Question: Does Sentence 1 paraphrase Sentence 2? Yes or No?275|||276{{answer_choices[label]}}277```278Notice that the choices `Yes or No` are not escaped. Yes/no, true/false279are choices that do not need to be escaped (unlike categories).280 281## Uploading Prompts282 283Once you save or modify a template, the corresponding file inside the `templates`284directory in the repo will be modified. To upload it, follow these steps:2851. Run `make style` and `make quality`.2862. Commit the modified template files (anything under `templates`) to git.2873. Push to your fork on GitHub.2884. Open a pull request against `main` on the PromptSource repo.289 290 291## Jinja Cookbook292 293- Accessing nested attributes of a dict294```jinja295{{ answers_spans.spans }}296```297 298- Joining list299```jinja=300{{ spans_list | join(", ") }}301```302 303- If conditions304```jinja305{% if label==0 %}306do_something307{% elif condition %}308do_something_else309{% endif %}310```311- Using `zip()` to zip multiple lists312```jinja313{% for a, b in zip(list_A, list_B) %}314do_something_with_a_and_b315{% endfor %}316```317 318 319Jinja includes lots of complex features but for most instances you likely only320need to use the methods above. If there's something you're not sure how to do,321just open an issue. We'll collect other frequent patterns here.322