tsi-org/tango

1<!---2Copyright 2023- The HuggingFace Team. All rights reserved.3 4Licensed under the Apache License, Version 2.0 (the "License");5you may not use this file except in compliance with the License.6You may obtain a copy of the License at7 8    http://www.apache.org/licenses/LICENSE-2.09 10Unless required by applicable law or agreed to in writing, software11distributed under the License is distributed on an "AS IS" BASIS,12WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.13See the License for the specific language governing permissions and14limitations under the License.15-->16 17# Generating the documentation18 19To generate the documentation, you first have to build it. Several packages are necessary to build the doc, 20you can install them with the following command, at the root of the code repository:21 22```bash23pip install -e ".[docs]"24```25 26Then you need to install our open source documentation builder tool:27 28```bash29pip install git+https://github.com/huggingface/doc-builder30```31 32---33**NOTE**34 35You only need to generate the documentation to inspect it locally (if you're planning changes and want to36check how they look before committing for instance). You don't have to commit the built documentation.37 38---39 40## Previewing the documentation41 42To preview the docs, first install the `watchdog` module with:43 44```bash45pip install watchdog46```47 48Then run the following command:49 50```bash51doc-builder preview {package_name} {path_to_docs}52```53 54For example:55 56```bash57doc-builder preview diffusers docs/source/en58```59 60The docs will be viewable at [http://localhost:3000](http://localhost:3000). You can also preview the docs once you have opened a PR. You will see a bot add a comment to a link where the documentation with your changes lives.61 62---63**NOTE**64 65The `preview` command only works with existing doc files. When you add a completely new file, you need to update `_toctree.yml` & restart `preview` command (`ctrl-c` to stop it & call `doc-builder preview ...` again).66 67---68 69## Adding a new element to the navigation bar70 71Accepted files are Markdown (.md or .mdx).72 73Create a file with its extension and put it in the source directory. You can then link it to the toc-tree by putting74the filename without the extension in the [`_toctree.yml`](https://github.com/huggingface/diffusers/blob/main/docs/source/_toctree.yml) file.75 76## Renaming section headers and moving sections77 78It helps to keep the old links working when renaming the section header and/or moving sections from one document to another. This is because the old links are likely to be used in Issues, Forums, and Social media and it'd make for a much more superior user experience if users reading those months later could still easily navigate to the originally intended information.79 80Therefore, we simply keep a little map of moved sections at the end of the document where the original section was. The key is to preserve the original anchor.81 82So if you renamed a section from: "Section A" to "Section B", then you can add at the end of the file:83 84```85Sections that were moved:86 87[ <a href="#section-b">Section A</a><a id="section-a"></a> ]88```89and of course, if you moved it to another file, then:90 91```92Sections that were moved:93 94[ <a href="../new-file#section-b">Section A</a><a id="section-a"></a> ]95```96 97Use the relative style to link to the new file so that the versioned docs continue to work.98 99For an example of a rich moved section set please see the very end of [the transformers Trainer doc](https://github.com/huggingface/transformers/blob/main/docs/source/en/main_classes/trainer.mdx).100 101 102## Writing Documentation - Specification103 104The `huggingface/diffusers` documentation follows the105[Google documentation](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) style for docstrings,106although we can write them directly in Markdown.107 108### Adding a new tutorial109 110Adding a new tutorial or section is done in two steps:111 112- Add a new file under `docs/source`. This file can either be ReStructuredText (.rst) or Markdown (.md).113- Link that file in `docs/source/_toctree.yml` on the correct toc-tree.114 115Make sure to put your new file under the proper section. It's unlikely to go in the first section (*Get Started*), so116depending on the intended targets (beginners, more advanced users, or researchers) it should go in sections two, three, or four.117 118### Adding a new pipeline/scheduler119 120When adding a new pipeline:121 122- create a file `xxx.mdx` under `docs/source/api/pipelines` (don't hesitate to copy an existing file as template).123- Link that file in (*Diffusers Summary*) section in `docs/source/api/pipelines/overview.mdx`, along with the link to the paper, and a colab notebook (if available).124- Write a short overview of the diffusion model:125    - Overview with paper & authors126    - Paper abstract127    - Tips and tricks and how to use it best128    - Possible an end-to-end example of how to use it129- Add all the pipeline classes that should be linked in the diffusion model. These classes should be added using our Markdown syntax. By default as follows:130 131```132## XXXPipeline133 134[[autodoc]] XXXPipeline135    - all136	- __call__137```138 139This will include every public method of the pipeline that is documented, as well as the  `__call__` method that is not documented by default. If you just want to add additional methods that are not documented, you can put the list of all methods to add in a list that contains `all`.140 141```142[[autodoc]] XXXPipeline143    - all144	- __call__145	- enable_attention_slicing146	- disable_attention_slicing147    - enable_xformers_memory_efficient_attention 148    - disable_xformers_memory_efficient_attention149```150 151You can follow the same process to create a new scheduler under the `docs/source/api/schedulers` folder152 153### Writing source documentation154 155Values that should be put in `code` should either be surrounded by backticks: \`like so\`. Note that argument names156and objects like True, None, or any strings should usually be put in `code`.157 158When mentioning a class, function, or method, it is recommended to use our syntax for internal links so that our tool159adds a link to its documentation with this syntax: \[\`XXXClass\`\] or \[\`function\`\]. This requires the class or 160function to be in the main package.161 162If you want to create a link to some internal class or function, you need to163provide its path. For instance: \[\`pipelines.ImagePipelineOutput\`\]. This will be converted into a link with164`pipelines.ImagePipelineOutput` in the description. To get rid of the path and only keep the name of the object you are165linking to in the description, add a ~: \[\`~pipelines.ImagePipelineOutput\`\] will generate a link with `ImagePipelineOutput` in the description.166 167The same works for methods so you can either use \[\`XXXClass.method\`\] or \[~\`XXXClass.method\`\].168 169#### Defining arguments in a method170 171Arguments should be defined with the `Args:` (or `Arguments:` or `Parameters:`) prefix, followed by a line return and172an indentation. The argument should be followed by its type, with its shape if it is a tensor, a colon, and its173description:174 175```176    Args:177        n_layers (`int`): The number of layers of the model.178```179 180If the description is too long to fit in one line, another indentation is necessary before writing the description181after the argument.182 183Here's an example showcasing everything so far:184 185```186    Args:187        input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):188            Indices of input sequence tokens in the vocabulary.189 190            Indices can be obtained using [`AlbertTokenizer`]. See [`~PreTrainedTokenizer.encode`] and191            [`~PreTrainedTokenizer.__call__`] for details.192 193            [What are input IDs?](../glossary#input-ids)194```195 196For optional arguments or arguments with defaults we follow the following syntax: imagine we have a function with the197following signature:198 199```200def my_function(x: str = None, a: float = 1):201```202 203then its documentation should look like this:204 205```206    Args:207        x (`str`, *optional*):208            This argument controls ...209        a (`float`, *optional*, defaults to 1):210            This argument is used to ...211```212 213Note that we always omit the "defaults to \`None\`" when None is the default for any argument. Also note that even214if the first line describing your argument type and its default gets long, you can't break it on several lines. You can215however write as many lines as you want in the indented description (see the example above with `input_ids`).216 217#### Writing a multi-line code block218 219Multi-line code blocks can be useful for displaying examples. They are done between two lines of three backticks as usual in Markdown:220 221 222````223```224# first line of code225# second line226# etc227```228````229 230#### Writing a return block231 232The return block should be introduced with the `Returns:` prefix, followed by a line return and an indentation.233The first line should be the type of the return, followed by a line return. No need to indent further for the elements234building the return.235 236Here's an example of a single value return:237 238```239    Returns:240        `List[int]`: A list of integers in the range [0, 1] --- 1 for a special token, 0 for a sequence token.241```242 243Here's an example of a tuple return, comprising several objects:244 245```246    Returns:247        `tuple(torch.FloatTensor)` comprising various elements depending on the configuration ([`BertConfig`]) and inputs:248        - ** loss** (*optional*, returned when `masked_lm_labels` is provided) `torch.FloatTensor` of shape `(1,)` --249          Total loss is the sum of the masked language modeling loss and the next sequence prediction (classification) loss.250        - **prediction_scores** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) --251          Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).252```253 254#### Adding an image255 256Due to the rapidly growing repository, it is important to make sure that no files that would significantly weigh down the repository are added. This includes images, videos, and other non-text files. We prefer to leverage a hf.co hosted `dataset` like257the ones hosted on [`hf-internal-testing`](https://huggingface.co/hf-internal-testing) in which to place these files and reference258them by URL. We recommend putting them in the following dataset: [huggingface/documentation-images](https://huggingface.co/datasets/huggingface/documentation-images).259If an external contribution, feel free to add the images to your PR and ask a Hugging Face member to migrate your images260to this dataset.261 262## Styling the docstring263 264We have an automatic script running with the `make style` command that will make sure that:265- the docstrings fully take advantage of the line width266- all code examples are formatted using black, like the code of the Transformers library267 268This script may have some weird failures if you made a syntax mistake or if you uncover a bug. Therefore, it's269recommended to commit your changes before running `make style`, so you can revert the changes done by that script270easily.271 272