bds2714/jukebox

1Tutorials2*********3 4What is tensorboard X?5----------------------6 7At first, the package was named tensorboard, and soon there are issues about8name confliction. The first alternative name came to my mind is9tensorboard-pytorch, but in order to make it more general, I chose tensorboardX10which stands for tensorboard for X.11 12Google's tensorflow's tensorboard is a web server to serve visualizations of the13training progress of a neural network, it visualizes scalar values, images,14text, etc.; these information are saved as events in tensorflow. It's a pity15that other deep learning frameworks lack of such tool, so there are already16packages letting users to log the events without tensorflow; however they only17provides basic functionalities. The purpose of this package is to let18researchers use a simple interface to log events within PyTorch (and then show19visualization in tensorboard). This package currently supports logging scalar,20image, audio, histogram, text, embedding, and the route of back-propagation. The21following manual is tested on Ubuntu and Mac, and the environment are anaconda's22python2 and python3.23 24 25Create a summary writer26-----------------------27Before logging anything, we need to create a writer instance. This can be done with:28 29.. code-block:: python30 31    from tensorboardX import SummaryWriter32    #SummaryWriter encapsulates everything33    writer = SummaryWriter('runs/exp-1')34    #creates writer object. The log will be saved in 'runs/exp-1'35    writer2 = SummaryWriter()36    #creates writer2 object with auto generated file name, the dir will be something like 'runs/Aug20-17-20-33'37    writer3 = SummaryWriter(comment='3x learning rate')38    #creates writer3 object with auto generated file name, the comment will be appended to the filename. The dir will be something like 'runs/Aug20-17-20-33-3xlearning rate'39 40Each subfolder will be treated as different experiments in tensorboard. Each41time you re-run the experiment with different settings, you should change the42name of the sub folder such as ``runs/exp2``, ``runs/myexp`` so that you can43easily compare different experiment settings. Type ``tensorboard runs`` to compare44different runs in tensorboard.45 46 47General api format48------------------49.. code-block:: python50 51    add_something(tag name, object, iteration number)52 53 54Add scalar55-----------56Scalar value is the most simple data type to deal with. Mostly we save the loss57value of each training step, or the accuracy after each epoch. Sometimes I save58the corresponding learning rate as well. It's cheap to save scalar value. Just59log anything you think is important. To log a scalar value, use60``writer.add_scalar('myscalar', value, iteration)``. Note that the program complains61if you feed a PyTorch tensor. Remember to extract the scalar value by62``x.item()`` if ``x`` is a torch scalar tensor.63 64 65Add image66---------67An image is represented as 3-dimensional tensor. The simplest case is save one68image at a time. In this case, the image should be passed as a 3-dimension69tensor of size ``[3, H, W]``. The three dimensions correspond to R, G, B channel of70an image. After your image is computed, use ``writer.add_image('imresult', x,71iteration)`` to save the image. If you have a batch of images to show, use72``torchvision``'s ``make_grid`` function to prepare the image array and send the result73to ``add_image(...)`` (``make_grid`` takes a 4D tensor and returns tiled images in 3D tensor).74 75.. Note::76	Remember to normalize your image.77 78 79Add histogram80-------------81Saving histograms is expensive. Both in computation time and storage. If training82slows down after using this package, check this first. To save a histogram,83convert the array into numpy array and save with ``writer.add_histogram('hist',84array, iteration)``.85 86 87Add figure88----------89You can save a matplotlib figure to tensorboard with the add_figure function. ``figure`` input should be ``matplotlib.pyplot.figure`` or a list of ``matplotlib.pyplot.figure``.90Check `<https://tensorboardx.readthedocs.io/en/latest/tensorboard.html#tensorboardX.SummaryWriter.add_figure>`_ for the detailed usage.91 92Add graph93---------94To visualize a model, you need a model ``m`` and the input ``t``. ``t`` can be a tensor or a list of tensors95depending on your model. If error happens, make sure that ``m(t)`` runs without problem first. See96`The graph demo <https://github.com/lanpa/tensorboardX/blob/master/examples/demo_graph.py>`_ for97complete example.98 99 100Add audio101---------102To log a single channel audio, use ``add_audio(tag, audio, iteration, sample_rate)``, where ``audio`` is an one dimensional array, and each element in the array represents the consecutive amplitude samples.103For a 2 seconds audio with ``sample_rate`` 44100 Hz, the input ``x`` should have 88200 elements.104Each element should lie in [−1, 1].105 106Add embedding107-------------108Embeddings, high dimensional data, can be visualized and converted109into human perceptible 3D data by tensorboard, which provides PCA and110t-sne to project the data into low dimensional space. What you need to do is111provide a bunch of points and tensorboard will do the rest for you. The bunch of112points is passed as a tensor of size ``n x d``, where ``n`` is the number of points and113``d`` is the feature dimension. The feature representation can either be raw data114(*e.g.* the MNIST image) or a representation learned by your network (extracted115feature). This determines how the points distributes. To make the visualization116more informative, you can pass optional metadata or ``label_imgs`` for each data117points. In this way you can see that neighboring point have similar label and118distant points have very different label (semantically or visually). Here the119metadata is a list of labels, and the length of the list should equal to ``n``, the120number of the points. The ``label_imgs`` is a 4D tensor of size ``NCHW``. ``N`` should equal121to ``n`` as well. See122`The embedding demo <https://github.com/lanpa/tensorboardX/blob/master/examples/demo_embedding.py>`_ for123complete example.124 125 126Useful commands127---------------128Install129=======130 131Simply type ``pip install tensorboardX`` in a unix shell to install this package.132To use the newest version, you might need to build from source or ``pip install133tensorboardX —-no-cache-dir`` .  To run tensorboard web server, you need134to install it using ``pip install tensorboard``.135After that, type ``tensorboard --logdir=<your_log_dir>`` to start the server, where136``your_log_dir`` is the parameter of the object constructor. I think this command is137tedious, so I add a line alias ``tb='tensorboard --logdir '`` in ``~/.bashrc``. In138this way, the above command is simplified as ``tb <your_log_dir>``. Use your favorite139browser to load the tensorboard page, the address will be shown in the terminal140after starting the server.141 142 143Misc144----145Performance issue146=================147Logging is cheap, but display is expensive.148For my experience, if there are 3 or more experiments to show at a time and each149experiment have, say, 50k points, tensorboard might need a lot of time to150present the data.151 152 153Grouping plots154==============155Usually, there are many numbers to log in one experiment. For example, when156training GANs you should log the loss of the generator, discriminator. If the157loss is composed of two other loss functions, say L1 and MSE, you might want to158log the value of the other two losses as well. In this case, you can write the159tags as Gen/L1, Gen/MSE, Desc/L1, Desc/MSE. In this way, tensorboard will group160the plots into two sections (Gen, Desc). You can also use the regular expression161to filter data.162