Qinghao/AcmeTrace

Acme Trace

This repository hosts the public releases of Acme traces from the Shanghai AI Lab, encompassing workloads spanning from March 2023 to August 2023. We encourage anyone to use the traces for academic purposes, and if you had any questions, feel free to send an email to us, or file an issue on Github.

Furthermore, we have conducted a thorough analysis of the Acme workloads, detailed in our NSDI '24 paper titled Characterization of Large Language Model Development in the Datacenter.

<!-- ### Note that due to space constraints on GitHub, our cluster utilization files are not hosted here. If you're interested in accessing these files, they are available on HuggingFace (~80GB). <span style="font-size:20px;font-weight:bold;"> Link:</span> <img src="https://huggingface.co/datasets/huggingface/badges/raw/main/dataset-on-hf-sm.svg"> -->

<!-- [Acme Full Dataset](https://huggingface.co/datasets/Qinghao/AcmeTrace) -->

Acme Dataset

The main trace characteristics, dataset structure and schema are:

Main Characteristics:

• —Full Dataset size: 80GB (on HuggingFace)
• —Dataset size: 109MB
• —Duration: 6 months
• —Number of independent GPU clusters: 2
• —Total number of jobs: 880,740
• —Total number of GPU jobs: 470,497

Dataset Structure

📦AcmeTrace
 ┣ 📂data
 ┃ ┣ 📂job_trace 
 ┃ ┃ ┣ 📂trace_previous_work              (Prior job traces for comparison)
 ┃ ┃ ┃ ┣ 📜helios_trace.csv
 ┃ ┃ ┃ ┣ 📜xxx.csv
 ┃ ┃ ┣ 📜trace_kalos.csv                  (Job trace file, collected from scheduler)
 ┃ ┃ ┗ 📜trace_seren.csv
 ┃ ┣ 📂utilization
 ┃ ┃ ┣ 📂ipmi                             (Power of different server models in Seren, collected from IPMI)
 ┃ ┃ ┃ ┣ 📜CPU_D_Power.csv
 ┃ ┃ ┃ ┣ 📜GPU_AB_Power.csv
 ┃ ┃ ┃ ┗ 📜GPU_C_Power.csv
 ┃ ┃ ┣ 📂kalos                            (Resource utilization logs, collected from DCGM & Prometheus)
 ┃ ┃ ┃ ┣ 📜DRAM_ACTIVE.csv
 ┃ ┃ ┃ ┣ 📜xxx.csv
 ┃ ┃ ┣ 📂seren
 ┃ ┃ ┃ ┣ 📜DRAM_ACTIVE.csv
 ┃ ┃ ┃ ┣ 📜xxx.csv
 ┃ ┃ ┣ 📂util_pkl                         (Processed pickle files for plotting)
 ┃ ┃ ┃ ┣ 📜gpu_power_kalos.pkl
 ┃ ┃ ┃ ┣ 📜xxx.pkl
 ┃ ┣ 📜cluster_summary.csv
 ┃ ┣ 📜generate_utilization_pkl.ipynb     (Parse utilization files and generate pickles)
 ┃ ┗ 📜utils.py
 ┣ 📂figure                               (Examples of trace visualization)
 ┃ ┣ 📜bar_job_state.pdf
 ┃ ┣ 📜xxx.pdf
 ┣ 📜LICENSE.txt
 ┣ 📜README.md
 ┗ 📜analysis.ipynb                       (Scripts for plotting)

Schema and Description

1. Job Trace

Description

Provides rich information on all jobs submitted to scheduler in each cluster.

• —trace_seren.csv Example

• —trace_kalos.csv Example

Schema

Only in Kalos: | Field | Description | | ------------- | --------------------------------------------------- | | mem_per_pod_GB | Pod memory resource configuration | | shared_mem_per_pod | Pod memory resource configuration | | fail_time | the time that failure occurs | | stop_time | the time that job stops |

Notes

1. 1.A job can end up with one of five statuses: (1) COMPLETED: it is finished successfully; (2) CANCELLED: it is terminated by the user; (3) FAILED: it is terminated due to internal or external errors; (4) TIMEOUT: the execution time is out of limit; (5) NODE_FAIL: it is terminated due to the node crash. TIMEOUT and NODE_FAIL are very rare in our traces, and are regarded as failed in our analysis.
2. 2.Calculated from the difference between end_time and start_time. (Unit: seconds)
3. 3.Calculated from the difference between start_time and submit_time. (Unit: seconds)
4. 4.Calculated from the product between duration and gpu_num.

2. Resource Utilization

Description

Cluster resource utilization monitoring data, collected from DCGM, IPMI and Prometheus.

• —NODE_CPU_UTILIZATION.csv Example

Schema