shawhin/HDFS_v1_blocks

HDFS v1 Block-Level Dataset

Resources:

• —Video Explainer
• —GitHub Repo
• —Dataset

Dataset Summary

This dataset is a block-level transformation of the HDFS_v1 log dataset. While the original dataset contains individual log lines (~11M rows), this version aggregates all log entries belonging to the same block into a single text sequence, making it suitable for LLM-based anomaly classification.

Each row represents a unique HDFS block with its complete log history concatenated in chronological order.

Supported Tasks

• —anomaly-detection: Binary text classification to predict whether a block experienced an anomaly based on its log sequence.

Dataset Structure

Data Splits

Data Fields

Text Format

Each log line within text follows the format:

<LEVEL> <COMPONENT>: <CONTENT>

Example:

INFO dfs.DataNode$DataXceiver: Receiving block blk_-1608999687919862906 src: /10.251.73.220:42557 dest: /10.251.73.220:50010
INFO dfs.DataNode$DataXceiver: Receiving block blk_-1608999687919862906 src: /10.251.73.220:55213 dest: /10.251.73.220:50010
INFO dfs.FSNamesystem: BLOCK* NameSystem.allocateBlock: /mnt/hadoop/mapred/system/job_200811092030_0001/job.jar. blk_-1608999687919862906
INFO dfs.DataNode$PacketResponder: PacketResponder 1 for block blk_-1608999687919862906 terminating
INFO dfs.DataNode$PacketResponder: Received block blk_-1608999687919862906 of size 67108864 from /10.251.73.220

Source Data

• —Original Dataset: logfit-project/HDFS_v1
• —Original Source: LogPAI/loghub

Dataset Creation

1. 1.Loaded the original line-level HDFS_v1 dataset
2. 2.Formatted each log line as <LEVEL> <COMPONENT>: <CONTENT>
3. 3.Grouped by block_id and concatenated log entries (ordered by line number)
4. 4.Aggregated anomaly labels (max per block)
5. 5.Created stratified 80/10/10 train/dev/test splits preserving class distribution

Citation

@inproceedings{xu2009detecting,
  title={Detecting Large-Scale System Problems by Mining Console Logs},
  author={Xu, Wei and Huang, Ling and Fox, Armando and Patterson, David and Jordan, Michael},
  booktitle={SOSP 2009}
}

@inproceedings{zhu2023loghub,
  title={Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics},
  author={Zhu, Jieming and He, Shilin and He, Pinjia and Liu, Jinyang and Lyu, Michael R.},
  booktitle={ISSRE 2023}
}

License

See original dataset license.