RSS3-Network/high_quality_open_web_content

1---2license: cc0-1.03dataset_info:4  features:5    - name: handle6      dtype: string7    - name: body8      dtype: string9    - name: media10      list:11        - name: address12          dtype: string13        - name: mime_type14          dtype: string15    - name: profile_id16      dtype: string17    - name: publication_id18      dtype: string19    - name: timestamp20      dtype: string21  splits:22    - name: batch_023      num_bytes: 22012246224      num_examples: 89111825    - name: batch_126      num_bytes: 88520913327      num_examples: 300000028    - name: batch_229      num_bytes: 64080899430      num_examples: 217220831    - name: batch_332      num_bytes: 92412094733      num_examples: 300000034    - name: batch_435      num_bytes: 76217388736      num_examples: 239624737    - name: batch_538      num_bytes: 750622339      num_examples: 2273540    - name: batch_641      num_bytes: 33527242      num_examples: 84243    - name: batch_744      num_bytes: 874860345      num_examples: 1835246  download_size: 222266798847  dataset_size: 344902552148configs:49  - config_name: default50    data_files:51      - split: batch_052        path: data/batch_0-*53      - split: batch_154        path: data/batch_1-*55      - split: batch_256        path: data/batch_2-*57      - split: batch_358        path: data/batch_3-*59      - split: batch_460        path: data/batch_4-*61      - split: batch_562        path: data/batch_5-*63      - split: batch_664        path: data/batch_6-*65      - split: batch_766        path: data/batch_7-*67---68 69# A High Quality Open Web Content Dataset70 71![](./assets/RSS3_LOGO_PRIMARY.svg)72 73This dataset is curated by the RSS3 Network.74It contains a large collection of content from a variety of decentralized platforms such as Farcaster and Lens.75 76The dataset has been structured and indexed by RSS3 Nodes, and is provided in a structured format for easy access and analysis.77 78**All content is available on the RSS3 Network's Data Sublayer.**79 80RSS3 ecosystem projects have been leveraging the dataset to build various applications and services, including finetuning large language models, training recommendation systems, and more.81 82## Fields83 841. **handle**: The handle of the author on the corresponding platform.852. **body**: The text content of the post.863. **media**: A list of media objects associated with the post.87   - **address**: The URL of the media.88   - **mime_type**: The MIME type of the media.894. **profile_id**: The profile id of the author on the corresponding platform.905. **publication_id**: The publication id of the post on the corresponding platform.916. **timestamp**: The timestamp of the post on the corresponding platform. When not available, it is the time in which the data was indexed and structured by RSS3 Nodes.92 93## Alternative Access94 95For enhanced filtering options and more control over data granularity, you may access the dataset through the RSS3 Network. For more details, please refer to our documentation: <https://docs.rss3.io>.96 97## Other Resources98 99In addition to providing data, RSS3 offers SDKs and frameworks for building AI solutions, learn more: <https://docs.rss3.io/>.100 101As a proud member of the Open Information Initiative, RSS3 also sponsors the Open Information Grant. If you have a great idea, apply here: <https://openinformation.io/grant>.102 103## License and Disclaimer104 105This dataset is marked with CC0 1.0.106 107By marking the work with a CC0 public domain dedication, the creator is giving up their copyright and allowing reusers to distribute, remix, adapt, and build upon the material in any medium or format, even for commercial purposes.108 109RSS3 Network and its Node operators does not own, produce and/or endorse any of the content in this dataset.110All content is owned by the respective authors and/or platforms, which is provided in a decentralized manner with unrestricted access.111At the time of publication, the respective platforms did not restrict the redistribution of the content via their terms of service and/or licenses.112