rocca/top-reddit-posts
The post-data-by-subreddit.tar file contains 5000 gzipped json files - one for each of the top 5000 subreddits (as roughly measured by subscriber count and comment activity). Each of those json files (e.g. askreddit.json) contains an array of the data for the top 1000 posts of all time. Notes: I stopped crawling a subreddit's top-posts list if I reached a batch that had a post with a score less than 5, so some subreddits won't have the full 1000 posts. No posts comments are included. Only the… See the full description on the dataset page: https://huggingface.co/datasets/rocca/top-reddit-posts.
1119
1import {exists} from "https://deno.land/std/fs/mod.ts";2 3let subreddits = JSON.parse(await Deno.writeTextFile(`./top-5k-subreddits.json`);4 5while(1) {6 7 await new Promise(r => setTimeout(r, 1000));8 9 try {10 11 let i = -1;12 for(let s of subreddits) {13 i++;14 if(await exists(`./post-data-by-subreddit/${s}.json`)) continue;15 let after = "";16 let posts = [];17 while(true) {18 let url = `https://www.reddit.com/r/${s}/top.json?t=all&limit=100&after=${after}`;19 console.log(i, url);20 let result = await fetch(url).then(r => r.json()).catch(e => (console.error(e), false));21 let data = result.data;22 23 if(!data.children) console.log(JSON.stringify(data));24 25 for(let post of data.children) {26 delete post.data.all_awardings; // awards take up a lot of space (e.g. 20MB with vs 4MB without for r/funny)27 }28 posts.push(...data.children);29 after = data.after;30 if(!after) break;31 if(!data.children.find(c => c.data.score > 5)) break;32 }33 await Deno.writeTextFile(`./post-data-by-subreddit/${s}.json`, JSON.stringify(posts));34 }35 36 console.log("FINISHED.");37 break;38 39 } catch(e) {40 console.error(e);41 await new Promise(r => setTimeout(r, 1000*60*15));42 }43 44}45