Bangoose/Geocities-Derivations

GeoCities Derived Datasets

This folder contains all the final dataset files derived.

Info

GeoId

GeoIds represent a GeoCities site, not individual page. They take the from of:

STRUCT(country VARCHAR, neighborhood VARCHAR, suburb VARCHAR, id VARCHAR)

Datasets

geocities-full.parquet & geocities-fulluser.parquet

Dataset containing all content related to individual GeoCities sites. The fulluser one only contains sites in which user data exists for.

• —geoid (GEOID): GeoID for the site.
• —incoming (GEOID[]): GeoIDs of sites linking to this site.
• —outgoing (GEOID[]): GeoIDs of sites linked from this site.
• —badges (VARCHAR[]): SHA1 hashes of web badges found on the site.
• —urls (VARCHAR[]): Observed URLs corresponding to the site.
• —index (VARCHAR): Inferred index url.
• —content (VARCHAR): Derived textual content (capped at 4kb).

geocities-meta.parquet

Dataset containing all content related to individual GeoCities sites, excluding the derived textual content.

• —geoid (GEOID): GeoID for the site.
• —incoming (GEOID[]): GeoIDs of sites linking to this site.
• —outgoing (GEOID[]): GeoIDs of sites linked from this site.
• —badges (VARCHAR[]): SHA1 hashes of web badges found on the site.
• —urls (VARCHAR[]): Observed URLs corresponding to the site.
• —index (VARCHAR): Inferred index url.

geocities-badges.parquet

Dataset containing all the hash and corresponding source GeoIds of all the web badges (that have at least one associated GeoId).

• —sha1 (VARCHAR): SHA1 hash of the web badge.
• —geoids (GEOID[]): GeoIDs of sites containing the badge.

badges.zip

Dataset containing all the images of the scraped badges. Each image's name is it's SHA1 hash.

geocities-users.paraquet

Dataset containing all users derived from guestbook entries.

• —user_id (VARCHAR): Identifier for the derived user.
• —geoid (GEOID): GeoID for the site associated with the user.

geocities-guestentries.paraquet

Dataset containing all the entries of the parsed guestbooks. Labels were only included if they appeared on multiple distinct sites.

• —geoid (GEOID): GeoID for the site.
• —entry_id (VARCHAR): Identifier for the guestbook entry.
• —field_id (INTEGER): Identifier for the field within the guestbook entry.
• —raw_label (VARCHAR): Raw field label.
• —canonical_label (VARCHAR): Canonicalized field label.
• —value (VARCHAR): Raw entry value.
• —crawl_date (BIGINT): Crawl timestamp.
• —url (VARCHAR): URL of the guestbook.
• —page_md5 (VARCHAR): MD5 hash of the guestbook's html.