shaikat005/medium-web-pentesting

Medium Web Pentesting Articles

Dataset Description

A curated collection of 357 Medium articles focused on web penetration testing, scraped from Medium's search results for the query web pentesting. Each record includes article metadata and the opening snippet of the article body.

This dataset is useful for NLP tasks such as topic modeling, text classification, content recommendation, and summarization within the cybersecurity domain.

Dataset Details

Dataset Summary

Supported Tasks

• —Text Classification — classify articles by topic, difficulty, or tool type
• —Text Generation — generate article intros in the pentesting domain
• —Feature Extraction — extract embeddings for semantic search or clustering
• —Information Retrieval — build search indexes over pentesting content

Dataset Structure

Data Fields

Data Splits

This dataset is provided as a single split:

Data Preprocessing

The raw scraped data underwent the following cleaning steps before upload:

1. 1.Dropped scraper metadata columns — web_scraper_order, web_scraper_start_url, and go to were removed as they contain no semantic value.
2. 2.Dropped 3 null-blog rows — Articles with no extractable body text (likely paywalled or member-only) were removed.
3. 3.Filled `claps` and `responses` nulls with `0` — Missing engagement metrics are treated as zero engagement.
4. 4.Normalized `claps` notation — Values like 1.4K were converted to integers (1400).
5. 5.Parsed `read_time_minutes` — Extracted the numeric minute value from strings like "7 min read".
6. 6.Standardized `date` — Converted from "Jan 29, 2025" format to ISO "2025-01-29".
7. 7.Renamed `blog` → `article_snippet` — To accurately reflect that this is the article's opening snippet, not the full text.

Note on `article_snippet`: This field contains only the opening paragraph of each article as captured by the scraper. Some entries are very short greetings (e.g., "Hello Everyone!", "Hey guys!") which reflect the actual article openings. The field is kept as-is to preserve fidelity to the source. Some articles are in languages other than English (Turkish, Portuguese, Arabic, etc.).

Dataset Statistics

Example Records

{
  "title": "Web Application Pentests & The Basics",
  "author": "Mike Smith",
  "date": "2025-01-22",
  "read_time_minutes": 7,
  "claps": 68,
  "responses": 3,
  "article_snippet": "Hello Everyone!"
}

{
  "title": "Hacking With Cookies",
  "author": "Teri Radichel",
  "date": "2025-03-18",
  "read_time_minutes": 19,
  "claps": 6,
  "responses": 0,
  "article_snippet": "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
}

Source Data

Data Collection

Data was collected using a web scraper targeting Medium's public search endpoint:

https://medium.com/search?q=web+pentesting

The scraper captured article cards including title, author, date, read time, engagement counts, and the opening body snippet.

Who are the source data producers?

The articles are authored by individual Medium writers sharing knowledge about web penetration testing, bug bounty hunting, CTF writeups, and cybersecurity tooling.

Considerations for Using the Data

Social Impact

This dataset is intended for educational and research purposes in the cybersecurity NLP domain. It may help researchers build tools that assist security professionals in finding relevant literature and knowledge.

Bias and Limitations

• —Snippet-only content: The article_snippet is not the full article; full content is behind Medium's paywall for many posts.
• —English-dominant but multilingual: Most articles are in English, but Turkish, Portuguese, and Arabic articles are present without language labels.
• —Engagement bias: Articles with more claps may represent more popular or sensationalist content rather than higher quality.
• —Scraper limitations: read_time_minutes = 0 indicates the read time was not available, not that the article has no content.

Citation

If you use this dataset in your work, please cite:

@dataset{medium_web_pentesting_2026,
  title     = {Medium Web Pentesting Articles},
  year      = {2026},
  note      = {Scraped from Medium.com search results for "web pentesting"},
  license   = {CC BY 4.0}
}

Dataset Card Contact

For issues, corrections, or contributions, please open a discussion on the dataset repository.