shaikat005/medium-web-pentesting

1---2license: cc-by-4.03task_categories:4  - text-classification5  - text-generation6  - feature-extraction7language:8  - en9tags:10  - medium11  - cybersecurity12  - web-pentesting13  - articles14  - nlp15  - blog16pretty_name: Medium Web Pentesting Articles17size_categories:18  - n<1K19---20 21# Medium Web Pentesting Articles22 23## Dataset Description24 25A curated collection of **357 Medium articles** focused on **web penetration testing**, scraped from Medium's search results for the query `web pentesting`. Each record includes article metadata and the opening snippet of the article body.26 27This dataset is useful for NLP tasks such as topic modeling, text classification, content recommendation, and summarization within the cybersecurity domain.28 29---30 31## Dataset Details32 33### Dataset Summary34 35| Property | Value |36|---|---|37| Source | Medium.com (`search?q=web+pentesting`) |38| Total Records | 357 |39| Date Range | 2015-09-24 to 2026-04-11 |40| Language | Multilingual (predominantly English) |41| License | CC BY 4.0 |42 43### Supported Tasks44 45- **Text Classification** — classify articles by topic, difficulty, or tool type46- **Text Generation** — generate article intros in the pentesting domain47- **Feature Extraction** — extract embeddings for semantic search or clustering48- **Information Retrieval** — build search indexes over pentesting content49 50---51 52## Dataset Structure53 54### Data Fields55 56| Column | Type | Description |57|---|---|---|58| `title` | `string` | Title of the Medium article |59| `author` | `string` | Username/display name of the article author |60| `date` | `string` | Publication date in `YYYY-MM-DD` format |61| `read_time_minutes` | `int` | Estimated reading time in minutes (0 if not available) |62| `claps` | `int` | Number of claps (Medium's engagement metric); values like `1.4K` have been converted to integers (e.g. `1400`) |63| `responses` | `int` | Number of reader responses/comments |64| `article_snippet` | `string` | Opening paragraph or intro snippet scraped from the article body |65 66### Data Splits67 68This dataset is provided as a single split:69 70| Split | Records |71|---|---|72| `train` | 357 |73 74---75 76## Data Preprocessing77 78The raw scraped data underwent the following cleaning steps before upload:79 801. **Dropped scraper metadata columns** — `web_scraper_order`, `web_scraper_start_url`, and `go to` were removed as they contain no semantic value.812. **Dropped 3 null-blog rows** — Articles with no extractable body text (likely paywalled or member-only) were removed.823. **Filled `claps` and `responses` nulls with `0`** — Missing engagement metrics are treated as zero engagement.834. **Normalized `claps` notation** — Values like `1.4K` were converted to integers (`1400`).845. **Parsed `read_time_minutes`** — Extracted the numeric minute value from strings like `"7 min read"`.856. **Standardized `date`** — Converted from `"Jan 29, 2025"` format to ISO `"2025-01-29"`.867. **Renamed `blog` → `article_snippet`** — To accurately reflect that this is the article's opening snippet, not the full text.87 88> **Note on `article_snippet`:** This field contains only the **opening paragraph** of each article as captured by the scraper. Some entries are very short greetings (e.g., "Hello Everyone!", "Hey guys!") which reflect the actual article openings. The field is kept as-is to preserve fidelity to the source. Some articles are in languages other than English (Turkish, Portuguese, Arabic, etc.).89 90---91 92## Dataset Statistics93 94| Metric | Value |95|---|---|96| Articles with 0 claps | ~21% |97| Median claps | 12 |98| Max claps | 1,400 |99| Median read time | 4 min |100| Max read time | 27 min |101| Articles with responses | ~17% |102 103---104 105## Example Records106 107```json108{109  "title": "Web Application Pentests & The Basics",110  "author": "Mike Smith",111  "date": "2025-01-22",112  "read_time_minutes": 7,113  "claps": 68,114  "responses": 3,115  "article_snippet": "Hello Everyone!"116}117```118 119```json120{121  "title": "Hacking With Cookies",122  "author": "Teri Radichel",123  "date": "2025-03-18",124  "read_time_minutes": 19,125  "claps": 6,126  "responses": 0,127  "article_snippet": "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"128}129```130 131---132 133## Source Data134 135### Data Collection136 137Data was collected using a web scraper targeting Medium's public search endpoint:138 139```140https://medium.com/search?q=web+pentesting141```142 143The scraper captured article cards including title, author, date, read time, engagement counts, and the opening body snippet.144 145### Who are the source data producers?146 147The articles are authored by individual Medium writers sharing knowledge about web penetration testing, bug bounty hunting, CTF writeups, and cybersecurity tooling.148 149---150 151## Considerations for Using the Data152 153### Social Impact154 155This dataset is intended for **educational and research purposes** in the cybersecurity NLP domain. It may help researchers build tools that assist security professionals in finding relevant literature and knowledge.156 157### Bias and Limitations158 159- **Snippet-only content**: The `article_snippet` is not the full article; full content is behind Medium's paywall for many posts.160- **English-dominant but multilingual**: Most articles are in English, but Turkish, Portuguese, and Arabic articles are present without language labels.161- **Engagement bias**: Articles with more claps may represent more popular or sensationalist content rather than higher quality.162- **Scraper limitations**: `read_time_minutes = 0` indicates the read time was not available, not that the article has no content.163 164---165 166## Citation167 168If you use this dataset in your work, please cite:169 170```bibtex171@dataset{medium_web_pentesting_2026,172  title     = {Medium Web Pentesting Articles},173  year      = {2026},174  note      = {Scraped from Medium.com search results for "web pentesting"},175  license   = {CC BY 4.0}176}177```178 179---180 181## Dataset Card Contact182 183For issues, corrections, or contributions, please open a discussion on the dataset repository.184