goker/comp-serp-data

Comprehensive SERP Data

This dataset contains comprehensive search engine ranking data collected from Google and Bing, along with extracted technical and content features for analyzing search engine ranking algorithms.

📊 Dataset Overview

• —Total Records: 14,465 search results
• —Search Engines: Google (5,895 results) and Bing (8,570 results)
• —Keywords: 500 diverse search queries
• —Features: 20 features including technical scores, content analysis, and ranking metadata

🎯 Research Purpose

This dataset was created to empirically characterize and compare the ranking environments of large-scale search engines (Google and Bing) through systematic data collection and feature extraction. It enables research on:

• —Search engine ranking algorithm analysis
• —Cross-engine comparison studies
• —Technical SEO feature importance
• —Content relevance analysis
• —Ranking prediction modeling

📁 Dataset Structure

Main Dataset File

• —File: datasets/dataset.csv
• —Format: CSV (Comma-separated values)
• —Encoding: UTF-8
• —Size: ~2MB

Column Descriptions

Search Metadata

Content Features

Technical Features (Lighthouse Scores)

Analysis Features

🔬 Data Collection Methodology

1. Keyword Selection

• —Source: raw/keywords/keywords.csv
• —Count: 500 diverse search queries
• —Categories: E-commerce, services, information, local search
• —Selection Criteria: High search volume, diverse intent types

2. Search Engine Data Collection

• —Google: Custom Search API (top 20 results per query)
• —Bing: Bing Search API (top 20 results per query)
• —Collection Period: Systematic collection with rate limiting
• —Error Handling: Retry mechanisms and exception tracking

3. Web Page Processing

• —HTML Extraction: Full page content capture using headless browsers
• —Screenshot Capture: Visual page representation
• —Performance Measurement: Lighthouse scores via PageSpeed Insights API
• —Content Analysis: NLP-based feature extraction

4. Feature Extraction

• —Technical Features: Automated Lighthouse scoring
• —Content Features: Natural language processing and semantic analysis
• —Query Matching: Exact and fuzzy matching algorithms
• —Semantic Similarity: Sentence transformer-based relevance scoring

📈 Dataset Statistics

Distribution by Search Engine

• —Google: 5,895 results (40.7%)
• —Bing: 8,570 results (59.3%)

Distribution by Ranking Tier

• —High Tier (positions 1-5): 3,784 results (26.2%)
• —Medium Tier (positions 6-10): 3,837 results (26.5%)
• —Low Tier (positions 11-20): 6,844 results (47.3%)

Cluster Distribution

• —Cluster 0: 2,314 results (16.0%)
• —Cluster 1: 2,122 results (14.7%)
• —Cluster 2: 1,522 results (10.5%)
• —Cluster 3: 5,142 results (35.6%)
• —Cluster 4: 2,311 results (16.0%)
• —Cluster 5: 1,054 results (7.3%)

🛠️ Technical Implementation

Data Collection Pipeline

The dataset was created using the SERP Profiler Kit, a comprehensive research framework with the following components:

1. 1.Data Collection Modules
2. 2.Google Custom Search API integration
3. 3.Bing Search API integration
4. 4.Web scraping with headless browsers
5. 5.PageSpeed Insights API integration

1. 1.Feature Extraction
2. 2.Natural language processing with sentence transformers
3. 3.Technical SEO analysis using Lighthouse
4. 4.Content relevance scoring
5. 5.Query matching algorithms

1. 1.Data Processing
2. 2.Quality validation and outlier detection
3. 3.Feature normalization and standardization
4. 4.Cluster analysis using K-means
5. 5.Statistical analysis and validation

Quality Assurance

• —Exception Tracking: Comprehensive error logging
• —Data Validation: Multi-stage quality checks
• —Outlier Detection: Statistical anomaly identification
• —Reproducibility: Deterministic processing with fixed random seeds

📊 Research Applications

1. Clustering Analysis (RQ1)

• —Identify distinct ranking profiles using K-means clustering
• —Analyze feature patterns across different ranking strategies
• —Validate cluster quality using multiple metrics

2. Feature Importance Analysis (RQ2)

• —Determine which features most strongly predict ranking positions
• —Compare feature importance across ranking tiers
• —Identify engine-specific ranking factors

3. Cross-Engine Comparison (RQ3)

• —Compare ranking characteristics between Google and Bing
• —Analyze feature distribution differences
• —Identify engine-specific ranking patterns

4. Ranking Prediction (RQ4)

• —Build ordinal logistic regression models for ranking prediction
• —Analyze feature coefficients and significance
• —Validate model assumptions and performance

🔧 Usage Examples

Python Usage

import pandas as pd

# Load the dataset
df = pd.read_csv('datasets/dataset.csv')

# Basic statistics
print(f"Dataset shape: {df.shape}")
print(f"Engines: {df['engine'].value_counts()}")
print(f"Rank tiers: {df['rank_tier'].value_counts()}")

# Filter by search engine
google_results = df[df['engine'] == 'google']
bing_results = df[df['engine'] == 'bing']

# Analyze technical features
tech_features = ['performance_score', 'accessibility_score', 
                'best-practices_score', 'seo_score']
print(df[tech_features].describe())

# Analyze content features
content_features = ['query_in_title', 'query_in_h1', 
                   'semantic_similarity_title_query', 'word_count']
print(df[content_features].describe())

R Usage

# Load the dataset
df <- read.csv('datasets/dataset.csv')

# Basic analysis
summary(df)
table(df$engine)
table(df$rank_tier)

# Technical features analysis
tech_cols <- c('performance_score', 'accessibility_score', 
               'best.practices_score', 'seo_score')
summary(df[tech_cols])

🔗 Related Resources

• —Source Code: SERP Profiler Kit Repository
• —Documentation: Complete methodology and implementation details
• —Analysis Results: Statistical analysis and visualization outputs

📄 License

This dataset is released under the MIT License. Please ensure compliance with search engine terms of service and website robots.txt files when using this data.

🤝 Contributing

For questions, issues, or contributions:

• —Open an issue on the repository
• —Review the documentation for methodology details
• —Check the analysis results for statistical insights

⚠️ Important Notes

1. 1.API Compliance: This dataset was collected following search engine API terms of service
2. 2.Ethical Scraping: All web scraping was performed with respect to robots.txt files
3. 3.Data Freshness: Search results may change over time; this dataset represents a snapshot
4. 4.Usage Limitations: This dataset is for research purposes only

Note: This dataset enables systematic analysis of search engine ranking algorithms and provides a foundation for understanding how different factors influence search result positioning across major search engines.