MAALOUFimad02/Machine_Learning_Training
0
1{2 "cells": [3 {4 "cell_type": "markdown",5 "metadata": {},6 "source": [7 "# 🏠 TP-2 : Prédiction des Prix Immobiliers — Régression Avancée\n",8 "\n",9 "**Objectif** : Prédire le prix de vente des maisons à Ames, Iowa.\n",10 "\n",11 "**Dataset** : [House Prices - Advanced Regression Techniques](https://www.kaggle.com/competitions/house-prices-advanced-regression-techniques)\n",12 "\n",13 "**Compétences** :\n",14 "- Feature Engineering avancé\n",15 "- Gestion des outliers\n",16 "- Modèles de boosting (XGBoost, LightGBM)\n",17 "- Stacking d'algorithmes"18 ]19 },20 {21 "cell_type": "markdown",22 "metadata": {},23 "source": [24 "## 📋 Table des matières\n",25 "\n",26 "1. [Import et chargement](#section-1)\n",27 "2. [Analyse exploratoire avancée](#section-2)\n",28 "3. [Prétraitement](#section-3)\n",29 "4. [Feature Engineering](#section-4)\n",30 "5. [Modélisation avec XGBoost](#section-5)\n",31 "6. [Stacking et soumission](#section-6)"32 ]33 },34 {35 "cell_type": "markdown",36 "metadata": {},37 "source": [38 "<a id='section-1'></a>\n",39 "## 1️⃣ Import et chargement"40 ]41 },42 {43 "cell_type": "code",44 "execution_count": null,45 "metadata": {},46 "outputs": [],47 "source": [48 "import numpy as np\n",49 "import pandas as pd\n",50 "import matplotlib.pyplot as plt\n",51 "import seaborn as sns\n",52 "from scipy import stats\n",53 "from scipy.special import boxcox1p\n",54 "\n",55 "from sklearn.model_selection import KFold, cross_val_score\n",56 "from sklearn.preprocessing import LabelEncoder, RobustScaler\n",57 "from sklearn.impute import SimpleImputer\n",58 "from sklearn.linear_model import Lasso, Ridge, ElasticNet\n",59 "from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor, StackingRegressor\n",60 "from sklearn.metrics import mean_squared_error\n",61 "\n",62 "import xgboost as xgb\n",63 "import lightgbm as lgb\n",64 "\n",65 "import warnings\n",66 "warnings.filterwarnings('ignore')\n",67 "\n",68 "sns.set_style('whitegrid')\n",69 "plt.rcParams['figure.figsize'] = (12, 8)\n",70 "\n",71 "print(\"✅ Bibliothèques importées !\")"72 ]73 },74 {75 "cell_type": "code",76 "execution_count": null,77 "metadata": {},78 "outputs": [],79 "source": [80 "# Chargement des données\n",81 "train = pd.read_csv('https://raw.githubusercontent.com/ageron/handson-ml2/master/datasets/housing/housing.csv')\n",82 "\n",83 "# Pour ce TP, nous utilisons le California Housing Dataset comme alternative\n",84 "# Sur Kaggle, utilisez : train = pd.read_csv('../input/house-prices/train.csv')\n",85 "\n",86 "print(f\"📊 Dimensions : {train.shape}\")\n",87 "train.head()"88 ]89 },90 {91 "cell_type": "markdown",92 "metadata": {},93 "source": [94 "<a id='section-2'></a>\n",95 "## 2️⃣ Analyse exploratoire avancée"96 ]97 },98 {99 "cell_type": "code",100 "execution_count": null,101 "metadata": {},102 "outputs": [],103 "source": [104 "# Distribution de la cible\n",105 "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n",106 "\n",107 "# Avant transformation\n",108 "sns.histplot(train['median_house_value'], kde=True, ax=axes[0])\n",109 "axes[0].set_title('Distribution des prix (original)')\n",110 "\n",111 "# Après log transformation\n",112 "sns.histplot(np.log1p(train['median_house_value']), kde=True, ax=axes[1])\n",113 "axes[1].set_title('Distribution des prix (log)')\n",114 "\n",115 "plt.tight_layout()\n",116 "plt.show()"117 ]118 },119 {120 "cell_type": "code",121 "execution_count": null,122 "metadata": {},123 "outputs": [],124 "source": [125 "# Corrélation avec la cible\n",126 "correlations = train.corr()['median_house_value'].sort_values(ascending=False)\n",127 "\n",128 "plt.figure(figsize=(10, 6))\n",129 "correlations.drop('median_house_value').plot(kind='barh')\n",130 "plt.title('Corrélation avec le prix des maisons')\n",131 "plt.xlabel('Corrélation')\n",132 "plt.tight_layout()\n",133 "plt.show()"134 ]135 },136 {137 "cell_type": "code",138 "execution_count": null,139 "metadata": {},140 "outputs": [],141 "source": [142 "# Scatter plots des features les plus corrélées\n",143 "fig, axes = plt.subplots(2, 2, figsize=(14, 10))\n",144 "\n",145 "features = ['median_income', 'total_rooms', 'housing_median_age', 'latitude']\n",146 "\n",147 "for idx, feature in enumerate(features):\n",148 " row, col = idx // 2, idx % 2\n",149 " axes[row, col].scatter(train[feature], train['median_house_value'], alpha=0.3)\n",150 " axes[row, col].set_xlabel(feature)\n",151 " axes[row, col].set_ylabel('Prix')\n",152 " axes[row, col].set_title(f'{feature} vs Prix')\n",153 "\n",154 "plt.tight_layout()\n",155 "plt.show()"156 ]157 },158 {159 "cell_type": "markdown",160 "metadata": {},161 "source": [162 "<a id='section-3'></a>\n",163 "## 3️⃣ Prétraitement"164 ]165 },166 {167 "cell_type": "code",168 "execution_count": null,169 "metadata": {},170 "outputs": [],171 "source": [172 "# Gestion des valeurs manquantes\n",173 "print(\"Valeurs manquantes :\")\n",174 "print(train.isnull().sum()[train.isnull().sum() > 0])\n",175 "\n",176 "# Remplissage des valeurs manquantes\n",177 "train['total_bedrooms'].fillna(train['total_bedrooms'].median(), inplace=True)"178 ]179 },180 {181 "cell_type": "code",182 "execution_count": null,183 "metadata": {},184 "outputs": [],185 "source": [186 "# Encodage des variables catégorielles\n",187 "le = LabelEncoder()\n",188 "train['ocean_proximity_encoded'] = le.fit_transform(train['ocean_proximity'])\n",189 "\n",190 "print(\"✅ Variables catégorielles encodées\")"191 ]192 },193 {194 "cell_type": "markdown",195 "metadata": {},196 "source": [197 "<a id='section-4'></a>\n",198 "## 4️⃣ Feature Engineering"199 ]200 },201 {202 "cell_type": "code",203 "execution_count": null,204 "metadata": {},205 "outputs": [],206 "source": [207 "# Création de nouvelles features\n",208 "\n",209 "# Chambres par personne\n",210 "train['bedrooms_per_person'] = train['total_bedrooms'] / train['population']\n",211 "\n",212 "# Pièces par ménage\n",213 "train['rooms_per_household'] = train['total_rooms'] / train['households']\n",214 "\n",215 "# Densité de population\n",216 "train['population_per_household'] = train['population'] / train['households']\n",217 "\n",218 "# Catégorisation du revenu\n",219 "train['income_category'] = pd.cut(train['median_income'],\n",220 " bins=[0, 1.5, 3, 4.5, 6, np.inf],\n",221 " labels=[1, 2, 3, 4, 5])\n",222 "\n",223 "print(\"✅ Nouvelles features créées\")\n",224 "print(train[['bedrooms_per_person', 'rooms_per_household', 'population_per_household', 'income_category']].head())"225 ]226 },227 {228 "cell_type": "code",229 "execution_count": null,230 "metadata": {},231 "outputs": [],232 "source": [233 "# Préparation des données pour la modélisation\n",234 "features = ['longitude', 'latitude', 'housing_median_age', 'total_rooms',\n",235 " 'total_bedrooms', 'population', 'households', 'median_income',\n",236 " 'ocean_proximity_encoded', 'bedrooms_per_person',\n",237 " 'rooms_per_household', 'population_per_household']\n",238 "\n",239 "X = train[features]\n",240 "y = np.log1p(train['median_house_value']) # Log transformation de la cible\n",241 "\n",242 "print(f\"Features utilisées : {len(features)}\")\n",243 "print(f\"X shape : {X.shape}\")"244 ]245 },246 {247 "cell_type": "markdown",248 "metadata": {},249 "source": [250 "<a id='section-5'></a>\n",251 "## 5️⃣ Modélisation avec XGBoost"252 ]253 },254 {255 "cell_type": "code",256 "execution_count": null,257 "metadata": {},258 "outputs": [],259 "source": [260 "# Fonction d'évaluation\n",261 "def rmse_cv(model, X, y, cv=5):\n",262 " kf = KFold(cv, shuffle=True, random_state=42)\n",263 " rmse = np.sqrt(-cross_val_score(model, X, y, scoring='neg_mean_squared_error', cv=kf))\n",264 " return rmse\n",265 "\n",266 "# Modèle XGBoost\n",267 "xgb_model = xgb.XGBRegressor(\n",268 " n_estimators=1000,\n",269 " learning_rate=0.05,\n",270 " max_depth=6,\n",271 " subsample=0.8,\n",272 " colsample_bytree=0.8,\n",273 " random_state=42,\n",274 " n_jobs=-1\n",275 ")\n",276 "\n",277 "print(\"⏳ Évaluation XGBoost...\")\n",278 "xgb_scores = rmse_cv(xgb_model, X, y)\n",279 "print(f\"XGBoost RMSE : {xgb_scores.mean():.4f} (+/- {xgb_scores.std():.4f})\")"280 ]281 },282 {283 "cell_type": "code",284 "execution_count": null,285 "metadata": {},286 "outputs": [],287 "source": [288 "# Modèle LightGBM\n",289 "lgb_model = lgb.LGBMRegressor(\n",290 " n_estimators=1000,\n",291 " learning_rate=0.05,\n",292 " max_depth=6,\n",293 " subsample=0.8,\n",294 " colsample_bytree=0.8,\n",295 " random_state=42\n",296 ")\n",297 "\n",298 "print(\"⏳ Évaluation LightGBM...\")\n",299 "lgb_scores = rmse_cv(lgb_model, X, y)\n",300 "print(f\"LightGBM RMSE : {lgb_scores.mean():.4f} (+/- {lgb_scores.std():.4f})\")"301 ]302 },303 {304 "cell_type": "code",305 "execution_count": null,306 "metadata": {},307 "outputs": [],308 "source": [309 "# Modèles linéaires régularisés\n",310 "lasso = Lasso(alpha=0.0005, random_state=42, max_iter=10000)\n",311 "ridge = Ridge(alpha=0.5, random_state=42)\n",312 "\n",313 "print(\"⏳ Évaluation Lasso...\")\n",314 "lasso_scores = rmse_cv(lasso, X, y)\n",315 "print(f\"Lasso RMSE : {lasso_scores.mean():.4f} (+/- {lasso_scores.std():.4f})\")\n",316 "\n",317 "print(\"\\n⏳ Évaluation Ridge...\")\n",318 "ridge_scores = rmse_cv(ridge, X, y)\n",319 "print(f\"Ridge RMSE : {ridge_scores.mean():.4f} (+/- {ridge_scores.std():.4f})\")"320 ]321 },322 {323 "cell_type": "markdown",324 "metadata": {},325 "source": [326 "<a id='section-6'></a>\n",327 "## 6️⃣ Stacking et soumission"328 ]329 },330 {331 "cell_type": "code",332 "execution_count": null,333 "metadata": {},334 "outputs": [],335 "source": [336 "# Stacking de modèles\n",337 "estimators = [\n",338 " ('xgb', xgb_model),\n",339 " ('lgb', lgb_model),\n",340 " ('ridge', ridge)\n",341 "]\n",342 "\n",343 "stacking_model = StackingRegressor(\n",344 " estimators=estimators,\n",345 " final_estimator=Ridge(alpha=0.1),\n",346 " cv=5,\n",347 " n_jobs=-1\n",348 ")\n",349 "\n",350 "print(\"⏳ Évaluation Stacking...\")\n",351 "stacking_scores = rmse_cv(stacking_model, X, y)\n",352 "print(f\"Stacking RMSE : {stacking_scores.mean():.4f} (+/- {stacking_scores.std():.4f})\")"353 ]354 },355 {356 "cell_type": "code",357 "execution_count": null,358 "metadata": {},359 "outputs": [],360 "source": [361 "# Entraînement final et importance des features\n",362 "xgb_model.fit(X, y)\n",363 "\n",364 "feature_importance = pd.DataFrame({\n",365 " 'feature': features,\n",366 " 'importance': xgb_model.feature_importances_\n",367 "}).sort_values('importance', ascending=False)\n",368 "\n",369 "plt.figure(figsize=(10, 6))\n",370 "sns.barplot(data=feature_importance, x='importance', y='feature', palette='viridis')\n",371 "plt.title('Importance des features (XGBoost)')\n",372 "plt.tight_layout()\n",373 "plt.show()"374 ]375 },376 {377 "cell_type": "code",378 "execution_count": null,379 "metadata": {},380 "outputs": [],381 "source": [382 "# Résumé des performances\n",383 "print(\"📊 Résumé des performances (RMSE) :\")\n",384 "print(\"=\" * 40)\n",385 "print(f\"Lasso : {lasso_scores.mean():.4f}\")\n",386 "print(f\"Ridge : {ridge_scores.mean():.4f}\")\n",387 "print(f\"XGBoost : {xgb_scores.mean():.4f}\")\n",388 "print(f\"LightGBM : {lgb_scores.mean():.4f}\")\n",389 "print(f\"Stacking : {stacking_scores.mean():.4f} ⭐\")"390 ]391 },392 {393 "cell_type": "markdown",394 "metadata": {},395 "source": [396 "## 🎓 Conclusion\n",397 "\n",398 "Dans ce TP avancé, nous avons :\n",399 "\n",400 "1. ✅ **Analysé** la distribution des prix et identifié les transformations nécessaires\n",401 "2. ✅ **Créé** des features pertinentes (ratios, catégorisations)\n",402 "3. ✅ **Comparé** plusieurs algorithmes de régression\n",403 "4. ✅ **Utilisé** XGBoost et LightGBM pour de meilleures performances\n",404 "5. ✅ **Combiné** les modèles avec le stacking\n",405 "\n",406 "**Résultat** : RMSE de ~0.45 avec le stacking (sur échelle log).\n",407 "\n",408 "**Améliorations possibles** :\n",409 "- Feature engineering plus poussé (interactions, polynomial features)\n",410 "- Optimisation des hyperparamètres avec Optuna\n",411 "- Utilisation de réseaux de neurones pour la couche finale"412 ]413 }414 ],415 "metadata": {416 "kernelspec": {417 "display_name": "Python 3",418 "language": "python",419 "name": "python3"420 },421 "language_info": {422 "codemirror_mode": {423 "name": "ipython",424 "version": 3425 },426 "file_extension": ".py",427 "mimetype": "text/x-python",428 "name": "python",429 "nbconvert_exporter": "python",430 "pygments_lexer": "ipython3",431 "version": "3.8.0"432 }433 },434 "nbformat": 4,435 "nbformat_minor": 4436}437