fmahezs/luma-lite
0
1{2 "cells": [3 {4 "cell_type": "markdown",5 "id": "17b9b0c0",6 "metadata": {},7 "source": [8 "# Demonstration of Luma Module \n",9 "This notebook contain the implementation of the source code for each module in the EPISTEM land cover mapping framework"10 ]11 },12 {13 "cell_type": "markdown",14 "id": "ed875980",15 "metadata": {},16 "source": [17 "## Library import and earth engine initialization\n",18 "If you have earth engine account you could used that to authenticate and initialize the earth engine. However, if you did not have the account, service account initialization is avaliable"19 ]20 },21 {22 "cell_type": "code",23 "execution_count": 1,24 "id": "1be9d7e5",25 "metadata": {},26 "outputs": [27 {28 "name": "stderr",29 "output_type": "stream",30 "text": [31 "\n",32 "[notice] A new release of pip is available: 25.2 -> 25.3\n",33 "[notice] To update, run: C:\\Users\\AFahrezi\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\\python.exe -m pip install --upgrade pip\n"34 ]35 }36 ],37 "source": [38 "#This code is used if the notebook is implemented in github codespace. Just remove the (#)\n",39 "!python -m pip install .. --quiet"40 ]41 },42 {43 "cell_type": "code",44 "execution_count": 1,45 "id": "d131ed9b",46 "metadata": {},47 "outputs": [48 {49 "name": "stdout",50 "output_type": "stream",51 "text": [52 "\n",53 " EARTH ENGINE AUTHENTICATION NOTES:\n",54 " \n",55 " 1. Make sure you already have a google cloud project that has enable the Earth Engine API and registered to \n",56 " commercial or non-commercial use. For more information visit: https://developers.google.com/earth-engine/guides/access \n",57 " \n",58 " 2. you can authenticate programmatically by calling: from luma_ge.ee_config import authenticate_manually\n",59 " authenticate_manually()\n",60 " \n",61 " 3. This will open a web browser. Sign in with your Google account that has Earth Engine access.\n",62 " \n",63 " 4. Copy the authorization code from the browser and paste it in the terminal.\n",64 " \n",65 " \n",66 " For more details, visit: https://developers.google.com/earth-engine/guides/python_install\n",67 " \n"68 ]69 },70 {71 "name": "stderr",72 "output_type": "stream",73 "text": [74 "Service account initialization failed: Caller does not have required permission to use project ee-epstm2024. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=ee-epstm2024 and then retry. Propagation of the new permission may take a few minutes.\n"75 ]76 },77 {78 "name": "stdout",79 "output_type": "stream",80 "text": [81 "Earth Engine initialized with service account successfully!\n",82 "Initialized: True\n",83 "Authenticated: True\n",84 "Project: projects/ee-epstm2024/assets/Reference_data_sumsel_test\n"85 ]86 }87 ],88 "source": [89 "import ee \n",90 "import luma_ge\n",91 "\n",92 "#Option 1: Manual authenticate using personal account\n",93 "#Instructions for manual authentication\n",94 "luma_ge.print_auth_instructions()\n",95 "#uncomment the below line and follow earth engine authentication process\n",96 "#luma_ge.authenticate_manually()\n",97 "\n",98 "#Option 2: Autheticate using service account (json file)\n",99 "#service_account_path = '../auth/earth-engine-451407-520c1ef64879.json' #(if failed use this)\n",100 "service_account_path = '../auth/ee-epstm2024.json'\n",101 "success = luma_ge.initialize_with_service_account(service_account_path)\n",102 "\n",103 "if success:\n",104 " print(\"Earth Engine initialized with service account successfully!\")\n",105 "else:\n",106 " print(\"Service account initialization failed. Try to authenticate earth engine manually\")\n",107 "\n",108 "#Check authentication status\n",109 "status = luma_ge.get_auth_status()\n",110 "print(f\"Initialized: {status['initialized']}\")\n",111 "print(f\"Authenticated: {status['authenticated']}\")\n",112 "if status['project']:\n",113 " print(f\"Project: {status['project']}\")"114 ]115 },116 {117 "cell_type": "code",118 "execution_count": 5,119 "id": "87e17a13",120 "metadata": {},121 "outputs": [122 {123 "name": "stderr",124 "output_type": "stream",125 "text": [126 "c:\\Users\\AFahrezi\\AppData\\Local\\anaconda3\\envs\\luma-lite\\Lib\\site-packages\\geemap\\conversion.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.\n",127 " import pkg_resources\n"128 ]129 }130 ],131 "source": [132 "import ee \n",133 "import luma_ge\n",134 "import geemap\n",135 "#luma_ge.authenticate_manually()\n",136 "from luma_ge.data_acquisition import Reflectance_Data, Reflectance_Stats, final_Image\n",137 "from luma_ge.helpers import get_aoi_from_gaul\n"138 ]139 },140 {141 "cell_type": "markdown",142 "id": "77a1466e",143 "metadata": {},144 "source": [145 "## Module 1: Acquisition of Near-Cloud-Free Satellite Imagery"146 ]147 },148 {149 "cell_type": "markdown",150 "id": "5114ef38",151 "metadata": {},152 "source": [153 "### System Response 1.1: Area of Interest Definition"154 ]155 },156 {157 "cell_type": "code",158 "execution_count": 6,159 "id": "d8507432",160 "metadata": {},161 "outputs": [162 {163 "data": {164 "text/html": [165 "\n",166 " <style>\n",167 " .geemap-dark {\n",168 " --jp-widgets-color: white;\n",169 " --jp-widgets-label-color: white;\n",170 " --jp-ui-font-color1: white;\n",171 " --jp-layout-color2: #454545;\n",172 " background-color: #383838;\n",173 " }\n",174 "\n",175 " .geemap-dark .jupyter-button {\n",176 " --jp-layout-color3: #383838;\n",177 " }\n",178 "\n",179 " .geemap-colab {\n",180 " background-color: var(--colab-primary-surface-color, white);\n",181 " }\n",182 "\n",183 " .geemap-colab .jupyter-button {\n",184 " --jp-layout-color3: var(--colab-primary-surface-color, white);\n",185 " }\n",186 " </style>\n",187 " "188 ],189 "text/plain": [190 "<IPython.core.display.HTML object>"191 ]192 },193 "metadata": {},194 "output_type": "display_data"195 }196 ],197 "source": [198 "#Set the country and province for the AOI using GAUL admin boundaries\n",199 "aoi = get_aoi_from_gaul(country=\"Indonesia\", province=\"Sumatera Selatan\")\n",200 "#Alternatively, used geemap_shp_to_ee to directly used shapefile in your local machine"201 ]202 },203 {204 "cell_type": "code",205 "execution_count": 7,206 "id": "b9e11db3",207 "metadata": {},208 "outputs": [209 {210 "data": {211 "text/html": [212 "\n",213 " <style>\n",214 " .geemap-dark {\n",215 " --jp-widgets-color: white;\n",216 " --jp-widgets-label-color: white;\n",217 " --jp-ui-font-color1: white;\n",218 " --jp-layout-color2: #454545;\n",219 " background-color: #383838;\n",220 " }\n",221 "\n",222 " .geemap-dark .jupyter-button {\n",223 " --jp-layout-color3: #383838;\n",224 " }\n",225 "\n",226 " .geemap-colab {\n",227 " background-color: var(--colab-primary-surface-color, white);\n",228 " }\n",229 "\n",230 " .geemap-colab .jupyter-button {\n",231 " --jp-layout-color3: var(--colab-primary-surface-color, white);\n",232 " }\n",233 " </style>\n",234 " "235 ],236 "text/plain": [237 "<IPython.core.display.HTML object>"238 ]239 },240 "metadata": {},241 "output_type": "display_data"242 },243 {244 "name": "stdout",245 "output_type": "stream",246 "text": [247 " WADMKK\n",248 "0 \n",249 "1 \n",250 "2 \n",251 "3 \n",252 "4 \n",253 ".. ...\n",254 "543 Boalemo\n",255 "544 Bone Bolango\n",256 "545 Pohuwato\n",257 "546 Gorontalo Utara\n",258 "547 Kota Gorontalo\n",259 "\n",260 "[548 rows x 1 columns]\n"261 ]262 }263 ],264 "source": [265 "#alternatively, you can also select smaller AOI using regency data below\n",266 "import pandas as pd\n",267 "indo_regency = ee.FeatureCollection('projects/ee-agilakbar/assets/Indonesian_Regency')\n",268 "#check Regency List\n",269 "regency = indo_regency.aggregate_array(\"WADMKK\").getInfo()\n",270 "print(pd.DataFrame(regency, columns=[\"WADMKK\"]))\n",271 "#province = indo_regency.aggregate_array(\"WADMPR\").getInfo()\n",272 "#Example for Pagar Alam\n",273 "regency_name = \"Bantul\"\n",274 "#Filter the FeatureCollection, used it for AOI\n",275 "aoi = indo_regency.filter(ee.Filter.eq(\"WADMKK\", regency_name)).geometry()"276 ]277 },278 {279 "cell_type": "markdown",280 "id": "186e3a9f",281 "metadata": {},282 "source": [283 "### System Response 1.2: Search and Filter Imagery\n",284 "The EPISTEM source code supports Landsat mission data, ranging from Landsat 1 to Landsat 9. For Landsat 1 - 3, the avaliable data is corrected radiance reflectance. The Landsat 5-9 used here is collection 2 surface reflectance (SR) analysis ready data.\n",285 "\n",286 "The retrival logic used here is as follow:\n",287 "1. Retrive multispectral bands (band 1 - 7) from landsat collection 2 SR data (if avaliable)\n",288 "2. Retrive thermal band from landsat collection 2 TOA data \n",289 "3. Create temporal composite for each data \n",290 "4. Stacked the final two data into a earth engine image (ee.image)"291 ]292 },293 {294 "cell_type": "code",295 "execution_count": 8,296 "id": "3027fb75",297 "metadata": {},298 "outputs": [299 {300 "data": {301 "text/html": [302 "\n",303 " <style>\n",304 " .geemap-dark {\n",305 " --jp-widgets-color: white;\n",306 " --jp-widgets-label-color: white;\n",307 " --jp-ui-font-color1: white;\n",308 " --jp-layout-color2: #454545;\n",309 " background-color: #383838;\n",310 " }\n",311 "\n",312 " .geemap-dark .jupyter-button {\n",313 " --jp-layout-color3: #383838;\n",314 " }\n",315 "\n",316 " .geemap-colab {\n",317 " background-color: var(--colab-primary-surface-color, white);\n",318 " }\n",319 "\n",320 " .geemap-colab .jupyter-button {\n",321 " --jp-layout-color3: var(--colab-primary-surface-color, white);\n",322 " }\n",323 " </style>\n",324 " "325 ],326 "text/plain": [327 "<IPython.core.display.HTML object>"328 ]329 },330 "metadata": {},331 "output_type": "display_data"332 },333 {334 "name": "stderr",335 "output_type": "stream",336 "text": [337 "2026-02-20 13:47:22,567 - Reflectance_Data - INFO - ReflectanceData initialized.\n",338 "2026-02-20 13:47:22,571 - final_Image - INFO - final_Image creation initialized.\n",339 "2026-02-20 13:47:22,572 - Reflectance_Data - INFO - Starting data fetch for Landsat 8 Operational Land Imager Surface Reflectance\n",340 "2026-02-20 13:47:22,572 - Reflectance_Data - INFO - Date range: 2017-01-01 to 2017-12-31\n",341 "2026-02-20 13:47:22,574 - Reflectance_Data - INFO - Cloud cover threshold: 40%\n",342 "2026-02-20 13:47:22,574 - Reflectance_Data - INFO - detailed statistics will not be computed\n",343 "2026-02-20 13:47:22,575 - Reflectance_Stats - INFO - Reflectance Stats initialized.\n",344 "2026-02-20 13:47:22,577 - Reflectance_Data - INFO - Filtered collection created (use compute_detailed_stats=True for more information)\n",345 "2026-02-20 13:47:23,723 - final_Image - INFO - Creating quality mosaic from 26 images using NDVI as quality metric\n",346 "2026-02-20 13:47:23,723 - final_Image - INFO - Quality mosaic created covering AOI with best available pixels\n",347 "2026-02-20 13:47:25,504 - final_Image - INFO - Mosaic date range: 2017-01-26 to 2017-11-10\n",348 "2026-02-20 13:47:26,338 - final_Image - INFO - Creating Median composite from 26 images\n",349 "2026-02-20 13:47:26,338 - final_Image - INFO - Composite clipped to AOI\n",350 "2026-02-20 13:47:28,116 - final_Image - INFO - Composite created from 2017-01-26 to 2017-11-10\n"351 ]352 }353 ],354 "source": [355 "#========== FIRST RETRIVE THE MULTISPECTRAL BAND===========\n",356 "#Intialize the relfectance class data function\n",357 "optical_reflectance = Reflectance_Data()\n",358 "#Initialize the final image class for composite creation\n",359 "composite = final_Image() #NEW FEATURE ADDED HERE\n",360 "#define the start and end date for imagery collection\n",361 "start = '2017-01-01'\n",362 "end = '2017-12-31'\n",363 "#get the image collection and corresponding statistics\n",364 "landsat_data, meta = optical_reflectance.get_optical_data(aoi, start, end, optical_data='L8_SR', \n",365 " cloud_cover=40, compute_detailed_stats=False)\n",366 "#create mosaic between image collection, and clip based on AOI\n",367 "mosaic_landsat = composite.get_quality_mosaic(landsat_data, aoi, quality_band= 'NDVI', calculate_coverage=False) #REPLACE OLD CODE WITH THE NEW ONE HERE\n",368 "#Alternatively you can use temporal aggregation (ee reducer) to create mode cloudless imagery\n",369 "#Add new functionality to calculate the coverage of the composite\n",370 "median_landsat = composite.get_temporal_composite(landsat_data, aoi, reducer='Median', calculate_coverage=False) #REPLACE OLD CODE WITH THE NEW ONE HERE\n",371 "#visualization parameter\n",372 "l8_sr_visparam = {'min': 0,'max': 0.4,'gamma': [0.95, 1.1, 1],'bands':['NIR', 'RED', 'GREEN']}\n",373 "#Add the data to the map\n",374 "Map = geemap.Map()\n",375 "Map.addLayer(mosaic_landsat, l8_sr_visparam, 'L8 SR Mosaic')\n",376 "Map.addLayer(median_landsat, l8_sr_visparam, 'L8 SR Median')\n",377 "Map.addLayer(landsat_data, l8_sr_visparam, 'L8 SR Image Collection')\n",378 "# set center of the map in the area of interest\n",379 "Map.centerObject(aoi, 7)"380 ]381 },382 {383 "cell_type": "code",384 "execution_count": 9,385 "id": "62243459",386 "metadata": {},387 "outputs": [388 {389 "data": {390 "text/html": [391 "\n",392 " <style>\n",393 " .geemap-dark {\n",394 " --jp-widgets-color: white;\n",395 " --jp-widgets-label-color: white;\n",396 " --jp-ui-font-color1: white;\n",397 " --jp-layout-color2: #454545;\n",398 " background-color: #383838;\n",399 " }\n",400 "\n",401 " .geemap-dark .jupyter-button {\n",402 " --jp-layout-color3: #383838;\n",403 " }\n",404 "\n",405 " .geemap-colab {\n",406 " background-color: var(--colab-primary-surface-color, white);\n",407 " }\n",408 "\n",409 " .geemap-colab .jupyter-button {\n",410 " --jp-layout-color3: var(--colab-primary-surface-color, white);\n",411 " }\n",412 " </style>\n",413 " "414 ],415 "text/plain": [416 "<IPython.core.display.HTML object>"417 ]418 },419 "metadata": {},420 "output_type": "display_data"421 },422 {423 "name": "stderr",424 "output_type": "stream",425 "text": [426 "2026-02-20 13:47:47,341 - Reflectance_Stats - INFO - Reflectance Stats initialized.\n",427 "2026-02-20 13:47:47,341 - Reflectance_Data - INFO - Starting thermal data fetch for Landsat 8 Top-of-atmosphere reflectance\n",428 "2026-02-20 13:47:47,343 - Reflectance_Data - INFO - Date range: 2017-01-01 to 2017-12-31\n",429 "2026-02-20 13:47:47,344 - Reflectance_Data - INFO - Cloud cover threshold: 40%\n",430 "2026-02-20 13:47:47,344 - Reflectance_Data - INFO - Fast mode enabled - detailed statistics will not be computed\n",431 "2026-02-20 13:47:47,346 - Reflectance_Data - INFO - Filtered collection created (use compute_detailed_stats=True for detailed info)\n",432 "2026-02-20 13:47:49,004 - final_Image - INFO - Creating Median composite from 26 images\n",433 "2026-02-20 13:47:49,004 - final_Image - INFO - Composite clipped to AOI\n",434 "2026-02-20 13:47:50,575 - final_Image - INFO - Composite created from 2017-01-26 to 2017-11-10\n"435 ]436 },437 {438 "data": {439 "application/vnd.jupyter.widget-view+json": {440 "model_id": "157e590e795846bfa639c4b6bd044327",441 "version_major": 2,442 "version_minor": 0443 },444 "text/plain": [445 "Map(center=[-7.901331625344606, 110.35488447577849], controls=(WidgetControl(options=['position', 'transparent…"446 ]447 },448 "execution_count": 9,449 "metadata": {},450 "output_type": "execute_result"451 }452 ],453 "source": [454 "#retive thermal bands from TOA\n",455 "thermal_bands, thermal_stats = optical_reflectance.get_thermal_bands(aoi, start, end, cloud_cover=40, compute_detailed_stats=False)\n",456 "median_thermal = composite.get_temporal_composite(thermal_bands, aoi, reducer='Median') #REPLACE THE OLD CODE WITH THE NEW ONE\n",457 "thermal_vis = {'min': 286,'max': 300,'gammma': 0.4}\n",458 "#stacked all landsat bands and convert to float(making sure all data type are compatible)\n",459 "drop_landsat = median_landsat.select(median_landsat.bandNames().remove('AEROSOL'))\n",460 "stacked_landsat = drop_landsat.addBands(median_thermal).toFloat()\n",461 "#visualize the thermal bands and multispectral bands\n",462 "Map.addLayer(median_thermal, thermal_vis, \"Thermal Bands\")\n",463 "Map"464 ]465 },466 {467 "cell_type": "code",468 "execution_count": 7,469 "id": "7bc9e785",470 "metadata": {},471 "outputs": [472 {473 "data": {474 "text/html": [475 "\n",476 " <style>\n",477 " .geemap-dark {\n",478 " --jp-widgets-color: white;\n",479 " --jp-widgets-label-color: white;\n",480 " --jp-ui-font-color1: white;\n",481 " --jp-layout-color2: #454545;\n",482 " background-color: #383838;\n",483 " }\n",484 "\n",485 " .geemap-dark .jupyter-button {\n",486 " --jp-layout-color3: #383838;\n",487 " }\n",488 "\n",489 " .geemap-colab {\n",490 " background-color: var(--colab-primary-surface-color, white);\n",491 " }\n",492 "\n",493 " .geemap-colab .jupyter-button {\n",494 " --jp-layout-color3: var(--colab-primary-surface-color, white);\n",495 " }\n",496 " </style>\n",497 " "498 ],499 "text/plain": [500 "<IPython.core.display.HTML object>"501 ]502 },503 "metadata": {},504 "output_type": "display_data"505 },506 {507 "name": "stdout",508 "output_type": "stream",509 "text": [510 "['BLUE', 'GREEN', 'RED', 'NIR', 'SWIR1', 'SWIR2', 'THERMAL']\n"511 ]512 }513 ],514 "source": [515 "print(stacked_landsat.bandNames().getInfo())"516 ]517 },518 {519 "cell_type": "markdown",520 "id": "f6d9542d",521 "metadata": {},522 "source": [523 "### Image retrival report (optional)"524 ]525 },526 {527 "cell_type": "code",528 "execution_count": 11,529 "id": "5dcdcd22",530 "metadata": {},531 "outputs": [532 {533 "data": {534 "text/html": [535 "\n",536 " <style>\n",537 " .geemap-dark {\n",538 " --jp-widgets-color: white;\n",539 " --jp-widgets-label-color: white;\n",540 " --jp-ui-font-color1: white;\n",541 " --jp-layout-color2: #454545;\n",542 " background-color: #383838;\n",543 " }\n",544 "\n",545 " .geemap-dark .jupyter-button {\n",546 " --jp-layout-color3: #383838;\n",547 " }\n",548 "\n",549 " .geemap-colab {\n",550 " background-color: var(--colab-primary-surface-color, white);\n",551 " }\n",552 "\n",553 " .geemap-colab .jupyter-button {\n",554 " --jp-layout-color3: var(--colab-primary-surface-color, white);\n",555 " }\n",556 " </style>\n",557 " "558 ],559 "text/plain": [560 "<IPython.core.display.HTML object>"561 ]562 },563 "metadata": {},564 "output_type": "display_data"565 },566 {567 "name": "stderr",568 "output_type": "stream",569 "text": [570 "2025-12-11 13:27:28,877 - Reflectance_Stats - INFO - Reflectance Stats initialized.\n"571 ]572 },573 {574 "name": "stdout",575 "output_type": "stream",576 "text": [577 "============================================================\n",578 " Landsat Data Collection Retrival Report\n",579 "============================================================\n",580 "Total Images Found: 53\n",581 "Date Range: 2017-01-13 to 2017-12-17\n",582 "Unique WRS Tiles: 10\n",583 "\n",584 "Scene Cloud Cover Statistics:\n",585 "------------------------------\n",586 "Average Cloud Cover: 26.8%\n",587 "Minimum Cloud Cover: 2.9%\n",588 "Maximum Cloud Cover: 40.0%\n",589 "\n",590 "WRS Path/Row Tiles:\n",591 "------------------------------\n",592 "Path 123/Row 062\n",593 "Path 123/Row 063\n",594 "Path 124/Row 061\n",595 "Path 124/Row 062\n",596 "Path 124/Row 063\n",597 "Path 124/Row 064\n",598 "Path 125/Row 061\n",599 "Path 125/Row 062\n",600 "Path 125/Row 063\n",601 "Path 126/Row 062\n",602 "\n",603 "Available Acqusition Date:\n",604 "------------------------------\n",605 "Date range: 2017-01-13 to 2017-12-17\n",606 "(53 total acquisition dates)\n",607 "\n",608 "Scene IDs (first 10):\n",609 "------------------------------\n",610 "• LC08_123062_20170405\n",611 "• LC08_123062_20170421\n",612 "• LC08_123062_20170726\n",613 "• LC08_123062_20170912\n",614 "• LC08_123062_20171014\n",615 "• LC08_123062_20171217\n",616 "• LC08_123063_20170710\n",617 "• LC08_123063_20170827\n",618 "• LC08_123063_20170912\n",619 "• LC08_123063_20171030\n",620 "... and 43 more scenes\n",621 "\n",622 "============================================================\n"623 ]624 }625 ],626 "source": [627 "#intialize the statistic class\n",628 "stats = Reflectance_Stats()\n",629 "#get the retrival report and automatically print them\n",630 "retrival_report = stats.get_collection_statistics(landsat_data, print_report=True)"631 ]632 },633 {634 "cell_type": "markdown",635 "id": "cb839e51",636 "metadata": {},637 "source": [638 "### System Response 1.3: Imagery Download"639 ]640 },641 {642 "cell_type": "code",643 "execution_count": 8,644 "id": "49d212e5",645 "metadata": {},646 "outputs": [647 {648 "data": {649 "text/html": [650 "\n",651 " <style>\n",652 " .geemap-dark {\n",653 " --jp-widgets-color: white;\n",654 " --jp-widgets-label-color: white;\n",655 " --jp-ui-font-color1: white;\n",656 " --jp-layout-color2: #454545;\n",657 " background-color: #383838;\n",658 " }\n",659 "\n",660 " .geemap-dark .jupyter-button {\n",661 " --jp-layout-color3: #383838;\n",662 " }\n",663 "\n",664 " .geemap-colab {\n",665 " background-color: var(--colab-primary-surface-color, white);\n",666 " }\n",667 "\n",668 " .geemap-colab .jupyter-button {\n",669 " --jp-layout-color3: var(--colab-primary-surface-color, white);\n",670 " }\n",671 " </style>\n",672 " "673 ],674 "text/plain": [675 "<IPython.core.display.HTML object>"676 ]677 },678 "metadata": {},679 "output_type": "display_data"680 },681 {682 "name": "stdout",683 "output_type": "stream",684 "text": [685 "Exporting... (status: READY)\n",686 "Exporting... (status: RUNNING)\n",687 "Exporting... (status: RUNNING)\n",688 "Exporting... (status: RUNNING)\n",689 "Exporting... (status: RUNNING)\n",690 "Exporting... (status: RUNNING)\n",691 "Exporting... (status: RUNNING)\n",692 "Exporting... (status: RUNNING)\n",693 "Exporting... (status: RUNNING)\n",694 "Exporting... (status: RUNNING)\n",695 "Exporting... (status: RUNNING)\n",696 "Exporting... (status: RUNNING)\n",697 "Exporting... (status: RUNNING)\n",698 "Exporting... (status: RUNNING)\n",699 "Exporting... (status: RUNNING)\n",700 "Exporting... (status: RUNNING)\n",701 "Exporting... (status: RUNNING)\n",702 "Export complete (status: COMPLETED)\n"703 ]704 }705 ],706 "source": [707 "export_task = ee.batch.Export.image.toDrive(\n",708 " image=stacked_landsat,\n",709 " description='Landsat_Bantul',\n",710 " folder='Earth Engine',\n",711 " fileNamePrefix='Landsat_Bantul',\n",712 " scale=30,\n",713 " region=aoi, # or aoi.geometry()\n",714 " maxPixels=1e13\n",715 ")\n",716 "export_task.start()\n",717 "import time\n",718 "\n",719 "while export_task.active():\n",720 " print('Exporting... (status: {})'.format(export_task.status()['state']))\n",721 " time.sleep(10)\n",722 "\n",723 "print('Export complete (status: {})'.format(export_task.status()['state']))"724 ]725 },726 {727 "cell_type": "markdown",728 "id": "1d798e32",729 "metadata": {},730 "source": [731 "## Module 2: Land-cover classification Scheme\n",732 "Three approach are provided to handle classification scheme:\n",733 "1. Upload a csv file \n",734 "2. Manual input the classification scheme\n",735 "3. Use default classification scheme (RESTORE+ project)"736 ]737 },738 {739 "cell_type": "markdown",740 "id": "e534c2be",741 "metadata": {},742 "source": [743 "### Import the module"744 ]745 },746 {747 "cell_type": "code",748 "execution_count": null,749 "id": "2240e73f",750 "metadata": {},751 "outputs": [],752 "source": [753 "from luma_ge.classification_scheme import LULC_Scheme_Manager\n",754 "#Initialize the LULC Scheme Manager\n",755 "manager = LULC_Scheme_Manager()\n",756 "print(\"Land Cover Classification Scheme Manager initialized!\")\n",757 "print(f\"Current class count: {manager.get_class_count()}\")\n",758 "#Temporary function to display the classiifcation scheme in notebook\n",759 "#Display current classification scheme\n",760 "def display_classification_scheme(manager):\n",761 " \"\"\"Display the current classification scheme in a readable format\"\"\"\n",762 " if not manager.has_classes():\n",763 " print(\"No classes defined yet.\")\n",764 " return\n",765 " \n",766 " print(\"\\n=== Current Classification Scheme ===\")\n",767 " df = manager.get_dataframe()\n",768 " print(df.to_string(index=False))\n",769 " \n",770 " return df\n",771 "\n",772 "# Display the scheme\n",773 "df = display_classification_scheme(manager)"774 ]775 },776 {777 "cell_type": "markdown",778 "id": "72d08dfb",779 "metadata": {},780 "source": [781 "### System Response 2.1a: Upload Classification Scheme"782 ]783 },784 {785 "cell_type": "code",786 "execution_count": null,787 "id": "2c972c28",788 "metadata": {},789 "outputs": [],790 "source": [791 "import pandas as pd\n",792 "#Reset manager for CSV upload example\n",793 "manager = LULC_Scheme_Manager()\n",794 "#path to csv \n",795 "csv_path = \"../data/Example_Classification_scheme.csv\"\n",796 "\n",797 "print(\"=== CSV Upload Process ===\")\n",798 "\n",799 "# Load the CSV\n",800 "df = pd.read_csv(csv_path, sep=None, engine=\"python\")\n",801 "print(\"Loaded CSV:\")\n",802 "print(df)\n",803 "\n",804 "# Auto-detect columns\n",805 "id_col, name_col, color_col = manager.auto_detect_csv_columns(df)\n",806 "print(f\"\\nAuto-detected columns:\")\n",807 "print(f\"ID column: {id_col}\")\n",808 "print(f\"Name column: {name_col}\")\n",809 "print(f\"Color column: {color_col}\")"810 ]811 },812 {813 "cell_type": "code",814 "execution_count": null,815 "id": "dccbff10",816 "metadata": {},817 "outputs": [],818 "source": [819 "# Process CSV upload\n",820 "success, message = manager.process_csv_upload(df, id_col, name_col, color_col)\n",821 "if success:\n",822 " print(f\"✅ {message}\")\n",823 " \n",824 " # Finalize the upload\n",825 " success, message = manager.finalize_csv_upload()\n",826 " if success:\n",827 " print(f\"✅ {message}\")\n",828 " else:\n",829 " print(f\"❌ {message}\")\n",830 "else:\n",831 " print(f\"❌ {message}\")\n",832 "\n",833 "# Display the loaded scheme\n",834 "display_classification_scheme(manager)"835 ]836 },837 {838 "cell_type": "markdown",839 "id": "fa1d8b9f",840 "metadata": {},841 "source": [842 "### System Response 2.1b: Manual Scheme Definition"843 ]844 },845 {846 "cell_type": "code",847 "execution_count": null,848 "id": "458a8020",849 "metadata": {},850 "outputs": [],851 "source": [852 "#Reset manager for manual input example\n",853 "manager = LULC_Scheme_Manager()\n",854 "#Manually add the class\n",855 "print(\"=== Manual Class Addition ===\")\n",856 "\n",857 "#Example of class to add\n",858 "classes_to_add = [\n",859 " (1, \"Hutan Lahan Kering\", \"#0E6D0E\"),\n",860 " (2, \"Pertanian Lahan Kering\", \"#E8F800\"),\n",861 " (3, \"Permukiman\", \"#F81D00\"),\n",862 " (4, \"Badan Air\", \"#1512F3\"),\n",863 " (5, \"Pertanian Lahan Basah\", \"#\")\n",864 "]\n",865 "\n",866 "for class_id, class_name, color_code in classes_to_add:\n",867 " success, message = manager.add_class(class_id, class_name, color_code)\n",868 " if success:\n",869 " print(f\"✅ {message}\")\n",870 " else:\n",871 " print(f\"❌ {message}\")\n",872 "\n",873 "print(f\"\\nTotal classes: {manager.get_class_count()}\")"874 ]875 },876 {877 "cell_type": "code",878 "execution_count": null,879 "id": "213558d3",880 "metadata": {},881 "outputs": [],882 "source": [883 "# Example: Edit an existing class\n",884 "print(\"=== Editing a Class ===\")\n",885 "\n",886 "# Edit the first class (index 0)\n",887 "class_to_edit = manager.edit_class(0)\n",888 "if class_to_edit:\n",889 " print(f\"Editing class: {class_to_edit}\")\n",890 " \n",891 " # Update the class with new information\n",892 " success, message = manager.add_class(1, \"HUtan Lahan Rendah\", \"#004D00\")\n",893 " if success:\n",894 " print(f\"✅ {message}\")\n",895 " else:\n",896 " print(f\"❌ {message}\")\n",897 "\n",898 "# Display updated scheme\n",899 "display_classification_scheme(manager)"900 ]901 },902 {903 "cell_type": "markdown",904 "id": "704bb924",905 "metadata": {},906 "source": [907 "### System Response 2.1c: Template Classification Scheme"908 ]909 },910 {911 "cell_type": "code",912 "execution_count": null,913 "id": "11b2702d",914 "metadata": {},915 "outputs": [],916 "source": [917 "# Reset manager for default scheme example\n",918 "manager = LULC_Scheme_Manager()\n",919 "\n",920 "print(\"=== Available Default Schemes ===\")\n",921 "default_schemes = manager.get_default_schemes()\n",922 "\n",923 "for scheme_name, classes in default_schemes.items():\n",924 " print(f\"\\n{scheme_name}: {len(classes)} classes\")\n",925 " for class_data in classes:\n",926 " print(f\" - ID {class_data['ID']}: {class_data['Class Name']} ({class_data['Color Code']})\")"927 ]928 },929 {930 "cell_type": "code",931 "execution_count": null,932 "id": "982940ad",933 "metadata": {},934 "outputs": [],935 "source": [936 "# Load the RESTORE+ default scheme\n",937 "scheme_name = \"RESTORE+ Project\"\n",938 "success, message = manager.load_default_scheme(scheme_name)\n",939 "\n",940 "if success:\n",941 " print(f\"✅ {message}\")\n",942 "else:\n",943 " print(f\"❌ {message}\")\n",944 "\n",945 "# Display the loaded scheme\n",946 "display_classification_scheme(manager)"947 ]948 },949 {950 "cell_type": "markdown",951 "id": "fdfc2278",952 "metadata": {},953 "source": [954 "### System Response 2.2: Download classification scheme"955 ]956 },957 {958 "cell_type": "code",959 "execution_count": null,960 "id": "fecc7b3e",961 "metadata": {},962 "outputs": [],963 "source": [964 "print(\"=== Export Classification Scheme ===\")\n",965 "#Convert the selected classification scheme manager to dataframe\n",966 "classification_df = manager.get_dataframe()\n",967 "print(\"Classification DataFrame:\")\n",968 "print(classification_df)\n",969 "#Save the file\n",970 "output_path = '../Selected_LC_Classification_Scheme.csv'\n",971 "classification_df.to_csv(output_path, index=False)\n",972 "print(f\"\\n✅ Classification scheme saved to: {output_path}\")"973 ]974 },975 {976 "cell_type": "markdown",977 "id": "449588c3",978 "metadata": {},979 "source": [980 "# Module 3: Generate Region Of Interest\n",981 "Three methods to generate ROI are supported in EPISTEM platform:\n",982 "1. **Upload Training Data** - Upload your own shapefile\n",983 "2. **On-screen Sampling** - Create samples using interactive map\n",984 "3. **Default Reference Data** - Use Epistem's default training data"985 ]986 },987 {988 "cell_type": "markdown",989 "id": "4acdddc0",990 "metadata": {},991 "source": [992 "## Library Import and Setup"993 ]994 },995 {996 "cell_type": "markdown",997 "id": "2df1d02e",998 "metadata": {},999 "source": [1000 "## System Response 3.1 Prerequisite Check"1001 ]1002 },1003 {1004 "cell_type": "code",1005 "execution_count": null,1006 "id": "d956b21d",1007 "metadata": {},1008 "outputs": [],1009 "source": [1010 "print(\"=== Checking Prerequisites ===\")\n",1011 "#Load from previous module\n",1012 "#From Module 1 - AOI data\n",1013 "try:\n",1014 " AOI = aoi\n",1015 " print(\"✅ AOI from Module 1 is available\")\n",1016 " aoi_available = True\n",1017 "except:\n",1018 " print(\"❌ AOI data not available, please run Module 1 first\")\n",1019 " aoi_available = False\n",1020 "\n",1021 "#From Module 2 - Classification scheme\n",1022 "try:\n",1023 " \n",1024 " # For demonstration, create sample classification scheme\n",1025 " LULCTable = classification_df\n",1026 " print(\"✅ Classification scheme from Module 2 is available\")\n",1027 " print(f\" - Number of classes: {len(LULCTable)}\")\n",1028 " scheme_available = True\n",1029 "except:\n",1030 " print(\"❌ Classification scheme not available, please run Module 2 first\")\n",1031 " scheme_available = False\n",1032 "\n",1033 "if aoi_available and scheme_available:\n",1034 " print(\"\\n✅ All prerequisites met! You can proceed with training data collection.\")\n",1035 "else:\n",1036 " print(\"\\n❌ Prerequisites not met. Please complete previous modules first.\")"1037 ]1038 },1039 {1040 "cell_type": "markdown",1041 "id": "92cede0c",1042 "metadata": {},1043 "source": [1044 "## System Response 3.2 ROI Upload and content Verification"1045 ]1046 },1047 {1048 "cell_type": "code",1049 "execution_count": null,1050 "id": "ae398a78",1051 "metadata": {},1052 "outputs": [],1053 "source": [1054 "# Modul 3a \n",1055 "# Import modules and functions\n",1056 "import ee\n",1057 "import pandas as pd\n",1058 "from luma_ge.sample_data import SyncTrainData, SplitTrainData"1059 ]1060 },1061 {1062 "cell_type": "code",1063 "execution_count": null,1064 "id": "7411b55c",1065 "metadata": {},1066 "outputs": [],1067 "source": [1068 "# ----- Data Input -----\n",1069 "# 1. Decision to upload data\n",1070 "UploadTrainData = True # set as 'true' to upload your own training data shapefile\n",1071 "# set as 'false' to either add train data by sampling on screen or use default training data\n",1072 "\n",1073 "# 2. Training data file path (if UploadTrainData is true)\n",1074 "TrainVectPath = '../data/Training_Sumsel_Data.shp'\n",1075 "TrainField = 'ID' \n",1076 " # Load and process training data\n",1077 "TrainDataDict = SyncTrainData.LoadTrainData(\n",1078 " landcover_df=LULCTable,\n",1079 " aoi_geometry=AOI,\n",1080 " training_shp_path=TrainVectPath\n",1081 " )"1082 ]1083 },1084 {1085 "cell_type": "code",1086 "execution_count": null,1087 "id": "f745ae9d",1088 "metadata": {},1089 "outputs": [],1090 "source": [1091 "# ----- System response 3.2.a -----\n",1092 "# Set class field\n",1093 "TrainDataDict = SyncTrainData.SetClassField(TrainDataDict, TrainField)\n",1094 "\n",1095 "# Validate classes\n",1096 "TrainDataDict = SyncTrainData.ValidClass(TrainDataDict, 1)\n",1097 "\n",1098 " # Check sample sufficiency\n",1099 "TrainDataDict = SyncTrainData.CheckSufficiency(TrainDataDict, min_samples=20)\n",1100 "\n",1101 " # Filter by AOI\n",1102 "TrainDataDict = SyncTrainData.FilterTrainAoi(TrainDataDict)\n",1103 "\n",1104 " # Create training data table\n",1105 "table_df, total_samples, insufficient_df = SyncTrainData.TrainDataRaw(\n",1106 " training_data=TrainDataDict.get('training_data'),\n",1107 " landcover_df=TrainDataDict.get('landcover_df'),\n",1108 " class_field=TrainDataDict.get('class_field'))\n",1109 "\n",1110 "#Summary result\n",1111 "vr = TrainDataDict.get('validation_results', {})\n",1112 "\n",1113 "print(\"=\" * 70)\n",1114 "print(\"TRAINING DATA SUMMARY\")\n",1115 "print(\"=\" * 70)\n",1116 "print(f\"Total training points loaded : {vr.get('total_points', 'N/A')}\")\n",1117 "print(f\"Points after class filtering : {vr.get('points_after_class_filter', 'N/A')}\")\n",1118 "print(f\"Valid points (inside AOI) : {vr.get('valid_points', 'N/A')}\")\n",1119 "print(f\"Invalid classes found : {len(vr.get('invalid_classes', []))}\")\n",1120 "print(f\"Points outside AOI : {len(vr.get('outside_aoi', []))}\")\n",1121 "print(\"=\" * 70)\n",1122 "\n",1123 " # --- Display the main table ---\n",1124 "if table_df is not None and not table_df.empty:\n",1125 " display_df = table_df.copy()\n",1126 " if 'Percentage' in display_df.columns:\n",1127 " display_df['Percentage'] = display_df['Percentage'].apply(\n",1128 " lambda x: f\"{x:.2f}%\" if isinstance(x, (int, float)) else x\n",1129 " )\n",1130 " display(display_df)\n",1131 "else:\n",1132 " print(\"No valid training data available to display.\")"1133 ]1134 },1135 {1136 "cell_type": "markdown",1137 "id": "0c16694e",1138 "metadata": {},1139 "source": [1140 "## System Response 3.2 Default ROI"1141 ]1142 },1143 {1144 "cell_type": "code",1145 "execution_count": null,1146 "id": "4869adce",1147 "metadata": {},1148 "outputs": [],1149 "source": [1150 "print(\" Loading default reference training data...\")\n",1151 "TrainEePath = 'projects/ee-rg2icraf/assets/Indonesia_lulc_Sample'\n",1152 "TrainField = 'kelas'\n",1153 "\n",1154 "# Stopgap solution: Rename 'Land Cover Class' column to 'LULC_Type' if it exists\n",1155 "if 'Land Cover Class' in LULCTable.columns:\n",1156 " LULCTable = LULCTable.rename(columns={'Land Cover Class': 'LULC_Type'})\n",1157 " print(\"Column 'Land Cover Class' renamed to 'LULC_Type'\")\n",1158 "\n",1159 " \n",1160 "try:\n",1161 " print(\"Loading reference training data from Earth Engine...\")\n",1162 " \n",1163 " # Load training data\n",1164 " TrainDataDict = SyncTrainData.LoadTrainData(\n",1165 " landcover_df=LULCTable,\n",1166 " aoi_geometry=AOI,\n",1167 " training_ee_path=TrainEePath\n",1168 " )\n",1169 " \n",1170 " print(\"Processing and validating reference data...\")\n",1171 " \n",1172 " # Set class field\n",1173 " TrainDataDict = SyncTrainData.SetClassField(TrainDataDict, TrainField)\n",1174 " \n",1175 " # Validate classes\n",1176 " TrainDataDict = SyncTrainData.ValidClass(TrainDataDict)\n",1177 " \n",1178 " # Check sufficiency\n",1179 " TrainDataDict = SyncTrainData.CheckSufficiency(TrainDataDict, min_samples=20)\n",1180 " \n",1181 " # Filter by AOI\n",1182 " TrainDataDict = SyncTrainData.FilterTrainAoi(TrainDataDict)\n",1183 " \n",1184 " # Create summary table\n",1185 " table_df, total_samples, insufficient_df = SyncTrainData.TrainDataRaw(\n",1186 " training_data=TrainDataDict.get('training_data'),\n",1187 " landcover_df=TrainDataDict.get('landcover_df'),\n",1188 " class_field=TrainDataDict.get('class_field')\n",1189 " )\n",1190 " \n",1191 " print(\"✅ Reference training data loaded and processed successfully!\")\n",1192 " print(f\"Total samples: {total_samples}\")\n",1193 " \n",1194 " # Display summary table\n",1195 " display(table_df)\n",1196 " \n",1197 " # Store final training data\n",1198 " TrainDataFinal = TrainDataDict.get('training_data')\n",1199 " \n",1200 " # Show validation results\n",