sidruud/agentic-etl-to-databricks-bot

title: Agentic ETL to Databricks Bot sdk: gradio sdkversion: 6.5.1 appfile: gradioapp.py pythonversion: 3.11 license: apache-2.0 ---

🚀 Agentic ETL to DataBricks Bot - HF Spaces Ready-to-Deploy Package

Version: 1.0 | Date: February 9, 2026 Status: ✅ Production Ready for Hugging Face Spaces

📦 What's in This Package

This folder contains everything you need to deploy the Agentic ETL to DataBricks Bot on Hugging Face Spaces in just 3 steps (5 minutes).

hf-spaces-deployment/
├── gradio_app.py ......................... Main web interface
├── requirements.txt ...................... Python dependencies (HF optimized)
├── .gitignore ........................... Exclude cache/logs/large files
├── HF_SPACES_DEPLOYMENT_GUIDE.md ........ Full step-by-step guide
├── PACKAGE_README.md .................... Quick overview
├── README.md (this file) ................ You are here
├── src/ ................................. Agent code (DataStage, ODS, S2T, Planner, etc.)
├── .github/skill/ ....................... Policies & Instructions
├── docs/skills/ ......................... Knowledge pack (70+ skill files)
├── references/databricks/ ............... Notebook templates
├── artifacts/ ........................... Runtime output directory (empty)
│   ├── bronze/
│   ├── silver/
│   ├── gold/
│   ├── tests/
│   ├── reports/
│   ├── metrics/
│   └── cache/
└── inputs/ ............................. User input directory (empty)
    ├── dsx/
    └── s2t/

🎯 Deploy in 3 Steps

Step 1: Create Hugging Face Space (1 minute)

1. 1.Go to: https://huggingface.co/spaces
2. 2.Click "Create New Space"
3. 3.Fill in:
4. 4.Space name: agentic-etl-to-databricks-bot
5. 5.Space SDK: Gradio (select from dropdown)
6. 6.Visibility: Public
7. 7.License: Apache 2.0

Step 2: Upload This Package (2 minutes)

Option A: Git (Recommended)

# 1. Clone your new Space
git clone https://huggingface.co/spaces/YOUR_USERNAME/agentic-etl-to-databricks-bot
cd agentic-etl-to-databricks-bot

# 2. Copy all files from this package
xcopy* ..\..\etl_to_databricks_agent_demo\hf-spaces-deployment\* . /E

# 3. Push to Hugging Face
git add .
git commit -m "Deploy agentic ETL bot - Initial release"
git push

Option B: Web Upload (Simpler - Drag & Drop)

• —Open your Space's web editor
• —Drag & drop all folders into the editor
• —Files auto-commit

Step 3: Done! (1-2 minutes for first build)

Your bot is live at:

https://huggingface.co/spaces/YOUR_USERNAME/agentic-etl-to-databricks-bot

First deployment: 30-60 seconds (installs dependencies) Subsequent loads: 2-5 seconds

✨ Features (Once Deployed)

Users can access these features via a web interface:

Auto NLP Router 🤖

Type: "migrate inputs/dsx/example.dsx silver only"
→ Auto-routed to correct agent

Agent Team (12 Active)

• —⚡ LeadAgent (Orchestrator)
• —📄 DataStageAgent (DSX Parser)
• —🔗 OdsDsxAgent (ODS Analysis)
• —📊 S2TAgent (Excel Mapper)
• —📋 PlannerAgent (Policy Planning)
• —🛠️ NotebookBuilderAgent (Code Gen)
• —✓ DataQualityAgent (DQ Checks)
• —🧪 UnitTestAgent (Test Gen)
• —👁️ CodeReviewAgent (Review)
• —📈 MetricsAgent (Metrics)
• —📊 ProjectManagerAgent (Reports)
• —⭐ ReferenceStandardAgent (Standards)

All Commands Reference

• —✅ Core: auto, lead, analyze, plan, migrate
• —✅ ODS/S2T: analyze-ods, plan-ods, migrate-ods, s2t
• —✅ Batch: migrate-batch, convert, list-templates
• —✅ Utils: memory, status, clear-cache
• —✅ Options: --llm, --mock, --layers, --template, --yes

Runtime Mode Toggle

• —🟢 Mock Mode (default, no API key needed)
• —🟦 LLM Mode (with OpenAI API key)

🔧 Environment Variables (Optional)

If you want LLM mode enabled:

1. 1.Go to your Space Settings → Repository secrets
2. 2.Add:

   OPENAI_API_KEY = sk-...
   LLM_PROVIDER = openai

Otherwise, bot runs in Mock mode (fully functional, no API key needed).

🎨 Customization (Optional)

Edit files in the Space web editor:

Change colors (futuristic cyan/orange theme):

• —File: gradio_app.py
• —Lines: 18-36 (CSS variables)

Change title:

• —File: gradio_app.py
• —Line: ~710 (hero-title)

Update agent roles:

• —File: gradio_app.py
• —Search for "Agent Team" section

📋 What's Included

✅ Included

• —✅ Source code (src/ folder - all agents)
• —✅ Policies & instructions (.github/skill/)
• —✅ Skills knowledge pack (docs/skills/ - 70+ files)
• —✅ Notebook templates (references/databricks/)
• —✅ Empty runtime directories (artifacts/, inputs/)

❌ NOT Included (by design)

• —❌ Virtual environments (.venv/ - recreated by HF)
• —❌ Python caches (__pycache__/ - auto-generated)
• —❌ Generated notebooks (*.ipynb - created at runtime)
• —❌ API key secrets (use Space secrets instead)
• —❌ Sample DSX files (add your own to inputs/dsx/)

📊 Size Reference

🚨 Troubleshooting

App takes a long time to start

Answer: First deployment installs dependencies (30-60s). Subsequent loads are instant.

"ModuleNotFoundError: No module named 'src'"

Solution: Ensure src/ folder is at the root level (same as gradio_app.py)

"Port already in use"

No action needed: HF Spaces auto-assigns ports

Missing dependencies

Fix: Verify requirements.txt is in the root directory

API key not working

Check:

1. 1.Space Settings → Repository secrets
2. 2.Correct key format: sk-...
3. 3.Restart the Space (Settings → Restart)

📖 Full Documentation

For detailed deployment instructions, see:

• —HF_SPACES_DEPLOYMENT_GUIDE.md - Step-by-step with screenshots
• —PACKAGE_README.md - Package overview

🔐 Security Notes

✅ No sensitive data in package ✅ API keys stored securely in Space secrets ✅ Input validation: Paths restricted to inputs/ and artifacts/ ✅ Source of truth: CLI (data_engineering_agent.py) for full control

🎓 How It Works

User Input (Web UI)
    ↓
gradio_app.py (Gradio interface)
    ↓
Auto NLP Router (natural language → structured commands)
    ↓
Lead Agent (auto-detect workflow: DSX/ODS/S2T)
    ↓
Specialized Agents (Parse → Plan → Generate → Test → Review)
    ↓
Output (Notebooks in artifacts/, reports, metrics)

💡 Pro Tips

Team Access

• —Share the URL publicly - no login needed!
• —Anyone with the link can use it

Embed in Documentation

Use the HF Embed button to embed the bot in your docs/wiki

Custom Domain (Paid)

• —Upgrade to HF Pro
• —Add custom domain in Space settings
• —Share professional-looking URL

Monitoring

• —Check Space Logs if something breaks
• —Logs show error messages and startup info

🤝 Contributing

If you improve the bot:

1. 1.Update the Space repository
2. 2.Test changes locally first
3. 3.Commit & push to update the live bot

📞 Support

• —Hugging Face Docs: https://huggingface.co/docs/hub/spaces
• —This repo: GitHub Issues on your fork
• —Community: Hugging Face Discussions

📜 License

Apache License 2.0

🎉 Ready to Deploy?

1. 1.✅ You have this package
2. 2.✅ Create HF Space (3 clicks)
3. 3.✅ Upload files (drag & drop)
4. 4.✅ Your bot is live!

Questions? Check HF_SPACES_DEPLOYMENT_GUIDE.md for detailed help.

Go forth and automate! 🚀

Made with ❤️ for data engineers