devaatmik/shopify-store-audit

Shopify Store Audit & Remediation — OpenEnv Environment

Train AI agents to find and fix real e-commerce store issues through the Shopify Admin API.

Motivation

Store auditing is a $5K–$15K consulting service that Shopify merchants regularly pay for. Every store accumulates issues: missing product descriptions, broken pricing, SEO gaps, inventory discrepancies, empty collections, stuck orders. This environment uses real Shopify product data (45 products from actual CSV exports) and lets AI agents learn to diagnose and fix them through API operations that map 1:1 to Shopify Admin GraphQL mutations.

Why this matters for the agent community:

• —Real data — 45 products loaded from real Shopify CSV exports (apparel + jewelry catalogs)
• —184 discoverable issues — auto-scanned from real data quality gaps + synthetic injections
• —Randomised episodes — different issues sampled each reset (seeded for reproducibility)
• —Shaped rewards — discovery, partial fix, efficiency bonus, regression & repetition penalties
• —Genuine difficulty progression — hint level scales from guided to fully autonomous exploration
• —18 API commands mirroring real Shopify Admin GraphQL mutations

How It Works

The environment loads real Shopify product exports (apparel.csv, jewelery.csv) containing 45 products across apparel, bags, footwear, jewelry, outdoor gear, and home goods. An IssuePool scans the catalog and discovers real data quality issues (0/45 products have SEO titles, 0/45 have image alt text, 20/20 jewelry products have no SKUs, plus handle typos and formatting artifacts). Synthetic issues (corrupted prices, draft products, negative inventory) are generated on top.

On each reset(seed=N), the pool randomly samples 8/12/20 issues depending on the task. Different seed = different bugs. The agent must discover and fix them through API commands.

Difficulty Tiers

The three tasks aren't just "more items" — they differ in how much the agent is told:

Reward Function

Multi-signal shaped reward that provides gradient throughout the episode:

This means a weak agent that explores but fails to fix still earns discovery rewards. A careless agent that breaks things gets punished. A perfect agent earns close to 1.0.

Action Space

Actions are JSON objects with a command and params:

{"command": "update_product_seo", "params": {"product_id": "ayers-chambray", "seo_title": "Ayres Chambray | Store"}}

Observation Space

Baseline Scores

The hard task genuinely challenges frontier models. An agent trained via RL on this environment would need to learn exploration strategies that gpt-4o doesn't exhibit out of the box.

Setup Instructions

Prerequisites

• —Python 3.10+
• —Docker
• —openenv-core (pip install openenv-core)

Local Development

cd /path/to/project
pip install -e .

# Start server
uvicorn server.app:app --host 0.0.0.0 --port 8000

# Test
curl http://localhost:8000/health
curl http://localhost:8000/tasks
curl -X POST http://localhost:8000/reset -H "Content-Type: application/json" -d '{}'

Docker

docker build -t shopify-store-audit .
docker run -p 8000:8000 shopify-store-audit

Run Inference

export API_BASE_URL="https://api.openai.com/v1"
export MODEL_NAME="gpt-4o"
export HF_TOKEN="your-api-key"
export ENV_URL="http://localhost:8000"

python inference.py

Validate

openenv validate

Shopify API Mapping

Every environment command maps to a real Shopify Admin GraphQL operation:

Agents trained here learn patterns directly transferable to real Shopify store management via Shopify MCP or Shopify CLI.

Architecture

├── apparel.csv, jewelery.csv    # Real Shopify product exports (45 products)
├── models.py                    # Pydantic Action & Observation types
├── client.py                    # EnvClient for WebSocket connection
├── openenv.yaml                 # OpenEnv spec metadata
├── pyproject.toml               # Dependencies
├── Dockerfile                   # Container definition
├── inference.py                 # Baseline agent (runs all 3 tasks)
├── test_live.py                 # WebSocket integration test
└── server/
    ├── app.py                   # FastAPI + /tasks + /grade endpoints
    ├── shopify_store_audit_environment.py  # Environment (reset/step/state)
    ├── store.py                 # CSV loader, IssuePool, ShopifyStore CRUD
    ├── tasks.py                 # TaskConfig (num_issues, hint_level, categories)
    └── graders.py               # Per-task grading functions

Extensibility

The architecture supports connecting to a real Shopify store via the Admin GraphQL API. The ShopifyStore class can be subclassed with a LiveShopifyStore that makes real API calls instead of in-memory mutations. Environment variables SHOPIFY_STORE_URL and SHOPIFY_ACCESS_TOKEN would enable live mode. The action space and observation format remain identical — the agent doesn't know which mode it's in.

License

MIT