CoolFace
Datasetpublic

Marti844/SaaS-Bench-docker

SaaS-Bench Docker Images Docker image archives for the SaaS-Bench benchmark — a suite of 23 self-hosted SaaS applications used to evaluate computer-use LLM agents on real, multi-step business workflows. This repository hosts the prebuilt .tar images (≈ 63 GB total) so you can reproduce the benchmark environment without rebuilding each app from source. The eval harness, task definitions, and verifiers live in the main SaaS-Bench repository. Paper: SaaS-Bench: Can Computer-Use… See the full description on the dataset page: https://huggingface.co/datasets/Marti844/SaaS-Bench-docker.

sourceHugging Faceapache-2.0updated 6d agoView on Hugging Face
0likes9.3kdownloads
Dataset Card

SaaS-Bench Docker Images

Docker image archives for the SaaS-Bench benchmark — a suite of 23 self-hosted SaaS applications used to evaluate computer-use LLM agents on real, multi-step business workflows.

This repository hosts the prebuilt `.tar` images (≈ 63 GB total) so you can reproduce the benchmark environment without rebuilding each app from source. The eval harness, task definitions, and verifiers live in the main SaaS-Bench repository.

Paper: SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?

Overview

SaaS-Bench evaluates browser-driving LLM agents on 106 task instances across 6 domains, running on 23 self-hosted SaaS applications. Each task asks the agent to complete a multi-step workflow (e.g. create a purchase order, configure a project board, schedule a patient visit); a per-task verify.py script then inspects the running application's state (DB rows, API responses, filesystem) and returns a pass/fail.

TrackDomainTasksRepresentative apps
uni-mBOF15Twenty, Bigcapital, HRMS, Pretix
uni-mHA16OpenEMR, OnlyOffice, OpnForm
uni-mSEPM31Baserow, OpenProject, code-server, Metabase
uni-mTCDW12OnlyOffice, Mattermost, RoundcubeMail, ownCloud
multi-mAASC12Grocy, farmOS, Recipya, e-label
multi-mIMC20SiYuan, Watcharr, BookLore, PhotoPrism, MediaCMS

Domains: BOF = Business Operations & Finance · HA = Healthcare & Administration · SEPM = Software Eng. & Project Mgmt. · TCDW = Team Comms & Document Workflows · AASC = Agriculture, Authoring & Supply Chain · IMC = Information Mgmt. & Creative.

Contents

23 Docker image archives (mw-*.tar) covering every app used by the benchmark.

FileApp / StackSize
mw-baserow.tarBaserow3.07 GB
mw-bigcapital.tarBigcapital2.94 GB
mw-booklore.tarBookLore1.55 GB
mw-code-server.tarcode-server8.46 GB
mw-elabel.tare-label2.00 GB
mw-farmos.tarfarmOS1.13 GB
mw-grocy.tarGrocy286 MB
mw-hrms.tarHRMS5.88 GB
mw-mattermost.tarMattermost (+Postgres)1.53 GB
mw-mediacms.tarMediaCMS1.76 GB
mw-metabase.tarMetabase889 MB
mw-onlyoffice.tarOnlyOffice (4-image stack)10.46 GB
mw-openemr.tarOpenEMR4.48 GB
mw-openproject.tarOpenProject2.27 GB
mw-opnform.tarOpnForm548 MB
mw-owncloud.tarownCloud2.14 GB
mw-photoprism.tarPhotoPrism3.82 GB
mw-pretix.tarPretix2.39 GB
mw-recipya.tarRecipya624 MB
mw-roundcubemail.tarRoundcube Mail1.34 GB
mw-siyuan.tarSiYuan Notes3.06 GB
mw-twenty.tarTwenty CRM2.11 GB
mw-watcharr.tarWatcharr250 MB

Each tar already contains the :latest tag; image names follow the mw-<app>[-<component>] convention so loaders can resolve them deterministically.

Download

python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="Marti844/SaaS-Bench-docker",
    repo_type="dataset",
    local_dir="docker/images",
    allow_patterns=["*.tar"],
)

Or with the CLI:

bash
hf download Marti844/SaaS-Bench-docker \
    --repo-type dataset --local-dir docker/images \
    --include "*.tar"

System requirements

  • —Disk: ≥ 130 GB free — ~63 GB for the archives plus the loaded images.
  • —RAM: ≥ 500 GB recommended if you run the full eval with the default 4-way parallelism — most stacks bundle their own DB / search / document-server, so total memory grows quickly under concurrency.
  • —Host OS: Linux. Tested on Ubuntu 22.04 and Alibaba Cloud Linux.
  • —Docker: 24+ with the compose plugin.

Licensing

  • —This card: Apache 2.0.
  • —Each bundled Docker image retains the license of its upstream project (e.g. OnlyOffice — AGPLv3, Mattermost — MIT/AGPLv3 dual, OpenEMR — GPLv3, etc.). The images are redistributed for benchmarking convenience only. Verify upstream terms before any non-research use.