CompileError/WebCoderBench

1from dataclasses import dataclass2from enum import Enum3 4 5@dataclass6class Task:7    benchmark: str8    metric: str9    col_name: str10 11 12# Select your tasks here13# ---------------------------------------------------14class Tasks(Enum):15    # task_key in the json file, metric_key in the json file, name to display in the leaderboard16    task0 = Task("anli_r1", "acc", "ANLI")17    task1 = Task("logiqa", "acc_norm", "LogiQA")18 19 20NUM_FEWSHOT = 0  # Change with your few shot21# ---------------------------------------------------22 23 24# Your leaderboard name25TITLE = """<h1 align="center" id="space-title">WebCoderBench</h1>"""26 27# What does your leaderboard evaluate?28INTRODUCTION_TEXT = """29"""30 31# Which evaluations are you running? how can people reproduce what you have?32LLM_BENCHMARKS_TEXT = """33## About WebCoderBench34 35Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate code generation capabilities and commercial potential. However, building a benchmark for LLM-generated web apps remains challenging due to:36 37- The need for real-world user requirements38- Generalizable evaluation metrics without relying on ground-truth implementations or test cases39- Interpretable evaluation results40 41---42 43## What Is Included44 45WebCoderBench is a real-world-collected, generalizable, and interpretable benchmark for web app generation.46 47**At a glance**48 49- **1,572** real-world user requirements50- Diverse modalities and expression styles reflecting realistic user intentions51- **24** fine-grained evaluation metrics across **9** perspectives52 53---54 55## How It Is Evaluated56 57WebCoderBench combines **rule-based evaluation** and the **LLM-as-a-judge** paradigm to enable fully automated, objective, and general evaluation.58 59To make the overall ranking more interpretable, we adopt **human-preference-aligned weights** over metrics to yield overall scores.60 61---62 63## Key Findings64 65Experiments across **12 representative LLMs** and **2 LLM-based agents** show that there is **no dominant model across all evaluation metrics**, offering opportunities for targeted optimization.66"""67 68EVALUATION_QUEUE_TEXT = """69## Some good practices before submitting a model70 71### 1) Make sure you can load your model and tokenizer using AutoClasses:72```python73from transformers import AutoConfig, AutoModel, AutoTokenizer74config = AutoConfig.from_pretrained("your model name", revision=revision)75model = AutoModel.from_pretrained("your model name", revision=revision)76tokenizer = AutoTokenizer.from_pretrained("your model name", revision=revision)77```78If this step fails, follow the error messages to debug your model before submitting it. It's likely your model has been improperly uploaded.79 80Note: make sure your model is public!81Note: if your model needs `use_remote_code=True`, we do not support this option yet but we are working on adding it, stay posted!82 83### 2) Convert your model weights to [safetensors](https://huggingface.co/docs/safetensors/index)84It's a new format for storing weights which is safer and faster to load and use. It will also allow us to add the number of parameters of your model to the `Extended Viewer`!85 86### 3) Make sure your model has an open license!87This is a leaderboard for Open LLMs, and we'd love for as many people as possible to know they can use your model 🤗88 89### 4) Fill up your model card90When we add extra information about models to the leaderboard, it will be automatically taken from the model card91 92## In case of model failure93If your model is displayed in the `FAILED` category, its execution stopped.94Make sure you have followed the above steps first.95If everything is done, check you can launch the EleutherAIHarness on your model locally, using the above command without modifications (you can add `--limit` to limit the number of examples per task).96"""97 98CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results"99CITATION_BUTTON_TEXT = r"""100"""101