ahagestedt/apex-devops

1spec_version: 12name: apex-devops3display_name: Apex DevOps Corp QA4domain: dev-sim5description: A technology operations company spanning project management, CI/CD, security,6  analytics, and collaboration tools. Evaluates customer support quality across 247  enterprise platforms.8tags:9- qa-evaluation10- dev-sim11- multi-step12- openenv13- rl-environment14env:15  action_type: rl_qa_base.models.QAAction16  observation_type: rl_qa_base.models.QAObservation17  state_type: rl_qa_base.models.QAState18scenarios:19- name: Jira (Atlassian)20  category: dev-sim21  description: Evaluate customer support quality for Jira (Atlassian)22  tasks: 2023- name: BMC Helix24  category: dev-sim25  description: Evaluate customer support quality for BMC Helix26  tasks: 1927- name: Databricks28  category: dev-sim29  description: Evaluate customer support quality for Databricks30  tasks: 1931- name: Jira Service Management32  category: dev-sim33  description: Evaluate customer support quality for Jira Service Management34  tasks: 1935- name: Okta36  category: dev-sim37  description: Evaluate customer support quality for Okta38  tasks: 1939- name: Asana40  category: dev-sim41  description: Evaluate customer support quality for Asana42  tasks: 1843- name: Slack44  category: dev-sim45  description: Evaluate customer support quality for Slack46  tasks: 1847- name: Snowflake48  category: dev-sim49  description: Evaluate customer support quality for Snowflake50  tasks: 1851- name: Tableau52  category: dev-sim53  description: Evaluate customer support quality for Tableau54  tasks: 1855- name: Canvas (Instructure)56  category: dev-sim57  description: Evaluate customer support quality for Canvas (Instructure)58  tasks: 1759- name: Confluence (Atlassian)60  category: dev-sim61  description: Evaluate customer support quality for Confluence (Atlassian)62  tasks: 1763- name: GitHub64  category: dev-sim65  description: Evaluate customer support quality for GitHub66  tasks: 1767- name: GitLab68  category: dev-sim69  description: Evaluate customer support quality for GitLab70  tasks: 1771- name: Jenkins72  category: dev-sim73  description: Evaluate customer support quality for Jenkins74  tasks: 1775- name: Marketo (Adobe)76  category: dev-sim77  description: Evaluate customer support quality for Marketo (Adobe)78  tasks: 1779- name: Pardot (Salesforce)80  category: dev-sim81  description: Evaluate customer support quality for Pardot (Salesforce)82  tasks: 1783- name: SharePoint84  category: dev-sim85  description: Evaluate customer support quality for SharePoint86  tasks: 1787- name: UiPath88  category: dev-sim89  description: Evaluate customer support quality for UiPath90  tasks: 1791- name: Box92  category: dev-sim93  description: Evaluate customer support quality for Box94  tasks: 1695- name: Microsoft Teams96  category: dev-sim97  description: Evaluate customer support quality for Microsoft Teams98  tasks: 1699- name: Zoom100  category: dev-sim101  description: Evaluate customer support quality for Zoom102  tasks: 16103- name: CrowdStrike Falcon104  category: dev-sim105  description: Evaluate customer support quality for CrowdStrike Falcon106  tasks: 14107- name: Blackboard108  category: dev-sim109  description: Evaluate customer support quality for Blackboard110  tasks: 10111- name: Splunk112  category: dev-sim113  description: Evaluate customer support quality for Splunk114  tasks: 10115verifiers:116- name: Billing Verifier117  type: rule-based118  description: Validates financial calculation correctness, invoice accuracy, and119    billing workflow compliance.120  params:121    accuracy_weight: 0.4122    completeness_weight: 0.3123    compliance_weight: 0.2124    efficiency_weight: 0.1125- name: API Integration Verifier126  type: rule-based127  description: Checks API endpoint correctness, request/response validation, and integration128    workflow accuracy.129  params:130    endpoint_weight: 0.4131    payload_weight: 0.3132    auth_weight: 0.2133    error_handling_weight: 0.1134- name: Compliance Verifier135  type: rule-based136  description: Validates regulatory compliance claims, policy adherence, and audit137    trail correctness.138  params:139    regulation_weight: 0.4140    documentation_weight: 0.3141    audit_weight: 0.2142    remediation_weight: 0.1143- name: QA Evaluation Judge144  type: llm-judge145  description: LLM-based assessment of customer support evaluation quality. Evaluates146    whether agent scoring and reasoning align with ground truth.147