> ## Documentation Index > Fetch the complete documentation index at: https://docs.eigenai.com/llms.txt > Use this file to discover all available pages before exploring further. # Demo Samples > A free sample of the Toolathlon dataset — single-turn, multi-app agent tasks over 32 MCP tool servers, with deterministic grading, SFT trajectories, and self-contained RL environments. These demo samples are a free **10-task slice** of the **Toolathlon** dataset — a corpus of single-turn, tool-using agent tasks set inside a shared **multi-application MCP workspace**. Each task gives the agent one user request spanning real-world services (Canvas LMS, GitHub, BigQuery, WooCommerce, arXiv, Notion, Google Workspace, and more), and grades the result **deterministically** on what the agent actually produced. The full dataset ships **4,300 RL environments** and **1,682 SFT trajectories** across **102 task families** — see the [Full Dataset](/products/eigendata-cli/datasets/toolathlon/overview) page. ## Overview | Property | Value | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | **Demo tasks** | 10 RL tasks across 10 families (one per family) | | **Demo SFT trajectories** | 10 graded-correct rollouts (`reward == 1.0`), one per task | | **Environment** | 32 MCP tool servers over one consistent world (the full \~32 GB shared environment ships with the full corpus) | | **Task format** | Single-turn: one user request → tool calls → deterministic grading | | **Grading** | Per-task deterministic grader — programmatic, no LLM judge | | **Trajectories** | OpenAI chat format with `reasoning_content` (chain-of-thought) + `tool_calls` | | **Full corpus** | 4,300 RL tasks · 1,682 SFT trajectories · 102 families — see [Full Dataset](/products/eigendata-cli/datasets/toolathlon/overview) | ## Environment Every task runs against **one reusable environment** — a multi-application workspace backed by **32 MCP tool servers** launched as local subprocesses, with each task overlaying its own mock data and launching only the servers it needs. The demo bundles the **10 tasks and their SFT trajectories**; the full \~32 GB shared environment (all 32 servers + fixtures) ships with the full corpus. ``` toolathlon/ ├── sft/ │ ├── trajectories.jsonl # 10 demo SFT rollouts, one JSON object per line │ └── stats.json # per-domain counts └── rl/ └── tasks/task_-rNNN/ # 10 demo task instances (one per family) ├── task.json # prompts, visible tools, limits, grading config ├── env/ │ ├── initial_workspace/ # files the agent starts with │ └── mock_mcp_output/ # task-specific mock data layered on the shared world └── verification/ ├── verify.py # the reward function └── grader_spec.json # the checks (the answer key) ``` The full corpus additionally ships `rl/shared/toolathlon/` — the one shared world all 4,300 tasks run against (`bundle.json` + `env/` \~32 GB + `mcp/` 32 servers + `tools/` registry + `verification_lib/` grader library). ## Data | Path | Description | | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `sft/trajectories.jsonl` | 1,682 SFT rollouts in OpenAI chat format — `tools`, `messages` (system / user / assistant / tool), and `metadata` | | `sft/stats.json` | Aggregate counts, per-domain breakdown, token distribution | | `rl/tasks//task.json` | Task spec: `system_prompt`, `user_prompt`, `env`, `tools[]`, `limits`, `metadata` | | `rl/tasks//env/initial_workspace/` | Files the agent starts with in its working directory | | `rl/tasks//env/mock_mcp_output/` | Task-specific mock data layered on the shared world | | `rl/tasks//verification/verify.py` | Deterministic reward function — reads the world end state, returns `[0.0, 1.0]` | | `rl/tasks//verification/grader_spec.json` | The concrete checks: file existence, CSV content, tool-call effects, service state | | `rl/shared/toolathlon/` | The shared environment — 32 MCP servers, fixture data, runner, grader library *(full corpus only; not bundled in the demo download)* | ## Sample The 102 families span education, research, data/ML, business, commerce, developer workflows, documents, and more — each instantiated across many scenario variants (the `rNNN` suffix): | Domain | Families | | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Education & LMS** | `canvas-arrange-exam`, `canvas-art-manager`, `canvas-art-quiz`, `canvas-do-quiz`, `canvas-homework-grader-python`, `canvas-list-test`, `canvas-new-students-notification`, `canvas-submit-late-work`, `course-assistant`, `course-schedule`, `courses-ta-hws`, `university-course-selection` | | **Research & academia** | `academic-pdf-report`, `academic-warning`, `add-bibtex`, `apply-phd-email`, `cvpr-research`, `find-alita-paper`, `hk-top-conf`, `paper-checker` | | **Data & ML** | `ab-testing`, `imagenet`, `llm-training-dataset`, `logical-datasets-collection`, `merge-hf-datasets`, `verl-dataset` | | **Business & finance** | `flagged-transactions`, `gdp-cr5-analysis`, `investment-decision-analysis`, `invoice-org`, `live-transactions`, `nvidia-market`, `nvidia-stock-analysis`, `oil-price`, `payable-invoice-checker`, `quantitative-financial-analysis`, `sales-accounting`, `stock-build-position`, `yahoo-analysis` | | **Commerce** | `filter-low-selling-products`, `price-comparison`, `woocommerce-customer-survey`, `woocommerce-new-product`, `woocommerce-new-welcome`, `woocommerce-product-recall`, `woocommerce-stock-alert`, `woocommerce-update-cover` | | **Productivity & communication** | `arrange-workspace`, `cooking-guidance`, `detect-revised-terms`, `dietary-health`, `email-paper-homepage`, `fillout-online-forms`, `game-statistics`, `identify-all-songs`, `interview-report`, `landing-task-reminder`, `meeting-assign`, `music-analysis`, `nhl-b2b-analysis`, `profile-update-online`, `set-conf-cr-ddl`, `student-interview`, `task-tracker` | | **Developer & infra** | `dataset-license-issue`, `git-bug-hunt`, `git-milestone`, `git-repo`, `k8s-safety-audit`, `personal-website-construct`, `sla-timeout-monitor`, `sync-todo-to-readme`, `youtube-repo` | | **Documents & analysis** | `excel-data-transformation`, `excel-market-research`, `ipad-edu-price`, `latex-prompt-box`, `machine-operating`, `ppt-analysis`, `privacy-desensitization`, `reimbursement-form-filler` | | **Travel & logistics** | `subway-planning`, `train-ticket-plan`, `travel-exchange`, `travel-expense-reimbursement`, `trip-adviser`, `trip-itinerary-generator`, `upenn-campus-route`, `search-ca-school` | | **Productivity apps** | `language-school`, `notion-find-job`, `notion-hr`, `notion-movies`, `notion-personal-website`, `update-material-inventory`, `vlm-history-completer`, `wandb-best-score`, `wandb-shortest-length`, `mrbeast-analysis`, `inter-final-performance-analysis`, `experiments-recordings`, `inventory-sync` | | Task | MCP servers | Description | | -------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ab-testing-r000` | BigQuery, Filesystem | Compute per-segment conversion rates from warehouse tables, fill a CSV, then conditionally create a GCS bucket or write a log entry based on the winner | | `canvas-do-quiz-r003` | Memory, Canvas, Filesystem | Look up a student identity, list unfinished quizzes across courses, fill a catch-up plan CSV, then actually submit each quiz on Canvas | | `find-alita-paper-r010` | arXiv, Filesystem, Scholarly | Search a paper catalog by title keyword and date cutoff, extract a code URL buried in the abstract, write a provenance card and a JSON record | | `invoice-org-r008` | PDF Tools, Filesystem, Yahoo Finance, Excel | Parse invoices from a document inbox, convert multi-currency amounts using a rate series, write a summary CSV and a JSON total | | `excel-data-transformation-r000` | Excel, Filesystem, Terminal | Flatten a stacked-banner-header workbook into a normalized table with derived columns | | `interview-report-r000` | Filesystem, Word | Read 6 per-item documents, synthesize an interview report in Word format | | `git-milestone-r000` | Filesystem, Terminal, Fetch | Vet open-source dependencies from a GitHub account — audit commits, releases, and license compliance | | `cooking-guidance-r000` | Filesystem, HowToCook | Plan a weekly dish lineup, pull recipes, and produce a consolidated grocery top-up list | | `machine-operating-r000` | BigQuery, Filesystem, Excel | Pull sensor readings from a wind-farm feed, identify calibration exceptions, produce a worklist | | `music-analysis-r000` | Excel, Google Sheets, Terminal, Filesystem | Analyze chart-streak data from a weekly grid, compute statistics, and build an A\&R briefing | **System prompt:** > Accessible workspace directory: /workspace/dumps/workspace \[...] > If you believe the task is completed, you can either call the `local-claim_done` tool or respond without calling any tool to indicate completion. **User prompt:** > We ran an A/B test on two product-recommendation email templates (A and B) across our catalog. Each catalog table in the warehouse logs per-window clicks and store\_views for one template variant. For each catalog and each variant, compute the conversion rate as total store\_views over total clicks, and fill the provided `segment_rates.csv` (keep its headers). Add one final OVERALL row per variant whose rate is the arithmetic mean of that variant's per-catalog rates (not the pooled totals). > > Then decide the winner by comparing the two OVERALL means. If template B's overall mean is higher, roll it out: create one GCS bucket named with prefix `winner-template-b`. Otherwise (A wins or a tie), make no bucket and instead write a single cloud\_logging entry with decision `template_A_retained` and note `No template change`. **MCP servers:** `google-cloud` (BigQuery + Cloud Storage + Logging), `filesystem` **Initial workspace:** a `segment_rates.csv` with headers and empty rate cells: ``` segment,variant,rate Apparel,A, Apparel,B, Books,A, ... overall,A, overall,B, ``` The agent must query BigQuery tables, compute rates, fill the CSV, compare the overall means, then take the correct conditional action (bucket creation or log entry — but not both). Each assistant message carries `reasoning_content` (chain-of-thought) alongside tool calls. This is a short task (3 tool calls) that illustrates the search → filter → write pattern. **System:** > Accessible workspace directory: /workspace/dumps/workspace \[...] **User:** > We're building provenance cards for the "Lumio" model series in the `papers` catalog. From the Lumio-titled records, take only those published on or before 2025-11-02, and pick the most recent of those. Write its provenance card to `paper_report.txt` (title / arxiv\_abs\_url / code\_url lines) \[...] Mirror the same id and abs link into `selected_record.json`. *** **Step 1 — search the paper catalog:** *Reasoning:* > We need to search the papers catalog for title containing "Lumio". Use arxiv\_local-search\_papers. *Tool call:* ```json theme={null} arxiv_local-search_papers({"query": "Lumio", "max_results": 20}) ``` *Result (4 papers found):* ```json theme={null} {"total": 4, "results": [ {"id": "1905.11946", "title": "Lumio: EfficientNet: ...", "published": "2025-12-12"}, {"id": "2605.05104v1", "title": "Lumio: Scaling Vision ...", "published": "2025-02-20"}, {"id": "2103.14023", "title": "Lumio: Swin Transformer ...", "published": "2024-11-21"}, {"id": "1409.4842", "title": "Lumio: Going Deeper with Convolutions", "published": "2025-05-11"} ]} ``` *** **Step 2 — filter by date, pick the most recent, write the report:** *Reasoning:* > Check dates: 1905.11946 published 2025-12-12 (after cutoff — exclude). 2605.05104v1 published 2025-02-20 (before). 2103.14023 published 2024-11-21 (before). 1409.4842 published 2025-05-11 (before). Among qualifying papers, the most recent is 1409.4842 (2025-05-11). Its abstract contains the code repo URL `https://github.com/Helios-Lab/Lumio`. *Tool call:* ```json theme={null} filesystem-write_file({ "path": "paper_report.txt", "content": "Lumio: Going Deeper with Convolutions\nhttps://arxiv.org/abs/1409.4842\nhttps://github.com/Helios-Lab/Lumio" }) ``` *Result:* `Successfully wrote to paper_report.txt` *** **Step 3 — write the JSON record:** *Tool call:* ```json theme={null} filesystem-write_file({ "path": "selected_record.json", "content": "{\"id\": \"1409.4842\", \"arxiv_abs_url\": \"https://arxiv.org/abs/1409.4842\", ...}" }) ``` *Result:* `Successfully wrote to selected_record.json` *** **Step 4 — final message:** > The provenance card for the most recent Lumio-titled paper published on or before 2025-11-02 has been created: **Lumio: Going Deeper with Convolutions** (arxiv.org/abs/1409.4842, code at github.com/Helios-Lab/Lumio). **Why it's hard:** The agent must (a) exclude one paper that falls after the date cutoff (a common off-by-one trap), (b) pick the most recent among the remaining three (not the first result), and (c) extract the code URL from the abstract text — it's not a stored field. The grader runs **6 concrete checks** — no LLM judge: | Check | Kind | What it verifies | | ----------------------------------------- | ----------------------- | --------------------------------------------------------------------- | | `file_exists:segment_rates.csv` | `file_exists` | The output CSV was written | | `csv_columns:segment_rates.csv` | `csv_columns` | Columns are exactly `segment`, `variant`, `rate` | | `csv_row_bounds:segment_rates.csv` | `csv_row_bounds` | Exactly 16 rows (7 segments × 2 variants + 2 overall) | | `tabular_expected_rows:segment_rates.csv` | `tabular_expected_rows` | Every row matches the gold values (e.g. `Apparel,A,70.936`) | | `mock_fired:storage_create_bucket` | `mock_tool_calls` | A bucket with prefix `winner-template-b` was created (because B wins) | | `mock_forbidden:logging_write_log` | `mock_tool_calls` | No log entry was written (wrong branch for B-wins) | The task passes only when **all 6 checks pass** (`score == 1.0`). A single wrong rate, a missing overall row, or taking the wrong conditional action fails the task. For `find-alita-paper-r010`, the grader checks that `paper_report.txt` contains the exact title (`Lumio: Going Deeper with Convolutions`), the correct arXiv URL (`arxiv.org/abs/1409.4842`), and the code repository link (`github.com/Helios-Lab/Lumio`). Picking the wrong paper or missing the GitHub URL from the abstract fails the task. ## Download ```bash theme={null} # Download Toolathlon demo data hf download jindidi/eigendata-demo-data --repo-type dataset --include "toolathlon/*" ``` View Toolathlon files For the complete Toolathlon corpus — all 4,300 tasks, 1,682 SFT trajectories, the full 32-server environment, and commercial licensing — see the [Full Dataset](/products/eigendata-cli/datasets/toolathlon/overview) page.