--- title: "Task capabilities: values and benchmark evidence" description: "Describes task capability values and the benchmark evidence that informs model routing for each capability." lang: en status: public-preview lastUpdated: 2026-10-21 url: https://duale.ai/en/docs/model-routing/task-capabilities --- ## AI-generated summary This page covers task capability selection, benchmark evidence mapping, response-quality benchmarks, permissiveness preferences, and scope caveats for model routing. - Task capability values and their intended results. - Benchmark evidence mapped to General and task capabilities. - Response-quality benchmarks for factual reliability and safety. - Target permissiveness for willingness to answer allowed requests. - Scope caveats for comparing benchmark measurements. Summaries were generated by AI. --- Choose task capabilities to describe the work your application needs, without choosing a model by name. Set `required_capabilities` for skills the Task depends on and `preferred_capabilities` for skills that would help. Both are best-effort routing preferences. Routing prefers evidence for every required capability and can fall back to the best available route when that combination is unavailable. A preferred capability improves a route's preference without excluding other routes. ## Choose a capability by the intended result | Value | Use it for | Example | | ----------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ | | `problem_solving` | Deriving conclusions through several logical or scientific steps. | Explain which hypothesis follows from a set of constraints. | | `math` | Mathematical problem solving and formal proofs. | Solve a competition-style problem or construct a checkable proof. | | `code` | Generating, inspecting, modifying, debugging, or migrating software. | Implement a function, review a patch, or port a program. | | `agentic` | Completing multiple steps with available Tools and feedback. | Research a question, inspect results, and revise a deliverable. | | `long_context` | Using information spread across extensive supplied material. | Reconcile facts from distant sections of a document set. | | `finance` | Financial analysis, modeling, and banking workflows. | Compare financial statements or prepare a spreadsheet model. | | `cyber_defensive` | Reproducing and remediating software vulnerabilities. | Reproduce a reported defect and prepare a corrective patch. | | `cyber_offensive` | Assessing whether a vulnerability can be exploited within an authorized scope. | Evaluate whether a known vulnerability can be exploited in a test environment. | | `medical` | Medical work; task-performance evidence focuses on clinical documentation. | Turn supplied encounter notes into a structured clinical document. | | `legal` | Legal analysis and reviewable legal documents. | Compare clauses or prepare a source-grounded review draft. | `problem_solving` emphasizes the steps needed to reach a conclusion. `long_context` adds the need to use extensive input. `agentic` adds iterative work with Tools and feedback. Combine them when the Task needs those distinct skills. General is the blended comparison score that the **LLM Coach** view of the Dashboard shows for your routes. It remains useful for comparing and sorting models, but is not a task capability. For a new Task without a dominant specialization, omit capability preferences. Broad benchmark evidence still informs routing. Instruction-following evidence measures a model's performance under benchmark conditions. Use a response schema and your application's checks when an output must satisfy a deterministic contract. ## Express the work the Task needs For conclusions that require combining evidence across several long documents, use this `RoutingPolicy` JSON excerpt: ```json { "required_capabilities": ["long_context"], "preferred_capabilities": ["problem_solving"] } ``` For a code change with iterative Tool use, require `code` and prefer `agentic`. Choose the capabilities that describe the work; adding every capability does not make a Task better specified. The SDK's `capabilities=` shorthand applies only when you omit `routing=`. See [application integration](https://duale.ai/en/docs/model-routing/application-integration.md) for a complete SDK example. In a continuation, omitted fields keep the parent's preferences. The [SDK reference](https://duale.ai/en/docs/sdk/reference.md#routingpolicy) explains how to override or clear them. Task capabilities do not enable Tools, authorize data access, or declare image or structured-output support. Use [route settings](https://duale.ai/en/docs/model-routing/provider-capabilities.md) for endpoint features, [Tools](https://duale.ai/en/docs/tools.md) for Tool access, and the [routing contract](https://duale.ai/en/docs/model-routing/contract-and-limits.md#separate-hard-requirements-from-preferences) for hard requirements. ## See the benchmark evidence for General and task capabilities The following associations distinguish General comparison evidence from named task preferences. A benchmark contributes when a measurement is available for a configured model. Its appearance here does not mean every model, Model Route, or customer pool has that measurement. | Benchmark | Comparison and task scopes | Evaluated work | | -------------------------------------------------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | MMLU-Pro | General, `problem_solving` | Broad academic knowledge and reasoning. | | Humanity's Last Exam (HLE) | General, `problem_solving` | Advanced multidisciplinary questions; coverage depends on the evaluated subset and mode. | | GPQA Diamond | General, `problem_solving` | Graduate-level scientific questions. | | [MMMU-Pro](https://arxiv.org/abs/2409.02813v3) | General, `problem_solving` | Multimodal understanding and visual reasoning, used as indirect evidence for broader reasoning work. | | IFBench | General | Following detailed, verifiable instructions. | | AA-Omniscience Accuracy | General | Factual knowledge; this is the accuracy measure. | | SimpleQA | General | Factual recall without external tools. | | AA-LCR, historical version; AA-LCR v1.1 | `long_context` | Reasoning over long inputs. The two versions are separate measurements. | | AIME 2025 | `math` | Competition mathematics. | | ProofBench v1; ProofBench v1.1 | `math` | Machine-checked Lean 4 theorem proving. Results from the two versions are not directly comparable. | | [CritPt](https://arxiv.org/abs/2509.26574v1) | `problem_solving` | Research-level physics reasoning across modern physics disciplines. | | [SciCode](https://arxiv.org/abs/2407.13168v1) | `code` | Scientific programming and implementation of scientific methods. | | [LiveCodeBench](https://arxiv.org/abs/2403.07974v2) | `code` | Code generation for recent programming problems; the evaluation window matters. | | DeepSWE v1.1 | `code`, `agentic` | Software-engineering tasks completed through iterative work. | | Vals Code Migration | `code`, `agentic` | CLI and COBOL-to-Java migration, evaluated through behavioral tests. | | Terminal-Bench Hard; Terminal-Bench 2.0, 2.1, and 4.0 | `agentic` | Multi-step terminal tasks. Each version remains distinct; the 4.0 measurements use mini-swe-agent. | | Terminal-Bench-Science 0.1 | `agentic` | Scientific workflows with mini-swe-agent; attempt counts and retry policies can differ between evaluations. | | τ²-Bench Telecom | `agentic` | Telecommunications workflows with tools and interaction. | | Berkeley Function Calling Leaderboard (BFCL), overall accuracy | `agentic` | Function selection and argument generation; interpret the result in its recorded evaluation version. | | GDPval-AA v2; GDPval-AA v2.1 | `agentic` | Professional work evaluated through Elo ratings. Compare ratings within the same version. | | BrowseComp-Plus Search Profile v1 | `agentic` | Research with search tools under the specified evaluation profile. | | [Vals Web Search Index (Exa)][vals-web-search] | `agentic` | Multi-step finance and legal research using the same Exa search tool across evaluated models. | | Toolathlon-Verified | `agentic` | Reliability across extended tool-use tasks. | | [Vending-Bench 2](https://andonlabs.com/evals/vending-bench-2) | `agentic` | Sustained operation of a simulated vending business, measured by final bank balance. | | EnterpriseOps-Gym-AA | `agentic` | Enterprise workflows. | | AutomationBench-AA | `agentic` | Business-workflow objectives completed without benchmark guardrail violations. | | MCP-Atlas | `agentic` | Tool use across multiple servers. | | APEX-Agents-AA | `agentic` | Extended professional-services tasks. | | Finance Agent v2 | `finance` | Financial-analysis tasks. | | τ³-Banking | `finance` | Multi-step banking workflows with tools. | | Excel Modeling Benchmark | `finance` | Financial modeling in spreadsheets. | | CyberGym Level 1 | `cyber_defensive` | Reproducing software vulnerabilities. | | CyberBench PATCH | `cyber_defensive` | Vulnerability remediation. | | SEC-Bench Pro | `cyber_offensive` | Vulnerability discovery and proof-of-concept generation. | | ExploitBench | `cyber_offensive` | Machine-verified browser exploitation capabilities. | | ExploitGym | `cyber_offensive` | Turning known software vulnerabilities into code execution. | | MedScribe | `medical` | Clinical documentation; it does not cover every medical task. | | Harvey LAB-AA, all-pass rate without a hallucination gate | `legal` | Legal work products. Dataset and judging versions matter; this measure is distinct from a hallucination-gated headline score. | [vals-web-search]: MMMU-Pro's visual tasks provide evidence of reasoning across images and text. A score does not establish endpoint image support; [Images](https://duale.ai/en/docs/model-routing/images.md) explains how to declare it. ## Understand the evidence for response quality These benchmarks provide response-quality evidence without adding capability values. They measure factual reliability and appropriate response behavior; they do not establish that a particular answer is correct or safe. | Benchmark | Evaluated behavior | How to interpret its relevance | | ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | [AA-Omniscience Hallucination Rate](https://arxiv.org/abs/2511.13029v1) | Incorrect answers among responses that were not fully correct. Lower is preferable. | Read alongside AA-Omniscience Accuracy; abstention can reduce hallucinations without answering the question. | | [HELM Safety](https://crfm.stanford.edu/2024/11/08/helm-safety.html) | A composite evaluation across several risk categories. Higher indicates lower measured risk. | This is not a refusal rate. | | [FORTRESS average risk](https://arxiv.org/abs/2506.14922v1) | Harmful criteria met by answers to adversarial requests. Lower is preferable. | Separate from FORTRESS's benign over-refusal measure. | | [StrongREJECT harmfulness](https://arxiv.org/abs/2402.10260v2) | How useful an answer is for carrying out a forbidden request. Lower is preferable. | Measures harmful assistance beyond whether an answer contains a refusal phrase. | | [SAFE-Teen health-advice failure rate][safe-teen] | Clinical boundaries in conversations with teens aged 13–17. Lower is preferable. | Applies when `medical` is requested; complements medical documentation evidence. | | [SAFE-Teen urgent-safety failure rate][safe-teen] | Recognizing danger and directing teens to immediate help. Lower is preferable. | Applies when `medical` is requested; does not establish clinical competence. | [safe-teen]: SAFE-Teen results inform `medical` routing. Duale AI uses the results of the October 2026, 106-scenario evaluation without an added system prompt. Partial ratings count as non-failures, so the complement of either failure rate is not a strict pass rate. These simulated teen conversations do not establish how a model will behave in adult care. ## Choose willingness to answer allowed requests Use `target_permissiveness` to express the preferred willingness to answer sensitive but allowed requests. Use `0` to favor withholding or qualification, `0.5` to balance withholding and directness, and `1` to favor direct engagement. Set `null` for no specific preference. The following measurements inform willingness across task types. FORTRESS evaluates benign requests; SpeechMap provides indirect evidence from engagement with sensitive political requests. | Benchmark measurement | Evaluated behavior | Relevance to the preference | | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------------------------------------- | | [FORTRESS over-refusal](https://arxiv.org/abs/2506.14922v1) | Refusal of benign requests paired with adversarial requests. | Less over-refusal indicates greater willingness to answer allowed requests. | | [SpeechMap complete](https://github.com/xlr8harder/llm-compliance/blob/d1cae524908790b2e25a05638f4b21ec82884403/README.md) | Complete answers in the evaluated political-request corpus. | A complete answer need not be correct or appropriate. | | [SpeechMap evasive](https://github.com/xlr8harder/llm-compliance/blob/d1cae524908790b2e25a05638f4b21ec82884403/README.md) | Evasive or deflective answers in that corpus. | Engagement that stops short of a complete answer. | | [SpeechMap denial](https://github.com/xlr8harder/llm-compliance/blob/d1cae524908790b2e25a05638f4b21ec82884403/README.md) | Refused requests in that corpus. | The label alone does not establish that refusing was wrong. | [SpeechMap's categories](https://github.com/xlr8harder/llm-compliance/blob/d1cae524908790b2e25a05638f4b21ec82884403/README.md) describe response outcomes from the same evaluation. They do not certify that every evaluated request was allowed. The preference does not set your application's allowed-use policy or measure protection against harmful requests. [Routing contract and limits](https://duale.ai/en/docs/model-routing/contract-and-limits.md) defines the setting and its limits. ## Apply the evidence within its measured scope Benchmarks differ in task coverage, evaluation tools, reasoning effort, attempt budgets, and scoring. Some results use private tasks or provider-reported measurements. A benchmark name alone does not establish that two scores are comparable or independently reproducible. Review the Model Route's **Benchmarks** fields for available measurements and their units. Source and evaluation details can be incomplete; use recorded versions and conditions when assessing comparability. Historical AA-LCR, ProofBench v1, and GDPval-AA v2 results do not become measurements of their newer versions. The stored Harvey LAB-AA measure is the all-pass rate before any hallucination gate; do not infer a dataset or judging revision from that name alone. Its coverage does not establish performance across every jurisdiction or legal task. A missing benchmark measurement is an evidence gap, not a measured zero. The available evidence can change as your model pool changes. Test the Task outcomes you need, including the fallback result, through the checks in [application integration](https://duale.ai/en/docs/model-routing/application-integration.md#validate-the-outcome-not-the-route). ## Related content - [Agent harness: Task execution model and Tool Call outcomes](https://duale.ai/en/docs/agent-harness.md) - [Product: Task execution and platform capabilities](https://duale.ai/en/product.md) - [Task lifecycle: submission, outcomes, and continuations](https://duale.ai/en/docs/sdk/concepts.md) - [Model routing contract and limits](https://duale.ai/en/docs/model-routing/contract-and-limits.md) - [Data protection: stored copies, Task reads, and data paths](https://duale.ai/en/docs/security/data-protection.md) - [Application integration: routing and outcomes](https://duale.ai/en/docs/model-routing/application-integration.md) --- ## Sitemap See the full [Markdown sitemap](https://duale.ai/sitemap.md) for all pages.