Hugging Face Publishes Harness Adapters Recipe: Qwen3.8-27B Rises from 39.2% to 64.7% on SWE-bench Pro with pi

Hugging Face describes harness adapters: LoRAs trained with RL inside the target agent harness. Qwen3.8-27B gains accuracy and makes fewer tool calls. Recipe and artifacts are public, but results are self-evaluated.

News date: About 8 min read

Contents
  1. 1. What do harness adapters change?
  2. 2. How were the training data and pipeline built?
  3. 3. How well does it work? How do the specialized adapters differ from the super-adapter?
  4. 4. Does it transfer across harnesses or to other tasks?
  5. 5. What should you know if you want to try it yourself?

On November 20, 2026, Hugging Face published an article introducing harness adapters. An agent harness is the execution framework that lets an AI model read and write files, run commands and call tools to complete coding tasks, such as pi, Hermes Agent and OpenCode. The approach trains a LoRA adapter inside a specific harness using reinforcement learning (RL, where the model tries repeatedly and learns from rewards based on outcomes). A LoRA leaves the original model weights unchanged and attaches a small set of extra parameters. The goal is to fit Qwen3.8-27B more closely to that harness and task type. Hugging Face says it has released the full training recipe, and the artifacts (adapters, GGUF model files, RL tasks, the safeagents plugin) can be browsed in its Hugging Face collection.

What do harness adapters change?

Hugging Face points out that models are usually trained for general performance, not optimized for a specific harness and real tasks. The article opens with an example: Qwen3.8-27B scores 39% on Terminal-Bench 2.1 (an agentic coding benchmark) in the pi harness, averaging 2.9K tokens per task (a token is the basic unit of text a model processes). With frozen weights plus an RL-trained LoRA adapter, pass@1 (success rate with one attempt per task) is 55%, averaging 1.5K tokens. Hugging Face says the main reasons are that the model reasons less, calls tools less, and does better on the first try.

According to Hugging Face, most RL frameworks train on a reimplemented agent loop. This approach instead runs the harness as-is, and the trainer learns from the tokens actually produced; those tokens are recorded by a proxy between the harness and vLLM (a model inference server). The versions used are Pi 1.5.2, OpenCode 2.2.7 and Hermes Agent v0.22.6.

How were the training data and pipeline built?

  • Three task sources: SWE-smith (bugs injected into Python and Go projects), SWE-Gym (real GitHub issues, packaged via TaskTrove), and tasks generated with Tasksmith from merged pull requests.
  • Validation: after filtering with openenv validate (a tool that checks tasks and graders are correct), 858 of 1,127 Hub candidate tasks passed.
  • Difficulty filtering: of 442 graded tasks, the base model solved 151 every time and never solved 120. The remaining 171 were solved only sometimes and were kept as eligible tasks, because tasks that are always right or always wrong give no learning signal.
  • The dataset harness-hub/agentic-coding-envs contains 871 tasks from 35 repositories.
  • Training: each harness adapter used 2 NVIDIA B200 GPUs (one running vLLM, one training), with LoRA in an all-linear configuration. Rollouts (the model's actual attempts during training) ran with thinking mode off (the mode where the model writes out reasoning before answering). The reward was correctness as judged by the verifier plus a 0.1 tool-efficiency reward; the final learning rate was 2e-5.
  • Safety: safeagents (a tool that intercepts suspicious model actions) was attached during training, checking 1.3 million tool calls and blocking 647.

How well does it work? How do the specialized adapters differ from the super-adapter?

Hugging Face says the three specialized adapters beat the base model on both benchmarks in their own harnesses. On tasks that both the base model and the adapter solved, tool calls fell 56–72% on SWE-bench Pro and 53–70% on Terminal-Bench 2.1. On Terminal-Bench 2.1, pi improved by 16.3 percentage points, hermes by 14.1, and opencode by 6.0 overall.

The super-adapter uses PEFT's cat method to concatenate the three adapters along the rank dimension. Rank can be understood as the adapter's capacity; after merging it is 96, three times the 32 of a single adapter. The final choice uses weight ½, with checkpoints pi step 160, hermes step 160 and opencode step 150. The table below shows Hugging Face's comparison of merge weights on Terminal-Bench 2.1 using step-120 adapters.

Terminal-Bench 2.1, all 89 tasks, merge weight comparison for step-120 adapters (source: Hugging Face)
SettingpihermesopencodeAll three combined / tool calls per task
Base model39.1%46.0%44.8%43.3% / 49.2
Weight ⅓ (average)49.4%51.2%47.1%49.2% / 35.6
Weight ½ (final choice)48.8%53.5%55.2%52.5% / 25.9
Weight 1 (direct sum)40.2%62.8%47.1%50.0% / 13.4

Hugging Face says that with thinking off, the super-adapter beats the base model across all six harness–benchmark combinations (average pass@1 rises from 39.5% to 49.0%). On tasks both solved, it makes 53–73% fewer tool calls. With thinking on, it beats the equivalent base model with thinking in five of six combinations and ties in one (average 55.5% rising to 63.5%), even though thinking was not enabled during training. If you use only a single harness with thinking off, Hugging Face recommends that harness's specialized adapter, since the specialized versions lead in four of six combinations.

Does it transfer across harnesses or to other tasks?

Hugging Face tested the pi adapter in other harnesses. On Terminal-Bench 2.1 in OpenCode, the score rose from 44.8% to 54.0% (the OpenCode-specific version went from 46.4% to 52.4%); in Hermes it rose from 46.3% to 56.1% (the specialized version from 44.7% to 58.8%). On the 300-task AutomationBench (business automation workflow tasks, using that benchmark's own agent loop), the base model scored 23.0%, pi 27.0%, opencode 25.3%, the merged version 23.7%, and hermes 17.7%. Hugging Face considers most differences to be within noise: RL neither clearly transferred to that setting nor broke the model.

What should you know if you want to try it yourself?

Hugging Face says it has released the full recipe and encourages developers to build RL tasks from their own codebases and apply them to their preferred models and harnesses. The article also notes that any Harbor-format task can be validated with openenv validate plus the --sandbox hf-sandbox option.

Frequently asked questions

What is a harness adapter?

According to Hugging Face, it is a LoRA adapter trained with RL inside the target agent harness. The base model weights stay frozen and only a small set of extra parameters is added, making the model better suited to that harness and task distribution.

With a single harness, should I choose a specialized adapter or the super-adapter?

Hugging Face recommends that if you use only one harness with thinking off, you use that harness's specialized adapter, since the specialized versions lead in four of six tests. The super-adapter can run unchanged under pi, Hermes Agent and OpenCode.

Which should I use with thinking mode on?

Hugging Face says that although the super-adapter was not trained with thinking on, it is the strongest model in its evaluations with thinking enabled. It beats the base model with thinking in five of six combinations and ties in one; on SWE-bench Pro in pi, both score 74%.

Where can I find the artifacts and data?

Hugging Face says the adapters, GGUF files, RL tasks and safeagents plugin can all be browsed through its Hugging Face collection. The task dataset is harness-hub/agentic-coding-envs, the safeagents plugin is harness-hub/qwen-safeagents-plugin, and the model trajectories from the final evaluation are in the public bucket harness-hub/harness-adapters-trajectories.

What hardware was used for training and evaluation?

Hugging Face says each harness adapter was trained on 2 NVIDIA B200 GPUs, one running vLLM and one handling training. For evaluation, each LoRA checkpoint was first merged into the base weights and then served with vLLM on a single GPU.

Browse the latest news in this topic

Found this guide helpful? Share it with someone who needs it

Sources

Compiled from the references and verification dates listed above; fares, schedules, product features, and terms can change, so please confirm the latest details on the relevant official website before travelling, booking, or purchasing.

Lifestyle