arbor

Arbor — Autonomous Optimization Skill for Hypothesis Tree Refinement

Skill Overview

Arbor is an autonomous optimization skill that implements the Hypothesis Tree Refinement (HTR) method. It helps you iteratively improve code, models, agent frameworks, or data pipelines through structured experiment management, avoiding overfitting while accumulating reusable insights.

Applicable Scenarios

1. Model Training Optimization

When you need to adjust an optimizer, architecture, or training recipe to reduce loss or reach a target in fewer steps, Arbor can automatically organize multiple experiments, compare different hypotheses, and merge only test-validated improvements into the main branch. It is suitable for training tasks that require repeated hyperparameter tuning.

2. Agent Framework Engineering

If your agent loop, search framework, or tool-use architecture needs improved success rates or accuracy, Arbor manages parallel experiments through a hypothesis tree, turning every attempt into auditable evidence and preventing the loss of experience caused by blind trial and error.

3. Benchmark Competition Optimization

For “improve the submission” tasks in MLE-bench or Kaggle-style settings, Arbor’s development/test separation mechanism effectively prevents overfitting to the development set, ensuring that improvements also work in the real test environment.

Core Features

1. Hypothesis Tree State Management

Use the scripts/tree.py tool to manage the entire optimization process: create hypothesis nodes, record experimental evidence, propagate insights to parent nodes, and prune failed directions. The hypothesis tree serves as persistent research memory, ensuring that decisions remain grounded in the complete experiment history even after context is compressed during long-running tasks.

2. Coordinator–Executor Model

As the coordinator, you own the hypothesis tree and decide the search direction. Short-lived executor sub-agents test individual hypotheses in isolated git worktrees and report their results. Isolation ensures that parallel experiments do not interfere with one another, keeping exploratory changes separate until they pass merge gating.

3. Development/Test Separation for Merge Gating

Use the development evaluator to freely guide the search direction, but allow only improvements validated by an independent test evaluator to become the new best version. This mechanism detects development/test divergence: candidates with high development scores but low test scores are treated as evidence that the candidate is exploiting the feedback signal rather than producing a transferable improvement.

Frequently Asked Questions

What kinds of tasks is the Arbor skill suitable for?

Arbor is appropriate when you have a modifiable artifact, a clear objective, an automatically scorable evaluator, and an expectation of running multiple experiments. Typical scenarios include model training and hyperparameter tuning, agent system optimization, data pipeline improvements, and benchmark competitions. If you only need a one-time fix or a single answer, complete the task directly. If you need open-ended ideation without an evaluator, use another skill.

How does Arbor prevent overfitting during optimization?

Arbor prevents overfitting through strict development/test separation: the development evaluator guides the search, while the test evaluator is used only for merge gating. Strong performance on the development set followed by failure on the test set is treated as a warning that the direction may be exploiting the feedback signal rather than producing a transferable improvement. If there is no clear development/test split, the system requires one to be established first.

How does Hypothesis Tree Refinement differ from traditional optimization methods?

The core of HTR is that research state is persisted in a hypothesis tree rather than in the conversation history. Each node is tied to a hypothesis, distilled insights, and the corresponding artifact version. Arbor’s advantage comes from how it organizes the budget—maintaining competing hypotheses, comparing sibling nodes, and passing lessons between them—rather than from spending more tokens. Structured search outperforms brute-force sampling because each experiment is conditioned on the knowledge already present in the tree.