SCALAR

A SCALAR agent in Craftax-Classic. The agent composes learned skills to gather resources, craft tools, and collect diamond—a task requiring a sequence of prerequisite achievements while maintaining health, hunger, and thirst.

Abstract

A core challenge in reinforcement learning (RL) is effective exploration, particularly for long-horizon tasks. Recent approaches have explored the utility of large language models (LLMs), combining capabilities to decompose objectives into skills and generate code such as rewards and verifiers. However, ad hoc prompt and program designs, as well as their reliance on single proxy rewards, can lead to reward hacking and hallucinations. Furthermore, synthesizing the correct functions remains challenging without actual environment interactions. To address these challenges, we propose SCALAR, an iterative, bi-directional framework that couples an LLM planner and low-level RL controllers through a skill library. The skill library is a set of skills that, when composed, define a set of furthest reachable states by the current agent. In SCALAR, the library is iteratively expanded by a high-level LLM planner in conjunction with low-level RL agents. In one direction, an LLM planner uses information in the skill library to propose new skills with (1) preconditions reachable through existing skill compositions and (2) termination conditions unachievable by current skills. In the other direction, the LLM planner refines its world knowledge concurrently with RL training by analyzing successful RL trajectories through Pivotal Trajectory Analysis. We evaluate SCALAR on the Crafter benchmark, achieving 86.3% diamond-collection success, more than doubling the previous state-of-the-art.

Key Results

SCALAR iteratively expands a frontier of reachable states. An LLM proposes the next frontier skill σ based on world knowledge (knowledge base K and skill library Σ). Existing skills are dynamically composed to reach σ's preconditions, then RL trains the new skill. Successful trajectories are analyzed to refine the knowledge base, closing the loop.

Method

SCALAR grounds skill discovery in the agent's evolving understanding of reachable states through experience. Rather than proposing skills from specifications alone, we systematically expand an exploration frontier—the boundary of states achievable with the current skill library. New skills are proposed to extend this frontier: their preconditions must be satisfiable from currently reachable states, while their effects must contribute novel capabilities beyond what existing skill compositions can achieve.

SCALAR's five-stage pipeline: (1) The LLM generates STRIPS-style skill operators from the knowledge base, (2) converts them to executable options with initiation/termination conditions and rewards, (3) composes existing skills to reach the new skill's preconditions, (4) trains the skill policy via RL, and (5) analyzes successful trajectories to refine the knowledge base.

1. Skill Proposal on the Frontier

The LLM proposes new skills by generating STRIPS-style operators σ = ⟨PRE, EFF⟩ that specify symbolic preconditions and effects. These operators form a dependency graph where each skill's preconditions may depend on the effects of other skills.

The LLM queries the knowledge base to propose a new skill with symbolic preconditions (e.g., "has wood, has iron, near crafting table") and expected effects (e.g., "+1 iron pickaxe").

Each symbolic operator is then converted into an executable option o = ⟨I, π, β⟩ by generating initiation conditions, a trainable policy, and termination conditions.

Converting a symbolic skill specification into an executable option with code for initiation conditions, reward shaping, and termination verification.

Before training, we verify each proposed skill against the current frontier:

• Novelty: The skill's effects must reach states beyond what existing skill compositions can achieve
• Feasibility: The skill's preconditions must be satisfiable by composing existing skills

2. First-Return-Then-Explore Training

To train a new skill, we first use existing skills to reach states satisfying its preconditions. A classical planner sequences learned skills to navigate from the initial state to valid starting states for the new skill.

Existing skills are composed via symbolic planning to reach the preconditions required for learning new skills. For example, to learn "craft iron pickaxe," we first execute skills for collecting wood, mining iron, and placing a crafting table.

Once the agent reaches a valid starting state, RL trains the new skill policy. This "first-return-then-explore" approach isolates exploration to relevant states and stabilizes credit assignment by separating the roll-in phase (executing existing skills) from learning the new behavior.

The new skill is trained via RL from states satisfying its preconditions. Each skill has its own policy network, avoiding interference between skills with different reward scales.

3. Pivotal Trajectory Analysis

After successful rollouts, an LLM analyzes the trajectories to refine skill specifications and correct errors in the knowledge base. The LLM's initial world knowledge may be incorrect—for instance, predicting "Iron Pickaxe requires 3 Iron" when the environment actually requires only 1 iron. By examining what resources were actually consumed in successful trajectories, we update preconditions and effects to match ground truth. This creates a bi-directional feedback loop: the LLM guides RL exploration, and RL experience corrects the LLM's knowledge.

Pivotal Trajectory Analysis: The LLM examines successful trajectories to identify the minimal resources actually required, updating skill preconditions and the knowledge base to reflect true environment dynamics.

Experiments

We evaluate SCALAR on Craftax-Classic, a challenging long-horizon survival and crafting benchmark where sparse rewards and long survival horizons are essential. The goal is to collect diamond—requiring a chain of resource gathering and tool crafting while maintaining health, hunger, and thirst.

Diamond Collection Performance

SCALAR achieves substantially higher diamond collection rates compared to strong PPO baselines including PPO-RNN, PPO-FC, and PPO-RND with intrinsic motivation. While all methods eventually learn the prerequisite skills (crafting tables, pickaxes, etc.), the critical difference emerges in the final diamond collection phase—SCALAR achieves 86.3% success compared to just 40.8% for the best baseline.

Diamond collection success rate (left) and episode length (right) during training. SCALAR variants achieve over 2x higher diamond collection rates than PPO baselines while maintaining longer episodes—indicating the agent learns to survive longer while pursuing the goal.

Qualitative Behavior Differences

Beyond success rates, we observe qualitative differences in learned behaviors. PPO agents exhibit redundant action patterns—repeatedly placing crafting tables and furnaces after tools have already been crafted. These spurious behaviors likely emerge from correlations with reward during training. In contrast, SCALAR agents demonstrate more directed exploration, executing task-relevant actions without superfluous object placements.

Both agents collect diamond, but the PPO baseline exhibits redundant object placements (crafting tables, furnaces) not required for task completion. SCALAR's skill-based decomposition produces more focused behavior.

Ablation Study

We ablate key components of SCALAR to understand their contributions. Removing trajectory analysis significantly hurts performance—without it, the LLM's prior knowledge overestimates material requirements, leading to systematic over-collection (19 wood instead of 9, 3 iron instead of 1). Shared networks also underperform, as skills with different reward scales interfere during learning.

Ablation study: SCALAR-Dense (blue) and SCALAR-Sparse (green) outperform variants without trajectory analysis (orange) or with shared networks (red). Trajectory analysis improves sample efficiency by correcting resource overestimates; per-skill networks prevent interference between skills.

Training Efficiency

Trajectory analysis also improves training efficiency by correcting the LLM's initial overestimates. Without analysis, agents allocate excessive training time to prerequisite skills (collecting more wood and iron than necessary) before training on diamond collection. With trajectory feedback, agents learn minimal resource requirements and allocate more training budget to the target skill.

Fraction of training budget allocated to the diamond skill (y-axis) after reaching the iron pickaxe frontier. Trajectory analysis (green, blue) enables more efficient allocation by correcting overestimated resource requirements, allowing more frames for the target task.

LLM Cost

An advantage of SCALAR's architecture is that the LLM is not invoked during RL training—only between training runs for planning, reward generation, and trajectory analysis. This amortizes LLM cost effectively: total cost for all experiments was $2.83.

Acknowledgements

This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under grant number DGE2140739. This research was also sponsored by the Department of the Navy, Office of Naval Research under grant number N00014-23-1-2840, the Army Research Laboratory under grant number W911QX24F0049, and the Defense Advanced Research Projects Agency (DARPA) under grant number FA8750-23-2-1015.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, the Department of Defense, or the U.S. Government.

Contact: renosz@cs.cmu.edu, ywu5@andrew.cmu.edu

Citation