JustRL-II-base-model: An RL Initialization Checkpoint and the 7-Point Gap It Exposes

What happened

openbmb published JustRL-II-base-model to the Hugging Face hub on 7 September 2026. It is an RL initialization checkpoint — the shared starting point for every run in the JustRL II blog — not a post-training model, and it scores roughly 61% on AIME 2025 before any reinforcement learning.

Context

GRPO has been the default RL recipe for reasoning LLMs through 2025, but it carries a structural limit: group-level reward signals assign the same advantage to every token in a rollout, so credit assignment degrades as chain-of-thought sequences grow past a few thousand tokens. JustRL II addresses that by adding a learned value model that produces token-level advantages through GAE with a length-adaptive discount factor. Publishing the initialization checkpoint separately from the recipe is deliberate — it lets anyone reproduce the 61% baseline and verify the gains are coming from the algorithm, not from a stronger base model.

How it works

The architecture is standard LlamaForCausalLM in bfloat16, shipped as a single pytorch_model.bin with a transformers-compatible config, tokenizer files, and a Jinja chat template. It loads with the transformers library and is tagged compatible with vLLM and SGLang. The chat template exposes an enable_thinking flag that toggles chain-of-thought output inside dedicated tags.

The RL recipe draws 8 rollouts per prompt at temperature 1.0 under a 128k-token generation budget, graded by a math-verify variant. A learned critic, initialized from the policy and bias-calibrated to the pool pass rate, replaces GRPO's flat group reward with token-level GAE advantages using a length-adaptive λ. Tail-only overlong control caps excessive rollouts without penalizing every long response. The training set is 32,412 problems, filtered from roughly 100k open-source math items across DAPO-Math, DeepScaleR, and DeepMath by removing any the checkpoint already solves 8 out of 8 times.

Two details that will bite you in practice. The config ships with max_position_embeddings = 65536, yet the RL runs use a 128k-token budget; the RoPE-scaling step that bridges the gap is not in the model card, only in the linked Notion page. And two end-of-sequence token IDs are configured — [1, 130073] — both of which must be passed to any generation or serving call or output will not terminate correctly.

Our read

The 81% AIME 2025 number in the card belongs to the recipe, not to these weights. What openbmb published is the 61% starting point, and the comparison that matters is the 74% ceiling of a standard GRPO run on the same 32,412 problems. The 7-point gap, closed in roughly 300 RL steps, is the actual result: a learned value model with token-level GAE advantages beats flat group reward on long chain-of-thought sequences. That is a credit-assignment improvement, not a data or compute story, and it is easy to dismiss because the absolute numbers still look like a math benchmark.

Three things the card leaves unsaid that should change how you read this. First, the model is explicitly not aligned for general assistant use and has been evaluated only on mathematical reasoning; it is a research instrument, not a general-purpose LLM. Second, the 128k-token generation budget in the RL runs exceeds the 65536 max_position_embeddings in the shipped config, and the RoPE-scaling step that bridges the two lives only in an external Notion page. Reproducing the run without finding that step will silently truncate long rollouts. Third, at collection time the model had 4 likes and 26 downloads. The ablation tables, critic diagnostics, and difficulty-tier breakdowns referenced in the card are not in this source, so the 81% figure is a single data point from one configuration, not a validated capability curve.

What this changes

For a studio running ComfyUI with local models, nothing changes on Monday. This is a text-generation checkpoint for mathematical reasoning. It has no cross-attention, no latent-diffusion components, and no role in a video or image pipeline. It is also not aligned for conversational use, so it cannot serve as a script-writer or prompt-rewriter out of the box.

If you do need a math-reasoning LLM as a sidecar, the standard-Llama architecture means it loads into vLLM or SGLang without custom code. Two gotchas: pass both EOS tokens, [1, 130073], explicitly or generation will not terminate, and cap your context well below the 128k budget the RL runs assume — a single 24–48 GB GPU will not sustain that. And until a licence appears on the model card, the weights are unusable in any client deliverable.

License

No licence is stated in the model card, the model index, or any other source. The weights are public on Hugging Face, but open access to weights does not imply open source. Do not build a commercial or client-facing product on this checkpoint until openbmb publishes a licence or terms-of-use statement on the model card.

Key takeaways

• The published checkpoint is an RL initialization point scoring ~61% on AIME 2025, not the 81% post-RL model; the 81% belongs to the JustRL II recipe.
• The recipe's core contribution is a learned value model with token-level GAE advantages that closes the 7-point gap over standard GRPO (~74%) in roughly 300 RL steps.
• The 32,412-problem training set was filtered from ~100k open-source math items by removing problems the checkpoint already solves 8 out of 8 times.
• The model is not aligned for general assistant use, is evaluated only on mathematical reasoning, and carries no stated licence.
• The shipped config's 65536 max_position_embeddings is lower than the 128k-token generation budget used in the RL runs; the bridging configuration is documented only in an external linked page.

Sources

1. openbmb/JustRL-II-base-model — text-generation on Hugging Face — tier 1