GPT-6 Astra ARC-AGI-3: Did ARC Prize Reject 99.9%?

Fact-checked: October 25, 2026

A near-perfect AI benchmark score is an easy headline to remember. The testing conditions usually disappear much faster.

That is the problem with reports saying ARC Prize rejected OpenAI’s 99.9% GPT-6 Astra ARC-AGI-3 claim after recording 62.7% in its standard test. The numbers are real. The suggestion that the benchmark operator has invalidated the higher result is not supported by the official records reviewed for this article.

ARC Prize publishes both results under separate harness labels. Understanding those labels changes the story—and gives developers a useful lesson about memory, reasoning effort, and the cost of completing a task.

There is also a newer development worth including: ARC Prize now publishes Provider Adapter results for Google’s Gemini 4 Argon. This makes the debate broader than one OpenAI launch claim.

Did ARC Prize Reject OpenAI’s 99.9% Claim?

The official ARC Prize records checked on October 25, 2026, do not substantiate a rejection of the 99.9% result. They list 62.7% for GPT-6 Astra with the Standard harness and 99.9% with the Provider Adapter harness. These are separately labeled evaluation conditions.

ARC Prize’s analysis is dated September 3 and reports both numbers. The official record reviewed does not establish a later invalidation that would justify presenting 62.7% as a newly imposed replacement score.

OpenAI’s launch announcement highlights 99.9% and includes a footnote explaining its use of a Responses API harness. A reader who sees only the headline number can easily miss that qualification.

There are three separate questions here:

• Did an evaluated configuration achieve the reported score?
• Can that score be compared directly with a different configuration?
• Does the result establish artificial general intelligence?

Each question needs its own answer. A score can be documented while still requiring careful comparison and a narrow interpretation.

What Does ARC-AGI-3 Actually Measure?

ARC-AGI-3 evaluates agents interacting with unfamiliar, abstract game environments. They must discover how the environment works, infer objectives, and plan actions without receiving a conventional instruction manual.

This differs from answering a static multiple-choice question. An agent might move an object, observe a change, revise its understanding, and try a new sequence.

The benchmark focuses on learning through interaction. Its environments are designed to be solvable by people encountering them for the first time.

The Percentage Includes Efficiency

ARC-AGI-3 uses Relative Human Action Efficiency, or RHAE. The score incorporates level completion and the number of actions needed relative to a human baseline.

Consider an illustrative level where the human baseline is 40 actions:

• An agent finishing in 40 actions receives a raw level efficiency score of 100%.
• An agent needing 80 actions receives 25%, because the ratio is squared: (40 ÷ 80)² = 0.25.

The full benchmark also applies level weights, caps, and aggregation across games.

Consequently, 62.7% should not be described as simply getting 62.7% of questions correct. Likewise, 99.9% is not a measured probability that the system will perform any workplace task successfully.

The distinction matters when a benchmark result travels from a research table into a product recommendation.

The GPT-6 Astra ARC-AGI-3 Results, With Conditions Attached

The following four configurations capture the central comparison. All refer to ARC-AGI-3 Semi-Private results.

These costs describe the reported benchmark evaluations. They are not the price of a ChatGPT subscription, a single response, or an ordinary business workflow.

The table also reveals an easily missed detail: the best observed score for each harness occurs at a different reasoning level. The widely repeated comparison pairs Standard at Max with Provider Adapter at High.

That is a legitimate comparison of the best observed configurations. It is not a comparison holding reasoning effort constant.

Why Does the Harness Change the Result?

A harness is the software surrounding the model during an evaluation. It delivers observations, accepts actions, manages the conversation, and determines what information carries forward.

ARC Prize’s testing policy distinguishes two approaches:

Standard harness: A minimal interface shared across providers. The model can preserve useful discoveries in visible notes.

Provider Adapter harness: Uses provider-specific context management, including retained reasoning state and compaction for longer conversations.

Calling the Standard setup completely memoryless would be inaccurate. Visible notes are available. The difference concerns how much of the ongoing working state survives and how it is represented.

Visible Notes and Retained Reasoning Are Different

Imagine investigating an unfamiliar website editor. You discover that publishing requires saving a draft, selecting a destination, and confirming a separate dialog.

A short note can preserve that sequence. A richer retained state may also preserve the failed approaches, the reasons for choosing the next step, and unresolved possibilities.

The practical hypothesis is that better continuity can reduce repeated investigation. The benchmark results motivate testing that hypothesis in a real application; they do not guarantee the same improvement everywhere.

Our context engineering guide explains the broader challenge of selecting, preserving, and compressing information during extended AI work.

There Was Evidence Before Astra

In July 2026, OpenAI reported a separate GPT-5.6 Sol experiment on the ARC-AGI-3 public set. Enabling retained reasoning and compaction changed its score from 13.3% to 38.3%.

Those figures concern an earlier model and a different task set from the Astra Semi-Private results. They should remain separate in any comparison.

Their relevance is the recurring engineering issue: an evaluation measures the interaction between a model and its surrounding software.

Three Details the Headline Comparison Misses

1. Matching Reasoning Effort Changes the Gap

Subtracting the headline scores gives 37.2 percentage points.

Holding the reasoning label constant produces different comparisons:

These are calculations from the published, rounded scores.

The benefit of this extra step is clarity. A reader can distinguish the best configuration available from the effect observed at a matched reasoning setting.

Neither comparison isolates every possible influence on performance. Repeated trials would be needed to estimate how stable the differences are.

2. The Higher-Scoring Setup Also Cost Less

Using the reported costs, the Provider Adapter evaluation was approximately:

• 33.6% less expensive at Max reasoning: $17,332 versus $26,098.
• 53.8% less expensive at High reasoning: $18,817 versus $40,705.

Those percentages are arithmetic comparisons, not new benchmark experiments.

The practical implication deserves attention. Spending more on each individual decision can sometimes reduce the total bill if it prevents enough repeated work. Conversely, a setting that appears economical per call can become expensive over a long sequence.

A procurement team should therefore measure the cost of an accepted result, including retries and review. Token prices alone cannot answer that question.

3. The Highest Reasoning Setting Did Not Produce Every Best Result

Provider Adapter at High recorded a higher score than Provider Adapter at Max in these published runs.

That observation supports a restrained conclusion: reasoning effort should be evaluated as a configuration choice. It does not establish that High will always outperform Max, or that additional reasoning is generally harmful.

For an application builder, a small comparison across representative tasks is more informative than assuming the largest setting is automatically the best purchase.

October Update: Google Also Has Provider Adapter Results

ARC Prize’s Gemini 4 Argon entry, dated October 13, adds useful context.

At Low reasoning, its published ARC-AGI-3 scores are 49.8% with Standard and 97.5% with Provider Adapter. That is direct evidence that the adapter reporting category extends beyond OpenAI.

There is an important qualification: ARC Prize reports API issues during Google’s Standard evaluations. At Low reasoning, runs completed on 42 of 55 tasks; the published score includes all 55 and retains partial credit from incomplete runs.

That makes a simple interpretation of the gap as a pure capability difference inappropriate.

This newer example highlights two reporting requirements: identify the harness and disclose incomplete execution. An infrastructure failure and an agent exhausting its problem-solving ability are different events, even when both affect the final table.

For the wider model landscape, see our guide to frontier LLMs after GPT-5 and Claude 4.

Does a Near-Perfect Score Mean AGI Has Arrived?

ARC Prize explicitly says it is not claiming Astra is AGI. Its analysis recognizes substantial progress while explaining that the benchmark has bounded environments and deterministic mechanics.

That limitation is consequential. A game can have a definite winning state. A workplace assignment may involve unclear requirements, conflicting stakeholders, changing information, and several defensible outcomes.

Consider an assistant asked to prepare a supplier recommendation. It must judge whether quotations remain current, recognize missing terms, explain uncertainty, and avoid committing the business without authorization.

A puzzle benchmark can provide evidence about learning and planning. Establishing dependable performance on that supplier task requires a separate evaluation.

Benchmark saturation should prompt harder questions about transfer and reliability. It cannot supply evidence about every task the benchmark never tested.

How to Evaluate an Agent Before Using It for Real Work

A useful evaluation should resemble the work the system will actually perform.

For an AI news publisher, that might mean checking a launch announcement, separating confirmed facts from interpretation, and producing a draft with accurate model names and dates.

For a coding team, it might mean fixing a reproducible defect and producing a change that passes review. Our comparison of AI coding agents and chatbots explains how tool access changes the work an assistant can attempt.

Record the Conditions Before Comparing Scores

Use a short record for each result:

1. Model and version: Identify the exact model tested.
2. Harness and memory: Record how notes, reasoning state, and compaction are handled.
3. Reasoning effort: Keep the setting visible beside the score.
4. Task set: Distinguish public demonstrations from held-out evaluations.
5. Execution coverage: Record timeouts, API errors, and unfinished attempts.
6. Cost and date: Include the full run cost and when the evaluation occurred.

This makes a comparison easier to reproduce and helps explain why apparently similar products behave differently.

Measure the Outcome You Need

For a small internal pilot, choose a fixed set of representative tasks and define what counts as an acceptable result before running them.

Track completion, factual errors, elapsed time, spending, and human correction. If you change context management, keep the task set and other settings as consistent as possible.

The main benefit is a decision tied to your workload. It can reveal whether an improvement survives beyond a benchmark screenshot.

Common Mistakes When Reporting AI Benchmark Scores

• Dropping the harness label. A percentage without its evaluation conditions is incomplete.
• Mixing reasoning settings. Best observed results and matched-setting comparisons answer different questions.
• Calling the Standard harness memoryless. It supports visible notes.
• Confusing efficiency with accuracy. RHAE incorporates actions as well as completion.
• Ignoring unfinished runs. Execution coverage can materially affect interpretation.
• Treating evaluation cost as a consumer price. The unit of measurement must travel with the dollar figure.
• Presenting a comparison as a disqualification. A rejection claim needs an explicit supporting record.

Avoiding these mistakes improves the article’s usefulness for readers and makes individual passages easier to quote accurately.

Personal Experience: What Checking the Tables Changed

For this article, my first-hand work consisted of reviewing the official announcements, testing policy, results tables, and methodology, then recalculating the score gaps and cost differences.

The most useful step was lining up the reasoning settings. The headline comparison mixes Max and High. Once I compared the rows separately, the analysis became more precise.

Checking Google’s entry added another lesson: the caveat beside a result can matter as much as the percentage. Its incomplete Standard runs deserve to remain visible whenever that comparison is discussed.

I did not independently rerun the full benchmark. These observations come from the document and data review performed for this article.

The habit I would carry into another model review is straightforward: write down the configuration before interpreting the number.

Frequently Asked Questions

What is GPT-6 Astra’s verified ARC-AGI-3 score?

ARC Prize lists multiple results by configuration. The best observed Standard score is 62.7% at Max reasoning. The best observed Provider Adapter score is 99.9% at High reasoning, on the Semi-Private set.

Has ARC Prize declared the 99.9% result invalid?

The official records reviewed on October 25, 2026, do not substantiate that claim. ARC Prize continues to publish the result with its Provider Adapter label.

Why does the same model receive different scores?

The surrounding software changes what state carries between actions. Retained reasoning and compaction can change how an agent handles a long sequence, even when the underlying model is unchanged.

Does the Standard harness prevent all memory?

No. It lets the model carry forward visible notes. That differs from preserving provider-native reasoning state.

Is the difference 37.2 percent?

It is 37.2 percentage points when subtracting 62.7% from 99.9%. That is different from a relative percentage increase.

Does 99.9% mean Astra is reliable on 99.9% of business tasks?

No. The score belongs to a particular benchmark and configuration. Business reliability must be measured against the actual tasks, tools, and acceptance criteria involved.

Is Provider Adapter testing exclusive to OpenAI?

No. ARC Prize also publishes Provider Adapter results for Google’s Gemini 4 Argon. The official benchmarking repository documents adapters for several providers.

Which result should developers use?

Use the Standard condition to examine a shared interface and the adapter condition to examine provider-native context management. Then test the configuration your application will actually use.

Conclusion

The GPT-6 Astra ARC-AGI-3 story needs its testing conditions attached. The official record supports two separately labeled results; it does not substantiate the claim that ARC Prize rejected the higher one.

The more useful findings sit inside the tables: matched reasoning settings produce different score gaps, the adapter evaluations cost less in the comparisons examined here, and newer Google results introduce an execution-coverage caveat.

Before relying on another benchmark headline, record the harness, reasoning effort, task set, coverage, cost, and date. Then evaluate the system on work with a clear acceptance standard. That is how an impressive score becomes evidence for a practical decision.

Editorial disclosure: This article reflects official materials checked on October 25, 2026. Percentage-point gaps and cost reductions were calculated from published figures. The Personal Experience section describes a document and data review; no independent full-benchmark rerun was performed. Benchmark results do not guarantee performance in a different application.