Most machine learning failures are traced back to the model. In practice, a large share of them originate much earlier, at the point where the data was gathered. Public web pages remain the most accessible source of large-scale text and structured data available to teams without a proprietary corpus. They are also the source most likely to introduce silent defects into a project, because the web is uneven, duplicated, geographically variable, and legally uneven in equal measure.
The scale involved is not trivial. Common Crawal report aintaining an open repository of more than 10 petabytes of crawl data collected since 2008, with monthly crawls typically exceeding two billion pages and citations in over 12,000 research papers. This article sets out a working sequence for assembling a training set from public pages, and identifies the decisions at each stage that determine whether the resulting dataset is usable.
Step 1: Define the Prediction Task First
Collection should never begin with a crawl. It should begin with a written specification of what the model is expected to predict and on what input. That specification determines the unit of analysis. A sentiment classifier needs individual reviews. A product matcher needs paired listings. A summarizer needs full documents with intact structure. These are three incompatible collection jobs.
Write down the label definition, the minimum viable sample size, and the class balance you expect before writing a single line of collection code.
Example: A team building a job-title normalizer specifies the unit as a posting title paired with its industry, targets 200,000 examples, and caps any single employer at 2% of the corpus. That cap is a collection instruction, not a cleaning step.
Takeaway: The specification is the collection plan. Without it, you are archiving, not sampling.
Step 2: Establish What You Are Permitted to Collect
Public and permitted are not synonyms. Three constraints operate at once: the site's stated crawling preferences, its terms of use, and the legal status of the content itself. The first of these is now a formal standard. RFC 9309 pecifies the Robots Exclusion Protocol, and it is explicit that crawlers should not rely on a cached robots.txt file for more than 24 hours, and that the rules it contains are a request rather than a form of access authorization.
Read that second point carefully. Compliance with robots.txt is necessary but not sufficient. Copyright, database rights, and personal data protection sit outside that file entirely.
Practical position: restrict collection to pages served without authentication, honor robots directives, apply conservative rate limits, and exclude personal data unless you have a lawful basis for processing it.
Takeaway: Decide your permission boundary before collection, and record it. Retrofitting it is far more expensive.
Step 3: Where Mobile Proxies Fit Into Collection Design
This is the stage most teams underestimate. The same URL does not serve the same content to every visitor. Pages vary by requesting region, device class, language headers, and sometimes network type. A corpus assembled entirely from one data center in one country is not a sample of the web. It is a sample of what the web looks like to a single origin, and the model inherits that narrowness.
SOAX publishes its network specifications covering 33M+ IPs across 195+ locations with city and ASN-level targeting, and those figures matter for a specific reason: a mobile proxies layer only improves your sample if it can actually reach the regions and carriers your corpus is currently missing.
For any task involving prices, availability, local listings, or regional language variation, geographic and device diversity is a sampling requirement rather than an infrastructure preference.
Two design rules follow. Distribute requests to match the population you intend to model, not to maximize throughput. And log the origin metadata for every fetch, because without it you cannot later demonstrate what your sample actually covered.
Example: A retail pricing dataset collected only from North American origins showed a 14-point accuracy drop when the model was deployed in European markets. The model was fine. The sample never contained the pages it was later asked to interpret.
Takeaway: Collection origin is a sampling parameter. Treat it with the same rigor as class balance.
Step 4: Deduplicate Before You Clean
Web corpora are heavily duplicated. Syndicated articles, template boilerplate, mirrored product pages, and pagination artifacts inflate volume without adding information. Deduplicate first, because cleaning duplicated data means paying the compute cost repeatedly and, worse, over-weighting whatever happens to be duplicated most.
Use exact hashing for identical documents and near-duplicate detection for templated pages that differ only in a field or two. Strip navigation, cookie notices, and footers before hashing, or you will match on boilerplate rather than content.
Takeaway: Volume is not signal. A deduplicated 2 million documents will usually outperform a raw 10 million.
Step 5: Document the Dataset While You Build It
Documentation written afterward is documentation written from memory, and it is invariably wrong. The established template here is the datasheet. The datasheet proposal ommends that every dataset be accompanied by a document recording its motivation, composition, collection process, and recommended uses, so that creators and consumers can communicate about what a dataset does and does not contain. Record at minimum: the date range of collection, the origins used, the exclusions applied, the deduplication method, and the known gaps.
Takeaway: A dataset nobody can describe is a dataset nobody can safely reuse.
Step 6: Place the Dataset Under Governance
Data collection is a lifecycle activity, not a one-off project, and web sources drift constantly.
TheNIST AI RMF Playbook organizes suggested actions around the framework's four functions — Govern, Map, Measure, and Manage — and offers a workable structure for assigning ownership of a dataset rather than leaving it unattended after release.
Assign a named owner, schedule a re-collection cadence, and define what triggers a rebuild.
Comparison of Collection Approaches
Criterion | Existing open corpus | Direct single-origin crawl | Distributed collection |
Setup effort | Low | Moderate | High |
Task specificity | Poor | Good | Good |
Geographic fidelity | Mixed | Weak | Strong |
Ongoing maintenance | None | Moderate | Moderate |
Best suited to | Pretraining, exploration | Narrow single-market tasks | Multi-market or localized tasks |
Conclusion
A usable training set is the product of decisions made before collection begins, not repairs applied afterward. Specify the task, settle the permission boundary, design the sample to match the population, remove duplication early, and document as you go.
Next Steps for You
- Write a one-page specification covering unit of analysis, label definition, and target sample size.
- Draft your permission boundary and have it reviewed before collection starts.
- Map the regions and device classes your target population actually uses.
- Build deduplication into the pipeline rather than the cleanup phase.
- Open the datasheet on day one and fill it in as you collect.
Pro Tip: Collect a 1,000-document pilot sample and label it manually before scaling. Almost every structural defect in a web corpus is visible at that size, and fixing it there costs hours rather than weeks.
