2026-07-15 · Morgan Ellis · 829 words
Public pages, slowly
Public pages, slowly. They optimize for volume. We optimize for staying inside what the site already offers the public, at a human pace. Authorized work only.
Volume blogs optimize for volume. We optimize for staying inside what the site already offers the public, at a human pace. This how-to is slower than the keyword "slow scraping" wants. That is the point.
What web scraping is already named the doors. This page is only the public-page door, walked without a rush.
What "public" means here
A URL a signed-out person can open. A marketing page. Documentation. A public Ads Library card. A blog post the publisher put on the open web. Not a feed behind a login. Not a pagination trap that only appears after you authenticate.
robots.txt and terms come first. If Disallow covers the path, it is not public to a client. If the terms forbid automated copies, a person in a chair can still read, and a program should not harvest.
Official APIs versus headed collection. If an API covers the same bytes, use the API. Slow HTML is not a cheaper official door.
What "slowly" means on a desk
Slowly means a person could have done the same clicks in the same afternoon. One page, a pause, a note. Not a queue of ten thousand URLs overnight.
If you write a client at all, the interval should look like reading, not like a pipeline. Rate limits are not a puzzle. A courtesy delay is not a trick to stay under a threshold so you can finish a harvest the terms forbade. If you need volume, you are on the wrong door.
User-agent honesty in research. The client says what it is. A random Chrome string is how volume clients hide. I will not write that string.
A captcha means you already left human pace, or the site does not want the client. Captchas are a stop sign. Stop. Do not tune the interval until the puzzle goes away.
| Pace | Looks like | I use it |
|---|---|---|
| Human afternoon | Five public pages, notes, kettle | Yes |
| Documented API window | Sleep until the header says resume | Yes, on the API door |
| Overnight queue | Thousands of URLs, retries | No, not this door |
The cold profile
Look from a profile that holds no work logins. No Ads Manager cookie. No Seller Central session. No brand inbox. The cold profile can open the public URL, take a screenshot, save a permitted page into a research folder, and close.
Research files stay outside maskwright-data. Do not download into the work profile's tray. Chromium user-data dirs are not encrypted by our code. A public-page PNG next to a live cookie is a mixed folder.
Facebook Ads Library research is the commercial example: public library, manual first, official tools second. Instagram public research is look, not a 2023 code harvest.
What an afternoon looks like
You name the question in one line. You open the cold profile. You read five public pages a person could have read before lunch. You type notes with the URL and the time. You close the profile before you open Seller Central.
If the fifth page is enough to decide, you stop. If it is not, you ask whether an official export would have been the right door all along. You do not keep paging because the folder looks thin. A thin folder that is allowed is better than a fat folder that is not.
A second monitor with a work login is how the afternoon fails. The public page and the live session do not share a glance. If you need both, you close one first.
What I will not optimize
Retry storms. Parallel tabs against one host. Residential rotation so the host sees many people. Headless fleets. "Polite" crawlers that still finish a site in an hour.
Scraping ethics we follow is the list. Pace is one line on that list. It is the line this URL owns.
MaskWright 0.1 can isolate the cold profile from the work profiles. It will not schedule the clicks. Bulk start is for authorized profiles you already run, not for a public-page fleet.
If the job cannot be done at a human pace, it is not this door. It is an API job, a licensed export, or a job you drop. The Scraping hub will not invent a fourth option that is "slow" in the title and fast in the script.
A timer on the desk is enough instrumentation. If you would be embarrassed to tell a publisher you read their public pages at that pace, you are not on this door anymore. Storing research files is where the notes go.
FAQ
How slow is slow enough?
If a person could have done the same reading before lunch, you are in range. If you are measuring the host to find the highest rate that does not trip a wall, you already left this door.
Do I need a script for public pages?
Usually no. A person in a cold profile and a notes file is the whole stack for most looks.
Related notes
- Rate limits are not a puzzleScraping
- Official APIs versus headed collectionScraping
- Captchas are a stop signScraping
- robots.txt and terms come firstScraping
- User-agent honesty in researchScraping
- Scraping ethics we followScraping
- Storing research filesScraping
- What web scraping isScraping