Scraping ethics we follow

Competitor blogs underwrite bypass because they sell it. A stealth browser, a residential pool, a solver. Our rules are the opposite list. Public pages, official feeds, and a stop when the site forbids the method. This is the policy page for the scraping desk.

I write it as an operator on Windows, not as a lawyer. It is not legal advice. It is the list I use when a keyword wants a hide and the useful article is a boundary instead.

The rules

Official channels first. If there is an API, a feed, or an export, that is the door. Official APIs versus headed collection. HTML is a worse client for the same job.

robots.txt and written terms come first. We read them before we write a client. robots.txt and terms come first. A crawl page that skips this step is not a teacher we will copy.

Public pages, slowly. If the page is public and permitted, the pace is human. Volume blogs optimize for volume. We do not. Public pages, slowly.

Rate limits are a contract. A 429 is not a puzzle. Rate limits are not a puzzle. We wait, we shrink the job, or we pay the tier the platform sells.

Captchas and anti-bot walls are stops. We do not buy a solver. We do not publish a Cloudflare walkthrough.

No hide. We will not dress a collector as a person. We will not hide a scraper. If you collect, say what you are.

No inbox harvest. We will not scrape LinkedIn inboxes. Behind-login collection is not research. Instagram public research is look, not harvest.

Personal data is refused when we have no right to it. Emails from a storefront, phone lists, messages. The privacy sibling holds the longer line. This page only needs the stop.

How this differs from the bypass blogs

They start from a blocked script and work backward to a disguise. We start from a job and ask whether a door exists. What web scraping is is the definition. If the door is an API, we stop talking about Chromium. If the door is a public page, we talk about pace and an honest client. If the door is closed, we stop talking.

A proxy is an exit you brought for a job you were already allowed to do. An exit is enough for some allowed clients. A headed profile is the wrong tool for a harvest. It is not a residential hide.

An antidetect profile is a folder for authorized logins and cold research tabs that must not mix. It is not a stealth skin. The commercial query has its own line.

What a local desk does with the rules

Work profiles stay closed during a run. Scripts live in a different directory than maskwright-data. Leftovers live in a research folder you can name on a tax form if you had to. Provenance is one line: source, time, signed-out or official export.

A contractor who asks for a "light scrape" of a login is asking you to break the list. The answer is the official export, a Page role, or no. A vendor who asks you to paste cookies into their cloud collector is asking for the session. The answer is no.

These rules cost datasets. That is acceptable. A dataset you cannot provenance is a liability that sits next to the live profiles on the same PC. I would rather keep the profiles.

MaskWright 0.1 isolates profiles. It does not ship a collector, a solver, or Playwright. Bulk start opens profiles you already run. It does not page through someone else's HTML.

The automation desk has a parallel policy. Automation ethics we follow. Same stops, different cluster.

What we will not add later to "be helpful"

A hide-the-scraper appendix. A solver bake-off. A list of anti-detect flags. A LinkedIn inbox tool comparison. A "for research only" cloak.

If the site forbade the method, the article ends. The Scraping hub is built out of those endings. This page is the list we check before we write the next one.

FAQ

Is this legal advice?

No. Read the terms, talk to counsel, and treat this as how we write.

Can I scrape if I use MaskWright?

Isolation does not grant permission. 0.1 launches profiles. Collection still needs a door the publisher published.