Case study · Web Scraping

Domain Scraper

Recursively crawl websites to discover, deduplicate, and classify domain names.

Overview

Extracts unique domains from supplied URLs while recursively exploring linked pages up to a configurable depth. Results include domain components and classification information such as ICANN and private-domain status.

Problem

Collecting domains from websites requires recursive discovery, duplicate prevention, and correctly separating domain components and public-suffix classifications.

Solution

Recursively discovers domains, removes duplicates, and returns structured domain metadata with configurable filtering for ICANN-approved and private domains.

How it works

- Domain extraction - Recursive crawling - Configurable maximum crawl depth - Automatic domain deduplication - ICANN domain filtering - Private-domain filtering - Public suffix information - Subdomain information - Proxy configuration - Configurable minimum and maximum concurrency