Case study · Developer Tools

Website Intelligence API

Analyze website technologies, comment systems, structured data, robots.txt, and XML sitemaps in one unified report.

Overview

A website intelligence and technical SEO API that combines multiple website analysis capabilities into a single structured report. It detects technologies and comment platforms, extracts and validates structured data, evaluates robots.txt rules, inspects XML sitemaps, and separates website findings from processing errors.

Problem

Website research and technical audits often require several separate tools to inspect technologies, structured data, comments, crawl rules, and sitemap configuration, resulting in fragmented data and more complex automation workflows.

Solution

Built a unified website analysis workflow that runs multiple checks against each submitted website and returns the results through a consistent data model. Individual analysis sections can succeed or fail independently, allowing partial findings to remain available when one check encounters an error.

How it works

- Website technology and product version detection - Comment system and comment form detection - JSON-LD, Microdata, and RDFa extraction - Structured data validation - Broken structured data checks - robots.txt evaluation - XML sitemap discovery and inspection - Sitemap index traversal - Configurable internal page discovery - Per-check configurable crawl depth - Technology detection confidence and optional evidence - Structured website issues and processing errors - Partial-result handling when individual checks fail - JavaScript-powered website support - Multi-website analysis - Configurable concurrency and retries - Proxy support - JSON, HTML, CSV, and Excel dataset exports