Screaming Frog SEO Spider is a website crawler used by SEO professionals for site audits and technical SEO analysis. It's a desktop-based tool that crawls websites' links, images, CSS, scripts and apps to evaluate onsite SEO. The crawler respects robots.txt and can be configured for crawl speed and behavior.