The Meta-ExternalAgent crawler crawls the web for use cases such as training AI models or improving products by indexing content directly.