Semrush’s New Traffic Insights Explains Changes In Competitor Traffic Performance
New AI-powered Traffic Insights in Traffic Analytics help teams identify competitor traffic spikes and drops, understand the drivers, and act faster.
June 11, 2026
New bot simulation capabilities let teams see their website exactly as AI models and search engines do. Without manual configuration.
Semrush announces the launch of Crawler Profiles in Enterprise Site Intelligence. The new feature enables teams to simulate how specific AI agents and search engine crawlers access and interpret their website, giving organizations precise, bot-level visibility into their site's technical readiness for both SEO and AI discovery.
At launch, three crawler profiles are available: the Semrush crawler, ChatGPT (including ChatGPT-User, OAI-SearchBot, and GPTBot), and Googlebot. Each profile applies bot-aligned configuration settings automatically, removing the need for manual setup and ensuring realistic, accurate simulation at the project level.
Unlocking Targeted Insights for Bots
Without the ability to simulate bot behavior directly, teams are left with blind spots that can silently erode discoverability in both search and AI-generated responses.
Crawler Profiles addresses this by letting users select a predefined profile and run an analysis that mirrors how a specific bot accesses their content. Two crawl types are supported:
- A standard crawler that comprehensively sweeps the entire website to find technical issues and opportunities.
- A list-based crawler for targeted validation of specific URL sets.
Crawls can be launched immediately or scheduled to run at recurring intervals, enabling continuous monitoring as sites evolve.
Designed for Technical and Marketing Teams Alike
The feature serves a range of use cases and teams, helping them to ensure that their key pages are parsable to all search bots.
SEO managers can validate crawl accessibility and prevent hidden visibility gaps. DevOps and engineering teams can confirm rendering and blocking behavior before releases. Enterprise SEO leads can maintain AI readiness across large-scale properties, while content and digital teams can ensure their content is accessible to AI systems that increasingly shape how audiences discover brands.
Expanding Site Intelligence for the AI Era
Crawler Profiles builds on Site Intelligence’s comprehensive SEO and AI search capabilities. When combined with existing capabilities such as Bot Analytics — which delivers comprehensive crawl health analysis across 30 bots — teams now have a layered approach to understanding and optimizing how automated systems interact with their websites.
Crawler Profiles for AI agents and search bots is available now in Enterprise Site Intelligence.
