User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php User-agent: * Disallow: /bhp-beheer/ User-agent: dotbot Disallow: / User-agent: AhrefsBot Disallow: / #To block SEMrushBot from crawling your site for different SEO and technical issues: User-agent: SemrushBot-SA Disallow: / #To block SEMrushBot from crawling your site for Backlink Audit tool: User-agent: SemrushBot-BA Disallow: / #To block SEMrushBot from crawling your site for On Page SEO Checker tool and similar tools: User-agent: SemrushBot-SI Disallow: / #To block SEMrushBot from checking URLs your site for SWA tool: User-agent: SemrushBot-SWA Disallow: / #To block SEMrushBot from crawling your site for Content Analyzer and Post Tracking tools: User-agent: SemrushBot-CT Disallow: / #To block SEMrushBot from crawling your site for Brand Monitoring: User-agent: SemrushBot-BM Disallow: /