User-agent: * Disallow: Crawl-delay: 5 User-agent: AhrefsBot Disallow: / User-agent: dotbot Disallow: / User-agent: duggmirror Disallow: / User-agent: ia_archiver Disallow: / User-agent: LinkpadBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: rogerbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Snapbot Disallow: / User-agent: spbot Disallow: / User-agent: SurveyBot Disallow: /