User-agent: * Disallow: /bundles/ Disallow: /uploads/ #Block bots User-agent: A6-Indexer Disallow: / User-agent: AlphaSeoBot Disallow: / User-agent: AlphaSeoBot-SA Disallow: / # RDH, 08.19.19: I really don't want to block Applebot, but for now, I am. It is crawling us too much User-agent: Applebot Disallow: / User-agent: AspiegelBot Disallow: / User-agent: barkrowler Disallow: / # RDH, 05.13.20: I really don't want to block bing, but for now, I am. It is also already in htaccess rules User-agent: bingbot/2.0 Disallow: / User-agent: Blackboard Safeassign Disallow: / User-agent: BLEXBot Disallow: / User-agent: Bytespider Disallow: / User-agent: crawler4j Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: DotBot Disallow: / User-agent: Gigabot Disallow: / User-agent: LieBaoFast Disallow: / User-agent: MauiBot Disallow:/ User-agent: MauiBot (crawler.feedback+wc@gmail.com) Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: / User-agent: MqqBrowser Disallow:/ User-agent: Nimbostratus-Bot/v1.3.2 Disallow: / User-agent: Qwant-news Disallow: / User-agent: Qwantify Disallow: / User-agent: Seekport Crawler Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: SeznamBot Disallow: / User-agent: SputnikBot/2.3 Disallow:/ User-agent: The Knowledge AI Disallow:/ User-agent: Timpibot/0.8 Disallow: / User-agent: TinyTestBot Disallow: / User-agent: UCBrowser Disallow: / User-agent: yacybot Disallow: / User-agent: YandexBot Disallow: / User-agent: YandexBot/3.0 Disallow: / User-agent: Yeti Disallow: / User-agent: YisouSpider Disallow: /