Боты от ЦМУ ССОП на 2024-08-31
Материал из 1GbWiki.
(Различия между версиями)
Версия 14:19, 31 августа 2024 (править) Dmih (Обсуждение | вклад) ← К предыдущему изменению |
Версия 14:23, 31 августа 2024 (править) (отменить) Dmih (Обсуждение | вклад) К следующему изменению → |
||
Строка 1: | Строка 1: | ||
__TOC__ | __TOC__ | ||
+ | |||
+ | Данный список ботов предоставлен ЦМУ ССОП. Используйте блокировки по этому списку на свой страх и риск. Мы рекомендуем самостоятельно оценивать целесообразность данных блокировок и их корректность. Помимо очевидных ботов, список блокирует многие средства автоматизации, API, и так далее, поэтому относитесь к этому внимательно. | ||
+ | |||
== Список блокировки для web.config == | == Список блокировки для web.config == | ||
Строка 5132: | Строка 5135: | ||
User-agent: ZyXel | User-agent: ZyXel | ||
Disallow: / | Disallow: / | ||
+ | |||
+ | == Сырой список ботов == | ||
+ | |||
+ | На основе этого списка построены правила выше. | ||
+ | |||
+ | 01h4x | ||
+ | 360Spider | ||
+ | 404checker | ||
+ | 404enemy | ||
+ | 80legs | ||
+ | Abonti | ||
+ | Aboundex | ||
+ | Aboundexbot | ||
+ | Acunetix | ||
+ | ADmantX | ||
+ | adscanner | ||
+ | AdsTxtCrawlerTP | ||
+ | AfD-Verbotsverfahren | ||
+ | AhrefsBot | ||
+ | AIBOT | ||
+ | AiHitBot | ||
+ | Aipbot | ||
+ | Alexibot | ||
+ | ALittle Client | ||
+ | Alligator | ||
+ | AllSubmitter | ||
+ | AlphaBot | ||
+ | Anarchie | ||
+ | Anarchy | ||
+ | Anarchy99 | ||
+ | Ankit | ||
+ | Anthill | ||
+ | anthropic-ai | ||
+ | Apexoo | ||
+ | archive | ||
+ | arquivo-web-crawler | ||
+ | arquivo | ||
+ | Aspiegel | ||
+ | ASPSeek | ||
+ | Asterias | ||
+ | Atomseobot | ||
+ | Attach | ||
+ | autoemailspider | ||
+ | awario | ||
+ | AwarioBot | ||
+ | AwarioRssBot | ||
+ | AwarioSmartBot | ||
+ | BackDoorBot | ||
+ | Backlink-Ceck | ||
+ | backlink-check | ||
+ | BacklinkCrawler | ||
+ | BackStreet | ||
+ | BackWeb | ||
+ | Badass | ||
+ | Bandit | ||
+ | Barkrowler | ||
+ | BatchFTP | ||
+ | Battleztar Bazinga | ||
+ | BBBike | ||
+ | BDCbot | ||
+ | BDFetch | ||
+ | BetaBot | ||
+ | Bigfoot | ||
+ | Bitacle | ||
+ | Black Hole | ||
+ | Blackboard | ||
+ | BlackWidow | ||
+ | BLEXBot | ||
+ | Blow | ||
+ | BlowFish | ||
+ | Boardreader | ||
+ | Bolt | ||
+ | BotALot | ||
+ | Brandprotect | ||
+ | Brandwatch | ||
+ | Buck | ||
+ | Buddy | ||
+ | BuiltBotTough | ||
+ | BuiltWith | ||
+ | Bullseye | ||
+ | BunnySlippers | ||
+ | BuzzSumo | ||
+ | Bytespider | ||
+ | cah | ||
+ | Calculon | ||
+ | CATExplorador | ||
+ | CazoodleBot | ||
+ | CCBot | ||
+ | Cegbfeieh | ||
+ | CensysInspect | ||
+ | ChatGPT-User | ||
+ | check1 | ||
+ | CheeseBot | ||
+ | CherryPicker | ||
+ | CheTeam | ||
+ | ChinaClaw | ||
+ | Chlooe | ||
+ | Citoid | ||
+ | Claritybot | ||
+ | clark-crawler | ||
+ | Cliqzbot | ||
+ | Cloud mapping | ||
+ | coccocbot | ||
+ | Cocolyzebot | ||
+ | CODE87 | ||
+ | Cogentbot | ||
+ | cognitiveseo | ||
+ | cohere-ai | ||
+ | Collector | ||
+ | com | ||
+ | Copier | ||
+ | CopyRightCheck | ||
+ | Copyscape | ||
+ | Cosmos | ||
+ | Craftbot | ||
+ | crawl | ||
+ | crawler | ||
+ | crawler4j | ||
+ | Crawling at Home Project | ||
+ | CrazyWebCrawler | ||
+ | Crescent | ||
+ | CrunchBot | ||
+ | CSHttp | ||
+ | Curious | ||
+ | Custo | ||
+ | CyotekWebCopy | ||
+ | DatabaseDriverMysqli | ||
+ | DataCha0s | ||
+ | dataforseo | ||
+ | dataforseobot | ||
+ | DBLBot | ||
+ | demandbase-bot | ||
+ | Demon | ||
+ | Deusu | ||
+ | Devil | ||
+ | Digincore | ||
+ | DigitalPebble | ||
+ | DIIbot | ||
+ | Dirbuster | ||
+ | Disco | ||
+ | Discobot | ||
+ | Discoverybot | ||
+ | Dispatch | ||
+ | DittoSpyder | ||
+ | DnBCrawler-Analytics | ||
+ | DnyzBot | ||
+ | DomainAppender | ||
+ | DomainCrawler | ||
+ | Domains Project | ||
+ | DomainSigmaCrawler | ||
+ | domainsproject | ||
+ | DomainStatsBot | ||
+ | DomCopBot | ||
+ | Dotbot | ||
+ | Download Wonder | ||
+ | Dragonfly | ||
+ | Drip | ||
+ | DSearch | ||
+ | DTS Agent | ||
+ | EasyDL | ||
+ | Ebingbong | ||
+ | eCatch | ||
+ | ECCP/1 | ||
+ | Ecxi | ||
+ | EirGrabber | ||
+ | EMail Siphon | ||
+ | EMail Wolf | ||
+ | EroCrawler | ||
+ | evc-batch | ||
+ | Evil | ||
+ | Exabot | ||
+ | Express WebPictures | ||
+ | ExtLinksBot | ||
+ | Extractor | ||
+ | ExtractorPro | ||
+ | Extreme Picture Finder | ||
+ | EyeNetIE | ||
+ | Ezooms | ||
+ | FacebookBot | ||
+ | facebookscraper | ||
+ | FDM | ||
+ | FemtosearchBot | ||
+ | FHscan | ||
+ | Fimap | ||
+ | FlashGet | ||
+ | Flunky | ||
+ | Foobot | ||
+ | Freeuploader | ||
+ | FrontPage | ||
+ | Fuzz | ||
+ | FyberSpider | ||
+ | Fyrebot | ||
+ | G-i-g-a-b-o-t | ||
+ | GalaxyBot | ||
+ | Genieo | ||
+ | GermCrawler | ||
+ | Getintent | ||
+ | GetRight | ||
+ | GetWeb | ||
+ | Gigabot | ||
+ | Go-Ahead-Got-It | ||
+ | Go!Zilla | ||
+ | gopher | ||
+ | Gotit | ||
+ | GoZilla | ||
+ | GPTBot | ||
+ | Grabber | ||
+ | GrabNet | ||
+ | Grafula | ||
+ | GrapeFX | ||
+ | GrapeshotCrawler | ||
+ | GridBot | ||
+ | Haansoft | ||
+ | HaosouSpider | ||
+ | Harvest | ||
+ | Havij | ||
+ | HEADMasterSEO | ||
+ | Heritrix | ||
+ | Hloader | ||
+ | HMView | ||
+ | HonoluluBot | ||
+ | HTMLparser | ||
+ | HTTrack | ||
+ | Humanlinks | ||
+ | HybridBot | ||
+ | Iblog | ||
+ | Id-search | ||
+ | IDBot | ||
+ | IDBTE4M | ||
+ | IlseBot | ||
+ | Image Fetch | ||
+ | Image Sucker | ||
+ | imagesift | ||
+ | ImagesiftBot | ||
+ | IndeedBot | ||
+ | Indy Library | ||
+ | InfoNaviRobot | ||
+ | Information Security Team InfraSec Scanner | ||
+ | InfoTekies | ||
+ | InfraSec Scanner | ||
+ | instabid | ||
+ | Intelliseek | ||
+ | InterGET | ||
+ | Internet Ninja | ||
+ | InternetMeasurement | ||
+ | InternetSeer | ||
+ | internetVista monitor | ||
+ | ips-agent | ||
+ | Iria | ||
+ | IRLbot | ||
+ | isitwp | ||
+ | Iskanie | ||
+ | IstellaBot | ||
+ | iubenda-radar | ||
+ | JamesBOT | ||
+ | Jbrofuzz | ||
+ | JennyBot | ||
+ | JetCar | ||
+ | Jetty | ||
+ | JikeSpider | ||
+ | JOC Web Spider | ||
+ | Joomla | ||
+ | Jorgee | ||
+ | JustView | ||
+ | Jyxobot | ||
+ | Kenjin Spider | ||
+ | Keybot Translation-Search-Machine | ||
+ | Keyword Density | ||
+ | Kinza | ||
+ | Kozmosbot | ||
+ | Lanshanbot | ||
+ | Larbin | ||
+ | Leap | ||
+ | LeechFTP | ||
+ | LeechGet | ||
+ | LexiBot | ||
+ | Lftp | ||
+ | LibWeb | ||
+ | Libwhisker | ||
+ | LieBaoFast | ||
+ | Lightspeedsystems | ||
+ | Likse | ||
+ | Linkbot | ||
+ | linkdexbot | ||
+ | LinkextractorPro | ||
+ | linkfluence | ||
+ | LinkpadBot | ||
+ | LinkScan | ||
+ | LinksManager | ||
+ | LinkWalker | ||
+ | LinqiaMetadataDownloaderBot | ||
+ | LinqiaRSSBot | ||
+ | LinqiaScrapeBot | ||
+ | Lipperhey | ||
+ | Lipperhey Spider | ||
+ | Litemage_walker | ||
+ | Lmspider | ||
+ | LNSpiderguy | ||
+ | Ltx71 | ||
+ | lwp-request | ||
+ | lwp-trivial | ||
+ | Mag-Net | ||
+ | Magnet | ||
+ | magpie-crawler | ||
+ | Majestic SEO | ||
+ | Majestic-SEO | ||
+ | Majestic12 | ||
+ | MarkMonitor | ||
+ | MarkWatch | ||
+ | Mass Downloader | ||
+ | Masscan | ||
+ | Mata Hari | ||
+ | MauiBot | ||
+ | Mb2345Browser | ||
+ | MeanPath Bot | ||
+ | Meanpathbot | ||
+ | Mediatoolkitbot | ||
+ | mediawords | ||
+ | MegaIndex | ||
+ | Metauri | ||
+ | MFC_Tear_Sample | ||
+ | MicroMessenger | ||
+ | MIDown tool | ||
+ | MIIxpc | ||
+ | Minefield | ||
+ | Mister PiX | ||
+ | MJ12bot | ||
+ | Mojeek | ||
+ | Mojolicious | ||
+ | MolokaiBot | ||
+ | Morfeus Fucking Scanner | ||
+ | Mozlila | ||
+ | Mr | ||
+ | MSFrontPage | ||
+ | MSIECrawler | ||
+ | Msrabot | ||
+ | MTRobot | ||
+ | muhstik-scan | ||
+ | Musobot | ||
+ | Name Intelligence | ||
+ | Nameprotect | ||
+ | Navroad | ||
+ | NearSite | ||
+ | Needle | ||
+ | Nessus | ||
+ | Net Vampire | ||
+ | NetAnts | ||
+ | Netcraft | ||
+ | netEstate NE Crawler | ||
+ | NetLyzer | ||
+ | NetMechanic | ||
+ | NetSpider | ||
+ | Nettrack | ||
+ | Netvibes | ||
+ | NetZIP | ||
+ | NextGenSearchBot | ||
+ | Nibbler | ||
+ | NICErsPRO | ||
+ | Niki-bot | ||
+ | Nikto | ||
+ | NimbleCrawler | ||
+ | Nimbostratus | ||
+ | Ninja | ||
+ | Nmap | ||
+ | NPbot | ||
+ | Nuclei | ||
+ | Nutch | ||
+ | oBot | ||
+ | Octopus | ||
+ | Offline Explorer | ||
+ | Offline Navigator | ||
+ | omgili | ||
+ | OnCrawl | ||
+ | openai | ||
+ | Openfind | ||
+ | OpenLinkProfiler | ||
+ | OpenVAS | ||
+ | OrangeBot | ||
+ | OrangeSpider | ||
+ | OutclicksBot | ||
+ | OutfoxBot | ||
+ | Page Analyzer | ||
+ | page scorer | ||
+ | PageAnalyzer | ||
+ | PageGrabber | ||
+ | PageScorer | ||
+ | PageThing | ||
+ | Pandalytics | ||
+ | Panscient | ||
+ | Papa Foto | ||
+ | Pavuk | ||
+ | pcBrowser | ||
+ | PeoplePal | ||
+ | Petalbot | ||
+ | PHPCrawl | ||
+ | Pi-Monster | ||
+ | Picscout | ||
+ | Picsearch | ||
+ | PictureFinder | ||
+ | Piepmatz | ||
+ | Pimonster | ||
+ | Pixray | ||
+ | PleaseCrawl | ||
+ | plumanalytics | ||
+ | Pockey | ||
+ | POE-Component-Client-HTTP | ||
+ | polaris version | ||
+ | probe-image-size | ||
+ | Probethenet | ||
+ | ProPowerBot | ||
+ | ProWebWalker | ||
+ | Proximic | ||
+ | Psbot | ||
+ | Pu_iN | ||
+ | Pump | ||
+ | PxBroker | ||
+ | PyCurl | ||
+ | QueryN Metasearch | ||
+ | Quick-Crawler | ||
+ | Rainbot | ||
+ | RankActive | ||
+ | RankActiveLinkBot | ||
+ | RankFlex | ||
+ | RankingBot | ||
+ | RankingBot2 | ||
+ | Rankivabot | ||
+ | RankurBot | ||
+ | Re-re | ||
+ | RealDownload | ||
+ | Reaper | ||
+ | RebelMouse | ||
+ | Recorder | ||
+ | RedesScrapy | ||
+ | ReGet | ||
+ | RepoMonkey | ||
+ | Ripper | ||
+ | ripz | ||
+ | RocketCrawler | ||
+ | Rogerbot | ||
+ | RSSingBot | ||
+ | s1z | ||
+ | SalesIntelligent | ||
+ | satoristudio | ||
+ | SBIder | ||
+ | scalaj-http | ||
+ | scan | ||
+ | ScanAlert | ||
+ | Scanbot | ||
+ | ScoutJet | ||
+ | Scrapy | ||
+ | Screaming | ||
+ | ScreenerBot | ||
+ | ScrepyBot | ||
+ | Searchestate | ||
+ | SearchmetricsBot | ||
+ | Seekport | ||
+ | SeekportBot | ||
+ | SemanticJuice | ||
+ | Semrush | ||
+ | SemrushBot | ||
+ | SentiBot | ||
+ | SenutoBot | ||
+ | seobility | ||
+ | SeobilityBot | ||
+ | seocompany | ||
+ | SEOkicks | ||
+ | SEOkicks-Robot | ||
+ | SEOlyticsCrawler | ||
+ | Seomoz | ||
+ | SEOprofiler | ||
+ | seoscanners | ||
+ | SeoSiteCheckup | ||
+ | seostar | ||
+ | SEOstats | ||
+ | serpstatbot | ||
+ | sexsearcher | ||
+ | Shodan | ||
+ | Siphon | ||
+ | SISTRIX | ||
+ | Site Sucker | ||
+ | Sitebeam | ||
+ | sitechecker | ||
+ | SiteCheckerBotCrawler | ||
+ | SiteExplorer | ||
+ | Siteimprove | ||
+ | SiteLockSpider | ||
+ | siteripz | ||
+ | SiteSnagger | ||
+ | SiteSucker | ||
+ | Sitevigil | ||
+ | SlySearch | ||
+ | SmartDownload | ||
+ | SMTBot | ||
+ | Snake | ||
+ | Snapbot | ||
+ | Snoopy | ||
+ | SocialRankIOBot | ||
+ | Sociscraper | ||
+ | Sogou web spider | ||
+ | sogouspider | ||
+ | Sosospider | ||
+ | Sottopop | ||
+ | sp_auditbot | ||
+ | SpaceBison | ||
+ | Spammen | ||
+ | SpankBot | ||
+ | Spanner | ||
+ | Spbot | ||
+ | Spinn3r | ||
+ | SputnikBot | ||
+ | spyfu | ||
+ | Sqlmap | ||
+ | Sqlworm | ||
+ | Sqworm | ||
+ | Steeler | ||
+ | Stripper | ||
+ | Sucker | ||
+ | Sucuri | ||
+ | SuperBot | ||
+ | SuperHTTP | ||
+ | Surfbot | ||
+ | SurveyBot | ||
+ | Suzuran | ||
+ | Swiftbot | ||
+ | sysscan | ||
+ | Szukacz | ||
+ | T0PHackTeam | ||
+ | T8Abot | ||
+ | tAkeOut | ||
+ | Teleport | ||
+ | TeleportPro | ||
+ | Telesoft | ||
+ | Telesphoreo | ||
+ | Telesphorep | ||
+ | The Intraformant | ||
+ | TheNomad | ||
+ | Thumbor | ||
+ | TightTwatBot | ||
+ | TinyTestBot | ||
+ | Titan | ||
+ | Toata | ||
+ | Toweyabot | ||
+ | Tracemyfile | ||
+ | Trendiction | ||
+ | Trendictionbot | ||
+ | True_Robot | ||
+ | Turingos | ||
+ | Turnitin | ||
+ | TurnitinBot | ||
+ | TwengaBot | ||
+ | Twice | ||
+ | Typhoeus | ||
+ | ubermetrics-technologies | ||
+ | UnisterBot | ||
+ | Upflow | ||
+ | URLy Warning | ||
+ | URLy | ||
+ | V-BOT | ||
+ | Vacuum | ||
+ | Vagabondo | ||
+ | VB Project | ||
+ | VCI | ||
+ | VelenPublicWebCrawler | ||
+ | VeriCiteCrawler | ||
+ | VidibleScraper | ||
+ | Virusdie | ||
+ | VoidEYE | ||
+ | Voil | ||
+ | Voltron | ||
+ | voyagerx | ||
+ | Wallpapers | ||
+ | Wallpapers/3 | ||
+ | WallpapersHD | ||
+ | WASALive-Bot | ||
+ | WBSearchBot | ||
+ | Web Auto | ||
+ | Web Collage | ||
+ | Web Enhancer | ||
+ | Web Fetch | ||
+ | Web Fuck | ||
+ | Web Pix | ||
+ | Web Sauger | ||
+ | Web Sucker | ||
+ | Webalta | ||
+ | WebAuto | ||
+ | WebBandit | ||
+ | WebCollage | ||
+ | WebCopier | ||
+ | WEBDAV | ||
+ | WebEnhancer | ||
+ | WebFetch | ||
+ | WebFuck | ||
+ | webgains-bot | ||
+ | WebGo IS | ||
+ | WebImageCollector | ||
+ | WebLeacher | ||
+ | WebmasterWorldForumBot | ||
+ | webmeup-crawler | ||
+ | WebPix | ||
+ | webpros | ||
+ | webprosbot | ||
+ | WebReaper | ||
+ | WebSauger | ||
+ | Webshag | ||
+ | Website Quester | ||
+ | WebsiteExtractor | ||
+ | WebsiteQuester | ||
+ | Webster | ||
+ | WebStripper | ||
+ | WebSucker | ||
+ | WebWhacker | ||
+ | WebZIP | ||
+ | WeSEE | ||
+ | Whack | ||
+ | Whacker | ||
+ | Whatweb | ||
+ | Who | ||
+ | Widow | ||
+ | WinHTTrack | ||
+ | WiseGuys Robot | ||
+ | WISENutbot | ||
+ | Wonderbot | ||
+ | Woobot | ||
+ | Wotbox | ||
+ | Wprecon | ||
+ | WPScan | ||
+ | WWW-Collector-E | ||
+ | WWW-Mechanize | ||
+ | WWWOFFLE | ||
+ | Xaldon WebSpider | ||
+ | Xaldon_WebSpider | ||
+ | Xenu | ||
+ | xpymep1 | ||
+ | YaK | ||
+ | YoudaoBot | ||
+ | Zade | ||
+ | Zauba | ||
+ | Zermelo | ||
+ | Zeus | ||
+ | zgrab | ||
+ | Zitebot | ||
+ | ZmEu | ||
+ | ZoominfoBot | ||
+ | ZumBot | ||
+ | ZyBorg | ||
+ | ZoomBot | ||
+ | WWW::Mechanize | ||
+ | GT::WWW | ||
+ | HTTP::Lite | ||
+ | LWP::Simple | ||
+ | PECL::HTTP | ||
+ | NoUserAgent | ||
+ | noxtrumbot | ||
+ | Nsauditor | ||
+ | Nuhk | ||
+ | Nuzzel | ||
+ | NuzzelFeedBot | ||
+ | NuzzelMediaBot | ||
+ | Nymesis | ||
+ | NymesisBot | ||
+ | Ocelli | ||
+ | OmahaBot | ||
+ | OmniExplorer | ||
+ | Onebot | ||
+ | OnetSzukaj | ||
+ | Onfolio | ||
+ | Openindexspider | ||
+ | OpenWebSpider | ||
+ | OxBot | ||
+ | PagePeeker | ||
+ | PageScrape | ||
+ | PagetScraper | ||
+ | Pcore-HTTPClient | ||
+ | Perl LWP | ||
+ | PersonaPilot | ||
+ | Phantom | ||
+ | PhantomJS | ||
+ | PHP::HTTPClient | ||
+ | Pingdom | ||
+ | PiplBot | ||
+ | Plukkie | ||
+ | Pogodak | ||
+ | ProCogBot | ||
+ | Python-httplib2 | ||
+ | Python-urllib | ||
+ | PyWebScraper | ||
+ | Qirina Hurdle | ||
+ | QResync | ||
+ | Qwantify | ||
+ | Rambler | ||
+ | RankSonicSiteAuditor | ||
+ | RankingHit | ||
+ | RealPlayer | ||
+ | Repo | ||
+ | RepoDev | ||
+ | RepoJob | ||
+ | Riddler | ||
+ | RivalIQ | ||
+ | RixBot | ||
+ | Robozilla | ||
+ | RoHe | ||
+ | Ruhm | ||
+ | SBL-Bot | ||
+ | Scrubby | ||
+ | scrutiny | ||
+ | ScrutinyBot | ||
+ | Seeker | ||
+ | semalt | ||
+ | SEO Suite | ||
+ | SEOENGBot | ||
+ | SEOlytics | ||
+ | SentiSumBot | ||
+ | seochat | ||
+ | SEOEngine | ||
+ | SEOMetriks | ||
+ | SerendeputyBot | ||
+ | Server Density Service Monitoring Bot | ||
+ | SeznamBot | ||
+ | SFR-Sharepoint | ||
+ | ShaMan | ||
+ | Shockwave Flash | ||
+ | ShopWiki | ||
+ | Siege | ||
+ | SimplePie | ||
+ | Sitebulb | ||
+ | Sixy Search | ||
+ | Sixy Search V2 | ||
+ | SkimBot | ||
+ | Skrybot | ||
+ | Skywalker | ||
+ | Slackerbot | ||
+ | SlackSocialBot | ||
+ | Slackbot | ||
+ | Slap | ||
+ | SliquaBot | ||
+ | Snoopie | ||
+ | Snufkinbot | ||
+ | Sogou | ||
+ | Sogou News Spider | ||
+ | Sogou Orion spider | ||
+ | Sogou Pic Spider | ||
+ | Sogou Spider | ||
+ | Sogou blog | ||
+ | Sogou inst spider | ||
+ | Sogou news | ||
+ | Sogou orion | ||
+ | Sogou web | ||
+ | Sogou-Pic-Spider | ||
+ | Sogou-Test-Spider | ||
+ | Sogou-Test | ||
+ | SoftHyde | ||
+ | SoMegaBot | ||
+ | Sonarcloud | ||
+ | SoNET | ||
+ | Sonic | ||
+ | SonicWall | ||
+ | Soso Spider | ||
+ | Spacy | ||
+ | SpamBot | ||
+ | SpamBayes | ||
+ | SpamMoose | ||
+ | SpamPatrol | ||
+ | Speedy | ||
+ | Speedy Spider | ||
+ | SpeedySpider | ||
+ | SpiderMan | ||
+ | Spider_Monkey | ||
+ | SpiderBot | ||
+ | SpiderLine | ||
+ | SpiderMonkey | ||
+ | SpiderProtector | ||
+ | SpiderSilk | ||
+ | sprinklr-spider | ||
+ | Spyder | ||
+ | SQLIHelper | ||
+ | Squider | ||
+ | SSIbot | ||
+ | SSLLabs | ||
+ | SSLTestBot | ||
+ | Startsearch | ||
+ | StartpageBot | ||
+ | StartSiteBot | ||
+ | Static | ||
+ | StellarBot | ||
+ | Strategistbot | ||
+ | Stratbot | ||
+ | stratosec-bot | ||
+ | StrokeBot | ||
+ | StudyBot | ||
+ | StudiotimeBot | ||
+ | SuggyBot | ||
+ | Supernova | ||
+ | Supersalonbot | ||
+ | Surebot | ||
+ | Surveillance | ||
+ | Suzie | ||
+ | SVM | ||
+ | SWEBot | ||
+ | SwiteBot | ||
+ | SXN | ||
+ | SyAgent | ||
+ | Syndex | ||
+ | Syndexbot | ||
+ | Synthesio | ||
+ | Sysomos | ||
+ | SzukajSe | ||
+ | Tactical | ||
+ | Tagasaurus | ||
+ | tAganalytics | ||
+ | Tailor4Less | ||
+ | Tailwind | ||
+ | Talkwalker | ||
+ | TalkwalkerBot | ||
+ | TalkwalkerRSSBot | ||
+ | TalkwalkerSmartBot | ||
+ | Tantalos | ||
+ | TaringaBot | ||
+ | Tarybot | ||
+ | Tazman-Agg | ||
+ | Tbot | ||
+ | Teamsoft | ||
+ | TeepsaBot | ||
+ | TelegramBot | ||
+ | Telekom-Bot | ||
+ | Teoma | ||
+ | TeraBot | ||
+ | Testbot | ||
+ | TextRazor | ||
+ | Thalesbot | ||
+ | The Answer | ||
+ | The Kraken | ||
+ | The Nomad | ||
+ | The-Schmidts | ||
+ | TheWorld | ||
+ | Thirdlightbot | ||
+ | Thunderstone | ||
+ | Tixati | ||
+ | Tobasco | ||
+ | Token | ||
+ | Toll Free | ||
+ | Tomato Bot | ||
+ | TomatoCat | ||
+ | Tonic | ||
+ | Topbot | ||
+ | Topwebsearch | ||
+ | Torbot | ||
+ | ToutiaoSpider | ||
+ | Tpbot | ||
+ | Trackback | ||
+ | Tracklock | ||
+ | Trahit | ||
+ | Tramp | ||
+ | Trams | ||
+ | Translate.ru bot | ||
+ | Translate.ru | ||
+ | Trax | ||
+ | Trendogic | ||
+ | Trendspottr | ||
+ | TridionBot | ||
+ | Triggitbot | ||
+ | Trilby | ||
+ | trufflehog | ||
+ | Truefinder | ||
+ | Truthybot | ||
+ | TUCOWS | ||
+ | Twingly | ||
+ | TwistedPageGetter | ||
+ | Twitterbot | ||
+ | U0xA-Bot | ||
+ | Ubermetrics | ||
+ | Ubrowse | ||
+ | uipbot | ||
+ | uLink | ||
+ | UnchaindSpider | ||
+ | UniteBot | ||
+ | UnshortenBot | ||
+ | Updownerbot | ||
+ | UpdatePatrol | ||
+ | UpsightBot | ||
+ | Urlfilebot | ||
+ | URLitor | ||
+ | URLMonitorBot | ||
+ | URLSpider | ||
+ | Urlwatch | ||
+ | User-Agent | ||
+ | Validator | ||
+ | VBProjectBot | ||
+ | Vespa | ||
+ | VestrSpider | ||
+ | ViaMichelin | ||
+ | Vider | ||
+ | VideoSurf | ||
+ | ViewState | ||
+ | VirusTotal | ||
+ | vkShare | ||
+ | Vortex | ||
+ | voyager | ||
+ | VSBot | ||
+ | VSAgent | ||
+ | VSE | ||
+ | VYU2 | ||
+ | W3C-checklink | ||
+ | W3C_Validator | ||
+ | W3C_Unicorn | ||
+ | W3C_I18n-Checker | ||
+ | W3C_CSS_Validator | ||
+ | W3C-linkchecker | ||
+ | W3C | ||
+ | W3C-Unicorn | ||
+ | w3m | ||
+ | w3mir | ||
+ | wappalyzer | ||
+ | WAPMAI | ||
+ | watchmouse | ||
+ | Webaroo | ||
+ | WebArche | ||
+ | WebBot | ||
+ | Web Capture | ||
+ | Web Data | ||
+ | WebEx | ||
+ | WebImages | ||
+ | WebImageBot | ||
+ | WebLibrarian | ||
+ | WebMiner | ||
+ | WebMonitor | ||
+ | WebPatrol | ||
+ | WebRankBot | ||
+ | WebScrape | ||
+ | WebScraper | ||
+ | WebSearch | ||
+ | WebShot | ||
+ | WebSniffer | ||
+ | WebSphinx | ||
+ | WebVac | ||
+ | WebIndexer | ||
+ | Wepalabs | ||
+ | WebMon | ||
+ | Webinator | ||
+ | WebUser | ||
+ | WebVac_bot | ||
+ | Whale | ||
+ | |||
+ | Whibse | ||
+ | Whiskers | ||
+ | Wicked | ||
+ | Wide | ||
+ | Willing | ||
+ | Windows-RSS-Platform | ||
+ | Wise | ||
+ | Wisebot | ||
+ | Wishy | ||
+ | Wixbot | ||
+ | WordBot | ||
+ | WordPress | ||
+ | Wscbot | ||
+ | Wsearch | ||
+ | WSGI | ||
+ | WWWBots | ||
+ | Wykop | ||
+ | Xaldon | ||
+ | Xerka | ||
+ | Xerox | ||
+ | XGET | ||
+ | XiangcBot | ||
+ | |||
+ | XingShares | ||
+ | Xing-Share | ||
+ | Xmeta | ||
+ | Xmith | ||
+ | XMLSitemapGenerator | ||
+ | Xml-RPC | ||
+ | XoviBot | ||
+ | Xpress | ||
+ | xStarter | ||
+ | Yahoo-Search | ||
+ | Yahoo-Slurp | ||
+ | YahooCacheSystem | ||
+ | Yahoo-Blogs | ||
+ | YahooYSMcm | ||
+ | Yanga | ||
+ | Yasaklibot | ||
+ | Yeti | ||
+ | Yioop | ||
+ | YisouSpider | ||
+ | YisouVideo | ||
+ | YisouImage | ||
+ | YisouBot | ||
+ | YisouSearchBot | ||
+ | YioopBot | ||
+ | YisouImageBot | ||
+ | YoudaoImageBot | ||
+ | YoudaoSearchBot | ||
+ | YoudaoSpider | ||
+ | YoudaoVideoSpider | ||
+ | Yottaa | ||
+ | Yottos | ||
+ | YQBot | ||
+ | YrBot | ||
+ | YSEBot | ||
+ | YSLBot | ||
+ | Yuilib | ||
+ | YunSeek | ||
+ | Yuntongzhi | ||
+ | Yunyun | ||
+ | Zabbix | ||
+ | Zao | ||
+ | Zebra | ||
+ | ZeBot | ||
+ | Zend2 | ||
+ | Zend_Http_Client | ||
+ | Zerx | ||
+ | Zibber | ||
+ | Ziibra | ||
+ | Zimmerman | ||
+ | Zing | ||
+ | Ziot | ||
+ | ZipBot | ||
+ | Zipsa | ||
+ | Zoinks | ||
+ | Zombie.js | ||
+ | ZooplaBot | ||
+ | Zotero | ||
+ | ZoomInfo | ||
+ | Zune | ||
+ | Zurpha | ||
+ | Zuupa | ||
+ | ZyngaBot | ||
+ | ZyXel | ||
Версия 14:23, 31 августа 2024
Содержание |
Данный список ботов предоставлен ЦМУ ССОП. Используйте блокировки по этому списку на свой страх и риск. Мы рекомендуем самостоятельно оценивать целесообразность данных блокировок и их корректность. Помимо очевидных ботов, список блокирует многие средства автоматизации, API, и так далее, поэтому относитесь к этому внимательно.
Список блокировки для web.config
Это подхоит к инструкции "Блокировка по заголовку User-Agent через конфиг внутри сайта".
<add input="{HTTP_USER_AGENT}" pattern="(.*)01h4x(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)360Spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)404checker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)404enemy(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)80legs(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Abonti(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Aboundex(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Aboundexbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Acunetix(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ADmantX(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)adscanner(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)AdsTxtCrawlerTP(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)AfD-Verbotsverfahren(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)AhrefsBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)AIBOT(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)AiHitBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Aipbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Alexibot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ALittle Client(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Alligator(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)AllSubmitter(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)AlphaBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Anarchie(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Anarchy(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Anarchy99(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Ankit(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Anthill(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)anthropic-ai(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Apexoo(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)archive(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)arquivo-web-crawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)arquivo(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Aspiegel(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ASPSeek(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Asterias(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Atomseobot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Attach(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)autoemailspider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)awario(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)AwarioBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)AwarioRssBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)AwarioSmartBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BackDoorBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Backlink-Ceck(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)backlink-check(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BacklinkCrawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BackStreet(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BackWeb(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Badass(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Bandit(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Barkrowler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BatchFTP(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Battleztar Bazinga(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BBBike(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BDCbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BDFetch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BetaBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Bigfoot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Bitacle(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Black Hole(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Blackboard(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BlackWidow(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BLEXBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Blow(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BlowFish(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Boardreader(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Bolt(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BotALot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Brandprotect(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Brandwatch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Buck(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Buddy(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BuiltBotTough(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BuiltWith(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Bullseye(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BunnySlippers(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)BuzzSumo(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Bytespider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)cah(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Calculon(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)CATExplorador(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)CazoodleBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)CCBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Cegbfeieh(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)CensysInspect(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ChatGPT-User(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)check1(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)CheeseBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)CherryPicker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)CheTeam(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ChinaClaw(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Chlooe(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Citoid(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Claritybot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)clark-crawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Cliqzbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Cloud mapping(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)coccocbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Cocolyzebot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)CODE87(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Cogentbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)cognitiveseo(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)cohere-ai(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Collector(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)com(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Copier(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)CopyRightCheck(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Copyscape(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Cosmos(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Craftbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)crawl(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)crawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)crawler4j(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Crawling at Home Project(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)CrazyWebCrawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Crescent(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)CrunchBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)CSHttp(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Curious(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Custo(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)CyotekWebCopy(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DatabaseDriverMysqli(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DataCha0s(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)dataforseo(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)dataforseobot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DBLBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)demandbase-bot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Demon(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Deusu(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Devil(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Digincore(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DigitalPebble(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DIIbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Dirbuster(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Disco(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Discobot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Discoverybot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Dispatch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DittoSpyder(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DnBCrawler-Analytics(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DnyzBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DomainAppender(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DomainCrawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Domains Project(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DomainSigmaCrawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)domainsproject(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DomainStatsBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DomCopBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Dotbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Download Wonder(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Dragonfly(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Drip(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DSearch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)DTS Agent(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)EasyDL(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Ebingbong(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)eCatch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ECCP/1(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Ecxi(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)EirGrabber(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)EMail Siphon(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)EMail Wolf(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)EroCrawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)evc-batch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Evil(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Exabot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Express WebPictures(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ExtLinksBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Extractor(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ExtractorPro(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Extreme Picture Finder(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)EyeNetIE(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Ezooms(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)FacebookBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)facebookscraper(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)FDM(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)FemtosearchBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)FHscan(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Fimap(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)FlashGet(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Flunky(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Foobot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Freeuploader(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)FrontPage(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Fuzz(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)FyberSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Fyrebot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)G-i-g-a-b-o-t(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)GalaxyBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Genieo(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)GermCrawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Getintent(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)GetRight(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)GetWeb(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Gigabot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Go-Ahead-Got-It(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Go\!Zilla(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)gopher(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Gotit(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)GoZilla(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)GPTBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Grabber(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)GrabNet(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Grafula(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)GrapeFX(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)GrapeshotCrawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)GridBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Haansoft(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)HaosouSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Harvest(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Havij(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)HEADMasterSEO(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Heritrix(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Hloader(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)HMView(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)HonoluluBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)HTMLparser(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)HTTrack(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Humanlinks(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)HybridBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Iblog(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Id-search(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)IDBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)IDBTE4M(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)IlseBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Image Fetch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Image Sucker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)imagesift(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ImagesiftBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)IndeedBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Indy Library(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)InfoNaviRobot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Information Security Team InfraSec Scanner(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)InfoTekies(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)InfraSec Scanner(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)instabid(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Intelliseek(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)InterGET(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Internet Ninja(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)InternetMeasurement(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)InternetSeer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)internetVista monitor(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ips-agent(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Iria(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)IRLbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)isitwp(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Iskanie(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)IstellaBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)iubenda-radar(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)JamesBOT(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Jbrofuzz(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)JennyBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)JetCar(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Jetty(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)JikeSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)JOC Web Spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Joomla(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Jorgee(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)JustView(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Jyxobot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Kenjin Spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Keybot Translation-Search-Machine(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Keyword Density(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Kinza(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Kozmosbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Lanshanbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Larbin(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Leap(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LeechFTP(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LeechGet(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LexiBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Lftp(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LibWeb(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Libwhisker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LieBaoFast(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Lightspeedsystems(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Likse(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Linkbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)linkdexbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LinkextractorPro(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)linkfluence(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LinkpadBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LinkScan(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LinksManager(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LinkWalker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LinqiaMetadataDownloaderBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LinqiaRSSBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LinqiaScrapeBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Lipperhey(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Lipperhey Spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Litemage_walker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Lmspider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LNSpiderguy(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Ltx71(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)lwp-request(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)lwp-trivial(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Mag-Net(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Magnet(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)magpie-crawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Majestic SEO(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Majestic-SEO(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Majestic12(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)MarkMonitor(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)MarkWatch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Mass Downloader(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Masscan(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Mata Hari(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)MauiBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Mb2345Browser(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)MeanPath Bot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Meanpathbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Mediatoolkitbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)mediawords(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)MegaIndex(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Metauri(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)MFC_Tear_Sample(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)MicroMessenger(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)MIDown tool(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)MIIxpc(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Minefield(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Mister PiX(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)MJ12bot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Mojeek(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Mojolicious(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)MolokaiBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Morfeus Fucking Scanner(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Mozlila(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Mr(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)MSFrontPage(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)MSIECrawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Msrabot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)MTRobot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)muhstik-scan(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Musobot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Name Intelligence(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Nameprotect(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Navroad(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)NearSite(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Needle(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Nessus(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Net Vampire(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)NetAnts(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Netcraft(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)netEstate NE Crawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)NetLyzer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)NetMechanic(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)NetSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Nettrack(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Netvibes(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)NetZIP(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)NextGenSearchBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Nibbler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)NICErsPRO(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Niki-bot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Nikto(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)NimbleCrawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Nimbostratus(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Ninja(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Nmap(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)NPbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Nuclei(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Nutch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)oBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Octopus(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Offline Explorer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Offline Navigator(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)omgili(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)OnCrawl(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)openai(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Openfind(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)OpenLinkProfiler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)OpenVAS(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)OrangeBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)OrangeSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)OutclicksBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)OutfoxBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Page Analyzer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)page scorer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PageAnalyzer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PageGrabber(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PageScorer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PageThing(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Pandalytics(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Panscient(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Papa Foto(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Pavuk(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)pcBrowser(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PeoplePal(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Petalbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PHPCrawl(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Pi-Monster(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Picscout(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Picsearch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PictureFinder(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Piepmatz(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Pimonster(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Pixray(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PleaseCrawl(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)plumanalytics(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Pockey(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)POE-Component-Client-HTTP(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)polaris version(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)probe-image-size(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Probethenet(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ProPowerBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ProWebWalker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Proximic(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Psbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Pu_iN(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Pump(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PxBroker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PyCurl(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)QueryN Metasearch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Quick-Crawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Rainbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RankActive(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RankActiveLinkBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RankFlex(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RankingBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RankingBot2(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Rankivabot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RankurBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Re-re(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RealDownload(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Reaper(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RebelMouse(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Recorder(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RedesScrapy(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ReGet(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RepoMonkey(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Ripper(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ripz(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RocketCrawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Rogerbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RSSingBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)s1z(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SalesIntelligent(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)satoristudio(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SBIder(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)scalaj-http(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)scan(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ScanAlert(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Scanbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ScoutJet(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Scrapy(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Screaming(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ScreenerBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ScrepyBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Searchestate(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SearchmetricsBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Seekport(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SeekportBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SemanticJuice(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Semrush(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SemrushBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SentiBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SenutoBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)seobility(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SeobilityBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)seocompany(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SEOkicks(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SEOkicks-Robot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SEOlyticsCrawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Seomoz(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SEOprofiler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)seoscanners(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SeoSiteCheckup(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)seostar(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SEOstats(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)serpstatbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)sexsearcher(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Shodan(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Siphon(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SISTRIX(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Site Sucker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sitebeam(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)sitechecker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SiteCheckerBotCrawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SiteExplorer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Siteimprove(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SiteLockSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)siteripz(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SiteSnagger(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SiteSucker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sitevigil(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SlySearch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SmartDownload(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SMTBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Snake(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Snapbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Snoopy(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SocialRankIOBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sociscraper(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sogou web spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)sogouspider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sosospider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sottopop(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)sp_auditbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SpaceBison(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Spammen(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SpankBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Spanner(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Spbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Spinn3r(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SputnikBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)spyfu(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sqlmap(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sqlworm(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sqworm(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Steeler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Stripper(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sucker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sucuri(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SuperBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SuperHTTP(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Surfbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SurveyBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Suzuran(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Swiftbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)sysscan(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Szukacz(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)T0PHackTeam(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)T8Abot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)tAkeOut(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Teleport(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TeleportPro(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Telesoft(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Telesphoreo(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Telesphorep(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)The Intraformant(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TheNomad(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Thumbor(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TightTwatBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TinyTestBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Titan(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Toata(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Toweyabot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tracemyfile(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Trendiction(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Trendictionbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)True_Robot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Turingos(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Turnitin(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TurnitinBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TwengaBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Twice(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Typhoeus(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ubermetrics-technologies(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)UnisterBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Upflow(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)URLy Warning(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)URLy(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)V-BOT(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Vacuum(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Vagabondo(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)VB Project(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)VCI(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)VelenPublicWebCrawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)VeriCiteCrawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)VidibleScraper(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Virusdie(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)VoidEYE(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Voil(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Voltron(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)voyagerx(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wallpapers(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wallpapers/3(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WallpapersHD(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WASALive-Bot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WBSearchBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Web Auto(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Web Collage(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Web Enhancer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Web Fetch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Web Fuck(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Web Pix(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Web Sauger(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Web Sucker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Webalta(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebAuto(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebBandit(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebCollage(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebCopier(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WEBDAV(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebEnhancer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebFetch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebFuck(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)webgains-bot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebGo IS(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebImageCollector(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebLeacher(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebmasterWorldForumBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)webmeup-crawler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebPix(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)webpros(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)webprosbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebReaper(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebSauger(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Webshag(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Website Quester(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebsiteExtractor(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebsiteQuester(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Webster(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebStripper(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebSucker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebWhacker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebZIP(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WeSEE(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Whack(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Whacker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Whatweb(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Who(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Widow(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WinHTTrack(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WiseGuys Robot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WISENutbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wonderbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Woobot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wotbox(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wprecon(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WPScan(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WWW-Collector-E(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WWW-Mechanize(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WWWOFFLE(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Xaldon WebSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Xaldon_WebSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Xenu(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)xpymep1(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YaK(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YoudaoBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zade(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zauba(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zermelo(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zeus(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)zgrab(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zitebot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ZmEu(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ZoominfoBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ZumBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ZyBorg(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ZoomBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WWW\:\:Mechanize(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)GT\:\:WWW(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)HTTP\:\:Lite(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)LWP\:\:Simple(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PECL\:\:HTTP(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)NoUserAgent(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)noxtrumbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Nsauditor(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Nuhk(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Nuzzel(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)NuzzelFeedBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)NuzzelMediaBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Nymesis(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)NymesisBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Ocelli(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)OmahaBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)OmniExplorer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Onebot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)OnetSzukaj(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Onfolio(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Openindexspider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)OpenWebSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)OxBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PagePeeker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PageScrape(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PagetScraper(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Pcore-HTTPClient(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Perl LWP(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PersonaPilot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Phantom(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PhantomJS(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PHP\:\:HTTPClient(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Pingdom(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PiplBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Plukkie(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Pogodak(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ProCogBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Python-httplib2(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Python-urllib(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)PyWebScraper(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Qirina Hurdle(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)QResync(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Qwantify(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Rambler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RankSonicSiteAuditor(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RankingHit(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RealPlayer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Repo(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RepoDev(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RepoJob(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Riddler(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RivalIQ(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RixBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Robozilla(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)RoHe(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Ruhm(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SBL-Bot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Scrubby(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)scrutiny(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ScrutinyBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Seeker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)semalt(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SEO Suite(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SEOENGBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SEOlytics(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SentiSumBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)seochat(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SEOEngine(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SEOMetriks(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SerendeputyBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Server Density Service Monitoring Bot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SeznamBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SFR-Sharepoint(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ShaMan(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Shockwave Flash(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ShopWiki(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Siege(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SimplePie(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sitebulb(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sixy Search(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sixy Search V2(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SkimBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Skrybot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Skywalker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Slackerbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SlackSocialBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Slackbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Slap(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SliquaBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Snoopie(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Snufkinbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sogou(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sogou News Spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sogou Orion spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sogou Pic Spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sogou Spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sogou blog(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sogou inst spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sogou news(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sogou orion(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sogou web(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sogou-Pic-Spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sogou-Test-Spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sogou-Test(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SoftHyde(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SoMegaBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sonarcloud(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SoNET(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sonic(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SonicWall(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Soso Spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Spacy(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SpamBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SpamBayes(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SpamMoose(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SpamPatrol(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Speedy(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Speedy Spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SpeedySpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SpiderMan(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Spider_Monkey(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SpiderBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SpiderLine(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SpiderMonkey(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SpiderProtector(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SpiderSilk(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)sprinklr-spider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Spyder(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SQLIHelper(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Squider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SSIbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SSLLabs(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SSLTestBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Startsearch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)StartpageBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)StartSiteBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Static(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)StellarBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Strategistbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Stratbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)stratosec-bot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)StrokeBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)StudyBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)StudiotimeBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SuggyBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Supernova(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Supersalonbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Surebot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Surveillance(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Suzie(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SVM(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SWEBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SwiteBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SXN(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SyAgent(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Syndex(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Syndexbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Synthesio(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Sysomos(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)SzukajSe(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tactical(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tagasaurus(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)tAganalytics(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tailor4Less(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tailwind(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Talkwalker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TalkwalkerBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TalkwalkerRSSBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TalkwalkerSmartBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tantalos(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TaringaBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tarybot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tazman-Agg(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Teamsoft(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TeepsaBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TelegramBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Telekom-Bot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Teoma(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TeraBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Testbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TextRazor(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Thalesbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)The Answer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)The Kraken(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)The Nomad(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)The-Schmidts(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TheWorld(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Thirdlightbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Thunderstone(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tixati(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tobasco(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Token(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Toll Free(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tomato Bot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TomatoCat(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tonic(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Topbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Topwebsearch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Torbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ToutiaoSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tpbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Trackback(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tracklock(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Trahit(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Tramp(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Trams(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Translate\.ru bot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Translate\.ru(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Trax(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Trendogic(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Trendspottr(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TridionBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Triggitbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Trilby(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)trufflehog(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Truefinder(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Truthybot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TUCOWS(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Twingly(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)TwistedPageGetter(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Twitterbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)U0xA-Bot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Ubermetrics(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Ubrowse(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)uipbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)uLink(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)UnchaindSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)UniteBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)UnshortenBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Updownerbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)UpdatePatrol(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)UpsightBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Urlfilebot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)URLitor(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)URLMonitorBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)URLSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Urlwatch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)User-Agent(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Validator(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)VBProjectBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Vespa(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)VestrSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ViaMichelin(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Vider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)VideoSurf(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ViewState(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)VirusTotal(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)vkShare(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Vortex(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)voyager(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)VSBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)VSAgent(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)VSE(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)VYU2(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)W3C-checklink(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)W3C_Validator(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)W3C_Unicorn(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)W3C_I18n-Checker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)W3C_CSS_Validator(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)W3C-linkchecker(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)W3C(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)W3C-Unicorn(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)w3m(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)w3mir(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)wappalyzer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WAPMAI(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)watchmouse(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Webaroo(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebArche(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Web Capture(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Web Data(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebEx(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebImages(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebImageBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebLibrarian(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebMiner(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebMonitor(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebPatrol(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebRankBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebScrape(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebScraper(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebSearch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebShot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebSniffer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebSphinx(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebVac(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebIndexer(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wepalabs(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebMon(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Webinator(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebUser(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WebVac_bot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Whale(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WhatsApp(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Whibse(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Whiskers(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wicked(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wide(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Willing(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Windows-RSS-Platform(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wise(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wisebot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wishy(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wixbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WordBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WordPress(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wscbot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wsearch(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WSGI(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)WWWBots(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Wykop(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Xaldon(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Xerka(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Xerox(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)XGET(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)XiangcBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Xing(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)XingShares(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Xing-Share(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Xmeta(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Xmith(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)XMLSitemapGenerator(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Xml-RPC(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)XoviBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Xpress(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)xStarter(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Yahoo-Search(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Yahoo-Slurp(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YahooCacheSystem(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Yahoo-Blogs(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YahooYSMcm(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Yanga(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Yasaklibot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Yeti(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Yioop(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YisouSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YisouVideo(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YisouImage(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YisouBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YisouSearchBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YioopBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YisouImageBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YoudaoImageBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YoudaoSearchBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YoudaoSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YoudaoVideoSpider(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Yottaa(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Yottos(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YQBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YrBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YSEBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YSLBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Yuilib(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)YunSeek(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Yuntongzhi(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Yunyun(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zabbix(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zao(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zebra(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ZeBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zend2(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zend_Http_Client(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zerx(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zibber(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Ziibra(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zimmerman(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zing(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Ziot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ZipBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zipsa(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zoinks(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zombie\.js(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ZooplaBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zotero(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ZoomInfo(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zune(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zurpha(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)Zuupa(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ZyngaBot(.*)" /> <add input="{HTTP_USER_AGENT}" pattern="(.*)ZyXel(.*)" />
Список блокировок для .htaccess
Это подхоит к инструкции "Блокировка по заголовку User-Agent через конфиг внутри сайта".
Поместите это в начало вашего файла .htaccess в корне сайта.
RewriteEngine On RewriteCond %{HTTP_USER_AGENT} "01h4x" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "360Spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "404checker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "404enemy" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "80legs" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Abonti" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Aboundex" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Aboundexbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Acunetix" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ADmantX" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "adscanner" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "AdsTxtCrawlerTP" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "AfD-Verbotsverfahren" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "AhrefsBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "AIBOT" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "AiHitBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Aipbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Alexibot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ALittle Client" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Alligator" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "AllSubmitter" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "AlphaBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Anarchie" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Anarchy" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Anarchy99" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Ankit" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Anthill" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "anthropic-ai" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Apexoo" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "archive" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "arquivo-web-crawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "arquivo" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Aspiegel" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ASPSeek" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Asterias" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Atomseobot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Attach" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "autoemailspider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "awario" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "AwarioBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "AwarioRssBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "AwarioSmartBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BackDoorBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Backlink-Ceck" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "backlink-check" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BacklinkCrawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BackStreet" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BackWeb" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Badass" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Bandit" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Barkrowler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BatchFTP" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Battleztar Bazinga" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BBBike" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BDCbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BDFetch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BetaBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Bigfoot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Bitacle" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Black Hole" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Blackboard" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BlackWidow" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BLEXBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Blow" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BlowFish" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Boardreader" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Bolt" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BotALot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Brandprotect" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Brandwatch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Buck" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Buddy" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BuiltBotTough" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BuiltWith" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Bullseye" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BunnySlippers" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "BuzzSumo" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Bytespider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "cah" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Calculon" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "CATExplorador" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "CazoodleBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "CCBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Cegbfeieh" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "CensysInspect" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ChatGPT-User" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "check1" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "CheeseBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "CherryPicker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "CheTeam" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ChinaClaw" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Chlooe" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Citoid" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Claritybot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "clark-crawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Cliqzbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Cloud mapping" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "coccocbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Cocolyzebot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "CODE87" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Cogentbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "cognitiveseo" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "cohere-ai" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Collector" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "com" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Copier" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "CopyRightCheck" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Copyscape" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Cosmos" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Craftbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "crawl" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "crawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "crawler4j" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Crawling at Home Project" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "CrazyWebCrawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Crescent" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "CrunchBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "CSHttp" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Curious" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Custo" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "CyotekWebCopy" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DatabaseDriverMysqli" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DataCha0s" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "dataforseo" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "dataforseobot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DBLBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "demandbase-bot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Demon" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Deusu" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Devil" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Digincore" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DigitalPebble" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DIIbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Dirbuster" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Disco" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Discobot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Discoverybot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Dispatch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DittoSpyder" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DnBCrawler-Analytics" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DnyzBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DomainAppender" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DomainCrawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Domains Project" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DomainSigmaCrawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "domainsproject" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DomainStatsBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DomCopBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Dotbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Download Wonder" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Dragonfly" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Drip" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DSearch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "DTS Agent" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "EasyDL" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Ebingbong" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "eCatch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ECCP/1" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Ecxi" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "EirGrabber" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "EMail Siphon" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "EMail Wolf" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "EroCrawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "evc-batch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Evil" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Exabot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Express WebPictures" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ExtLinksBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Extractor" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ExtractorPro" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Extreme Picture Finder" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "EyeNetIE" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Ezooms" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "FacebookBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "facebookscraper" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "FDM" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "FemtosearchBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "FHscan" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Fimap" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "FlashGet" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Flunky" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Foobot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Freeuploader" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "FrontPage" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Fuzz" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "FyberSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Fyrebot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "G-i-g-a-b-o-t" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "GalaxyBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Genieo" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "GermCrawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Getintent" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "GetRight" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "GetWeb" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Gigabot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Go-Ahead-Got-It" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Go!Zilla" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "gopher" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Gotit" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "GoZilla" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "GPTBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Grabber" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "GrabNet" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Grafula" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "GrapeFX" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "GrapeshotCrawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "GridBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Haansoft" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "HaosouSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Harvest" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Havij" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "HEADMasterSEO" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Heritrix" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Hloader" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "HMView" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "HonoluluBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "HTMLparser" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "HTTrack" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Humanlinks" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "HybridBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Iblog" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Id-search" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "IDBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "IDBTE4M" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "IlseBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Image Fetch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Image Sucker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "imagesift" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ImagesiftBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "IndeedBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Indy Library" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "InfoNaviRobot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Information Security Team InfraSec Scanner" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "InfoTekies" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "InfraSec Scanner" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "instabid" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Intelliseek" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "InterGET" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Internet Ninja" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "InternetMeasurement" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "InternetSeer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "internetVista monitor" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ips-agent" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Iria" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "IRLbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "isitwp" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Iskanie" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "IstellaBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "iubenda-radar" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "JamesBOT" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Jbrofuzz" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "JennyBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "JetCar" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Jetty" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "JikeSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "JOC Web Spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Joomla" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Jorgee" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "JustView" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Jyxobot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Kenjin Spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Keybot Translation-Search-Machine" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Keyword Density" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Kinza" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Kozmosbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Lanshanbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Larbin" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Leap" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LeechFTP" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LeechGet" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LexiBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Lftp" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LibWeb" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Libwhisker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LieBaoFast" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Lightspeedsystems" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Likse" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Linkbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "linkdexbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LinkextractorPro" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "linkfluence" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LinkpadBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LinkScan" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LinksManager" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LinkWalker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LinqiaMetadataDownloaderBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LinqiaRSSBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LinqiaScrapeBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Lipperhey" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Lipperhey Spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Litemage_walker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Lmspider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LNSpiderguy" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Ltx71" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "lwp-request" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "lwp-trivial" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Mag-Net" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Magnet" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "magpie-crawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Majestic SEO" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Majestic-SEO" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Majestic12" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "MarkMonitor" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "MarkWatch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Mass Downloader" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Masscan" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Mata Hari" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "MauiBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Mb2345Browser" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "MeanPath Bot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Meanpathbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Mediatoolkitbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "mediawords" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "MegaIndex" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Metauri" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "MFC_Tear_Sample" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "MicroMessenger" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "MIDown tool" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "MIIxpc" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Minefield" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Mister PiX" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "MJ12bot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Mojeek" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Mojolicious" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "MolokaiBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Morfeus Fucking Scanner" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Mozlila" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Mr" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "MSFrontPage" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "MSIECrawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Msrabot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "MTRobot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "muhstik-scan" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Musobot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Name Intelligence" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Nameprotect" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Navroad" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "NearSite" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Needle" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Nessus" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Net Vampire" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "NetAnts" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Netcraft" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "netEstate NE Crawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "NetLyzer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "NetMechanic" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "NetSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Nettrack" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Netvibes" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "NetZIP" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "NextGenSearchBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Nibbler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "NICErsPRO" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Niki-bot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Nikto" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "NimbleCrawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Nimbostratus" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Ninja" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Nmap" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "NPbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Nuclei" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Nutch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "oBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Octopus" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Offline Explorer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Offline Navigator" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "omgili" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "OnCrawl" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "openai" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Openfind" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "OpenLinkProfiler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "OpenVAS" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "OrangeBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "OrangeSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "OutclicksBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "OutfoxBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Page Analyzer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "page scorer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PageAnalyzer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PageGrabber" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PageScorer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PageThing" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Pandalytics" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Panscient" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Papa Foto" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Pavuk" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "pcBrowser" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PeoplePal" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Petalbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PHPCrawl" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Pi-Monster" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Picscout" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Picsearch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PictureFinder" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Piepmatz" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Pimonster" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Pixray" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PleaseCrawl" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "plumanalytics" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Pockey" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "POE-Component-Client-HTTP" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "polaris version" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "probe-image-size" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Probethenet" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ProPowerBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ProWebWalker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Proximic" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Psbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Pu_iN" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Pump" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PxBroker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PyCurl" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "QueryN Metasearch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Quick-Crawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Rainbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RankActive" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RankActiveLinkBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RankFlex" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RankingBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RankingBot2" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Rankivabot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RankurBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Re-re" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RealDownload" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Reaper" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RebelMouse" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Recorder" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RedesScrapy" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ReGet" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RepoMonkey" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Ripper" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ripz" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RocketCrawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Rogerbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RSSingBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "s1z" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SalesIntelligent" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "satoristudio" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SBIder" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "scalaj-http" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "scan" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ScanAlert" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Scanbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ScoutJet" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Scrapy" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Screaming" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ScreenerBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ScrepyBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Searchestate" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SearchmetricsBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Seekport" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SeekportBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SemanticJuice" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Semrush" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SemrushBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SentiBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SenutoBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "seobility" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SeobilityBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "seocompany" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SEOkicks" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SEOkicks-Robot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SEOlyticsCrawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Seomoz" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SEOprofiler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "seoscanners" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SeoSiteCheckup" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "seostar" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SEOstats" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "serpstatbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "sexsearcher" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Shodan" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Siphon" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SISTRIX" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Site Sucker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sitebeam" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "sitechecker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SiteCheckerBotCrawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SiteExplorer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Siteimprove" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SiteLockSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "siteripz" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SiteSnagger" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SiteSucker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sitevigil" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SlySearch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SmartDownload" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SMTBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Snake" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Snapbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Snoopy" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SocialRankIOBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sociscraper" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sogou web spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "sogouspider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sosospider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sottopop" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "sp_auditbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SpaceBison" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Spammen" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SpankBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Spanner" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Spbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Spinn3r" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SputnikBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "spyfu" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sqlmap" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sqlworm" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sqworm" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Steeler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Stripper" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sucker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sucuri" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SuperBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SuperHTTP" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Surfbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SurveyBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Suzuran" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Swiftbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "sysscan" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Szukacz" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "T0PHackTeam" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "T8Abot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "tAkeOut" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Teleport" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TeleportPro" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Telesoft" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Telesphoreo" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Telesphorep" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "The Intraformant" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TheNomad" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Thumbor" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TightTwatBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TinyTestBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Titan" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Toata" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Toweyabot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tracemyfile" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Trendiction" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Trendictionbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "True_Robot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Turingos" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Turnitin" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TurnitinBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TwengaBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Twice" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Typhoeus" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ubermetrics-technologies" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "UnisterBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Upflow" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "URLy Warning" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "URLy" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "V-BOT" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Vacuum" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Vagabondo" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "VB Project" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "VCI" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "VelenPublicWebCrawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "VeriCiteCrawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "VidibleScraper" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Virusdie" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "VoidEYE" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Voil" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Voltron" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "voyagerx" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wallpapers" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wallpapers/3" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WallpapersHD" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WASALive-Bot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WBSearchBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Web Auto" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Web Collage" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Web Enhancer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Web Fetch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Web Fuck" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Web Pix" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Web Sauger" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Web Sucker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Webalta" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebAuto" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebBandit" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebCollage" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebCopier" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WEBDAV" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebEnhancer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebFetch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebFuck" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "webgains-bot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebGo IS" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebImageCollector" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebLeacher" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebmasterWorldForumBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "webmeup-crawler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebPix" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "webpros" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "webprosbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebReaper" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebSauger" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Webshag" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Website Quester" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebsiteExtractor" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebsiteQuester" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Webster" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebStripper" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebSucker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebWhacker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebZIP" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WeSEE" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Whack" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Whacker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Whatweb" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Who" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Widow" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WinHTTrack" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WiseGuys Robot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WISENutbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wonderbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Woobot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wotbox" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wprecon" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WPScan" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WWW-Collector-E" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WWW-Mechanize" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WWWOFFLE" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Xaldon WebSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Xaldon_WebSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Xenu" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "xpymep1" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YaK" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YoudaoBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zade" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zauba" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zermelo" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zeus" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "zgrab" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zitebot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ZmEu" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ZoominfoBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ZumBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ZyBorg" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ZoomBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WWW::Mechanize" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "GT::WWW" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "HTTP::Lite" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "LWP::Simple" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PECL::HTTP" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "NoUserAgent" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "noxtrumbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Nsauditor" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Nuhk" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Nuzzel" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "NuzzelFeedBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "NuzzelMediaBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Nymesis" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "NymesisBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Ocelli" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "OmahaBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "OmniExplorer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Onebot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "OnetSzukaj" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Onfolio" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Openindexspider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "OpenWebSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "OxBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PagePeeker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PageScrape" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PagetScraper" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Pcore-HTTPClient" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Perl LWP" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PersonaPilot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Phantom" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PhantomJS" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PHP::HTTPClient" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Pingdom" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PiplBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Plukkie" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Pogodak" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ProCogBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Python-httplib2" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Python-urllib" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "PyWebScraper" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Qirina Hurdle" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "QResync" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Qwantify" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Rambler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RankSonicSiteAuditor" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RankingHit" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RealPlayer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Repo" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RepoDev" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RepoJob" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Riddler" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RivalIQ" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RixBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Robozilla" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "RoHe" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Ruhm" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SBL-Bot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Scrubby" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "scrutiny" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ScrutinyBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Seeker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "semalt" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SEO Suite" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SEOENGBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SEOlytics" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SentiSumBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "seochat" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SEOEngine" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SEOMetriks" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SerendeputyBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Server Density Service Monitoring Bot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SeznamBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SFR-Sharepoint" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ShaMan" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Shockwave Flash" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ShopWiki" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Siege" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SimplePie" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sitebulb" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sixy Search" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sixy Search V2" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SkimBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Skrybot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Skywalker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Slackerbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SlackSocialBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Slackbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Slap" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SliquaBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Snoopie" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Snufkinbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sogou" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sogou News Spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sogou Orion spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sogou Pic Spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sogou Spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sogou blog" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sogou inst spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sogou news" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sogou orion" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sogou web" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sogou-Pic-Spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sogou-Test-Spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sogou-Test" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SoftHyde" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SoMegaBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sonarcloud" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SoNET" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sonic" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SonicWall" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Soso Spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Spacy" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SpamBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SpamBayes" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SpamMoose" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SpamPatrol" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Speedy" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Speedy Spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SpeedySpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SpiderMan" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Spider_Monkey" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SpiderBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SpiderLine" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SpiderMonkey" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SpiderProtector" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SpiderSilk" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "sprinklr-spider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Spyder" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SQLIHelper" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Squider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SSIbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SSLLabs" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SSLTestBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Startsearch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "StartpageBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "StartSiteBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Static" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "StellarBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Strategistbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Stratbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "stratosec-bot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "StrokeBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "StudyBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "StudiotimeBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SuggyBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Supernova" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Supersalonbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Surebot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Surveillance" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Suzie" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SVM" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SWEBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SwiteBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SXN" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SyAgent" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Syndex" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Syndexbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Synthesio" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Sysomos" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "SzukajSe" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tactical" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tagasaurus" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "tAganalytics" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tailor4Less" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tailwind" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Talkwalker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TalkwalkerBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TalkwalkerRSSBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TalkwalkerSmartBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tantalos" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TaringaBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tarybot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tazman-Agg" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Teamsoft" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TeepsaBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TelegramBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Telekom-Bot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Teoma" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TeraBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Testbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TextRazor" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Thalesbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "The Answer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "The Kraken" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "The Nomad" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "The-Schmidts" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TheWorld" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Thirdlightbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Thunderstone" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tixati" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tobasco" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Token" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Toll Free" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tomato Bot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TomatoCat" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tonic" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Topbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Topwebsearch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Torbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ToutiaoSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tpbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Trackback" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tracklock" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Trahit" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Tramp" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Trams" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Translate.ru bot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Translate.ru" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Trax" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Trendogic" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Trendspottr" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TridionBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Triggitbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Trilby" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "trufflehog" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Truefinder" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Truthybot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TUCOWS" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Twingly" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "TwistedPageGetter" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Twitterbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "U0xA-Bot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Ubermetrics" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Ubrowse" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "uipbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "uLink" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "UnchaindSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "UniteBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "UnshortenBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Updownerbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "UpdatePatrol" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "UpsightBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Urlfilebot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "URLitor" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "URLMonitorBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "URLSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Urlwatch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "User-Agent" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Validator" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "VBProjectBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Vespa" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "VestrSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ViaMichelin" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Vider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "VideoSurf" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ViewState" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "VirusTotal" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "vkShare" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Vortex" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "voyager" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "VSBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "VSAgent" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "VSE" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "VYU2" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "W3C-checklink" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "W3C_Validator" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "W3C_Unicorn" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "W3C_I18n-Checker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "W3C_CSS_Validator" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "W3C-linkchecker" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "W3C" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "W3C-Unicorn" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "w3m" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "w3mir" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "wappalyzer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WAPMAI" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "watchmouse" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Webaroo" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebArche" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Web Capture" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Web Data" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebEx" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebImages" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebImageBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebLibrarian" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebMiner" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebMonitor" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebPatrol" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebRankBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebScrape" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebScraper" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebSearch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebShot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebSniffer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebSphinx" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebVac" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebIndexer" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wepalabs" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebMon" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Webinator" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebUser" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WebVac_bot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Whale" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WhatsApp" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Whibse" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Whiskers" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wicked" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wide" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Willing" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Windows-RSS-Platform" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wise" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wisebot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wishy" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wixbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WordBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WordPress" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wscbot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wsearch" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WSGI" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "WWWBots" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Wykop" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Xaldon" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Xerka" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Xerox" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "XGET" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "XiangcBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Xing" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "XingShares" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Xing-Share" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Xmeta" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Xmith" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "XMLSitemapGenerator" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Xml-RPC" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "XoviBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Xpress" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "xStarter" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Yahoo-Search" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Yahoo-Slurp" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YahooCacheSystem" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Yahoo-Blogs" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YahooYSMcm" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Yanga" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Yasaklibot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Yeti" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Yioop" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YisouSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YisouVideo" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YisouImage" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YisouBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YisouSearchBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YioopBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YisouImageBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YoudaoImageBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YoudaoSearchBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YoudaoSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YoudaoVideoSpider" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Yottaa" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Yottos" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YQBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YrBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YSEBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YSLBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Yuilib" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "YunSeek" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Yuntongzhi" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Yunyun" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zabbix" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zao" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zebra" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ZeBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zend2" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zend_Http_Client" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zerx" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zibber" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Ziibra" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zimmerman" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zing" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Ziot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ZipBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zipsa" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zoinks" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zombie.js" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ZooplaBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zotero" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ZoomInfo" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zune" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zurpha" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "Zuupa" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ZyngaBot" [NC] RewriteRule ^ - [F,L] RewriteCond %{HTTP_USER_AGENT} "ZyXel" [NC] RewriteRule ^ - [F,L]
Файл robots.txt
Это надо поместить в файл robots.txt в корне вашего сайта. Собственные инструкции допишите после этих строк, в конец.
User-agent: 01h4x Disallow: / User-agent: 360Spider Disallow: / User-agent: 404checker Disallow: / User-agent: 404enemy Disallow: / User-agent: 80legs Disallow: / User-agent: Abonti Disallow: / User-agent: Aboundex Disallow: / User-agent: Aboundexbot Disallow: / User-agent: Acunetix Disallow: / User-agent: ADmantX Disallow: / User-agent: adscanner Disallow: / User-agent: AdsTxtCrawlerTP Disallow: / User-agent: AfD-Verbotsverfahren Disallow: / User-agent: AhrefsBot Disallow: / User-agent: AIBOT Disallow: / User-agent: AiHitBot Disallow: / User-agent: Aipbot Disallow: / User-agent: Alexibot Disallow: / User-agent: ALittle Client Disallow: / User-agent: Alligator Disallow: / User-agent: AllSubmitter Disallow: / User-agent: AlphaBot Disallow: / User-agent: Anarchie Disallow: / User-agent: Anarchy Disallow: / User-agent: Anarchy99 Disallow: / User-agent: Ankit Disallow: / User-agent: Anthill Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Apexoo Disallow: / User-agent: archive Disallow: / User-agent: arquivo-web-crawler Disallow: / User-agent: arquivo Disallow: / User-agent: Aspiegel Disallow: / User-agent: ASPSeek Disallow: / User-agent: Asterias Disallow: / User-agent: Atomseobot Disallow: / User-agent: Attach Disallow: / User-agent: autoemailspider Disallow: / User-agent: awario Disallow: / User-agent: AwarioBot Disallow: / User-agent: AwarioRssBot Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: BackDoorBot Disallow: / User-agent: Backlink-Ceck Disallow: / User-agent: backlink-check Disallow: / User-agent: BacklinkCrawler Disallow: / User-agent: BackStreet Disallow: / User-agent: BackWeb Disallow: / User-agent: Badass Disallow: / User-agent: Bandit Disallow: / User-agent: Barkrowler Disallow: / User-agent: BatchFTP Disallow: / User-agent: Battleztar Bazinga Disallow: / User-agent: BBBike Disallow: / User-agent: BDCbot Disallow: / User-agent: BDFetch Disallow: / User-agent: BetaBot Disallow: / User-agent: Bigfoot Disallow: / User-agent: Bitacle Disallow: / User-agent: Black Hole Disallow: / User-agent: Blackboard Disallow: / User-agent: BlackWidow Disallow: / User-agent: BLEXBot Disallow: / User-agent: Blow Disallow: / User-agent: BlowFish Disallow: / User-agent: Boardreader Disallow: / User-agent: Bolt Disallow: / User-agent: BotALot Disallow: / User-agent: Brandprotect Disallow: / User-agent: Brandwatch Disallow: / User-agent: Buck Disallow: / User-agent: Buddy Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: BuiltWith Disallow: / User-agent: Bullseye Disallow: / User-agent: BunnySlippers Disallow: / User-agent: BuzzSumo Disallow: / User-agent: Bytespider Disallow: / User-agent: cah Disallow: / User-agent: Calculon Disallow: / User-agent: CATExplorador Disallow: / User-agent: CazoodleBot Disallow: / User-agent: CCBot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: CensysInspect Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: check1 Disallow: / User-agent: CheeseBot Disallow: / User-agent: CherryPicker Disallow: / User-agent: CheTeam Disallow: / User-agent: ChinaClaw Disallow: / User-agent: Chlooe Disallow: / User-agent: Citoid Disallow: / User-agent: Claritybot Disallow: / User-agent: clark-crawler Disallow: / User-agent: Cliqzbot Disallow: / User-agent: Cloud mapping Disallow: / User-agent: coccocbot Disallow: / User-agent: Cocolyzebot Disallow: / User-agent: CODE87 Disallow: / User-agent: Cogentbot Disallow: / User-agent: cognitiveseo Disallow: / User-agent: cohere-ai Disallow: / User-agent: Collector Disallow: / User-agent: com Disallow: / User-agent: Copier Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: Copyscape Disallow: / User-agent: Cosmos Disallow: / User-agent: Craftbot Disallow: / User-agent: crawl Disallow: / User-agent: crawler Disallow: / User-agent: crawler4j Disallow: / User-agent: Crawling at Home Project Disallow: / User-agent: CrazyWebCrawler Disallow: / User-agent: Crescent Disallow: / User-agent: CrunchBot Disallow: / User-agent: CSHttp Disallow: / User-agent: Curious Disallow: / User-agent: Custo Disallow: / User-agent: CyotekWebCopy Disallow: / User-agent: DatabaseDriverMysqli Disallow: / User-agent: DataCha0s Disallow: / User-agent: dataforseo Disallow: / User-agent: dataforseobot Disallow: / User-agent: DBLBot Disallow: / User-agent: demandbase-bot Disallow: / User-agent: Demon Disallow: / User-agent: Deusu Disallow: / User-agent: Devil Disallow: / User-agent: Digincore Disallow: / User-agent: DigitalPebble Disallow: / User-agent: DIIbot Disallow: / User-agent: Dirbuster Disallow: / User-agent: Disco Disallow: / User-agent: Discobot Disallow: / User-agent: Discoverybot Disallow: / User-agent: Dispatch Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DnBCrawler-Analytics Disallow: / User-agent: DnyzBot Disallow: / User-agent: DomainAppender Disallow: / User-agent: DomainCrawler Disallow: / User-agent: Domains Project Disallow: / User-agent: DomainSigmaCrawler Disallow: / User-agent: domainsproject Disallow: / User-agent: DomainStatsBot Disallow: / User-agent: DomCopBot Disallow: / User-agent: Dotbot Disallow: / User-agent: Download Wonder Disallow: / User-agent: Dragonfly Disallow: / User-agent: Drip Disallow: / User-agent: DSearch Disallow: / User-agent: DTS Agent Disallow: / User-agent: EasyDL Disallow: / User-agent: Ebingbong Disallow: / User-agent: eCatch Disallow: / User-agent: ECCP/1 Disallow: / User-agent: Ecxi Disallow: / User-agent: EirGrabber Disallow: / User-agent: EMail Siphon Disallow: / User-agent: EMail Wolf Disallow: / User-agent: EroCrawler Disallow: / User-agent: evc-batch Disallow: / User-agent: Evil Disallow: / User-agent: Exabot Disallow: / User-agent: Express WebPictures Disallow: / User-agent: ExtLinksBot Disallow: / User-agent: Extractor Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Extreme Picture Finder Disallow: / User-agent: EyeNetIE Disallow: / User-agent: Ezooms Disallow: / User-agent: FacebookBot Disallow: / User-agent: facebookscraper Disallow: / User-agent: FDM Disallow: / User-agent: FemtosearchBot Disallow: / User-agent: FHscan Disallow: / User-agent: Fimap Disallow: / User-agent: FlashGet Disallow: / User-agent: Flunky Disallow: / User-agent: Foobot Disallow: / User-agent: Freeuploader Disallow: / User-agent: FrontPage Disallow: / User-agent: Fuzz Disallow: / User-agent: FyberSpider Disallow: / User-agent: Fyrebot Disallow: / User-agent: G-i-g-a-b-o-t Disallow: / User-agent: GalaxyBot Disallow: / User-agent: Genieo Disallow: / User-agent: GermCrawler Disallow: / User-agent: Getintent Disallow: / User-agent: GetRight Disallow: / User-agent: GetWeb Disallow: / User-agent: Gigabot Disallow: / User-agent: Go-Ahead-Got-It Disallow: / User-agent: Go!Zilla Disallow: / User-agent: gopher Disallow: / User-agent: Gotit Disallow: / User-agent: GoZilla Disallow: / User-agent: GPTBot Disallow: / User-agent: Grabber Disallow: / User-agent: GrabNet Disallow: / User-agent: Grafula Disallow: / User-agent: GrapeFX Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: GridBot Disallow: / User-agent: Haansoft Disallow: / User-agent: HaosouSpider Disallow: / User-agent: Harvest Disallow: / User-agent: Havij Disallow: / User-agent: HEADMasterSEO Disallow: / User-agent: Heritrix Disallow: / User-agent: Hloader Disallow: / User-agent: HMView Disallow: / User-agent: HonoluluBot Disallow: / User-agent: HTMLparser Disallow: / User-agent: HTTrack Disallow: / User-agent: Humanlinks Disallow: / User-agent: HybridBot Disallow: / User-agent: Iblog Disallow: / User-agent: Id-search Disallow: / User-agent: IDBot Disallow: / User-agent: IDBTE4M Disallow: / User-agent: IlseBot Disallow: / User-agent: Image Fetch Disallow: / User-agent: Image Sucker Disallow: / User-agent: imagesift Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: IndeedBot Disallow: / User-agent: Indy Library Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: Information Security Team InfraSec Scanner Disallow: / User-agent: InfoTekies Disallow: / User-agent: InfraSec Scanner Disallow: / User-agent: instabid Disallow: / User-agent: Intelliseek Disallow: / User-agent: InterGET Disallow: / User-agent: Internet Ninja Disallow: / User-agent: InternetMeasurement Disallow: / User-agent: InternetSeer Disallow: / User-agent: internetVista monitor Disallow: / User-agent: ips-agent Disallow: / User-agent: Iria Disallow: / User-agent: IRLbot Disallow: / User-agent: isitwp Disallow: / User-agent: Iskanie Disallow: / User-agent: IstellaBot Disallow: / User-agent: iubenda-radar Disallow: / User-agent: JamesBOT Disallow: / User-agent: Jbrofuzz Disallow: / User-agent: JennyBot Disallow: / User-agent: JetCar Disallow: / User-agent: Jetty Disallow: / User-agent: JikeSpider Disallow: / User-agent: JOC Web Spider Disallow: / User-agent: Joomla Disallow: / User-agent: Jorgee Disallow: / User-agent: JustView Disallow: / User-agent: Jyxobot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keybot Translation-Search-Machine Disallow: / User-agent: Keyword Density Disallow: / User-agent: Kinza Disallow: / User-agent: Kozmosbot Disallow: / User-agent: Lanshanbot Disallow: / User-agent: Larbin Disallow: / User-agent: Leap Disallow: / User-agent: LeechFTP Disallow: / User-agent: LeechGet Disallow: / User-agent: LexiBot Disallow: / User-agent: Lftp Disallow: / User-agent: LibWeb Disallow: / User-agent: Libwhisker Disallow: / User-agent: LieBaoFast Disallow: / User-agent: Lightspeedsystems Disallow: / User-agent: Likse Disallow: / User-agent: Linkbot Disallow: / User-agent: linkdexbot Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: linkfluence Disallow: / User-agent: LinkpadBot Disallow: / User-agent: LinkScan Disallow: / User-agent: LinksManager Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinqiaMetadataDownloaderBot Disallow: / User-agent: LinqiaRSSBot Disallow: / User-agent: LinqiaScrapeBot Disallow: / User-agent: Lipperhey Disallow: / User-agent: Lipperhey Spider Disallow: / User-agent: Litemage_walker Disallow: / User-agent: Lmspider Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: Ltx71 Disallow: / User-agent: lwp-request Disallow: / User-agent: lwp-trivial Disallow: / User-agent: Mag-Net Disallow: / User-agent: Magnet Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Majestic SEO Disallow: / User-agent: Majestic-SEO Disallow: / User-agent: Majestic12 Disallow: / User-agent: MarkMonitor Disallow: / User-agent: MarkWatch Disallow: / User-agent: Mass Downloader Disallow: / User-agent: Masscan Disallow: / User-agent: Mata Hari Disallow: / User-agent: MauiBot Disallow: / User-agent: Mb2345Browser Disallow: / User-agent: MeanPath Bot Disallow: / User-agent: Meanpathbot Disallow: / User-agent: Mediatoolkitbot Disallow: / User-agent: mediawords Disallow: / User-agent: MegaIndex Disallow: / User-agent: Metauri Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: MicroMessenger Disallow: / User-agent: MIDown tool Disallow: / User-agent: MIIxpc Disallow: / User-agent: Minefield Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: Mojeek Disallow: / User-agent: Mojolicious Disallow: / User-agent: MolokaiBot Disallow: / User-agent: Morfeus Fucking Scanner Disallow: / User-agent: Mozlila Disallow: / User-agent: Mr Disallow: / User-agent: MSFrontPage Disallow: / User-agent: MSIECrawler Disallow: / User-agent: Msrabot Disallow: / User-agent: MTRobot Disallow: / User-agent: muhstik-scan Disallow: / User-agent: Musobot Disallow: / User-agent: Name Intelligence Disallow: / User-agent: Nameprotect Disallow: / User-agent: Navroad Disallow: / User-agent: NearSite Disallow: / User-agent: Needle Disallow: / User-agent: Nessus Disallow: / User-agent: Net Vampire Disallow: / User-agent: NetAnts Disallow: / User-agent: Netcraft Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: NetLyzer Disallow: / User-agent: NetMechanic Disallow: / User-agent: NetSpider Disallow: / User-agent: Nettrack Disallow: / User-agent: Netvibes Disallow: / User-agent: NetZIP Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: Nibbler Disallow: / User-agent: NICErsPRO Disallow: / User-agent: Niki-bot Disallow: / User-agent: Nikto Disallow: / User-agent: NimbleCrawler Disallow: / User-agent: Nimbostratus Disallow: / User-agent: Ninja Disallow: / User-agent: Nmap Disallow: / User-agent: NPbot Disallow: / User-agent: Nuclei Disallow: / User-agent: Nutch Disallow: / User-agent: oBot Disallow: / User-agent: Octopus Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Navigator Disallow: / User-agent: omgili Disallow: / User-agent: OnCrawl Disallow: / User-agent: openai Disallow: / User-agent: Openfind Disallow: / User-agent: OpenLinkProfiler Disallow: / User-agent: OpenVAS Disallow: / User-agent: OrangeBot Disallow: / User-agent: OrangeSpider Disallow: / User-agent: OutclicksBot Disallow: / User-agent: OutfoxBot Disallow: / User-agent: Page Analyzer Disallow: / User-agent: page scorer Disallow: / User-agent: PageAnalyzer Disallow: / User-agent: PageGrabber Disallow: / User-agent: PageScorer Disallow: / User-agent: PageThing Disallow: / User-agent: Pandalytics Disallow: / User-agent: Panscient Disallow: / User-agent: Papa Foto Disallow: / User-agent: Pavuk Disallow: / User-agent: pcBrowser Disallow: / User-agent: PeoplePal Disallow: / User-agent: Petalbot Disallow: / User-agent: PHPCrawl Disallow: / User-agent: Pi-Monster Disallow: / User-agent: Picscout Disallow: / User-agent: Picsearch Disallow: / User-agent: PictureFinder Disallow: / User-agent: Piepmatz Disallow: / User-agent: Pimonster Disallow: / User-agent: Pixray Disallow: / User-agent: PleaseCrawl Disallow: / User-agent: plumanalytics Disallow: / User-agent: Pockey Disallow: / User-agent: POE-Component-Client-HTTP Disallow: / User-agent: polaris version Disallow: / User-agent: probe-image-size Disallow: / User-agent: Probethenet Disallow: / User-agent: ProPowerBot Disallow: / User-agent: ProWebWalker Disallow: / User-agent: Proximic Disallow: / User-agent: Psbot Disallow: / User-agent: Pu_iN Disallow: / User-agent: Pump Disallow: / User-agent: PxBroker Disallow: / User-agent: PyCurl Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Quick-Crawler Disallow: / User-agent: Rainbot Disallow: / User-agent: RankActive Disallow: / User-agent: RankActiveLinkBot Disallow: / User-agent: RankFlex Disallow: / User-agent: RankingBot Disallow: / User-agent: RankingBot2 Disallow: / User-agent: Rankivabot Disallow: / User-agent: RankurBot Disallow: / User-agent: Re-re Disallow: / User-agent: RealDownload Disallow: / User-agent: Reaper Disallow: / User-agent: RebelMouse Disallow: / User-agent: Recorder Disallow: / User-agent: RedesScrapy Disallow: / User-agent: ReGet Disallow: / User-agent: RepoMonkey Disallow: / User-agent: Ripper Disallow: / User-agent: ripz Disallow: / User-agent: RocketCrawler Disallow: / User-agent: Rogerbot Disallow: / User-agent: RSSingBot Disallow: / User-agent: s1z Disallow: / User-agent: SalesIntelligent Disallow: / User-agent: satoristudio Disallow: / User-agent: SBIder Disallow: / User-agent: scalaj-http Disallow: / User-agent: scan Disallow: / User-agent: ScanAlert Disallow: / User-agent: Scanbot Disallow: / User-agent: ScoutJet Disallow: / User-agent: Scrapy Disallow: / User-agent: Screaming Disallow: / User-agent: ScreenerBot Disallow: / User-agent: ScrepyBot Disallow: / User-agent: Searchestate Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: Seekport Disallow: / User-agent: SeekportBot Disallow: / User-agent: SemanticJuice Disallow: / User-agent: Semrush Disallow: / User-agent: SemrushBot Disallow: / User-agent: SentiBot Disallow: / User-agent: SenutoBot Disallow: / User-agent: seobility Disallow: / User-agent: SeobilityBot Disallow: / User-agent: seocompany Disallow: / User-agent: SEOkicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: SEOlyticsCrawler Disallow: / User-agent: Seomoz Disallow: / User-agent: SEOprofiler Disallow: / User-agent: seoscanners Disallow: / User-agent: SeoSiteCheckup Disallow: / User-agent: seostar Disallow: / User-agent: SEOstats Disallow: / User-agent: serpstatbot Disallow: / User-agent: sexsearcher Disallow: / User-agent: Shodan Disallow: / User-agent: Siphon Disallow: / User-agent: SISTRIX Disallow: / User-agent: Site Sucker Disallow: / User-agent: Sitebeam Disallow: / User-agent: sitechecker Disallow: / User-agent: SiteCheckerBotCrawler Disallow: / User-agent: SiteExplorer Disallow: / User-agent: Siteimprove Disallow: / User-agent: SiteLockSpider Disallow: / User-agent: siteripz Disallow: / User-agent: SiteSnagger Disallow: / User-agent: SiteSucker Disallow: / User-agent: Sitevigil Disallow: / User-agent: SlySearch Disallow: / User-agent: SmartDownload Disallow: / User-agent: SMTBot Disallow: / User-agent: Snake Disallow: / User-agent: Snapbot Disallow: / User-agent: Snoopy Disallow: / User-agent: SocialRankIOBot Disallow: / User-agent: Sociscraper Disallow: / User-agent: Sogou web spider Disallow: / User-agent: sogouspider Disallow: / User-agent: Sosospider Disallow: / User-agent: Sottopop Disallow: / User-agent: sp_auditbot Disallow: / User-agent: SpaceBison Disallow: / User-agent: Spammen Disallow: / User-agent: SpankBot Disallow: / User-agent: Spanner Disallow: / User-agent: Spbot Disallow: / User-agent: Spinn3r Disallow: / User-agent: SputnikBot Disallow: / User-agent: spyfu Disallow: / User-agent: Sqlmap Disallow: / User-agent: Sqlworm Disallow: / User-agent: Sqworm Disallow: / User-agent: Steeler Disallow: / User-agent: Stripper Disallow: / User-agent: Sucker Disallow: / User-agent: Sucuri Disallow: / User-agent: SuperBot Disallow: / User-agent: SuperHTTP Disallow: / User-agent: Surfbot Disallow: / User-agent: SurveyBot Disallow: / User-agent: Suzuran Disallow: / User-agent: Swiftbot Disallow: / User-agent: sysscan Disallow: / User-agent: Szukacz Disallow: / User-agent: T0PHackTeam Disallow: / User-agent: T8Abot Disallow: / User-agent: tAkeOut Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: Telesoft Disallow: / User-agent: Telesphoreo Disallow: / User-agent: Telesphorep Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: Thumbor Disallow: / User-agent: TightTwatBot Disallow: / User-agent: TinyTestBot Disallow: / User-agent: Titan Disallow: / User-agent: Toata Disallow: / User-agent: Toweyabot Disallow: / User-agent: Tracemyfile Disallow: / User-agent: Trendiction Disallow: / User-agent: Trendictionbot Disallow: / User-agent: True_Robot Disallow: / User-agent: Turingos Disallow: / User-agent: Turnitin Disallow: / User-agent: TurnitinBot Disallow: / User-agent: TwengaBot Disallow: / User-agent: Twice Disallow: / User-agent: Typhoeus Disallow: / User-agent: ubermetrics-technologies Disallow: / User-agent: UnisterBot Disallow: / User-agent: Upflow Disallow: / User-agent: URLy Warning Disallow: / User-agent: URLy Disallow: / User-agent: V-BOT Disallow: / User-agent: Vacuum Disallow: / User-agent: Vagabondo Disallow: / User-agent: VB Project Disallow: / User-agent: VCI Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: VeriCiteCrawler Disallow: / User-agent: VidibleScraper Disallow: / User-agent: Virusdie Disallow: / User-agent: VoidEYE Disallow: / User-agent: Voil Disallow: / User-agent: Voltron Disallow: / User-agent: voyagerx Disallow: / User-agent: Wallpapers Disallow: / User-agent: Wallpapers/3 Disallow: / User-agent: WallpapersHD Disallow: / User-agent: WASALive-Bot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: Web Auto Disallow: / User-agent: Web Collage Disallow: / User-agent: Web Enhancer Disallow: / User-agent: Web Fetch Disallow: / User-agent: Web Fuck Disallow: / User-agent: Web Pix Disallow: / User-agent: Web Sauger Disallow: / User-agent: Web Sucker Disallow: / User-agent: Webalta Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebCollage Disallow: / User-agent: WebCopier Disallow: / User-agent: WEBDAV Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebFetch Disallow: / User-agent: WebFuck Disallow: / User-agent: webgains-bot Disallow: / User-agent: WebGo IS Disallow: / User-agent: WebImageCollector Disallow: / User-agent: WebLeacher Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmeup-crawler Disallow: / User-agent: WebPix Disallow: / User-agent: webpros Disallow: / User-agent: webprosbot Disallow: / User-agent: WebReaper Disallow: / User-agent: WebSauger Disallow: / User-agent: Webshag Disallow: / User-agent: Website Quester Disallow: / User-agent: WebsiteExtractor Disallow: / User-agent: WebsiteQuester Disallow: / User-agent: Webster Disallow: / User-agent: WebStripper Disallow: / User-agent: WebSucker Disallow: / User-agent: WebWhacker Disallow: / User-agent: WebZIP Disallow: / User-agent: WeSEE Disallow: / User-agent: Whack Disallow: / User-agent: Whacker Disallow: / User-agent: Whatweb Disallow: / User-agent: Who Disallow: / User-agent: Widow Disallow: / User-agent: WinHTTrack Disallow: / User-agent: WiseGuys Robot Disallow: / User-agent: WISENutbot Disallow: / User-agent: Wonderbot Disallow: / User-agent: Woobot Disallow: / User-agent: Wotbox Disallow: / User-agent: Wprecon Disallow: / User-agent: WPScan Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: WWW-Mechanize Disallow: / User-agent: WWWOFFLE Disallow: / User-agent: Xaldon WebSpider Disallow: / User-agent: Xaldon_WebSpider Disallow: / User-agent: Xenu Disallow: / User-agent: xpymep1 Disallow: / User-agent: YaK Disallow: / User-agent: YoudaoBot Disallow: / User-agent: Zade Disallow: / User-agent: Zauba Disallow: / User-agent: Zermelo Disallow: / User-agent: Zeus Disallow: / User-agent: zgrab Disallow: / User-agent: Zitebot Disallow: / User-agent: ZmEu Disallow: / User-agent: ZoominfoBot Disallow: / User-agent: ZumBot Disallow: / User-agent: ZyBorg Disallow: / User-agent: ZoomBot Disallow: / User-agent: WWW::Mechanize Disallow: / User-agent: GT::WWW Disallow: / User-agent: HTTP::Lite Disallow: / User-agent: LWP::Simple Disallow: / User-agent: PECL::HTTP Disallow: / User-agent: NoUserAgent Disallow: / User-agent: noxtrumbot Disallow: / User-agent: Nsauditor Disallow: / User-agent: Nuhk Disallow: / User-agent: Nuzzel Disallow: / User-agent: NuzzelFeedBot Disallow: / User-agent: NuzzelMediaBot Disallow: / User-agent: Nymesis Disallow: / User-agent: NymesisBot Disallow: / User-agent: Ocelli Disallow: / User-agent: OmahaBot Disallow: / User-agent: OmniExplorer Disallow: / User-agent: Onebot Disallow: / User-agent: OnetSzukaj Disallow: / User-agent: Onfolio Disallow: / User-agent: Openindexspider Disallow: / User-agent: OpenWebSpider Disallow: / User-agent: OxBot Disallow: / User-agent: PagePeeker Disallow: / User-agent: PageScrape Disallow: / User-agent: PagetScraper Disallow: / User-agent: Pcore-HTTPClient Disallow: / User-agent: Perl LWP Disallow: / User-agent: PersonaPilot Disallow: / User-agent: Phantom Disallow: / User-agent: PhantomJS Disallow: / User-agent: PHP::HTTPClient Disallow: / User-agent: Pingdom Disallow: / User-agent: PiplBot Disallow: / User-agent: Plukkie Disallow: / User-agent: Pogodak Disallow: / User-agent: ProCogBot Disallow: / User-agent: Python-httplib2 Disallow: / User-agent: Python-urllib Disallow: / User-agent: PyWebScraper Disallow: / User-agent: Qirina Hurdle Disallow: / User-agent: QResync Disallow: / User-agent: Qwantify Disallow: / User-agent: Rambler Disallow: / User-agent: RankSonicSiteAuditor Disallow: / User-agent: RankingHit Disallow: / User-agent: RealPlayer Disallow: / User-agent: Repo Disallow: / User-agent: RepoDev Disallow: / User-agent: RepoJob Disallow: / User-agent: Riddler Disallow: / User-agent: RivalIQ Disallow: / User-agent: RixBot Disallow: / User-agent: Robozilla Disallow: / User-agent: RoHe Disallow: / User-agent: Ruhm Disallow: / User-agent: SBL-Bot Disallow: / User-agent: Scrubby Disallow: / User-agent: scrutiny Disallow: / User-agent: ScrutinyBot Disallow: / User-agent: Seeker Disallow: / User-agent: semalt Disallow: / User-agent: SEO Suite Disallow: / User-agent: SEOENGBot Disallow: / User-agent: SEOlytics Disallow: / User-agent: SentiSumBot Disallow: / User-agent: seochat Disallow: / User-agent: SEOEngine Disallow: / User-agent: SEOMetriks Disallow: / User-agent: SerendeputyBot Disallow: / User-agent: Server Density Service Monitoring Bot Disallow: / User-agent: SeznamBot Disallow: / User-agent: SFR-Sharepoint Disallow: / User-agent: ShaMan Disallow: / User-agent: Shockwave Flash Disallow: / User-agent: ShopWiki Disallow: / User-agent: Siege Disallow: / User-agent: SimplePie Disallow: / User-agent: Sitebulb Disallow: / User-agent: Sixy Search Disallow: / User-agent: Sixy Search V2 Disallow: / User-agent: SkimBot Disallow: / User-agent: Skrybot Disallow: / User-agent: Skywalker Disallow: / User-agent: Slackerbot Disallow: / User-agent: SlackSocialBot Disallow: / User-agent: Slackbot Disallow: / User-agent: Slap Disallow: / User-agent: SliquaBot Disallow: / User-agent: Snoopie Disallow: / User-agent: Snufkinbot Disallow: / User-agent: Sogou Disallow: / User-agent: Sogou News Spider Disallow: / User-agent: Sogou Orion spider Disallow: / User-agent: Sogou Pic Spider Disallow: / User-agent: Sogou Spider Disallow: / User-agent: Sogou blog Disallow: / User-agent: Sogou inst spider Disallow: / User-agent: Sogou news Disallow: / User-agent: Sogou orion Disallow: / User-agent: Sogou web Disallow: / User-agent: Sogou-Pic-Spider Disallow: / User-agent: Sogou-Test-Spider Disallow: / User-agent: Sogou-Test Disallow: / User-agent: SoftHyde Disallow: / User-agent: SoMegaBot Disallow: / User-agent: Sonarcloud Disallow: / User-agent: SoNET Disallow: / User-agent: Sonic Disallow: / User-agent: SonicWall Disallow: / User-agent: Soso Spider Disallow: / User-agent: Spacy Disallow: / User-agent: SpamBot Disallow: / User-agent: SpamBayes Disallow: / User-agent: SpamMoose Disallow: / User-agent: SpamPatrol Disallow: / User-agent: Speedy Disallow: / User-agent: Speedy Spider Disallow: / User-agent: SpeedySpider Disallow: / User-agent: SpiderMan Disallow: / User-agent: Spider_Monkey Disallow: / User-agent: SpiderBot Disallow: / User-agent: SpiderLine Disallow: / User-agent: SpiderMonkey Disallow: / User-agent: SpiderProtector Disallow: / User-agent: SpiderSilk Disallow: / User-agent: sprinklr-spider Disallow: / User-agent: Spyder Disallow: / User-agent: SQLIHelper Disallow: / User-agent: Squider Disallow: / User-agent: SSIbot Disallow: / User-agent: SSLLabs Disallow: / User-agent: SSLTestBot Disallow: / User-agent: Startsearch Disallow: / User-agent: StartpageBot Disallow: / User-agent: StartSiteBot Disallow: / User-agent: Static Disallow: / User-agent: StellarBot Disallow: / User-agent: Strategistbot Disallow: / User-agent: Stratbot Disallow: / User-agent: stratosec-bot Disallow: / User-agent: StrokeBot Disallow: / User-agent: StudyBot Disallow: / User-agent: StudiotimeBot Disallow: / User-agent: SuggyBot Disallow: / User-agent: Supernova Disallow: / User-agent: Supersalonbot Disallow: / User-agent: Surebot Disallow: / User-agent: Surveillance Disallow: / User-agent: Suzie Disallow: / User-agent: SVM Disallow: / User-agent: SWEBot Disallow: / User-agent: SwiteBot Disallow: / User-agent: SXN Disallow: / User-agent: SyAgent Disallow: / User-agent: Syndex Disallow: / User-agent: Syndexbot Disallow: / User-agent: Synthesio Disallow: / User-agent: Sysomos Disallow: / User-agent: SzukajSe Disallow: / User-agent: Tactical Disallow: / User-agent: Tagasaurus Disallow: / User-agent: tAganalytics Disallow: / User-agent: Tailor4Less Disallow: / User-agent: Tailwind Disallow: / User-agent: Talkwalker Disallow: / User-agent: TalkwalkerBot Disallow: / User-agent: TalkwalkerRSSBot Disallow: / User-agent: TalkwalkerSmartBot Disallow: / User-agent: Tantalos Disallow: / User-agent: TaringaBot Disallow: / User-agent: Tarybot Disallow: / User-agent: Tazman-Agg Disallow: / User-agent: Tbot Disallow: / User-agent: Teamsoft Disallow: / User-agent: TeepsaBot Disallow: / User-agent: TelegramBot Disallow: / User-agent: Telekom-Bot Disallow: / User-agent: Teoma Disallow: / User-agent: TeraBot Disallow: / User-agent: Testbot Disallow: / User-agent: TextRazor Disallow: / User-agent: Thalesbot Disallow: / User-agent: The Answer Disallow: / User-agent: The Kraken Disallow: / User-agent: The Nomad Disallow: / User-agent: The-Schmidts Disallow: / User-agent: TheWorld Disallow: / User-agent: Thirdlightbot Disallow: / User-agent: Thunderstone Disallow: / User-agent: Tixati Disallow: / User-agent: Tobasco Disallow: / User-agent: Token Disallow: / User-agent: Toll Free Disallow: / User-agent: Tomato Bot Disallow: / User-agent: TomatoCat Disallow: / User-agent: Tonic Disallow: / User-agent: Topbot Disallow: / User-agent: Topwebsearch Disallow: / User-agent: Torbot Disallow: / User-agent: ToutiaoSpider Disallow: / User-agent: Tpbot Disallow: / User-agent: Trackback Disallow: / User-agent: Tracklock Disallow: / User-agent: Trahit Disallow: / User-agent: Tramp Disallow: / User-agent: Trams Disallow: / User-agent: Translate.ru bot Disallow: / User-agent: Translate.ru Disallow: / User-agent: Trax Disallow: / User-agent: Trendogic Disallow: / User-agent: Trendspottr Disallow: / User-agent: TridionBot Disallow: / User-agent: Triggitbot Disallow: / User-agent: Trilby Disallow: / User-agent: trufflehog Disallow: / User-agent: Truefinder Disallow: / User-agent: Truthybot Disallow: / User-agent: TUCOWS Disallow: / User-agent: Twingly Disallow: / User-agent: TwistedPageGetter Disallow: / User-agent: Twitterbot Disallow: / User-agent: U0xA-Bot Disallow: / User-agent: Ubermetrics Disallow: / User-agent: Ubrowse Disallow: / User-agent: uipbot Disallow: / User-agent: uLink Disallow: / User-agent: UnchaindSpider Disallow: / User-agent: UniteBot Disallow: / User-agent: UnshortenBot Disallow: / User-agent: Updownerbot Disallow: / User-agent: UpdatePatrol Disallow: / User-agent: UpsightBot Disallow: / User-agent: Urlfilebot Disallow: / User-agent: URLitor Disallow: / User-agent: URLMonitorBot Disallow: / User-agent: URLSpider Disallow: / User-agent: Urlwatch Disallow: / User-agent: User-agent Disallow: / User-agent: Validator Disallow: / User-agent: VBProjectBot Disallow: / User-agent: Vespa Disallow: / User-agent: VestrSpider Disallow: / User-agent: ViaMichelin Disallow: / User-agent: Vider Disallow: / User-agent: VideoSurf Disallow: / User-agent: ViewState Disallow: / User-agent: VirusTotal Disallow: / User-agent: vkShare Disallow: / User-agent: Vortex Disallow: / User-agent: voyager Disallow: / User-agent: VSBot Disallow: / User-agent: VSAgent Disallow: / User-agent: VSE Disallow: / User-agent: VYU2 Disallow: / User-agent: W3C-checklink Disallow: / User-agent: W3C_Validator Disallow: / User-agent: W3C_Unicorn Disallow: / User-agent: W3C_I18n-Checker Disallow: / User-agent: W3C_CSS_Validator Disallow: / User-agent: W3C-linkchecker Disallow: / User-agent: W3C Disallow: / User-agent: W3C-Unicorn Disallow: / User-agent: w3m Disallow: / User-agent: w3mir Disallow: / User-agent: wappalyzer Disallow: / User-agent: WAPMAI Disallow: / User-agent: watchmouse Disallow: / User-agent: Webaroo Disallow: / User-agent: WebArche Disallow: / User-agent: WebBot Disallow: / User-agent: Web Capture Disallow: / User-agent: Web Data Disallow: / User-agent: WebEx Disallow: / User-agent: WebImages Disallow: / User-agent: WebImageBot Disallow: / User-agent: WebLibrarian Disallow: / User-agent: WebMiner Disallow: / User-agent: WebMonitor Disallow: / User-agent: WebPatrol Disallow: / User-agent: WebRankBot Disallow: / User-agent: WebScrape Disallow: / User-agent: WebScraper Disallow: / User-agent: WebSearch Disallow: / User-agent: WebShot Disallow: / User-agent: WebSniffer Disallow: / User-agent: WebSphinx Disallow: / User-agent: WebVac Disallow: / User-agent: WebIndexer Disallow: / User-agent: Wepalabs Disallow: / User-agent: WebMon Disallow: / User-agent: Webinator Disallow: / User-agent: WebUser Disallow: / User-agent: WebVac_bot Disallow: / User-agent: Whale Disallow: / User-agent: WhatsApp Disallow: / User-agent: Whibse Disallow: / User-agent: Whiskers Disallow: / User-agent: Wicked Disallow: / User-agent: Wide Disallow: / User-agent: Willing Disallow: / User-agent: Windows-RSS-Platform Disallow: / User-agent: Wise Disallow: / User-agent: Wisebot Disallow: / User-agent: Wishy Disallow: / User-agent: Wixbot Disallow: / User-agent: WordBot Disallow: / User-agent: WordPress Disallow: / User-agent: Wscbot Disallow: / User-agent: Wsearch Disallow: / User-agent: WSGI Disallow: / User-agent: WWWBots Disallow: / User-agent: Wykop Disallow: / User-agent: Xaldon Disallow: / User-agent: Xerka Disallow: / User-agent: Xerox Disallow: / User-agent: XGET Disallow: / User-agent: XiangcBot Disallow: / User-agent: Xing Disallow: / User-agent: XingShares Disallow: / User-agent: Xing-Share Disallow: / User-agent: Xmeta Disallow: / User-agent: Xmith Disallow: / User-agent: XMLSitemapGenerator Disallow: / User-agent: Xml-RPC Disallow: / User-agent: XoviBot Disallow: / User-agent: Xpress Disallow: / User-agent: xStarter Disallow: / User-agent: Yahoo-Search Disallow: / User-agent: Yahoo-Slurp Disallow: / User-agent: YahooCacheSystem Disallow: / User-agent: Yahoo-Blogs Disallow: / User-agent: YahooYSMcm Disallow: / User-agent: Yanga Disallow: / User-agent: Yasaklibot Disallow: / User-agent: Yeti Disallow: / User-agent: Yioop Disallow: / User-agent: YisouSpider Disallow: / User-agent: YisouVideo Disallow: / User-agent: YisouImage Disallow: / User-agent: YisouBot Disallow: / User-agent: YisouSearchBot Disallow: / User-agent: YioopBot Disallow: / User-agent: YisouImageBot Disallow: / User-agent: YoudaoImageBot Disallow: / User-agent: YoudaoSearchBot Disallow: / User-agent: YoudaoSpider Disallow: / User-agent: YoudaoVideoSpider Disallow: / User-agent: Yottaa Disallow: / User-agent: Yottos Disallow: / User-agent: YQBot Disallow: / User-agent: YrBot Disallow: / User-agent: YSEBot Disallow: / User-agent: YSLBot Disallow: / User-agent: Yuilib Disallow: / User-agent: YunSeek Disallow: / User-agent: Yuntongzhi Disallow: / User-agent: Yunyun Disallow: / User-agent: Zabbix Disallow: / User-agent: Zao Disallow: / User-agent: Zebra Disallow: / User-agent: ZeBot Disallow: / User-agent: Zend2 Disallow: / User-agent: Zend_Http_Client Disallow: / User-agent: Zerx Disallow: / User-agent: Zibber Disallow: / User-agent: Ziibra Disallow: / User-agent: Zimmerman Disallow: / User-agent: Zing Disallow: / User-agent: Ziot Disallow: / User-agent: ZipBot Disallow: / User-agent: Zipsa Disallow: / User-agent: Zoinks Disallow: / User-agent: Zombie.js Disallow: / User-agent: ZooplaBot Disallow: / User-agent: Zotero Disallow: / User-agent: ZoomInfo Disallow: / User-agent: Zune Disallow: / User-agent: Zurpha Disallow: / User-agent: Zuupa Disallow: / User-agent: ZyngaBot Disallow: / User-agent: ZyXel Disallow: /
Сырой список ботов
На основе этого списка построены правила выше.
01h4x 360Spider 404checker 404enemy 80legs Abonti Aboundex Aboundexbot Acunetix ADmantX adscanner AdsTxtCrawlerTP AfD-Verbotsverfahren AhrefsBot AIBOT AiHitBot Aipbot Alexibot ALittle Client Alligator AllSubmitter AlphaBot Anarchie Anarchy Anarchy99 Ankit Anthill anthropic-ai Apexoo archive arquivo-web-crawler arquivo Aspiegel ASPSeek Asterias Atomseobot Attach autoemailspider awario AwarioBot AwarioRssBot AwarioSmartBot BackDoorBot Backlink-Ceck backlink-check BacklinkCrawler BackStreet BackWeb Badass Bandit Barkrowler BatchFTP Battleztar Bazinga BBBike BDCbot BDFetch BetaBot Bigfoot Bitacle Black Hole Blackboard BlackWidow BLEXBot Blow BlowFish Boardreader Bolt BotALot Brandprotect Brandwatch Buck Buddy BuiltBotTough BuiltWith Bullseye BunnySlippers BuzzSumo Bytespider cah Calculon CATExplorador CazoodleBot CCBot Cegbfeieh CensysInspect ChatGPT-User check1 CheeseBot CherryPicker CheTeam ChinaClaw Chlooe Citoid Claritybot clark-crawler Cliqzbot Cloud mapping coccocbot Cocolyzebot CODE87 Cogentbot cognitiveseo cohere-ai Collector com Copier CopyRightCheck Copyscape Cosmos Craftbot crawl crawler crawler4j Crawling at Home Project CrazyWebCrawler Crescent CrunchBot CSHttp Curious Custo CyotekWebCopy DatabaseDriverMysqli DataCha0s dataforseo dataforseobot DBLBot demandbase-bot Demon Deusu Devil Digincore DigitalPebble DIIbot Dirbuster Disco Discobot Discoverybot Dispatch DittoSpyder DnBCrawler-Analytics DnyzBot DomainAppender DomainCrawler Domains Project DomainSigmaCrawler domainsproject DomainStatsBot DomCopBot Dotbot Download Wonder Dragonfly Drip DSearch DTS Agent EasyDL Ebingbong eCatch ECCP/1 Ecxi EirGrabber EMail Siphon EMail Wolf EroCrawler evc-batch Evil Exabot Express WebPictures ExtLinksBot Extractor ExtractorPro Extreme Picture Finder EyeNetIE Ezooms FacebookBot facebookscraper FDM FemtosearchBot FHscan Fimap FlashGet Flunky Foobot Freeuploader FrontPage Fuzz FyberSpider Fyrebot G-i-g-a-b-o-t GalaxyBot Genieo GermCrawler Getintent GetRight GetWeb Gigabot Go-Ahead-Got-It Go!Zilla gopher Gotit GoZilla GPTBot Grabber GrabNet Grafula GrapeFX GrapeshotCrawler GridBot Haansoft HaosouSpider Harvest Havij HEADMasterSEO Heritrix Hloader HMView HonoluluBot HTMLparser HTTrack Humanlinks HybridBot Iblog Id-search IDBot IDBTE4M IlseBot Image Fetch Image Sucker imagesift ImagesiftBot IndeedBot Indy Library InfoNaviRobot Information Security Team InfraSec Scanner InfoTekies InfraSec Scanner instabid Intelliseek InterGET Internet Ninja InternetMeasurement InternetSeer internetVista monitor ips-agent Iria IRLbot isitwp Iskanie IstellaBot iubenda-radar JamesBOT Jbrofuzz JennyBot JetCar Jetty JikeSpider JOC Web Spider Joomla Jorgee JustView Jyxobot Kenjin Spider Keybot Translation-Search-Machine Keyword Density Kinza Kozmosbot Lanshanbot Larbin Leap LeechFTP LeechGet LexiBot Lftp LibWeb Libwhisker LieBaoFast Lightspeedsystems Likse Linkbot linkdexbot LinkextractorPro linkfluence LinkpadBot LinkScan LinksManager LinkWalker LinqiaMetadataDownloaderBot LinqiaRSSBot LinqiaScrapeBot Lipperhey Lipperhey Spider Litemage_walker Lmspider LNSpiderguy Ltx71 lwp-request lwp-trivial Mag-Net Magnet magpie-crawler Majestic SEO Majestic-SEO Majestic12 MarkMonitor MarkWatch Mass Downloader Masscan Mata Hari MauiBot Mb2345Browser MeanPath Bot Meanpathbot Mediatoolkitbot mediawords MegaIndex Metauri MFC_Tear_Sample MicroMessenger MIDown tool MIIxpc Minefield Mister PiX MJ12bot Mojeek Mojolicious MolokaiBot Morfeus Fucking Scanner Mozlila Mr MSFrontPage MSIECrawler Msrabot MTRobot muhstik-scan Musobot Name Intelligence Nameprotect Navroad NearSite Needle Nessus Net Vampire NetAnts Netcraft netEstate NE Crawler NetLyzer NetMechanic NetSpider Nettrack Netvibes NetZIP NextGenSearchBot Nibbler NICErsPRO Niki-bot Nikto NimbleCrawler Nimbostratus Ninja Nmap NPbot Nuclei Nutch oBot Octopus Offline Explorer Offline Navigator omgili OnCrawl openai Openfind OpenLinkProfiler OpenVAS OrangeBot OrangeSpider OutclicksBot OutfoxBot Page Analyzer page scorer PageAnalyzer PageGrabber PageScorer PageThing Pandalytics Panscient Papa Foto Pavuk pcBrowser PeoplePal Petalbot PHPCrawl Pi-Monster Picscout Picsearch PictureFinder Piepmatz Pimonster Pixray PleaseCrawl plumanalytics Pockey POE-Component-Client-HTTP polaris version probe-image-size Probethenet ProPowerBot ProWebWalker Proximic Psbot Pu_iN Pump PxBroker PyCurl QueryN Metasearch Quick-Crawler Rainbot RankActive RankActiveLinkBot RankFlex RankingBot RankingBot2 Rankivabot RankurBot Re-re RealDownload Reaper RebelMouse Recorder RedesScrapy ReGet RepoMonkey Ripper ripz RocketCrawler Rogerbot RSSingBot s1z SalesIntelligent satoristudio SBIder scalaj-http scan ScanAlert Scanbot ScoutJet Scrapy Screaming ScreenerBot ScrepyBot Searchestate SearchmetricsBot Seekport SeekportBot SemanticJuice Semrush SemrushBot SentiBot SenutoBot seobility SeobilityBot seocompany SEOkicks SEOkicks-Robot SEOlyticsCrawler Seomoz SEOprofiler seoscanners SeoSiteCheckup seostar SEOstats serpstatbot sexsearcher Shodan Siphon SISTRIX Site Sucker Sitebeam sitechecker SiteCheckerBotCrawler SiteExplorer Siteimprove SiteLockSpider siteripz SiteSnagger SiteSucker Sitevigil SlySearch SmartDownload SMTBot Snake Snapbot Snoopy SocialRankIOBot Sociscraper Sogou web spider sogouspider Sosospider Sottopop sp_auditbot SpaceBison Spammen SpankBot Spanner Spbot Spinn3r SputnikBot spyfu Sqlmap Sqlworm Sqworm Steeler Stripper Sucker Sucuri SuperBot SuperHTTP Surfbot SurveyBot Suzuran Swiftbot sysscan Szukacz T0PHackTeam T8Abot tAkeOut Teleport TeleportPro Telesoft Telesphoreo Telesphorep The Intraformant TheNomad Thumbor TightTwatBot TinyTestBot Titan Toata Toweyabot Tracemyfile Trendiction Trendictionbot True_Robot Turingos Turnitin TurnitinBot TwengaBot Twice Typhoeus ubermetrics-technologies UnisterBot Upflow URLy Warning URLy V-BOT Vacuum Vagabondo VB Project VCI VelenPublicWebCrawler VeriCiteCrawler VidibleScraper Virusdie VoidEYE Voil Voltron voyagerx Wallpapers Wallpapers/3 WallpapersHD WASALive-Bot WBSearchBot Web Auto Web Collage Web Enhancer Web Fetch Web Fuck Web Pix Web Sauger Web Sucker Webalta WebAuto WebBandit WebCollage WebCopier WEBDAV WebEnhancer WebFetch WebFuck webgains-bot WebGo IS WebImageCollector WebLeacher WebmasterWorldForumBot webmeup-crawler WebPix webpros webprosbot WebReaper WebSauger Webshag Website Quester WebsiteExtractor WebsiteQuester Webster WebStripper WebSucker WebWhacker WebZIP WeSEE Whack Whacker Whatweb Who Widow WinHTTrack WiseGuys Robot WISENutbot Wonderbot Woobot Wotbox Wprecon WPScan WWW-Collector-E WWW-Mechanize WWWOFFLE Xaldon WebSpider Xaldon_WebSpider Xenu xpymep1 YaK YoudaoBot Zade Zauba Zermelo Zeus zgrab Zitebot ZmEu ZoominfoBot ZumBot ZyBorg ZoomBot WWW::Mechanize GT::WWW HTTP::Lite LWP::Simple PECL::HTTP NoUserAgent noxtrumbot Nsauditor Nuhk Nuzzel NuzzelFeedBot NuzzelMediaBot Nymesis NymesisBot Ocelli OmahaBot OmniExplorer Onebot OnetSzukaj Onfolio Openindexspider OpenWebSpider OxBot PagePeeker PageScrape PagetScraper Pcore-HTTPClient Perl LWP PersonaPilot Phantom PhantomJS PHP::HTTPClient Pingdom PiplBot Plukkie Pogodak ProCogBot Python-httplib2 Python-urllib PyWebScraper Qirina Hurdle QResync Qwantify Rambler RankSonicSiteAuditor RankingHit RealPlayer Repo RepoDev RepoJob Riddler RivalIQ RixBot Robozilla RoHe Ruhm SBL-Bot Scrubby scrutiny ScrutinyBot Seeker semalt SEO Suite SEOENGBot SEOlytics SentiSumBot seochat SEOEngine SEOMetriks SerendeputyBot Server Density Service Monitoring Bot SeznamBot SFR-Sharepoint ShaMan Shockwave Flash ShopWiki Siege SimplePie Sitebulb Sixy Search Sixy Search V2 SkimBot Skrybot Skywalker Slackerbot SlackSocialBot Slackbot Slap SliquaBot Snoopie Snufkinbot Sogou Sogou News Spider Sogou Orion spider Sogou Pic Spider Sogou Spider Sogou blog Sogou inst spider Sogou news Sogou orion Sogou web Sogou-Pic-Spider Sogou-Test-Spider Sogou-Test SoftHyde SoMegaBot Sonarcloud SoNET Sonic SonicWall Soso Spider Spacy SpamBot SpamBayes SpamMoose SpamPatrol Speedy Speedy Spider SpeedySpider SpiderMan Spider_Monkey SpiderBot SpiderLine SpiderMonkey SpiderProtector SpiderSilk sprinklr-spider Spyder SQLIHelper Squider SSIbot SSLLabs SSLTestBot Startsearch StartpageBot StartSiteBot Static StellarBot Strategistbot Stratbot stratosec-bot StrokeBot StudyBot StudiotimeBot SuggyBot Supernova Supersalonbot Surebot Surveillance Suzie SVM SWEBot SwiteBot SXN SyAgent Syndex Syndexbot Synthesio Sysomos SzukajSe Tactical Tagasaurus tAganalytics Tailor4Less Tailwind Talkwalker TalkwalkerBot TalkwalkerRSSBot TalkwalkerSmartBot Tantalos TaringaBot Tarybot Tazman-Agg Tbot Teamsoft TeepsaBot TelegramBot Telekom-Bot Teoma TeraBot Testbot TextRazor Thalesbot The Answer The Kraken The Nomad The-Schmidts TheWorld Thirdlightbot Thunderstone Tixati Tobasco Token Toll Free Tomato Bot TomatoCat Tonic Topbot Topwebsearch Torbot ToutiaoSpider Tpbot Trackback Tracklock Trahit Tramp Trams Translate.ru bot Translate.ru Trax Trendogic Trendspottr TridionBot Triggitbot Trilby trufflehog Truefinder Truthybot TUCOWS Twingly TwistedPageGetter Twitterbot U0xA-Bot Ubermetrics Ubrowse uipbot uLink UnchaindSpider UniteBot UnshortenBot Updownerbot UpdatePatrol UpsightBot Urlfilebot URLitor URLMonitorBot URLSpider Urlwatch User-Agent Validator VBProjectBot Vespa VestrSpider ViaMichelin Vider VideoSurf ViewState VirusTotal vkShare Vortex voyager VSBot VSAgent VSE VYU2 W3C-checklink W3C_Validator W3C_Unicorn W3C_I18n-Checker W3C_CSS_Validator W3C-linkchecker W3C W3C-Unicorn w3m w3mir wappalyzer WAPMAI watchmouse Webaroo WebArche WebBot Web Capture Web Data WebEx WebImages WebImageBot WebLibrarian WebMiner WebMonitor WebPatrol WebRankBot WebScrape WebScraper WebSearch WebShot WebSniffer WebSphinx WebVac WebIndexer Wepalabs WebMon Webinator WebUser WebVac_bot Whale WhatsApp Whibse Whiskers Wicked Wide Willing Windows-RSS-Platform Wise Wisebot Wishy Wixbot WordBot WordPress Wscbot Wsearch WSGI WWWBots Wykop Xaldon Xerka Xerox XGET XiangcBot Xing XingShares Xing-Share Xmeta Xmith XMLSitemapGenerator Xml-RPC XoviBot Xpress xStarter Yahoo-Search Yahoo-Slurp YahooCacheSystem Yahoo-Blogs YahooYSMcm Yanga Yasaklibot Yeti Yioop YisouSpider YisouVideo YisouImage YisouBot YisouSearchBot YioopBot YisouImageBot YoudaoImageBot YoudaoSearchBot YoudaoSpider YoudaoVideoSpider Yottaa Yottos YQBot YrBot YSEBot YSLBot Yuilib YunSeek Yuntongzhi Yunyun Zabbix Zao Zebra ZeBot Zend2 Zend_Http_Client Zerx Zibber Ziibra Zimmerman Zing Ziot ZipBot Zipsa Zoinks Zombie.js ZooplaBot Zotero ZoomInfo Zune Zurpha Zuupa ZyngaBot ZyXel