%PDF- %PDF-
Direktori : /home1/dimen328/.trash/espace/verification/snd/cet/ |
Current File : //home1/dimen328/.trash/espace/verification/snd/cet/robots.txt |
# RODONDO robots.txt User-agent: * Disallow: Disallow: / Disallow: // User-agent: googlebot User-agent: Googlebot User-agent: Mediapartners-Google User-agent: Adsbot-Google User-agent: Googlebot-Image User-agent: Googlebot-Mobile User-agent: Googlebot-Video User-agent: bingbot User-agent: Go-http-client User-agent: Go-http-client/1.1 User-agent: http-banner-detection User-agent: zgrab User-agent: Baiduspider User-agent: Baiduspider/2.0 User-agent: Baiduspider-mobile User-agent: Baiduspider-video User-agent: Baiduspider-image User-agent: NaverBot User-agent: Yeti User-agent: Yandex User-agent: YandexBot User-agent: YandexMobileBot User-agent: YandexVideo User-agent: YandexWebmaster User-agent: YandexSitelinks User-agent: SeznamBot User-agent: Twitterbot User-agent: Adidxbot User-agent: Yahoo Pipes 1.0 User-agent: Facebot User-agent: externalfacebookhit User-agent: facebookexternalhit User-agent: python-requests User-agent: python User-agent: Python/3.7 User-agent: aiohttp/3.6.0 User-agent: aiohttp User-agent: htdig/3.1.6 User-agent: Teleport Pro User-agent: yacybot User-agent: Browsershots User-agent: Slurp User-agent: Exabot User-agent: Fast corporate crawler User-agent: HTTrack User-agent: psnsearch User-agent: MagpieRSS User-agent: magpie-crawler/1.1 User-agent: magpie-crawler User-agent: 008 User-agent: HaoSouSpider User-agent: Sogou web spider User-agent: Sogou inst spider User-agent: Sogou spider2 User-agent: HCGOV-PRODTWO-gsa-crawler User-agent: HCGOV-PRODONE-gsa-crawler User-agent: CensysInspect/1.1 User-agent: CensysInspect User-agent: Censys User-agent: masscan/1.0 User-agent: masscan User-agent: crawler User-agent: slurp User-agent: DuckDuckBot User-agent: ia_archiver User-agent: webtech User-agent: webtech/1.2.8 Disallow: /secure Disallow: /Sbou Disallow: / Disallow: /* Disallow: // Crawl-delay: 99999