User-agent: Twitterbot User-agent: Bingbot User-agent: Slurp User-agent: DuckDuckBot User-agent: Baiduspider User-agent: YandexBot User-agent: facebot User-agent: ia_archiver User-agent: facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php) User-agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) User-agent: Googlebot User-agent: Googlebot-Image User-agent: MSNBot Allow: / User-agent: * Disallow: /