# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file User-agent: * Disallow: /admins/ Disallow: /auth/ Disallow: /payments/ Disallow: /cdn-cgi/ Disallow: /remote-companies/*/posting_successful User-agent: barkrowler Disallow: / Sitemap: https://rubyonremote.com/sitemap.xml