# Robots.txt file from http://www.24x.com/robots.txt # All robots will spider the domain User-Agent: * Disallow: