The robots.txt file allows you to specify what parts of your website taht spiders can access.  You can also specify which spiders are allowed.  Here is a concise, well written article with great ideas and example robots.txt code.