Robots.txt is a plain text file that search engines use to see if there are areas within your website which should not be indexed.
This file must be placed in your /www directory in order for a search engine to see it.
The sample below tells search engines not to index pages in the specified folders.
robots.txt:
This file must be placed in your /www directory in order for a search engine to see it.
The sample below tells search engines not to index pages in the specified folders.
robots.txt:
1 | User-agent: * |
2 | Disallow: /cgi-bin/ |
3 | Disallow: /images |
4 | Disallow: /secret |
5 | Disallow: /members |
