| David B 2005-03-21, 6:40 pm |
| I was reading up on robots.txt, which sounds like a really simple way to
prevent certain files from being indexed by search engines. Can anyone
tell me if I'm doing this correctly?
I would open up Notepad and save a file with the name "robots" into my
website, www.mysite.org. If I type the following...
User-agent: *
Disallow: /includes/
Disallow: /usa/ads/
This would exclude all search engines from www.mysite.org/includes and
www.mysite.org/usa/ads/
However, www.mysite.org/usa/other_folders/ would still be indexed. And
if I include a file from www.mysite.org/usa/ads/ on a page at
www.mysite.org/usa/news/today/, the search engines would index
www.mysite.org/usa/news/today/, including the included text.
Am I right on all counts? Thanks.
--
http://www.geobop.org/ (Geobop) | http://www.geoworld.org/ (GeoWorld) |
http://www.geozoo.org (GeoZoo) |
http://www.geosymbols.org (GeoSymbols) | http://www.politix.us/ (Kids'
Politix) |
http://geowebworks.geobop.org/ (Ethical Web Design) |
http://www.freedomware.us/ (Open Source Activism)
|