Webmaster forum - robots.txt

Author

robots.txt

DBLEXPOSURE

2005-07-29, 7:25 pm

I want to keep the bots out of a subdirectory called dealer

Disallow: /store2/dealer/

But I do want the store2 directory to be spidered.

I think I know the answer but to be on the safe side, will the above line
acomplish what I want? My fear is the store2 directory will disapear form
google and others..

Thank you all...

GreyWyvern

2005-07-29, 7:25 pm

And lo, DBLEXPOSURE didst speak in alt.www.webmaster:

> I want to keep the bots out of a subdirectory called dealer
>
> Disallow: /store2/dealer/
>
> But I do want the store2 directory to be spidered.

Spidered:
/
/store
/store2
/store2/
/store2/deal
/store2/dealer

Not spidered:
/store2/dealer/
/store2/dealer/index.html

This only applies to well-behaved bots, however. If you *really* don't
want /store2/dealer/ in any search engines, password protect it or don't
put it online.

Grey

--
The technical axiom that nothing is impossible sinisterly implies the
pitfall corollory that nothing is ridiculous.
- http://www.greywyvern.com - Orca Knowledgebase: Completely CSS styleable
Knowledgebase/FAQ system

DBLEXPOSURE

2005-07-29, 7:25 pm

"GreyWyvern" <spam@greywyvern.com> wrote in message
news:op.suoybllpsl6xfd@news.nas.net...
> And lo, DBLEXPOSURE didst speak in alt.www.webmaster:
>
>
> Spidered:
> /
> /store
> /store2
> /store2/
> /store2/deal
> /store2/dealer
>
> Not spidered:
> /store2/dealer/
> /store2/dealer/index.html
>
> This only applies to well-behaved bots, however. If you *really* don't
> want /store2/dealer/ in any search engines, password protect it or don't
> put it online.
>
> Grey
>
> --
> The technical axiom that nothing is impossible sinisterly implies the
> pitfall corollory that nothing is ridiculous.
> - http://www.greywyvern.com - Orca Knowledgebase: Completely CSS styleable
> Knowledgebase/FAQ system

Thank Grey!

So, I am to understand that the way I have it written, "Disallow:
/store2/dealer/" Is okay?

It's not the end of the world if some do find the dealer sub dir, just
anoying. My main concern is Googlebot and keeping him interested in the
store2 dir. and out of the store2/dealer/ sub. was not a problem until
google started using sitemaps. Now googlebot has found my dealer sub dir
and it shows up in some results. Like I said not the end of the world but I
would like to clean it up..

Thanks again

PZ

GreyWyvern

2005-07-29, 7:25 pm

And lo, DBLEXPOSURE didst speak in alt.www.webmaster:

> So, I am to understand that the way I have it written, "Disallow:
> /store2/dealer/" Is okay?

Correct. If you main concern is Google, they have a de-listing service, I
believe:
More info here: <http://www.google.com/intl/en/webmasters/remove.html>

Grey

--
The technical axiom that nothing is impossible sinisterly implies the
pitfall corollory that nothing is ridiculous.
- http://www.greywyvern.com - Orca Knowledgebase: Completely CSS styleable
Knowledgebase/FAQ system

Ken

2005-07-29, 11:22 pm

Hi -

On Fri, 29 Jul 2005 16:10:36 -0500, "DBLEXPOSURE"
<celstuff@hotmail.com> wrote:

>So, I am to understand that the way I have it written, "Disallow:
>/store2/dealer/" Is okay?
>
>It's not the end of the world if some do find the dealer sub dir, just
>anoying.

You don't have to use the full directory name. For example, if you
don't have any pages or subdirectories in store2 that start with d,
other than the dealer subdirectory, you could use:

Disallow: /store2/d

Then if someone looks at your robots.txt file, they will have no clue
that there is a dealer subdirectory.

You just have to make the string long enough that you don't disallow
anything you aren't wanting to.

--
Ken
http://www.ke9nr.net/