Robots

[GWD]

Can anyone tell me do I need a robot.txt file and if so, why? I've heard that it just blocks search engines which is no use to me.

[-=Seth=-]

use it to tell search engines which pages to include and which to exclude, if you dont have one search engines just go ahead and list what they like, however having one is advisable just for completeness even if its just to say include everything

[jonra01]

The search bots request the robots.txt file everytime they visit your site. If you don't have one it generates a 404 error. If you also don't have a 404 page then it generates another error. These all add to the server load. It also fills up your error log with unimportant errors making it harder to find real problems you might have on your site. It's a good idea to have a robots.txt file even if it is a blank file. Just stick a plain-text file named robots.txt in the root directory of your site.

[rotarysteve]

I use a blank/empty robots file on my site. Since I have access to my server logs, it reduced the 404 errors for me. Reducing those errors, made life much easier to find the really important 404's.

It's a good idea to have one I believe, then down the road if you do want to add something, you still can.

[GWD]

Thanks. I've uploaded my blank robots.txt file. Is this basically the same as telling the engines to view everything?

[JWJ]

To inform all robots that they are permitted to view all of your pages put these two lines in your file.

User-agent: *
Disallow:

An empty file achieves the same thing because it doesn't tell them they can't.

[Nomad]

Originally Posted by JWJ

To inform all robots that they are permitted to view all of your pages put these two lines in your file.

User-agent: *
Disallow:

An empty file achieves the same thing because it doesn't tell them they can't.

em jw? by using that you have just disallowed all files...

you need to be specific about what your disallowing and allowing otherwise leave it blank...

[JWJ]

No ... this disallows all pages ...

User-agent: *
Disallow: /

If there is nothing at all after the disallow it means 'disallow nothing'.

[webado]

Originally Posted by JWJ

To inform all robots that they are permitted to view all of your pages put these two lines in your file.

User-agent: *
Disallow:

An empty file achieves the same thing because it doesn't tell them they can't.

That's the default in any case. If no robots.txt file is available or if it's empty, it defaults to this directive.


It's good to have it though so as not to lose track of it.