Search Engine question

[JWJ]

I have two web sites ... an old unused one and a new one. They are:-
old = http://jwj.members.beeb.net
new = http://members.aol.com/JWJarts

For a variety of perfectly good reasons (or so it seemed at the time), I switched isp from Beeb to AOL about 12 months ago. I left my old web site in place at Beeb to avoid creating dead links, and copied my site to AOL where I have continued it's development. Google quickly spotted the duplication and indexed my AOL site dropping the Beeb one. Yahoo, MSN and a few others have still not found my new site and send traffic to the old one.

Some weeks ago I tackled this problem by adding a .htaccess and robots.txt file at the old site, and all traffic nows comes to the new one. However, most search engines are still finding the old account despite my efforts to develop and raise the profile of the new site.

Today I used some Search Engine tools (kindly promoted by DSL Guy in another thread) and was horrified to discover that no search engines seem to be aware that my new site even exists, yet most of them give a reasonable result for the old dead site.

These results cause me to wonder why my new site isn't being indexed, and why my old one is despite the .htaccess and robots.txt files. I am wondering whether I would be better off moving my newly developed site back to the old isp to take advantage of its existing visibility, or whether I should remove that old site completely regardless of any dead-linking caused.

Any thoughts would be most welcome.

[Nomad]

Just put a re-direct at the old site..that way you will get the traffic from there at your new site...

Nomad

[JWJ]

I have ... that's what the .htaccess is doing.

As I said, all of the traffic to my old site IS being sent to the new one ... but I still don't understand why the search engines are pointing at the old site. The point is, my newest site content will not be found by the engines while they keep ignoring the new site.

[webado]

I have ... that's what the .htaccess is doing.

As I said, all of the traffic to my old site IS being sent to the new one ... but I still don't understand why the search engines are pointing at the old site. The point is, my newest site content will not be found by the engines while they keep ignoring the new site.

I think there are 2 ways of specifying redirection in the .htacces file. One is temporary and another one is permanent. I don't have the specifics handy now. You'll need the permanent redirection.

Otherwise, all the search engines have your old site cached. Not much you can do about it.


----LATER -----

The .htacess file placed in the root folder of the "dead" web site should contain:


redirect permanent / http://www.new-site.com/


Put in the actual new site's url.

[DSL Guy]

Hi again JWJ,

Just a hypothetical- :idea:

If you discontinue redirecting traffic from the old site, and can somehow produce dead links from that site causing no action with that site as far as the crawlers can tell, then the search engines will begin to drop it. Does that make sense :?: Some engines and directories will allow removal of your old site using their prescribed method if they have one, at your request. You'll have to look for it in their policies. I believe that's the only other way you can do it.

Or you can do what I recently did :D - Let me explain: a while back I had a so-called all purpose url site to promo my broadband quote engine. However, I decided I wanted to approach my marketing in a more specific way creating more of a "web like site presence" and so I created a network of specific service urls for 4 of the broadband services I offer. That's when I decided to deactivate hosting for the all purpose url, or so I thought it was at the time. I even went as far as deleting it from my statcounter projects.

A couple days ago I was doing some keyword verifcations of one of the new urls and surprise :shock: up came the old one. I was shocked because I requested to stop hosting for that url and was told it would not be hosted. That was over a month ago. Somehow no one flipped the switch and it was getting larger than some of my other new urls so I decided to re-activate it. I figured it may be better to not waste all that hard work I had put into it and just weaved it back into the group.

What do you think?

[JWJ]

Chris, because of the changed structure of my site I was advised to redirect page to page. So this is what's in my .htaccess file redirect permanent /index.htm http:// members.aol.com/***arts/index.htm
redirect permanent /mary.htm http:// members.aol.com/***arts/Flagpin.htm
redirect permanent /Comparisons.htm http:// members.aol.com/***arts/Poems.htm
redirect permanent /links.htm http:// members.aol.com/***arts/links.htm
redirect permanent /Watercolours.htm http:// members.aol.com/***arts/Watercolours.htm
redirect permanent /scottish.htm http:// members.aol.com/***arts/scottish.htm
redirect permanent /Results.htm http:// members.aol.com/***arts/Results.htm
redirect permanent /Northampton.htm http:// members.aol.com/***arts/Northampton.htm
redirect permanent /Gallery02.htm http:// members.aol.com/***arts/Gallery02.htm
redirect permanent /Gallery01.htm http:// members.aol.com/***arts/Gallery01.htm
redirect permanent /Explanation.htm http:// members.aol.com/***arts/Explanation.htm
redirect permanent /England.htm http:// members.aol.com/JWJarts/England.htm
redirect permanent /Design.htm http:// members.aol.com/JWJarts/Design.htm
redirect permanent /AOL.htm http:// members.aol.com/JWJarts/AOL.htm
The robots file contains User-agent: *
Disallow:

They are working fine at redirecting visitors, it's just the search engines that seem to be ignoring them.

DSL, I had considered trying to do a 'remove url' from some of the key engines, or even simply deleting all the pages in my old site, but it's not entirely statisfactory.

However, if I am understanding you correctly, I may have had an idea last night that fits with what you have described to me this morning. It occurred to me that if these search engines persist in indexing my old site, then all I have to do is upload my new content to the old location. The .htaccess file will still redirect visitors to my new site, the search engines will be indexing my new content, and I would also be able to simplify the .htaccess file to a single statement.

The only other idea I have had is to split my site content and have an Art site and a Personal site, with lots of links between them so they could feed each other. This would avoid duplication of content and dead links, but would be harder to administer. The general feeling here was that that was not a very good idea.

In view of your comments, I think I'll upload the new content to the old site. As I see it, I have nothing to lose in trying it, and hopefully in time that old site will fall out of the frame.

[webado]

John, I think your old site is not getting reindexed, what must be seen by queries are cached results based on the old site, in addition to new ones.
The redirects have to divert traffic from the old site, whether page by page or for the entire site. Therefore there is no traffic to the old site, only to the cached images.

Do you have access to the server logs of the old site? Take a look and see what pages are being visited. None of the redirected ones should appear. If they do, there is a problem with the .htaccess file.

However I just visited your old site and got redirected to the new one, so the redirect works. I'd think it might be better to have the global redirect as well, using just the slash. The regardless of what page is requestes, whether it exists or not, traffic would go to the new site looking for that page.

[JWJ]

Thanks Chris,

I tried the global statement but I immediatley ran into trouble because of the case-dependant files at the new site, which I have been actively changing .... trying to be 'organised' but actually digging myself a deeper hole :?

Maybe I'll try placing a global redirect to my new index page at the end of the htaccess file to act as a catch-all. See if that works.

I have no access to server logs, but Statcounter has not moved since I put the redirect in place so I guess it's working.

[webado]

Thanks Chris,

I tried the global statement but I immediatley ran into trouble because of the case-dependant files at the new site, which I have been actively changing .... trying to be 'organised' but actually digging myself a deeper hole :?

Maybe I'll try placing a global redirect to my new index page at the end of the htaccess file to act as a catch-all. See if that works.

I have no access to server logs, but Statcounter has not moved since I put the redirect in place so I guess it's working.
Aha! I think there's a way to control the case (i.e.a let both upper and lower to default to same) with .htaccess at the new site. Not very celan as it allows one to get sloppy. Alternately, try renaming everything to what it used to be respecting the case, and change the corresponding links throughout.

[JWJ]

Nope ... thought of that :(

AOL (my new site) does not allow .htaccess files. Any control on redirects will have to be done at the old site.

I may well take your advice of changing all of my file names back to what they were in order to match with the old site. A biggish job, but it'll only need doing once.

[DSL Guy]

Just a question -

What or who could change all of my Site meta tags? I discovered this after checking my search engine saturation and seeing my site drop out of site.

Before this I think I had some great keyword saturation going in my tags since I had over a few hundred pages indexed in various engines. I was coming up within the first couple of pages in Google and Yahoo and MSN and some other lower ranked engines. Now those numbers have dropped to 1 page per engine in most cases.

This confuses me :? . Maybe my site admin got hacked into and someone did this. Maybe a competitor did this. I can't begin to imagine who else or what else could have happened. Now that it did though, any comments on what I can do?

[trackerm]

I did that .hta redirect thingo. However its no use putting it on my website as it needed to be on my ISP's root server (is thems the right words?). They told me they didn't support .hta files and the did a redirect for me.

I am having the same problem that Yahoo etc have taken ages to update to the new one and discard the old one. The redirect bacame effective on 10 August 2004 but they only cut the old site out last week. The new site is listed at number 12, where the old site was at number 1.


Google was, of course much quicker and left my lsiting at No1.
No wonder they are making huge bucks...

[webado]

I did that .hta redirect thingo. However its no use putting it on my website as it needed to be on my ISP's root server (is thems the right words?). They told me they didn't support .hta files and the did a redirect for me.

I am having the same problem that Yahoo etc have taken ages to update to the new one and discard the old one. The redirect bacame effective on 10 August 2004 but they only cut the old site out last week. The new site is listed at number 12, where the old site was at number 1.


Google was, of course much quicker and left my lsiting at No1.
No wonder they are making huge bucks...

Hmmm, this sounds a bit weird that they shouldn't support .htaccess at your level but do so higher up. My guess is it's really that they don't want to solve people's problems if they should ever get into trouble because of an incorrect .htaccess file. It is possible to mess up all forms of acccess.

Use an FTP program and take a good look at all the files appearing in your site's root folder. Make sure you specify -a so you can see hidden files as well. Ugh, I don't know whare you have to spcify it in all FTP programs, but in ws_ftp tehre's an inncuous little white box, with no label at all on the server panel. This is where you put that -a. Then do a refresh. You may be surprized to see there's already an .htaccess file sitting there. You may want to download it (in ascii mode) and look at it, because the changes you make will have to be incorporated into the existing .htaccess file contents. If there's one, rename it to some safe name like my-htaccess.txt . Then upload your new .htaccess file (make sure all is correct!) using ascii transfer mode. If that doesn't work, rename your .htaccess file to htaccess.txt and upload it. Then rename it on the server to .htaccess.

If none of this works then yes, they may be doing something funny there. I'd be quite miffed at that myself.


Actually to really feel safe, create a new folder on your site and try uploading an .htaccess file there and see how that goes. Tread very carefully with that file. Test the behaviour of that folder with the .htaccess file it has. If it's good, then you may consider uploading that file to the root folder.

[trackerm]

Hmmmmm Christina,

You may be right (yet again!)

I found this spicey little bit on the net... maybe its why I couldn't do it and my ISP had to.

IF YOU ARE NOT USING FRONTPAGE, you can set this up as a Mime Type in an .htaccess file. Open the .htaccess file in the domain-www directory and add this line, or create a file named .htaccess and insert just this line:

AddType text/x-server-parsed-html .html.

If you are using FrontPage, this is not available. Using an .htaccess file will mess up FrontPage.



I use FrontPage Express. But I am trying to learn... I downloaded HTML Kit and Ist Page 2000. My brain fried. I went to a bar and looked into the bottom of a glass.

[webado]

LOL! Even back in the days I was trying to use Frontpage I never uploaded directly to the web. Just used it on the pc and then uploaded using a separate FTP program.

Now I have no idea why an .htaccess file would mess up Frontpage unless Frontpage needs an .htaccess file with specific settings. Got to look into it a bit more closely since I admit my education is a bit (ok, a lot) lacking in this area.

Get your FTP program and snoop around your web site and see whether there is such a file there and half the msytery will be solved.

[JWJ]

Sorry to bring back an old thread, but the problem continues.

I have followed all the advice given so far, although I am still working my way through the meta tags making sure they are as good as I can get them, but it is still very much as if search engines are not finding anything other than my main index page. Googling for anything on my main page gets me a fairly good ranking ... Googling for anything else gets zilch.

What I want to ask now is this. I cannot see any url's in my html source, and therefore I can't see how a search engine would be able to follow the links. I am using a Frontpage Navigation bar to link my pages, but I only ftp the pages individually. In other words, I do not publish the entire site using FP. It seems to me, if I am understanding it correctly, that each url is 'constructed' by string manipulation using the parameters in the line beginning <webbot bot=. If so, then I think I have finally figured out why the search engines are passing me by.

I would be very grateful for any explanation of the webbot tag, and any suggestions as to how I ought to do my navigation so as to give the SE's links they can follow.

Thanks in anticipation.

[webado]

Well, even though your top menu does not offer outright links, you do have links scattered in the text so those should be followed. ESpecially since you do have the meta tag for ROBOTS to follow.

Of course, you could always modify that top menu to be just a straightforward set of links.

Maybe a little more patience?

----EDIT ----
Actually I think your top menu is ok anyway. I didn't look at it properly. It's true though I'm not familiar with the webbot thing. So to Google I go :)

[webado]

Hmm, Google says webbot it something that goes hand in hand with FrontPage. Not much detail I can chew on. The way I understand it by using that you let loose (or invite) this particular bot by specifying it there. Still not clear :?

Apparently webbot is isntrumental on submitting to some search engines. Need to read up more. :?

[JWJ]

Not a lack of patience, a lack of knowledge. I thought I'd bumped into a possible poor coding issue that might be detrimental to search engines. Thank you for explaining that's it's fine as it is. I shall leave it alone.
___________________________________

Ooops ... we crossed in the post, so to speak.

I have also been googling and have found out something about webbot bot "Include". I wonder if I need that as well. Need to read more. Thanks.

[webado]

Not a lack of patience, a lack of knowledge. I thought I'd bumped into a possible poor coding issue that might be detrimental to search engines. Thank you for explaining that's it's fine as it is. I shall leave it alone.

Well, would you consider taking out the webbot specification? It should leave the menu alone as it is. Maybe other bots are shy to follow if they are not webbot. It's just possible from the look of that construct that it is designed to be totally for the webbot ..... or not :?

[JWJ]

A shy bot ... there's no such thing :D

From what I've found out, webbot is not a crawler/spider but something peculiar to FP. I'll try removing it from one of my pages later and see what happens. I suspect the menu won't work, but we'll found out, eh?

[webado]

A shy bot ... there's no such thing :D

From what I've found out, webbot is not a crawler/spider but something peculiar to FP. I'll try removing it from one of my pages later and see what happens. I suspect the menu won't work, but we'll found out, eh?

You'll find otu, yes.

But you could put in the menu just like this (perhaps fix the format if it's tied in to webbot somehow):


<p class="webnorm" align="center" style="margin-top: 0; margin-bottom: 0">
<font color="#990033" face="Verdana" size="1"> <nobr>Home</nobr>|<nobr>Watercolours (watercolours.htm)</nobr>|<nobr>Drawings (pencil.htm)</nobr>|<nobr>Northampton (northampton.htm)</nobr>|<nobr>Poems+Stories (poems.htm)</nobr>|<nobr>Humour (humour.htm)</nobr>|<nobr>Miscellany (miscellany.htm)</nobr>|<nobr>Site Map (SiteMap.htm)</nobr></font></p>

[JWJ]

Wow, Thanks Chris.

I've had a play with FP and found a way I can have the complete url showing in the source. I've uploaded it. Which do you think I should stick with ... the code that's there now or yours?

[webado]

Wow, Thanks Chris.

I've had a play with FP and found a way I can have the complete url showing in the source. I've uploaded it. Which do you think I should stick with ... the code that's there now or yours?
Actually it',s not a good idea to use the entire url when it refers to pages on the same site, in the same path. This slows down access as it has to go and resolve all over again the domain to an ip address and re-access the site. It's not wrong, just not efficient.

What I gave you simply removed the webbot thing so it's open to all and sundry to go follow the links if they so feel.

[JWJ]

OK .... Thanks :D

[paulzon]

Just put a re-direct at the old site..that way you will get the traffic from there at your new site...

Nomad
As you are doing this for the benefit of Search Engines, make sure that it is a permanent "301" style redirect, rather than a temporary "302" style one. It is important, and should correct your problem. Google's advice for webmasters, as well as lots of other places can give you further guidance on this.
Ah! you have! (sorry Christina!) well so long as your old host supports this, then you shouldn't have any problems apart from waiting for a month or so, whilst it all works through.

regards from Paul www.zonicweb.net