September 2nd, 2006 by metapilot

With Google steady at the 9000 to 12000 (old-site, supplimental) pages showing for the site:metapilot.com search and up until today showing only two pages from my site (at the very bottom of the all the other supplimental pages), and Yahoo up the page count of the new site by page or two a week, I getting figitty.Google had shown one of the blog pages in that search, but today that is gone and only the home page, indexed Aug. 21, is showing. I’m not at all sure how google found that blog page since it wasn’t until at least a week later that I put any sort of public link to the blog. The only thing I can think of is that in messing around during the installation, and working on making a static home page (coming) I had an index.php page in the root along with the index.html page and somehow that got picked up? Thing is it wasn’t even the index.php page that was indexed, it was an archive page. Anyhow, It’s gone at least for the moment.

Since I am getting figitty, I re-crawled the site with my trusty dusty sitemap tool (I use the one over at auditmypc.com) so that my sitemaps included all the pages currently in the blog. I have two sitemaps, urllist.txt and sitemap.txt because urllist.txt used to be the only filename that Yahoo look when you submitted a sitemap as a feed–that was before Yahoo Site Explorer, which doesn’t clearly define a specific file name. Sitemap.txt is the file name Google suggests if you are using a text file as the sitemap you submit to Google Webmaster Tools and rather than telling Google to look for a file created spefically for Yahoo Sitemapts, I make one with that name too. For the time being it is faster and easier to have these two sitemaps than to figure out the ideal way to have a single one. (Did you get all that?)

So the new sitemaps were submitted to Google and Yahoo yesterday and the domain submitted to MSN, as well. I’m hoping this jogs some changes in the index in the next week or so. Of course, I won’t be able to be sure that this is what caused them but at least, it’s gives the felling of doing something to help push the process along.

September 1st, 2006 by metapilot

Following Google’s lead, Yahoo has came out with their own version of a sitemaps tool and rolled into the the Yahoo Site Explorer Beta tool. There is a lot of debate over the value of Google’s sitemap tool and Yahoo doesn’t really make any advances over it. One nice thing about it, though, is that you can now easily get the last-crawled date for any page that you have listed in “My Sites” (you have to have a free yahoo acount in order to access My Sites information).
In order to set up a new site in Yahoo Site Explorer, enter the site’s URL and click “Add My Site”, afterwhich you’re presented with links to Manage and Authenticate your new site. The coolest thing about the tool is that you can communicate to Yahoo that you want it to visit your site and it will go there and grab your feed in “real time”. Your feed can be RSS, Atom, a txt file or a compressed text file (.gz only) and by real time, I mean that you might have to wait a few minutes for it to refresh your screen and show you that it’s verified your feed exists.

The feed is the conduit through/by which you are telling Yahoo about URLs you want it to crawl. Google Sitemaps adds an additional conduit, or feed choice– a .xml file with which you can make your list of URLs dynamic by running a python script on your web server and I expect to see something of that nature coming from Yahoo in the near future. Very basically, though, all you need for Yahoo Site Explorer is a .txt file with a list all the URLs on your site that you wnat crawled (one URL per line) named urllist.txt.. Upload it to your root directory and after you click on “Manage Site” in Yahoo Site Explorer, type “urllist.txt” into the field and off goes the bot to check it out.
Before you can get to the “good” information about your site, you have to authenticate your site. This lets Yahoo know that you currently have access to the site’s rood directory, which means you’re likely to worthy of knowing the any little insights Yahoo Site Explorer might provide you. Whey you click on the “Authenticate” link, you can choose to download an authentication file (which you can save directly to your root directory, if you want) or make your own authentication file with the file name and contents presented. Once the file is placed in your root directory, click “Authenticate” and your site gets put into a pending authenication que until Yahoo crawls the feed. Within 24 hours, I could see that my status was no longer “Pending” but rather, I was now a “processed” site.

On to the real business, Yahoo has racheted up the number of indexed pages to 12. It’s good to see things filling in there. Over on the marginally more useful Google Webmaster Tools, I can see that the index is ranking the old site for some odd keywords, however zero traffic comes from any of his old link partners or search engine listings–at least not from anyone directly clicking on a link.

August 26th, 2006 by metapilot

As I noted back on Aug 17, all my site:domain searches required that I be logged into my Yahoo account. Since then I noticed, in passing, that I seemed to be getting link:domain searches back in that same interface. Now I realize that you do have to be logged in and you do have to use the Yahoo Site Explorer for the link: searches as well.It doesn’t make that much difference to me which I use, although in some of the forums, people have remarked that the site:domain search gives you different results depending on which interface is used and he the Yahoo Site Explorer is, to some degree, deficient, compared to getting results through the standard search interface.

That’s not been an issue for me but what seems to be an issue is some my research tools the make use of Yahoo link information have been eratic and in some cases not working at all. I can’t help but figure that it has something to do with changes in structuring the queries for the new interface. Some of my tool vendors seem to have come up with a fix already but some have not.

August 20th, 2006 by metapilot

At least I can now say that the site is not banned in Yahoo. Yahoo site explorer shows 12 pages still in the .net index and it has been stuck on showing only two of the new .com pages for several days, now. It is also showing one of the previous owner’s .pdf files in the index (although not in the cache).At least there are two new pages now showing in Yahoo, though. For a couple of days, only the home page was showing as indexed and current wisdom states that when just your home page shows in the Yahoo index, it is a sure sign that the site has been banned. Of course, being very aware of the possibility that my site will suffer from the legacy of the previous domain owner’s “poor neighborhood” practices and the resulting penalties placed on the domain, I was holding my breath while wating for more pages to show up. When another page did show up along with the home page in the Yahoo Site Explorer, I could finally let out a little sigh of relief. Now, only in Google is there an indication that an all-out ban is still possible.
Google is not cooperating at all. Though the site was submitted two weeks ago, there is no aknowledgemnt from Google that a site exists at that domian. Granted, two weeks is no time at all but I was hoping for a best case scenario–an older siter (even though it was purchased as an expired domain), with a lot of back links (even though they are from irrelevant, bad neighborhood sites), with fully indexed pages from another site being 301?ed to same page name pages on this site–it’s a good scenario–it’s a very good scenario, except for the litte issue of the domain having been BANNED.

Over the past six months to a year, or so, however, Google has greatly increased it’s grip on expired domains. For “domainer”- types, who have tended to rely on continuity of domain name aging (regardless of whether the registration had expired and ownership had changed hands) to help their site’s ranking ability, this turn of events hasn’t been particularly advantageous. For me, however, it means I can have a greater expection of being able to seperate my site and my owership of this domain from those that came before me. It is a double-edged sword, though. For it also means that I’ll most likely be subject to the euphamistically named “aging delay”, aka, the “sand box penalty”. Bleecchk, I hate that thing but it is better than being banned and not going through it.

MSN, on the other hand, currently lists 27 pages for the .net site, including the metapilot.net/index.html page. Eleven of the cached pages point to the “Permanently Moved” cached pages. The rest still point to standard caches of the originals. On the site:domain search for the .com site, it shows 29 pages in in the index, including the metapilot.com/index.html page(4 of them are from the previous owner’s site) .

Interestingly, when you click on the “Cached page” link in any of those 4 results from the previous owner, you get an empty page–not an actual true version of that previous page.

August 19th, 2006 by metapilot

Yahoo starts to show .com results and as it does, it starts removing pages from its “site:metapilot.net” results. Currently, just the index page from the .com domain is showing up. Notice I was (unintentionally) doing the search without using “www”.

Note: This was the last date that I was able to use the site:domain command and get standard Yahoo results. For the past week, Yahoo has fluctuated between serving me standard site:domain results and forcing me to be logged in and serving me results via the new Site Explorer (Beta) interface. Hence, all site:domain querries are redirected to the Yahoo Site Explorer interface. This is something that has been talked about in the forums over the past week or two and seems to be the case for a growing number of searchers who use the site:domain query.