Comments on: Does site:domain.com/* show you non-supplemental Google indexed pages?

By: Jennifer Hayden

Jennifer Hayden — Fri, 17 Sep 2010 13:18:20 +0000

Hey I found some fast pagerank tools, may be seo friends would like them.

seo41.com/pagerank-checker.php
seo41.com/internal-pagerank.php

one is bulk page rank checker that can check 1000 of pages PR with single click and another is internal PR checker to check pagerank of all internal pages on a website.

There is lot more helpful things for search engine optimization.

By: Fionnd

Fionnd — Thu, 24 Jun 2010 16:28:15 +0000

Would this explain why sites rank very well in ROW but the exact same site is nowhere to be seen in the US. I have a client with several sites ranking very well in UK and indee anywhere else in the world that I check but not in the US?

By: Halfdeck

Halfdeck — Sat, 09 Jan 2010 10:49:38 +0000

As Andy said, /* and regular site: query should theoretically return the same number but they don’t. Sebastian uncovered another query using date filters a while ago – might wanna look that one up too. John Mu once said regarding the normal site: query that it was only an approximation and suggested Webmaster Tools as a better alternative for monitoring indexation, so I’d also interpret /* numbers as no more than ballpark figures.

Another way to check index penetration is cache date of course, but those numbers are also influenced by things like number of page crawls per day so you have inaccuracies there too.

Average pages crawled per day is also a reflection of domain strength since crawling depth is mainly controlled by PageRank.

By: Andy Beard

Andy Beard — Fri, 08 Jan 2010 13:55:36 +0000

This would also show only subdomains

site:bbc.co.uk/ -www

Numbers are different though, as no pattern matching
http://www.google.com/support/customsearch/bin/answer.py?hl=en&answer=71826

The numbers using * are slightly different to another method I am aware of, but the ratios are typically very similar.

Conventional wisdom is that site:domain.com and site:domain.com/* should be the same but they are not

Do you notice a similarity to this other method?

http://search.aol.co.uk/aol/search?s_it=topsearchbox.search&query=site%3Aandybeard.eu&rp=
http://www.google.co.uk/search?q=site:andybeard.eu/*&pws=0&gl=UK

Bigger sites the disparity is often much greater (60%) but with the BBC it isn’t actualy that far off
http://www.google.co.uk/search?q=site%3Abbc.co.uk%2F*&pws=0&gl=UK
http://search.aol.co.uk/aol/search?query=site%3Abbc.co.uk

By: Sanjay

Sanjay — Fri, 08 Jan 2010 13:43:00 +0000

I don’t see any major changes after using both queries as per your blog explanation. I feel that Google’s algo uses a lot of factors, and not just some calculation like “we can only give this query this many spots”.

By: Tad Chef

Tad Chef — Fri, 08 Jan 2010 11:41:55 +0000

I’m afraid this syntax is just a buggy regular expression and does NOT show the non-supplemental results.
Compare it with this one here: “/*.”
It shows all pages on a site after the slash that have one word file name strings before that first dot.

It hides subdomains, directories, even homepages lacking a filename like “index.php” and even complex filenames consisting of more than one word combined with a minus.

So you’ll see

example.com/filename.php

but won’t

subdomain.example.com/filename.php
example.com
example.com/directory/
example.com/file-name.php

etc.

When you take away the dot you’ll see more pages as the regular expression is less strict but still it’s just a regular expression.

Also compare this query: site:*.bbc.co.uk/ -www
where you only see the BBC subdomains.

I hope that helps.

By: Dave

Dave — Fri, 08 Jan 2010 09:19:05 +0000

Interesting find, I’ve not noticed that before. I’ve checked against a small (lowish authority) site I have that ranks fairly well for some core terms, it’s a site I use quite often for testing. The site only has 80 pages on the site:domain.com search, but has just 14 for site:domain.com/*.

Checking the stats I can see that in the last week, Google has sent visits to 58 landing pages (all organic traffic). Don’t forget that that’s just a one off example, but that implies that Google will still rank and be willing to send traffic to pages even if they don’t appear in your site:domain.com/* search.

In summary and conclusion though, I still have no idea what the /* is showing…

By: Phill Midwinter

Phill Midwinter — Fri, 08 Jan 2010 09:03:00 +0000

We’re fairly sure that the Santa Clara data centre is now also running Caffeine. The index difference people are seeing seems to match the server that Google earlier confirmed as Caffeine.

By: Malte Landwehr

Malte Landwehr — Fri, 08 Jan 2010 08:42:57 +0000

Something else: When I search for “site:example.com/*” I see the “Similar” link on pages, that don’t show a “Similar” link when I search “site:example.com/“. I’m talking about the exact same URLs and tested it multiple times with two domains. It might be those to searches are served by two different DCs but why? There is definitely something going on with the *-parameter in site:-search.

By: Fred

Fred — Fri, 08 Jan 2010 08:41:14 +0000

it looks like that site:domain.com/* displays the first (important) results of site:domain.com/ . So I don’t see big similarities with supplemental index

By: Kevin Gibbons

Kevin Gibbons — Fri, 08 Jan 2010 08:27:23 +0000

Thanks for the comment Andy, interesting stuff – strange that there’s a difference between US and ROW.

And what you said about PageRank makes sense, obviously direct external links to content will help significantly here too (as opposed to using PageRank strength filtered down to get as many pages as possible indexed).

By: Andy Beard

Andy Beard — Fri, 08 Jan 2010 00:32:44 +0000

There is currently a difference in reported indexing size between US and ROW (rest of world), or at least seems to be.
For instance this might, depending on data center result in around 8M indexed pages for the BBC
http://www.google.com/search?q=site:bbc.co.uk&pws=0&gl=US

/* is one method, there are other queries that are similar which can be more or less consistant.
/* returns the same for both US and ROW currently

At least one of the alternatives returns similar though lower numbers with some variance based on geographical factors.

In many ways it is a chicken/egg situation with indexation

Pagerank helps get pages in the index, but you need authority/trust to get pages in the primary index.
More pages in the primary index is a good indication of higher trust, but does not directly imply significantly more search traffic
It is like the difference between 1 in 10 and 11 in 100 or 101 in 1000 – more Pages in the index = more PageRank but not necessarily more trust/authority.