SEO Blog - Internet Marketing News

Wouldn’t it be great if you could convince your competitors that they had to spend big in order to keep up with you? If they use SpyFu to estimate your ad spend and CPC then you can. I’ve looked at what SpyFu say about their own algorithm and techniques and then I’ve combined this knowledge with what I observed last week to come up with these ideas about how to turn SpyFu into a double agent.

Firstly let’s look at how SpyFu calculate their Ad Spend and CPC estimates. Then I’ll show you how to use this knowledge to increase SpyFu’s estimates of your Ad Spend and CPC without spending any more money.

How Does SpyFu Estimate Daily Ad Spend?
The onsite help, also available in The SpyFu Instruction Manual says this about calculating Daily Ad Spend:

When we calculate Daily Ad Budget, we start with all the keywords that we have seen a domain advertise on. We eliminate overlapping keywords. For example, “race cars”, “luxury cars”, and “cars” becomes “cars”. Then, we take into account the current and historical positions that we have seen the domain’s ad appear for each given keyword. Based on the position of each ad, we estimate the price that the domain likely pays for the keyword. Basically, we then add up all the custom individual keyword costs per day and we arrive at the Daily Ad Budget.

Let’s add in what we already know about SpyFu to our new knowledge from the above passage:

• SpyFu uses its web scraping database to get a list of all the keywords a site advertises on.
• Overlapping keywords are eliminated.
• A daily cost is estimated for each keyword given its ad position.
• The total can then be easily found.

I’ll look at this step by step:

Web Scraping: Reading the Internet
Velocityscape (who own SpyFu) have a massive web scraped database. If your ad appears on one of the first few SERPs for a term then they will know about it and have a record of it, unless the term is relatively unusual. This is why estimates are better for accounts with a large number of impressions; such accounts either bid on lots of relatively common terms that will be scraped by SpyFu or they bid on a few very general terms which are also covered. Accounts that bid on lots of low traffic, long tail keywords might not get picked up at this stage so SpyFu will not have very good data on them.

Ignoring the Long Tail
The example that SpyFu gives for eliminating overlapping keywords is clearly far too restrictive; I doubt they would lump “race cars” and “luxury cars” into “cars” but you rarely see three word keywords whilst using SpyFu and, personally, I’ve never seen a four word one. Accounts which bid mainly on long tail keywords will have lower impressions and a lower daily ad spend but SpyFu does not take this into account, ignoring the long tail and lumping everything into a more general query.

Counting the Cost
The accuracy of the daily cost estimate for each keyword is a tricky one. If they work it out by considering an average CPC for each keyword multiplied by an estimated number of clicks then their accuracy depends not only on their CPC estimates (see below) but also their estimates of CTR’s. Since all this is done by a computer CTR estimates will depend only on ad position and not on the ad copy which the folks at the Mind Valley Labs tell us can make a massive difference.

What About CPC?
I’ve just said that the Daily Ad Spend estimate depends on the accuracy of the CPC estimate. So if you can manipulate what SpyFu thinks about your CPC then you can manipulate their Daily Ad Spend estimate as well. So how does SpyFu estimate CPC?

If you take the Average Cost per Click of every keyword that a domain advertises on, add them all up, and divide by the total number of keywords, you will have the Avg Cost/Click for a domain. For example, if a domain advertises on 3 keywords with Avg Cost/Clicks of $1, $2, and $3, respectively then the Avg Cost per Click for the domain would be $2.

This is clearly a load of rubbish. The following table shows that SpyFu’s method only gives the correct answer if there is the same number of clicks for each keyword:

Now look at the table if each keyword gets a different number of clicks:

In this case the average CPC is 21/9=2.33 not 2 as SpyFu’s method tells us.

At $70/month for a subscription I hope that SpyFu doesn’t really estimate CPC in this way.

Assuming the SpyFu algorithm does actually estimate CPC in this way then their estimate will be most accurate when the standard deviation in the number of clicks for each keyword in an account is small. As we have seen the method gives the right answer in the case where all keywords receive the same number of clicks, it also wouldn’t be far off if the number of clicks was dominated by a few keywords.

Estimates for Individual Keywords
Of course, all this depends on the accuracy of their estimates of average CPC for each keyword but to say any keyword has an average CPC ignores one of the most fundamental parts of AdWords. No one can accurately predict the CPC for an ad without knowing how the quality score system works. I’m sure SpyFu has the technology and capacity to scrape landing pages as well as search results; they can then apply their own quality score algorithm but I can’t imagine their results are very accurate.

The AdWords API has a service that will estimate CPC for a given keyword but I doubt SpyFu use it: The API service is designed to be accurate when it can access campaign and ad group quality scores for the account it is being used for; this information will not be available for SpyFu.

Get on With It!
Enough waffle. Now we’ve looked at how SpyFu’s estimates are calculated lets look at what you can do to change them, without actually changing you account of course. None of these methods have been tested and some of them require information that I don’t know, but if what SpyFu says about itself is true then at least the first two of these methods should work.

Bid High
SpyFu do not know what your click through rates are. They calculate daily spend on an estimated daily spend for a given keyword which is probably based on an average CTR depending on the ad position. So if you use a high traffic keyword with and bid enough that your ad gets a good position then SpyFu think you are spending a lot of money. But what if your ad text was so bland that the real CTR was incredibly low? Then you’d be spending a lot less than SpyFu thought.

Bid on the Long Tail
Bid on the long tail of a really expensive keyword. SpyFu even published a list of the most expensive keywords to make this easier for you. The idea here is to use the way SpyFu lumps long tail searches together in order to convince them that your CPC is huge. For example top of their top list is “conference calling companies” at $52 a pop. As I’ve said, I’ve never seen a four word keyword in SpyFu so try bidding on something like “free conference calling companies.” The term “free” is common enough that SpyFu will probably scrape for it so their algorithm should record you bidding on “conference calling companies” but your actual CPC will be lower because people will bid less for the word “free.”

Of course the algorithm might decide to cut your keyword down to “free conference calling” instead so you should bid on a variety of “x conference calling companies” to make sure your long tail is shortened to the keywords with the maximum CPC.

Bid at the Right Time
SpyFu’s database is not updated very frequently. Most things I’ve read say it’s only redone every 60-90 days. Spend a lot when SpyFu does its update and you’ll appear to be spending that much for the next three months. Take these update figures with a pinch of salt; I can’t find anything official about this and quite a few blog posts about SpyFu appear quite biased against it.

When is SpyFu’s next database update due? I don’t know. It is also possible that their data gathering goes on all the time but they only do their calculations every three months. If this is the case then this method won’t work. Unfortunately I think this probably is the case.

Bid on the Right Place
SpyFu’s owners, Velocityscape are based in Phoenix, Arizona. Assuming their scraping is all done from there then a high spend ad group targeted at their location will be scraped and analysed by them as if it covers the whole web. Unfortunately things are unlikely to be that simple.

I doubt the whole of Velocityscape’s scraping operations are centralised; SpyFu UK would indicate that they have at least one other location. This is not a big barrier, campaigns can be targeted at more than one location but if they scrape using something like the AdWords Ad Preview Tool this method won’t work at all. Does anyone know anything about this?

Control the Double Agent
Always remember that SpyFu works both ways; while you’re using it to spy on your competitors, they are using it to spy on you. But also remember that you have complete control over what information SpyFu can gather about you and, to a certain extent, you can manipulate that information.

It would be an interesting challenge to see who can get the highest estimated daily ad spend for the lowest actual budget. Let the games begin!