SEO Blog - Internet Marketing News

As shown in my last blog post, now that Google are using expanded broad match to trigger ads from “travel ppc” for a search query on “shooting holidays USA”, getting a comprehensive list of negative keywords is a good idea. In this post I list five good resources for finding negative keywords; some of them are not intended to be used in this way but they still give useful information about possible negative keywords.

1. SEO Book Forums
Most campaigns will want a generic list of negative keywords. Things like “free” or “reviews” are good examples of negative keywords for any sort of campaign that sells a service or product online. The SEO Book has a huge list of negative keywords on their SEO Community Forums. You need a subscription to access the forum; I don’t think it’s worth getting one just for this but if you have one already then make use of this resource which is a great negative keyword starting point.

2. Google Keyword Tool
This is an obvious one but it’s still worth mentioning. When you’re using the tool to look for keyword ideas and you see something that isn’t relevant to your ad group then select “negative” from the drop down menu and prevent your ads from showing.  You can also use any other keyword tool in a similar way.

3. Google Analytics
Once you’ve implemented the Google Analytics filters I talked about last week you’ll have a list of the search queries people used when they clicked your ads. Not only is this a great for finding new keyword variants it’s also a good source of possible negative keywords.

4. SpyFu
Get a SpyFu report on your own domain name. SpyFu works by doings its own Google searches and scraping the results. If it says you’re bidding on “Price searches” then an ad with your domain as a display URL is showing an ad for that term. You can also have fun trying to guess what broad match terms your competitors are using based on their paid keywords.

5. The Google SERPs
Everyone knows that when you do a Google search the search query appears in bold whenever it is written on the results page. What people may not have noticed is that Google also emboldens (is that even a word?) related terms that it thinks are semantically linked to the search query. For example if you search “SEO” then “search engine optimisation” also appears in bold. If you see something in bold that isn’t relevant then add it as a negative.

6. Google Search Based Keyword Tool
As you’ve probably read yesterday, the new Google Search Based Keyword Tool is designed to help you spot missed opportunities in your AdWords campaign. Like most keyword tools it can also be used to find good negative keywords. This one is worth mentioning in its own point because it also gives you suggested landing pages for each new keyword. If one of your existing PPC landing pages is appearing as a suggestion for a lot of negative keywords then this suggests that the page should be more tightly optimised to prevent Google matching it with other PPC search terms.

Another useful thing to remember is to keep thinking in terms of your ad groups when adding negative keywords; if you have an ad group for “red cars” and an ad group for “blue cars” you should add “blue” as a negative in the “red cars” ad group. Otherwise expanded broad match might decide that since your “red cars” ad group has such a great quality score it might display that ad on the query “blue cars” even if you have [blue cars] as an exact match in your other ad group.

There is also a useful summary of a discussion on how negative keywords are matched over at seroundtable which is definately worth a look if you’re adding negative key phrases rather than just keywords.

It can be very difficult at times to find the actual search terms your PPC traffic arrives from, so this is a Google Analytics trick all advertisers should know. Google’s search query report can be useful but for high-traffic phrase or broad match keywords being told that 8 of your clicks arrived on “85 unique queries” doesn’t really give you the complete picture!

Since the introduction of expanded broad match Google can (and does) match your broad match keywords to just about anything vaguely relevant; knowing these queries is important, either to negative match them or to reduce CPCs by using an exact match. The image below really does highlight this point, notice the extremely irrelevant term “shooting holidays USA” was triggered by a broad match of travel PPC!

This report was setup last week and shows the AdWords keywords (either exact, phrase or broad match) followed by the actual search term which triggered the clickthrough in brackets:

(Click for full-size image)

Step by step guide on how to setup a Search Query report in Google Analytics
This information can easily be found in Google Analytics but, although the method is simple, it is not obvious; to be able to access this PPC goldmine you have to use filters. Until last week I didn’t even know the filters feature existed and even if I had I wouldn’t have been able to do the regular expressions stuff that our filters will need. For this reason I’d like to thank the Google Analytics Experts and the linklove blog for giving me some simple step by step instructions.

• In the above case we’ve set up a new profile before messing around, just to ensure that if a mistake was made none of the data is affected. There’s an “Add a Website Profile” option on the Analytics settings page; you want to add a profile for an existing site and then name it.
• Then you want to write the two filters; click the “Filter Manager” button and then add a filter.
• This first filter will get the search query and place it in a user defined field. I call it “Get Search Query” but you can name it whatever you want to. Select “Custom Filter” from the filter type drop down menu and select the “Advanced” radio button. You should see some input fields named “Field A -> Extract A” and similar.
• In the “Field A -> Extract A” drop down menu select “Referral”; this will pull out the SERP’s URL on which the ad was shown. In the box to the left on the drop down menu write “(\?|&)(q|p)=([^&]*)” without the quotation marks. This is a regular expression which extracts the search query from the SERP’s URL.
• In the “Field B -> Extract B” drop down menu select “Campaign Medium” and write “ppc|cpc” in the box. This filters out all the organic clicks.
• In the “Output To -> Constructor” drop down choose “Customized Field 1” and enter “$A3” in the box. This just tells Google Analytics where to store the data. Finally you need to click the button to make field B required and the one to turn off case sensitivity. Then apply the filter to your new profile.
• The 2nd filter includes this new data in the keyword report. Again, you want to set up an advanced custom filter but this time choose “Customized Field 1” from the “Field A -> Extract A” drop down. In the box write “(.*)”
• For “Field B -> Extract B” select “Campaign Term” to find out which of your keywords the search query matched and enter “(.*)” again in the box.
• Finally in the “Output To -> Constructor” menu choose “Campaign Term” or wherever you want your data to go and then enter “$B1, $A1” The space after the comma means that you can export your data to a .csv and have a separate field for the actual search term.
• If you’ve followed the steps as I’ve laid them out then the filters should be applied in the right order; if you want to check the information is there when you click to edit the new profile from the “Analytics Settings” page.

As always, it’ll be a little while before Google Analytics starts to register the new data so don’t be too impatient. Unfortunately the filters can’t be applied retrospectively so you can’t start using them on all your old data but as far as I’m concerned this is the only downside. Set up the filters and start refining your AdWords campaigns!

In my ppc for travel post from a couple of weeks ago I recommended splittester for testing ad variations. I got a bit fed up with only being able to test two ads at once and having to type the information in for each test. I though it would be a lot easier if I could do the same thing on a spreadsheet. I emailed Brian Teasley the creator of splittester, to find out what sort of statistical test he used so that I could implement it in my own spreadsheet. He offered to sell me his own spreadsheet for $950. NINE hundred and fifty dollars! Nine HUNDRED and fifty dollars! Nine hundred and FIFTY! For a spreadsheet! Brian, I could buy a bank for that.

I wasn’t going to pay $950 for a spreadsheet so I began to do my own research about statistical testing with the aim of using it for our own ads. Having read quite a few SEO blog posts on the subject I think that quite a lot of people don’t really know what they’re talking about when it comes to ad testing. Throughout this article I’m talking as if you’re optimising for CTR. The procedure for optimising for conversion rate is very similar; for impressions read clicks, and for clicks read conversions. Here are 5 of the most insidious errors:

1. You must have x clicks (mistake 1).
It isn’t a higher number of clicks that makes a result more statistically significant; the important thing, your sample size, is the number of impressions. Think of it this way; if you had an ad with 1,000,000 impressions and no clicks you wouldn’t wait for it to get a certain number (30 seems popular for some reason) of clicks before deciding it was a bad ad. In my own model I treat each impression as a Bernoulli trial with a click being a success. Then I estimate the binomial parameter p, the variance of which gets smaller as the number of impressions increases.

2. You must test two ads.
People only talk about A/B testing of adverts. You can still get valid results if you’re testing 1000 ad variations at the same time. The main problem with this is that your test would have to run for a long time to get significant results. It is also hard to say why the best ad might be best which makes it difficult to write your next ad variation.

3. Use a two-tailed test.
Some of you might not know what a two tailed test is. Imagine you have two ads called A and B that you’re testing for CTR. To test the hypothesis “A and B have different CTRs” you’d use a two tailed test to sum the probabilities that A was better than B and that B was better then A. If the test tells you your hypothesis is true what do you do? You’re no further forward because you already thought the ads would have a different CTR otherwise you wouldn’t be testing them.  To establish how sure you can be that ad A is better than ad B you must use a one tailed test. Telling someone that “You are approximately 99% confident that the ads will have different long term response rates” is useless.

4. You must have x clicks (mistake 2).
It is true that if the test runs for long enough then it will be obvious which ad is actually the best. But how long is long enough? Waiting 6 months to see if an ad with an apparent CTR of 0.5% is going to have a late surge to beat one with a CTR of 4.5% just causes your business to miss out on clicks. To solve this problem and know when a split test has been running long enough you must use a statistical test. I’d say there are three possibilities to choose from:

• The simplest test to use is known as a z-test. To use a z-test you must assume that CTR is normally distributed and your sample size must be big enough so that the sample CTR variance is a close approximation to the true variance.
• For small sample sizes use a t-test. Or more specifically, Welch’s t-test. This test does not assume that you know the population variance so it is a better test than the z-test. For large sample sizes the t-distribution matches the normal distribution (used in the z-test) so for large samples, since the t-test is more complicated to use, I’d use the z-test.
• For the above two tests there is an implicit assumption that CTR is normally distributed (on a bell curve). I think this is actually the case but if you disagree let me know why on the comments below and then start using the Mann-Whitney U test. Wikipedia says that for normally distributed data a Mann-Whitney U test is 95% as good as a t-test and it is less likely to give spurious results based on outliers. I would consider using this test when the CTR is small since then anyone who clicked the ad could be considered an outlier.

5. Any difference in CTR is because of the Ad.
I haven’t seen this view all that much but its one I believed myself until quite recently. I thought that any difference between the CTR of the ads being tested must be due entirely to the difference between the ad texts. I didn’t think that one ad might’ve been shown on slightly more relevant keywords or at a time of day when it was more likely to get clicked. How do you compensate for this when you’re testing? For a useful campaign ‘in the wild’ I don’t think it is possible to completely avoid this problem; it is impractical to have only one exact matched keyword per ad group. Instead do the best you can by following AdWords best practice and using tightly grouped keywords in each ad group.

I hope to blog a bit more about ad variations and statistical testing; I’ve had some pretty weird things come up as being statistically significant. Put any questions or comments in the form below and I’ll try to address them in my next post.

It is a PPC truism the ads in the top two positions get too many “curiosity clicks” and tend to convert poorly in comparison to ads lower down the page. I guess the thinking is that people who bother to read past the first couple of lines on the SERP are serious buyers. (So much of a truism that I can’t seem to find one blog post to link to that deals with this. Maybe it is a Richard Fergie truism rather than an industry wide one).
This all seemed to make sense so I introduced position preference bidding for one of our clients last week. There was no lower limit on ad position but the ads should not have appeared higher than position 3.  In the clients own words the results were “a little grim to say the least.” Conversion rates dropped over 10%. O dear.

I should’ve used my head and done some number crunching beforehand. Instead I’ll do it now and share the results with you. I know how much you love graphs, so I’ll even throw in one of them. Just for you.
The data comes from clients that use conversion tracking and for whom I also had last months data already cached in AdWords editor. For each ad group (over 2000 in total) I’ve plotted its average ad position against its conversion rate. Conversion rates can be very different across different verticals so this figure has been expressed as a percentage of the account average. A logarithmic scale has been used for the conversion rate percentage to try and space the data points out a bit.

Kevin’s comment when he saw this was “that looks like a bit of a mess” and I think he’s right. The graph is dominated by travel pay per click campaigns and recruitment accounts which make a big cloud in the middle. There seems to be a dramatic tail off in conversion rates for the DNA testing vertical about position 5.2 but this data is from only one small account so it may not be that accurate.

So what does this mean in terms of position preference bidding? It all depends on how much you have to pay for the top spot.  As this next graph shows, there is a definite increase in traffic associated with the top ad positions (the trend is exaggerated when a logarithmic scale is not used) so the higher placed your ad the more conversions you’ll get. The trend is not as pronounced as I thought it would be, but there is a correlation there.

If the cost/conversion is low enough you stand to make a lot of money, but if the CPC pushes your cost/conversion too high then it might not be worth bidding for the top spot.

To calculate if bidding for the top spot is worthwhile you need to decide if the tighter margins you’ll get as a result of increased ad spend is worth the extra sales. In some cases it will be, in some cases it won’t. I think my first graph shows that, at least in the verticals tested, there is no sweet spot for ads where conversion rates are high and prices are cheap. It comes down to a quite simple equation; make more on a tighter margin with a higher CPC or make less but have more profit on each transaction.

My own placement preference bidding experiment was a failure; let me know if you’ve had any successful ones.

Do you do any sort of pay-per-click management for the online travel vertical? I recently attended a Google AdWords Webinar about online travel trends which they based on a ComScore study of 50,000 UK web users. Our Google rep sent me a copy of the ComScore study which I have used to bring you my top ten useful tips for running a travel PPC campaign.

• Nearly half of all travel searches are brand related; can you afford to miss out on all this traffic? 36% of people who buy holidays use a brand search first and use a brand search immediately before purchasing so bid on branded keywords in your PPC campaigns
• Use day parting for PPC; people are 30% more likely to purchase a holiday on a Monday or Tuesday. Increase your bids then to capture this traffic and lower them at the weekend. Only 7% of purchasers buy a holiday on a Saturday.
• Get them early; 15.9% of purchasers buy their holiday from the first site they visit. Only 1.6% will buy immediately, but around 14.3% will return at some point for a conversion. Forget what you’ve learnt about the buying cycle; bidding on keywords that customers use in the research phase can get you a 15.9% conversion rate!
• Make sure your URL’s are memorable; 35% of transactions occur without a search on the same day. These people must’ve seen something they liked then gone away to think about it. Make sure they can remember where they were.
• Destinations aren’t as important as you think. 45% of online travel purchases are made without a destination search. Of course this means that 55% do use a destination related search term but I used to think that just about everybody would search for their destination at some point.
• Save some money for January. For the last few years there has been a massive peak in travel searches every January. Look on Google Trends with the travel query of your choice. Or don’t; trust me, there will be a peak in January.
• Ad variations are always a bit of a mystery. Test everything. Once I misspelled “hotels” as “hotsel” is an ad which turned out to have a (statistically) significantly better CTR. I thought I’d found something great so I rolled similar variations out across other ad groups. A few weeks later I checked to see what was going on, using splittester to judge which results were significant. Some ad groups it was better, some ad groups it was worse. I have no idea why. Test everything all the time.
• Most purchasers will visit your site at least twice before purchasing; make repeat visits more likely by including new and interesting content for them.
• Be patient. You’ve made all these changes, but on average it takes 29 days between first search and transaction for a holiday buyer. 30% of purchases occur more than 6 weeks after the initial search.
• Don’t want to be patient? Want to get the 17% of users who purchase after only one search? Then ideally you’re from easyjet, ryanair or some other well known airline. Branded searches tend to convert quicker (63% of single search transactions are branded) so build your brand if you want the shortest gap between click and conversion

I only got to look at the study this week so there hasn’t been time to see if all of these tips really work. I’ll let you know if any big surprises come along as I collect more data.

Update: We have now published a travel SEO article which looks at how to target searchers at the right stage of the buying cycle.

Its Friday afternoon, no one’s blogged on here for a while and I’ve been asked to step up to the plate without mentioning SpyFu! I should’ve been thinking of what to post for a few days now but I’m afraid I’ve been a bit distracted…

So, I’m sitting in a bar with a nice cool pint. I lean back in my chair and look around; what do I see?

That guy over there, he’s got one friend opposite him but he’s talking so the whole room can hear.”He’s not targeting his efforts” I think to myself, “expensive strategy but I guess he’s making a lot of impressions. I suppose his CTR can be quite low and he can still have a lot of success.” At this point I realise that AdWords has ruined my life but I can’t stop myself…

Thin skinny guy, black coat, black hair, band t-shirt. Leaning against the bar, nearly finished his drink. Looks like he’s watching the girls in the booth by the window.But he’s too slow, they’re leaving. Hang on, they’re not, some of them are just going to the toilet. But not all of them. Black haired guy finishes his drink and walks nervously over. There are two girls left, one them’s wearing a band t-shirt. I can’t hear what he says but I think “That’s more like it. He’s targeting his market”

Now who’s that? One guy, group of girls. The leggy blond doesn’t look particularly interested but one of her friends is flicking her hair and touching his arm. The guy is slowly ignoring the rest of the group and turning to face her. “Bidding too low” I remark, “he can’t keep his position with blondie.” Then I give him the benefit of the doubt; “placement targeting; he knows what converts best for him.”

Back to the skinny guy. He’s sitting on his own now; “high bounce rate?” I think as he gets up to go to the bar. But maybe not; band girl is coming out of the toilets, she looks at the booth where her friends are and then scans the rest of the bar. “Rookie error, he should’ve 301′d to his new location.” The girl pauses, lonely by the toilet door but not for long; “Are you looking for someone?” I hear. “Is that bidding on a competitors keyword?” I think. Skinny guy comes back around the bar with two drinks, his new friend smiles and walks over. “Brand recognition in action. And he gave her an incentive for a repeat visit.”

I see my own date in the door so I stand up and wave. I’m unusually tall, so my ad copy stands out from the crowd. I know she like this place, one of her friends told me, I hope just being here is good enough landing page optimisation. How forward should I be? Calls to action are good right? Perhaps it is at this point that my analogy breaks down, I don’t want to be too forward. Still, I hope I get a conversion.

Wouldn’t it be great if you could convince your competitors that they had to spend big in order to keep up with you? If they use SpyFu to estimate your ad spend and CPC then you can. I’ve looked at what SpyFu say about their own algorithm and techniques and then I’ve combined this knowledge with what I observed last week to come up with these ideas about how to turn SpyFu into a double agent.

Firstly let’s look at how SpyFu calculate their Ad Spend and CPC estimates. Then I’ll show you how to use this knowledge to increase SpyFu’s estimates of your Ad Spend and CPC without spending any more money.

How Does SpyFu Estimate Daily Ad Spend?
The onsite help, also available in The SpyFu Instruction Manual says this about calculating Daily Ad Spend:

When we calculate Daily Ad Budget, we start with all the keywords that we have seen a domain advertise on. We eliminate overlapping keywords. For example, “race cars”, “luxury cars”, and “cars” becomes “cars”. Then, we take into account the current and historical positions that we have seen the domain’s ad appear for each given keyword. Based on the position of each ad, we estimate the price that the domain likely pays for the keyword. Basically, we then add up all the custom individual keyword costs per day and we arrive at the Daily Ad Budget.

Let’s add in what we already know about SpyFu to our new knowledge from the above passage:

• SpyFu uses its web scraping database to get a list of all the keywords a site advertises on.
• Overlapping keywords are eliminated.
• A daily cost is estimated for each keyword given its ad position.
• The total can then be easily found.

I’ll look at this step by step:

Web Scraping: Reading the Internet
Velocityscape (who own SpyFu) have a massive web scraped database. If your ad appears on one of the first few SERPs for a term then they will know about it and have a record of it, unless the term is relatively unusual. This is why estimates are better for accounts with a large number of impressions; such accounts either bid on lots of relatively common terms that will be scraped by SpyFu or they bid on a few very general terms which are also covered. Accounts that bid on lots of low traffic, long tail keywords might not get picked up at this stage so SpyFu will not have very good data on them.

Ignoring the Long Tail
The example that SpyFu gives for eliminating overlapping keywords is clearly far too restrictive; I doubt they would lump “race cars” and “luxury cars” into “cars” but you rarely see three word keywords whilst using SpyFu and, personally, I’ve never seen a four word one. Accounts which bid mainly on long tail keywords will have lower impressions and a lower daily ad spend but SpyFu does not take this into account, ignoring the long tail and lumping everything into a more general query.

Counting the Cost
The accuracy of the daily cost estimate for each keyword is a tricky one. If they work it out by considering an average CPC for each keyword multiplied by an estimated number of clicks then their accuracy depends not only on their CPC estimates (see below) but also their estimates of CTR’s. Since all this is done by a computer CTR estimates will depend only on ad position and not on the ad copy which the folks at the Mind Valley Labs tell us can make a massive difference.

What About CPC?
I’ve just said that the Daily Ad Spend estimate depends on the accuracy of the CPC estimate. So if you can manipulate what SpyFu thinks about your CPC then you can manipulate their Daily Ad Spend estimate as well. So how does SpyFu estimate CPC?

If you take the Average Cost per Click of every keyword that a domain advertises on, add them all up, and divide by the total number of keywords, you will have the Avg Cost/Click for a domain. For example, if a domain advertises on 3 keywords with Avg Cost/Clicks of $1, $2, and $3, respectively then the Avg Cost per Click for the domain would be $2.

This is clearly a load of rubbish. The following table shows that SpyFu’s method only gives the correct answer if there is the same number of clicks for each keyword:

Now look at the table if each keyword gets a different number of clicks:

In this case the average CPC is 21/9=2.33 not 2 as SpyFu’s method tells us.

At $70/month for a subscription I hope that SpyFu doesn’t really estimate CPC in this way.

Assuming the SpyFu algorithm does actually estimate CPC in this way then their estimate will be most accurate when the standard deviation in the number of clicks for each keyword in an account is small. As we have seen the method gives the right answer in the case where all keywords receive the same number of clicks, it also wouldn’t be far off if the number of clicks was dominated by a few keywords.

Estimates for Individual Keywords
Of course, all this depends on the accuracy of their estimates of average CPC for each keyword but to say any keyword has an average CPC ignores one of the most fundamental parts of AdWords. No one can accurately predict the CPC for an ad without knowing how the quality score system works. I’m sure SpyFu has the technology and capacity to scrape landing pages as well as search results; they can then apply their own quality score algorithm but I can’t imagine their results are very accurate.

The AdWords API has a service that will estimate CPC for a given keyword but I doubt SpyFu use it: The API service is designed to be accurate when it can access campaign and ad group quality scores for the account it is being used for; this information will not be available for SpyFu.

Get on With It!
Enough waffle. Now we’ve looked at how SpyFu’s estimates are calculated lets look at what you can do to change them, without actually changing you account of course. None of these methods have been tested and some of them require information that I don’t know, but if what SpyFu says about itself is true then at least the first two of these methods should work.

Bid High
SpyFu do not know what your click through rates are. They calculate daily spend on an estimated daily spend for a given keyword which is probably based on an average CTR depending on the ad position. So if you use a high traffic keyword with and bid enough that your ad gets a good position then SpyFu think you are spending a lot of money. But what if your ad text was so bland that the real CTR was incredibly low? Then you’d be spending a lot less than SpyFu thought.

Bid on the Long Tail
Bid on the long tail of a really expensive keyword. SpyFu even published a list of the most expensive keywords to make this easier for you. The idea here is to use the way SpyFu lumps long tail searches together in order to convince them that your CPC is huge. For example top of their top list is “conference calling companies” at $52 a pop. As I’ve said, I’ve never seen a four word keyword in SpyFu so try bidding on something like “free conference calling companies.” The term “free” is common enough that SpyFu will probably scrape for it so their algorithm should record you bidding on “conference calling companies” but your actual CPC will be lower because people will bid less for the word “free.”

Of course the algorithm might decide to cut your keyword down to “free conference calling” instead so you should bid on a variety of “x conference calling companies” to make sure your long tail is shortened to the keywords with the maximum CPC.

Bid at the Right Time
SpyFu’s database is not updated very frequently. Most things I’ve read say it’s only redone every 60-90 days. Spend a lot when SpyFu does its update and you’ll appear to be spending that much for the next three months. Take these update figures with a pinch of salt; I can’t find anything official about this and quite a few blog posts about SpyFu appear quite biased against it.

When is SpyFu’s next database update due? I don’t know. It is also possible that their data gathering goes on all the time but they only do their calculations every three months. If this is the case then this method won’t work. Unfortunately I think this probably is the case.

Bid on the Right Place
SpyFu’s owners, Velocityscape are based in Phoenix, Arizona. Assuming their scraping is all done from there then a high spend ad group targeted at their location will be scraped and analysed by them as if it covers the whole web. Unfortunately things are unlikely to be that simple.

I doubt the whole of Velocityscape’s scraping operations are centralised; SpyFu UK would indicate that they have at least one other location. This is not a big barrier, campaigns can be targeted at more than one location but if they scrape using something like the AdWords Ad Preview Tool this method won’t work at all. Does anyone know anything about this?

Control the Double Agent
Always remember that SpyFu works both ways; while you’re using it to spy on your competitors, they are using it to spy on you. But also remember that you have complete control over what information SpyFu can gather about you and, to a certain extent, you can manipulate that information.

It would be an interesting challenge to see who can get the highest estimated daily ad spend for the lowest actual budget. Let the games begin!

I was doing some keyword maintenance today and noticed something new when I hovered over the magnifying glass tool beside a keyword:

It looks like Google are being more explicit with their quality scores. I haven’t noticed this before but rather than being an accidental leak this time it’s been integrated into Google’s changes to make quality score more transparent.

I’m not sure how much this will change advertiser behaviour. Perhaps knowing that a keyword is nearly “Ok” will encourage people to optimise their landing pages and ad descriptions further, rather than just pausing.

More information is also given for keywords with poor quality scores:

In this case we are told that no ad will be run on broad match, but that the ad can be triggered on a phrase match.

How will this new information affect the way you run your AdWords campaigns?

For those of you not in the loop, SpyFu.com is a web service that provides information on who is bidding on what keywords as well as further information on competitors’ daily ad spend as well as their average CPC. The startup squad have a great article on what spyfu does and how it does it.

No Mr Bond, I Expect you to Disagree.

Everyone seems to have an opinion on how useful spyfu is, ranging from the standard “it works/doesn’t work for me” to the commonly held belief that it is only really useful for large accounts. People disagreeing on the Internet? It’s time to try science!

The Experiment

I selected 20 of SEOptimise’s clients, discarded two of them for using foreign currencies (spyfu has a UK and US version but I didn’t want the hassle of fiddling with exchange rates) then ignored a further two when spyfu provided no data on them. I then compared the percentage accuracy for SpyFu’s estimates for daily ad spend and CPC with the actual daily spend and average CPC as well as the number of impressions the ad generates. I even drew graphs.

The Perils of Percentages

I decided to work with percentages rather than actual pounds and pence for two reasons; client confidentiality and because there is such a massive variation in budgets and CPC’s across the selected accounts. Considering percentage variations means that a £10 error on an account spending £1000/day counts less than for one on a spend of £10/day. But you knew that already, didn’t you?

One side effect of this is that, particularly on a graph, it looks like the lower estimates are a lot more accurate then the upper estimates. However, this is because the lower estimate is never less than zero so it can never be more than 100% different from the actual value.

Enough writing. Time for some pictures.

Bigger is Better?

With a bigger account, spyfu has more data so its estimates will be more accurate. Right?

This first graph is of the error in spyfu’s ad spend estimates against total ad spend. I expected that the error would be less for big spending accounts.

As you can see, it looks like my hypothesis is correct (at least for the upper estimate, which tends to dominate these charts) but then the error increases again for the account with the largest ad spend. There are three possibilities here:

1. Accuracy is unrelated to ad spend.

2. Accuracy increases with ad spend and the last result is just a freak

3. Spyfu is actually most accurate somewhere in the middle range of ad spends.

To me, option 3 seems the most unrealistic and option 2 the most likely. Let’s check by looking at CPC. . .

O dear, this graph of CPC error against total ad spend shows no correlation of any sort whatsoever.

You’re Using the Wrong Big

Ok, so it might be unfair to say an account is big just because it spends a lot of money. Spyfu uses web scraping so it should be more accurate for account with a large number of impressions.

The following graph shows the percentage error ad spend against the number of impressions the account has generated:

This graph looks very similar to the Ad Spend Error vs. Ad Spend graph, probably because daily ad spend is quite closely linked to the number of impressions an account gets.

The graph of CPC error against number of impressions, below, also doesn’t show any correlations.

The graphs so far, particularly for CPC show that there is not a strong relationship between the size of an account and the accuracy of spyfu.

Assuming our largest account is just an anomalous result I think there is a case for saying that estimates of daily ad spend are more accurate for the larger accounts but that the error in CPC does not improve (or get worse) as account size increases.

Spyfu’s algorithm is a bit of a mystery, perhaps it calculates CPC entirely separately from daily ad spend. Perhaps comparing CPC with the size of the account is like comparing apples and pears.

Comparing Apples and Apples

Now let’s look at graphs for the CPC error against actual CPC:

My goodness, there might even be a trend there. It looks as though the upper estimate gets more accurate as CPC increases. The account with the highest CPC is an anomaly since spyfu has given both a lower and upper estimate of 0; any PPC marketer with common sense would ignore this anyway.

This next graph shows the same information, but with the upper estimate plot removed so that it is easier to spot trends in the lower estimate:

So as CPC increases the lower estimate actually gets worse. This is all very confusing, what does it all mean?

What’s Going On?

So far the results seem to indicate the following:

1. Spyfu’s estimate of ad spend is more accurate for larger accounts whether size is measured by ad spend or number of impressions.

2. The accuracy of the CPC estimate does not depend on the account size.

3. The upper estimate for CPC improves as CPC increase, but…

4. The lower CPC estimate gets worse as CPC increases.

What Does This Tell Me?

Nothing at all. The sample size is small and the correlations are weak; you would be a fool to use this data to decide how accurate spyfu is. Besides, it’s not like the four conclusions written above are even quantitative. It would’ve been much more useful if I’d been able to say “The lower estimate of ad spend for an account is within 50% of the actual value 90% of the time” but as you have seen the results do not support anything like that. The best I can hope for is that someone will be intrigued by what they’ve seen here and decide to test clients from their own MCC. If they have more data, or if lots of people with not much data arrive at the same conclusions then perhaps I can say I was onto something.

So SpyFu is Useless?

Far from it. When it provided information, true ad spend was within spyfu’s estimates 71% of the time (ignoring the accounts for which it returned 0) with CPC being slightly better at 79%. I will continue to advise and act as if true ad spend and CPC lie within spyfu’s estimates because the information it provides is better than no information at all.

Following XL Airways going into administration last week and yesterday’s announcement that Lehman Brothers are filing for bankruptcy, many competitors have quickly setup PPC ads targeting brand queries to take advantage of this.

Below are the ads appearing on Google for XL and ads have quickly started appearing today for Lehman Brothers as well:

This obviously wouldn’t have been possible before Google opened up brand bidding, but is this a good thing? Personally I’m quite impressed with how quickly competitors have acted and surely for people searching for alternative flights to XL this has to be useful.