Yahoo’s New Paid Inclusion

At Search Engine Strategies earlier this month, all the spider-based search engines confirmed their commitment to a comprehensive search index containing as much relevant content as their spiders can find.

Google confirmed its stance against paid inclusion, saying its spiders should find all relevant content. Teoma/Ask Jeeves backed away from XML paid inclusion. Its communities-focused algorithm had problems accurately targeting data structured in an XML format. The company retains a per-URL paid inclusion program to facilitate more rapid inclusion of URLs and content into the index with content collected by the spider.

Yahoo, meanwhile, announced a completely new paid inclusion program, Site Match, part of its content acquisition program (CAP). Site Match is a shift from the earlier Inktomi, AltaVisa, and FAST inclusion programs. It introduces CPC to all commercial inclusion (other than the Yahoo directory).

Yahoo’s new paid inclusion philosophy eliminates the free ride for clicks of past per-URL inclusion programs. Yahoo created a similar pricing model for XML and per-URL inclusion programs. A lack of per-click fees for traffic meant low incentive for Webmasters or Web marketers to keep page content relevant for earlier per-URL paid inclusion programs at Yahoo divisions (Inktomi, AltaVista, and FAST).

Marketers (particularly affiliate marketers, who build pages mostly with links to affiliate merchants) optimized pages for position and, therefore, high click volume, with little thought to relevance or eventual conversion. Other than a one-time fee, there was no cost.

Some per-URL inclusion resellers even let marketers change URLs within a domain on a single annual inclusion. The ability to swap URLs within a domain may seem trivial, but some marketers used it as a method to tune pages for high position, regardless of relevance. They then turned off paid inclusion. Usually, search engine spiders found the optimized page and included it anyway. Yahoo probably found this abuse of the inclusion system resulted in suboptimal search experiences. A preponderance of affiliate and other poor content pages proliferated in its index.

Now, all commercial paid inclusion clicks generated through the Yahoo/Overture Site Match program will be charged at CPC. No one wants to pay for irrelevant traffic that won’t convert.

All the major engines are committed to a free crawl of Web site pages and data. Those of you with older sites may believe search engine index inclusion is a right, not a privilege, and should remain free (not counting search engine optimization, or SEO, work to make the site search-engine friendly, of course). New York sidewalk vendors selling sunglasses, T-shirts, and handbags on busy streets and paying no rent (not to mention sales or income tax) feel the same. Like the merchant with a great position on the corner of Broadway and 42nd Street, Web marketers who use luck and skill to get good organic search position should savor the free traffic and associated revenue and be thankful for every day the gravy train delivers free of charge.

Now, back to reality. The landscape has changed. Both per-URL and XML forms of paid inclusion in the Yahoo/Overture database have a PPC component. You must answer questions about your Web site, business goals, and business objectives. The following may help you make the best decisions regarding both per-URL and XML paid inclusion in the Yahoo/Overture/Inktomi environment:

• Will I be included without paying? Perhaps. It’s likely the new Yahoo SLURP spider, like its Inktomi predecessor, will follow links between and within sites. Look in your log files (manually or with an analysis tool) and see which site pages (if any) are visited by Yahoo’s SLURP spider. If that’s the case, it appears you’re at least being indexed. You can also try the following command in Yahoo’s search box: inurl:domain.com site:domain.com. This command works similarly in Google and reports URL counts.
• If many pages of my site are already indexed regularly (typically once a month), why should I pay for inclusion through Overture’s Site Match?
  • Relevance tuning: With a frequent refresh of a spider or the ability to tune an XML feed for relevance, you’ll get more relevant traffic and possibly higher positions on relevant searches. At the CPC for paid inclusion traffic, return on investment (ROI) may be high enough so additional quality traffic pays for itself (even when taking into account the free traffic you got before).
  • Tuning for presentation/display: Your client’s or prospect’s first interaction with your company may be a search result listing. Make the right impression. Articulate what you offer the searcher who clicks.
  • Freshness: Organic (free) spiders are on different schedules, ranging from daily to monthly, perhaps longer. Paid inclusion guarantees freshness. Change title, descriptions, or copy to reflect the most recent information. Even price can be placed in the title.
  • Reporting: If your log or Web analytics program doesn’t adequately count clicks by URL or mix of keyword phrases, paid inclusion reports provide this data. Of course, to determine ROI on paid inclusion, it helps to track behavior beyond the click and tie it back.

   

• If site content (products, inventory, copy, or prices) change regularly, is paid inclusion a good idea? Assuming ROI is there, paid inclusion may be a very effective and efficient part of a search engine marketing (SEM) marketing mix.
• Should I include all my pages in a paid inclusion feed or just the “important” ones? Best practice is to apply whatever success metrics are in place for other online and search campaigns to paid inclusion. Both XML and per-URL inclusion programs can be paused on a URL basis. That way, you pay only for traffic that results in good ROI.

If you have a good Google organic rank and relevance, chances are you’ve seen a 25 to 35 percent drop in traffic if your site isn’t in Yahoo’s index. That drop in traffic volume alone could drive you to consider paid inclusion. First, consider the ROI of a paid inclusion program.

At the CPCs charged, does it make sense for you, your business, and your specific objectives? If so, the more you understand about paid inclusion, the more you can make it work for you. Talk to an Overture representative or a professional who understands the new inclusion landscape.

Want more search information? ClickZ SEM Archives contain all our search columns, organized by topic.