How Google's 'Panda' update put some websites on endangered species list
A tweak in the way Google ranks websites has had serious consequences for many web-based businesses and has led to accusations of anticompetitive practices
Google antitrust inquiry: Eric Schmidt to meet Europe's competition chief
The European commission is expected to announce whether it will formally object to what some see as Google’s abuse of its power. Photograph: Karen Bleier/AFP/Getty Images
Sophie Lee got a shock when she checked her website's statistics data on 24 February. The number of visitors to her site about irritable bowel syndrome, IBStales.com, which she has run since 2003 and relies on for her income, had dropped precipitously. "It was so sudden," she says. "They had halved overnight."
Some examination quickly showed what had happened: Google's engineers had tweaked the company's search ranking system, in what later became known as the "Panda update", and demoted Lee's site so that rather than appearing, as it had, among the key first 10 results when people put "IBS" or "irritable bowel syndrome" into Google's search box, it was now about 300th.
And being on that first page is crucial: a recent study showed that 89% of clicks are on those 10 results. Suddenly, Lee's site – which is her livelihood – had effectively vanished from the sight of searchers in the US.
Worse was to come: a couple of weeks later Google extended the Panda update to the UK. And in October it tweaked its search algorithms again, as it says it does constantly – and Lee saw another fall in visitors.
"We were crushed by the Panda update, and the site is now gasping for breath," she said. "Traffic is down 75%, revenue is down 90%, and I'm getting seriously worried about the future." She has never bought adverts on Google, and isn't thinking of doing so now. "Everybody says not to rely on Google, but there's no way around it," she says.
Though Google's search system is often thought of as purely machine-driven, in fact its engineers are constantly changing the ways in which sites are ranked.
"Panda," they indicated, would downgrade so-called "content farms" of copied or useless content. The implication was that Lee might have fallen foul of that - though she insists that IBStales.com consists of original stories from fellow sufferers of the gut condition and some advice.
But to some, Lee's drop in income – which has been mirrored by a number of British technology news websites – is seen as collateral damage in a far bigger battle being fought over Google's dominance of the search industry, where about 95% of searches in Europe) and 82% in North America go via Google.
Any day now, the European commission is expected to announce whether it will formally object to what some see as Google's abuse of its power in the way that it treats smaller sites that offer the same sorts of services as it does.
If that happens, Google could be forced to comply with strictures on the way it treats rival sites offering particular sorts of search – for news, products, maps, shopping, images or videos – rather than pushing its own on the site. Alternatively, it could face fines of millions of pounds.
The EC began its investigation almost a year ago, after complaints from a number of companies, including Foundem.com, a British "vertical search engine" company that offers comparisons for a huge range of products and services, such as flights, hotels, gadgets and jobs.
Foundem says that it was repeatedly demoted in Google's search rankings despite breaking none of its publicly-stated rules. Shivaun and Adam Raff, its founders, woke up in 2006 to discover they had suffered the same fate as Lee: their site had vanished.
It stayed that way for three years, even while they won awards for being the UK's best comparison site. Then Google tweaked its system again, and Foundem reappeared. But now the Raffs were determined to make Google treat them fairly. Google said that it "de-indexed" Foundem because too much of its content was copied from other sites, which leads to automatic downgrading.
But the issue of "copying" is a mysterious one – because Google itself effectively "copies" everything, and products such as Google News do not originate any content at all. Yet they aren't downgraded, and they always feature ahead of any rival. Why, then, was Foundem downgraded, asks Shivaun Raff?
She thinks that the Panda alteration was part of a preparation by Google for a battle with the EC – and, potentially, the US, where the Federal Trade Commission (FTC) has begun an antitrust investigation of its share of search advertising, and a Senate committee quizzed the Google chairman, Eric Schmidt, about its dominance in the field in September.
Schmidt insisted that Google faces a "tremendously dynamic and competitive" environment, citing Microsoft's Bing – which has about a 3.6% share worldwide, compared with Google's 91% – and Yahoo and Yelp, now being used by Apple for searches done with its Siri voice assistant.
"Panda is a collection of disparate updates," Raff says. She says that though panda was widely touted as an attack on content farms, "it also marks an aggressive escalation of Google's war on rival vertical search services. First, vertical search services are in many ways the polar opposite of content farms" - because they link to multiple different sites, rather than containing content on one site.
"Second, Google's anticompetitive demotion of rival vertical search services, while simultaneously promoting its own, lies at the heart of ongoing antitrust investigations on both sides of the Atlantic.
"Panda wasn't just deployed in the midst of these investigations; we suggest that it was deployed in direct response to them. By bundling these diametrically opposed updates together, the 'content farm' elements could be viewed as providing cover for the vertical search targeted elements."
For Google, the stakes are high. If the EC or the US finds it is abusing its dominance – by pushing sites such as Foundem down – then it could face close oversight of its search algorithms and might be obliged to check them with the EC before implementing them.
A spokesman for Google declined to comment on Lee's experience, but said: "Quality is obviously a core factor in how highly a site ranks. In search, we're constantly updating and fine tuning our algorithms to help high-quality publishers get more traffic.
"For some context, last year we tested about 6,000 changes and launched about 500. As you can imagine, with each of these changes some websites will see more traffic and some will see less."
Lee, meanwhile, just hopes she can climb back up the search rankings. "It's almost impossible not to be reliant on Google," she says. "It has such enormous power. I'm trying to use Facebook and Twitter, to get less reliant on it. But I need this to work. It's my job."
Charles Arthur
guardian.co.uk, Monday 5 December 2011 16.24 GMT
© 2011 Guardian News and Media Limited or its affiliated companies. All rights reserved.
How Google's 'Panda' update put some websites on endangered species list | Technology | The Guardian
A tweak in the way Google ranks websites has had serious consequences for many web-based businesses and has led to accusations of anticompetitive practices
Google antitrust inquiry: Eric Schmidt to meet Europe's competition chief
The European commission is expected to announce whether it will formally object to what some see as Google’s abuse of its power. Photograph: Karen Bleier/AFP/Getty Images
Sophie Lee got a shock when she checked her website's statistics data on 24 February. The number of visitors to her site about irritable bowel syndrome, IBStales.com, which she has run since 2003 and relies on for her income, had dropped precipitously. "It was so sudden," she says. "They had halved overnight."
Some examination quickly showed what had happened: Google's engineers had tweaked the company's search ranking system, in what later became known as the "Panda update", and demoted Lee's site so that rather than appearing, as it had, among the key first 10 results when people put "IBS" or "irritable bowel syndrome" into Google's search box, it was now about 300th.
And being on that first page is crucial: a recent study showed that 89% of clicks are on those 10 results. Suddenly, Lee's site – which is her livelihood – had effectively vanished from the sight of searchers in the US.
Worse was to come: a couple of weeks later Google extended the Panda update to the UK. And in October it tweaked its search algorithms again, as it says it does constantly – and Lee saw another fall in visitors.
"We were crushed by the Panda update, and the site is now gasping for breath," she said. "Traffic is down 75%, revenue is down 90%, and I'm getting seriously worried about the future." She has never bought adverts on Google, and isn't thinking of doing so now. "Everybody says not to rely on Google, but there's no way around it," she says.
Though Google's search system is often thought of as purely machine-driven, in fact its engineers are constantly changing the ways in which sites are ranked.
"Panda," they indicated, would downgrade so-called "content farms" of copied or useless content. The implication was that Lee might have fallen foul of that - though she insists that IBStales.com consists of original stories from fellow sufferers of the gut condition and some advice.
But to some, Lee's drop in income – which has been mirrored by a number of British technology news websites – is seen as collateral damage in a far bigger battle being fought over Google's dominance of the search industry, where about 95% of searches in Europe) and 82% in North America go via Google.
Any day now, the European commission is expected to announce whether it will formally object to what some see as Google's abuse of its power in the way that it treats smaller sites that offer the same sorts of services as it does.
If that happens, Google could be forced to comply with strictures on the way it treats rival sites offering particular sorts of search – for news, products, maps, shopping, images or videos – rather than pushing its own on the site. Alternatively, it could face fines of millions of pounds.
The EC began its investigation almost a year ago, after complaints from a number of companies, including Foundem.com, a British "vertical search engine" company that offers comparisons for a huge range of products and services, such as flights, hotels, gadgets and jobs.
Foundem says that it was repeatedly demoted in Google's search rankings despite breaking none of its publicly-stated rules. Shivaun and Adam Raff, its founders, woke up in 2006 to discover they had suffered the same fate as Lee: their site had vanished.
It stayed that way for three years, even while they won awards for being the UK's best comparison site. Then Google tweaked its system again, and Foundem reappeared. But now the Raffs were determined to make Google treat them fairly. Google said that it "de-indexed" Foundem because too much of its content was copied from other sites, which leads to automatic downgrading.
But the issue of "copying" is a mysterious one – because Google itself effectively "copies" everything, and products such as Google News do not originate any content at all. Yet they aren't downgraded, and they always feature ahead of any rival. Why, then, was Foundem downgraded, asks Shivaun Raff?
She thinks that the Panda alteration was part of a preparation by Google for a battle with the EC – and, potentially, the US, where the Federal Trade Commission (FTC) has begun an antitrust investigation of its share of search advertising, and a Senate committee quizzed the Google chairman, Eric Schmidt, about its dominance in the field in September.
Schmidt insisted that Google faces a "tremendously dynamic and competitive" environment, citing Microsoft's Bing – which has about a 3.6% share worldwide, compared with Google's 91% – and Yahoo and Yelp, now being used by Apple for searches done with its Siri voice assistant.
"Panda is a collection of disparate updates," Raff says. She says that though panda was widely touted as an attack on content farms, "it also marks an aggressive escalation of Google's war on rival vertical search services. First, vertical search services are in many ways the polar opposite of content farms" - because they link to multiple different sites, rather than containing content on one site.
"Second, Google's anticompetitive demotion of rival vertical search services, while simultaneously promoting its own, lies at the heart of ongoing antitrust investigations on both sides of the Atlantic.
"Panda wasn't just deployed in the midst of these investigations; we suggest that it was deployed in direct response to them. By bundling these diametrically opposed updates together, the 'content farm' elements could be viewed as providing cover for the vertical search targeted elements."
For Google, the stakes are high. If the EC or the US finds it is abusing its dominance – by pushing sites such as Foundem down – then it could face close oversight of its search algorithms and might be obliged to check them with the EC before implementing them.
A spokesman for Google declined to comment on Lee's experience, but said: "Quality is obviously a core factor in how highly a site ranks. In search, we're constantly updating and fine tuning our algorithms to help high-quality publishers get more traffic.
"For some context, last year we tested about 6,000 changes and launched about 500. As you can imagine, with each of these changes some websites will see more traffic and some will see less."
Lee, meanwhile, just hopes she can climb back up the search rankings. "It's almost impossible not to be reliant on Google," she says. "It has such enormous power. I'm trying to use Facebook and Twitter, to get less reliant on it. But I need this to work. It's my job."
Charles Arthur
guardian.co.uk, Monday 5 December 2011 16.24 GMT
© 2011 Guardian News and Media Limited or its affiliated companies. All rights reserved.
How Google's 'Panda' update put some websites on endangered species list | Technology | The Guardian
Last edited:
