How my site disappeared from Google search

Blog, , , , ,

Seen my personal blog lately? Probably not, if you were searching via Google. Major sections of my site have been disappearing from the search index over the past three weeks. My homepage, my blog and many of the most recent articles on it no longer showed up in result pages. I’m no Matt Cutts, but I get a fair number of people coming to my site when searching for info about Google search, avoiding scams, and how to name their baby. All that traffic has been slipping away.

You can probably imagine how you would feel if this was happening to you. Does Google hate me? Was my site hacked? What do I do, and how much will it cost to get this fixed?

I will answer all of those questions, starting with the first:

My site is falling out of the index, does Google hate me?

Probably not. My situation is actually pretty illustrative – I’m pretty sure Google doesn’t hate me and isn’t unfairly slapping my site down because, well, I work at Google.

That’s right, Google was kicking pages from one of its own employees out of search results. I’m sure I’m not the first. Google doesn’t treat my site any differently than anyone else’s. BTW, standard disclaimers apply to this post.

So I knew there was probably a logical reason for the dropped pages, which brings me to the next question:

Is Google dropping my pages from search results because my site got hacked?

This is a very, very good question to ask – hacking is unfortunately common. This very site has been hacked before. I don’t want to go into a lot of detail on how to tell if your site was hacked in this post, but the Webmaster Central Blog has some good pointers.

Really, this is part of a broader question – what has changed with my site? In my case it wasn’t hacking – it turns out out that Google was getting tons of crawl errors over the past couple weeks. As Googlebot tried to recrawl my site looking for new content, it kept getting network unreachable errors. After days and days of this, Googlebot figured I had closed up shop. We don’t want to send searchers to pages that have disappeared.

Why all the errors? Here’s where I had to do some digging – I have a multi-site account with a web host. It turns out my site had exceeded my bandwidth quota. The worst thing is it’s not the max quota I paid for, but just the amount I had portioned off for my blog. I didn’t think I’d ever get enough visitors to worry about it. D’oh.

What do I do, and how much will it cost to get this fixed?

If your site starts disappearing from Google search results, how can you figure out what’s going on without access to all the uber-powerful, super-secret Google tools that I used?

It turns out I only needed one Google tool to diagnose the problem, and I’ll let you in on the secret. In fact, I’ll leak the url (SEO bloggers get ready to tweet!):

http://www.google.com/webmasters/tools/

Yep, good old Webmaster Tools. Between the reports there and my own host’s dashboard, I was able to figure out and fix the problem in less than 20 minutes. Googlebot will take longer than that to reindex everything but I noticed progress almost immediately:

I looked under Site Diagnostics -> Crawl Errors to see all the pages that Google couldn’t reach. After I fixed the bandwidth cap on my side, I noticed that under Sitemaps, my sitemap status had a little red “X” as well. I logged into to my blog and regenerated the sitemap, checked that it was there, and clicked “Resubmit” in Webmaster Tools.

Voila! Total cost: $0.

Bonus Question: This sounds embarrassing, why are you sharing it with everyone?

Having my own site start disappearing from my employer’s search engine isn’t exactly something to brag about. It’s doubly embarrassing to admit how long it took for me to notice what was going on, though I bet a lot of other site owners are in the same boat, to busy doing their day jobs to constantly check search rankings.

I wanted to share my story because #1, it might help someone figure out what to do if they have a similar issue, and #2, it illustrates a bit about how Google tries to do business.

Everything is set up to give users the best results we can, and to cope with all the spam and abuse on the web. Sites that are irrelevant, unresponsive, or violate the webmaster guidelines might not end up in front of searchers, even if it’s my site. Heck, even if it’s Google Japan.

52 thoughts on “How my site disappeared from Google search”

  1. Really good example of how to use the webmaster tools effectively. Great write up Jason. It’s easy to forget about bandwidth limitations and quota with web hosts….

    Thanks

    Shahid

  2. Great post, thanks for being humble and sharing. Didn’t know about the shared web host potential.. Good to know owning over a thousand sites..

    Did you up to dedicated server vs shared? or upgrade your hosting?

  4. Nice (and funny) post. Unfortunately you super mega tool often is not enough. It’s six month I’m trying to figure out why a web site of mine had gone after the 10th page. And Gwt doesn’t disclose anything.
    No hack, crawl time regular, sitemap ok.

    All pages are on the index, the old dear site: revealed to me.

    So what’s the matter? Google really hate that web site? Maybe, because other competitor with worse link and bad onpage seo stand up of the crowd without given reason.

    So, which is your real secret tool? :D

  5. This is a good example of a site that has been indexed but has fallen out of index. In this case a good investigative crawl issue has resolved the problem.

    The issue that I get across so many times is with old domains, or maybe those that has expired and is being revived. The site has been indexed by other SEs but does not get indexed by Google. Stats shows Google spiders the site, has proper sitemap and has relatively good content but to no avail.

    Are those perfect candidates for reconsideration?

  7. I feel sorry for you now that Matt tweeted this post. Maybe you’ll start getting new crawl errors soon and exceed your quota? Better be ready for it :D

  10. > If your site starts disappearing from Google search results, how can you figure out what’s going on without access to all the uber-powerful, super-secret Google tools that I used?
    > It turns out I only needed one Google tool to diagnose the problem, and I’ll let you in on the secret.

    Would you have known to check Webmaster Tools if you hadn’t first used all the uber-powerful, super-secret Google tools that I used?

    It’s great that Webmaster Tools was all you needed for this particular problem, but the rest of us aren’t so lucky. Webmaster Tools has lost ~6,000 links to my website – a check of which shows the vast majority are still there. In the span of a month I lost 2 points of PR… so that a site in my niche that hasn’t been updated since 2001, has about 1% of the content, and 1/5 the pageviews now ranks way higher.

    Webmaster Tools provides some insight, but it’s no silver bullet.

  11. “Googlebot will take longer than that to reindex everything but I noticed progress almost immediately”

    FYI Google never removes anything from it’s indexes. They just don’t show it. If you go for reinclusion, you can have all your pages restored in a matter of minutes, regardless of the crawler. (At least this is what my experience tells me.)

  12. Just read that you work for teh goog. Perhaps that puts you in a better position to confirm my previous statement. Any chance you could get back to me on that?

  16. Looks like it took a couple of week for it to fall off the index… i guess if you keep a closer eye on the site, it wont happen again.

    I assume its not a big money earner, so your commitment to the site is limited, otherwise you would have picked this up much soon.

  18. I feel your pain. Just after creating our new company website, it disappeared from search totally after 2 months after being near top and not sure what I did. Wrote an email to Google explaining we were a legit company and voila, back at #2 for company name at least. Wish they would have said why we were removed in the explanation.

  20. BTW, thanks to everyone for the grammar and link nitpicks, they should be fixed now. That’s what I get for writing a blog at midnight on a work day.

  21. I don’t make any money off this site, it’s just my personal blog, but I imagine there are a lot of profitable sites that don’t often check rankings or think to check for crawl errors. Hopefully if anyone ends up in the same situation as me they’ll find their way to Webmaster Tools and get everything back in shape.

  23. Nice post…always an education to read about you. All the best to the spouse and kinder….Kent is still here.

    Linda

  25. Hi Jason,

    Great post. I have been evangelizing to my small businesses the importance of these various ‘blocking & tackling’ Best Practices, such as authorizing AND USING Google’s Webmaster Tools and more.

    I reiterated my thoughts using your blog as a great example. I did summarize, but mostly linking back to you. Hope that’s fine and thanks again!
    Best regards,
    Chandler Hall
    http://www.AstuteMarketing.Net

  26. Stijn Vogels,

    A reinclusion request is intended for sites that have been removed from the index for violating the webmaster guidelines. It’s not intended for other types of issues, such as the one Jason describes here.

    For crawling issues, if the bot can’t access pages (because they’re blocked with robots, return a 404, return network errors for a considerable length of time, require login, etc.), those pages will eventually fall out of the index. Can you imagine how large the index would have to be if it included every single page that ever existed?

    In those cases, it can certainly take a while for the pages to return to the index as the bot has to recrawl them (and the recrawl interval will be longer than it would be for pages that are known to be functional and full of content) and index them.

    Jason, glad the tool was helpful for you. Certainly webmaster tools can’t diagnose all problems, but it can pinpoint the most common ones — accidentally blocking with robots, network errors, misconfigured servers, hacking…

  27. Pingback: Even Google Employees Get Delisted

  28. Great post and thanks for the explanations on how bandwidth limitations and crawl errors can in fact cause ranking woes. This just reinforces the fact that many issues are algorithmic and not always a manual penalty enforced by a Google employee.

  29. When I checked for site:jasonmorrison.net on 24th Feb, your homepage wasn’t on 1st position, now it’s back.

    Lucky enough to get back to SERP faster than others.

  30. Pingback: Google también penaliza las webs de sus propios empleados

  31. The same thing has just happened to me. I was in the top three of the SERPs for the phrase ‘ishigaki japan’ and on the top page for ‘ishigaki’. My homepage isn’t being presented in the SERPS anymore for these phrases.

    The problem started on Thursday and my site is showing no sign of re-appearing. This is the second time that this has happened this month. On the first occasion the problem disappered within 24 hours – and without any changes to my site. This time I’ve hit 72 hours. Very annoying and worrying.

  33. ” Why all the errors? Here’s where I had to do some digging – I have a multi-site account with a web host. ”

    I have a few websites, and want to setup a multi-site account with NetworkSolutions. But afraid to migrate my sites to a single multi-site account – concerned that if I do they’ll fall off the Google search results (one of them is in the 2nd results return).

    Would you e-mail me and tell me how a multi-site account works? Thanks, David

  34. PS: When you e-mail me ( HonakerDAvid at hotmail) would you let me know who your hosting service is? Since google “finds” your websites, maybe I should use the same hosting service and not try to “re-invent” the wheel. Thanks again, David

  35. I am just starting my web site. I published my site about a week ago but it does not show in google search. I have typed content of the site exactly in search box with quotes around but google does not find it. I don’t know if my site never listed or just dissapeared even before I found it.

  36. We’ve not been so lucky with Webmaster tools. 2 months ago our home page disappeared from the index for our main key term. Only that one term and only on Google (it’s still page 1 on Bing, Yahoo and every other non-Google SE) and we’re still page 1 on Google for other important terms. The site hadn’t changed in months.

    The only thing we found was another site that had copied our meta title and description exactly. They corrected this 3 weeks ago and the site has been spidered twice since but we’re still nowhere. Interestingly, their domain is the 3 words of this one specific search phrase welded together.

    It almost looks as if Google has seen the duplicate tags and through its own logic decided that the other site is “right” because its domain matches the search term – penalising us in the process.

    I know Google claims to “do no evil” but when evil is done – albeit perhaps inadvertnetly – the response seems to be “tough”. How does one get action on something so crucial to the success of a business.

  37. Peter – Same thing is happening to me. Have you found a resolution yet? I held the #1 rank for over 3 years and now I don’t exist… I’m losing a lot of busines.

  38. The past 5 years when someone would put the name of my company into google my website was #1 on the first page, and for the past month its now on the 4th page, ever since 2 days ago it went down to the 9th page. I heard its happening with plenty of other sites too. I HOPE ITS ONLY A ‘TEMPORARY’ GLITCH!!!! Can someone please explain this to me, because this is having a devastating effect on my business so far. http://www.google.com/webmasters/tools/ wont work for me, because i work in a real estate company and i have my own site on the company’s site (for example: realestate.com/johnsmith), other agents in my office that also have a website on the company’s site are still on top, but my page is disappearing.

  39. Thymio – no, no resolution as yet. I tried posting similarly on Matt Cutt’s blog but the comment has been in moderation for 2 weeks now. Are you seeing the same – you home page gone entirely for only a single phrase when all other phrases still rank well and you’re still highlight placed on Bing/Yahoo? Try using Copyscape to check for duplicate content on other sites – see if it shows anything.

    I think it t must be some form of keyword filter but the problem is there is simply no guidance and no route of escalation to honest businesses and that’s what’s frustrating. The author obviously has access to the uber-tool and I was just hoping that he could provide us with at least a clue as to the problem so we can ask Google to address it (or we can address it).

    I know blogs like this attract posters complaining about things like dropping a place or three for a term. I can fully understand Google brushing that aside because the world is constantly changing and hence competitors strengths and algorithm changes will cause minor fluctuations in the resutls – it wouldn’t be a good search engine if it didn’t. But to drop out of the index entirely for only a single phrase on single page is obviously a very definite punitive action.

    Even Google must acknowledge that the software it writes must have bugs that inadvertently penalise genuine parties. Those bugs have to be found and highlighted because they genuinely affect the futures of business and people’s livelihoods. It’s not good that one company has a global monopoloy (with no regulation) on the alogorithm that can make or break the future of millions of businesses and has almost no requirement to explain itself or provide a genuine complaints process.

    All I’m asking for is a human to review the issue and to respond in some manner. A company with a global monopoloy should adequaltey staff a proper complaints process or be broken up in accordance with anit-competition laws.

  40. Just thought you’d like an update – after having our home page out of the index completely for 3 months on our key term (only that term), the home page returned today out of the blue on page 1 of Google (where it had been in January 2010)

    The bad news – there’s nothing in Webmaster tools to indicate that anything has “happened” from the Google end and there’s nothing particular that we did. We’ve published a new news story and made some small tweaks to the home page (adding a link to our Facebook page for instance) but nothing that we wouldn’t do normally. We also re-published our sitemap (but we’ve done that before since the problem without success). No small tweak would cause a page to jump from nowhere at all to page 1 anyway – it has to be a lifting of a penalty.

    So it appears as though, after 3 long months, someone at Google has finally taken notice of our complaint, seen that the penalty was indeed wrong and lifted it. However, I can’t say for sure because we’ve had no communication from Google at all throghout the entire period other than to say our re-submission request was accepted (2 months ago).

    So, I’m sorry if anyone else is having the same problem and looking for a silver bullet. Looks like you just have to plod on and hope that at some point Google decide to review complaints about faults with their engine which affect businesses, revenue and ultimately people’s livelihoods.

    Nice work Google. Do no evil? Give me a break.

  41. As others have said you are back in the search results having been looking for an answer to a problem that hit me this morning. Almost all my 400+pages have vanished from the search results even though they are still showing as indexed. 95% of terms that were ranking as high as no 3 are no longer showing although one of two terms are still ranking but bit lower. Looked on webmaster tools and can’t find an issue so for me unfortunately webmaster tools is not the answer so off to continue my search of the internet for possible answers.

  42. Update: My website came back to Google for all previously held ranking positions. I reviewed my site and web host to ensure it was not hacked or compromised. I reviewed my content per Google’s instructions and submitted a request for a ‘Site Reconsideration’. I recieved a response 9 days later stating they processed my request. The ranking returned shortly after that. Google has been a great marketing tool. I cannot complain when my clicks are free ;-).

  43. HI,

    My blog visibility has recently decreased. The keywords with which my blog was searched is now not visible. I am new to seo thing and this is really discouraging.

    regards

  44. Jason thank you for great advice with Crawl errors, what happens if your site disappeared and in Crawl errors there is No data available?

    Thank you

    Marios

  45. Thanks for the tips, and I was with you all the way until “I work for Google.” Suddenly, Kool-aid. Sure, in your case, there was a bandwidth issue that had to be resolved no matter what. But don’t average webmasters and bloggers always eat their hearts out wondering “what did I do wrong?” Google’s blanket change with Panda really threw a wrench in some people’s carefully planned SEO, especially if they relied on article sites to popularize their websites. Money and time down the drain because of a stridently arbitrary change.

  46. I’ve also had a blog site suddenly disappear from Google SERP. Two weeks ago several articles placed in the top 10 or higher (positions not SER Pages). I have been wondering why view stats suddenly declined from over 50 /day 2 weeks ago, to very few right now.

    So today a tried a search for the terms I know were getting the top places – no results anywhere (checked around 20 Google results pages in each case) then tried a search for the sub-domain name (this blog is name.wordpress.com) – still no results.

    Webmaster tools found no crawl errors, no notifications, and no malware. A look at the code showed no sign of hacking. And a site:domain check found that the pages were still indexed, and cached! There is some cross-over with content on my business site, not duplicated, just discussing the same topics from a different perspective – at the most that should only get that content downgraded, not the whole site removed

    The only problem was a red X against the site map.

    I am aware of several articles that have been re-posted, with referrals to the original articles, and a number of link-backs from other sites – outside my control. It seems a bit too much that other people using my content, or linking to it should get the site removed from SERP. In any case it has been stated often that Google can tell the date and time of the page, and determine which the original was.

    There are no links to Spam sites, most links are to my own business site, other technical sites or blogs, and a few to Wikipedia, W3C and so on.

    Also interesting, an article published earlier this month did get returned in Google SERP, but nothing from before the end of September.

    The site map was re-submitted and now has a nice green correct mark, and I submitted a request for re-inclusion today….

    I guess the only thing now is to wait and see what happens!

  48. If a website is penalized because the creator and Not the owner put into some links what actions will give solution? Its not my mistake, its the webdesigner who cheat… What can i do to save my website reputation???

  49. Having my own site start disappearing from my employer’s search engine isn’t exactly something to brag about. It’s doubly embarrassing to admit how long it took for me to notice what was going on, though I bet a lot of other site owners are in the same boat, to busy doing their day jobs to constantly check search rankings

  50. hey jason,
    wow would i be upset if my websites stoped showing up on google, it would definitely be a pain in the butt. i also wanted to thank you for all the nice tips!

    all the best!

Comments are closed.