Does reading this title give you a mini panic attack?
Having gone through exactly what the title suggests, I can guarantee that your fear is fully justified.
If you want to relive my nightmare with me — maybe equal parts catharsis and SEO study — let’s go through the events chronologically.
Are you ready?
04.08.2019
It was a Sunday morning. I drank my coffee and tweaked our SEO tools as usual, not expecting anything. Then… BAMM!
What. That. Hell?
As SEOs, we are all used to seeing natural fluctuations in rankings. fluctuations, not disappearances.
Step 1: rejection
Immediately my thoughts go to one place: It’s a mistake. So I jumped into some other tools to confirm whether or not Ahrefs was going insane.
Google Analytics also showed a corresponding drop in traffic, confirming that something was definitely afoot. As an SEO, I naturally assumed the worst…
Step 2: Algo Panic
Algorithm update. Please please don’t let it be an algo update.
I jumped into Barracuda’s Panguin Tool to see if our issue coincided with a confirmed update.
No updates. phew
Step 3: Diagnosis
Nobody ever thinks straight when their reptilian brain is busy. You panic, think irrationally, and make bad decisions. zero cold.
I’ve finally gathered some presence of mind to think clearly about what happened: it’s highly unusual for keyword rankings to disappear entirely. It has to be technical.
It has to be an index.
A quick Google search for the pages that had lost keyword rankings confirmed that the pages were indeed gone. Search Console reported the same:
Note the warning below:
No: ‘noindex’ detected in ‘robots’ meta tag
Now we got somewhere. Next it was time to confirm this finding in the source code.
Our pages have been flagged for de-indexing. But how many pages have actually been de-indexed so far?
Step 4: Assessment of the damage
All of them. After sending our developer some frantic notes, he confirmed that a sprint deployed on Thursday evening (August 1, 2019), almost three days earlier, had accidentally made the code live on every page.
But has the entire site been de-indexed?
This is highly unlikely, as it would have required Google to crawl every page on the site within three days to find the “noindex” markup. Search Console would be of no help in this regard as their data is always laggy and the changes may never pick up before they are fixed.
Even looking back now, we see that Search Console only picked up a maximum of 249 affected pages out of over 8,000 indexed. Which is impossible considering our search presence was reduced by a third a full week after the incident was resolved.
Note: I’ll never be sure how many pages were fully de-indexed in Google, but what I do know is that EVERY page had “noindex” markup, and I vaguely remember reading “site:braton.com” googled and saw about an eighth of them indexed our pages. I wish I had a screenshot, of course. We’re sorry.
Step 1: Fix the problem
Once the issue was identified, our developer rolled back the update and brought the site live as it was before the “noindex” markup. Next came the issue of re-indexing our content.
Step 2: Re-crawl the website as soon as possible
I deleted the old sitemap, created a new one and re-uploaded it to Search Console. I also grabbed most of our core products’ landing pages and manually requested a reindex (which I don’t think really works since the last SC update).
Step 3: Wait
There was nothing we could do at the moment but wait. There were so many questions:
- Will the pages rank for the same keywords as before?
- Will they rank in the same positions?
- Will Google somehow “penalize” pages for short disappearances?
Only time would tell.
August 8, 2019 (one week) – 33% drop in search exposure
In assessing the damage, I will use the date the buggy code was fully deployed and released to the live pages (2 August) as ground zero. So the first measurement will be completed for seven days, from August 2nd to August 8th.
Search Console would probably give me the best indication of how badly our search presence has suffered.
We had lost about 33.2% of our search traffic. Ouch.
Luckily, that would mark the culmination of the damage we experienced throughout the ordeal.
August 15, 2019 (two weeks) – 23% less traffic
During that time, I kept two things in mind: search traffic and indexed pages. Even though I resubmitted my sitemap and fetched pages manually in Search Console, many pages were still not indexed – even the most important landing pages. This will become an issue throughout this timeline.
Due to our remaining unindexed pages, our traffic was still suffering.
Two weeks after the incident, we were still down 8% and our revenue-generating conversions were falling with traffic (despite increased conversion rates).
August 22, 2019 (three weeks) – 13% less traffic
Our pages were still indexing slowly. Painfully slow as I watched my commercial goals fall through the floor.
At least it was clear that our search presence was recovering. but how It was of particular interest to me to recover.
Have all pages been re-indexed but with reduced search presence?
Was only a subset of pages re-indexed with fully restored search presence?
To answer this question, I looked at pages that were individually de-indexed and re-indexed. Here is an example of one of those pages:
Here’s an example of a page that’s been deindexed for a much shorter period of time:
In every case I could find, each page was fully restored to its original search presence. So it didn’t seem to be a question of whether pages are restored or not, but rather when pages are re-indexed.
Speaking of which, Search Console has a new feature that “validates” bad pages. I started this process on August 26th. After that point, SC slowly re-crawled these pages (I assume) at a rate of about 10 pages per week. Is that even faster than a normally planned crawl? Do these tools in SC do anything at all?
What I did know for sure was that a number of pages were still deindexed after three weeks, including commercial landing pages that I had been relying on to drive traffic. More on that later.
August 29, 2019 (four weeks) – 9% less traffic
At this point I was very frustrated because there was only about 150 pages left to reindex and no matter how many times I checked Search Console and requested reindexing, it wouldn’t work.
These pages were able to be fully indexed (as reported by SC URL Inspection) but were not crawled. As a result, almost a month later we were still 9% down from baseline.
A particular page simply refused to be reindexed. This was a high commercial value product page that I relied on for conversions.
In my attempts to force a reindex, I’ve tried the following:
- URL inspection and indexing request (15 times per month).
- Update the release date and then request indexing.
- Updating the content and publication date, and then requesting indexing.
- Resubmitting sitemaps to SC.
Nothing worked. This page would not be reindexed. The same goes for over a hundred other URLs with less commercial impact.
Note: This page would not be re-indexed until October 1st, two full months after it was de-indexed.
By the way, here’s our overall recovery progress after four weeks:
September 5, 2019 (five weeks) – 10.4% less traffic
The Great Plateau. At this point we had re-indexed all of our pages except for the 150 or so pages that were said to have been “validated”.
They weren’t. And they weren’t re-crawled either.
It seemed like we were likely to make a full recovery, but the timing was in Google’s hands and there was nothing I could do about it.
September 12, 2019 (six weeks) – 5.3% increase in traffic
It took us about six weeks to fully restore our traffic.
But in truth we still hadn’t fully recovered our traffic as some content was overperforming and overcompensating for a number of pages that weren’t indexed yet. Specifically our product page which would not be indexed for another ~2.5 weeks.
Overall, our search presence recovered after six weeks. But our content was not fully re-indexed until more than eight weeks after the issue was resolved.
Conclusion
For starters, you should definitely not de-index your site by accident, as an experiment, or for any other reason. It stings. I estimate that we deleted about 12% of all organic traffic, which is an equally proportional drop in commercial conversions.
What did we learn??
Once the pages were re-indexed, they were fully restored in terms of search visibility. The biggest problem was re-indexing them.
Some main questions we answered with this random experiment:
Have we recovered?
Yes we have fully recovered and all urls seem to be getting the same search visibility.
How long did it take?
Search visibility returned to baseline after six weeks. All pages re-indexed after about eight to nine weeks.
source
