So, we had a bit of a crazy thing happen to us at work (Convurgency) this week. We have a very advanced website – Flash interface for people with full browsers, but then the very same content has a basic HTML rendering for mobile devices and search spiders.
We noticed some incredibly strange errors in how Google and Bing were indexing our site this week. Both went from seeing all the code and content to seeing nothing at all, a virtually blank page.
Upon deeper investigation, we figured out why. Any of you out there performing mobile/spider rendering for your site should be aware of these:
- Bing is no longer using the MSNbot to do its spidering, but now has finally launched (without telling anyone) their own bingbot
- Google recently launched their Site Preview tool. Because this is more of a browser-like tool, and not a real spider, it doesn't carry 'bot' in its user agent. The user agent for reference at this time is: Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13
So, because our spider detection scripts didn't know about either of these new bots, they were indexing our Flash site, and seeing absolutely nothing at all. We have now resolved this of course, and I recommend you do the same!
However, even though it's all resolved, I still find it extremely odd that the Google Preview Tool actually affected the indexed content of the website. What I mean is that the page code/content seen by the preview toolÂ supersededÂ what the googlebot had indexed and previously showed in SERPs. I think this is a huge mistake on Google's part, and I hope they resolve it soon!