posted under blog-tips by admin

Cuil Search Engine, The Self Claimed “Google Killer” is not going to kill Google, instead Cuil will kill your sites. Yes, unfortunately Cuil is not as Cool as it’s name.

There have been so many bad negative story about Cuil. The New Google Killer Serch Engine was said killing so many websites while indexing them.

It’s Twiceler, the Cuil Crawler Bot. Seems Twiceler has still so many bugs on it’s code. Instead of indexing our exists pages, Cuil is creating a stupid address for files / pages that never exists on our sites.

Jazzychad reported that Cuil is trying to index a really lot of files or pages that have never been existing on his sites. And this is really lot of pages that end up with 404 pages.

That is a really bad thing because it will consume so many bandwith and another resources of our sites. For all Media Temple customers, it will just be a nightmare.

As MT Customer know, 404 page will eat a lot of their GPU resource, which the price is so expensive. Each time the site generate 404 pages, their GPU resource will decreased. If finally they run out their GPU, they will be charged for so many money. That’s why this will be a real nightmare for Media Temple’s Users.

And now, the only way we can do to handle this nightmare is just by blocking Twiceler, the Cuil bot of indexing our websites.

How to block Twiceler from indexing our website?

Just put this 2 lines of code into your robots.txt

User-agent: twiceler
Disallow: /

That’s it. Now you can save your bad ass from being attack by Cuil. btw, no wonder Cuil claimed that they have 3 times bigger then Google indexed pages. Cuil has even indexing our pages that don’t even exists.