Search Engines – Robots
Spiders, crawlers and robots are soulless lines of code that perform miraculous feats every single second of every day. In spite of that, they are not godly beings, and instead, rely on instructions from humans to function. So take advantage of that and teach robots visiting your website how to behave – or just give them instructions on how to crawl your website. Here are a few useful meta tags which you could include in the <head> section or URL HTTP header of your website or webpages. Many consultants specialising in SEO services have looked into how SEO robots work and the effect their processes could have on a sites rankings.
Sadly, web crawlers don’t look like this, but they are still exceptionally powerful little lines of code. Image courtesy of Pixabay
To be indexed or not to be indexed
The noindex meta tag tells crawlers whether you would like particular websites or webpages to be indexed or not. Websites with databases or private information will definitely need to use this code. Forums or media-heavy websites may also prefer their content to be excluded from indexes. The tag is also useful if you would like to exclude specific pages on your site from being indexed.
Protip: The ‘index’ tag is not required since bots will crawl websites on default.
I will follow you, wherever you may go
No, crawlers don’t listen to Ricky Nelson, but they will listen to instructions on which outbound links they may follow. Outbound links bleed link authority, which is a commodity not many webmasters can afford to lose. So a simple nofollow attribute will prevent crawlers from following all the outbound links emanating from your website.
Don’t keep all brochures
Supermarkets and retailers hate it when their customers bring old brochures or flyers with outdated pricing. Similarly, if you’re running an e-commerce site, caches of old pricing lying around for everyone to see is the last thing you want. So gently, but firmly, tell search engines, ‘noarchive’! This will prevent them from making cache copies of your website. This will also prevent archived websites from stories complete copies of your website on their servers.