Search Engines – Architecture

The information architecture, code structure and site navigation of your website not only help with maintenance, user experience and operational efficiency, they also make it easier for robots and spiders to crawl through your website. If robots cannot roam free on your website, or certain sections are blocked, they will leave. They will come back and try again in the future, but your website ranking would have been affected by then. So how do you ensure bots and spiders feel welcomed at your site?

Sitemaps play an important role in SEO. Image courtesy of Pixabay

Use a sitemap

A sitemap is a hierarchical textual layout of the content on your website. It provides web crawlers with instructions on where to go, and where it can’t go. Since crawlers no longer need to waste time following all internal links from the homepage, they will work faster and more efficiently – and return the favour by giving websites bonus ranking points. They will also consume less resources from the website’s server – which is important for really large websites. Using a sitemap also ensures that bots won’t crawl duplicate content, which Google doesn’t particularly like.

Minimising crawl errors

Robots sometimes encounter errors while crawling through websites. The simplest way to find out more about these errors is by going to a search engine’s webmasters tool. Some of the most common errors include:

• 4xx error codes: This means a crawler followed a non-existent URL. This URL may belong to a deleted page, or the URL may have been changed. Webmasters should write codes or install a plugin for a permanent 301 redirect to resolve the issue. In terms of SEO, a 301 redirect will preserve the link authority of the old page and transfer it to the new page.

• 5xx error codes: This is when a crawler is unable to access a website due to server-related issues. This could signal underlying issues with your hosting provider, some please spend some time to investigate the issue.

Have an expert review your site architecture

Site architecture is a massive topic and issues and opportunities here can be quite complex. This is especially true when looking at a larger site (some sites have tens of thousands, hundreds of thousands or even millions of URLs indexed after all!). It could be a good idea to have a search engine optimisation company take a look at your site in greater detail to find any potential issues.