Search Engines – Index

Search engines are the tour guides of the internet. They know and have visited most of the places, they can identify which are the good and bad places, and they can take you where you need to go with just a few words. Of course, this is an oversimplification, but you get the idea.

Technically speaking though, search engines are software algorithms which crawl through the majority of the visible portion of the World Wide Web, before cataloguing and sorting the millions of websites they found for easy retrieval. About a third of all internet traffic originates from search engines. Currently, the global search engine market is dominated by Google with a 92% share of the market, followed by Yahoo (2.67%) and Bing (2.39%).

The search engine landscape before the arrival of Google. Image courtesy of Philip Bradley

What is crawling?

Crawling is a layman’s term for the activities of software bots (robots) and spiders of search engines. The robots and spiders are deployed by search engines to navigate through the internet and collect data about websites. The data include URLs, code structure, size of webpages, types of content, keywords, backlinks, and much more. This process is called indexing, and the amount of data collected enormous.

What happens next?

These data are then stored inside a search engine index, which are then fed to the algorithms used by search engines. The algorithms would subsequently classify and rank the websites. When users make queries on search engines, the data will be retrieved and presented on search engine result pages in a human-readable sorted list structure. These queries would also be used by search algorithms to further refine their search results.

To maintain accuracy, search engines will periodically send the robots and crawlers out again to record new websites and changes to current ones.

You spend more time trying to rank! Once you better understand how the search engine result pages (SERPs) work, it is time to start ranking. It could be a good idea to seek out a trusted seo company now.