How does web crawling work?

Writing and publishing compelling content for your website is not the end of the story when it comes to promoting your business. The major problem – as any businessman knows – is promotion and getting people to notice you exist. While out in the “real world” this involves an advertising budget and carefully crafted campaigns the show potential customers your worth, on the internet it’s a completely different business.

We all know the old advertising cliché that “content is king”, and it remains true to this day. However, the content only remains king on the internet as long as it is seen, and to make sure that’s the case, it has to be picked up by the leading search engines and indexed in such a way that potential customers see your site as high up the results as humanly possible. This means making sure that the search engine web crawlers find your site and index its contents that way you want it to be.

Before a search engine can tell you where a page is, it has to go out and find it. Back in the early days of the internet, it was down to site owners to submit their pages to search engines for indexing, and they’d get round to it – eventually. It didn’t take long for these companies to develop special tools which we call “spiders” which search the web building up lists of the words found on websites. They do this millions of times each day, in order to keep up with the hugely dynamic nature of today’s web.

Spiders start off at popular pages, reading the content and following links that it finds. Eventually it moves on to other sites, doing the same, moving in an organic process from site to site, building up lists of key words. Your site will become one of these pages, indexed into the search engine’s huge list of words, contextualised alongside the pages to which it is linked.

Web spiders are relatively simple programs, and it soon became apparent that they could be tricked by clever use of key words and the seeding of huge numbers of incoming links seeded across the internet. To this end, the big search engines such as Google arm their indexing programs with tools to detect attempts to fool the system and penalise those that they think are operating unfairly.

This is where SEO specialists come in. They know exactly what the likes of Google are looking for, which tactics move a site nearer the top of the index, and which ones are likely to result in a penalty. As Google’s system is a closely guarded secret, and changes often, SEO companies have to stay on their toes and be ready to react when things change.

As long as you remember that web bots are looking at your site all the time, it should be a relatively straightforward process. Keep your keywords up to date, make sure there aren’t any dead links, and ensure that promotional links don’t penalise your ranking. Simple really…

This entry was posted in General SEO Stuff. Bookmark the permalink.