How search engines work – crawling, indexing and ranking

Welcome to chapter 3 of AreteBlog’s SEO guide.

In this chapter we’ll look into how search engines actually work.

Search engines exist to discover, understand, and organize the content over the internet for the purpose of providing the most relevant content to the users.

How does Search Engines work?

Search engines have three fundamental functions:

• Crawling – Comb the internet for content, follow through the links, looking for the code for each URL they find.
• Indexing – The page analyzed for meaning and content by Googlebot (Google crawler) is stored in Google index. An indexed page is then ready to appear upon relevant queries made.
• Ranking – Provide the most relevant search results to queries. Such that the search results appear from most relevant to least relevant.

Search Engine Crawling

Crawling is the process of exploring new and updated content using bots known as crawlers or spiders.

Content varies from web pages, texts, images, pdfs and videos. The crawlers start by enticing a few web pages and then follow the links on those web pages to find new URLs.Hence, in this way web crawlers find new content and add that content to their index.

Search Engine Indexing

Googlebots save crawled data in an index called Caffeine after they have crawled it. Indexed pages can now show up in search results with the condition that they follow Google webmaster’s guidelines.

Search Engine Ranking

Search engines work to serve the user with the best matches to their searches. When someone executes an online search, search engines scour their index and display the most relevant content to the users.

Tell search engines how to crawl your site

Use GoogleSearchConsole or “site:domain.com” advanced search operator to find out whether your are indexed or not. 

If you find out that some of your important pages are yet not indexed, or any of your unimportant pages are indexed, there are some optimizations you can opt for.

You can use these optimizations to better direct Googlebot how to crawl your content.

Telling Googlebots how to crawl your pages can give you a better control of how your pages get indexed.

How to get indexed by Google

Found that some of your pages or your entire website is not indexed?

Here’s what you need to do:

1. Go to Google Search Console
2. Click on URL inspection tool
3. Enter URL of the page or site you want to get indexed
4. Wait for the tool to check your provided URL
5. Click the “Request Indexing” button

Doing so is a good practice whenever you publish/post something new, such that you let Google know that you have added something new. However, this does not solve bottom line issues.

Here are a few tips to solve such underlying problems:

Remove Crawl blocks on your Robot.txt file

One of the prime reasons that your site or page is not indexed by Google is that it might be crawl blocked by a robots.txt file.

To check whether its the case or not, go to www.yourdomain.com/robots.txt

Look for either of the two snippet codes:

User-agent: Googlebot

Disallow: /

---

User-agent: *

Disallow: /

Any of the above means that they are not allowed to crawl any pages on your site. In that case, remove them. Its simple!

Remove rogue noindex tags

You might want Google not to crawl some of your pages. This will happen if you tell Googlebots not to!

There are two methods to do so:

1. Meta tags

Pages with either of these meta tags in their <heads> won’t be indexed by Google:

<meta name=“robots” content=“noindex”>

<meta name=“googlebot” content=“noindex”>

2. X-Robots-tags

Crawlers also do not go for the X‑Robots-Tag HTTP response header. You can implement this using a server-side scripting language like PHP, or in your .htaccess file, or by changing your server configuration.

You can use the URL inspection tool in the Google Search Console to check whether the page is blocked or not. Just enter your URL, then look for the “Indexing allowed? No: ‘noindex’ detected in ‘X‑Robots-Tag’ http header”

Include the page in your Sitemap

Sitemap helps Google identify which pages in your site are important and which are not. Also, how often a page must be re-crawled.

Although Google still crawls your pages if they are not in your site map, still it is a good practice to make.

To check if a page is in your sitemap, use the URL inspection tool in Search Console. If you see the “URL is not on Google” error and “Sitemap: N/A,” then it isn’t in your sitemap or indexed.

Fix no follow internal links

Nofollow links are links with a rel=“nofollow” tag. They prevent the transfer of PageRank to the destination URL. Google also doesn’t crawl nofollow links.

In short, you should make sure that all internal links to indexable pages are followed.

Build high quality backlinks

Backlinks are the source for you to build trust in the eyes of Google. These tell Google that your page is valuable. Of Course if someone is linking to it, it must hold some value in it. 

Pages with high quality backlinks are likely to be crawled and re-crawled more faster then those with no backlinks.

Now you know about how Google crawls, indexes and ranks your pages. Three cheers!

Let’s Continue the journey and hop into Chapter 4 of this all-inclusive SEO guide.