Meta robot tags are directives that search engines will always respect. Adding these robots tags can make the indexation of your website easier.
There is a difference between directives and suggestions.
Meta robots tags or robots.txt files are directives and will always be obeyed. ** Canonical tags ** are recommendations that Google can decide to obey or not.
There are many options when it comes to page-level meta tags, but the following are examples commonly associated with SEO:
<meta name="robots" content="noindex,nofollow" />
The robots tag is probably the most common tag you will see. By default it will have the value index,follow so it does not need to specified, all is also a valid alternative version:
<meta name="robots" content="all" />
By setting a robots tag to noindex,nofollow as in the example above, it will indicate to search engines:
- noindex: To not show this page in search results. Omitting
noindexwill indicate the page can be indexed and shown in search results. When building a website, you might not want to index certain pages. Common use cases include settings pages, internal search pages, policies, and more. - nofollow: To not follow links on this page. Omitting this will allow robots to crawl and follow links on this page. Links found on other pages may enable crawling, so omitting
nofollowwill allow Google to crawl and follow links on this page. Links found on other pages may enable crawling, so if appears in pages and , and has a robots tag, but doesn't, Google may decide to crawl the link.
