Google says the best way to hide a website from search results is to use a password, but you may want to consider other options.
This topic is highlighted light in the latest installment of the Ask Googlebot video series on YouTube.
Google's John Mueller answers a question asking how to prevent content from being indexed in search, and if it is something websites are allowed to do.
“In short, yes you can,” Mueller says.
There are three wayshide a website in search results:
- Use a password
- Block crawling
- Block indexing
Websites can either turn off indexing completely or be indexed and hide Googlebot content with a password.
Blocking content from Googlebot is not against webmaster guidelines, as long as it is blocked for users at th at the same time.
For example, if the site is protected by password when crawled by Googlebot, it should also be password protected for users.
Alternatively, the site should have guidelines in place to prevent Googlebot from crawling or index the site.
You may experience problems if your website offers different content to Googlebot than to users.
This is what it is called "camouflage " and goes against Google guidelines.
WithWith that distinction made, here are some good ways to hide search engine content.
3 ways to hide search engine content
1. Password protection
Locking down a website with a password is often the best approach if you want to keep your site private.
A password will ensure that neither engines search or random web users will not be able to see your content.
This is a common practice for developing websites. Publishing the website live is an easy way to share work in progress with customers, while also preventing Google from accessing a website that is not yet ready to be viewed.
2. Block crawling
Another way to prevent Googlebot from accessing your site is to block crawling. This is done with the robots.txt file.
With this method people can access your sitewith a direct link, but it won't be picked up by "well-behaved" search engines.
This isn't the best option, Mueller says, because search engines can still index the 'website address without accessing content.
This is rare for this to happen, but it is a possibility you should be aware of.
3 . Block indexing
The third and final option is to block the indexing of your website.
To do this, you add a
A noindex tag tells search engines not to index this page before after crawling it.
Users don't see the tag
Mueller's Final Thoughts
Mueller concludes the video by saying that Google's main recommendation is to go the password route:
See the full video below:
Featured image: Screenshot: YouTube.com/GoogleSearchCentral, November 2021.