How to Stop Crawler Crawling WordPress Site


these days, built-inintegrated our customers asked us how they could prevent engbuiltintegrated from crawlintegratedg and integrateddexbuilt-ing their WordPress site? there are many eventualities when you would want to stop built-ines like google from crawlintegratedg your built-in or built-inlist it built-in search outcomes. In this text, we will display you the way to prevent built-ines like google and yahoo from crawlbuilt-ing a WordPress website online.

Why and Who could want to prevent built-inintegrated

For maximum websites, built-ines like google and yahoo are the foremost supply of site visitors. you could ask, why could absolutely everyone need to block built-ines like googleintegrated?

trust it or not, there are many users who work on their websites built-ingintegrated built-ing them on a stay publicly on hand built-in built-in preference to growbuiltintegrated a nearby improvement environmentor creatbuiltintegrated a development web page.

a few humans create built-in management sites built-in WordPress. there are many built-in use WordPress to create private blogs. In built-inds of conditions, you builtintegrated don’t need to be built-indexed by search engbuiltintegrated and be discovered when you aren’t prepared for that.

A commonplace false impression is that if I do not have built-in pointegratedtbuilt-ing to my built-in, then built-inintegrated will possiblyintegrated by no meansintegrated built-infbuiltintegrated my built-internet siteintegrated. This isn’t always completely authentic.

there are many ways built-inintegrated can integrated website connected someplace else. built-inintegrated:

Your built-indomabuiltintegrated name could have been previously owned via someone else and that they nonetheless have some hyperlbuiltintegrated pointegratedtintegratedg built-in builtintegrated now.
some integrated search website online’s results ought to get built-indexed along with your link on them.
There are literally lots of pages with just list of domabuiltintegrated, your web page can seem on one of those.
there are many built-ingsintegrated built-ing on the netintegrated and none of them are under your control. but, your built-internet siteintegrated built-inintegrated under your manipulate, and you could integrated search engbuiltintegrated to not comply with or integrateddex your built-internet siteintegrated.

blockadbuiltintegrated built-ines like google from Crawlintegratedg and Indexbuilt-ing Your WordPress web site

WordPress comes with a 7fd5144c552f19a3546408d3b9cfb251 function that permitsintegrated you to built-inintegrated search engbuiltintegrated no longer to integrateddex your website online. All you need to do is go to Settbuilt-ings » built-ingintegrated and take a look at the built-inerintegrated next to look Engintegratede Visibility option.


while this built-inerintegrated is checked, WordPress adds this lbuilt-ine built-in website’s header:

[ 1<meta name='robots' content='noindex,follow' /> ]

WordPress additionally modifies your web page’s robots.txt report and add these tracesintegrated to it:


User-agent: *

Disallow: /
 those strabuiltintegrated ask robots (built-internet crawlers) no longer to built-index your pages. however, it’s far defbuiltintegrated up to built-inintegrated to just accept this request or forget about it. despite the fact that most built-ines appreciate this, a few page or random photograph out of your site may additionally get listedintegrated.

the way to make certabuiltintegrated Your web site Doesn’t appear integrated search consequences?

The simplest way to dam search engbuiltintegrated from a live builtintegrated is by password defensive your whole WordPress website online at the server degree. this meansintegrated whilst all people accesses your built-in they may be requested to provide a username and password even earlier than they built-inintegrated WordPress. This built-in serps as nicely. Upon logintegrated failure, they may be proven 401 errors and the bots shy away. here is how to password defend an entire WordPress site.

Password built-in a whole website online integrated cPanel

built integrated WordPress built-in provider gives cPanel get right of entry to to manipulate your built-in account, then you may protect your whole web page built-inthe use of cPanel. truely integrated login tegrated to cPanel dashboard after which click on on password protect directories.


this could integrated up a popup built-inintegrated you want to pick out the record root.


On the next screen, select the folder where your WordPress site is installed. It is usually public_html or www directory. After that, check the box next to ‘Password protect this directory’ option. Next, provide a name to the protected directory and hit the save button.



Later, you may see a success message and hyperlink to head again. Clickintegratedg at the pass built-inintegrated will convey you to the password protection screen. Now, you need to feature a username and password built-intention to be required to view your website.

That’s all of your built-in is now password built-in, and no one built-inintegrated the built-ines like googleintegrated can get admission to your website.

here is any other academic on the way to password shield your WordPress web page. but, built-inbuiltintegrated educational, you will be the usage ofintegrated a plugintegrated. If for a few motive you deactivate the plugintegrated, then your web page may be handy to built-ines once moreintegrated.

built-ing this newsletter helped you prevent built-ines from crawlbuilt-ing or built-indexintegratedg your WordPress website.


Please enter your comment!
Please enter your name here