Mondoze Knowledge Base

Search our articles or browse by category below

I’m getting a crawl error on Cloudflare cdn-cgi files.

Last modified: October 5, 2022
You are here:
Estimated reading time: 1 min

Crawl Error on Cloudflare cdn-cgi Files

If you are getting a crawl error in Google Webmaster Tools for a cdn-cgi file, please disallow Google from crawling that with a rule in robots.txt for: 

Disallow: /cdn-cgi/

An additional option using WordPress:

Solution provided by Ben

You can make modifications in your /wp-includes/functions.php file. to create an additional Disallow rule. A sample of what it would look like is below:

function do_robots() {
header( ‘Content-Type: text/plain; charset=utf-8’ );

do_action( ‘do_robotstxt’ );

$output = “User-agent: *\n”;
$public = get_option( ‘blog_public’ );
if ( ‘0’ == $public ) {
$output .= “Disallow: /\n”;
} else {
$site_url = parse_url( site_url() );
$path = ( !empty( $site_url[‘path’] ) ) ? $site_url[‘path’] : ”;
$output .= “Disallow: $path/wp-admin/\n”;
$output .= “Disallow: $path/wp-includes/\n”;
$output .= “Disallow: $path/cdn-cgi/\n”;
}

Note: This won’t impact the actual crawl for your site at all. The error for /cdn-cgi/ is also not going to impact your rankings at all. Google is attempting to crawl a Cloudflare resource. One that doesn’t need to be crawl, and they possibly can’t read the DOM or JavaScript in that resource.

Was this article helpful?
Dislike 0
Views: 2053