The "Create Project" pop-up in the Site Audit tool

[sharminakter]

Errors in the robots.txt section mean that Google was unable to crawl your site for a technical reason. Robots.txt fetch tells you the failure rate that bots experienced when crawling your robots.txt file. While your site doesn't need a robots.txt file, it should return a 200 or 404 response when requested. If Googlebot has a connection problem, such as an HTTP 503 status , it will stop crawling your site.

DNS resolution tells you when your DNS server didn't recognize your hostname or didn't respond during a crawl. If you encounter a problem here, contact your registrar to verify that your site is configured correctly and that your server has an active Internet connection.

Server connectivity tells you when your server has not responded or has not provided the complete response. If y

ou see spikes or consistent connectivity issues here, you may want to ask

your provider to increase capacity or address availability issues.
How to check the crawlability of your website with Site Audit?
Use the Site Audit tool to have SemrushBots crawl nigeria phone data your website and check its health.

The tool flags over 140 issues, including duplicate content and broken links , that you can identify and fix.

Here's what you need to do:

Finalize basic settings
Go to the Site Audit tool and create a new project if this is your first audit.

Then select your site's audit settings.

Site Audit Settings Page
The crawl scope allows you to refine your search.

You can also adjust the number of URLs to be checked per audit depending on your subscription level and how often you will inspect the website.