How to create and implement robots txt file in your website

Syntax

User-agent: *
Allow: /
Sitemap: https://www.----.com/sitemap.xml

Disallow :
User-agent: [user-agent name]- Links , Images
Disallow: [URL string not to be crawled],Image Folder

Save File Robots.txt
how to check robots.txt - mydomain.com/robots.txt

Top

Visitor Counter =