Indeks
Robot.Text file allows you to manage which pages would you like to crawl or not to crawl by spiders or GoogleBot. Some website owner is not aware of this and maybe this is one of the main reason why their website pages are not indexed by Search Engines. But how can you add and edit your robot.text file? I will list down mine how to add and edit robot.text file for WordPress users here.
Create A Robots.Text File Manually
1. Open Notepad
2. Add the following
User-agent: *
Disallow: /
Note: right after the forward slash add the particular page or folder that you want to disallowed or not to crawled by GoogleBot for indexing. You can add more lines for Disallow if you have multiple that you want to disallowed.
3. Save the file in your computer add use a file name robots.txt (do not misspelled it)
4. Upload the file to your main directory using FTP or File Manger in your cPanel
5. Save it and your done
Use A WordPress Plugin
There is an available plugin that you can use to edit your robots.txt file. The plugin is available here and free to download.
1. Install WP Robot Txt plugin. You can manually upload it or you can install directly to your plugins directory.
2. Activate the plugin from your plugins directory.
3. Once activated, go under your Dashboard Settings and click Readings.
4. In this section you can edit your robots.text file.
5. Click Save.
As I have mentioned above you can add more lines for which pages you want to disallow by GoogleBot. However, it still depends on them if your pages is going to be index or not. That why creating quality content or pages is recommend because even if you allow the spider to crawl your pages but it doesn't have a good content, then its still doesn't work out for your indexing.
Create A Robots.Text File Manually
1. Open Notepad
2. Add the following
User-agent: *
Disallow: /
Note: right after the forward slash add the particular page or folder that you want to disallowed or not to crawled by GoogleBot for indexing. You can add more lines for Disallow if you have multiple that you want to disallowed.
3. Save the file in your computer add use a file name robots.txt (do not misspelled it)
4. Upload the file to your main directory using FTP or File Manger in your cPanel
5. Save it and your done
Use A WordPress Plugin
There is an available plugin that you can use to edit your robots.txt file. The plugin is available here and free to download.
1. Install WP Robot Txt plugin. You can manually upload it or you can install directly to your plugins directory.
2. Activate the plugin from your plugins directory.
3. Once activated, go under your Dashboard Settings and click Readings.
4. In this section you can edit your robots.text file.
5. Click Save.
As I have mentioned above you can add more lines for which pages you want to disallow by GoogleBot. However, it still depends on them if your pages is going to be index or not. That why creating quality content or pages is recommend because even if you allow the spider to crawl your pages but it doesn't have a good content, then its still doesn't work out for your indexing.
