Taurus Robots
taurus-media/module-robots
Automatically adds Disallow rules to robots.txt for all filterable product attributes, preventing search engines from crawling layered navigation filter combinations that cause duplicate content.
Build Tests
Code Quality
Tested on Magento 2.4.8-p4
Recent Test History
Each release is tested against the latest Magento version at that time.
Top Contributors
View Leaderboard
Share This Module's Status
README
Loaded from GitHubTaurus Robots
A Magento 2 module that automatically adds filterable product attributes to the robots.txt file to prevent search engines from crawling layered navigation filter combinations, which can lead to duplicate content issues.
Description
The Taurus_Robots module extends the default Magento 2 Robots functionality. It identifies all product attributes configured to be "Use in Layered Navigation" (filterable) and adds a Disallow rule for each of them under the User-agent: * section.
For example, if you have a filterable attribute color, it will add:
Disallow: /*color=*
Features
- Automatically detects filterable product attributes.
- Dynamically injects
Disallowrules into the generatedrobots.txt. - Sorts rules alphabetically for better readability.
- Ensures rules are placed under the global
User-agent: *block.
Installation
composer require taurus-media/module-robots
bin/magento module:enable Taurus_Robots
bin/magento setup:upgrade
bin/magento setup:static-content:deploy
This content is fetched directly from the module's GitHub repository. We are not the authors of this content and take no responsibility for its accuracy, completeness, or any consequences arising from its use.