Добрый день!
У меня старый сайт на редкой CMS (S.Builder) и им много кто "занимался". В итоге огромное количество дублей страниц/разделов.
сайт http://dreamboutique.ru
Я так понял есть две тактики для устранения индексации данных дублей через robots.txt:
1. Всем поисковикам все разрешить и запрещать дубли. Но, постоянно появляются новые, другие дублирующие разделы каталога. (этот вариант используется сейчас http://dreamboutique.ru/robots.txt)
2. Всем все запретить, и писать allow только уникальным страницам/разделам каталога.
И, при добавлении новой страницы или нового раздела - прописывать очередной allow.
Этот вариант точно устранит все дубли. Беспокоит одно. Не может ли этот вариант как-то навредить. и Нужно ли отдельно все allow прописывать для User-Agent:Yandex, User-Agent:Googlebot, User-Agent:Googlebot*.
Пример (ну в общем почти карта сайта):
User-agent: * Disallow: / Allow: /catalog/evening-dresses/$ Allow: /catalog/evening-dresses/vechernie-platya-na-vypusknoj/$ Allow: /catalog/evening-dresses/korotkie-vechernie-platya/$ Allow: /catalog/evening-dresses/novinki-vechernih-platiev/$ Allow: /catalog/evening-dresses/vechernie-platya-na-korporativ/$ Allow: /catalog/evening-dresses/speredi-korotkie-vechernie-platia/$ Allow: /catalog/evening-dresses/klubnye-vechernie-platya/$ Allow: /catalog/evening-dresses/koktejlnye-vechernie-platya/$ Allow: /catalog/evening-dresses/nedorogie-vechernie-platya/$ Allow: /catalog/evening-dresses/dlinnye-vechernie-platya/$ Allow: /catalog/evening-dresses/index.php$ Allow: /catalog/evening-dresses/vechernie-platya-na-vypusknoj/index.php$ Allow: /catalog/evening-dresses/korotkie-vechernie-platya/index.php$ Allow: /catalog/evening-dresses/novinki-vechernih-platiev/index.php$ Allow: /catalog/evening-dresses/vechernie-platya-na-korporativ/index.php$ Allow: /catalog/evening-dresses/speredi-korotkie-vechernie-platia/index.php$ Allow: /catalog/evening-dresses/klubnye-vechernie-platya/index.php$ Allow: /catalog/evening-dresses/koktejlnye-vechernie-platya/index.php$ Allow: /catalog/evening-dresses/nedorogie-vechernie-platya/index.php$ Allow: /catalog/evening-dresses/dlinnye-vechernie-platya/index.php$ Allow: /catalog/evening-dresses/?page_4=* Allow: /catalog/evening-dresses/view/evening-dresses/* Allow: /catalog/wedding-dresses/grecheskie-svadebnye-platya/$ Allow: /catalog/wedding-dresses/svadebnye-platya-dlya-beremennyh/$ Allow: /catalog/wedding-dresses/svadebnye-platya-rybka-rusalka/$ Allow: /catalog/wedding-dresses/svadebnye-platya-a-siluet/$ Allow: /catalog/wedding-dresses/kruzhevnye-svadebnye-platya/$ Allow: /catalog/wedding-dresses/pyshnye-svadebnye-platya/$ Allow: /catalog/wedding-dresses/korotkie-svadebnie-platia/$ Allow: /catalog/wedding-dresses/zakrytye-svadebnie-platya/$ Allow: /catalog/wedding-dresses/svadebnye-platya-s-volanami/$ Allow: /catalog/wedding-dresses/svadebnye-platya-s-zanizhennoy-taliey/$ Allow: /catalog/wedding-dresses/svadebnye-platya-s-vysokoj-taliej/$ Allow: /catalog/wedding-dresses/assimetrichnye-svadebnye-platya/$ Allow: /catalog/wedding-dresses/rasprodazha-svadebnyh-platjev/$ Allow: /catalog/wedding-dresses/nedorogie-svadebnye-platya/$ Allow: /catalog/wedding-dresses/grecheskie-svadebnye-platya/index.php$ Allow: /catalog/wedding-dresses/svadebnye-platya-dlya-beremennyh/index.php$ Allow: /catalog/wedding-dresses/svadebnye-platya-rybka-rusalka/index.php$ Allow: /catalog/wedding-dresses/svadebnye-platya-a-siluet/index.php$ Allow: /catalog/wedding-dresses/kruzhevnye-svadebnye-platya/index.php$ Allow: /catalog/wedding-dresses/pyshnye-svadebnye-platya/index.php$ Allow: /catalog/wedding-dresses/korotkie-svadebnie-platia/index.php$ Allow: /catalog/wedding-dresses/zakrytye-svadebnie-platya/index.php$ Allow: /catalog/wedding-dresses/svadebnye-platya-s-volanami/index.php$ Allow: /catalog/wedding-dresses/svadebnye-platya-s-zanizhennoy-taliey/index.php$ Allow: /catalog/wedding-dresses/svadebnye-platya-s-vysokoj-taliej/index.php$ Allow: /catalog/wedding-dresses/assimetrichnye-svadebnye-platya/index.php$ Allow: /catalog/wedding-dresses/rasprodazha-svadebnyh-platjev/index.php$ Allow: /catalog/wedding-dresses/nedorogie-svadebnye-platya/index.php$ Allow: /catalog/wedding-dresses/$ Allow: /catalog/wedding-dresses/index.php$ Allow: /catalog/wedding-dresses/love-story/?page_2=* Allow: /catalog/wedding-dresses/knightly/?page_2=* Allow: /catalog/wedding-dresses/dream-collection/?page_2=* Allow: /catalog/wedding-dresses/pronovias1/?page_2=* Allow: /catalog/wedding-dresses/pepe_botella/?page_2=* Allow: /catalog/wedding-dresses/haute-couture/?page_2=* Allow: /catalog/wedding-dresses/econom-collection/?page_2=* Allow: /catalog/wedding-dresses/nedorogie-svadebnye-platya/?page_2=* Allow: /catalog/wedding-dresses/love-story/$ Allow: /catalog/wedding-dresses/knightly/$ Allow: /catalog/wedding-dresses/dream-collection/$ Allow: /catalog/wedding-dresses/pronovias1/$ Allow: /catalog/wedding-dresses/pepe_botella/$ Allow: /catalog/wedding-dresses/haute-couture/$ Allow: /catalog/wedding-dresses/econom-collection/$ Allow: /catalog/wedding-dresses/nedorogie-svadebnye-platya/$ Allow: /catalog/wedding-dresses/love-story/index.php$ Allow: /catalog/wedding-dresses/knightly/index.php$ Allow: /catalog/wedding-dresses/dream-collection/index.php$ Allow: /catalog/wedding-dresses/pronovias1/index.php$ Allow: /catalog/wedding-dresses/pepe_botella/index.php$ Allow: /catalog/wedding-dresses/haute-couture/index.php$ Allow: /catalog/wedding-dresses/econom-collection/index.php$ Allow: /catalog/wedding-dresses/nedorogie-svadebnye-platya/index.php$ Allow: /catalog/wedding-dresses/view/love-story/* Allow: /catalog/wedding-dresses/view/knightly/* Allow: /catalog/wedding-dresses/view/dream-collection/* Allow: /catalog/wedding-dresses/view/pronovias1/* Allow: /catalog/wedding-dresses/view/pepe_botella/* Allow: /catalog/wedding-dresses/view/haute-couture/* Allow: /catalog/wedding-dresses/view/econom-collection/* Allow: /catalog/wedding-dresses/view/nedorogie-svadebnye-platya/* Allow: /atelier/$ Allow: /atelier/index.php$ Allow: /style/$ Allow: /style/index.php$ Allow: /fotoden/$ Allow: /fotoden/index.php$ Allow: /contacts/$ Allow: /contacts/index.php$ Allow: /style/price.php$ Allow: /style/svadebnaja-pricheska-v-grecheskom-stile/$ Allow: /style/svadebnaja-pricheska-gollivudskie-lokony/$ Allow: /style/svadebnaja-pricheska-na-dlinnye-volosy/$ Allow: /style/svadebnaja-pricheska-na-korotkie-volosy/$ Allow: /style/svadebnaja-pricheska-na-srednie-volosy/$ Allow: /style/svadebnaja-pricheska-s-diademoj/$ Allow: /style/svadebnaja-pricheska-s-fatoj/$ Allow: /style/vysokaja-svadebnaja-pricheska-s-cvetami/$ Allow: /style/svadebnaja-pricheska-s-fatoj-na-raspushhennye-volosy/$ Allow: /style/svadebnaja-pricheska-puchok/$ Allow: /style/svadebnaja-pricheska-v-grecheskom-stile/index.php$ Allow: /style/svadebnaja-pricheska-gollivudskie-lokony/index.php$ Allow: /style/svadebnaja-pricheska-na-dlinnye-volosy/index.php$ Allow: /style/svadebnaja-pricheska-na-korotkie-volosy/index.php$ Allow: /style/svadebnaja-pricheska-na-srednie-volosy/index.php$ Allow: /style/svadebnaja-pricheska-s-diademoj/index.php$ Allow: /style/svadebnaja-pricheska-s-fatoj/index.php$ Allow: /style/vysokaja-svadebnaja-pricheska-s-cvetami/index.php$ Allow: /style/svadebnaja-pricheska-s-fatoj-na-raspushhennye-volosy/index.php$ Allow: /style/svadebnaja-pricheska-puchok/index.php$ Allow: /style/svadebnaja-pricheska-v-grecheskom-stile/view/* Allow: /style/svadebnaja-pricheska-gollivudskie-lokony/view/* Allow: /style/svadebnaja-pricheska-na-dlinnye-volosy/view/* Allow: /style/svadebnaja-pricheska-na-korotkie-volosy/view/* Allow: /style/svadebnaja-pricheska-na-srednie-volosy/view/* Allow: /style/svadebnaja-pricheska-s-diademoj/view/* Allow: /style/svadebnaja-pricheska-s-fatoj/view/* Allow: /style/vysokaja-svadebnaja-pricheska-s-cvetami/view/* Allow: /style/svadebnaja-pricheska-s-fatoj-na-raspushhennye-volosy/view/* Allow: /style/svadebnaja-pricheska-puchok/view/* Allow: /catalog/yubki/index.php$ Allow: /catalog/yubki/$ Allow: /catalog/yubki/view/* Sitemap: http://www.dreamboutique.ru/sitemap.xml user-agent: AhrefsBot disallow: /
Заранее спасибо за помощь.
