Proxy y'imbere ikoresha robots.txt ku crawler?
Gumya ingingo ngenderwako za robots muri crawler no gutahura ingano z'ikoreshwa ryayo kuri proxy.
Guha crawler proxy ya HTTP bihindura ingene ibisabwa vyayo bigera kuri website. Ariko ntibigira ko proxy izana robots.txt canke igatora itsinda ry'amategeko ya crawler.
Amategeko ya robots akwiye kugumya muri crawler nyene. Soma ikiganiro mu congereza.