Proxy yekumberi inoshandisa robots.txt kune crawler here?
Chengetedza mutemo we robots mukati me crawler uye nzwisisa miganhu yekuuisa kushanda pane proxy.
Kupa crawler proxy ye HTTP kunochinja nzira iyo zvikumbiro zvayo zvinosvika nezvayo kuwebhusaiti. Asi hazviite kuti proxy itore robots.txt kana kusarudza boka remitemo ye crawler.
Mitemo ye robots inofanira kuramba iri mukati me crawler pachayo. Verenga chinyorwa nechirungu.