I-proxy eqondileyo iyawusebenzisa robots.txt kum-crawler?
Gcina umthetho we-robots kum-crawler njalo uqonde imida yokuwusebenzisa ku-proxy.
Ukunika i-crawler i-HTTP proxy kuguqula indlela izicelo zayo ezifika ngayo ku-website. Kodwa akukwenzi i-proxy ithathe i-robots.txt loba ikhethe iqembu lemithertho ye-crawler.
Umthetho we-robots kufanele ugcinwe kum-crawler ngokwakho. Funda isihloko ngesiNgisi.