Nzaghachi proxy ọ na-etinye robots.txt n'ọrụ maka crawler?
Debe iwu robots n'ime crawler ma ghọta oke itinye ya n'ọrụ na proxy.
Inye crawler HTTP proxy na-agbanwe otú arịrịọ ya si erute weebụsaịtị. Ma ọ naghị eme ka proxy were robots.txt, họrọ otu iwu crawler, ma ọ bụ hazie arịrịọ ya nke ọma.
Iwu Robots Exclusion Protocol na-akọwa iwu maka crawler, tinyere ndakọrịta na mwepu. Ọ dị mkpa ka iwu a nọrọ n'ime crawler n'onwe ya. Gụọ isiokwu Bekee.