HTTP proxy bɛ robots.txt labɛn crawler ye wa?
Robots sariya togɔdon crawler kɔnɔ, ani ka famu ko proxy tɛ se ka a labɛn a yɛrɛ ye.
Ni i ye HTTP proxy di crawler ma, a bɛna a ɲɛsili yɛlɛma, nka a tɛna proxy wajibi ye ka robots.txt ta, walima ka crawler ka sariya kulu sugandi, walima ka a ka ɲininkali ɲɛnajɛla kɛ.
Robots sariya togɔdon crawler yɛrɛ kɔnɔ, ɲininkali sariya ni sisanw ye. Proxy tɛ se ka o sariya labɛn crawler ye. Kalan Anglɛ kan.