Ingabe i-proxy yokuqondisa iyawusebenzisa robots.txt kum-crawler?
Gcina inqubomgomo ye-robots kum-crawler futhi uqonde imikhawulo yokuyisebenzisa ku-proxy yokudlulisa.
Ukunika i-crawler i-proxy ye-HTTP kushintsha indlela izicelo zayo ezifika ngayo kuwebhusayithi. Kodwa akwenzi ukuthi i-proxy ilande i-robots.txt, ikhethe iqembu lemithertho ye-crawler noma ihlele izicelo zayo ngokuzithoba.
I-Robots Exclusion Protocol ichaza imithertho yama-crawler. Ihlanganisa ukufanisa, ukwehluleka kokulanda, ukuhambisa kabusha kanye nokuziphatha kwe-cache. I-proxy edlulisa kuphela idokhumenti eceliwe ayizange isebenzise leyo nqubomgomo egameni le-crawler.
Gcina isinqumo semithertho eduze nomugqa wezicelo we-crawler. Yilapho ubunikazi obuhlosiwe, i-URI eqondiwe kanye nesimo sokuhlela kutholakala khona ngaphambi kokuthunyelwa komsebenzi.
Kungani ukuhlunga ohlangothini lwe-proxy kunzima kunalokho okuzwakala
Ku-HTTP evamile, isihlungi esakhiwe ngokukhethekile singabona indlela eqondiwe kanye ne-User-Agent. Noma kunjalo sidinga ukusetshenziswa kwangempela kwemithertho kanye nenqubomgomo ecacisiwe yokulanda, yokugcina kanye nokubuyekeza ifayela le-robots elisebenzayo. Ukuguqula imigqa embalwa ye-Disallow ibe imigoqo yekheli emile akuwona umthetho wokuxhumana ophelele.
Ku-HTTPS ngokusebenzisa i-CONNECT evamile, i-proxy yokuqondisa ibona indawo okuphela kuyo umhubhe, hhayi indlela yekhasi ngalinye elibethelwe kanye ne-User-Agent yangaphakathi. Ayinakho ulwazi oludingekayo ezinqumweni ze-robots ezingeni lendlela, ngaphandle uma okunye ukuhlelwa ngamabomu kukunikeza.
Ukushintsha ikheli lokuphuma nakho akwenzi indlela evinjiwe ibe indlela evunyelwe. Xhumanisa ijubane phakathi kwabasebenzi ababelana ngomsebenzi we-crawler; ukushintshanisa imizila akuyona indawo yaleyo nqubomgomo.
Ekugcineni, imithertho ye-robots akuyona imvume yokufinyelela. Indlela evunyelwe ayidluli ukuqinisekiswa noma ezinye izidingo zokufinyelela zewebhusayithi, futhi ifayela le-robots lomphakathi alivikeli okuqukethwe okuyimfihlo.