So proxy a ɛwɔ anim bɛhyɛ robots.txt so ma crawler?
Fa robots nhyehyɛe no to crawler no nsam, na te nneɛma a ɛyɛ den a ɛwɔ sɛ wɔde hyɛ so wɔ proxy ase no ase.
HTTP proxy a wɔde bɛma crawler no sesa kwan a ɔfa so kɔ website no so. Nanso ɛmma proxy no nkɔfa robots.txt, na ɛnyɛ ɔno na ɛpaw crawler no mmara kuo anaa ɛhyehyɛ n’abusua nsɛm yiye.
Robots Exclusion Protocol kyerɛ mmara a ɛfa crawlers ho, a bi ne sɛnea wɔsusu mu, mfomso a ɛba berɛ a wɔrefa fael, ne kwan a wɔsakra so. Ɛho hia sɛ wosiesie mmara no wɔ crawler no ankasa, ɛfiri sɛ ɛhɔ na wunim ne din, ne botae URI ne berɛ nhyehyɛe. Kenkan nsɛm no nyinaa wɔ Borɔfo kasa mu.