ʻOku fakahoko ʻe he proxy ki muʻa ʻa e robots.txt ki he crawler?
Tauhi ʻa e tuʻutuʻuni robots ʻi loto ʻi he crawler pea ʻilo ʻa e ngataʻanga ʻo hono fakahoko ʻi he proxy.
ʻOku liliu ʻe he foaki ʻo ha HTTP proxy ki ha crawler ʻa e founga ʻoku aʻu ai ʻene ngaahi kole ki he uepisaiti. Ka ʻoku ʻikai te ne fakaʻaiʻai ʻa e proxy ke ʻomai ʻa e robots.txt pe fili ʻa e kulupu lao ʻo e crawler.
ʻOku totonu ke nofo ʻa e lao robots ʻi loto ʻi he crawler. Lau ʻa e fakamatala ʻi he lea faka-Pilitānia.