A Scrapy Cookie Jar Belongs to a Session, Not Automatically to a Proxy

Switching the proxy does not automatically switch the spider's cookies. That can be useful for one continuing session and surprising when several independent sessions share the spider.

Scrapy's cookie middleware supports separate jars selected by the request's cookiejar metadata. Give that key the identity you actually intend to preserve:

yield scrapy.Request(
    'https://example.com/start',
    meta={'proxy': selected_proxy, 'cookiejar': 'workflow-17'},
)

The endpoint and jar key are independent choices. An unauthenticated public crawl may take its selected HTTP endpoint from the HTTP proxy list where that route is permitted; the workflow still owns the jar key. A follow-up request that should remain in this session needs the same jar key; the metadata is not automatically sticky for every new request you create. Propagate the specific fields required by the workflow rather than blindly copying all middleware metadata.

Using the proxy address as the jar key is only one policy

That shortcut groups state by endpoint. It does not necessarily group it by person, account, or public exit. One provider endpoint can rotate exits, and several accounts can use the same endpoint.

For an authorized workflow that needs stable state, record the account/session identifier, jar identifier, and selected route separately. If you decide to change the route halfway through, decide what should happen to the application session as well.

Separate jars prevent accidental cookie sharing between the groups you defined. They do not promise that a destination cannot relate two requests through an account or other application data.

If the wrong page persists after changing IPs, inspect which jar sent the request before deleting every cookie or replacing the entire proxy pool. The retained state may explain the page perfectly well.