本文基于Scrapy 2.17.0，所有指纹值都是本机实跑出来的，不是推演。文中会给出两个反直觉的结论（默认端口、相对路径），如果你用 Scrapy 做过增量爬取，建议重点看第六节。前言上一篇 讲Scheduler时，enqueue_request的第一行一笔带过了：ifnotrequest.dont_filterandself.df.request_seen(request):self.df.log(request,self.spider)returnFalseself.df就是去重器。这一行决定了哪些请求根本进不了队列——去重规则稍有偏差，轻则重复抓取浪费带宽，重则该抓的链接被静默丢弃，而且日志里几乎看不出来。这篇我们把它拆到底：指纹怎么算、归一化做了什么、以及它没做什么。一、BaseDupeFilter：一个什么都不做的基类classBaseDupeFilter:"""Dummy duplicate request filtering class (:setting:`DUPEFILTER_CLASS`) that does not filter out any request."""@classmethoddeffrom_crawler(cls,crawler:Crawler)-Self:returncls()defrequest_seen(self,request:Request)-bool:returnFalsedefopen(self)-Deferred[None]|None:passdefclose(self,reason:str)-Deferred[None]|None:passdeflog(self,request:Request,spider:Spider)-None:"""Log that a request has been filtered"""warn("Calling BaseDupeFilter.log() is deprecated.",ScrapyDeprecationWarning,stacklevel=2,)request_seen永远返回False——它是「关闭去重」的现成实现。想全局禁用去重，不用给每个请求写dont_filter=True：# settings.pyDUPEFILTER_CLASS="scrapy.dupefilters.BaseDupeFilter"注意log()已经被标记废弃了。自己写去重器时不要再调super().log()。二、RFPDupeFilter：一个 set 加一个文件真正的默认实现只有几十行：classRFPDupeFilter(BaseDupeFilter):def__init__(self,path=None,debug=False,*,fingerprinter=None)-None:self.file=Noneself.fingerprinter:RequestFingerprinterProtocol=(fingerprinterorRequestFingerprinter())self.fingerprints:set[str]=set()self.logdupes=Trueself.debug=debug self.logger=logging.getLogger(__name__)ifpath:# line-by-line writing, see: https://github.com/scrapy/scrapy/issues/6019self.file=Path(path,"requests.seen").open("a+",buffering=1,encoding="utf-8")self.file.reconfigure(write_through=True)self.file.seek(0)self.fingerprints.update(x.rstrip()forxinself.file)defrequest_seen(self,request:Request)-bool:fp=self.request_fingerprint(request)iffpinself.fingerprints:returnTrueself.fingerprints.add(fp)ifself.file:self.file.write(fp+"\n")returnFalsedefrequest_fingerprint(self,request:Request)-str:"" 
长河君 · 2026/8/25 14:20:55 · 阅读 1.2k RFPDupeFilter 与请求指纹算法详解
本文基于Scrapy 2.17.0，所有指纹值都是本机实跑出来的，不是推演。文中会给出两个反直觉的结论（默认端口、相对路径），如果你用 Scrapy 做过增量爬取，建议重点看第六节。前言上一篇 讲Scheduler时，enqueue_request的第一行一笔带过了：ifnotrequest.dont_filterandself.df.request_seen(request):self.df.log(request,self.spider)returnFalseself.df就是去重器。这一行决定了哪些请求根本进不了队列——去重规则稍有偏差，轻则重复抓取浪费带宽，重则该抓的链接被静默丢弃，而且日志里几乎看不出来。这篇我们把它拆到底：指纹怎么算、归一化做了什么、以及它没做什么。一、BaseDupeFilter：一个什么都不做的基类classBaseDupeFilter:"""Dummy duplicate request filtering class (:setting:`DUPEFILTER_CLASS`) that does not filter out any request."""@classmethoddeffrom_crawler(cls,crawler:Crawler)-Self:returncls()defrequest_seen(self,request:Request)-bool:returnFalsedefopen(self)-Deferred[None]|None:passdefclose(self,reason:str)-Deferred[None]|None:passdeflog(self,request:Request,spider:Spider)-None:"""Log that a request has been filtered"""warn("Calling BaseDupeFilter.log() is deprecated.",ScrapyDeprecationWarning,stacklevel=2,)request_seen永远返回False——它是「关闭去重」的现成实现。想全局禁用去重，不用给每个请求写dont_filter=True：# settings.pyDUPEFILTER_CLASS="scrapy.dupefilters.BaseDupeFilter"注意log()已经被标记废弃了。自己写去重器时不要再调super().log()。二、RFPDupeFilter：一个 set 加一个文件真正的默认实现只有几十行：classRFPDupeFilter(BaseDupeFilter):def__init__(self,path=None,debug=False,*,fingerprinter=None)-None:self.file=Noneself.fingerprinter:RequestFingerprinterProtocol=(fingerprinterorRequestFingerprinter())self.fingerprints:set[str]=set()self.logdupes=Trueself.debug=debug self.logger=logging.getLogger(__name__)ifpath:# line-by-line writing, see: https://github.com/scrapy/scrapy/issues/6019self.file=Path(path,"requests.seen").open("a+",buffering=1,encoding="utf-8")self.file.reconfigure(write_through=True)self.file.seek(0)self.fingerprints.update(x.rstrip()forxinself.file)defrequest_seen(self,request:Request)-bool:fp=self.request_fingerprint(request)iffpinself.fingerprints:returnTrueself.fingerprints.add(fp)ifself.file:self.file.write(fp+"\n")returnFalsedefrequest_fingerprint(self,request:Request)-str:""