scrapy: 将数据写入 json 文件

Write items to a JSON file

定义一个 pipline

import json

from itemadapter import ItemAdapter

class JsonWriterPipeline:

    def open_spider(self, spider):
        self.file = open('items.jl', 'w')

    def close_spider(self, spider):
        self.file.close()

    def process_item(self, item, spider):
        line = json.dumps(ItemAdapter(item).asdict()) + "\n"
        self.file.write(line)
        return item

添加到 settings.py 中

ITEM_PIPELINES = {
  'myspider.pipelines.JsonWriterPipeline': 300,
}

参考

• https://docs.scrapy.org/en/latest/topics/item-pipeline.html#item-pipeline-example