How can I use different pipelines for different spiders in a single Scrapy project

Question

I have a scrapy project which contains multiple spiders. Is there any way I can define which pipelines to use for which spider? Not all the pipelines i have defined are applicable for every spider.

Thanks

Answer 1

Just remove all pipelines from main settings and use this inside spider.

This will define the pipeline to user per spider

class testSpider(InitSpider):
    name = 'test'
    custom_settings = {
        'ITEM_PIPELINES': {
            'app.MyPipeline': 400
        }
    }

Answer 2

Building on the solution from Pablo Hoffman, you can use the following decorator on the process_item method of a Pipeline object so that it checks the pipeline attribute of your spider for whether or not it should be executed. For example:

def check_spider_pipeline(process_item_method):

    @functools.wraps(process_item_method)
    def wrapper(self, item, spider):

        # message template for debugging
        msg = '%%s %s pipeline step' % (self.__class__.__name__,)

        # if class is in the spider's pipeline, then use the
        # process_item method normally.
        if self.__class__ in spider.pipeline:
            spider.log(msg % 'executing', level=log.DEBUG)
            return process_item_method(self, item, spider)

        # otherwise, just return the untouched item (skip this step in
        # the pipeline)
        else:
            spider.log(msg % 'skipping', level=log.DEBUG)
            return item

    return wrapper

For this decorator to work correctly, the spider must have a pipeline attribute with a container of the Pipeline objects that you want to use to process the item, for example:

class MySpider(BaseSpider):

    pipeline = set([
        pipelines.Save,
        pipelines.Validate,
    ])

    def parse(self, response):
        # insert scrapy goodness here
        return item

And then in a pipelines.py file:

class Save(object):

    @check_spider_pipeline
    def process_item(self, item, spider):
        # do saving here
        return item

class Validate(object):

    @check_spider_pipeline
    def process_item(self, item, spider):
        # do validating here
        return item

All Pipeline objects should still be defined in ITEM_PIPELINES in settings (in the correct order -- would be nice to change so that the order could be specified on the Spider, too).

Answer 3

The other solutions given here are good, but I think they could be slow, because we are not really not using the pipeline per spider, instead we are checking if a pipeline exists every time an item is returned (and in some cases this could reach millions).

A good way to completely disable (or enable) a feature per spider is using custom_setting and from_crawler for all extensions like this:

pipelines.py

from scrapy.exceptions import NotConfigured

class SomePipeline(object):
    def __init__(self):
        pass

    @classmethod
    def from_crawler(cls, crawler):
        if not crawler.settings.getbool('SOMEPIPELINE_ENABLED'):
            # if this isn't specified in settings, the pipeline will be completely disabled
            raise NotConfigured
        return cls()

    def process_item(self, item, spider):
        # change my item
        return item

settings.py

ITEM_PIPELINES = {
   'myproject.pipelines.SomePipeline': 300,
}
SOMEPIPELINE_ENABLED = True # you could have the pipeline enabled by default

spider1.py

class Spider1(Spider):

    name = 'spider1'

    start_urls = ["http://example.com"]

    custom_settings = {
        'SOMEPIPELINE_ENABLED': False
    }

As you check, we have specified custom_settings that will override the things specified in settings.py, and we are disabling SOMEPIPELINE_ENABLED for this spider.

Now when you run this spider, check for something like:

[scrapy] INFO: Enabled item pipelines: []

Now scrapy has completely disabled the pipeline, not bothering of its existence for the whole run. Check that this also works for scrapy extensions and middlewares.

Answer 4

You can use the name attribute of the spider in your pipeline

class CustomPipeline(object)

    def process_item(self, item, spider)
         if spider.name == 'spider1':
             # do something
             return item
         return item

Defining all pipelines this way can accomplish what you want.

Answer 5

I can think of at least four approaches:

1. Use a different scrapy project per set of spiders+pipelines (might be appropriate if your spiders are different enough warrant being in different projects)
2. On the scrapy tool command line, change the pipeline setting with scrapy settings in between each invocation of your spider
3. Isolate your spiders into their own scrapy tool commands, and define the default_settings['ITEM_PIPELINES'] on your command class to the pipeline list you want for that command. See line 6 of this example.
4. In the pipeline classes themselves, have process_item() check what spider it's running against, and do nothing if it should be ignored for that spider. See the example using resources per spider to get you started. (This seems like an ugly solution because it tightly couples spiders and item pipelines. You probably shouldn't use this one.)

Answer 6

The most simple and effective solution is to set custom settings in each spider itself.

custom_settings = {'ITEM_PIPELINES': {'project_name.pipelines.SecondPipeline': 300}}

After that you need to set them in the settings.py file

ITEM_PIPELINES = {
   'project_name.pipelines.FistPipeline': 300,
   'project_name.pipelines.SecondPipeline': 400
}

in that way each spider will use the respective pipeline.

Answer 7

You can just set the item pipelines settings inside of the spider like this:

class CustomSpider(Spider):
    name = 'custom_spider'
    custom_settings = {
        'ITEM_PIPELINES': {
            '__main__.PagePipeline': 400,
            '__main__.ProductPipeline': 300,
        },
        'CONCURRENT_REQUESTS_PER_DOMAIN': 2
    }

I can then split up a pipeline (or even use multiple pipelines) by adding a value to the loader/returned item that identifies which part of the spider sent items over. This way I won’t get any KeyError exceptions and I know which items should be available.

    ...
    def scrape_stuff(self, response):
        pageloader = PageLoader(
                PageItem(), response=response)

        pageloader.add_xpath('entire_page', '/html//text()')
        pageloader.add_value('item_type', 'page')
        yield pageloader.load_item()

        productloader = ProductLoader(
                ProductItem(), response=response)

        productloader.add_xpath('product_name', '//span[contains(text(), "Example")]')
        productloader.add_value('item_type', 'product')
        yield productloader.load_item()

class PagePipeline:
    def process_item(self, item, spider):
        if item['item_type'] == 'product':
            # do product stuff

        if item['item_type'] == 'page':
            # do page stuff

Answer 8

I am using two pipelines, one for image download (MyImagesPipeline) and second for save data in mongodb (MongoPipeline).

suppose we have many spiders(spider1,spider2,...........),in my example spider1 and spider5 can not use MyImagesPipeline

settings.py

ITEM_PIPELINES = {'scrapycrawler.pipelines.MyImagesPipeline' : 1,'scrapycrawler.pipelines.MongoPipeline' : 2}
IMAGES_STORE = '/var/www/scrapycrawler/dowload'

And bellow complete code of pipeline

import scrapy
import string
import pymongo
from scrapy.pipelines.images import ImagesPipeline

class MyImagesPipeline(ImagesPipeline):
    def process_item(self, item, spider):
        if spider.name not in ['spider1', 'spider5']:
            return super(ImagesPipeline, self).process_item(item, spider)
        else:
           return item 

    def file_path(self, request, response=None, info=None):
        image_name = string.split(request.url, '/')[-1]
        dir1 = image_name[0]
        dir2 = image_name[1]
        return dir1 + '/' + dir2 + '/' +image_name

class MongoPipeline(object):

    collection_name = 'scrapy_items'
    collection_url='snapdeal_urls'

    def __init__(self, mongo_uri, mongo_db):
        self.mongo_uri = mongo_uri
        self.mongo_db = mongo_db

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            mongo_uri=crawler.settings.get('MONGO_URI'),
            mongo_db=crawler.settings.get('MONGO_DATABASE', 'scraping')
        )

    def open_spider(self, spider):
        self.client = pymongo.MongoClient(self.mongo_uri)
        self.db = self.client[self.mongo_db]

    def close_spider(self, spider):
        self.client.close()

    def process_item(self, item, spider):
        #self.db[self.collection_name].insert(dict(item))
        collection_name=item.get( 'collection_name', self.collection_name )
        self.db[collection_name].insert(dict(item))
        data = {}
        data['base_id'] = item['base_id']
        self.db[self.collection_url].update({
            'base_id': item['base_id']
        }, {
            '$set': {
            'image_download': 1
            }
        }, upsert=False, multi=True)
        return item

Answer 9

we can use some conditions in pipeline as this

    # -*- coding: utf-8 -*-
from scrapy_app.items import x

class SaveItemPipeline(object):
    def process_item(self, item, spider):
        if isinstance(item, x,):
            item.save()
        return item

Answer 10

Overriding 'ITEM_PIPELINES' with custom settings per spider, as others have suggested, works well. However, I found I had a few distinct groups of pipelines I wanted to use for different categories of spiders. I wanted to be able to easily define the pipeline for a particular category of spider without a lot of thought, and I wanted to be able to update a pipeline category without editing each spider in that category individually.

So I created a new file called pipeline_definitions.py in the same directory as settings.py. pipeline_definitions.py contains functions like this:

def episode_pipelines():
    return {
        'radio_scrape.pipelines.SaveEpisode': 100,
    }

def show_pipelines():
    return {
        'radio_scrape.pipelines.SaveShow': 100,
    }

Then in each spider I would import the specific function relevant for the spider:

from radio_scrape.pipeline_definitions import episode_pipelines

I then use that function in the custom settings assignment:

class RadioStationAEspisodesSpider(scrapy.Spider):
    name = 'radio_station_A_episodes'        
    custom_settings = {
        'ITEM_PIPELINES': episode_pipelines()
    }

Answer 11

Simple but still useful solution.

Spider code

    def parse(self, response):
        item = {}
        ... do parse stuff
        item['info'] = {'spider': 'Spider2'}

pipeline code

    def process_item(self, item, spider):
        if item['info']['spider'] == 'Spider1':
            logging.error('Spider1 pipeline works')
        elif item['info']['spider'] == 'Spider2':
            logging.error('Spider2 pipeline works')
        elif item['info']['spider'] == 'Spider3':
            logging.error('Spider3 pipeline works')

Hope this save some time for somebody!