python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Here is what the above code is Doing:
1. Define a class that inherits from scrapy.Spider.
2. Give it a name.
3. Define a start_urls variable that contains a list of URLs to scrape.
4. Define a parse() method that will be called on the start URLs.
5. In the parse() method, use CSS selectors to extract the blog titles from the response.
6. Use the scrapy.Request class to follow links to the next page.
7. In the parse() method, use CSS selectors to extract the blog titles from the response.
8. Use the scrapy.Request class to follow links to the next page.