Learning Scrapy : learn the art of efficient web scraping and crawling with Python / Dimitrios Kouzis-Loukas.

Online resource; title from cover (Safari, viewed February 12, 2016).

Includes index.

Annotation Learn the art of efficient web scraping and crawling with PythonAbout This Book Extract data from any source to perform real time analytics. Full of techniques and examples to help you crawl websites and extract data within hours. A hands-on guide to web scraping and crawling with real-life problems and solutionsWho This Book Is ForIf you are a software developer, data scientist, NLP or machine-learning enthusiast or just need to migrate your company's wiki from a legacy platform, then this book is for you. It is perfect for someone, who needs instant access to large amounts of semi-structured data effortlessly. What You Will Learn Understand HTML pages and write XPath to extract the data you need Write Scrapy spiders with simple Python and do web crawls Push your data into any database, search engine or analytics system Configure your spider to download files, images and use proxies Create efficient pipelines that shape data in precisely the form you want Use Twisted Asynchronous API to process hundreds of items concurrently Make your crawler super-fast by learning how to tune Scrapy's performance Perform large scale distributed crawls with scrapyd and scrapinghubIn DetailThis book covers the long awaited Scrapy v 1.0 that empowers you to extract useful data from virtually any source with very little effort. It starts off by explaining the fundamentals of Scrapy framework, followed by a thorough description of how to extract data from any source, clean it up, shape it as per your requirement using Python and 3rd party APIs. Next you will be familiarised with the process of storing the scrapped data in databases as well as search engines and performing real time analytics on them with Spark Streaming. By the end of this book, you will perfect the art of scarping data for your applications with easeStyle and approachIt is a hands on guide, with first few chapters written as a tutorial, aiming to motivate you and get you started quickly. As the book progresses, more advanced features are explained with real world examples that can be reffered while developing your own web applications.

Cover ; Copyright; Credits; About the Author; About the Reviewer; www.PacktPub.com; Table of Contents; Preface; Chapter 1: Introducing Scrapy; Hello Scrapy; More reasons to love Scrapy; About this book: aim and usage; The importance of mastering automated data scraping; Developing robust, quality applications, and providing realistic schedules; Developing quality minimum viable products quickly; Scraping gives you scale; Google couldn't use forms; Discovering and integrating into your ecosystem; Being a good citizen in a world full of spiders; What Scrapy is not; Summary

Chapter 2 : Understanding HTML and XPathHTML, the DOM tree representation, and the XPath; The URL; The HTML document; The tree representation; What you see on the screen; Selecting HTML elements with XPath; Useful XPath expressions; Using Chrome to get XPath expressions; Examples of common tasks; Anticipating changes; Summary; Chapter 3 : Basic Crawling; Installing Scrapy; MacOS; Windows; Linux; Ubuntu or Debian Linux; Red Hat or CentOS Linux; From the latest source; Upgrading Scrapy; Vagrant: this book's official way to run examples; UR2IM -- the fundamental scraping process; The URL

The request and the responseThe Items; A Scrapy project; Defining items; Writing spiders; Populating an item; Saving to files; Cleaning up -- item loaders and housekeeping fields; Creating contracts; Extracting more URLs; Two-direction crawling with a spider; Two-direction crawling with a CrawlSpider; Summary; Chapter 4 : From Scrapy to a Mobile App; Choosing a mobile application framework; Creating a database and a collection; Populating the database with Scrapy; Creating a mobile application; Creating a database access service; Setting up the user interface

Mapping data to the User InterfaceMappings between database fields and User Interface controls; Testing, sharing, and exporting your mobile app; Summary; Chapter 5 : Quick Spider Recipes; A spider that logs in; A spider that uses JSON APIs and AJAX pages; Passing arguments between responses; A 30-times faster property spider; A spider that crawls based on an Excel file; Summary; Chapter 6 : Deploying to Scrapinghub; Signing up, signing in, and starting a project; Deploying our spiders and scheduling runs; Accessing our items; Scheduling recurring crawls; Summary

Chapter 7 : Configuration and ManagementUsing Scrapy settings; Essential settings; Analysis; Logging; Stats; Telnet; Performance; Stopping crawls early; HTTP caching; Example 2 -- working offline by using the cache; Crawling style; Feeds; Downloading media; Other media; Amazon Web Services; Proxying; Example 4 -- using proxies and Crawlera's clever proxy; Further settings; Project-related settings; Extending Scrapy settings; Fine-tuning downloading; Autothrottle extension settings; Memory UsageExtension settings; Logging and debugging; Summary; Chapter 8 : Programming Scrapy

eBooks on EBSCOhost EBSCO eBook Subscription Academic Collection - Worldwide