Scrapy: Powerful Web Scraping & Crawling with Python
Why take this course?
很高興你對於Scrapy這個強大的Python網頁抓取框架感興趣，並且希望透過本教程來學習如何使用它。Scrapy是一個開源的Web爬蟲框架，專為複雜的Web爬蟲任務而設計，以提供高性能和可擴展性。它不僅僅是用於從網站中提取內容的工具，還包括了網頁爬取（crawling）的功能。
以下是這個Python Scrapy教程的主要內容概述：
-
Scrapy介紹 - Scrapy和其他Python網頁抓取庫（如BeautifulSoup, LXML, Requests, Selenium等）的比較，以及何時選擇使用Scrapy。
-
安裝和創建Scrapy項目 - 學習如何在你的開發環境中安裝Scrapy，並創建一個新的Scrapy項目。
-
基本Spider - 創建一個基本的Spider來從網站提取數據。
-
XPath命令 - 學習如何使用XPath命令與Scrapy一起使用，以精確提取所需數據。
-
高級Spider - 構建一個更複雜的Spider，能夠遍歷多個網頁頁面並從每一頁提取數據。
-
Scrapy架構 - 探討Scrapy項目的整體架構，理解各部分的作用和如何在Spider代碼中使用它們。
-
最佳實踐 - 學習如何避免因為不當行為而被網站封禁的最佳實踐。
-
部署到Scrapy Cloud - 學習如何將你的Scrapy爬蟲部署到Scrapy Cloud，這是一個無需自行設置伺服器的平台。
-
認証會話 - 探索Scrapy在需要用戶登錄才能訪問數據的網站上進行爬取時的使用情境。
-
與Splash/Selenium結合 - 當你需要抓取包含許多AJAX調用和JavaScript執行的動態網頁時，學習如何結合Scrapy、Splash或Selenium。
-
數據輸出 - 學習如何將提取的數據寫入CSV、JSON和XML文件。
-
數據存儲 - 探索如何將Scrapy提取的數據存儲到MySQL和MongoDB數據庫中。
這個教程旨在幫助你從基礎到高級，全面理解並使用Scrapy進行網頁抓取。通过实际操作和案例分析，你將能够构建、调试和优化自己的网络爬虫项目。
Loading charts...
Comidoc Review
Our Verdict
This 4.25-star rated Scrapy course on Udemy provides in-depth, detailed instruction and real-life examples that enable learners to apply their web scraping skills in various contexts. With over 16K subscribers and updated as recently as early 2020, the course covers Scrapy's powerful features while providing Q&A support for students. However, prepare to invest time into fully understanding the content due to occasional pacing problems and potentially under-explained code rationales. Make sure you dedicate the required effort to ensure you derive the maximum benefit from this engaging, informative course. Note that the codes' functionality can vary, occasionally requiring additional work to achieve desired outcomes.
What We Liked
- In-depth coverage of Scrapy and web scraping, touching on advanced topics like editing and using Scrapy parameters, building Scrapy advanced spiders, and more
- Real-life web scraping projects utilizing Craigslist, LinkedIn, and others, enabling students to apply their knowledge in practical contexts
- Rich supporting materials, such as downloadable Python source code for all exercises and Q&A board for queries
- Comprehensive video delivery that covers numerous aspects of Scrapy without leaving essential details unexplained
Potential Drawbacks
- Pacing and organization concerns raised by several users; some key points and information may require rewinding to fully grasp
- Lack of clear motivation behind written code, which impacts students' understanding of the rationale and purpose behind specific actions taken within Scrapy files
- The occasional issue with provided code not functioning as intended, resulting in time lost troubleshooting or learning from alternative sources
- Subtitles need improvement for better clarity