Scrapy, a fast high-level web crawling & scraping framework for Python.

A Powerful Spider(Web Crawler) System in Python.

Elegant Scraper and Crawler Framework for Golang

👾 Fast, simple and clean video downloader

Python爬虫代理IP池(proxy pool)

News, full-text, and article metadata extraction in Python 3. Advanced docs:

A scalable web crawler framework for Java.

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

Incredibly fast crawler designed for OSINT.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

Distributed crawler powered by Headless Chrome

基于搜狗微信搜索的微信公众号爬虫接口

Redis-based components for Scrapy.

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

A collection of awesome web crawler,spider in different languages

Declarative web scraping

Every web site provides APIs.

The DomCrawler component eases DOM navigation for HTML and XML documents.

Intelligent proxy pool for Humans™ (Maintainer needed)

Web Application Security Scanner Framework

DotnetSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

Web crawling framework based on asyncio.

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目: