爬虫集合

Elegant Scraper and Crawler Framework for Golang

Python爬虫代理IP池(proxy pool)

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

Incredibly fast crawler designed for OSINT.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Pholcus is a distributed high-concurrency crawler software written in pure golang

越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）（因工作原因，项目暂停）

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

A collection of awesome web crawler,spider in different languages

今日热榜，一个获取各大热门网站热门头条的聚合网站，使用Go语言编写，多协程异步快速抓取信息，预览:https://mo.fish

Every web site provides APIs.

admin ui for scrapy/open source scrapinghub

BitTorrent DHT Protocol && DHT Spider.

🔞 JAVClub - 让你的大姐姐不再走丢

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

Web crawling framework based on asyncio.

owllook-在线网络小说阅读网站&小说搜索引擎&小说推荐系统[搜索、追书、收藏、追更、小说API]

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO 👉

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 🚁，点星支持

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

简单易用的Python爬虫框架，QQ交流群：597510560

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

Async Python 3.6+ web scraping micro-framework based on asyncio（Python3.6+异步爬虫框架）

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.