A Powerful Spider(Web Crawler) System in Python.

👾 Fast, simple and clean video downloader

Elegant Scraper and Crawler Framework for Golang

Python爬虫代理IP池(proxy pool)

News, full-text, and article metadata extraction in Python 3. Advanced docs:

A scalable web crawler framework for Java.

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

Incredibly fast crawler designed for OSINT.

Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

Distributed crawler powered by Headless Chrome

基于搜狗微信搜索的微信公众号爬虫接口

Redis-based components for Scrapy.

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

A collection of awesome web crawler,spider in different languages

The DomCrawler component eases DOM navigation for HTML and XML documents.

Every web site provides APIs.

Intelligent proxy pool for Humans™ (Maintainer needed)

Web Application Security Scanner Framework

DotnetSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:

Web crawling framework based on asyncio.