crawler

👾 Fast and simple video download library and CLI tool written in Go

A Powerful Spider(Web Crawler) System in Python.

Elegant Scraper and Crawler Framework for Golang

Python爬虫代理IP池(proxy pool)

News, full-text, and article metadata extraction in Python 3. Advanced docs:

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

A scalable web crawler framework for Java.

Incredibly fast crawler designed for OSINT.

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

基于搜狗微信搜索的微信公众号爬虫接口

Distributed crawler powered by Headless Chrome

Redis-based components for Scrapy.

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

A collection of awesome web crawler,spider in different languages

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

The DomCrawler component eases DOM navigation for HTML and XML documents.

Intelligent proxy pool for Humans™ (Maintainer needed)

Every web site provides APIs.

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

Web Application Security Scanner Framework

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作的爬虫行业从业者了解我国相关法律，避免触碰数据合规红线。 [AD]中文知识图谱门户