The Wayback Machine - http://web.archive.org/web/20200913212551/https://github.com/topics/spider

#

spider

Here are 2,120 public repositories matching this topic...

facert / awesome-spider

Star

爬虫集合

python awesome spider

Updated Aug 6, 2020

gocolly / colly

Star

Elegant Scraper and Crawler Framework for Golang

go golang crawler scraper framework spider scraping crawling

Updated Sep 12, 2020
Go

jhao104 / proxy_pool

Star

Python爬虫代理IP池(proxy pool)

redis flask crawler spider proxy crawl proxypool ssdb

Updated Sep 2, 2020
Python

shengqiangzhang / examples-of-web-crawlers

Star

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

python crawler spider example selenium multithreading stock wechat taobao pyquery tmall fund agent-pool wechat-report

Updated May 15, 2020
Python

guyueyingmu / avbook

Star

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

crawler scraper laravel database spider magnet-link guzzlehttp magnet adult javbus javlibrary avmoo adult-video

Updated Sep 8, 2020
PHP

crawlab

crawlab-team / crawlab

Star

Open

不能使用非crawlab里面mongodb么?

6

52Lau commented Jun 30, 2020

不能使用非crawlab里面mongodb么?

Read more

good first issue

Open

可配置爬虫，在界面上对进行修改，提示保存成功，然而并没有

1

Open

docker安装的任务执行有问题

5

s0md3v / Photon

Star

Incredibly fast crawler designed for OSINT.

python crawler osint spider information-gathering

Updated Jul 22, 2020
Python

henrylee2cn / pholcus

Star

Pholcus is a distributed high-concurrency crawler software written in pure golang

Updated Feb 15, 2020
Go

luyishisi / Anti-Anti-Spider

Star

越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）（因工作原因，项目暂停）

python spider geek

Updated Jun 29, 2020
Python

bda-research / node-crawler

Star

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

nodejs javascript jquery crawler spider cheerio extract-data

Updated Jun 24, 2020
JavaScript

SpiderClub / haipproxy

Star

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

redis crawler spider scheduler distributed scrapy high-availability ipproxy

Updated Nov 1, 2019
Python

BruceDone / awesome-crawler

Star

A collection of awesome web crawler,spider in different languages

crawler scraper awesome spider web-crawler web-scraper node-crawler

Updated Aug 5, 2020

tophubs / TopList

Star

今日热榜，一个获取各大热门网站热门头条的聚合网站，使用Go语言编写，多协程异步快速抓取信息，预览:https://mo.fish

golang spider hot today-s-hot-list hotlist

Updated May 6, 2020
Go

gaojiuli / toapi

Star

Every web site provides APIs.

python html api flask json crawler web spider toapi

Updated Feb 29, 2020
Python

DormyMo / SpiderKeeper

Star

admin ui for scrapy/open source scrapinghub

spider dashboard scrapy scrapyd scrapy-ui scrapyd-ui scrapyd-dashboard

Updated Nov 4, 2019
Python

core

JAVClub / core

Star

🔞 JAVClub - 让你的大姐姐不再走丢

spider japanese google-drive jav video-streaming magnet adult porn adult-content javbus javiewer

Updated Sep 12, 2020
JavaScript

shiyanhui / dht

Star

BitTorrent DHT Protocol && DHT Spider.

go spider dht bittorrent-dht-protocol

Updated Apr 26, 2020
Go

Gerapy / Gerapy

Star

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

docker vuejs django spider dashboard vue distributed scrapy scrapyd webspider gerapy

Updated Sep 13, 2020
Python

jae-jae / QueryList

Star

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

crawler scraper spider querylist

Updated Jul 18, 2020
PHP

gaojiuli / gain

Star

Web crawling framework based on asyncio.

python crawler spider aiohttp asyncio uvloop

Updated Jun 1, 2019
Python

howie6879 / owllook

Star

owllook-在线网络小说阅读网站&小说搜索引擎&小说推荐系统[搜索、追书、收藏、追更、小说API]

spider schedule book sanic aiohttp reader asyncio novels uvloop novels-search owllook

Updated May 3, 2020
Python

my8100 / scrapydweb

Star

Open

User Guide | Q&A | 用户指南 | 问答

77

LWsmile commented Nov 27, 2018

linux：HTTPConnectionPool(host='192.168.0.24', port=6801): Max retries exceeded with url: /listprojects.json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f0a78b2d828>: Failed to establish a new connection: [Errno 111] Connection refused',))
windows：HTTPConnectionPool(host='localhost', port=6801): Max retries exceeded with url: /jobs (Caused by Ne

Read more

good first issue question

kangvcar / InfoSpider

Star

INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。

automation spider selenium python3 crawl tkinter wxpython

Updated Sep 12, 2020
Python

jumper2014 / lianjia-beike-spider

Star

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 🚁，点星支持

crawler spider lianjia

Updated Jul 24, 2019
Python

sjdirect / abot

Star

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

c-sharp unit-testing crawler spider csharp parsing cross-platform web-crawler netcore log4net takes-care flexibility pluggable spiders csharp-library abot netcore2 netstandard20 netcore3 javascript-renderer netstandard21 abot-nuget icrawldecisionmaker netsta

Updated Aug 31, 2020
C#

hu17889 / go_spider

Star

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

go crawler spider schedule pipeline

Updated Nov 16, 2017
Go

xianhu / PSpider

Star

简单易用的Python爬虫框架，QQ交流群：597510560

python crawler multi-threading spider multiprocessing web-crawler proxies python-spider web-spider

Updated Mar 3, 2020
Python

wkunzhi / Python3-Spider

Star

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

python crawler spider selenium crawl scrapy splash geek taobao scrapy-crawler meituan dianping pyppeteer

Updated Jul 24, 2020
Python

JayBizzle / Crawler-Detect

Star

Open

Potential bots

3

JayBizzle commented Jan 21, 2020

Filestack
Google-Ads-Overview Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Google-Ads-Overview Mozilla/5.0 (Linux; U; Android 6.0.1; generic) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Version/4.0 Mobile Safari/537.36
Google-Ads-Overview Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleW

Read more

good first issue

howie6879 / ruia

Star

Async Python 3.6+ web scraping micro-framework based on asyncio（Python3.6+异步爬虫框架）

crawler spider aiohttp asyncio uvloop crawling-framework asyncio-spider ruia

Updated Aug 15, 2020
Python

Improve this page

Add a description, image, and links to the spider topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the spider topic, visit your repo's landing page and select "manage topics."

You can’t perform that action at this time.