The Wayback Machine - http://web.archive.org/web/20220605210238/https://github.com/topics/scraping
Skip to content
#

scraping

Star

Here are 3,522 public repositories matching this topic...

Language: All
Filter by language
All 3,522 Python 1,730 JavaScript 449 Jupyter Notebook 349 HTML 162 TypeScript 121 Go 98 PHP 95 Java 86 C# 65 Ruby 50
Sort: Best match
Sort options
Best match Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

psf / requests-html

Star 12.6k
Open

Make pyppeteer use proxies

11
oldani
oldani commented Feb 18, 2019

If you're using proxies with requests-html and rendering JS sites is all good. Once you render a website pyppeteer don't know about this proxies and will expose your IP. This is an undesired behavior when scraping with proxies.

The idea is that whenever someone passes in proxies to the session object or any method call, make pyppeteer also use these proxies. #265

Read more
enhancement good first issue
ferret

MontFerret / ferret

Star 5k
Open

Documentation: add examples of queries made using `XPATH`

3
ngirard
ngirard commented Mar 31, 2021

Unless I missed something, the documentation doesn't explain how to query document metadata (searching "site:montferret.dev metadata" through Google returned nothing, neither did grepping the source code).

As an example, I tried to query the og:url metadata.
I tried variations of //meta[property='og:url']::attr(content), with or without the leading //, and with or without the `attr(conte

Read more
help wanted good first issue type/documentation
Find more good first issues
apify-js

apify / apify-js

Star 3.3k
Open

Update main examples to include DOM manipulation

1
mtrunkat
mtrunkat commented Sep 17, 2019

Main examples at Apify SDK webpage, Github repo and CLI templates should demonstrate how to manipulate with DOM and retrieve data from it.

Also add one example of scraping with Apify SDK + jQuery to https://sdk.apify.com/docs/examples/basiccrawler

Feedback from: https://medium.com/better-programming/do-i-need-python-scrapy-to-build-a-web-scraper-7cc7cac2081d

I lost an hour trying to make

Read more
good first issue
Find more good first issues

aapatre / Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE

Star 2.7k
Open

[FEATURE REQUEST] Add architecture.md

3
fakeid30
fakeid30 commented Feb 7, 2021

Is your feature request related to a problem? Please describe.
A new contributor will feel overwhelmed when they will try to contribute to this project.

Describe the solution you'd like
Add a architecture.md as described in this blog post

Describe alternatives you've considered
Not having architecture.md.

**Add

Read more
enhancement help wanted good first issue

Improve this page

Add a description, image, and links to the scraping topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the scraping topic, visit your repo's landing page and select "manage topics."

Learn more