The Wayback Machine - http://web.archive.org/web/20210725082954/https://github.com/topics/scraping
Skip to content
#

scraping

Star

Here are 2,715 public repositories matching this topic...

Language: All
Filter by language
All 2,715 Python 1,320 JavaScript 372 Jupyter Notebook 235 HTML 125 Go 88 Java 80 PHP 78 TypeScript 76 Ruby 52 C# 50
Sort: Best match
Sort options
Best match Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated
scrapy

scrapy / scrapy

Star
Find more good first issues →

psf / requests-html

Star
Open

Make pyppeteer use proxies

9
oldani
oldani commented Feb 18, 2019

If you're using proxies with requests-html and rendering JS sites is all good. Once you render a website pyppeteer don't know about this proxies and will expose your IP. This is an undesired behavior when scraping with proxies.

The idea is that whenever someone passes in proxies to the session object or any method call, make pyppeteer also use these proxies. #265

Read more
enhancement good first issue
apify-js

apify / apify-js

Star
Open

Update main examples to include DOM manipulation

1
mtrunkat
mtrunkat commented Sep 17, 2019

Main examples at Apify SDK webpage, Github repo and CLI templates should demonstrate how to manipulate with DOM and retrieve data from it.

Also add one example of scraping with Apify SDK + jQuery to https://sdk.apify.com/docs/examples/basiccrawler

Feedback from: https://medium.com/better-programming/do-i-need-python-scrapy-to-build-a-web-scraper-7cc7cac2081d

I lost an hour trying to make

Read more
good first issue
Find more good first issues →

kevinzg / facebook-scraper

Star
Open

How use "Optional parameters"

2
jeverah
jeverah commented Feb 16, 2021

Hi,

I have read that it is possible to write extra parameters.
Where exactly do I do that?
Do I create a new file or can I add a line to my command in the terminal? Like --extra_info true?

### Optional parameters
*(For the 'get_posts' function)*.

- **group**: group id, to scrape groups instead of pages. Default is `None`.
- **pages**: how many pages of posts to request, usua
Read more
good first issue

Improve this page

Add a description, image, and links to the scraping topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the scraping topic, visit your repo's landing page and select "manage topics."

Learn more