Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
A node.js powered scrapper
It works on CSR pages (React, Angular) with dynamic urls.
Once it is done it generates a sitemap.xml file with all the urls found, ready to be uploaded to Google Search Console.
Usage:
$ sitemap https://vvlog.devParams:
| Parameter | type | default | description |
|---|---|---|---|
| --wait | integer | 1500 | Specify the time (milliseconds) to wait (So the fetches are completed) before starting to parse the page. |
| --limit | integer | 999999 | Specify the limit of urls to parse before stopping the scrapper. |
Todo:
- Make it a NPM package.
- Make wait time dynamic in response of fetches inside url.
- New params that lets you specify how deep you want to go inside the url.
- Integrate it as part of build process of a create-react-app.
- Clean old code.