Scraping

Scraping Limitations / Road Blockers:
How many requests can their server take
How long does it take for the server to handle a request
Request/second
How many requests can you parallelise:
From a single process
From multiple process
How do you track what needs to be scraped
Authentication
Watch out for
Password change reminders
Account being locked out
If the page uses JavaScript
Goals:
Maximize number of requests/sec
Less compute resources used
Tools:
Database
Queue
Database:
Avoid nosql
Use a SQL database from the start, since you’ll most likely be exporting/querying it
Easier to change field names
Run SQL queries to fix
One table per “type” of page
One table for the pagination results
Another table for page results
Another table for consolidated results
This can be the source of truth
Hard part may be figuring out what should exist in the consolidated table, but doesn’t
Log to the console:
page/id scraped
time scraped
Time it took to scrape page

Ruby / Ruby on Rails

Slack

Last modified 1yr ago

Copy link