Paddy/Betterzon
1
0
Fork 0
mirror of https://github.com/Mueller-Patrick/Betterzon.git synced 2024-12-23 12:15:11 +00:00
Code Issues Projects Releases Wiki Activity
0a25edd58f
Betterzon/Crawler/requirements.txt

7 lines
80 B
Plaintext
Raw Normal View History

BETTERZON-49: Creating Module for Crawler
2021-04-01 08:34:35 +00:00
pymysql
BETTERZON-58 (#53) * BETTERZON-58: Basic Functionality with scrapy * Added independent crawler function, yielding price * moved logic to amazon.py * . * moved scrapy files to unused folder * Added basic amazon crawler using beautifulsoup4 * Connected Api to Crawler * Fixed string concatenation for sql statement in getProductLinksForProduct * BETTERZON-58: Fixing SQL insert * BETTERZON-58: Adding access key verification * BETTERZON-58: Fixing API endpoint of the crawler - The list of products in the API request was treated like a string and henceforth, only the first product has been crawled * Added another selector for price on amazon (does not work for books) Co-authored-by: root <root@DESKTOP-ARBPL82.localdomain> Co-authored-by: Patrick Müller <patrick@mueller-patrick.tech> Co-authored-by: Patrick <50352812+Mueller-Patrick@users.noreply.github.com>
2021-05-18 22:46:14 +00:00
flask==1.1.2
BETTERZON-49 (#24) * BETTERZON-49: Creating Dockerfile * BETTERZON-49: Added minimal Flask API as Docker container Co-authored-by: Patrick Müller <patrick@mueller-patrick.tech>
2021-04-07 21:34:08 +00:00
flask-sqlalchemy
BETTERZON-57: Adding utility sql functions
2021-04-13 19:10:02 +00:00
flask_restful
BETTERZON-58 (#53) * BETTERZON-58: Basic Functionality with scrapy * Added independent crawler function, yielding price * moved logic to amazon.py * . * moved scrapy files to unused folder * Added basic amazon crawler using beautifulsoup4 * Connected Api to Crawler * Fixed string concatenation for sql statement in getProductLinksForProduct * BETTERZON-58: Fixing SQL insert * BETTERZON-58: Adding access key verification * BETTERZON-58: Fixing API endpoint of the crawler - The list of products in the API request was treated like a string and henceforth, only the first product has been crawled * Added another selector for price on amazon (does not work for books) Co-authored-by: root <root@DESKTOP-ARBPL82.localdomain> Co-authored-by: Patrick Müller <patrick@mueller-patrick.tech> Co-authored-by: Patrick <50352812+Mueller-Patrick@users.noreply.github.com>
2021-05-18 22:46:14 +00:00
beautifulsoup4
requests
lxml
Reference in New Issue Copy Permalink