Commit Graph

6 Commits

Author SHA1 Message Date
henningxtro
26ba21156a
BETTERZON-58 (#53)
* BETTERZON-58: Basic Functionality with scrapy

* Added independent crawler function, yielding price

* moved logic to amazon.py

* .

* moved scrapy files to unused folder

* Added basic amazon crawler using beautifulsoup4

* Connected Api to Crawler

* Fixed string concatenation for sql statement in getProductLinksForProduct

* BETTERZON-58: Fixing SQL insert

* BETTERZON-58: Adding access key verification

* BETTERZON-58: Fixing API endpoint of the crawler
- The list of products in the API request was treated like a string and henceforth, only the first product has been crawled

* Added another selector for price on amazon (does not work for books)

Co-authored-by: root <root@DESKTOP-ARBPL82.localdomain>
Co-authored-by: Patrick Müller <patrick@mueller-patrick.tech>
Co-authored-by: Patrick <50352812+Mueller-Patrick@users.noreply.github.com>
2021-05-19 00:46:14 +02:00
henningxtro
9e9b442f35
BETTERZON-58: Basic Functionality with scrapy (#33) 2021-04-28 22:24:13 +02:00
Patrick
f5fd1825d7
BETTERZON-56: Adding crawler load-balancing script (#28) 2021-04-14 18:52:22 +02:00
fafacdd942 BETTERZON-57: Adding utility sql functions 2021-04-13 21:10:02 +02:00
55a019d217
BETTERZON-49 (#24)
* BETTERZON-49: Creating Dockerfile

* BETTERZON-49: Added minimal Flask API as Docker container

Co-authored-by: Patrick Müller <patrick@mueller-patrick.tech>
2021-04-07 23:34:08 +02:00
05d4795f9d BETTERZON-49: Creating Module for Crawler 2021-04-01 10:34:35 +02:00