mirror of
https://github.com/Mueller-Patrick/Betterzon.git
synced 2025-11-04 02:25:48 +00:00
26ba21156a
* BETTERZON-58: Basic Functionality with scrapy * Added independent crawler function, yielding price * moved logic to amazon.py * . * moved scrapy files to unused folder * Added basic amazon crawler using beautifulsoup4 * Connected Api to Crawler * Fixed string concatenation for sql statement in getProductLinksForProduct * BETTERZON-58: Fixing SQL insert * BETTERZON-58: Adding access key verification * BETTERZON-58: Fixing API endpoint of the crawler - The list of products in the API request was treated like a string and henceforth, only the first product has been crawled * Added another selector for price on amazon (does not work for books) Co-authored-by: root <root@DESKTOP-ARBPL82.localdomain> Co-authored-by: Patrick Müller <patrick@mueller-patrick.tech> Co-authored-by: Patrick <50352812+Mueller-Patrick@users.noreply.github.com>
13 lines
263 B
Python
13 lines
263 B
Python
# Define here the models for your scraped items
|
|
#
|
|
# See documentation in:
|
|
# https://docs.scrapy.org/en/latest/topics/items.html
|
|
|
|
import scrapy
|
|
|
|
|
|
class CrawlerItem(scrapy.Item):
|
|
# define the fields for your item here like:
|
|
# name = scrapy.Field()
|
|
pass
|