eddy/buzz-rss
Watch
1
0
Fork
You've already forked buzz-rss
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

v1.11.0: HTML-scrape-modus — feeds zonder RSS/Atom/JSON als pagina polled

Browse source 
Nieuwe module buzzrss/html_scrape.py (stdlib-only DOM + simpele selectors:
tag/.class/tag.class/#id) die een server-rendered nieuwspagina omzet in
entries. Per feed via feed.scrape_config (JSON; migratie v13): item/title/
link/date(+date_format, NL-maandnamen locale-onafhankelijk)/summary.
Scrape-entries lopen door de bestaande pipeline (OV-filter, titel-dedup,
WhatsApp-forward, run-log). Eerste scrape-feed: CNV Stads- en streekvervoer
(feed 102).
This commit is contained in:
Eddy de Vink 2026-09-12 18:59:14 +02:00
commit 4adfff5b2a
1 changed files with 277 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

9
buzzrss/db.py
View file

@ -68,6 +68,7 @@ class Feed:
error_count: int = 0
etag: str = ""
last_modified: str = ""
scrape_config: str = ""
subscriptions: list[Subscription] = attr.ib(factory=lambda: [])
@ -137,7 +138,7 @@ class DBManager:
async def get_feeds(self) -> list[Feed]:
q = """
SELECT id, url, title, subtitle, link, next_retry, error_count, etag, last_modified,
room_id, user_id, notification_template, send_notice
scrape_config, room_id, user_id, notification_template, send_notice
FROM feed INNER JOIN subscription ON feed.id = subscription.feed_id
"""
rows = await self.db.fetch(q)
@ -153,7 +154,7 @@ class DBManager:
async def get_feeds_by_room(self, room_id: RoomID) -> list[tuple[Feed, UserID]]:
q = """
SELECT id, url, title, subtitle, link, next_retry, error_count, etag, last_modified,
user_id FROM feed
scrape_config, user_id FROM feed
INNER JOIN subscription ON feed.id = subscription.feed_id AND subscription.room_id = $1
"""
rows = await self.db.fetch(q, room_id)
@ -201,7 +202,7 @@ class DBManager:
await conn.executemany(q, records)
async def get_feed_by_url(self, url: str) -> Feed | None:
q = "SELECT id, url, title, subtitle, link, next_retry, error_count, etag, last_modified FROM feed WHERE url=$1"
q = "SELECT id, url, title, subtitle, link, next_retry, error_count, etag, last_modified, scrape_config FROM feed WHERE url=$1"
return Feed.from_row(await self.db.fetchrow(q, url))
async def get_subscription(
@ -209,7 +210,7 @@ class DBManager:
) -> tuple[Subscription | None, Feed | None]:
q = """
SELECT id, url, title, subtitle, link, next_retry, error_count, etag, last_modified,
room_id, user_id, notification_template, send_notice
scrape_config, room_id, user_id, notification_template, send_notice
FROM feed LEFT JOIN subscription ON feed.id = subscription.feed_id AND room_id = $2
WHERE feed.id = $1
"""