eddy/buzz-rss
Watch
1
0
Fork
You've already forked buzz-rss
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

v1.11.0: HTML-scrape-modus — feeds zonder RSS/Atom/JSON als pagina polled

Browse source 
Nieuwe module buzzrss/html_scrape.py (stdlib-only DOM + simpele selectors:
tag/.class/tag.class/#id) die een server-rendered nieuwspagina omzet in
entries. Per feed via feed.scrape_config (JSON; migratie v13): item/title/
link/date(+date_format, NL-maandnamen locale-onafhankelijk)/summary.
Scrape-entries lopen door de bestaande pipeline (OV-filter, titel-dedup,
WhatsApp-forward, run-log). Eerste scrape-feed: CNV Stads- en streekvervoer
(feed 102).
This commit is contained in:
Eddy de Vink 2026-09-12 18:59:14 +02:00
commit 4adfff5b2a
5 changed files with 277 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

7
buzzrss/migrations.py
View file

@ -144,3 +144,10 @@ async def upgrade_v12(conn: Connection, scheme: Scheme) -> None:
# entry_date_idx diende alleen de oude dedup-query.
await conn.execute("DROP INDEX IF EXISTS entry_date_idx")
await conn.execute("CREATE INDEX entry_posted_at_idx ON entry (posted_at)")
@upgrade_table.register(description="Add HTML scrape config to feeds")
async def upgrade_v13(conn: Connection) -> None:
# scrape_config = JSON met per-feed HTML-selectors voor nieuwspagina's zonder
# RSS/Atom/JSON-feed. Leeg = normale feed. Zie buzzrss/html_scrape.py.
await conn.execute("ALTER TABLE feed ADD COLUMN scrape_config TEXT NOT NULL DEFAULT ''")