How to Use rs-trafilatura with crawl4ai
crawl4ai is an async web crawler built for producing LLM-friendly output. By default, it converts pages to Markdown using its own scraping pipeline. But if you want page-type-aware content extracti...

Source: DEV Community
crawl4ai is an async web crawler built for producing LLM-friendly output. By default, it converts pages to Markdown using its own scraping pipeline. But if you want page-type-aware content extraction with quality scoring, you can swap in rs-trafilatura as the extraction strategy. This tutorial shows how to set that up. Install pip install rs-trafilatura crawl4ai If this is your first time with crawl4ai, you also need Playwright browsers: python -m playwright install chromium Basic Usage rs-trafilatura provides RsTrafilaturaStrategy, a drop-in replacement for crawl4ai's built-in extraction strategies: import json import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig from rs_trafilatura.crawl4ai import RsTrafilaturaStrategy async def main(): strategy = RsTrafilaturaStrategy() config = CrawlerRunConfig(extraction_strategy=strategy) async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://example.com", config=config) data = json.loads(result.extract