Overview
The Trix Web Crawler is a core feature that allows you to import web content directly into your memory store. Crawl entire websites or fetch individual pages, with automatic content extraction and markdown conversion.
Multi-Page Crawling
Crawl entire websites with configurable depth, link discovery, and domain filtering.
Clean Extraction
Mozilla Readability extracts article content, removing navigation, ads, and boilerplate.
Robots.txt Compliant
RFC 9309 compliant robots.txt parsing with crawl-delay support and sitemap discovery.
Markdown Output
Content is converted to clean markdown, preserving structure for LLM consumption.
Quick Start
Import a single page as a memory:
import { Trix } from '@trixdb/client';
const client = new Trix({ apiKey: 'your_api_key' });
// Fetch a single page
const result = await client.crawler.fetch({
url: 'https://example.com/article',
tags: ['documentation', 'imported']
});
console.log('Imported:', result.memory.id);
console.log('Title:', result.source.title);API Endpoints
POST /v1/crawler/fetch
Fetch and import a single web page as a memory.
| Parameter | Type | Required | Description |
|---|---|---|---|
url | string | Yes | URL to fetch (must be a valid URI) |
spaceId | string | No | Space to save the memory to |
tags | string[] | No | Tags to add to the imported memory |
respectRobotsTxt | boolean | No | Whether to respect robots.txt rules (default: true) |
Response
{
"success": true,
"memory": {
"id": "mem_abc123",
"content": "# Article Title\n\nArticle content in markdown...",
"content_type": "text",
"metadata": {
"source": "web",
"url": "https://example.com/article",
"title": "Article Title",
"author": "John Doe"
},
"tags": ["documentation", "imported"],
"created_at": "2024-01-15T10:30:00Z"
},
"source": {
"url": "https://example.com/article",
"title": "Article Title",
"excerpt": "A brief description of the article...",
"author": "John Doe",
"language": "en"
}
}Multi-Page Crawling
POST /v1/crawler/crawl
Start a multi-page website crawl. The crawler will discover and follow links, respecting the configured limits.
// Crawl a documentation site
const result = await client.crawler.crawl({
seedUrls: ['https://docs.example.com'],
maxDepth: 3,
maxPages: 50,
allowedDomains: ['docs.example.com'],
tags: ['documentation']
});
console.log(`Imported ${result.memories.length} pages`);
result.memories.forEach(memory => {
console.log(` - ${memory.title}`);
});Crawl Parameters
| Parameter | Type | Required | Description |
|---|---|---|---|
seedUrls | string[] | Yes | URLs to start crawling from (1-10 URLs) |
maxDepth | integer | No | Maximum link depth to follow (0-10, default: 3) |
maxPages | integer | No | Maximum pages to crawl (1-500, default: 100) |
allowedDomains | string[] | No | Restrict crawling to these domains only |
excludePatterns | string[] | No | URL patterns to skip (regex) |
crawlDelay | integer | No | Delay between requests in ms (100-60000, default: 1000) |
respectRobotsTxt | boolean | No | Follow robots.txt rules (default: true) |
followLinks | boolean | No | Discover and follow links (default: true) |
spaceId | string | No | Space to save memories to |
tags | string[] | No | Tags to add to all imported memories |
Crawl Response
{
"status": "completed",
"message": "Crawl completed. Imported 42 pages.",
"stats": {
"pages_crawled": 42,
"pages_failed": 3,
"total_content_length": 284500,
"duration_ms": 45230
},
"memories": [
{ "id": "mem_abc1", "url": "https://docs.example.com/", "title": "Home" },
{ "id": "mem_abc2", "url": "https://docs.example.com/getting-started", "title": "Getting Started" }
],
"config": {
"seedUrls": ["https://docs.example.com"],
"maxDepth": 3,
"maxPages": 50
}
}Preview Extraction
POST /v1/crawler/preview
Preview what content will be extracted from a URL without saving it as a memory. Useful for testing before importing.
// Preview extraction without saving
const preview = await client.crawler.preview({
url: 'https://example.com/article'
});
console.log('Title:', preview.preview.title);
console.log('Content length:', preview.preview.content_length, 'chars');
console.log('Preview:', preview.preview.content_preview.slice(0, 200), '...');Preview Response
{
"success": true,
"preview": {
"url": "https://example.com/article",
"title": "Article Title",
"excerpt": "A brief description of the article content...",
"author": "John Doe",
"language": "en",
"content_length": 4523,
"content_preview": "# Article Title\n\nThis is the beginning of the article content...",
"crawled_at": "2024-01-15T10:30:00Z"
}
}Crawler Features
Content Extraction
The crawler uses Mozilla Readability to extract article content, automatically removing:
- Navigation menus and headers
- Sidebars and advertisements
- Footer content and boilerplate
- Scripts and styling
Markdown Conversion
Extracted HTML is converted to clean Markdown using Turndown, preserving:
- Headings and structure
- Lists (ordered and unordered)
- Links and images
- Code blocks and inline code
- Bold, italic, and other formatting
Trap Detection
The crawler includes intelligent trap detection to avoid infinite crawling:
- Calendar traps: Detects calendar navigation that generates infinite date URLs
- Session IDs: Identifies and skips URLs with session tokens
- Infinite patterns: Detects repeating path segments and overly deep URLs
Rate Limiting
The crawler automatically respects rate limits:
- Configurable delay between requests (default: 1000ms)
- Honors
Crawl-delaydirective from robots.txt - Per-domain rate limiting when crawling multiple domains
Robots.txt Compliance
The crawler follows the RFC 9309 specification
for robots.txt parsing. When respectRobotsTxt is enabled (default), the crawler will:
- Fetch and cache robots.txt per domain
- Respect
User-agentrules forTrixBotand* - Honor
DisallowandAllowdirectives - Follow
Crawl-delayspecifications - Discover sitemaps from
Sitemapdirectives
The crawler identifies itself as TrixBot/1.0 (+https://trixdb.com/bot). Site owners can use
this user-agent to create specific rules for Trix.
Best Practices
Domain Filtering
Always specify allowedDomains when crawling to prevent the crawler from
following external links to other websites:
{
"seedUrls": ["https://docs.example.com"],
"allowedDomains": ["docs.example.com", "example.com"],
"maxPages": 100
}Exclude Patterns
Use excludePatterns to skip URLs that don't contain useful content:
{
"seedUrls": ["https://example.com"],
"excludePatterns": [
"/login",
"/signup",
"/admin/.*",
"\\?page=",
"/tag/.*"
]
}Crawl Delay
Be a good web citizen. Use appropriate crawl delays to avoid overloading servers:
- Public documentation: 500-1000ms delay
- Personal/small sites: 2000-5000ms delay
- Large enterprise sites: May allow faster crawling
Start Small
When crawling a new site, start with low limits to test the extraction quality:
Step 1: Test with 5 pages first
{
"seedUrls": ["https://docs.example.com"],
"maxDepth": 1,
"maxPages": 5
}Step 2: If results look good, increase limits
{
"seedUrls": ["https://docs.example.com"],
"maxDepth": 3,
"maxPages": 100
}Use Cases
Documentation Import
Crawl documentation sites to create a searchable knowledge base. Your AI can then answer questions based on the imported docs.
Research Collection
Import articles, blog posts, and research papers. Organize with tags and search semantically across all content.
Competitive Intelligence
Track competitor content and industry news. Keep your AI updated with the latest public information.
Content Archival
Archive important web content before it disappears. Convert to clean markdown for long-term storage.
Limits
| Limit | Value | Notes |
|---|---|---|
| Max seed URLs | 10 | Per crawl request |
| Max pages per crawl | 500 | Configurable via maxPages |
| Max depth | 10 | Link depth from seed URLs |
| Min crawl delay | 100ms | Between requests to same domain |
| Request timeout | 30s | Per page fetch |