llms.txt Generator from Screaming Frog Export

SEO experts, website owners, and content managers who crawl sites with Screaming Frog and want to create llms.txt files for AI content discovery. This workflow takes a Screaming Frog CSV export, extracts key page data, filters for quality indexable pages, optionally uses AI to classify useful content, formats into llms.txt structure, and generates a downloadable file or uploads it to Google Drive. 1. Set up n8n workflow. 2. Provide Screaming Frog credentials if needed (none directly). 3. Trigger via form: input site name, description, and upload internal_html.csv or internal_all.csv. 4. Optional: Configure Google Gemini for AI classification and Google Drive for upload. 5. Run workflow to get llms.txt. - n8n instance. - Screaming Frog CSV export (internal\_html.csv preferred). - Optional: Google Gemini API credentials for AI node. - Optional: Google Drive OAuth for upload. - Enable/disable AI Text Classifier for smarter filtering. - Add custom filters (e.g., word count > 300, specific paths). - Modify row format in Set nodes. - Replace upload with other storage (e.g., Dropbox). - Handle large sites with Loop Over Items to avoid timeouts. Key Technologies: n8n Workflow Automation,Screaming Frog CSV Parsing,Google Gemini AI (for classification),Google Drive Integration,LangChain Text Classifier Automation Value: Automates tedious manual extraction and formatting of crawl data into llms.txt, saving hours of work. AI integration intelligently filters high-value pages, improving llms.txt quality for LLMs without manual review. Handles multi-language column names, making it versatile for global sites. Scalable output: direct download or cloud upload, with easy customization for filters and destinations. Reduces errors in SEO file generation, ensuring only indexable, 200-status HTML pages are included. Best Practices: For large sites (>1000 pages), insert a Loop Over Items node after filtering to handle API quotas and timeouts. Monitor LLM token usage: enable AI only for essential classification to control costs. Test with small CSV first to verify column mappings, especially for non-English exports.