Multi-Page Web-Scraper Chatbot

Users, researchers, or businesses needing quick answers from multiple websites without manual browsing. This workflow builds an AI chatbot that takes user queries with website URLs, scrapes content from multiple pages, stores chat history, and uses AI to summarize and answer questions clearly. Set up a webhook endpoint in n8n. Connect to Google Gemini API, Postgres for memory, Supabase for message storage, and Jina AI for scraping. Link the sub-workflow as a tool in the AI Agent. n8n account, Google Gemini API key, Postgres and Supabase credentials, active webhook URL. Handles up to 10 URLs per query with rate limiting. Edit the AI prompt for different teaching styles. Add more tools like summarizers. Change scraping URLs or databases for specific needs. Key Technologies: n8n,Google Gemini AI,Postgres Chat Memory,Supabase Database,Jina AI Web Scraping,Webhook Triggers Automation Value: Automates web research across sites, saving hours of manual reading. Provides clear, organized answers with chronological patterns for better learning. Maintains conversation history for ongoing chats without repeating info. Scales to multiple URLs, reducing errors in data collection. Best Practices: Use authentication on webhooks to secure access. Implement retry logic on scraping to handle network issues. Limit context window in memory to manage costs and speed. Validate URLs in prompts to avoid invalid scrapes. Monitor database storage for large chat histories. Test with multiple URLs to ensure loop stability.