Automatic Web Crawler w/ OpenAI

Workflow Analysis: Autonomous AI Web Crawler Overview Brief This workflow automates the process of finding and extracting social media profile links from a list of company websites. It begins by fetching company names and their corresponding websites from a database. For each company, it deploys an autonomous AI agent to intelligently crawl the website, locate social media links, and structure the findings. The final, enriched data—containing the company name, website, and a structured list of their social media profiles—is then saved back into a database. This automation eliminates the highly manual and time-consuming task of lead enrichment and data gathering, allowing for the rapid processing of large lists of companies without human intervention. Technologies Used - n8n: The core automation platform coordinating the entire process. - Supabase: A PostgreSQL-based database used for both fetching the initial list of companies and storing the final, enriched data. - OpenAI: The AI provider whose gpt-4o model acts as the intelligent agent for crawling. - LangChain: The framework used (via n8n's integrated nodes) to build the AI agent, provide it with tools, and manage its interaction with the websites. - Web Scraping (HTTP, HTML Parsing): Core methods used by the AI's tools to fetch and extract data from web pages. How AI Integrates into the Flow The AI is the centerpiece of this workflow, acting as an intelligent, autonomous agent rather than a simple data processor. Here’s how it's integrated: 1. Autonomous Decision-Making: The LangChain Agent node, powered by OpenAI's gpt-4o, is given a primary objective: "Retrieve social media profile URLs." It is not given a rigid, step-by-step set of instructions. Instead, it is equipped with two specialized tools: one for reading text on a page and one for extracting all links. 2. Dynamic Tool Use: The AI agent analyzes the content of a webpage and decides which tool to use next. For example, it might first extract all links. If it sees a link with text like "Contact Us" or "About," it can decide to use its URL-retrieval tool again on that specific link to navigate deeper into the site.