Skip to content
FlowHub
FluxonLab
RAG & Knowledge Basesfree

From sitemap crawling to vector storage: Creating an efficient workflow for RAG

by Mariela SlavenovaUpdated Aug 2026
RequiresOpenAISupabaseSupabasePostgresPostgres
Share Post Share
MaWhen clicking ‘Test workflow’When clicking ‘…HRHTTP RequestXmXMLSOSplit OutSILoop Over ItemsWaWaitIfIfDDDefault Data LoaderDefault Data Lo…TSCharacter Text SplitterCharacter Text …Embeddings OpenAIEmbeddings Open…SeEdit FieldsHRCrawl4AI_Task StatusCrawl4AI_Task S…SILoop Over Items1IfIf2SOSplit Out1SeFormat the URLCheck if the URL is in the Supabase TableCheck if the UR…CoFormat the Output from the Supabase nodeFormat the Outp…IfIf "shouldInsert" is trueIf "shouldInser…URL in a new rowCREATE TABLE scrape_queue in SupabaseCREATE TABLE sc…CREATE TABLE scrape_queue in Supabase1CREATE TABLE sc…HRCrawl4ai Web Page ScrapeCrawl4ai Web Pa…CoRemove redundant data from the scrapingRemove redundan…VSSupabase Vector Store_documentsSupabase Vector…Get a row - scrape_queue TableGet a row - scr…Update a row in scrape_queue TableUpdate a row in…Update a row in scrape_queue Table1Update a row in…WaWait1CoQuality Filter NodeQuality Filter …CoContent Type DetectionContent Type De…CoBetter Metadata ExtractionBetter Metadata…IfIf1Update a row in scrape_queue Table2Update a row in…CoTask_id Counter12345678910111213141516171819202122232425262728293031323334353637383940
1/5
FLOWS
STEPS · 40
Run manually by an operator

On manual trigger, crawls a sitemap via HTTP, splits and embeds the content with OpenAI, and stores vectors in Supabase.

Tags

manualadvancedOpenAISupabaseMarket ResearchAI RAGdiscoveredpending-review
Connects
OpenAIsupabaseSupabasepostgresPostgres
CategoryRAG & Knowledge Bases
Triggermanual
Complexityadvanced
Nodes35
AddedJun 27, 2026