Here at WordLift, we primarily focus on structured data and improving the data quality of content knowledge graphs. We depend on crawling to cope with missing and messy data on various use cases. <\/p>\n\n\n\n
Extracting Structured Data from Web Pages using Large Language Models<\/h2>\n\n\n\n
Recently, I\u2019ve been exploring the potential of OpenAI function calling<\/strong> for extracting structured data<\/a> from web pages<\/strong>. This could be a game-changer for those who, like us, are actively looking to synergize Large Language Models (#LLMs) with Knowledge Graphs<\/a> (#KGs).<\/p>\n\n\n\n Why is this exciting? Because the integration of LLMs with KGs is fast becoming a hot topic in tech, and developing a unified framework that can enrich both LLMs and KGs simultaneously is of significant importance.<\/p>\n\n\n\n By using this Colab Notebook<\/a>, you can extract entity attributes from a list of URLs – even from pages built in JavaScript! I used in this implementation the schema for LodgingBusiness<\/a> (hotels, b&b and resorts).<\/p>\n\n\n\n A few lessons learned from this exploration:<\/p>\n\n\n\n ScrapeGraphAI – the New Frontier in Web Scraping<\/strong><\/p>\n\n\n\n I have recently discovered a new fantastic library for AI scraping called ScrapeGraphAI<\/strong><\/a>. This Python library uses LLM and direct graph logic<\/strong> to create scraping pipelines for websites and any type of document (XML, HTML, JSON, etc.). <\/p>\n\n\n\n This library – at a first glance – has proven to be powerful, adapting seamlessly to various web pages, which prompted me to update the Streamlit<\/a> web application<\/strong> that you can now immediately use. <\/p>\n\n\n\n Jump to the web application here [using now ScrapeGraphAI] \ud83c\udf88<\/a><\/strong><\/p>\n\n\n\n This is a demonstrative web app<\/strong>. The UI is a bit clunky when you start refining rules, and in general, it is limited to crawling only a few URLs<\/strong>. If you are looking for something that scales, I would recommend Advertools,<\/strong><\/a> a well-known python library developed by the mythical Elias Dabbas<\/a><\/strong>. <\/p>\n\n\n\n If you want to see how you can use it, watch this webinar. Here, Elias Dabbas and Doreid Haddad show how to build a Knowledge Graph using Advertools and WordLift<\/strong>. <\/p>\n\n\n\n\n
![\"\"](\"https:\/\/wordlift.io\/blog\/en\/wp-content\/uploads\/sites\/3\/2024\/05\/download-2024-05-10T184409.188.png\")
<\/figure>\n\n\n\nHere is how the scraping app works <\/h2>\n\n\n\n
\n
![\"\"](\"https:\/\/wordlift.io\/blog\/en\/wp-content\/uploads\/sites\/3\/2024\/05\/web-scraping-seo.jpg\")
<\/figure>\n\n\n\nExisting limitations<\/h2>\n\n\n\n