In this blog post, I will share how we\u2019re seeding and enriching taxonomies for e-commerce websites<\/strong> using natural language processing<\/a>.<\/p>\r\n\r\n\r\n\r\n In general with retail products, improving a graph is much more difficult. We typically don\u2019t deal with clearly defined entities but rather we work with unbounded attributes – lenses for glasses, materials for furniture, and so on. Let\u2019s first review a few concepts and some helpful SEO best practices for creating category pages. This will help us understand the workflow and how we can fine-tune it for our needs.<\/p>\r\n\r\n\r\n\r\n ?\u200d? Want to jump right to the Colab? It’s <\/strong>here<\/strong><\/a>!<\/strong><\/p>\r\n\r\n\r\n\r\n What is a product taxonomy?<\/a><\/p>\r\n\r\n\r\n\r\n SEO best practices for creating and optimizing category pages<\/a><\/p>\r\n\r\n\r\n\r\n Overview of the workflow to automate the creation of subcategories<\/a><\/p>\r\n\r\n\r\n\r\n Conclusions and future work<\/a><\/p>\r\n\r\n\r\n\r\n A taxonomy for products acts like shelves in a supermarket. It is the structure that organizes all products in the most accessible way so that customers can easily find them with the least cognitive effort <\/strong>(and the least amount of clicks). Generally, a product taxonomy is a combination of hierarchies of categories and subcategories and product attributes that are specific for each category (e.g. color<\/a> or size for t-shirts and frame shape for sunglasses).<\/p>\r\n\r\n\r\n\r\n In an ideal world, you want to have clean attributes for every product but in reality, this rarely is the case. Data is sparse, incomplete, and of low quality. Structure sparsity is a very common problem to deal with, you would like to have every item clearly classified but you don\u2019t and only a subset of the catalog has been organized correctly. In general, the knowledge domain is also highly sophisticated and attributes tend to be specific for each category. You also need to deal with sub-types, overlapping elements, and synonyms (i.e. \u201cI call it googles but you call them glasses\u201d).<\/p>\r\n\r\n\r\n\r\n In knowledge organization, taxonomies don\u2019t grow like plants on their own. They are the results of highly specialized human labor and can be created using different approaches: top-down, bottom-up, or a combination of both. We can either rely on domain experts that guide us through the maze of how to turn more abstract concepts like \u201caccessories\u201d into fine-grained subcategories such as \u201caccessories > sunglasses > sports activewear > running\u201d and so on or we can discover these terms by ourselves and playfully combine them by learning how they relate to each other.<\/p>\r\n\r\n\r\n\r\n As anticipated, we want to automate the creation of these taxonomies by leveraging machine learning<\/a> and our understanding of core SEO principles. In general with AI, our main focus is not on the technology that we\u2019re going to use, but rather on how we plan to involve and interact with the humans involved in the process. We want to augment our product knowledge graphs and make them helpful to both, the consumer looking for a product to buy and the merchant that is selling that product.<\/p>\r\n\r\n\r\n\r\n Before reviewing our framework let\u2019s highlight the SEO best practices when creating a product taxonomy. They will help us understand the code and the workflow that we envisioned.<\/p>\r\n\r\n\r\n\r\n We have two options:<\/p>\r\n\r\n\r\n\r\n We can work with both targets in mind but the steps will be different and in general, it greatly helps to go one step at a time. You want to measure the impact and work against measurable results. Doing too many things at once never helps. <\/p>\r\n\r\n\r\n\r\n There is no golden rule here but you can use your common sense. You want to be relevant for searchers and provide for each new category page a minimum number of products. Establishing the threshold has to do with competition and user experience. If your competitors have 100 different products on average it is unlucky that you will rank with just 10 products for the same intent. At the same time setting this threshold is also highly dependent on the user experience. If you are able to provide more details at a first glance and can always bring to the top the products that people love you can still win the game even with fewer products.<\/p>\r\n\r\n\r\n\r\n A textual introduction to the listing is vital. It doesn\u2019t need to be too long (that would simply not work especially on mobile), ideally the user should be able to collapse it but you need to be very informative, personalized, and crystal clear. What are we selling? To whom are we selling (what is the target audience)? And what we want people to remember about these products. We have been also successful in demonstrating tractions (in both traffic and sales) when adding, at the bottom of categories pages, relevant question-answer pairs using FAQ markup.<\/p>\r\n\r\n\r\n\r\n FAQ traffic related to a retail website in the US.<\/em><\/p>\r\n\r\n\r\n\r\n Needless to say, you don\u2019t want to generate traffic in overlap with already existing pages. Quite the opposite, you might want to distribute traffic more evenly across all pages. This basically means that on one side you want to ensure that any new category is not going to overlap with existing pages, and on the other side, you want to ensure that highly-trafficked pages can link to contextually relevant, and less visible, sub-category pages.<\/p>\r\n\r\n\r\n\r\n Check out the example below. The intro text under home > men > bags of a famous luxury fashion store, provides the user with relevant links to more specific (and less trafficked landing pages).<\/p>\r\n\r\n\r\n\r\n This is smart as intents are very well defined (1. \u201cI want to buy a designer bag for man\u201d for the main category page and 2 .\u201cI want to buy a Guggi|Off-White\u2019s|Saint Laurent bag for man\u201d) and we can more evenly distribute the link equity from the main page to the sub-pages. <\/p>\r\n\r\n\r\n\r\n One last note can be added to address concerns on the crawl budget. When creating new pages people are usually afraid of hitting the crawl budget. Now, while it is true that Google tends to be more conservative on indexing the web (also to reduce its carbon footprint) we shall always keep in mind that if you have a million or fewer web pages you shouldn\u2019t worry about a crawl budget. Don\u2019t trust me, read instead what Gary Illyes said on this topic<\/a> some time ago. <\/p>\r\n\r\n\r\n\r\n The basic idea is the following:<\/p>\r\n\r\n\r\n\r\n Finally, choose the terms that will cover the highest number of products<\/strong> as sub-categories. <\/p>\r\n\r\n\r\n\r\n Keyword extraction is a crucial text mining task in our workflow. Provided a search<\/a> query, the extraction algorithm shall identify a set of terms that represent at best the query. Before choosing KeyBERT<\/a>, I have tested: YAKE!, an unsupervised keyword extraction library that supports multilingual content (you can give it a quick try here<\/a>), T Zero,<\/a> a large language model optimized for zero-shot tasks, and GPT-3. For the sake of this tutorial, we will analyze the category for Ray-Ban on the FarFetch website, a well-established British-Portuguese online luxury fashion retail brand. <\/p>\r\n\r\n\r\n\r\n The category for Ray-Ban on FF<\/em><\/p>\r\n\r\n\r\n\r\n We always want to use AI<\/a> to help humans connect and easily find what they are looking for. In other words, we need to use machines to augment human intelligence. To build a taxonomy that makes sense I envisioned the following human-AI interaction. We start by accessing queries from either the Google Search Console or SEMrush. For this tutorial, I have prepared the keyword<\/a> analysis behind the category page of FarFetch\u2019s Ray-Ban sunglasses in Google Drive (here<\/a>). We will read this spreadsheet and extract the queries using BERTopic and KeyBERT. <\/p>\r\n\r\n\r\n\r\n The advantages I found in KeyBERT, for this project, are the following:<\/p>\r\n\r\n\r\n\r\n If you have used SBERT you will immediately find yourself at ease with KeyBERT. The library uses BERT-embeddings and cosine similarity to extract terms from sentences (queries in our case). KeyBERT is a library by Maarten Grootendorst<\/a> and you can immediately experiment with the BERT Keyword extractor<\/a> developed by my dear friend Charly Wargnier ?. <\/p>\r\n\r\n\r\n\r\n I used BERTopic to explore the queries. In the end, you can simply skip the clustering<\/a> part, and directly translate queries into keywords using KeyBERT. The advantage of going through the clustering process is that you will be able to better understand the data. If you don\u2019t have enough knowledge of the domain I would suggest going through this step. <\/p>\r\n\r\n\r\n\r\n There is another reason for clustering the data, we might want to analyze how queries change over time (this is particularly useful when products are affected by seasonality).<\/p>\r\n\r\n\r\n\r\n As anticipated we will also use another well-known library by Maarten<\/a> called BERTopic for the clustering of the terms. BERTopic also is based on SBERT.<\/p>\r\n\r\n\r\n\r\n BERTopic<\/a> is well documented and you can experiment with various settings. What I ended up using is the seed_topic_list parameter to have the model converge around a list of seed terms. <\/p>\r\n\r\n\r\n\r\n Clusters are visualized in a two-dimensional diagram<\/em>.<\/p>\r\n\r\n\r\n\r\n You can use the various visualizations provided by the library to see the queries behind the page. If we look at the diagram below, while not perfectly organized, we can spot some potential subcategories worth creating. <\/p>\r\n\r\n\r\n\r\n A barchart of the clusters.<\/em><\/p>\r\n\r\n\r\n\r\n The iconic wayfarer versus the classic aviators but also the shapes of the frames (round, circle glasses, etc.) seem to be valuable subcategories to be created. <\/p>\r\n\r\n\r\n\r\n\r\n
\r\n
\r\n
\r\n
\r\n
What is a product taxonomy?<\/h2>\r\n\r\n\r\n\r\n
What are the main challenges in creating a product taxonomy?<\/h3>\r\n\r\n\r\n\r\n
How to create a product taxonomy?<\/h3>\r\n\r\n\r\n\r\n
SEO best practices for creating and optimizing category pages<\/h2>\r\n\r\n\r\n\r\n
1. Having a clear goal in mind, SEO needs to be measurable<\/h3>\r\n\r\n\r\n\r\n
\r\n
2. Setting a minimum number of products for new categories<\/h3>\r\n\r\n\r\n\r\n
3. Working on content that helps<\/h3>\r\n\r\n\r\n\r\n
![](\"https:\/\/lh5.googleusercontent.com\/jyr9_7sq5u9ZR6d7H6UXWqMWhKHr34edBhN01B2edvAicHrNawxySKmdN-oyCG6-eQzs-B2x3iI0jYuBHNGPzsjoeS9eysRcxfYYHWx1mlorlR-cul3Kvrk0zYPqGDJQTwxKeqDT\")
4. Preventing keyword cannibalization while improving internal links<\/h3>\r\n\r\n\r\n\r\n
![](\"https:\/\/lh6.googleusercontent.com\/uVMU6KXtUF07wiQvdvVAM-KMsvz8L7N1OVR_u3l57vePalYTHsbjpyHYVDB0bpmk33gUvkjPJuSHqcpLay1_YfiFpViy2iFBqMnYhTRagP4qm1K1QVUacsV2TUzRCbmc5uRp0Eqb\")
Overview of the workflow to automate the creation of subcategories<\/h2>\r\n\r\n\r\n\r\n
\r\n
![\"\"](\"https:\/\/wordlift.io\/blog\/en\/wp-content\/uploads\/sites\/3\/2022\/02\/Schermata-2022-02-08-alle-14.13.31.png\")
<\/figure>\r\n\r\n\r\n\r\nKeyword extraction techniques<\/h3>\r\n\r\n\r\n\r\n
![](\"https:\/\/lh5.googleusercontent.com\/YgXOZH5inGAtT1lgrMZZEpd_Wk97IxYcAK6ET8w7hfepPsnAwfJNmc6cVvIfnjH8wJjUXfXOmYu5kUZQYFk5FZUVCUvKUU0Tl0hyWWK_S1E333Urs8SLT7wCpQ9Lw1El9_AVcSpz\")
1. Analyzing search demand<\/h3>\r\n\r\n\r\n\r\n
KeyBERT<\/h4>\r\n\r\n\r\n\r\n
![](\"https:\/\/lh3.googleusercontent.com\/Pqg-t0rnE1mqJ5bIqesIdlVd4URSXfdKokMh6V2qrw-q5WdVYZfj8miB1um0tHjDhvsEe-S_BA-zAGtW4xBbTk89rrLKUPAZNWdxlkJ9eKXhLdurSDz0OQVc5GsxOmEFUp9T-fmv\")
\r\n
2. Clustering queries (optional)<\/h3>\r\n\r\n\r\n\r\n
Topic modeling<\/h4>\r\n\r\n\r\n\r\n
![](\"https:\/\/lh3.googleusercontent.com\/UjvanHO5lHylqxssY9qKsSIkI56fsjMf1B-AnKTQdNGreRyMkkeObUemJLHy1BngfXaLaugEXmME5PxOePAfijJ8HAYfDqqBetmwP7WY_aFiG6nwjJQY8zXGuP0kBpfTy03QWpzq\")
![](\"https:\/\/lh3.googleusercontent.com\/ndBwczSufJfnQf1lQUBAvic347aGU30HxsRoKORoCvniIyqpaFVhLN6XJxl3Sga36cBGrHPYKRqfbkvVKk5j6Ue74hy7AYnLg61WzIih8Iv2tvh4no9us4jA0siWjzPEIwUZvd7x\")
![](\"https:\/\/lh3.googleusercontent.com\/eFBzJ8UGEyhl4Hb44WhwTJFxAODF_NGEqtS7hEFs4nd_Y9Sma5thn_ROYqn60yoXBRj5Aib01Vx9ynJL_PScF325cg-HTpAEzikvwLN4SKhhUGPbdkY8OSZ8celriQr2ziAC8fZV\")
![](\"https:\/\/lh4.googleusercontent.com\/z1fSMao6mA1z6xTaA8GhDxHjPm3MEa08rD0-p8bb4P3yBhe5GNEmJew0nFZmhWYcdMMDyWGy5x4QI2yQyZrn0WsgyXtDZD0uRUudXH0asqQPV9Svpc1UxvWjMwh5NF9_gQqUzTai\")
<\/td>![](\"https:\/\/lh5.googleusercontent.com\/y2spabZHyKQ_RzMzt0uq8cDTnlXuv9A-HUTUmN6LNDHkSGxPqAb2nNXYHVadNgTJPweAdWNGbzg5LpfjlPK87xLC84_zjMJN_E4hEI8R8mg1WEDZVhKloJPSjrtZAhmOmxHZovN_\")
<\/td><\/tr>![](\"https:\/\/lh5.googleusercontent.com\/dxjslq50yRQhBysBj9026xgoZNw8_TkZTN4UusU1YZSZRr5rY5IXpVcsBFpuyOyzenRGDX2i-POUL9qHs_iP6ByfY7bR6jZq_3-nu8vqW-pBeemY6tfzqXW_l7oAOmUtQ1qB1hvR\")
<\/p>\r\n\r\n\r\n\r\n![](\"https:\/\/lh6.googleusercontent.com\/ugFRdaN0vvo-T_K8Q_1HGK2peRDN3KkJ73VFosd6HaviYG7sknsjUB3xn1WcRWurMHEHHF7w4rqjsNwW85cR-Z0dFCw0yZyN96lk5A5hhko9Uk7u_8wC-iu78j2gsWTLsJMmVu4z\")
<\/p>\r\n\r\n\r\n\r\n![](\"https:\/\/lh3.googleusercontent.com\/0A6bkYg6z4hETGgnHzfHJpdb1X3SH97G5UPM9O5eIFyE1cZSrSAATJPGDaoPgb8-wGy2ctvCSGwgS64eUEpheRRrPXIZU3ovSf1PgxyAjDo3BFUJx2BZTTbcNMFYG7SnCJr2rgXu\")
<\/p>\r\n\r\n\r\n\r\n![](\"https:\/\/lh3.googleusercontent.com\/DZ2XuIu_XA0PptzvsPCokgw8oWrhl6puwlO0hQnnXlOCB8VbIDlEy3wJeAi5j9crYTJNRn5ER0efYzG-EC79OPCfjqE5M_7aJjPTib2N8WySLqszvI3D8PszdxSMT14_nFCcCFRS\")
<\/p>\r\n\r\n\r\n\r\n![](\"https:\/\/lh5.googleusercontent.com\/CcMY_-SSok5F7IKbTvfxLQ0EJeJ5eOyoZxhv6zeOVTcSIxBTEtfotPM4SYzBKLXDh1xIFU_04k_6HlPbV6VlJXdvT1M-SZj8ujUsUdUqleiqWAUWQXqhXdqVOLOeNuJ6DQESl_MD\")
<\/p>\r\n\r\n\r\n\r\n![](\"https:\/\/lh4.googleusercontent.com\/WfDfvuBVvl8TXDZbI-nHuWd5UYmNWoQ5xfqk0YrNnHbrLvN0QK2_05lVh32lmYBgtcJhwgfmhzG4Mi59FFgL8xIyXcI6bfjsZ60qi2b0j_5bBU7MBG7kmEyQyt8_9EjdReJD_al3\")
<\/p>\r\n\r\n\r\n\r\n