Machine Learning in Action for Search Engine Optimization

Machine Learning in Action for Search Engine Optimization

In this post, I’ll walk through the analysis of Google Search Console data combined with a machine learning clustering technique to provide an indication on what pages can be optimized to improve the organic traffic of a company website. I will also highlight the lessons I learned while using machine learning for an SEO task.

Interestingly, website owners when I propose to use their data are usually very relieved that AI can take care of the mundane, repetitive SEO work like analyzing GSC data; this allows the clients of our SEO management service and our own team, to focus on more complex, value-adding work such as content writing, content enrichment, and monetization.

Machine learning is fun

This experiment is designed for anyone: no specific coding skill is required. A good grip on Google Sheets is more than enough to get you started in the data mining of your website’s GSC data.

We will use Orange, an open source tool built on top of Python, for data mining and analysis that uses a visual programming front-end (a graphical user interface that lets you do what a developer would do using a Jupyter notebook environment and Python, yay!).

You can install Orange from Anaconda, a popular Python data science platform, or simply download it and install it from their website. We will also use data from a web crawler to extract information about the length of the title and the length of the meta description. This can be done using a WooRank account, Sitebulb or any other web crawler of your choosing.  

Stand on the shoulder of giants

Dealing with machine learning is indeed a paradigm shift. The basic idea is that we provide highly curated data to a machine and the machine will learn from this data, it will program itself and it will help us in the analysis by either grouping data points, making predictions or extracting relevant patterns from our dataset. Choosing the data points and curating the dataset, in machine learning, is as strategic as writing the computer program in traditional computer science. By deciding the type of data you will feed the machine you are transferring the knowledge required to train the machine. To do so, you need the so-called domain experts and when I started with this experiment I came across a tweet from Bill Slawski that indicated me the importance of comparing search impressions to clicks on a page as the most valuable piece of data from the Google Search Console.

I also spotted another valuable conversation on the topic between Aleyda Solis and Cyrus Shepard.

By reading this I decided to compile a dataset composed of the following attributes. The first 6 coming from GSC and the other 2 coming out of the crawling of the pages.

The overall idea, as explained by Bill Slawski, is to rewrite the title and the meta description of pages that receive a good number of impression and a low number of clicks.

“Willing to know more about what data is provided by Google Search Console? Read it all here on the WooRank’s Blog.”

As we learned from Aleyda another important aspect to winning the game is to focus only on pages that have already a strong position (between 3 and 5 she says). This is extremely important, as it will speed up the process and bring almost immediate results. Of course, the bracket might be different for smaller websites (in some cases working with pages with a position between 3 and 10 might also be valuable).

How do I get the data from Google Search Console into Google Sheet?

Luckily GSC provides fast and reliable access to your data via APIs, and you can use a Google Sheet Add On called searchanalyticsforsheets.com that automatically retrieve the data and stores it in Google Sheet without writing a line of code. It is free, super simple to use and well documented (kudos for the developing team ?).

If you are more familiar with Python you can also use this script by Stephan Solomonidis on GitHub that would do pretty much the same work with only a few lines of code.

In my dataset, I wanted to have both queries and pages in the same file. A page usually ranks for multiple intents and it is important to see what is the main query we want to optimize for.

How can I merge two datasets in one?

Aggregating data from the crawler with data from GSC can be done directly in Orange using the merge data widget that horizontally combines two datasets by using the page as a matching attribute. I used, instead, Google Sheets with a combination of ARRAYFORMULA (it will run the function on an entire column) and VLOOKUP (this does the actual match and brings both title length and meta description length in the same table).  

=ARRAYFORMULA(VLOOKUP(A2:A,crawl_html!A6:C501,{1,2},false))
ARRAYFORMULA(VLOOKUP(search_key,range,index,[is_sorted]))
  • search_key (the attribute used in the matching)
  • range (the sheet with the data from the crawler)
  • index (the columns from the crawler dataset that we want to import  for the length of the title and of the meta description)   
  • is_sorted (typically set to FALSE since the two tables we’re merging don’t follow the same order)

Prepare data with loving care

Data curation is essential to obtain any valid results with artificial intelligence. Data preparation also is different for each algorithm. Each machine learning algorithm requires data to be formatted in a very specific way and before finding the right combination of column and yield useful insights I did several iterations. Missing data and wrong formatting (when migrating data in Orange in our case) have been issues to deal with. Generally speaking for missing data there are two options, either remove the data points or fill it up with average values (there are a lot more options to consider but this is basically what I did in the various iterations). Formatting is quite straightforward, we simply want Orange to properly see each informative feature as a number (and not as a string).

The dataset

The dataset we’re working with is made of 15784 rows each one containing a specific combination of page and query. We have 3 informative features in the dataset (clicks, impression, and position) and 5 labels (page, query, CTR, title and meta description length). Page and query are categorical labels (we can group the data by the same query or by the same page). CTR is a formula that calculates clicks/impression * 100 and for this reason is not an informative feature. Labels or calculated values are not informative: they don’t help the algorithm in clustering the data. At the same time, they are extremely useful to help us understand and read the patterns in the data.  

Dataset configuration in Orange

Dataset configuration in Orange

Introducing k-Means for clustering search queries

When looking at thousands of combination of queries across hundreds of web pages selecting the pages that have the highest potential in terms of SEO optimization is an intimidating task. This is particularly true when you have never done such analysis before or when you are approaching a website that you don’t know (as we do – in most cases – when we start a project with one new client that is using our technology).

We want to be able to group the combination of pages that can be more easily improved by updating the title and the snippet that describes the article. We also want to learn something new from the data that we collected to improve the overall quality of the content that we will produce in the future. Clustering is a good approach as it will break down the opportunities in a limited number of groups and it will unveil the underlying patterns in the data.

A cluster refers to a collection of data points aggregated together by a certain degree of similarity.

What is k-Means Clustering?

K-Means clustering is one of the simplest and most popular unsupervised machine learning algorithms. It will make inferences using only input features (data points like the numbers of impressions or the number of clicks) without requiring any labeled outcome.

K-Means will average the data by identifying a centroid for each group and by grouping all records in a limited number of cluster. A centroid is the imaginary center of each cluster.  

The pipeline in Orange

Here is how the flow looks like in Orange. We’re importing the CSV data that we have created using the File widget, we’re quickly analyzing the data using the Distribution Widget. We have the k-Means Widget at the center of the workflow that receives data from the Select Rows Widget (this is a simple filter to work only on records that are positioned in SERP between 3 and 10) and  sends the output to a Scatter Plot that will help us visualize the clusters and understand the underlying patterns. On another end, the k-Means sends the data to a Data Table widget that will produce the final report with the list of pages we need to work on and their respective queries. Here we also use a Select Rows widget to bring in our final report only the most relevant cluster.  

The data analysis pipeline in Orange

The data analysis pipeline in Orange

The distribution of rankings.

Here is how the distribution of rankings looks like.

The silhouette score in k-Means helps us understand how similar each combination is to its own cluster (cohesion) compared to other clusters (separation).

The silhouette score ranges from 0 to 1 (a high value indicates that the object is well matched to its own cluster). By using this value the algorithm can define how many clusters we need (unless we specify otherwise) and the level of cohesion of each group. In our case 3 cluster represent the best way to organize our data and to prioritize our work. From the initial 15784 samples (the rows in our dataset) we have now selected 1010 instances (these are all the combination with pages in position 3-10) that have been grouped by k-Means.   

k-Means configuration

k-Means configuration parameters

What is the data telling us

We will use Orange’s intelligent data visualization to find informative projections. In this way, we can see how the data has been clustered. The projections are a list of attribute pairs by average classification accuracy score that shows us the underlying patterns in our dataset. Here are the top 4 I have chosen to evaluate.

1. Focus on high impressions and low CTR and here is the list of pages to optimize

CTR vs Impressions

Scatter Plot #1 – CTR vs Impressions  (the size of the symbols indicates the CTR)

There is no point in working on cluster C1, either there are very little impressions or the CTR is already high. Where it hurts the most is on C3 and following we have C2 cluster.

We have now a total of 56 combinations of pages and queries that really deserve our attention (C2 and C3). Out of this batch, there are 18 instances in C3 (the most relevant group to address) and this basically means working on 16 pages (2 pages are getting traffic from 2 queries each).

The final report with the pages to work on

The final report with the pages to work on

This is the list for our content team to optimize. New titles and improved meta description will yield better results in a few weeks.

2. Positions don’t matter as much as impressions

Scatter Plot #2 - Positions vs Impressions  

Scatter Plot #2 – Positions vs Impressions

Our three clusters are well distributed across all selected positions. We might prefer – unless there are strategic reasons to do otherwise – to improve the CTR of a page with a lower position but a strong exposure rather than improving the clicks on a higher ranking result on a low volume keyword.

3. Write titles with a length between 40 and 80 characters

Google usually displays the first 50–60 characters of a title tag. MOZ research suggests that you can expect about 90% of your titles to display properly when contained under the 60 characters. From the data we gathered we could see that, while the vast majority is working under 60 characters we can still get a healthy CTR with titles up to 78 characters and not shorter than 38 characters.   

Scatter Plot #3 - CTR vs Title Length

Scatter Plot #3 – CTR vs Title Length

4. Write Meta Description with a length between 140 and 160 characters

At the beginning of May last year, the length of meta description on Google has been shortened after the last update in December 2017, when the length was extended up to 290 characters. In other words, Google is still testing various length and if on a desktop it displays 920 pixels (158 characters) on mobile you will see up to 120 characters in most cases.   

Meta description length in 2019 according to blog.spotibo.com

Meta description length in 2019 according to blog.spotibo.com

This means that the correct length is also dependent on the percentage of mobile users currently accessing the website. Once again we can ask the data what should be the preferred number of characters by looking at clusters C2 and C3. Here we can immediately see that the winning length is between 140 and 160 chars (highest CTR = bigger size of the shapes).   

Scatter Plot #4 - CTR vs Meta Description Length

Scatter Plot #4 – CTR vs Meta Description Length

Make Your Website Smarter with AI-Powered SEO: just focus on your content and let the AI grow the traffic on your website!

Courtney McGhee

WooRank

What’s next?

These are really the first steps towards a future where SEOs and marketers have instant access to insights provided by machine learning that can drive a stronger and sustainable growth of web traffic without requiring a massive amount of time in sifting through spreadsheets and web metrics.

While it took a few weeks to set up the initial environment, to test the right combination of features and to share this blog post with you, now to process hundreds of thousands of combinations anyone can do it out in just a few minutes! This is also the beauty of using a tool like Orange that, after the initial setup, requires no coding skills.    

We will continue to improve the methodology while working for our VIP clients, validating the results from these type of analysis and eventually improve our product to bring these results to an increasing number of people (all the clients of our semantic technology).

Keep following us and drop me a line to learn more about AI for SEO!

10 Artificial Intelligence Software for SEO

10 Artificial Intelligence Software for SEO

Artificial Intelligence is all around us. From Siri to Alexa, to Google Home, it’s consuming the age we live in. We have found ourselves relying on a voice in a device to help us with the simplest of tasks. Luckily, content marketers can utilize this advanced technology to assist with search engine optimization techniques.

WordLift has mastered the art of Semantic AI, and we are excited to see this process grow beyond just our company. All over the web, companies are utilizing this to cut down the time and effort needed from SEO specialists, by the click of a button.

We have divulged into the top 10 Artificial Intelligence Search Engine Optimization Software, showing you exactly what makes each unique from the others. Jim Yu, the CEO and founder of Bright Edge, recently released an article in which he divided these SEO tools into three categories:

  • insight,
  • automation and
  • personalization.

We have broken these tools into the corresponding categories to help you understand how you can integrate them into your SEO workflow.

Make Your Website Smarter with AI-Powered SEO: just focus on your content and let the AI grow the traffic on your website!

Courtney McGhee

WooRank

Insight Tools

Bright Edge

Bright Edge is a platform that contains several modules to help content marketers with optimizing their content. The software includes; DataCube, Hyperlocal, Intent Signal, Keyword reporting, Page reporting, Content recommendations, Share of voice, Site reporting and story builder.

The most unique feature is their Hyperlocal add-in. This aspect allows users to map out keywords in a specific region; either a country or city. Bright Edge’s Content Recommendations gives the opportunity to read through precise suggestions on each page. It personalizes each page on your site according to what that specific page contains.

The platform provides a unique way to view how various SEO changes impact the brand. Story builder combines data from several pieces of the website to create aesthetic tables and charts, making it easier to decipher the data.

MarketBrew

This software is unique in how quickly it distributes information to the consumer. MarketBrew provides each company with step by step on-site training, as well as a breezy plan to implement the program. The software prides itself on its search engine modeling, producing information in only 1 and a half hours.

Their process involves coding a base search model, and in turn adjusting it so that it fits your target search engine; to which they claim they can accommodate any search engine. Their machine learns the exact algorithms that include which search engine you are wanting to use. This tool provides the user with a precise description of what distinguishes the first result from the second one; such as the HTML content or even the META description. This cuts off time that a user spends manually analyzes the inner workings of the results.

MarketBrew also conveniently provides the user with exact ways to resolve the issues with your ranking, which can then be tested again within hours. This software overall provides a great visual explanation as well as step-by-step ways to swiftly and resourcefully improve your site.

Can I Rank?

Can I Rank gathers information from various Search Engine Optimization websites, then takes the extra step to elaborate with suggestions. Their artificial intelligence method works with providing the user with data that leads them in the right direction to boosting their content, backing it up with more than 200,000 websites.

Can I Rank offers a keyword difficulty score to allow the user to judge which exact keyword will work for their specific website. The analysis is all done by a machine-learning system that focuses heavily on data as opposed to strict opinions. This website is efficient for those who want that data to back up why they should change and doesn’t leave you clueless on what to adjust.

Overall, Can I Rank lives up to their name by showing users exactly what sets them apart, and what they can do to improve that.

Pave AI

Pave AI is an Artificial Intelligence based tool that turns Google Analytics data into helpful insights to improve your everyday marketing strategy. Its algorithm integrates marketing data from different platforms (such as Adwords, Facebook Ads & Twitter Ads) and analyzes them, making it easy to understand what works and what can be improved.

Pave AI offers personalized reports and data-driven recommendations, crossing your data with 16+ million possible combinations to identify the most relevant insights across all marketing channels. We recommend this tool if you wish to cut the time spent on analytics and you’re in need of a quick tailor-made solution to turn meaningful insights into effective marketing strategies.

Automation tools

Wordlift

logo WordLift

WordLift offers Artificial Intelligence for three facets of websites on WordPress; editorial, business, and personal blogger. Receiving a 4.7 out of 5 stars from WordPress itself, this plug-in analyzes your content into categories of; who, what, when, and where. WordLift processes your information by creating new entities, allowing you to accept them and select internal links for your content. This program also suggests open license images, which reduces the time used on Googling for images.

WordLift publishes a knowledge graph with your linked data (just like Google does!) and this data can be used in many unique features, such as:

  • creating timelines for events,
  • utilizing Geomaps for locations,
  • making chords to show which topic relates to the others.

WordLift, above all other of these platforms, adds the most distinctive effects to your WordPress website.

 

Dialogflow

Dialogflow is the basis of voice search on any platform such as; Google Assistant, Alexa, Cortana or even Facebook Messenger. This program is supported by Google and runs with natural language processing.

Dialogflow uses named entity recognition to analyze the spoken phrases from user to process the requests. The process includes providing the machine with several examples of how particular question could be phrased. In each case, the user must define an “entity” to show what is the most pertinent part of the statement spoken. From here, the information is spoken and relayed back to the consumer.

Dialogflow provides a helpful guide on their website to help users with the beginning process of getting Alexa or Siri to do just what you want them to do!

Curious to see a use case? Meet Sir Jason Link, the first Google Action that integrates Dialogflow and WordLift AI.

Andrea Volpini

Alli AI

Alli AI offers several AI-powered SEO features to improve and optimize your website content strategies. The tool provides the user with an easy and powerful way to increase traffic, build quality backlink and scale business outreach.

Alli AI uses Machine Learning technology to simplify SEO process through an all-in-one software tailored for each client and packaged into a pretty nice UI. The process includes planning your SEO strategy, finding backlinks, getting code and content optimizations in addition to tracking your traffic progress.

Furthermore, Alli AI boasts of having created a human tool, as it gives users the feeling of actually dealing with a person and not a machine.

Albert

Albert is an Artificial Intelligence powered software designed to manage your digital marketing campaigns and maintain a constant level of optimization in order to reach your business goals.

The software provides an out-and-out self-learning digital marketing ally designed to take care of every aspect of digital campaigns. Its features include autonomous targeting, media buying, cross channel execution, analytics and insights.

Albert is the perfect match for those who usually spend a lot of time on digital campaign optimization and who are looking for a powerful tool to reach a better allocation of budget between channels. Albert will advise which time and place engage with more customers and provides a constant growth of campaigns towards the set goal. The software also offers suitable recommendations for improvements that require human action such as best practice recommendations, budget shifts, creative performance etc.

Personalization tools

Acrolinx

Acrolinx is a game changer for those in the content marketing and advertising sector. The thought process drastically changes when it comes to optimizing search results. Developed at the German Research Center for Artificial Intelligence, Acrolinx works with 30 tools across the web; such as Microsoft Word or Google Docs, giving you much flexibility with how you promote your content. However, Acrolinx only supports; English, German, French, Swedish, Chinese and Japanese.

The software defines their evaluation technique with a “scorecard.” They make sure to ask what type of voice you are trying to achieve, to make accurate suggestions for you. Acrolinx works alongside, Salesfore.com, WordPress, Drupal, Adobe Marketing Cloud, and many more. The company provides an efficient guide to make sure that you are creating good content.

OneSpot

This software is unique from the others in that it focuses mainly on the consumer journey, with it’s patented “content sequencing” section. OneSpot generates personalized content after viewing a website user’s history on the internet. The company structures itself into three segments; OneSpot OnSite, OneSpot InBox, and OneSpot ReAct. Each facet of the company focuses specifically on that medium.

Through all of these, OneSpot creates a unique “content interest profile” for each user who visits your site. This profile allows the software user to create a deeper connection with consumers and be able to better target new visitors. OneSpot gives users a great way to expand a relationship with consumers through multiple mediums.

Follow us at Wordlift for more insights on SEO, or sign up for a free trial and get the full AI SEO experience.

WordLift Plugin is now available in Arabic

WordLift Plugin is now available in Arabic

We’re thrilled to announce that the Arabic Translation for WordLift is now in action and has been completed thanks to the amazing efforts of our colleague Nevine Adel Abdel Rehim, who works for our sister company in Egypt, insideout10. Kudos to Nevine!

While WordLift’s Natural Language APIs, the pre-trained machine learning models that WordLift uses to reveal the structure and meaning of your articles, already supported the Arabic language, the user interface of the WordPress plugin was available only in English and Italian (some of it is also translated into German and Danish).

Thanks to Nevine’s help now, clients like the American University of Cairo and Merck in Egypt, along with thousands of other Arabic speaking WordPress users, can now automate their SEO using WordLift in their own native language.

WordLift in Arabic

Here is a screenshot of SalamTak: a blog curated by Merck and focused on Multiple sclerosis (MS).

Why localization is so important?

Localization is key. In order to let our Arabic colleagues, partners and clients, that are already using WordPress in their native language, use WordLift with full confidence we decided to fully localize the plugin in Arabic.

Helping Translate WordPress in Your Language

WordPress is available in many languages and can be translated into other languages as well. WordLift as any other plugin on WordPress can be translated into other languages using translate.wordpress.org a web-based translation tool that allows anyone to contribute translations of WordPress core and any Themes or Plugins hosted on WordPress.org.  

When you install a new plugin, you certainly feel more comfortable if they are in your native language. Your user experience is better if you can read everything – from the readme to the meta-boxes and buttons – in your own language.

2019 has just begun and this is really the first result to celebrate: from now on our semantic plugin can be used also in Arabic ? Isn’t that awesome?

Stand out on search in 2019. Get 50% off WordLift until January 7th Buy Now!

x