{"id":10603,"date":"2019-04-10T17:07:08","date_gmt":"2019-04-10T17:07:08","guid":{"rendered":"https:\/\/wordlift.io\/blog\/en\/?p=10603"},"modified":"2021-03-02T08:05:46","modified_gmt":"2021-03-02T07:05:46","slug":"keyword-suggestion-tool-tensorflow","status":"publish","type":"post","link":"https:\/\/wordlift.io\/blog\/en\/keyword-suggestion-tool-tensorflow\/","title":{"rendered":"How to build a keyword suggestion tool using TensorFlow"},"content":{"rendered":"

One of the most fascinating features of deep neural networks applied to NLP<\/a> is that, provided with enough examples of human language, they can generate text and help us discover many of the subtle variations in meanings. In a <\/span>recent blog post<\/span><\/a> by Google research scientist Brian Strope and engineering director Ray Kurzweil<\/a> we read:<\/span><\/p>\n

\u201cThe content of language is deeply hierarchical, reflected in the structure of language itself, going from letters to words to phrases to sentences to paragraphs to sections to chapters to books to authors to libraries, etc.\u201d <\/span><\/p><\/blockquote>\n

Following this hierarchical structure, new computational language models, aim at simplifying the way we communicate and have silently entered our daily lives; from Gmail \u201cSmart Reply\u201d feature to the keyboard in our smartphones, recurrent neural network, and character-word level prediction using LSTM (<\/span>Long Short Term Memory<\/span><\/a>) have paved the way for a new generation of agentive applications. <\/span><\/p>\n

From keyword research to keyword generation<\/span><\/h2>\n

As usual with my AI<\/a>-powered SEO experiments, I started with a concrete use-case. One of our strongest publishers in the tech sector was asking us <\/span>new unexplored search intents<\/b> to invest on with articles and how to guides. Search marketers, copywriters and SEOs, in the last 20 years have been scouting for the right keyword to connect with their audience. While there is a large number of available tools for doing keyword research I thought, wouldn\u2019t it be better if our client could have a smart auto-complete to generate any number of keywords<\/strong> in their semantic domain, instead than keyword data generated by us?<\/span><\/span> <\/b>The way a search intent (or query) can be generated, I also thought, is also quite similar to the way a title could be suggested during the editing phase of an article. And titles (or SEO titles), with a trained language model that takes into account what people search, could help us find the audience we\u2019re looking for in a simpler way.<\/span><\/p>\n\n\n\n
\"\"<\/td>\nJump directly to the code: Interactive textgenrnn Demo w\/ GPU for keyword generation<\/strong><\/a><\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

The unfair advantage of Recurrent Neural Networks<\/span><\/h2>\n

What makes an RNNs \u201cmore intelligent\u201d when compared to feed-forward networks, is that <\/span>rather than working on a fixed number of steps they compute sequences of vectors. They are not limited to process only the current input, but also everything that they have perceived previously in time.<\/span><\/p>\n

\"A
A diagram of a Simple Recurring Network by Jeff Helman<\/span><\/a><\/figcaption><\/figure>\n

This characteristic makes them particularly efficient in processing human language (a sequence of letters, words, sentences, and paragraphs) as well as music (a sequence of notes, measures, and phrases) or videos (a sequence of images). <\/span><\/p>\n

RNN, I learned in the <\/span>seminal blog post by Andrej Karpathy<\/span><\/a> on their effectiveness, are considered <\/span>Turing-Complete<\/span><\/a>: this basically means that they can potentially build complete programs.  <\/span><\/p>\n

\"RNN<\/p>\n

Here above you can see the difference between a recurrent neural network and a feed-forward neural network. Basically, RNNs have a short-memory that allow them to store the information processed by the previous layers. The hidden state is looped back as part of the input. LSTMs are an extension of RNNs whose goal is to \u201cprolong\u201d or \u201cextend\u201d this internal memory – hence allowing them to remember previous words, previous sentences or any other value from the beginning of a long sequence.<\/span><\/p>\n

\"\"
The LSTM cell where each gate works like a perceptron.<\/figcaption><\/figure>\n

Imagine a long article where I explained that I am Italian at the beginning of it and then this information is followed by other let\u2019s say 2.000 words. An LSTM is designed in such a way that it can \u201crecall<\/a>\u201d that piece of information while processing the last sentence of the article and use it to infer, for example, that I speak Italian. A common LSTM cell is made of an <\/span>input gate<\/b><\/span>, an output gate<\/b><\/span> and a forget gate<\/b><\/span>. The cell remembers values over a time interval and the three gates regulate the flow of information<\/span><\/i> into and out of the cell much like a mini neural network. In this way, LSTMs can overcome the <\/span>vanishing gradient problem<\/span><\/i><\/a> of traditional RNNs. <\/span> <\/span>  <\/span><\/p>\n

If you want to learn more in-depth on the mathematics behind recurrent neural networks and LSTMs, go ahead and read <\/span>this article<\/span><\/a> by Christopher Olah.  <\/span><\/p>\n

Let\u2019s get started: \u201cIo sono un compleanno!\u201d<\/span><\/h2>\n

After reading Andrej Karpathy\u2019s blog post I found <\/span>a terrific Python library called <\/b>textgenrnn<\/b><\/a> by <\/span>Max Woolf<\/span><\/a>. This library is developed on top of TensorFlow and makes it super easy to experiment with Recurrent Neural Network for text generation.<\/span><\/p>\n

Before looking at generating keywords for our client I decided to learn text generation and how to tune the <\/span>hyperparameters in textgenrnn <\/span>by doing a few experiments. <\/span><\/p>\n

AI is interdisciplinary<\/b> by definition, the goal of every project is to bridge the gap between computer science and human intelligence. <\/span><\/p><\/blockquote>\n

I started my tests by throwing in the process a large text file in English that I found on Peter Norvig\u2019s website (<\/span>https:\/\/norvig.com\/big.txt<\/span><\/a>) and I end up, thanks to the help of <\/span>Priscilla<\/span><\/a> (a clever content writer collaborating with us), \u201c<\/span>resurrecting<\/span><\/i>\u201d <\/span>David Foster Wallace<\/b><\/span> with its monumental Infinite Jest (provided in Italian from Priscilla\u2019s ebook library and spiced up with some of her random writings).<\/p>\n

\"\"<\/p>\n

At the beginning of the training process – in a character by character configuration – you can see exactly what the network sees: a nonsensical sequence of characters that few epochs (training iteration cycles) after will transform into proper words.  <\/span><\/p>\n

As I became more accustomed to the training process I was able to generate the following phrase: <\/span><\/p>\n

\u201cIo sono un compleanno. Io non voglio temere niente? Come no, ancora per Lenz.\u201d<\/b><\/p>\n

\u201c<\/span>I’m a birthday. I don’t want to fear anything? And, of course, still for Lenz.<\/span><\/i>\u201d<\/span><\/p><\/blockquote>\n

\"David
David Foster Wallace<\/figcaption><\/figure>\n

Unquestionably a great piece of literature ?that gave me the confidence to move ahead in creating <\/span>a smart keyword suggest tool<\/b> for our tech magazine.<\/span><\/p>\n

The dataset used to train the model<\/span><\/h2>\n

As soon as I was confident enough to get things working (this means basically being able to find a configuration that – with the given dataset – could produce a language model with a loss value equal or below 1.0), I asked Doreid, our SEO expert to work on WooRank\u2019s API and to prepare <\/span>a list of 100.000 search queries<\/b> that could be relevant for the website. <\/span><\/p>\n

To scale up the number we began by querying Wikidata to get a list of software for Windows that our readers might be interested to read about. As for any ML<\/a>, project <\/span>data is the most strategic asset<\/b>. So while we want to be able to generate <\/span>never-seen-before queries<\/span><\/i> we also want to train the machine on something that is unquestionably good from the start. <\/span><\/p>\n

The best way to connect words to concepts is to define a context for these words. In our specific use case, the context is primarily represented by <\/span>software applications that run on the Microsoft Windows operating system<\/b>. We began by slicing the Wikidata graph with <\/span>a simple query<\/span><\/a> that provided us with the list of 3.780+ software apps that runs on Windows and 470+ related software categories. By expanding this list of keywords and categories, Doreid came up with a CSV file containing the training dataset for our generator.<\/span><\/p>\n

\"The
The first rows in the training dataset.<\/figcaption><\/figure>\n

After several iterations, I was able to define the top performing configuration<\/strong> by applying the values below. I moved from character-level to word-level and this greatly increased the speed of the training. As you can see I have 6 layers with 128 cells on each layer and I am running the training for 100 epochs. This is indeed limited, depending on the size of the dataset, by the fact that <\/span>Google Colab after 4 hours of training stops the session<\/b><\/span> (this is also a gentle reminder<\/span><\/i> that it might be the right time to move from Google Colab to Cloud Datalab – the paid version in Google Cloud). <\/span><\/p>\n

\"Textgenrnn
Textgenrnn configuration<\/figcaption><\/figure>\n
\"\"
Here we see the initial keywords being generated while training the model<\/figcaption><\/figure>\n

Rock & Roll, the fun part<\/span><\/h2>\n

After a few hours of training, the model was ready to generate our <\/span>never-seen-before search intents<\/strong> with a simple python script containing the following lines.  <\/span><\/p>\n

Here a few examples of generated queries:<\/span><\/p>\n\n\n\n\n\n\n\n\n\n\n
where to find google drive downloads<\/span><\/td>\n<\/tr>\n
where to find my bookmarks on google chrome<\/span><\/td>\n<\/tr>\n
how to change your turn on google chrome<\/span><\/td>\n<\/tr>\n
how to remove invalid server certificate error in google chrome<\/span><\/td>\n<\/tr>\n
how to delete a google account from chrome<\/span><\/td>\n<\/tr>\n
how to remove google chrome from windows 8 mode<\/span><\/td>\n<\/tr>\n
how to completely remove google chrome from windows 7<\/span><\/td>\n<\/tr>\n
how do i remove google chrome from my laptop<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

You can play with <\/span>temperatures<\/span> to <\/span>improve the creativity<\/span><\/i><\/span> of the results or provide a prefix<\/span> to indicate the first words of the keyword that you might have in mind and let the generator figure out the rest.<\/span><\/p>\n

Takeaways and future work<\/span><\/h2>\n

\u201cSmart Reply\u201d suggestions can be applied to <\/b>keyword research work<\/i><\/b> and is worth assessing in a systematic way the quality of these suggestions in terms of:<\/span><\/p>\n