How Natural Language Processing and Structured Data Make Quora Smarter

Anywhere we look, there is an area where Artificial Intelligence is changing the rules of the game. From personal assistants that are changing the way we interact with machines; to self-driving cars and diagnostics systems that can diagnose certain diseases more accurately than doctors. That isn’t only due to the buzz of the media (that also contributes). This is due instead to the fact that AI is a vast area that touches several disciplines.

In this article, I want to show you how a field of Artificial Intelligence, called Natural Language Processing (NLP) helped Quora to become one of the most popular Q&A sites in the world. In fact, NLP has become so critical to Quora that it also has open vacancies for NLP engineers.

As WordLift was born as a university project from a Natural Language Processing research, we always look for best practices to see how the industry is evolving and helping the web become smarter.

Natural Language Processing applications

The main aim of NLP is to help computers’ program to process large amounts of natural language data by making sense of that. On platforms like Quora, with hundreds of millions of users keeping the quality of its content high is critical.

As specified on Quora engineering blog:

Hundreds of millions of people use Quora to discover high-quality answers to questions important to them. The quality of our content and the civility of our community are two important factors that make Quora special. We want to maintain that quality even as billions of people start using Quora.

The most effective way to be able to keep high-quality standards, while growing the user base is the ability to process that data in a way that makes it more valuable to its users. In fact, as explained further:

Such a rich dataset puts us in a unique position to use various Natural Language Processing (NLP) techniques to solve exciting problems critical to our success.

How’s Quora applying NLP? Here are 13 interesting ways.

Start using NLP on your website today!

Get Started with WordLift ?

Answer Quality

Quality is critical for any platform to survive. For a Q&A platform like Quora, this is even more important. In fact, where users contribute all Quora content, how does it make sure to keep its quality high? First, we must define quality. Quora looks at things like writing style, readability, completeness, and trustworthiness.

This is a ranking issue. In fact, Quora has to look at many variables and give it a ranking based on its relevance and helpfulness such that most helpful answers show at the top.

To define the relevance an helpfulness Quora it needs to have five properties:

• Answers the question that was asked.
• Provides knowledge that is reusable by anyone interested in the question.
• Answers that are supported by rationale.
• Demonstrates credibility and is factually correct.
• Is clear and easy to read.

Quora also gives an example of how it uses Natural Language Processing to extract relevant data to assess and rank answers:

Source: engineering.quora.com

As you can see Quora looks at various things; most of those are in the form of text. However, the text needs to be converted into data. Once that content becomes structured data that is how NLP helps turn that text in machine-readable data easily processed by its algorithms.

This gives Quora the opportunity to be more sophisticated in creating ranking systems by considering things such as author credibility, formatting, upvotes, and many other variables.

Question Quality

The flip of the coin of answers’ quality is questions’ quality. In fact, if you know Quora chances are you found it through Google’s search. In fact, if we look at its marketing mix more than 80% of traffic comes from search:

That’s because Quora is well positioned on so-called long-tail keywords that allow Quora to take over the SERP. Of course, this is also thanks to the fact that Quora can provide quality content to those answers. Yet, if Quora didn’t use a process driven by AI and machine learning, it would have been impossible to leverage on such a mole of natural language data.

That is why Quora uses the same ranking system that we saw above also to assess the relevance of Questions.

Get Started with WordLift

Start using NLP on your website today

Search

When you type something in Quora’s box, that has several functions:

In fact, this isn’t only a Q&A tool that allows anyone to ask something but also a way to search anything on the platform. You might think that the retrieval of information from that search box is mainly based on keyword matching. However, that is not the case as specified by Quora engineering team:

We use NLP techniques in this information retrieval problem space to help us better understand user queries and questions, as well as better rank content in the form of questions, answers, topics and user biographies. Unlike regular search engines with simple keyword matching, we can also support searches done with longer queries that are in the form of questions well.

Other NLP applications that make Quora smarter

When you transform the text into structured data, suddenly that knowledge which before was only accessible to humans becomes easily accessible to machines. Natural language processing helps make that transition, which translates human text in machine-readable data that can be fed to a system to make it more relevant for its users.

In this article, we saw how Quora uses NLP in three key areas. However, that is just the beginning. There are other areas in which NLP is crucial for Quora’s success:

• Automatic Grammar Correction
• Duplicate Question Detection
• Related Question Generation
• Topic Biography Quality
• Answer Summaries
• Automatic Answer Wikis
• Hate SpeechHarassment Detection
• Spam Detection
• Question Edit Quality

At WordLift we also use NLP to automate an important part of the digital marketing strategy, the so-called SEO. This article wanted to show you all the practical way in which AI is helping startups to build smarter systems that become more useful to their users beyond the buzz and hype created by media.

If you want to try NLP on your website, book a demo and let’s talk about your project. ?