BERT by Google


By now you would have heard about Google’s BERT update. If not, you might have at least experienced its benefits while searching with “longer and more conversational queries”. Google stated that “BERT is a core update impacting 10% of the global search results in SERP (Search Engine Results Page). The Algorithm of BERT was created and open sourced by Google in 2018 and was rolled out on October 21st 2019 to all the featured snippet results for queries in English.


For complex long tail/conversational queries, Google was struggling to provide the right answer as it wasn’t able to understand the contextual meaning of the query. The best results are obtained when a search engine is capable of understanding the language of the search. Google’s ultimate goal is to provide useful results, no matter how you spell or put together the words in the query. Google says that almost 15% of the queries happening on a daily basis are completely new and haven’t occurred earlier. This led to the creation of a new, smart and powerful algorithm named BERT. It’s basically a Machine Learning algorithm which is focussed on improving the search result quality. To understand better, let’s have a look at the following cases:

  1. The Term “Bucket” used in the following sentences
    1. He kicked the bucket.
    2. The bucket is filled with water.
    3. I haven’t yet crossed it from my bucket list.

Each of the above sentences has a completely different context, but they also share the same term “bucket”. When we enter them in a search bar, the results coming up might be similar. So the semantic context matters and that is where BERT comes into play.

2. Earlier, the search results for the term “Assistant to CFO“ was showing results for CFO jobs too just because the keyword ‘to’ is not being understood.

3. The term ‘Server Jobs in Bangalore’, when searched earlier would only result in all the database & tech related jobs. But after the introduction of BERT, it got improvised and now, based on the intent of the user it shows hotel server as well as tech related server jobs.

What is BERT?

BERT stands for Bidirectional Encoder Representations from Transformers. It is not just an algorithmic update, but also a machine learning natural language processing framework. To understand better, we have to dig a bit deeper.

Bidirectional: [B]

Basically in Google searches, you might have observed that the predictions generally happen to the right as we type in. When you type “sentence is harder than”, the normal model usually starts giving various predictions to the right for completing that sentence in the search. These predictions arise from previous search databases or might be common successors to the search term as shown below.

Unlike the above case, BERT uses the following methodology:

This means that BERT has the capability to understand both left & right hand sides of the targeted search term.

Encoder Representations:[E R]

For easier understanding of Encoder Representations, we should first learn more about Masked Language Modelling [MLM].

What is MLM?

When someone is typing the word “character ” in Google, algorithms start predicting on both sides of the words by randomly masking words (creating Dummies) in the sentence.

In the above image ‘?’ Represents- Character

The outcome of prediction is much better in this case as masking forces the algorithm to use information from the entire sentence simultaneously, regardless of the position of the words ie. it goes through a wider database searching for matching successors and predecessors of the above mentioned character or ‘?’. If the algorithm has the capability to understand the context of the keywords, then it will be easy for it to mask the other words in the sentence.

So how does a keyword work in Google?

All the keywords we use, technically have a vector space representation using models like Word2VEC & GLOVE2VEC. The purpose of this vector space is for the Google algorithm to easily understand the relationship between the keywords typed in by a user.

Hence, the BERT update can now be considered as a virtual 3-Dimensional space with each keyword having more than one contextual meaning.

Ex: For the word “Bank”, there are different meanings according to the context as shown below:

The Term "Bank"

3-D Context Verb-Noun Space

The BERT algorithm is already pre-trained with the entire database of Wikipedia (about 2500 million words) so that it is easier to identify the contextual differences via virtual vector space relationships for almost any keyword searched by a user. Google also started saving keywords in a format called Word Pieces (Ex: Printing—> Print + ing, Printed—> Print + ed) instead of just words, which is actually an effective way for reducing the size of the vocabulary.

So when a user types on Google, the keywords are being encoded & matched across word pieces, which in turn helps in decoding the contextual usage of the words with the help of various parameters like vector space, time, location of the search etc.


Transformer is a novel neural network architecture for understanding a language. It is a powerful state of the art architecture which uses self attention rather than the sequence flow methodology.

For Example:

Here, when we look at the above sentence, what does “it” denote? Is it referring to ‘animal’ or ‘street’? It’s a simple question for a human to answer, but for an algorithm, it’s not. This is where Self Attention helps the algorithm to match “animal” with “it” in the above sentence. Self Attention improvises the search results by giving the algorithm a better understanding of the sequence of the words in a sentence.

Impact on SEO?

A) The accuracy of voice searches were improvised. {Almost 50% 0f the searches will be through voice around 2022}

Ex: For instance, “four candles” and “fork handles” will have a closely same accent, which is very confusing for Google to search, which actually got improvised with BERT.

B) The core ideology of blogging will now transform from giving a walkthrough across a broad topic to giving an in-depth description on specifics. So, the better you answer a specific question, the better results it will yield in searches.

C) Bert update is intended to provide relevant traffic to your site by focusing on “Long Tail Keywords”. This actually helps in optimizing voice searches which basically comes with longer search words. To be more precise, the average number of words in a query is 3 – 4 for most of the normal searches happening in Google, whereas it is 6 – 8 mostly for voice based queries.

The customer satisfaction for searches will depend on factors like average visit duration, bounce rate, click per impression and much more parameters.

For Example : Antique furniture – Short Tail

Gandhian era antique couch with spring cushion – Long Tail

Long Tail keywords can contribute heavily towards leads & sales. Hence, BERT’s capability to optimize the Long Tail Keywords generally enhances the quality of leads coming from search.

D) BERT update also enhances local language based searches which will be a big leap in International SEO. Searches happening in any language will have the best possible results online. BERT was given this multi-linguistic ability since a lot of patterns in one language might resemble in other languages as well. There is a possibility to transfer a lot more learnings to different languages, so that we will be able to get hold of an even bigger database obtained from different unknown languages.

E) BERT will help the search engine to understand Conversational Queries in a better manner, which in turn reduces the irrelevant traffic drawn to your sites.

For Example:

The purpose of this search is for a Brazillian to get a visa for travelling to the US. But earlier, since the preposition “to” is being neglected in searches the results might appear somewhat irrelevant. But now with BERT, Google is capable of gathering the nuances better.

Do You need to optimize for BERT?

The answer is NO! In order to stay ahead of your competitors in search, you need to keep on producing quality content that enlightens your customers, fits their needs, and solves their problems. So it is advisable to focus on optimizing your content for users rather than for an Algorithmic Update. This will automatically get BERT working in your favour. Our team at The RoarUs Marketing helps you in producing this quality content which is drafted exclusively for your services and target audiences.

The RoarUs Marketing is a data-driven Digital Marketing agency which is highly focused on ROI. Our motto is “Say Digital, We Roar”. We are currently working with some industry leaders in the hospitality sector providing them with marketing services like Web Design and Analytics, SEO, Local SEO, Social Media Management, Creative Design, Ad Campaigns, Citation Building, Online Reputation Management and much more. We also undertake On-Demand marketing projects. Our entire team is indebted to provide our clients with continuous improvement across all stages of a project and thereby letting them enjoy the perks of world class marketing!

Share the thought

Write a comment