{"id":1534,"date":"2020-09-14T19:03:21","date_gmt":"2020-09-14T19:03:21","guid":{"rendered":"https:\/\/data-science.gotoauthority.com\/2020\/09\/14\/20-popular-nlp-text-preprocessing-techniques-implementation-in-python\/"},"modified":"2020-09-14T19:03:21","modified_gmt":"2020-09-14T19:03:21","slug":"20-popular-nlp-text-preprocessing-techniques-implementation-in-python","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2020\/09\/14\/20-popular-nlp-text-preprocessing-techniques-implementation-in-python\/","title":{"rendered":"20+ Popular NLP Text Preprocessing Techniques Implementation In Python"},"content":{"rendered":"
\n
\"NLP\"NLP<\/span><\/div>\n
\n

Using the text preprocessing techniques we can remove noise from raw data and makes raw data more valuable for building models.\u00a0<\/p>\n

Here, raw data is nothing but data we collect from different sources like reviews from websites, documents, social media, twitter tweets<\/a>, news articles etc.\u00a0<\/p>\n

Data preprocessing is the primary and most crucial step in any data science problems or project. Preprocessing the collected data is the integral part of any Natural Language Processing, Computer Vision, deep learning and machine learning problems. Based on the type of dataset, we have to follow different preprocessing methods.\u00a0<\/p>\n

Which means machine learning data preprocessing techniques vary from the deep learning, natural language or nlp\u00a0 data preprocessing techniques.<\/p>\n

So there is a need to learn these techniques to build effective natural language processing models<\/a>.<\/p>\n

In this article we will discuss different text preprocessing techniques or methods like normalization, stemming, lemmatization<\/strong>, etc. for \u00a0handling text to build various Natural Language Processing problems\/models.\u00a0<\/p>\n<\/div>\n

\n
\n
\n

Popular Text Preprocessing Techniques Implementation in Python #nlp, #datascience #machinelearning<\/p>\n<\/div>\n

\n\t\t\t
\n\t\t\t\t<\/i>
\n\t\t\t\tClick to Tweet<\/span>
\n\t\t\t<\/span>\n\t\t<\/p>\n<\/div>\n<\/div>\n

\n

Moreover we don’t limit ourself with the theory part but we will also implement these technique in \u00a0python.<\/p>\n

Before we go further below are the list of topics you will learn in this article.<\/p>\n<\/div>\n

\n

Text Preprocessing Importance in NLP<\/h2>\n

As we said before text preprocessing is the first step in the Natural Language Processing pipeline. The importance of preprocessing is increasing in NLP due to noise or unclear data extracted or collected from different sources.\u00a0<\/p>\n

Most of the text data collected from reviews of E-commerce websites like Amazon or Flipkart, tweets from twitter,\u00a0 comments from Facebook or Instagram, and other websites like Wikipedia, etc.\u00a0<\/p>\n

We can observe users use short forms, emojis, misspelling<\/strong> of words, etc. in their comments, tweets, and so on.<\/p>\n

We should not feed raw data without preprocessing to \u00a0build models because the preprocessing of text directly improves the model’s performance<\/a>.<\/p>\n

If we feed data without performing any text preprocessing techniques, the build models will not learn the real significance of the data. In some cases, if we feed raw data without any preprocessing techniques the models will get confused and give random results.\u00a0<\/p>\n

In that confusion, the model will learn harmful patterns that are not valuable. Due to this, the model’s performance will be affected, which means the model performance will reduce significantly.<\/p>\n

So we should remove all these noises from the text and make it a more clear and structured form for building models.<\/p>\n

Here we have to know one thing.<\/p>\n

The natural language text preprocessing techniques will vary from problem to problem. This means we cannot apply the same text preprocessing techniques used for one NLP problem to another NLP problem.\u00a0<\/p>\n

For example, in sentiment analysis classification problems<\/strong>, we can remove or ignore numbers within the text because numbers are not significant in this problem statement. <\/p>\n

However, we should not ignore the numbers if we are dealing with financial related problems. Because numbers play a key role in these kinds of problems.<\/p>\n

So while performing NLP text preprocessing techniques. We need to focus more on the domain we are applying these NLP techniques and the order<\/strong> of methods also plays a key role.<\/p>\n

Don’t worry about the order of these techniques for now.\u00a0 We will give the generic order in which you need to apply these techniques.<\/p>\n

Our suggestion is to use preprocessing methods or techniques on a subset of aggregate data<\/strong> (take a few sentences randomly). We can easily observe whether it is in our expected form or not. If it is in our expected form, then apply on a complete dataset; otherwise, change the order of preprocessing techniques.<\/p>\n

We will provide a python file<\/strong> with a preprocess class of all preprocessing techniques at the end of this article. <\/p>\n

You can download and import that class to your code. We can get preprocessed text by calling preprocess class with a list of sentences and sequences of preprocessing techniques we need to use.<\/p>\n

Again the order of technique we need to use will differ from problem to problem.<\/p>\n

Different Text Preprocessing Techniques<\/h2>\n

Let us jump to learn different types of text preprocessing techniques.\u00a0<\/p>\n

In the next few minutes, we will discuss and learn the importance and implementation of these techniques.<\/p>\n

Converting to Lower case<\/h3>\n

Converting all our text into the lower case is a simple and most effective approach. \u00a0If we are not applying lower case conversion on words like NLP, nlp, Nlp, we are treating all these words as different<\/strong> words.\u00a0<\/p>\n

After using the lower casing, all three words are treated as a single word that is nlp.<\/strong><\/p>\n<\/div>\n

\n\"Converting\"Converting<\/span><\/p>\n

Converting To Lower Case<\/p>\n<\/div>\n

\n

This method is useful for problems that are dependent on the frequency<\/strong> of words such as document classification.\u00a0<\/p>\n

In this case, we count the frequency of words by using bag-of-words, TFIDF, etc.<\/a><\/p>\n

It is better to perform lower case<\/strong> the text as the first step in this text preprocessing. Because if we are trying to remove stop words all words need to be in lower case. <\/p>\n

For example, few sentences have the starting word as “The”<\/strong> if we are not performing the lower casing technique before that technique, we can not remove all stopwords<\/strong>.\u00a0<\/p>\n

The other case is for calculating the frequency count. If we not<\/strong> converted the text into lower case Data Science<\/strong> and data science<\/strong> will treat as different tokens. <\/p>\n

In natural language processing the lower dimension of text which is words<\/strong> called as tokens<\/strong>.<\/p>\n

We can apply this method to most of the text related problems. Still, it may not be suitable for different projects like Parts-Of-Speech tag<\/strong> recognition or dependency parsing<\/strong>, where proper word casing is essential to recognize nouns, verbs, etc.<\/p>\n

Implementation of lower case conversion<\/h4>\n<\/div>\n

Removal of HTML tags<\/h3>\n
\n\"Removing\"Removing<\/span><\/p>\n

Html Tags Removal<\/p>\n<\/div>\n

\n

This is the second essential preprocessing technique. The chances to get HTML tags in our text data is quite common when we are extracting or scraping data from different websites.\u00a0<\/p>\n

We don’t get any valuable information from these HTML tags. So it is better to remove them from our text data. We can remove these tags by using regex<\/strong> and we can also use the BeautifulSoup<\/strong> module from bs4 libraries.\u00a0<\/p>\n

Let us see the implementation using python.<\/p>\n

HTML tags removal Implementation using regex module<\/h4>\n<\/div>\n
\n\"Html\"Html<\/span><\/p>\n

HTML tags removal example<\/p>\n<\/div>\n

Implementation of Removing HTML tags using bs4 library<\/h4>\n
\n

We can observe both<\/strong> the functions are giving the same<\/strong> result after removing HTML tags from our example text.<\/p>\n

Removal of URLs<\/h3>\n<\/div>\n
\n\"Remove\"Remove<\/span><\/p>\n

Remove Urls<\/p>\n<\/div>\n

\n

URL is the short-form of Uniform Resource Locator<\/strong>. The URLs within the text refer to the location of another website or anything else. <\/p>\n

If we are performing any website backlinks analysis<\/strong>, twitter or Facebook in that case, URLs are an excellent choice to keep in text. <\/p>\n

Otherwise, from URLs also we can not get any information. So we can remove it from our text. We can remove URLs from the text by using the python Regex<\/strong> library.<\/p>\n<\/div>\n

\n\"Urls\"Urls<\/span><\/p>\n

Urls removal Example<\/p>\n<\/div>\n

\n

Implementation of Removing URLs \u00a0using python regex<\/h4>\n

In the below script. We take example text with URLs and then call the 2 functions with that example text. In the remove_urls<\/strong> function, assign a regular expression to remove URLs to url_pattern after That, substitute URLs within the text with space by calling the re library’s sub-function<\/strong>.<\/p>\n<\/div>\n

\n

Removing Numbers<\/h3>\n

We can remove numbers from the text if our problem statement doesn’t<\/strong> require numbers.\u00a0<\/p>\n

For example, if we are working on financial<\/strong> related problems like banking or insurance-related sectors. We may get information from numbers. <\/p>\n

In those cases, we shouldn’t<\/strong> remove numbers.<\/p>\n<\/div>\n

\n\"Removing\"Removing<\/span><\/p>\n

Removing Numbers<\/p>\n<\/div>\n

\n

Implementation of Removing numbers \u00a0using python regex<\/h4>\n

In the code below, we will call the remove_numbers<\/strong> function with example text, which contains numbers. <\/p>\n

Let’s see how to implement it.<\/p>\n<\/div>\n

\n

In the above removing_numbers<\/strong> function. We mentioned a pattern to recognize numbers within the text and then substitute numbers with space using the re library’s sub-function. <\/p>\n

And then return text after removing the number to numbers_result<\/strong> variable.<\/p>\n

Converting numbers to words<\/h3>\n

If our problem statement need<\/strong> valuable information from numbers in that case, we have to convert numbers to words. Similar problem statements which are discussed at the removing numbers (above section).<\/p>\n<\/div>\n

\n\"Converting\"Converting<\/span><\/p>\n

Converting Numbers to Words<\/p>\n<\/div>\n

\n

Implementation of Converting numbers to words using python num2words library<\/h4>\n

We can convert numbers to words by just importing the num2words<\/strong> library. In the code below, we will call the num_to_words<\/strong> function with example text. Example text has numbers.<\/p>\n<\/div>\n

\n

In the above code, the num_to_words function is getting the text as input. In that, we are splitting text using a python string function of a split with space to get words individually. \u00a0<\/p>\n

Taking each word and checking if that word is digit or not. If the word is digit then convert that into words.<\/p>\n

Apply spelling correction<\/h3>\n<\/div>\n
\n\"Spelling\"Spelling<\/span><\/p>\n

Checking Spelling<\/p>\n<\/div>\n

\n

Spelling correction is another important preprocessing technique while working with tweets, comments, etc. Because we can see incorrect spelling<\/strong> words in those areas of text. We need to make those misspelling words to correct spelling words.<\/p>\n

We can check and replace misspelling words with correct spelling by using two python libraries, one is pyspellchecker, and another one is autocorrect.<\/p>\n<\/div>\n

\n\"Example\"Example<\/span><\/p>\n

Example of Spelling Correction<\/p>\n<\/div>\n

\n

Implementation of spelling correction using python pyspellchecker library<\/h4>\n

Below we are calling a spell_correction<\/strong> function with example text. Example text has incorrect spelling words to check whether the spell_correction function gives correct words or not.<\/p>\n<\/div>\n

Implementation of spelling correction using python autocorrect library<\/h4>\n
\n

We can observe both methods given correct or expected solutions.<\/p>\n

Convert accented characters to ASCII characters<\/h3>\n

This is another common preprocessing technique in NLP. We can observe special characters at the top of the common letter or characters if we press a longtime while typing, for example, r\u00e9sum\u00e9.<\/strong>\u00a0<\/p>\n

If we are not removing these types of noise from the text, then the model will consider resume and r\u00e9sum\u00e9; both are two different words. <\/p>\n

Even if both are the same. We can convert this accented character to ASCII <\/strong>characters by using the unidecode<\/strong> library.<\/p>\n<\/div>\n

\n\"Convert\"Convert<\/span><\/p>\n

Convert accented characters to ASCII<\/p>\n<\/div>\n

\n

Implementation of accented text to ASCII converter in python<\/h4>\n

We will define the accented_to_ascii<\/strong> function to convert accented characters to their ASCII values in the below script. \u00a0<\/p>\n

We will do this function with example text.<\/p>\n<\/div>\n

\n

In the above code, we use the unidecode method of the unidecode library with input text. Which converts<\/strong> accented characters to ASCII values.<\/p>\n

Converting chat conversion words to normal words<\/h3>\n

This is another essential preprocessing technique if we work with chat conversions<\/strong>, or our problem statement requires chat conversion analysis. We need to handle short-form. As nowadays, people use short-form<\/strong> words in their chatting conversions for their simplicity.<\/p>\n<\/div>\n

\n\"Chat\"Chat<\/span><\/p>\n

Chat conversion to normal words<\/p>\n<\/div>\n

\n

A better way to work with those words is to replace short-form words to their original words. <\/p>\n

We can find all those short-form words and its actual words in this Github Repo<\/a> to save that file into our system; click right click and then press on save as option.<\/p>\n

Implementation of python script<\/h4>\n<\/div>\n

Expanding Contractions<\/h3>\n
\n\"Expanding\"Expanding<\/span><\/p>\n

Expanding Contractions<\/p>\n<\/div>\n

\n

Contractions are words or combinations of words created by dropping a few letters and replacing those letters by an apostrophe. <\/p>\n

An example of a contraction word.<\/p>\n