Hey, in this video we'll go further and practice regular expressions and that will clear any doubts
you might have about them.
So in this video, we will answer the question of what are the most used words in the miracle in the
end?
This book.
How can we find that out?
Well, you should think about the data source.
Where do you start?
So you are working on words, but all you've got is a string, one single string.
And so you need a better data structure.
You need a list of words, not a string of words, because that allows you to do calculations on the
items of that list of the words of that list.
So to get the list of words, you can use the split method.
It's a string method.
But there is a problem with the split method because for example, you see OCT here is found together
with a period because the split method splits the book, text up to the spaces at white spaces.
So you have October here, but in another occurrence, October might be without a period and that means
October with a period and October without periods will not be counted as one word.
They will be counted as different words.
And that is not what we want.
So the split method is, again, not enough to get accurate results.
So we need to use regular expressions again.
So let's create a pattern object using the r e library.
So make sure you have imported our E previously in the cells and make sure that if you have just open
Jupyter, make sure to run that all the cells.
So our e dot compile.
And here we've got to enter a pattern.
So previously in the previous video, we found all the sentences that contain the word love.
So that matches a sentence, right?
And this here matches a chapter followed by a number.
Now we want to match words, and that would be A to Z or A to Z.
And then findings is equal to or e dot find all pattern book.
And let's see what the findings are now.
So with that, we get letters because what we have told this compiled method to do is to find one single
letter, which could be a letter from A to Z, a lowercase or uppercase one single letter of both.
What we want to do is we want to say, give me at least one or more letters.
So we need to use that plus.
Press again.
Enter.
And this time you're going to get all those.
Words as a list.
We can improve this by working on the lowercase of the book, because otherwise a word such as Friday
here and another word such as Friday in lowercase could be counted as different words.
So if you work on the lowercase version of the text of the book, then you are converting all the texts
in loud letters, lowercase letters.
So that means you are comparing lowercase letters against lowercase letters.
That means you'll get more accurate results.
So press enter.
And now we get this.
And the length of the findings.
So.
The number of words in the book is this 86,798 words.
Right.
But our question is, what are the most used words rates?
So this gives us only a list of words, and these words are repeated.
So we should now figure out a way.
To count the occurrences of each of these words in the list.
So let me just print out just five of them, because it would be too much here and then in another cell.
We want to iterate for words and findings.
So we want to iterate over that findings lists, which looks like this, right?
So for each word, then we want to end up basically with a dictionary which will look like this, let's
say chapter.
Three occurrences and then we have before.
Five four currencies and 856 of currencies and so on.
So we need to construct such a dictionary.
Therefore, in here we start with an empty dictionary.
Before the loop starts and then we go through each word, for example, for chapter in findings, and
then we see D chapter.
So words, words, that means chapter, for example, or before or it is equal to.
The words.
Plus one now.
What I'm doing here basically, is that.
I'm iterating over this list, and then and then I'm adding the chapter words, the chapter string as
a key in the dictionary.
So that adds the chapter key, and its value is going to be the current value of the count, which could
be zero plus one.
So let me execute this to see what we got so we get an error.
It's a key chapter because in here at the first iteration, the key chapter will not exist.
So we say here, if words, if chapter in D, that's keys.
So if that exists, then we do this.
Else we do D.
Word is equal to just one.
So execute this.
A new cell D is now this dictionary.
So.
There we have the chapter and the occurrences of the chapter, words before the occurrences of the before
word, and so on.
Let me explain you how this loop works.
So again, a D is equal to an empty dictionary.
Then we go through the findings list.
For example, we get the chapter.
Full chapter in findings.
If chapter is not in the Keys, this will be executed if it's not in the Keys.
So we will add to one if we just met the chapter Words in the Loop.
Otherwise, if we have already added the chapter to the dictionary, then we will add plus one and in
each other occurrence of the chapter.
Right.
So we will add that one plus one.
It will be two.
In another loop it will be three.
And so on and so forth.
So then we end up with this, and then we want to say this is equal to now we want to to sort this dictionary,
but to to be able to sort it first, you need to convert it into a list of tuples of value key.
Four key value in detailed items.
So the list will be this now.
So it's the same data as we had in the dictionary, but we have this list of tuples now because this
now allows us to apply sorted methods, right like this.
And then we have all the words and their occurrences.
And to make this reverse, you say reverse is equal to true.
And there we go.
So the.
Words has this number of occurrences in the book and has this number of occurrences.
And so we have very common words.
And then we have things like Roberto, which was the best friend of the writer who writes the book.
So he mentioned him a lot of time.
And so on.
So this was a list comprehension.
And basically we did construct this list using tuples.
So this list will contain multiple tuples.
Each table will contain the value and the key of the dictionary for key value in the items.
So we're going to the items of the dictionary to each key and value, and we are creating a value and
the key in that list of tuples.
And then we sort out that tuple and the table will be sorted out by these numbers since these numbers
are the first in the tuple.
Right.
That's why we wrote here.
Value Key and note key value.
Because if we did that, this would be sorted out by the keys.
You see zones, the Z is then Y, and then we have X and W, So we want value key here.
And there we have that.
So with that, we have answered the question of what are the most used words?
So there we go.
These are the most used words.
Now, there is a problem here.
In every book you analyze, you would find that these words are actually the most common ones.
So these are articles like V and and and I pronoun.
So these words are found everywhere.
So we need to figure out a way to ignore such words.
And these such words are known as stop words in natural language processing.
So what I'm trying to say is that now we need to use some techniques known as natural language processing.
So techniques which Python uses to recognize language basically.
So these techniques are able to find that this is an article.
So these are like intelligent techniques and they ignore these articles and they can give you the actual
words which are not articles, so which are not stop words.
In other words, we need to filter out the stop words, but that is something you learned in the next
videos tomorrow.
So that will be natural language processing.
Today we focused on regular expressions and that's about this video.
Thanks a lot.
I'll see you later.