Hello guys.
So we are going to continue the discussion with respect to multithreading.
Now in this video we will be implementing some real world use case uh which is called as web scraping.
And here we are going to specifically use multithreading.
Uh, again understand one thing is that here I have given the problem statement.
So we basically say this as multithreading for I o bound task okay.
The scenario is web scraping.
So here you can see that web scraping often involves making numerous network requests to fetch web pages.
Okay, this task are I o bound because they spend a lot of time waiting for responses from the server,
right?
So multithreading can significantly improve the performance by allowing the multiple web pages to be
fetched concurrently.
So if I have multiple web pages, I can probably create that many number of threads and parallelly bring
all the content out of it.
Okay.
So in this particular video I will be talking about this.
We will be implementing this entire thing, um, how you can actually implement multi threading in the
case of web scraping.
So first of all, let me do one thing.
Let me just show you the web page that I will be scraping.
Okay.
So this is my web page over here.
You can see it.
This is a long chain page over here.
So this one page we will take.
And let's consider this another page.
So this two URL I will be going ahead and using it.
So let me just copy and paste this two URL.
Let's say this is my two URL.
Not only two I'll take three URLs okay.
So let's say I have one more tutorial.
Let me just go ahead and read this.
Let's consider this three URL okay.
So I'm going to use this three URL.
Everything in short.
So let me tell.
Then let me just comment it down.
But later on I will be using this as a I'll be copying and pasting it over there.
Okay.
So first of all what we are going to do is that we are going to quickly import threading.
Okay.
Since we are going to implement a multi threading right.
So import threading.
Along with this we will go ahead and import requests and request along with this, uh, what I'm actually
going to do is that I'm going to probably import one important library, which is called as BS4.
That is nothing but beautiful soup for.
So for that I will be going ahead and using this BS4 library over here.
And let me quickly go ahead inside this.
Right now I'm in my multithreading and multiprocessing folder.
I will go ahead and write CD dot dot to go back.
Because my requirement dot txt file is present in this particular folder.
So I will just go ahead and write pip install minus our requirement dot txt.
So we are installing this BS4 library which is also called as beautiful soup for.
And this is specifically used for web scraping.
So I will just go ahead and do the requirement.
Dot txt will do the installation.
And here you can see the requirement is already satisfied because I have already done the installation
because I don't want to waste more time.
Um, just showing you that how the installation is basically taking place, right?
So once we do the installation now, I will go ahead and write my code over here.
Now here.
What I'm actually going to do I'm going to write from BS4.
I'm going to import Beautifulsoup.
Okay.
So this will be the library that will be used for, you know, web scraping any URL.
So here I'll be taking the list of URL.
And I have actually used all the list of URL over here.
So I will copy and I will paste it.
Okay.
Now with respect to this URLs what I'm actually going to do I will be using strings.
Okay.
So this will be my one URL.
I will use a comma.
Then let's go back to my second element of the list which will be this one.
Again I'll be using comma.
Okay over here.
And finally I'll go and use this third one which will be my another URL.
Right.
So this three URLs we are going to do the web scraping.
And we'll try to retrieve the content.
Or at least we'll try to see how many characters are there using multithreading.
Now the importance of multithreading over here is that I will probably create three threads, and at
a time I will probably say that Go and hit this all URLs.
Okay.
Now let me just go ahead and create a function which is called as fetch underscore contents.
Okay.
And now I'm just going to go ahead and give my URL over here.
So let's quickly go ahead and create my response.
And I'll say hey use this request library and use dot.
Get with this specific URL that I'm giving.
Okay then once I have this I will be using my Beautifulsoup to do the web scraping.
So in order to do the web scraping I just need to use response.
So I'll be using this library.
Inside this I'll be giving response dot content.
So this response dot content which I'm actually hitting it through request dot dot this content will
be having the entire content over there, uh, with respect to HTML.
And I'll say, hey, go ahead and use an HTML parser so that we get the content that is present inside
the HTML.
So this is the parser that we specifically use.
And it is available in Beautifulsoup.
Okay.
Next statement is that I will just go ahead and print Here, I will not just display the entire content.
It will be of no use because I just, uh, I, uh, I don't want to see the content in short.
Instead, what I'll do is I'll just count all the characters, okay?
So you can go ahead and display the content.
Also, you can just go ahead and write soup dot text.
Okay.
Then you'll be able to get the content.
But what I will do I will go ahead and fetch uh, and I'll go ahead and write length of soup of text.
Okay.
And this I will go ahead and write in this brackets.
Okay.
Since I'm using a F string.
And here let me just go ahead and write soup dot text.
So this will basically count each and every word.
And it will be giving me that many number of words.
Right.
So here what I will do I'll remove this length also brackets I don't require it.
So this many characters will be there from this all.
You all right.
So all the URLs that I'm actually specifically giving okay.
So these are the information that I'm actually getting it from this entire URL itself.
And this is what I'm printing okay.
In short I'm just printing.
How many characters are there over there.
Now I'll go ahead and create my threads.
So let's say I will go ahead and create a list of list of thread.
Now I'll say for URL in URLs, since I have to probably go in each and every URL and run this entire
fetch content.
Okay.
So first of all, I will go ahead and create my thread.
Okay.
So first of all, in order to create the thread I will right trading dot thread.
Okay.
So this will be threading dot thread.
I will give my target.
My target will be my function which is called as fetch content.
This I have already shown you.
And my arguments.
The arguments that I'm actually going to give over here will be my URL.
Okay.
So these are the two parameters that we need to give.
What is the content.
What is the function that I'm calling and on which all URLs I'm basically calling.
Right.
So that is the reason I'm giving each and every single URL over here.
Then I will say, hey, once this operation is done, I will go ahead and append this particular thread
inside my threads itself.
Okay.
So I will go ahead and write thread.
Okay.
Then finally let's go ahead and start this thread right.
So here how am I creating this particular thread.
See when this particular operation will be taking place.
Right.
Uh, so here you'll be able to see that I'm appending the thread, and then I'm starting the thread
till I get the entire response.
It will go back to the next thread.
It will create a thread and again it will continue this particular process.
Now I will wait for all the thread execution to get completed.
So here I will say for thread in threads.
And here I will just go ahead and write thread dot join.
That's it okay.
And finally I will just go ahead and print all web pages fetched okay.
So yes this is it.
I will go ahead and open my terminal.
I think now it should work.
Now what we are going to do is that we are going to extract all the information from this particular,
each and every URL, and we are going to see the number of characters, right, number of text over
there.
So here I will first of all go to my CD 16 folder.
I will clear my screen.
Now let me just go ahead and write Python Python.
And here I'm going to use my use case multithreading dot p y.
So once I execute this, see how fast it is?
C fetch 7477 characters from this URL.
This was responsible by one of the thread.
Then it went with another thread over here where I was able to get eight nine, eight, six character.
And finally, from this particular URL, I was able to extract 66004044 uh, characters.
Right?
But in short, you can just say that this three threads were parallelly working and probably retrieving
all these characters from here.
Right?
And this is one very good example where you can actually see that how I was able to create this threading.
Right.
You can add any number of URLs and that many number of threads will get keep on getting created.
Right.
Because we are creating each and every one with respect to the URL.
Right?
And let's say one thread is probably taking some amount of time.
Then parallelly all the threads will also start working right concurrently they are working and they
are fetching the results out of it.
So uh, I hope you are able to understand this particular use case again.
This was an amazing use case with respect to multi threading.
This was it from my side.
I will see you all in the next video.
Thank you.