Hello guys.
So we are going to continue the discussion with respect to pandas.
And in this video we are going to see how we can read data from different different sources by using
some amazing inbuilt functions that are available in pandas.
Now to start with what I'm actually going to do, let's I've just copied and pasted this specific code
over here.
So here you can see I have imported pandas I have imported from IO string io.
And here I have given some kind of JSON.
Right.
So employee name James email, James email job profile.
And here is some list of details that I am able to see.
Right.
Like title one team title two senior developer.
Something like this.
Right.
I'm taking this particular data.
And first of all this data is in the form of JSON right.
Key value pairs.
So what I'm doing is that I will take this entire data, convert it into a string string by using string
IO string input output.
And I'm just writing pd dot read underscore JSON.
Once I do this, I will be able to get a data frame in this particular way right.
So here you can see employee underscore name.
Uh email uh job profile.
So here you are able to see this right.
And this is basically my data frame altogether since I'm using PD dot read underscore JSON.
Let me just go ahead and save this in df variable.
So this is nothing but this is my DF.
Okay.
Now the reason why I am showing you this is that I'm just trying to show you that yes, if you have
a key value pairs, then also you will be able to read that and convert into a data frame.
Right now let's say I want to convert this data frame back into a JSON.
So for this I will be using df dot two underscore JSON again.
So two underscore JSON.
And I'll go ahead and execute it.
Now as soon as I go ahead and execute it.
See the difference how this JSON is basically got converted.
Right.
And here you'll be able to see initially we had something like this.
Now you have converted this into this kind of format right.
What is this all about.
So here if you go ahead and see here I have not given any parameters.
So there is one very important parameter which is called as orient okay.
And by default orient will be based on index value.
Right.
So orient basically means see here what it has done.
It has taken this employee name as one key.
Then now it is taking zero colon James okay.
Then it went to this email key.
And now it is basically using the combination of index and but this particular value.
So zero colon James.
And at the rate gmail.com.
Then finally you had job profile with zero colon.
So in short when we converted this into a JSON it is taking this as a column name.
And it is taking key value pairs based on the index number.
Okay.
If I try to convert this into JSON based on records, like if I use a parameter which is called as orient,
and here if I just go ahead and and if you probably go ahead and see this right parameter, let's see
this parameter here you will be able to see this.
Okay I will just go ahead and write something like this.
And here I will just go ahead and say Orient.
Uh now with respect to Orient, let's say I'm going to basically use a parameter which is like index.
Okay.
So index is the default one, right?
Same thing.
Right.
So by default the index is that is getting applied over here is specifically with respect to or um this
particular index value or it will be index.
Okay.
Uh now you can also change the Orient value to some other thing.
And we will see what all values again we can probably match with it.
Right.
And again uh, you can also see the documentation with respect to two underscore JSON function in data
frame.
So now let me go ahead and copy this same thing and paste it over here.
Instead of writing Orient is equal to index I will go ahead and write.
Orient is equal to records.
Now you can see how we had actually given in over here based on record.
Similarly it is basically getting displayed right.
So here you can see employee underscore name James email this.
So record by record you'll be able to see in this particular manner.
Okay.
So this was some of the examples.
Now what I'm actually going to do.
Uh I'll show you one more way of reading a specific data.
So let me do one thing.
Uh, let me quickly show open my browser.
So here I'll be showing you one URL.
Okay.
This URL you can actually see it is something like wine dot data right.
And this URL will be able to see it is in comma separated values like CSV format.
So I will take this particular uh uh entire URL.
Right.
And I'll go to my code.
And here I will just go ahead and write PD dot read underscore CSV.
Right.
And here I will just go ahead and give my URL.
And let's say that I'm going to say that hey don't consider anything as my header.
So header will be none right.
So if I go ahead and just see this write DF is equal to I will be able to see this.
It will take some time to read it okay.
Once it read it's uh I will just go ahead and write DF dot head.
And this is how my entire data set looks like.
Okay.
So by default, since I wrote header is equal to none, it is just going to make the default column
indexes by 0123.
And similarly row index is 01234.
Okay.
Um this was how to read a specific, uh data from a specific URL if it is in the comma separated values.
I can also convert this back into into my CSV file.
So if I go ahead and write dot or dot two, underscore CSV and let me just go ahead and write wine dot
CSV, okay.
And if I execute it here, you will be able to see in my same folder location wine .21. CSV has been
created, right?
So you can also convert back a data frame into a CSV file.
Okay.
Now let's go ahead and see some more examples.
I have already spoken about two underscore JSON with respect to orient and index.
You can also do this okay.
Now let's say I have a specific HTML file.
Now this HTML file I will just show it to you.
I can also read data from HTML file.
That is the most amazing thing.
So let's say this is my HTML file over here.
Uh, it is FDIC government uh, failed bank list.
Now in this particular HTML file, I definitely have tables.
Right.
And with the help of data frame, we can also go ahead and read this entire table also, or we can also
read the content inside this HTML.
So first of all, what I will do I will go to my code.
I will just say, hey, this is my URL.
So let me just go ahead and define my URL.
And now with this specific URL, what I am actually going to do, I'll just go ahead and use PD dot
read underscore HTML okay.
And I'll give this particular URL okay.
Now if I go ahead and create my data frame okay.
This will basically give me.
So here you can see mission or missing optional dependency l XML.
So it is saying hey you need to install uh l XML.
So let's go ahead and install this.
Uh, there are some dependencies with respect to some of the libraries.
So I'll go ahead and write pip install uh pip install l XML.
Okay.
So I will just go ahead and install this.
So let's do the installation.
Probably once I install this it will start working okay.
So here we have installed it now.
I think it should be working fine.
Now I will go ahead and read my HTML.
Okay.
Again it is giving me an error.
Okay, I have to also go ahead and install HTML five.
Okay.
So these are some of the dependencies.
Whenever you work with uh websites you have to do this installation okay.
So I'll go ahead and write pip install HTML five I think now this is the last library.
If there is any dependencies again it will be telling us okay.
Now I'll go ahead and execute it.
Now I think it should not give us an error.
It is still giving us an error because it is saying hey, we need to also have Beautifulsoup four.
Okay, so let's go ahead and install this also.
Okay.
So I will go ahead and install pip install um Beautifulsoup four once I do this installation uh web
encodings and all.
So here you can see Beautifulsoup four is also encoded.
Now I think it should work.
So whenever you get this kind of errors please go ahead and see the error okay.
And then you'll be able to see this.
No tables found okay.
No table found in this bank list HTML.
Let me do one thing.
I think the URL has been changed or what?
So this is my failed bank list.
Okay, it is not HTML, it is this URL.
Okay.
So now let me go ahead and execute this.
Now you can see if I go ahead and execute DF I'll be getting a list of tables.
See this is in this particular format.
If I go ahead and write DF of zero.
So here I'm actually going to get my entire table.
And this is how it looks like in the form of data frame.
Okay.
So here I've just shown you that how simply if you have any HTML page which has tables over there,
you can definitely go ahead and explore this and it will be able to capture the entire table and it
will be able to give it to you.
Okay, now not only this, let's say I also may have a Wikipedia page.
I may have some other things I may have.
Uh, again, there may be different different pages.
Let me just show you one more HTML, okay.
Content.
And here you'll be able to understand.
So here if I go ahead and execute this this in this Wikipedia you basically have mobile country code,
right?
So these are my tables that are present over here with respect to this.
Right.
And let's say with the help of pandas, I really want to quickly go ahead and use all this particular
data.
Okay.
So what I'm actually going to do over here is that I will say, hey, this is my URL now.
And based on this URL, I will take up this entire country code.
And now I will just go ahead and write PD, underscore read underscore HTML.
And here I'm just going to give my URL.
And one more thing that read underscore HTML has one amazing thing right.
That is nothing.
But it basically also has some kind of match keywords.
Okay.
So this will basically be my URL and it also has some match keywords.
What does this basically mean?
That let's say I want to match it with some main keys like this country.
Okay, I want to probably match only this particular table.
I don't want this table.
Right.
Uh, so what I will do, I will just go ahead and write.
Hey, based on the country, let's go ahead and do the match.
So here I will just go ahead and write my match keyword like country and I'll say, hey, I don't even
want my header, so I'll make it as header zero.
So if I go ahead and execute it you'll be able to see that.
Yeah, I'm able to get it it in the form of list.
Uh, so if I go ahead and take the first element of the list.
So here is my entire data frame.
And this is the same country.
We are able to get it and it is able to do the match.
Let's say I want the above table.
I can do that specific match based on the keyword that I can actually use over here.
And here you can see I'm able to get all the inputs.
And with respect to this, I'm able to easily scrap the entire data itself.
Right.
And one more, uh, type of data that you want to probably create.
Let's say, uh, you can also go ahead and use one Excel data.
So here I will just go ahead and write Excel.
Excel.
Oh this will be my sample underscore data dot Excel x okay.
So this data I will just go ahead and open it.
If you reveal in this explorer.
And if I open this particular data Excel file also, I can probably open it.
Okay.
It is not opening it.
Let's see.
I will just open this over here.
I will open a file.
Let's say here I go ahead and write, uh, name and age.
I'm just creating it manually.
So here I can go ahead and write name and age.
Let's say here I go ahead and write crush 32, Jack 34, John 31.
Okay.
And let me just save this.
Save as browse.
Okay.
I'll save it in my.
Folder.
Data analysis with Python.
And let me just go ahead and write over here as data dot xlsx.
Okay.
And I'll close this.
Okay.
So here, what I can actually do over here, uh, is that I will just go ahead and read this Excel file.
Okay.
In order to read this Excel file, I have to just use PD underscore, read Excel and say.
And here I am just going to use that particular Excel file name.
Right.
And it is nothing but data dot xls x.
Right.
The second parameter that I can give is something called a sheet name by default.
Uh, I have the first sheet over there, so it will be the first sheet name itself.
So let me just go ahead and execute it.
Uh, it says uh okay.
No module opened by Excel.
Okay.
So when you are basically using read underscore Excel you also have to install this.
So I'll go ahead and do the pip install this one.
So and the reason why I'm showing you all this error because this is for the first time that you really
need to do right.
And when you are doing this you may be facing some kind of error.
Again some error may come.
So here you can see that I'm also able to read all these things right now.
uh, one very important thing that I really want to show it to you after reading Excel that is converting
a pickle file.
Now, what exactly is a pickle file?
Okay, so if I go ahead and see what exactly is a pickle file?
Okay.
Pickle file.
What is a pickle file?
Okay.
If I just Google it, you can see that it is a process of converting a Python object into a byte stream
to store it in a file or database, maintain state across sessions or transport data over the network.
So let's say that I want to probably quickly go ahead and create some kind of pickle file where I do
some kind of serialization.
So this will basically be my Excel.
I can also convert this into a pickle file and I can also load it.
So two underscore pickle is the function name that I'm going to use.
And I will just go ahead and use my pickle file name over here something like this.
So if you go over here you will be also able to see this pickle file.
It will be completely serialized.
You will not be able to open this right.
And let's say if I want to quickly go ahead and read this pickle file.
So what I can actually do, I can just write PD, dot read, underscore, uh, pickle file.
And they are all different different files which you can definitely explore.
It is up to you.
And here I'll just go ahead and write DF Dot Excel.
Once I do that again I'll be able to convert that particular data into this.
But at the end of the day, this pickle file will also be very handy when we are developing machine
learning projects there.
Let's say we have to probably develop the, uh, pickle files or there we basically train the model.
We need to save it somewhere.
So we will also be using a pickle file over there.
Again along with this.
As I said guys, there are a lot of option if I just go ahead and write PD, dot read, write.
There are so many different different options read, pass, read, sequel, read SQL query, read table,
read XML.
So just go ahead and explore it.
Anyhow, I have tried to explain as much as I could in this, along with the installation, the errors
that we face, uh, the HTML pages that we have actually read.
So I hope you like this particular video.
I will see you all in the next video.
Thank you.