Hello guys.
So we are going to continue the discussion with respect to Python.
And in this video we are going to see data manipulation and analysis with pandas library.
Now pandas is a very important library where you can actually perform exploratory data analysis.
You can also perform a lot of feature engineering.
So in this video we will be talking about all this particular topics.
And what we are going to do is that I have also taken some kind of CSV file, a sample data set, which
we will try to perform some kind of EDA.
We will do some analysis.
Let's say if there is a missing values, I will show you how you can basically handle it.
So let's quickly go ahead.
And first of all, as usual we have to go ahead and import pandas.
So I will just go ahead and write uh import pandas as PD.
Um, and this is what is the library that we are going to specifically use.
Then uh, I will just go ahead and write PD dot read underscore CSV.
So let me just go ahead and use this read underscore CSV.
And here I'm going to basically use data dot CSV.
Right.
So data dot CSV is a file name.
And if you see over here I have already uploaded data dot CSV and these are my entire data.
You know, it also has a lot of missing values and all, so that will be amazing to check this.
Anyhow, I will be giving you this in the entire material section.
Okay, now let's go ahead and fetch the first five rows okay.
And for basically getting this particular rows what I'm going to write, I will just go ahead and write
DF dot head of five and let's see this.
So I have data like data category value product sales and region.
So these are some of the information that we have regarding that specific data.
If you want to also go ahead and check out the last five rows I can go ahead and write DF dot tail.
And here I can just give the last five or last ten whatever things, whatever number you give over here,
that many number of records you'll be able to see from the end.
Right?
So from the end, these are my last five records that you are able to see over here.
Perfect.
Uh, one more thing is that let's say you want to go ahead and see the, uh, uh, you know, the entire
if I just go ahead and write DF dot, describe all the statistical Mystical things that you want, right?
Like count.
Mean standard deviation.
Minimum 25 percentile, 50th percentile, 75 percentile.
And over here you'll be able to see only values.
And sales has been taken remaining all it will just be considering as a categorical column.
Okay.
Uh, but if I probably consider value and sales these are numerical columns.
So it will just try to find out all the statistical analysis that we basically require for both of this
particular column.
Now, uh, this is there.
Now let's say I want to also go ahead and see the data type.
So I'll just go ahead and write df dot df dot d data types.
That is nothing but d types.
So here you can see data date is nothing but object.
Object is nothing but string.
Category is object float is 64.
Product is object sales is float 64 region is object.
Let's say if I want to probably go ahead and convert this into date time, we can actually do it with
the help of, uh, data analysis.
Again with the help of pandas.
Also we'll be doing and we'll see those kind of projects when I will be creating a separate section
for EDA.
Okay.
And how you can probably do some of the amazing EDA.
All those things I will be discussing about.
Okay, now let's go ahead and discuss about handling missing values.
Okay.
Handling missing values.
So first of all, what I'm actually going to do is that once we read any data right, they will be chances
that whether there is any missing values or not.
So if I go ahead and write DF dot is null, right.
And I'll just go ahead and execute it.
So here you'll be able to see with respect to my entire CSV file, wherever there will be false.
That basically means there is no missing values.
And let's say if there is some values like true, that basically means in this particular column, in
this record, this value is missing and it does not have any any values over there.
Right.
So obviously this is a missing value.
Similarly you can see one more true is over here.
One more true is over here.
But just seeing like this it becomes very much difficult.
So what I will do is that, uh, you know, to find out the records, I will just go ahead and write,
uh, something like this.
And whichever are true, I will be able to get the answer.
Just a second.
I'm able to get an error.
No worries.
So I'll just go ahead and see this.
Now you can see over here.
Uh, Nan.
Nan Nan values are specifically coming, so I think I've done some kind of mistake over here.
So I'm writing.
Hey, DF dot is null or or.
Let me do one thing.
Very simple.
DF dot is null if I'm going to write and I'll use dot any operation.
And let's say I'll go ahead and write.
Access is equal to one.
So here you will be able to see that, uh, you know with respect to every record.
Right.
See ax is equal to one basically means what right.
With respect to every record, you will be able to see where there is a missing values and where there
is not a missing values.
Okay.
So these are some of the ways that you can probably check it out.
Okay.
And uh, if I remove this ax is equal to one by default it will take ax is equal to zero.
So here you can see for date it is false.
That is date is basically having all the values category.
It is false.
Value is basically having a missing value.
So it is showing true.
sales is basically their true, right?
So all these things are if you also go ahead and use one more inbuilt function called as dot sum.
Okay.
So if I go ahead and use dot sum here you'll be able to see like in value.
There are three missing values in sales.
There are four missing values right.
So all these missing values are specifically visible just by doing this.
Like I'm just over here.
What I'm calculating DF dot is null wherever it is true.
All the summation is basically going to happen.
And this is by default going to happen with the help of column wise okay.
So here you can basically see that okay I feel like it is there.
Okay.
Now uh, the thing is that, uh, you know, value is basically having a missing value and sales is
basically having a missing value.
So can I go ahead and fill those missing values.
So it is very much simple.
Let's say if I don't know what all missing values are there, I can basically go ahead and right fill
in a right fill Na.
And if I just go ahead and write fill Na with zero right by default some zero value, then you'll be
able to see that all these values, wherever there is a missing value, it will be replaced with zero.
It will be replaced with zero.
It will be replaced with 0000.
Everywhere it will be replaced with zero.
And if you are doing this kind of replacement, you have to also make sure that you go ahead and fill
this in another, or you create a copy of that particular data frame.
And usually in most of the scenario we don't fill with zero, because if I have a numerical value based
on a distribution, I may want to probably if it is a numerical value itself, I may want to replace
the missing value with the mean of that specific column.
Okay.
So that also I will try to show it to you.
Okay.
Now how to uh fill.
And that may be the scenario over there filling missing values okay.
With the mean of the column okay.
Let's try to do this okay.
So here what I will do I will just say hey DF of let's say I know what all missing values are specifically
there.
I'll say okay, sales has a missing value.
They are on four values over there I will write hey DF of sales right will be nothing but.
Or let me just go ahead and write sales underscore fill in.
Okay.
So this will be my new column.
And here what I'm actually going to do I'll say hey DF dot of sales okay dot fill in.
Now if I want to really do the fill na over here, how do I do it if I'm using this particular function
fill n a.
So here I will just go ahead and write df of sales.
Right.
And I know sales has that particular missing value.
So I will just go ahead and take dot mean right.
So what I'm actually doing wherever the Nan value is basically present in that sales column, I'm going
to replace with DF of sales dot mean the mean value.
And now if I go ahead and see it and if I just go ahead and print the DF.
So this is my new column that you'll be able to see sales underscore fill in.
So it was 754 over here.
Then you can see wherever there is a Nan here I'm able to get the mean value of this particular column
okay.
So here also you will be able to see somewhere you'll be able to see Nan.
Again I'm having this particular value.
Again there is a nan.
Nan basically means not a number okay.
And here we are specifically having the values wherever there is a missing value.
That is basically considered as Nan by the data frame and how easily you.
I have actually created this particular column.
And just by writing one line of code, you'll be able to see just by using this dot fill.
And I've replaced all the missing values with mean.
Okay, so these are some of the amazing methods that are already present in data frames.
But in EDA we will be looking into more examples where we'll be also working with time, data and all.
So first of all, let's do this entire analysis with the pandas.
And later on you'll be seeing how I'll be combining NumPy, pandas and some of the visualization libraries
to do some amazing EDA.
Okay, now let's do one thing.
Uh, again, um, let's say that you want to change the data type.
See, when I go went and wrote DF dot d types.
Right.
D types.
So here you can see data is an object category right.
And here you can see categories and object right.
And product is an object.
Object basically means string.
And let's say I want to probably go ahead and convert this into a numerical values, or see if I probably
consider date.
I want to convert this into date time.
I will be showing you this example later on when we do the EDA, but let's say that I have some kind
of categories product and all I want to change.
Uh, or here I have Float64.
I want to probably convert this into an integer value.
How do I probably do that?
Okay.
And let's say if I have a specific column I want to rename that column.
How do I do that okay.
So some of the analysis that we specifically do let's start with renaming columns okay.
And here what I'll do I'll just use this date column and I'll try to rename it.
Right.
So here we have in DF a rename function right.
Rename a function itself.
And here you'll be seeing that I'll be providing my column name like column name which I really need
to rename.
So here I will just go ahead and say okay it is date and I need to rename this to Sales date.
Okay.
So let's say I want to probably go ahead and rename in this specific way and all this entire thing will
be replaced back in the DF.
So if I go ahead and now see DF dot head, you'll be able to see, hey, I will be able to get all the
answer.
Okay, I'm getting an A set object is not callable.
We have to probably see okay I have to use colon okay.
So this should be a key value pair right with respect to the column.
Now if I go ahead and see DF dot head you can see sale data or sales data I will just go ahead and write.
But this time again I can replace this sale date with sales data.
And here you can see that I have actually replaced the entire column name.
And similarly you can also do it.
And remember this.
You really need to give in the form of key value pairs because this is completely dictionary type okay.
Now let's go ahead and change the data type.
Let's say I will go ahead and change data types.
Now in data types let's say I want to probably change the sales which is in float 64 back to in 32 or
in 16, whatever things I want.
So I will just go ahead and write DF of uh, let's say value underscore new.
So let's go ahead and create this new column.
And what I'm actually going to do is that I'm going to probably take this entire value.
And I'm just going to change the data type.
So I will go ahead and write as type width nothing but width float.
Okay.
Now if I go ahead and write df dot head, and here you'll be able to see that I will be able to get
a new column that is value underscore know.
And I've just converted this into float.
But what I will do I'll just go ahead and convert this into int right.
Uh cannot convert.
Uh okay.
It is having a missing values also.
Okay.
The main thing is that it is having a missing value over here.
So what I will do I will just go ahead and write fill any.
Okay.
And uh with fill in I will just go ahead and write uh df of value.
Value.
I'll replace with mean okay.
Mean dot mean df of value, dot mean okay.
And then I will convert that into type int.
Okay.
Now let's go ahead and see this.
So here you can see that now I'm able to get all the integer values.
And here what I did I just filled all the Nan values with this mean.
And I try to replace with that integer value okay.
So here you can probably see this is my new thing with a new column.
Okay.
Um, let's say I want to apply a function to one of the columns.
Right.
So for that obviously I've seen lambda right.
Let's say um, I will go ahead and say this will be my new value.
And let's say my new value rate has got increased by 10%.
Okay, 10% or IT or double of this particular value something.
Right.
So here I will take this DF of new value or value.
Okay.
Uh whatever is the numerical column I'm creating a new column.
Now here I'm going to apply any function.
So in order to apply I'll just use this dot apply function.
I'll call the function name inside this.
So you can go ahead and create any function that you want.
But I'll show you one application with the help of Lambda.
So here I will go ahead and use Lambda function.
I will go ahead and apply x.
And I'll just give my expression x multiplied by two.
Okay.
So if I go ahead and see this and now if I go ahead and write DF dot head.
So here you can see initially my value was 28.
Now it is 56.0 78.0 64.0.
So that basically means I'm able to apply all this kind of operations specifically in, in this, uh,
just by using one kind of function.
And this can be an inbuilt function.
It can also be a custom function with respect to custom function.
You just need to call it by providing the parameters.
I think that will be more than sufficient.
Okay.
Uh, so this was with respect to the uh, some of the manipulation operations.
But when we see the full fledged EDA, we will get to know more about it.
Okay.
Uh, now coming to the next thing, uh, a very important topic altogether.
And most common things that we specifically do is, uh, here I'm going to basically go to write data
aggregating And grouping.
Data aggregating and grouping.
Okay.
Now data aggregating and grouping basically means that with respect to any column, let's say I want
to probably go ahead and aggregate it.
I want to probably go and do the things right.
Uh, I want to probably group based on different different categories.
I can actually do it.
Okay.
So let's say this is my DF dot head okay.
DF dot head.
Okay.
Let's say based on products.
Okay.
I want to do an aggregation and I want to do the mean of this specific value.
Can I do it.
So let's see this.
So here I will just go ahead and write grouped underscore mean is equal to DF of product.
Okay.
I'm just going to group by this specific product.
Or let me just go ahead and write DF dot.
There'll be a method which is called as group by.
And inside this group by I am going to give my categorical column.
Let's say my categorical column is product okay.
And with respect to this categorical column now when I'm grouping based on product I have to also make
sure that I need to calculate the mean like product one.
What is the average mean product two what is the average mean?
Something like that right.
So here I'm going to take my value okay.
Dot mean right.
So once I probably do this you'll be able to see I will just go ahead and print my grouped underscore
mean.
So here based on product one it is 46.21 based on product two it is 52.8.
And based on product three it is 55.16.
So that is what is the mean I am able to get it.
Let's say I want to probably, uh, you know, uh, do the group by based on two categorical variables
like let's say product or sale, product or region.
I can also do that.
So here what I'm actually going to write I'll just go ahead and write DF dot group by okay.
And here I'm basically going to use my first uh first column that is uh, let's say uh one first categorical
column that is product.
And here I will give it in the form of list.
Right.
So here the next parameter that I'm actually going to specifically give.
Okay.
So let's see what is the next column region.
Okay.
So let's say I am trying to group by by two different categorical variables.
And then I go ahead and take my value.
And I after giving the value I will just do the dot sum.
And this time I'll not do an aggregation function dot mean.
Instead I'll do dot sum like total sale based on products and region based on regions.
I'm trying to just trying to find out the what is the sale with respect to that.
Okay.
And now, uh, this will basically be my grouped sum group underscore sum.
And here I'm just going to print my grouped underscore sum.
So here you can see.
So based on product one you can see in East there is 292.
In north there is nine South 100 west 246.
So based on two categories I'm actually grouping it.
And I'm doing the sum of all the values.
I can also go ahead and do the mean similarly.
Okay.
So let's go ahead and do this And this time I will just.
Oops.
I will just take the mean.
Okay, so based on every region and every product, I can also go ahead and calculate the mean.
Okay.
So here you can see in each there is 41.71.
In North there is 4.450.
In south there is 50.0.
So all these kind of things are basically there.
Okay.
Now uh, let's say I want to apply multiple aggregate functions.
Okay.
So aggregate multiple functions.
So you that also you can actually do it over here.
So I will just go ahead and write grouped underscore agg aggregate.
And here I'm going to write group by operation DF dot group by let's say one of the categorical column.
This time I will just take based on region okay.
So this will basically be my region.
And based on region what I will do I will take the value okay.
I will take the value.
And after taking this value I'm just going to apply all my aggregate function.
And this will be in the form of list like Mean some.
And let's say one more aggregated function that I want to use is count.
Okay.
So all this operations I'm going to basically apply okay.
And now if I go ahead and just go and open it.
So here you can see based on East the mean is 42 north this much south this much west this much.
And again based on east some is this much north.
Is this much south.
Is this much west is this much.
And count you have all this particular data.
So this was some of the amazing mechanisms or probably creating or doing some kind of aggregation function.
Again.
Uh, that's more about practice.
The more you perform different, different EDA, you'll be able to do all these things, but just understand
whatever things we are able to do in SQL, right?
All the things we are also able to do it over here.
Now, uh, let's go ahead and discuss about merging and joining data frames.
Okay.
So again, a set of operations that we can actually do with the help of merging and joining data frames.
So let's go ahead and see some examples with respect to this.
So guys in order to understand merging and joining data frames, what we will do is that we'll go ahead
and create some sample data frames okay.
So here you can see with respect to the F1 and F2 I have keys like ABC and keys are nothing but columns
A, B, D and values are one, two, three and four five, six.
Let's say I'm going to print this.
Let's say this is my df1.
And along with this, this is my df2.
Okay.
Now, you know, over here is that, uh, these are my columns, right?
Keys and value.
Now, can I merge?
Can I merge a two data frame and on a specific key, you know, and like I want to merge this two data
frame.
And here you can see one of the column is key.
Right.
And it is let's say there are some of the keys that are matching.
And we need to merge based on this okay.
So first of all we will try to merge this data frame data frame on on the key columns.
Okay.
On the key columns.
So here what I'm actually going to do I'm basically going to write pd dot merge df one comma df two
okay.
Now if you probably go ahead and see each and every parameters.
Right.
So if I'm just going to merge this Df1 and Df2 right up here, there are a lot of parameters.
Let's say something like left left on uh there may be another parameter like on, you know.
So let's try one by one with respect to different different parameters.
So here I will just go ahead and write on okay.
So I'm going to probably merge Df1 and Df2 in such a way that I will use a specific key.
Let's say that particular key is my key column okay.
And the kind of join that I really want to do between this particular data frame is inner.
So I will just go ahead and use a parameter called as how okay.
So once I execute this see the magic.
So this is my key.
This is my value one.
This is my value two.
So in my data frame one I had key value one.
In my data frame I had key value two.
Now based on a single column I'm trying to merge both this particular data frame.
So here one first parameter that I have given is obviously Df1 and df2.
The two data frames, which is basically getting merged and coming to the next one that is on which
column name.
Right.
Column name is key, right?
Because this is the this is the column which is matching between both this particular data frames.
Then how the join should happen.
So obviously I'm going to use the inner join.
Inner join basically means from the left hand side.
From the right hand side both will be basically considered.
So here you can see a b.
So if I go ahead and see based on the inner join a B is matching in both this particular column.
Right.
Both this particular data frame a b a b value one values are one two.
So value 112 is basically getting displayed.
Value two will be four five.
So value two will be four five.
So this is how the merger is basically happened.
Now what we are going to do we are going to do this same merger on different different joining parameters.
How?
Let's say if I go ahead and do outer outer join.
Okay.
So here you'll be able to see both the data frame will be considered.
So key value one and value two.
So matching key is nothing but a b.
So that is the reason you will be able to see both this particular data.
But with respect to see I do not have anything as in my data frame.
Two I do not have anything like C key, right?
So that is the reason it is basically showing you Nan value.
And similarly in the case of data frame one when our key value was d in data frame two, nothing was
there in data frame one.
So here you don't have anything like D is equal to something.
So that is the reason here you'll be able to see that the value is nan okay.
Now in outer join both the data frame is considered and each and every rows and keys will be probably
considered.
It will be joined if there is a not if there is no value matching with respect to that, it is just
going to give you a Nan.
Okay.
So this is also one type of join that we specifically do.
The other join that I'm actually going to use is like left outer join right now in the left outer join
left data frame will be considered or given more priority with respect to all the keys that are available
right in Df1.
Over here I have given left Df1 and the right it is Df2.
So when I am doing left join Df1 will be given more priority based on all the keys that is probably
present.
So here you can see a b.
So this value is getting matched.
But with respect to key I do not have anything in my data frame two.
So that is the reason Nan value is basically given.
Okay.
So this is also one more example.
One more join is there which is called as right outer join.
So here on the same key if I'm doing right outer join now right side data frame will be given more priority.
So that is the reason you'll be able to see these over here.
Value one will be Nan because in Df1 you don't have anything something related to key D right.
And value two it is basically displaying six.
So here right side data frame is considered more important.
And all the keys are basically taken over there.
And based on that, whatever is matching from the left data frame that is just written.
If it is not matching, none will be given.
Right.
So these are some of the examples with respect to merging the data frame.
Uh.
Uh, and this will be very much important because whenever you work with any kind of data.
Right.
You will be able to see that.
Hey.
Uh, yeah.
Some or the other way.
We can really do a lot of data analysis.
Okay.
Now, finally, uh, there is also one more sales data that I've actually created for you.
Just go.
Ahead and try to do the same thing that we probably did it over here.
Right.
Group by.
Operations and all.
You can practice it.
I will be putting some kind of data sets over there.
In this particular material itself.
So just go ahead and do some data aggregation operation like how I did product by and.
All right.
Um, you also need to calculate the average sales based on the product.
And.
All right.
So I hope you like this particular video.
Uh, I think we have discussed a lot with respect to this and more things will be discussing when we
go ahead with doing EDA.
Right.
So yes, this was it.
I will see you all in the next video.
Thank you.