In this video
we're going to scrape this text from this website, automated.pythonanywhere.com.
So that's the current webpage is the homepage and we will be using selenium for this.
So selenium is a Python library, a third party python library that is used to automate browser actions.
That includes things like clicking on a link and then on the next page doing something
else, such as copying some text.
Or perhaps you want to log in.
You're going to learn that's doing the next videos.
So you can write a Python script to log into this website, enter the username and password
of course, everything is automated.
Selenium and Python will do that through that Python script and then do something else on the next web
page after the login and so on.
But for this particular video, we're going to do something simple.
So just scrape this text with selenium.
So which means that we're going to get the text and load it in our Python program and maybe print it
out, or you can do whatever you want then with that text.
So let's go to Repl.
And here you can see that I have created an empty folder for this section.
So browser automation and web scraping.
And then I'm going to create a new Repl in this folder using Python and let's name it Scrape Simple
Text with...
With
Selenium, create Repl.
And so this is our empty Repl.
It's a good idea to install selenium beforehand.
Selenium,
like that, press the plus icon and wait until the installation completes.
Sometimes though the installation can hang, so you'll see that these messages will not disappear.
If that's the case, you might want to reload the browser and see here if selenium has been installed,
if it's listed here, if not, try again until it is installed.
And always you can also just import selenium here and when you run the script it will install automatically.
So you have these two options.
Use the packages menu or directly import.
My installation was completed.
And now before we import Selenium and start writing the code, we need to do another configuration.
But you need to do this configuration only if you created a new Repl.
If you forked my Repl, which is attached in the lecture resources as a link,
you don't need to do this step.
So for those who created a new Repl,
you can go to these three dots and go to show hidden files and you'll see these two files.
If you fork my Repl, you will not see this files when you show the hidden files.
So this is only for those people, again, who created a new Repl.
So you need to go to
replit.nix.
And then you need to go in here, make a new line and add pkgs.chromium and then
again pkgs.chromedriver.
So we need these to packages to be able to scrape data with selenium.
So close that file.
You can then hide those files again.
So you'll see only those working files of your projects.
And now we can go ahead and import selenium here.
But what I'll do is I'll just import from selenium
the webdriver.
Web driver is basically a tool which allows us to instruct the behavior of the web browser.
So the web driver makes everything possible, and we have access to that web driver from Python through
selenium.
And the first thing, the first logical thing to do is to create a driver variable and use webdriver.
Actually, we don't use the webdriver.
We use the Chrome class of the webdriver.
Normally, Chrome, this class expects the executable path of a chrome driver, which you can download
and install from the web.
So that's the chrome, the driver for the Google Chrome Web browser.
Every Web browser have a different driver.
However, here on Repl, you don't have to provide that path to an executable because it's automatically
configured, which makes our programming job easier so we don't have to go through downloads and configuration
actions.
So just like that.
And while this is the minimum thing you can do to configure a driver, there are other things, other
options that you add to your driver.
So this is what I'm going to do now.
I'm going to create some options using webdriver.ChromeOptions.
So this can be an empty class call.
So we call the class, but no need to provide any arguments.
And so this is an instance of this.
Now it's the ChromeOptions object, and in that options
we add arguments.
These are basically flags and these are what I was referring as the options.
The first flag we want to add is disable-infobars.
So disable dash infobars.
What this will do is on a browser sometimes we have these bars here which display some information
such as of a memory is low or things like that.
Now those bars may interfere with the script accessing, taking actions on the browser.
So the script may want to click somewhere, but that popup, that info bar could confuse the cursor
and so on.
So we add that's to disable them.
Other options that is good to have is to add arguments.
The next one is start
maximized.
So that will start the browser as maximized.
That's also a good thing because some web pages, for example, may change the contents when you resize
when you make them smaller.
So we want to access the maximized version of the browser.
Another one is.
Disable-dev-shm
-usage.
This particular option is to avoid some issues that occur when you interact with a browser on a Linux
computer.
And this here or Repl is a Linux machine, so it's good to have that to avoid some particular issues.
The next one is a let me be consistent here with the quotes.
Just for consistency.
And next, no-sandbox.
Now browsers for security.
They use something called sandboxing.
And if we want our program, our scripts to have greater privileges on that particular web page that
the script is going to access and scrape, we want to add this option so that sandboxing is disabled.
Next, we have some experimental options which are new and we have exclude.
Switches and in a lists
comes enable
automation.
Now, this option and another one that I'm going to add here.
So these experimental options are options that help selenium to avoid detection from the browser.
So some browsers, web pages, they don't like scripts.
So we enable our scripts to access those browsers.
So if the other one is not actually that. It's an add-argument option, it's not an experimental one.
So this is disable-blink-
features=.
Equal to out to AutomationControlled.
And that's it.
So these are the options.
Set options to make browsing easier.
Right.
And then we need to provide that options variable in here.
So options now has all these arguments.
This variable contains all those arguments.
Now, what I would prefer to do here is to just select all these, indent them with a tab and then put
them inside a function which I'd like to name get_driver.
So I'm trying to keep things organized here.
So this function is responsible to create a driver and return that driver.
That's good.
However, the driver is currently empty.
It's not connected to a webpage.
So we want to connect it to a web page just inside the function.
So driver.get and here this gets the URL, so the URL is this web page, right?
So we said that we want to scrape this text and that text is in this web page.
Therefore, we provide that web page as a string in here.
So, it starts with HTTP and so on.
Now we return the driver for this particular web page.
And then I would like to create another function.
So let's name this main.
And here I'm going to get the driver from get_driver.
So we call that function.
Therefore this variable now holds this driver because that function returns that.
So it makes sense.
And with that driver, now we extract an element from that driver.
So driver dot find
element by X path which is done here.
Yes.
And this gets the X path as a string.
What is the x path?
Well, let's find that out.
So we want the x path of this text, right?
Therefore, we select somewhere over here and go to inspect element.
You may need to enable your browser developer options if this inspect element options doesn't show up.
So you have to maybe google something.
How to show inspect menu on your particular browser, which could be Firefox, Chrome or Safari, etc..
Anyway, so a web page is made of HTML elements.
For example, this element that you see here.
This creates this text.
means the text is quite big, the font.
And if I click over here, you see that the text is a inside this tag.
You see here, slash here.
So these tags, html tags
hold this text and if I click over this tag and I go to copy and I go to X path, I just copied the
Xpath of that text and if I pasted there,
you see, we're going to get this string.
So that's what I copied from there.
Now,
let's return to this element and see what it is made of, how it looks like.
And therefore I'm going to print the call of the main function.
So I'm calling main here.
Right.
Print main with a print to print out that element and run.
It seems that we have an error.
So let's look at it. Experienced string,
bytes or path like object, not options.
So the error started in line 22 when we called the function and it changed back to line 18 when we get
the driver, which goes back to line 13, options.
And that's where the roots of error is in our script.
Then we have library errors here.
So line 13 is these options?
So it seems that options is actually a positional argument.
So we have to mention it's name options equals two options.
So the name of the arguments and the variable,
this one in here.
Right.
So let's run again.
And this is the output.
So that's the element.
This one.
And we also have.
A warning, deprecation warning which means this find element by, so this methods has been
deprecated, which means it's a good idea to remove it because in the later versions of selenium,
it's going to be removed as a method.
So we can take measures now and use what this suggests, find element.
So let's do find elements which gets this two arguments by and value.
So I think without even looking at the documentation of this value, I think what they want us to
do is say
by xPath.
So we declare that we are accessing this elements using the X path options and not other options
such as fines, element by text or by class, etc..
So by xpath is good, and this would be value.
So yeah, that should do it, run again.
And here is a browser also.
It's being access.
We don't see anything now because this is just a simple operation.
So that's the element.
Right.
And to give the actual texts, you can just do element.text here in the return and run.
Yeah.
Here is the text.
So that's one in there.
And this concludes this video, so you learn how to set up selenium.
It was a long one, probably because we have to configure lots of options.
But the good news is that you going to use this function just as it is in future videos of this section.
So I'll talk to you in the next video.
See you.