Selenium is a hard guy if you get to use it extensively.
I’ve been there.
It will at some point make you feel like you’re doing everything wrong and before you know it, frustration sets in.
Like i said, I’ve been there.
The thing is:
There’s no documentation that tells you what issues you might encounter with selenium with time.
You only get to know this issue eventually.
I’ve been using selenium for some years majorly for web scraping and testing.
So, i’m writing this article to show you those problems you’re 90% likely to ram into with selenium and their solutions according to my own experience across the years.
Thanks to Stackoverflow and trial and error with no directions.
It’s funny but you might have already encounter some of this selenium issues before.
This post will cover most selenium issue and their solutions.
Let’s get to it.
Mismatched Web Driver
There are a lot of time, i mean a lot. I just get the below message out of the blue
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 83
This issue does occur from time to time even after your projects has been running fine for some time.
It happens when the version of your browser is different from that of your selenium driver.
So what’s the cause ?
It’s caused by update to the browser and that’s why it’s more frequent on the Windows OS.
For example, installed chrome is version 79 but your chrome driver is version 78 or version 80.
This usually happens when your browser update it’s self.
So how do we fix this ?
By either upgrading or downgrading either of them. Do whichever .
All that matters for you to fix this issue is to make sure the browser version matches that of the selenium driver.
If your chrome browser version is 80, make sure your chrome driver is updated to 80 as well.
Or, If your Firefox browser version is 80, make sure your gecko driver is updated to 80 as well.
Also If your driver were to be 79, you could as well just downgrade your browser to 79 but i wouldn’t recommend downgrading.
And everything should be back running fine.
You can download chrome webdriver here
And geckodriver for Firefox here
Missing Web Driver
See the below error:
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
This happens mostly with newbies.
This issue of missing driver doesn’t happen except only if selenium can’t find your chrome driver.
You have to make sure your chromedriver is in path or simply just copy it in the same directory as script .
if you want to step up and handle it like an elite, you need to explicitly pass the location of your driver to selenium.
something like this:
browser = webdriver.Chrome(executable_path=".\\assets\\chromedriver.exe")
This way you can place the driver wherever you want but just need to pass in the path like i did above.
I recommend this method. You should stick to it.
It sounds weird . LoL
No such thing as untimely timing. I actually coined that term.
So what is it ?
It’s a situation where your script looks for elements on the page when it’s not yet ready leading to crashes.
This happens alot.
So, how do we handle this ?
By waiting patiently for the script by using the selenium explicit wait.
Note that this is different from implicit wait. Implicit wait applies to all elements. So specifying an implicit wait of 2 seconds will make selenium wait 2 seconds before every element on the page.
There are different reasons for waiting for an element but the most common is waiting for the element to be located on the page.
The below is a python code that waits for an element with ID attribute “page-head” until it is located on the page.
WebDriverWait(c, 15).until( EC.presence_of_element_located((By.ID, 'page-head')))
If it isn’t found within the wait time specified, a Timeout exception will be raised.
Tip: If you’re sure the element will always show, you can use a longer wait time .. or a short one but you need to handle your Exception properly.
That way you prevent your program from crashing.
These are errors that are specific to the browser and has nothing to do with selenium itself.
The bad news is there’s no way your script can communicate the presence of this error neither does it get notify of it.
It’s the hardest, you can only handle it by watching out for indicators that might point out that something went wrong.
It just tries to move to the next instruction in your program.
So it means no Selenium exception handling for you. Sorry.
The good news is, after many years on this, i came up with my own custom solutions.
It goes thus:
It’s by using the error constant shown by your browser to identify the state of the browser.
A lot of While loop, checks and exception handling is involve.
Using Google Chrome as example, One common browser issue is when the browser itself timeout due to internet break or bad coverage or a proxy failure.
We can watch out for this by looking for the text ” ERR_CONNECTION_TIMED_OUT” in the browser page source and act accordingly.
something like this:
if " ERR_CONNECTION_TIMED_OUT " in browser.page_source: print('Browser has time out .. refreshing') browser.refresh()
The above code refreshes the browser.
if not, the program just goes ahead and crashes because selenium wasn’t aware of the problem.
There are several ways i go about this.
One way is running this checks as a watchdog thread in the program process.
You should drop your comment if you need me to elaborate on a specific type.
User, Password Proxy Authentication
The process of using proxy password authentication with your program might go smoothly with libraries like requests, but this isn’t the case with selenium.
Before we proceed.
You should always use IP proxy authentication if you’re working with selenium if it’s an option for you. User:Password authentication is a pain in the jaw.
Especially when you need to rotate the proxies on the fly.
Okay, let’s say this is the last resort and there’s no way out except to use it.
There are two ways to go about it.
- First is passing the proxy authentication and IP to the browser itself.
- Second is loading the proxy as a plugin on browser startup
The first one is a browser specific solution. See the below code for Google chrome browser solution