Website Automation with Python, Firefox and Selenium

An appropriate alternate title would be: How to control a web page or test your website with Python. Recently, I was given the following assignment:

1) Login to a website
2) click on a toolbar and load a specific search form
3) enter some data into one of the fields on the form and search
4) if found, click another button
5) repeat

Of course, I was supposed to do all this with Python, my favorite programming language. My first thought was to use the Mechanize package, but while I could login to the website using that, I was unable to get it to load the form. Why? Well, unfortunately the aforementioned toolbar was made using javascript and it seemed to be generating the form too. For whatever reason, the Python port doesn't support loading javascript elements, although I did find out that the project it's based on has a javascript plugin, so there is hope that it might eventually. Thus, I went looking for another solution and recalled that Selenium might fit the bill. In the end, it worked quite well. Since I won't be able to show you what I did exactly because it was for an internal project, we'll be automating Gmail instead. Let's get cracking!

What You'll Need

• The Selenium IDE Firefox plugin (which is actually a set of plugins)
• The Python bindings for Selenium which can be found on PyPI or you just easy_install

Automating a website

I experimented a bit trying to come up with a good example where I would login to Gmail or Yahoo, but I had problems with each. Gmail's frames are uniquely named every time you login, so I couldn't find the frame when I'd run my code. Yahoo is somehow blocking the body portion of a new email from selenium, so I wasn't able to record what I typed there. As you've probably figured out, I was hoping to show you how to send a bunch of emails in an automated way just for the fun of it. Oh well.

Instead, I decided to get a little meta and just script my own blog's contact form. It works well enough for this example, but you can't actually submit the form because of the Captcha. However, this should give you the general idea. Here's how you go about it:

Step One: Open a new instance of Firefox and then start your Selenium IDE plugin (Tools menu, Selenium IDE)
Step Two: Go to your website of choice and do what you want to automate. In this case, I clicked on my contact link and then filled out the form. (see screenshot below)
Step ThreeIn the Selenium IDE window, go to File, Export Test Case As and choose Python 2 (WebDriver). There's also a Python 2 (Remote Control) that you can export too, but you won't be able to run that script without some extra software.
Step Four: Run the web driver script and see if it works. I usually export the remote too as it helped me figure out some of the field or CSS names that the web driver script didn't have. For example, in my original project, I didn't realize I needed to switch frames until I looked in the remote code.

Now let's take a look at the code that Selenium IDE created based on my silly blog example:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
import unittest, time, re

class BlogDriver(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Firefox()
        self.driver.implicitly_wait(30)
        self.base_url = "https://dev.pythonlibrary.org/"
        self.verificationErrors = []
    
    def test_blog_driver(self):
        driver = self.driver
        driver.get(self.base_url + "/")
        driver.find_element_by_link_text("Contact the Author").click()
        driver.find_element_by_id("si_contact_name1").clear()
        driver.find_element_by_id("si_contact_name1").send_keys("Mike Driscoll")
        driver.find_element_by_id("si_contact_subject1").clear()
        driver.find_element_by_id("si_contact_subject1").send_keys("I love your blog!")
        driver.find_element_by_id("si_contact_message1").clear()
        driver.find_element_by_id("si_contact_message1").send_keys("Python rules and your website rocks!")
        driver.find_element_by_id("si_contact_captcha_code1").clear()
        driver.find_element_by_id("si_contact_captcha_code1").send_keys("22HU")
        driver.find_element_by_id("fsc-submit-1").click()
    
    def is_element_present(self, how, what):
        try: self.driver.find_element(by=how, value=what)
        except NoSuchElementException, e: return False
        return True
    
    def tearDown(self):
        self.driver.quit()
        self.assertEqual([], self.verificationErrors)

if __name__ == "__main__":
    unittest.main()

As you can see, Selenium IDE created a simple unittest script. If you run it, a new Firefox instance will load and it will fill in all that stuff and try to submit it. Of course you'll receive an error from the website since the CAPTCHA I recorded will most likely be different from the one you see. The script runs pretty fast and it's fun to use Selenium to learn about other people's web element naming conventions. In the script I created for our internal project, I removed all the unittest stuff and created my own class that looped over a whole bunch of inputs to look up so we could scrape the data.

The one thing that Selenium doesn't do is actually automate Firefox itself. You can't make Selenium click any of Firefox's menus or bookmarks. However, if all you need to do is automate a website, then this works great! On the other hand, if you really need to be able to click Firefox's menus and what-not, then you might want to look into the SendKeys package. It's for Windows only though. The Selenium script should work on Windows, Linux or Mac.

Hopefully this article will help you get started testing your website or get you going on an automation project of your own.