One Amazon reviewer recently mentioned this in a review:
But how? If you've read the book, you already know the answer: Selenium and PhantomJS.
from selenium import webdriver
driver = webdriver.PhantomJS(executable_path='')
These seven lines (including the print statement) can solve your Ajax loading problems. Note: There are also ways of waiting to return content by checking to see if a particular element on the page has loaded or not before returning, but waiting a few seconds usually works fine as well.
Key to this sort of testing is the concept of Selenium elements. This object was briefly encountered in Chapter 10, and is returned by calls like:
usernameField = driver.find_element_by_name('username')
Just as there are a number of actions you can take on various elements of a website in your browser, there are many actions Selenium can perform on any given element. Among these are:
myElement.send_keys_to_element("content to enter")
All of these actions can be strung together in chains, put into functions to act on variable elements, and can even be used to drag and drop elements (see Github: https://github.com/REMitchell/python-scraping/blob/master/chapter13/4-dr...)
*I know someone's going to take this as an opportunity to mention Flash, Silverlight, or other third-party browser plugins. I know, I know. You don't have to mention it. I'm hoping they go away! Sans extra software you have to add to your browser to make it work however, this principle holds true.