This guide is for folks new to data collection through Python web scraping, where installing development tools on your laptop remains a learning curve for you. 

I have a similar article where we install and run ChromeDriver on Windows.

I will provide some demo code for you to use in my public Github repository: https://github.com/jonathan-moo/python-webscraping-demo.

Why are we using ChromeDriver?

To scrape a website, we would load a website accessible via the Internet on a browser.

As a browser renders the content (HTML tags), we can extract information programmatically through the browser and load it into our Python environment as variables in our development environment.

This Chrome app is being controlled by my Python script or Jupyter Notebook.

There arises a question. Which browser should we use to render the content?

As the name suggests, it would be Chrome, and Google provided a dev kit to spin up a browser programmatically.

Prerequisites

You’ll need Homebrew installed in your Terminal, where Homebrew will manage the ChromeDriver package for you. Check the Homebrew webpage to find out how to install the package manager.

For a quick reference, you can run the following code in your macOS Terminal to install Homebrew:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

If you have already installed Homebrew, it is good practice to update your Homebrew installation by running the following commands:

  • brew update
  • brew upgrade

Install ChromeDriver using Homebrew

Run brew install chromedriver to install ChromeDriver. By default, it will download the binary file to the /usr/local/bin/ folder.

MacOS will automatically quarantine the ChromeDriver app within your local machine, which makes it unusable because it will not allow you to run it. It will suggest that the ChromeDriver developer is not trusted.

It is a safety mechanism that MacOS has to protect its users from downloading and using malicious apps, but in this particular scenario, it is inhibiting us from using ChromeDriver. You’ll need to ignore the warning and remove the quarantine by running the following code:

xattr -d com.apple.quarantine $(which chromedriver)

Every time you upgrade the ChromeDriver version, you must remove the quarantine limitation to use the app.

To test if the ChromeDriver app works and to check its version number, run chromedriver –version. You should see something like this:

(base) Jonathans-MacBook-Pro:~ jonathanmoo$ chromedriver --version

ChromeDriver 123.0.6312.122 (31f8248cdd90acbac59f700b603fed0b5967ca50-refs/branch-heads/6312@{#824})

Install Selenium Binding With Splinter

While installing splinter as a Python package, you’ll need to add bindings for Selenium.

pip install splinter[selenium]

You can check the official docs here: https://splinter.readthedocs.io/en/0.21.0/install/driver_install.html

Test run your ChromeDriver

Run the /notebooks/test_your_chromedriver_setup.ipynb in your Jupyter notebook to test.

The code would start a Chrome browser with the heading, “Chrome is being controlled by automated test software.”

This is where you can continue to build on your code to scrap data from websites.