This script creates a proxy server between the Tor Browser and Tor to capture requests/responses, using the seleniumwire library. You can access and modify the HTTP headers that are being sent/received, including the onion services. Note: You need to have Tor installed and running on the localhost while running this script.

tor-browser-selenium-wire.py

import os
from seleniumwire import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary

# Uncomment these if you need additional information for debugging
#import logging
#logging.basicConfig(level=logging.DEBUG)

# The location of the Tor Browser bundle
# Update this to match the location on your computer
tbb_dir = "/home/woswos/tor-browser_en-US/"

# Disable Tor Launcher to prevent it connecting the Tor Browser to Tor directly
os.environ['TOR_SKIP_LAUNCH'] = '1'
os.environ['TOR_TRANSPROXY'] = '1'

# Set the Tor Browser binary and profile
tb_binary = os.path.join(tbb_dir, 'Browser/firefox')
tb_profile = os.path.join(tbb_dir, 'Browser/TorBrowser/Data/Browser/profile.default')
binary = FirefoxBinary(os.path.join(tbb_dir, 'Browser/firefox'))
profile = FirefoxProfile(tb_profile)

# We need to disable HTTP Strict Transport Security (HSTS) in order to have
#   seleniumwire between the browser and Tor. Otherwise, we will not be able
#   to capture the requests and responses using seleniumwire.
profile.set_preference("security.cert_pinning.enforcement_level", 0)
profile.set_preference("network.stricttransportsecurity.preloadlist", False)

# Tell Tor Button it is OK to use seleniumwire
profile.set_preference("extensions.torbutton.local_tor_check", False)
profile.set_preference("extensions.torbutton.use_nontor_proxy", True)

# Required if you need JavaScript at all, otherwise JS stays disabled regardless
#   of the Tor Browser's security slider value
profile.set_preference("browser.startup.homepage_override.mstone", "68.8.0");

# Configure seleniumwire to upstream traffic to Tor running on port 9050
#   You might want to increase/decrease the timeout if you are trying
#   to a load page that requires a lot of requests. It is in seconds.
options = {
    'proxy': {
        'http': 'socks5h://127.0.0.1:9050',
        'https': 'socks5h://127.0.0.1:9050',
        'connection_timeout': 10
    }
}

driver = webdriver.Firefox(firefox_profile=profile,
                           firefox_binary=binary,
                           seleniumwire_options=options)

driver.get("https://check.torproject.org/")

for request in driver.requests:
    if request.response:
        print(
            request.path,
            request.response.status_code,
            request.response.headers['Content-Type']
        )

diver.quit()

If you want to use this setup without Tor for testing purposes, you need to remove Tor from the seleniumwire's upstream proxy. When there is no upstream proxy, seleniumwire will route the traffic through the regular internet.

You can achieve this by removing the seleniumwire_options from the webdriver constructor:

driver = webdriver.Firefox(firefox_profile=profile,
                           firefox_binary=binary)

Hi,

first of all, thanks for posting this snippet, it's really helpful.
For some reason, if I use this and request "https://check.torproject.org/", it works and confirms tor usage but NOT with any .onion url.
They fail with an error 502 - Bad gateway. My guesses so far...

• seleniumwire has a problem with the socks5h upstream proxy?
• some system time misconfiguration?
• incompatible tb update?

I had no luck messing aroung with those guesses.

Any idea what the culprit could be?

@woswos

// Edit: By the way, I made sure it is not a server side error (which 500s should be actually). Using my alternative setup (tor-browser-selenium) without seleniumwire works without any issues, same settings as far as I can tell.

You saved my life,thanks really much, at first it didnt work but i added this preferences and it worked like a charm:

profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks_host', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.update_preferences()

Is there a similar way to get this to work with tbselenium driver, especially with their STEM module? I can use the code I had before, without using Firefox objects, but I keep getting
selenium.common.exceptions.,WebDriverException: Message: Reached error page: about:neterror?e=dnsNotFound&u=...