AU.← Engineering answers

Python automation · Practical answer

How do you make Selenium automation recover from network interruptions?

Treat browser automation as a recoverable workflow, not one fragile sequence of clicks.

By Anhaj Uwaisulkarni7 min read
Direct answer

Use explicit waits for observable page states, split the workflow into idempotent business steps, save a checkpoint after each completed step and retry only transient failures with bounded exponential backoff. If the WebDriver session is unhealthy, quit it, create a clean browser and resume from the last verified checkpoint.

1. Wait for states, not seconds

A fixed sleep guesses how long the page needs. An explicit wait observes what the automation actually needs: a visible element, a clickable control, a changed URL or a completed request reflected in the UI.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 15)
submit = wait.until(EC.element_to_be_clickable((By.ID, "submit")))
submit.click()
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".success")))

Selenium’s documentation warns against mixing implicit and explicit waits because the combined timing can become unpredictable. Pick a clear waiting strategy and use it consistently.

2. Retry the business step

Do not retry every low-level click independently. Wrap a complete, verifiable step such as “open account page” or “submit record.” After an exception, check whether the intended result already happened before repeating an action that could create a duplicate.

The helper below is for read-only or genuinely idempotent operations. Supply the specific exception types that your application has classified as temporary; it deliberately does not catch every exception. It is a reusable control-flow example, not a complete Selenium application.

import time


def run_with_recovery(step, verify, *, transient_errors,
                      attempts=3, sleep=time.sleep):
    if attempts < 1:
        raise ValueError("attempts must be at least 1")

    for attempt in range(attempts):
        try:
            if verify():
                return
            step()
            if verify():
                return
        except transient_errors:
            if attempt == attempts - 1:
                raise

        if attempt < attempts - 1:
            sleep(min(2 ** attempt, 8))

    raise RuntimeError("Step did not reach its verified state")

A timeout does not prove that a write failed. If a form submission times out after the server saved it, retrying can create a duplicate. For payments, record creation or other non-idempotent writes, reconcile using a stable record ID or a server-supported idempotency key. If you cannot determine the result, stop for review instead of automatically submitting again.

In a Selenium integration, a TimeoutException may be temporary, but it can also mean the selector is wrong. Classify it using the page state and logs. Re-find stale elements inside each attempt; retrying the same stale element reference will not repair it.

3. Save checkpoints outside the browser

Cookies and page state are not enough. Store a small checkpoint with the record ID, last completed phase and timestamp in SQLite, JSON or a remote database. On restart, query the target system when possible and resume from the last confirmed state.

Design repeated steps to be idempotent: running them twice should either be safe or detect that the work is already complete. This matters more than the retry library you choose.

4. Separate transient and permanent failures

  • Retry: page-load timeouts, temporary DNS failures, stale elements after a re-render and brief service unavailability.
  • Stop and report: invalid credentials, permission failures, changed selectors, validation errors and unexpected business rules.

For each final failure, record a sanitized URL, current checkpoint and exception trace. Capture screenshots or page source only when needed, redact credentials and personal data, restrict access and set a retention limit. Those artifacts turn “it stopped” into a diagnosable incident without creating an unnecessary store of sensitive information.

5. Replace a broken session cleanly

First test whether the driver can read a simple property such as the current URL. If the browser has crashed or the session is invalid, call quit() in a protected cleanup block, create a new driver and navigate back to a known entry point. Reuse a persistent browser profile only when the target system permits it and the profile can be protected.

Never automate around access controls, multi-factor authentication or a site’s usage rules. Reliable automation should reduce repetitive work without weakening security boundaries.

Primary references