Introduction

When Selenium tests fail unexpectedly, screenshots provide valuable visual information about the application’s state. However, there are situations where the problem exists within the HTML structure itself rather than the user interface. Missing elements, dynamically generated content, incorrect attributes, and synchronization issues often require developers to inspect the webpage’s underlying HTML.

This is where capturing the page source becomes extremely useful. Selenium’s page_source property allows automation engineers to save the complete HTML document at any point during test execution.

Capturing the page source helps developers to:

Verify whether elements actually exist in the DOM.
Investigate locator failures.
Analyze dynamically generated HTML.
Debug synchronization issues.
Preserve failure information for later investigation.
Improve troubleshooting capabilities in automation frameworks.

In this tutorial, you will learn how Selenium captures page source information, understand when it should be used, explore practical examples, common mistakes, best practices, and frequently asked interview questions.

What is Capturing Page Source?

Capturing page source refers to saving the complete HTML document currently loaded inside the browser during test execution.

Instead of:

Test Fails
     │
     ▼
Element Not Found
     │
     ▼
No HTML Information Available
     │
     ▼
Investigate Manually

we can capture the page source:

Test Executes
      │
      ▼
Capture HTML Source
      │
      ▼
Save HTML File
      │
      ▼
Inspect the DOM
      │
      ▼
Identify the Problem Easily

Page source files significantly simplify troubleshooting efforts in Selenium automation frameworks.

Why Should We Capture Page Source?

Capturing page source allows developers to:

Inspect the DOM structure.
Verify whether elements exist.
Debug locator failures.
Investigate dynamic webpage behavior.
Preserve failure information.
Improve reporting mechanisms.

Large automation frameworks frequently capture page source files automatically whenever test failures occur.

Practical Example

The following example demonstrates how Selenium captures and stores the current HTML source of a webpage.

from pathlib import Path

from selenium import webdriver


# Topic: Capturing Page Source
# Practice site:
# https://www.testmuai.com/selenium-playground/
# Run:
# pytest -s 64_examples/test_03_capturing_page_source.py
#
# Saving driver.page_source preserves the complete HTML document during test
# execution and greatly simplifies troubleshooting locator-related failures.


def test_capturing_page_source(
    tmp_path
):

    driver = webdriver.Chrome()

    try:
        driver.get(
            "https://www.testmuai.com/"
            "selenium-playground/"
        )

        page_source_file = (
            tmp_path /
            "page_source.html"
        )

        page_source_file.write_text(
            driver.page_source,
            encoding="utf-8"
        )

        content = (
            page_source_file.read_text(
                encoding="utf-8"
            )
        )

        assert (
            "<html"
            in content.lower()
        )

        assert (
            page_source_file.exists()
        )

    finally:
        driver.quit()

Output

Chrome browser launched successfully.

Website opened successfully.

HTML source captured successfully.

HTML file saved successfully.


page_source.html


Assertions Passed.

Test Executed Successfully.

Note: The contents of the generated HTML file will vary depending on the webpage displayed during execution.

Understanding the Code

Import Required Modules

from pathlib import Path

from selenium import webdriver

Imports:

Selenium WebDriver.
Path handling utilities used for managing files and directories.

Launch Chrome Browser

driver = webdriver.Chrome()

Creates a new Chrome browser session.

Open the Website

driver.get(
    "https://www.testmuai.com/"
    "selenium-playground/"
)

Opens the Selenium Playground webpage.

Create the HTML File Path

page_source_file = (
    tmp_path /
    "page_source.html"
)

This creates the following file path:

Temporary Directory
        │
        ▼
page_source.html

which will store the captured HTML source.

Capture the Page Source

driver.page_source

returns the complete HTML document currently loaded inside the browser.

For example:

<html>
    ...
    ...
    ...
</html>

This includes:

HTML elements.
Attributes.
Generated DOM content.
Dynamically rendered markup.

Save the HTML File

page_source_file.write_text(
    driver.page_source,
    encoding="utf-8"
)

Selenium performs the following steps:

Browser Window
       │
       ▼
Retrieve HTML Source
       │
       ▼
Generate HTML File
       │
       ▼
Save the Page Source
       │
       ▼
Store the File Successfully

Saving the HTML source preserves the application’s state during execution.

Read the Saved File

content = (
    page_source_file.read_text(
        encoding="utf-8"
    )
)

This retrieves the contents of:

page_source.html

for validation purposes.

Verify the HTML Source

assert (
    "<html"
    in content.lower()
)

This assertion verifies that:

The HTML document was captured successfully.
The saved file contains valid webpage content.

Verify the File Exists

assert (
    page_source_file.exists()
)

This confirms that:

The HTML file was generated successfully.
Selenium saved the page source correctly.

Close the Browser

driver.quit()

Closes all browser windows and properly ends the WebDriver session.

Running the Example

Execute the following command:

py -3 -m pytest -s ^
"64_examples/test_03_capturing_page_source.py"

Run all troubleshooting examples together:

py -3 -m pytest -s ^
"61_examples/"

Note: The -s option displays console output generated during test execution.

Execution Flow

Launch Browser
       │
       ▼
Open Website
       │
       ▼
Retrieve Page Source
       │
       ▼
Generate HTML File
       │
       ▼
Save HTML Content
       │
       ▼
Verify the File Exists
       │
       ▼
Assertions Passed
       │
       ▼
Close Browser

Relationship with Troubleshooting Test Failures

This section also includes topics such as:

Python Logging.
Capturing Screenshots.
Browser Console Logs.
Debugging Failed Test Cases.

The shared conftest.py file automatically captures the page source whenever a test fails.

Test Failure
      │
      ▼
PyTest Detects Failure
      │
      ▼
Capture Screenshot
      │
      ▼
Capture HTML Source
      │
      ▼
Generate HTML File
      │
      ▼
Store Failure Artifacts

For example:

screenshots
      │
      ├── failed_test.png
      │
      └── failed_test.html

Note: Automatic page source capture using conftest.py is covered separately under Debugging Failed Test Cases. The current example focuses only on manually capturing HTML using driver.page_source.

Automation Testing Example

Suppose Selenium cannot locate an element.

Element Not Found
       │
       ▼
Capture HTML Source
       │
       ▼
Open page_source.html
       │
       ▼
Search for the Element
       │
       ▼
Identify the Problem

Common findings include:

Incorrect element IDs.
Missing elements.
Dynamic HTML rendering.
Synchronization issues.
Unexpected webpage states.

Capturing page source significantly simplifies such investigations.

Real-World Example

Large automation frameworks commonly capture page source files for:

Failed assertions.
Locator failures.
Regression failures.
CI/CD executions.
Synchronization problems.
Reporting mechanisms.

For example:

Automation Failure
        │
        ▼
Capture HTML Source
        │
        ▼
Store Failure Information
        │
        ▼
Generate Reports
        │
        ▼
Simplify Troubleshooting

Page source files provide valuable information that may not always be visible in screenshots.

Common Mistakes Beginners Make

Capturing Page Source Too Late

Avoid attempting to capture:

driver.page_source

after:

driver.quit()

The browser session must remain active when retrieving the HTML source.

Ignoring Dynamic Content

Some developers assume that:

driver.page_source

always matches the original webpage source.

However, Selenium returns the current DOM state after JavaScript execution and page updates, which makes page source capture particularly useful for debugging dynamic applications.

Ignoring Saved HTML Files

Page source files frequently reveal:

Incorrect locators.
Missing elements.
DOM changes.
Application failures.
Synchronization problems.

Always inspect saved HTML files carefully during troubleshooting.

Best Practices

Capture page source whenever locator-related failures occur.
Preserve HTML files during CI/CD executions whenever possible.
Combine page source capture with screenshots and logging mechanisms.
Maintain meaningful file names.
Capture HTML before closing the browser.
Review saved HTML files carefully while debugging automation failures.
Integrate page source capture with reporting tools whenever appropriate.

Conclusion

Capturing page source is one of the most valuable troubleshooting techniques available in Selenium automation testing. By preserving the complete HTML document during test execution, automation engineers can investigate locator failures, synchronization issues, and unexpected application behavior efficiently.

When combined with screenshots, logging mechanisms, and reporting tools, page source capture significantly improves debugging capabilities and framework maintainability in real-world automation projects.

Mastering page source capture techniques is an essential Selenium automation and interview skill.

Frequently Asked Questions (FAQs)

What is `driver.page_source`?

driver.page_source returns the complete HTML document currently loaded inside the browser.

Can page source be captured automatically?

Yes.

Large automation frameworks frequently capture page source files automatically whenever tests fail using PyTest fixtures and hooks.

Is page source useful for debugging locator failures?

Yes.

It is one of the most valuable tools for investigating missing elements, DOM changes, and synchronization issues.

Does `driver.page_source` include dynamically generated HTML?

Yes.

Selenium returns the current DOM state, including content generated after JavaScript execution whenever applicable.

Should page source capture be combined with screenshots?

Yes.

Combining:

Page source files.
Screenshots.
Logging mechanisms.
Reporting tools.

provides significantly better troubleshooting capabilities.

Key Takeaways

driver.page_source preserves the complete HTML document during Selenium test execution.
Page source files significantly simplify troubleshooting locator-related failures.
Capturing HTML files is particularly useful when debugging dynamic webpages.
Page source capture complements screenshots and logging mechanisms effectively.
Automatic page source capture is commonly implemented in large automation frameworks.
Proper page source management substantially improves debugging capabilities.
Understanding the current DOM state greatly simplifies failure analysis.
Capturing Page Source is an important Selenium automation and interview topic.

Introduction

What is Capturing Page Source?

Why Should We Capture Page Source?

Practical Example

Output

Understanding the Code

Import Required Modules

Launch Chrome Browser

Open the Website

Create the HTML File Path

Capture the Page Source

Save the HTML File

Read the Saved File

Verify the HTML Source

Verify the File Exists

Close the Browser

Running the Example

Execution Flow

Relationship with Troubleshooting Test Failures

Automation Testing Example

Real-World Example

Common Mistakes Beginners Make

Capturing Page Source Too Late

Ignoring Dynamic Content

Ignoring Saved HTML Files

Best Practices

Conclusion

Frequently Asked Questions (FAQs)

What is driver.page_source?

Can page source be captured automatically?

Is page source useful for debugging locator failures?

Does driver.page_source include dynamically generated HTML?

Should page source capture be combined with screenshots?

Key Takeaways

What is `driver.page_source`?

Does `driver.page_source` include dynamically generated HTML?