ServiceNow/Selenium – Shadow DOMs

Introduction

If you’re interested in a better solution to this problem, take a look at my follow up blog on this specific subject! There we utilize the shadow_root attribute as it was designed to be used to achieve the same goal.

When you’re developing software to last, building tests along the way is integral in maintaining a robust inflexible code-base. ServiceNow’s ATF is an efficient, cost-effective way of doing regression testing in the ServiceNow platform, but it missed the mark on navigating Custom UI pages as well as third-party integration testing. That’s where using Selenium come’s in. Selenium is an automated testing tool that multiple languages and multiple browsers, allowing developers to create tests by navigating the page and integrate those tests using frameworks such as TestNG and JUnit.

We set out taking on the simple task of logging in, navigating from the home page to the Incidents table and logging out using just Selenium in order to tackle one of the biggest hurdles of Selenium in ServiceNow, the Shadow DOM.

Understanding Shadow DOM

Shadow DOM is a web development technique that provides a way to encapsulate or “hide” elements of a web page’s Document Object Model (DOM), offering a level of separation between an element’s functionality and style from the rest of the page. This encapsulation makes it possible to keep an element’s features private, allowing developers to style and script elements without interfering with other elements on the page. However, it introduces challenges of compatibility with Selenium’s testing library. The elements that are “hidden” within the Shadow DOM are difficult to identify with traditional Selenium methods.

Difficulties of Selenium in the Shadow DOM

Using Selenium, you can use use the find_element() function to grab practically any element. It gives you the option of choosing unique identifiers of said elements such as CSS, Name, ID, ClassName, XPATH, etc. This is always worth trying first, but with elements that are within the ShadowDOM it often falls short in being able to identify and acquire the obfuscated element.

Below is the traditional method to grab an element using Selenium using find_element:

all_button = snTest.driver.find_element(By.XPATH'//*[@id="d6e462a5c3533010cbd77096e940dd8c"]')

Plus, Selenium has provided the shadow_root() function to give user’s the identify the shadow DOM

This method, however, provides little help as it doesn’t give the developer any real way to interact with the Shadow DOM, and, in the case of ServiceNow, there are multiple shadow roots nested on top of each other. It becomes very complex, trying to identify and interact with each shadow root to get to the element you’re searching for.

Using JavaScript with Selenium

In order to interact with elements inside the shadow DOM, we had to explore the only thing that could touch them: JavaScript. Since Shadow DOMs are a browser standard, they can be interacted with using JavaScript directly. After some experimentation and learning, we found that we could execute JavaScript code directly from Selenium.

The solution was to utilize Selenium’s execute_script() function to invoke JavaScript code that could access and interact with elements inside the Shadow DOM. We crafted custom JavaScript paths to locate and interact with shadow elements, effectively ‘bypassing’ the encapsulation to access the elements we needed for our tests. To acquire the JavaScript path of an element, it’s as simple as using DevTools and copying the JS path directly.

Using this method provides the developer with JS script using querySelector() functions to traverse the shadowRoots until they’ve reached the desired element. From this point, you can call whatever JS function you’d like, including a simple .click() and it will be returned in the execute_script() function.

Overcoming the time.sleep() Dilemma: Reintroducing Selenium Explicit Waits

Initially, time.sleep() was used as a quick workaround to deal with latency issues. It was an easy solution for letting the page load or waiting for elements to become visible. However, it soon became clear that this was not a scalable or reliable solution. time.sleep() introduces arbitrary wait times which can make the tests slower than necessary and may lead to false negatives if the wait time isn’t long enough.

To overcome this, we implemented an explicit wait strategy using the is_element_present() function. This method checks for the presence of an element each time it is called. When coupled with Seleniums explicit wait function WebDriverWait() we can still verify that the element exists by continually polling the is_element_present() function. This strategy ensures we wait only as long as necessary for an element to appear, increasing test efficiency and reliability.

Selenium/JavaScript Solution

The click_shadow_element function in the ServiceNowSelenium class is an example of how we implemented this solution. It uses JavaScript to wait for and click a shadow DOM element based on the JavaScript path provided.

ServiceNowSelenium Class:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoAlertPresentException, UnexpectedAlertPresentException, TimeoutException
from selenium.common.exceptions import ElementNotVisibleException, ElementNotInteractableException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

class ServiceNowSelenium:

# constructor
# contains driver, url and login credentials
def __init__(self, url, username, password):
    self.driver = webdriver.Chrome()
    self.url = url
    self.username = username
    self.password = password

# Login function, initializes driver and logs in based on values provided
def login(self):
    # Initialize 
    self.driver.maximize_window() # Full screen window
    self.driver.get(self.url) # Navigate to url given by constructor
    self.accept_alert() # Catch any alerts

    # Login using username/password
    username_field = self.driver.find_element(By.NAME, 'user_name')
    username_field.send_keys(self.username)

    password_field = self.driver.find_element(By.NAME, 'user_password')
    password_field.send_keys(self.password)

    login_button = self.driver.find_element(By.ID, 'sysverb_login')
    login_button.click()

# Logout and quit driver function
def logout(self):
    user_menu_path = '''[copy/paste js path]'''
    self.click_shadow_element("user menu button", user_menu_path)

    logout_path = '''[copy/paste js path]'''
    self.click_shadow_element("logout button", logout_path)

    self.driver.quit()

# Check if an element is present using JS path.
# Particularly useful for shadow DOM elements
def is_element_present(self, js_path):
    try:
        element = self.driver.execute_script(f"return {js_path}")
        return element is not None
    except Exception:
        pass
    try:
        outer_html = self.driver.execute_script(f"return {js_path}.outerHTML;")
        if outer_html:
            return True;<br />    except Exception:
        return False

# used to click shadow elements w/ js path.
# uses javascript to allow for explicit waits 
def click_shadow_element(self, errorName, js_path, wait_time = 10):
    try:
        WebDriverWait(self.driver, wait_time).until(lambda x: self.is_element_present(js_path))
        self.driver.execute_script(f"return {js_path}.click()")
    except (TimeoutException, ElementNotVisibleException, ElementNotInteractableException) as e:
        raise Exception(f"The {errorName} button was not found. Additional info: {str(e)}")

# Wait and accept alerts as they come
def accept_alert(self):
    try:
        WebDriverWait(self.driver, .5).until(EC.alert_is_present())
        alert = self.driver.switch_to.alert
        alert.accept()
        print("Alert accepted")
    except (NoAlertPresentException, TimeoutException, UnexpectedAlertPresentException):
        pass