0

This image represents the web page from which I am trying to extract the hrefs, hiddens under the details button. On the right hand side you can see the inspection of the web page

I am using Selenium library in Python but despite several attempts I can't get the href. Code used:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import ElementClickInterceptedException
import time

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument('--disable-gpu')

driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
driver.get("https://webgate.ec.europa.eu/rasff-window/screen/search")

cookies = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.XPATH, "//a[text()='Accept all cookies']"))).click()
search = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.XPATH,'//*[@id="search"]/div[3]/div[4]/button'))).click()
next_page = driver.find_element(By.XPATH,'//*[@id="main-content"]/app-search-component/div/app-list-component/div/div[3]/mat-paginator/div/div/div[2]/button[2]')

counter = 0
 #15075
while counter <= 50:

    details = WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[contains(text(),'Details')]")))
    for detail in details:
        print(detail.get_attribute("href"))

    counter += 25
    next_page.click()
    time.sleep(1)

print(details)

driver.close()

I get the following error: cookies = WebDriverWait(driver,30).until(EC.element_to_be_clickable((By.XPATH, "//a[text()='Accept all cookies']"))).click()

selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element ... is not clickable at point (422, 528). Other element would receive the click: ...

Improve this question
1

1 Answer 1

Reset to default
0

Refer the code below:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.maximize_window()
driver.get('https://webgate.ec.europa.eu/rasff-window/screen/search')
wait = WebDriverWait(driver, 30)

# accept cookies
wait.until(EC.element_to_be_clickable((By.XPATH, "//a[text()='Accept all cookies']"))).click()
# Click on Search button
WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.XPATH, "//button[contains(text(),'Search')]"))).click()
# Store all 'Details' web elements
elements = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//a[contains(text(),'Details')]")))

for element in elements:
    print(element.get_attribute("href"))

Console output:

https://webgate.ec.europa.eu/rasff-window/screen/notification/620237
https://webgate.ec.europa.eu/rasff-window/screen/notification/620545
https://webgate.ec.europa.eu/rasff-window/screen/notification/620512
https://webgate.ec.europa.eu/rasff-window/screen/notification/620588
https://webgate.ec.europa.eu/rasff-window/screen/notification/620544
https://webgate.ec.europa.eu/rasff-window/screen/notification/620422
https://webgate.ec.europa.eu/rasff-window/screen/notification/620420
https://webgate.ec.europa.eu/rasff-window/screen/notification/620548
https://webgate.ec.europa.eu/rasff-window/screen/notification/620506
https://webgate.ec.europa.eu/rasff-window/screen/notification/620577
https://webgate.ec.europa.eu/rasff-window/screen/notification/620568
https://webgate.ec.europa.eu/rasff-window/screen/notification/620379
https://webgate.ec.europa.eu/rasff-window/screen/notification/620397
https://webgate.ec.europa.eu/rasff-window/screen/notification/620523
https://webgate.ec.europa.eu/rasff-window/screen/notification/620383
https://webgate.ec.europa.eu/rasff-window/screen/notification/620508
https://webgate.ec.europa.eu/rasff-window/screen/notification/618752
https://webgate.ec.europa.eu/rasff-window/screen/notification/620374
https://webgate.ec.europa.eu/rasff-window/screen/notification/620248
https://webgate.ec.europa.eu/rasff-window/screen/notification/620258
https://webgate.ec.europa.eu/rasff-window/screen/notification/620201
https://webgate.ec.europa.eu/rasff-window/screen/notification/620312
https://webgate.ec.europa.eu/rasff-window/screen/notification/620179
https://webgate.ec.europa.eu/rasff-window/screen/notification/620064
https://webgate.ec.europa.eu/rasff-window/screen/notification/620187

Process finished with exit code 0

UPDATE:

Root cause of the error: In headless mode, Accept All cookies button was overlapped by some other element. To fix this, you need to maximise the window using following code: chrome_options.add_argument("window-size=1920,1080")

There were more than one issues in your code. Check below the corrected code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import ElementClickInterceptedException
import time

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument("window-size=1920,1080")
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument('--disable-gpu')

driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
driver.get("https://webgate.ec.europa.eu/rasff-window/screen/search")

cookies = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.XPATH, "//a[text()='Accept all cookies']"))).click()
search = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.XPATH,'//*[@id="search"]/div[3]/div[4]/button'))).click()
next_page = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.XPATH,"//button[@aria-label='Next page']//span")))

counter = 0
 #15075
while counter <= 50:

    details = WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[contains(text(),'Details')]")))
    for detail in details:
        print(detail.get_attribute("href"))

    counter += 25
    # next_page.click()
    driver.execute_script("arguments[0].click();", next_page)
    time.sleep(10)

print(details)

driver.close()
Improve this answer
5
  • Edit the question and share your code in the question.
    – Shawn
    Commented Jul 4, 2023 at 8:30
  • Check the updated answer, use the code from the answer. You will notice that I have made few changes to your code.
    – Shawn
    Commented Jul 4, 2023 at 9:15
  • The code you provided worked the first time then I got the same error reported in the question with the following small difference: not clickable at point (1183, 1032)
    – Davide
    Commented Jul 4, 2023 at 10:07
  • Asking multiple follow-up questions on a same question is a bad idea. If the original issue which you reported has resolved then accept this answer. And then ask a new question with the specific error and your code trials, this way you will get answers from a wider community.
    – Shawn
    Commented Jul 4, 2023 at 10:10
  • I got the same error reported in the question with the following small difference: not clickable at point (1183, 1032 - Clearly its not same error. Its giving similar error on some other line of your code.
    – Shawn
    Commented Jul 4, 2023 at 10:22

Not the answer you're looking for? Browse other questions tagged or ask your own question.