Skip to main content

Problem with list extraction on Webflow websites

Feature Request

Seems sites built on Webflow don’t provide a page URL for listing pages (cards) when running a List Extraction. They seem to be using a CSS class, but I can’t identify the right class to add to a List Extraction manually, resulting in no URLs collected. As a result, Page Extractions fail.

Here’s an example of a site causing problems: hotel list

Posted by MrKhaki ·
1 comment

Log in to comment and vote

Comments1

  • MrKhaki

    •

    Sep 25

    I spent some time trying to troubleshoot this issue. The problem on Webflow websites is that links are generated with JavaScript. I threw the issue at AI and it came up with:

    See my thinking

    Overview of the problem

    Context: I’m trying to scrape hotel/property listings that are implemented as card components on a site using Webflow-style front‑end patterns and a RootRez‑like booking engine. These cards:

    • Are rendered client‑side via JavaScript (React/Vue/Webflow interactions).

    • Do not expose standard <a href="..."> links for detail pages.

    • Often store the key data (IDs, URLs, prices, etc.) in:

      • data-* attributes on card elements, or

      • a JSON state object (e.g. window.__INITIAL_STATE__) inside <script> tags.

    Ultimate Web Scraper currently only supports static CSS selectors on the initial HTML, without:

    • Executing JavaScript to get the rendered DOM.

    • Extracting attributes (like data-id) separately from the element.

    • Running custom JavaScript to transform card data into a list of URLs or rows.

    Because of this, any selector I add in UWS returns no data for these card‑based lists—the HTML delivered to the browser doesn’t contain the cards or their data yet.

    Why this breaks scraping for card-based Webflow sites

    On these sites, the workflow is:

    1. Server sends a minimal HTML shell:

      html

      <div id="app"></div> <script>window.__INITIAL_STATE__ = { properties: [...] }</script>

    2. JavaScript framework reads __INITIAL_STATE__ and renders:

      html

      <div class="property-card" data-id="1234"> <div class="property-name">Hotel Name</div> <img src="..."> </div>

    3. Detail URLs are implicit, e.g.:

      js

      const url = `/property/${property.id}`;

    Ultimate Web Scraper:

    • Sees only the pre‑render HTML (no .property-card elements).

    • Cannot access window.__INITIAL_STATE__ or run JS to build the cards.

    • Cannot construct URLs from IDs because it doesn’t support JS transforms.

    Result: Scraping returns empty results, even though the browser visibly shows a full list of cards.

    Requested feature additions for Ultimate Web Scraper

    To successfully scrape lists from card‑based Webflow/JS sites, I’d like Ultimate Web Scraper to support:

    1. JavaScript execution / rendered DOM mode

      • Ability to load a page, execute its JavaScript, and then run CSS selectors against the post‑render DOM.

      • This would allow selectors like:

        • div.property-card

        • div.property-card .property-name

        • div.property-card img

    2. Attribute extraction for selected elements

      • For each matched element, allow extracting attributes such as:

        • data-id

        • data-url

        • data-price

      • Example:

        • Selector: div.property-card

        • Extract attribute: data-id

        • Save as: property_id

    3. Custom JavaScript transforms per row

      • After selecting elements and attributes, allow running a small JS snippet to construct fields.

      • Example:

        js

        // Given property_id const detail_url = `https://lodging.visittruckeetahoe.com/property/${property_id}`;

      • This would let me turn card IDs into full detail URLs, or combine multiple attributes into one field.

    4. Access to page-level JSON state

      • Ability to read and parse JSON embedded in <script> tags (e.g. window.__INITIAL_STATE__) and treat arrays like properties[] as rows.

      • This is often the source of truth for card data on JS/Webflow sites.

    Example use case (what I’m trying to do)

    On a card-based lodging page:

    • Goal: Generate a CSV with:

      • hotel_name

      • property_id

      • detail_url

      • image_url

    • Needed capabilities:

      • Execute JS to render .property-card elements.

      • Select .property-card and extract data-id.

      • Select .property-card .property-name (text) and img[src].

      • Run JS to build:

        js

        detail_url = `https://lodging.example.com/property/${property_id}`;

    Without these features, Ultimate Web Scraper cannot scrape modern Webflow/JS card lists, even though the data is clearly visible in the browser.

    If Ultimate Web Scraper adds rendered DOM support, attribute extraction, and per‑row JS transforms, it would unlock scraping for a large class of modern, card‑based Webflow and JavaScript sites like the one I’m working with.