Mon, Jul 27, 26

search engine optimization Shopify

Product Data Hooks for Crawling Shopify’s Product Pages

Every time we audit an ecommerce website, we need to collect a lot of product-level data.

Product pages are one of the most important parts of an ecommerce site. They are where users evaluate products, compare options, view images, check availability, read descriptions, and ultimately decide whether to add something to the cart or complete a purchase.

From an SEO and merchandising perspective, product pages also contain a lot of useful audit data. For example, we may want to know:

Product name
Product availability
Product type
Product vendor or brand
Product description
Number of product images
Product collections
Available color options
SKU
Barcode
Variant information
Whether key product data is missing or inconsistent

This information helps with many ecommerce SEO checks. For example, it can identify products with thin descriptions, only one image, weak or generic collections, missing SKUs or brand/vendor data, or unavailable products still receiving organic traffic.

The challenge is that collecting this data across hundreds or thousands of product pages can be time-consuming.

In the past, I used XPath expressions with Screaming Frog’s Custom Extraction feature. That works well but can be slow to set up because every Shopify theme is slightly different.

One store may output the product title in an H1.
Another may wrap it in a custom component.
Another may load variant data differently.
Another may have product information hidden inside JavaScript.
Another may use custom metafields, apps, or theme-specific markup.

Even though the underlying Shopify product data may be similar, the front-end HTML can vary a lot from site to site. This means XPath or CSSPath extraction rules often need rebuilding for each new audit.

At some point, to save time or because I was tired of rewriting extraction rules from scratch, I decided there had to be a quicker way.

Shopify Liquid to the rescue.

Shopify themes are built with Liquid, and Shopify’s product object gives access to useful product data directly inside the product template. This means we can output the data we need in a consistent format, regardless of how the visible theme markup is structured.

The idea is simple:

Add a hidden block of audit-only HTML to the product template.
Output useful Shopify product fields inside that block.
Add predictable data-hook attributes to each field.
Crawl the product URLs with Screaming Frog.
Use Custom Extraction to extract the values by their data-hook attributes.
Remove the hidden block once the audit is finished.

This creates a consistent extraction layer that is easier to crawl than the visible product page layout.

Instead of writing a different XPath for each theme’s product title, image gallery, vendor field, or variant picker, you create predictable markup once and crawl that.

Here is the basic idea:

Each useful piece of product data is placed inside an HTML element.
Each element gets a data-hook attribute.
Screaming Frog can then extract the value of each data-hook.

For example:

data-hook="product_name"
data-hook="product_available"
data-hook="product_type"
data-hook="product_vendor"
data-hook="product_images_num"
data-hook="first_variant_sku"

This makes crawling much more predictable.

The excludeColls variable in the code below contains a list of collections I do not really want to extract for each product.

For example, collections like Sale, New Arrivals, and Back In Stock can be useful on the website but do not always show what the product actually is. For audits, I usually focus more on collections that describe the product category, group, use case, or merchandising structure.

You can edit this list depending on the client site.

For example, you may want to exclude collections such as:

Sale
New Arrivals
Back In Stock
Best Sellers
Featured
Homepage
Staff Picks
Clearance
Final Sale

The exact list will depend on the store.

Here is the Liquid snippet:

<li data-hook="product_name">{{ product.title | escape }}</li>

<li data-hook="product_available">{{ product.available }}</li>

<li data-hook="product_type">{{ product.type | escape }}</li>

<li data-hook="product_vendor">{{ product.vendor | escape }}</li>

<li data-hook="product_description">{{ product.description | strip_html | strip | escape }}</li>

<li data-hook="product_images_num">{{ product.images.size }}</li>

<li>

{% assign excludedCols = "Sale,New Arrivals,Back In Stock" | split: "," %}

{% for collection in product.collections %}

{% unless excludedCols contains collection.title %}

<li>{{ collection.title | escape }}</li>

{% endunless %}

{% endfor %}

</ul>

</li>

<li>

{% if product.options_by_name['Color'] %}

{% for color in product.options_by_name['Color'].values %}

<li>{{ color | escape }}</li>

{% endfor %}

{% elsif product.options_by_name['color'] %}

{% for color in product.options_by_name['color'].values %}

<li>{{ color | escape }}</li>

{% endfor %}

{% endif %}

</ul>

</li>

{{ product.selected_or_first_available_variant.barcode | escape }}

</li>

</li>

</ul>

The code is intentionally simple.

It creates a hidden unordered list and outputs product data into list items. The markup is hidden with display: none, so visitors should not see it. However, the data remains in the HTML, allowing a crawler like Screaming Frog to extract it.

There are a few important notes.

First, use this as a temporary audit helper, not permanent production code. Add it, crawl the site, collect the data, and remove it when done.

Second, place it only where the Shopify product object is available. In most Shopify themes, that means adding it to the main product template or product section.

Depending on the theme, this could be a file such as:

main-product.liquid
product-template.liquid
product.liquid

A product section used by the product JSON template

Shopify themes vary, so the exact file name may differ. If the theme is an Online Store 2.0 theme, the product page may be controlled by a JSON template referencing a Liquid section. In that case, the code usually belongs in the relevant product section, not the JSON template.

Third, do not add this directly to a live theme without thinking it through.

A safer workflow is:

Duplicate the theme.
Add the snippet to the duplicate theme.
Preview the duplicate theme.
Crawl the preview URLs if possible.
If needed, publish the change briefly during a controlled audit window.
Remove the snippet once the crawl is complete.

Sometimes you may need to place the code on the live theme temporarily so Screaming Frog can crawl the public URLs users and search engines see. If you do, keep the code hidden, test carefully, and remove it as soon as the audit finishes.

Fourth, remember that display: none hides the data visually, but it does not make the data private.

Do not output sensitive information, internal-only notes, cost data, supplier data, margin data, or anything else that should not be visible in the page source.

The fields in the example above are generally product-facing fields or operational identifiers commonly associated with products, but you should still review them before using the snippet on a client site.

Once the snippet is added, you can crawl the product pages with Screaming Frog and configure Custom Extraction.

For example, you can use CSSPath selectors such as:

[data-hook="product_name"]

[data-hook="product_available"]

[data-hook="product_type"]

[data-hook="product_vendor"]

[data-hook="product_description"]

[data-hook="product_images_num"]

[data-hook="product_collections"]

[data-hook="product_colors"]

[data-hook="selected_or_first_variant_sku"]

[data-hook="selected_or_first_variant_barcode"]

You can also use XPath if that is your preference.

For example:

//*[@data-hook="product_name"]

//*[@data-hook="product_available"]

//*[@data-hook="product_images_num"]

For simple fields, extracting the inner text is usually enough.

For nested fields like product collections or product colors, Screaming Frog may return the combined text of all child list items. Depending on how you want the export to look, you can keep that combined value, extract individual list items, or modify the Liquid to output a comma-separated list.

For example, if you prefer a comma-separated collection list, you could output collections like this:

{% assign excludedCols = "Sale,New Arrivals,Back In Stock" | split: "," %}

{% assign auditCollections = "" %}

{% for collection in product.collections %}

{% unless excludedCols contains collection.title %}

{% capture auditCollections %}

{{ auditCollections }}{% unless auditCollections == blank %}, {% endunless %}{{ collection.title }}

{% endcapture %}

{% endunless %}

{% endfor %}

</li>

The same idea can be used for colors or other product options.

The color option is worth noting because option names vary across Shopify stores. One store may use Color, another Colour, and others may use Finish, Fabric, Material, or something custom.

The example code checks for Color and color, but you may need to adjust this depending on how the client manages product options.

For example, on a UK store, you may need:

product.options_by_name['Colour']

For a furniture store, you may need:

product.options_by_name['Fabric']

For a flooring store, you may need:

product.options_by_name['Finish']

Not every store will have the option you’re used to.

Shopify product options can be accessed through Liquid and output in a consistent audit-friendly format.

You can also extend this snippet with additional fields depending on the audit.

For example, you may want to add:

Product ID
Product handle
Product URL
Featured image
Number of variants
Selected variant price
Compare-at price
Tags
Media count
Whether the product has only a default variant
Whether the product is assigned to any collections

Here are a few examples:

<li data-hook="product_id">{{ product.id }}</li>

<li data-hook="product_handle">{{ product.handle | escape }}</li>

<li data-hook="product_url">{{ product.url | escape }}</li>

<li data-hook="product_variants_num">{{ product.variants.size }}</li>

<li data-hook="product_tags">{{ product.tags | join: ", " | escape }}</li>

<li data-hook="product_has_only_default_variant">{{ product.has_only_default_variant }}</li>

These extra fields can be especially useful when you need to join Screaming Frog crawl data with Shopify exports, Google Search Console data, analytics data, rankings, or other datasets.

For example, you could use the crawl to answer questions like:

Which organic landing pages have products with fewer than three images?
Which indexed product pages are currently unavailable?
Which product pages have missing or weak descriptions?
Which vendors or product types generate the most SEO issues?
Which product pages belong only to promotional collections and not meaningful category collections?
Which products have missing SKUs or barcodes?
Which product pages have many variants but very little descriptive content?
Which product pages are live but not assigned to important collections?

This approach is not meant to replace proper Shopify data exports or API access. If you have direct access to Shopify admin exports, product feeds, or the Shopify API, those may provide cleaner product data.

This method is useful, however, when you are already crawling the site for SEO and want product-level context directly in the crawl export.

It is also useful when you want to connect what search engines and crawlers see on the front end with the underlying Shopify product data.

A few final reminders:

Use this as a temporary audit tool.
Do not expose sensitive data.
Add the snippet only where the product object is available.
Test on a duplicate theme when possible.
Keep the data-hook names consistent.
Adjust option names such as Color or Colour for each store.
Remove the code when the crawl is complete.
The main benefit is speed.

Instead of rebuilding custom XPath rules for every Shopify theme, you can create a predictable audit layer using Shopify Liquid itself. Once the hooks are in place, Screaming Frog can collect the product data in a much more consistent way.