With millions of stores built on Shopify, the platform holds valuable market data. Therefore, scraping Shopify stores offers a powerful way to gather this valuable information automatically. By extracting public details such as pricing, product descriptions, and inventory levels, you can easily monitor competitors and identify emerging market trends. This process can turn raw web pages into actionable insights for your business.
It might sound technical, but it doesn't have to be complicated. This guide explores the most effective techniques for every skill level. We will cover everything from the simple JSON URL trick to Python scripting and advanced headless browsers to help you get started.
Before jumping into Shopify scrapers, it is important to understand how these online shops are built. Knowing this foundational knowledge will save you time and help you avoid errors during the extraction process.
Most Shopify stores have a unique design, but they share the same underlying framework. This means you can often predict the URLs for important pages. For example, you will commonly find structures like:
example.com/products/[product-name]
example.com/collections/[collection-name]
example.com/pages/[page-title]
This standardization is a huge advantage because it allows you to create a scraping script that works across multiple stores with minimal changes.
Once you understand the structure, you can identify what information is available. Here are the key data points you can gather from these Shopify stores:
Product Information: It includes product titles, detailed descriptions, unique SKUs, and links to all associated images.
Pricing Data: You can easily track the current selling price, original price, and discount amounts.
Variants: Different options for each item, like size, color, material, or style.
Inventory Levels: You can check if an item is in stock or estimate how many units remain.
Customer Reviews: Star ratings and written feedback from real buyers.
Store Policies: You can also gather information on a store's shipping, return, and privacy policies.
This method is perfect if you have zero coding experience. It works great for Excel users, marketers, and anyone who just needs a quick way to grab product data. You do not need to install any software or write a single line of code.
Step 1: Access Raw Product Data
The process is incredibly straightforward. You can access raw product data by simply adding /products.json to the end of any store URL. This returns all the product information in a structured format called JSON. For example, it would look like this:
https://examplestore.com/products.jsonPress "Enter," and you will see a page full of raw data. It might look messy at first glance, but it contains everything you need. You will find product titles, prices, images, and inventory counts all in one place.
Step 2: View Product Data for More Products (Optional)
If you want to view more products, you can add a page parameter to the URL. By default, Shopify only shows 50 products at maximum per page. To see more, you can add a page number to the URL. For instance, to view the second page of products, use:
https://example-store.myshopify.com/products.json?page=2You can also ask for more products on a single page, up to 250 at a time. Just modify the URL like this:
https://example-store.myshopify.com/products.json?limit=250Step 3: Save the JSON as CSV
Once you have the data on your screen, you can copy it. Then, paste it into a free online JSON to CSV converter. This will quickly turn the raw data into a spreadsheet you can open with Excel or Google Sheets.
The main advantage of this method is its incredible simplicity. You can get useful data in just a few clicks. It is also completely free and works on many Shopify stores without any special setup. However, this method is not very scalable. If you need to collect data from many stores or gather a huge amount of products, you will quickly find it too slow and manual. For larger projects, you will need more advanced techniques.
If the JSON method feels too limited, scraping Shopify stores with Python scripting is your next step. This method is a great fit for students learning Python, data analysts, or anyone comfortable with basic programming. It gives you more control and allows you to automate the entire extraction process.
Before you start, make sure you have a few things ready on your computer.
Python installed: You can download it for free from the official Python website. Version 3.8 or higher is recommended.
The Requests library: This popular library lets Python fetch web pages. Install it by running pip install requests in your terminal.
The Pandas library (optional): This helps you organize and save data into a spreadsheet. Install it with pip install pandas.
Once these are set up, you are ready to write your first Shopify scraper.
The goal of this script is to connect to a Shopify store's JSON endpoint. Then you can pull the product data, and save it into a CSV file.
Step 1: Import the libraries
Start by importing the tools you just installed.
import requests
import pandas as pd
import time # For adding delays between requests
import json # For JSON parsing error handlingStep 2: Define the target URL
Next, specify the Shopify store you want to scrape. We will use the /products.json endpoint.
# REPLACE THIS WITH YOUR TARGET STORE URL
base_url = "https://example-store.myshopify.com/products.json"
# Headers to mimic a real browser visit
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130+ Safari/537.36",
"Accept": "application/json",
"Accept-Language": "en-US,en;q=0.9"
}Step 3: Fetch the data
For stores with few products, you can start with a simple script:
try:
# Send request with timeout and headers
response = requests.get(base_url, headers=headers, timeout=10)
response.raise_for_status() # Raise exception for bad status codes
# Parse JSON response
data = response.json()
products = data.get('products', [])
product_list = []
for product in products:
# SAFE data extraction with error handling
variants = product.get('variants', [])
product_info = {
'title': product.get('title', 'N/A'),
'vendor': product.get('vendor', 'N/A'),
'price': variants[0].get('price') if variants else 'N/A',
'sku': variants[0].get('sku') if variants else 'N/A'
}
product_list.append(product_info)
# Save to CSV
if product_list:
df = pd.DataFrame(product_list)
df.to_csv('shopify_products.csv', index=False, encoding='utf-8')
print(f"Successfully saved {len(product_list)} products!")
else:
print("No products found.")
except requests.exceptions.RequestException as e:
print(f"Request failed: {e}")
except json.JSONDecodeError as e:
print(f"Failed to parse JSON: {e}")
except Exception as e:
print(f"Unexpected error: {e}")Most Shopify stores have more products than fit on one page. Here's a robust script that handles pagination properly:
import requests
import pandas as pd
import time
import json
def scrape_shopify_store(store_url, max_pages=50):
"""
Safely scrape products from a Shopify store with pagination support
Args:
store_url: The store's products.json URL
max_pages: Maximum pages to scrape (safety limit)
"""
# Store configuration
base_url = store_url
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
"Accept": "application/json"
}
all_products = []
page = 1
limit = 250 # Shopify's maximum per page
print(f"Starting to scrape: {base_url}")
while page <= max_pages:
print(f"Fetching page {page}...")
try:
# Construct URL with pagination
# Note: Some stores may use different pagination methods
url = f"{base_url}?limit={limit}&page={page}"
# Send request with timeout
response = requests.get(url, headers=headers, timeout=15)
# Handle common HTTP status codes
if response.status_code == 429:
print("Rate limit exceeded. Waiting 5 seconds...")
time.sleep(5)
continue # Retry current page
elif response.status_code == 403:
print("Access denied. The store may have anti-scraping measures.")
break
elif response.status_code == 404:
print("Page not found. Check the store URL.")
break
elif response.status_code != 200:
print(f"Request failed with status: {response.status_code}")
break
# Parse JSON response
try:
data = response.json()
except json.JSONDecodeError:
print("Invalid JSON response received")
break
products = data.get('products', [])
# Stop if no products returned
if not products:
print("No more products available.")
break
# Extract product information safely
for product in products:
variants = product.get('variants', [])
product_info = {
'id': product.get('id'),
'title': product.get('title'),
'vendor': product.get('vendor'),
'handle': product.get('handle'),
'product_type': product.get('product_type'),
'price': variants[0].get('price') if variants else None,
'sku': variants[0].get('sku') if variants else None,
'inventory': variants[0].get('inventory_quantity') if variants else None,
'created_at': product.get('created_at'),
'page': page
}
all_products.append(product_info)
print(f" Found {len(products)} products on page {page}")
# 🚨 CRITICAL: Check if we've reached the last page
# If returned products are fewer than limit, it's the last page
if len(products) < limit:
print("Reached the last page of products.")
break
# 🚨 IMPORTANT: Add delay to be polite and avoid being blocked
time.sleep(1) # 1 second delay between requests
page += 1
except requests.exceptions.Timeout:
print(f"Timeout on page {page}, skipping...")
page += 1
continue
except requests.exceptions.RequestException as e:
print(f"Network error: {e}")
break
except Exception as e:
print(f"Unexpected error: {e}")
break
# Save results if we collected any data
if all_products:
df = pd.DataFrame(all_products)
# Remove any duplicate products based on ID
if 'id' in df.columns:
initial_count = len(df)
df = df.drop_duplicates(subset=['id'], keep='first')
removed = initial_count - len(df)
if removed > 0:
print(f"Removed {removed} duplicate products")
# Save to CSV
filename = 'shopify_products_complete.csv'
df.to_csv(filename, index=False, encoding='utf-8')
# Print summary
print("\n" + "="*50)
print("SCRAPING COMPLETED SUCCESSFULLY!")
print(f"Total products saved: {len(df)}")
print(f"Total pages scraped: {page}")
print(f"File saved as: {filename}")
print("="*50)
else:
print("No products were collected.")
# Example usage
if __name__ == "__main__":
# ALWAYS check these before running:
print("="*60)
print("IMPORTANT: Before running this script, ensure you:")
print("1. Have permission to scrape the target store")
print("2. Have checked the store's robots.txt file")
print("3. Are complying with their terms of service")
print("4. Are using appropriate delays between requests")
print("="*60)
# Replace with your target store URL
target_store = "https://example-store.myshopify.com/products.json"
# Optional: Add a small random delay before starting
time.sleep(2)
# Run the scraper
scrape_shopify_store(target_store)Step 4: Save to a CSV file
Finally, use pandas to convert your list into a clean spreadsheet.
df = pd.DataFrame(product_list)
df.to_csv('shopify_products.csv', index=False)
print("Data saved successfully!")Sometimes the JSON endpoint is inaccessible, or a store loads its content dynamically through JavaScript. When this happens, the previous methods will not work. You will then need a more powerful approach to scrape Shopify stores. This is where headless browsers come in. They allow you to interact with websites just like a real browser would, but without the graphical interface.
This method works well for professional scraping engineers, full-stack developers, and experienced programmers. It requires more setup and runs slower than the JSON approach, but it can handle almost any website.
To get started with headless browser scraping, you will need to set up a few tools on your system.
A Headless Browser Library: The most popular choices are:
Puppeteer: Developed by Google, it provides a high-level API to control Chrome or Chromium over the DevTools Protocol.
Playwright: Developed by Microsoft, it supports multiple browsers (Chromium, Firefox, WebKit) and offers a robust API.
Selenium: A long-standing tool for browser automation, also capable of running in headless mode.
For this guide, we'll focus on Playwright (for Python) due to its modern API and a strong developer experience compared to older automation tools.
The pandas Library: This data analysis library will help organize and export the scraped product data.
A Code Editor: Like VS Code, Sublime Text, or Atom, to write your JavaScript code.
Python: Ensure you have Python 3.8 or newer installed. Playwright for Python and pandas both require a working Python environment.
Instead of hitting the JSON endpoint, we will visit the actual store page. Let it fully render, and then extract product information directly from the HTML elements.
Step 1. Install essential tools
First, install the Playwright library for Python:
pip install playwright pandasThen, download the browser binaries:
playwright install chromiumStep 2. Import the libraries and configure the browser
We set up the browser to mimic a modern desktop browsing environment and reduce basic bot detection.
from playwright.sync_api import sync_playwright
import pandas as pd
import time
import random
from urllib.parse import urljoin, urlparse
def create_browser():
"""Launch a headless browser with realistic settings."""
playwright = sync_playwright().start()
browser = playwright.chromium.launch(
headless=True,
args=[
'--no-sandbox',
'--disable-setuid-sandbox',
'--disable-dev-shm-usage',
'--disable-blink-features=AutomationControlled'
]
)
# Modern 2025 user agents
user_agents = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
]
selected_ua = random.choice(user_agents)
context = browser.new_context(
viewport={"width": 1920, "height": 1080},
user_agent=selected_ua,
extra_http_headers={
'Accept-Language': 'en-US,en;q=0.9',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Sec-CH-UA': '"Chromium";v="125", "Not.A/Brand";v="24"',
'Sec-CH-UA-Platform': '"Windows"'
}
)
# Basic stealth: avoid exposing navigator.webdriver to naive checks
context.add_init_script(
"Object.defineProperty(navigator, 'webdriver', {get: () => undefined})"
)
return playwright, browser, contextStep 3: Navigate to the store and wait for the content to load
def make_absolute_url(relative_url, base_url):
"""Convert relative URL to absolute URL."""
if not relative_url or relative_url == "N/A":
return "N/A"
if relative_url.startswith(("http://", "https://")):
return relative_url
if relative_url.startswith("//"):
return f"https:{relative_url}"
parsed_base = urlparse(base_url)
base_domain = f"{parsed_base.scheme}://{parsed_base.netloc}"
return urljoin(base_domain, relative_url)
def visit_store_page(context, url):
"""Open the store page and wait for products to appear."""
page = context.new_page()
print(f"Visiting: {url}")
try:
# 'domcontentloaded' is faster and more stable than 'networkidle'
page.goto(url, wait_until="domcontentloaded", timeout=45000)
# Allow time for client-side rendering
page.wait_for_timeout(2000)
product_selectors = [
".product-card",
".product-item",
".grid__item",
'[data-product-id]',
'[data-product-handle]'
]
found_elements = False
for selector in product_selectors:
elements = page.query_selector_all(selector)
if elements:
print(f"Found {len(elements)} products using selector: {selector}")
found_elements = True
break
if not found_elements:
links = page.query_selector_all('a[href*="/products/"]')
if links:
print(f"Found {len(links)} product links")
found_elements = True
if found_elements:
page.mouse.wheel(0, 500)
page.wait_for_timeout(1000)
except Exception as e:
print(f"Navigation warning: {e}")
page.close()
return None
return pageStep 4: Extract product data from the rendered HTML
This is the core of the method. Note that different Shopify themes use different class names, so you may need to adjust these selectors for your target store.
def extract_products(page):
"""Pull product information from the current page."""
products = []
product_selectors = [
".product-card",
".product-item",
".grid__item",
".product-grid-item",
'[data-product-id]',
'[data-product-handle]'
]
product_elements = []
for selector in product_selectors:
elements = page.query_selector_all(selector)
if elements:
print(f"Using selector: {selector}")
product_elements = elements
break
if not product_elements:
all_links = page.query_selector_all('a[href*="/products/"]')
unique_containers = []
for link in all_links:
container = link.evaluate_handle("""
element => {
let container = element;
for (let i = 0; i < 3; i++) {
if (container.parentElement) {
container = container.parentElement;
}
}
return container;
}
""")
if container not in unique_containers:
unique_containers.append(container)
product_elements = unique_containers
if product_elements:
print(f"Found {len(product_elements)} product containers via links")
print(f"Found {len(product_elements)} product elements on this page.")
base_url = page.url.split("?")[0]
for element in product_elements:
try:
product_id = element.get_attribute("data-product-id") or "N/A"
product_handle = element.get_attribute("data-product-handle") or "N/A"
title = "N/A"
for selector in [
".product-card__title",
".product__title",
".product-item__title",
"h3 a",
"h2 a",
".card__heading",
"h3",
"h2"
]:
el = element.query_selector(selector)
if el and el.text_content().strip():
title = el.text_content().strip()
break
price = "N/A"
for selector in [
".price",
".product-price",
".money",
".price__regular",
".card-information__text"
]:
el = element.query_selector(selector)
if el and el.text_content().strip():
price = el.text_content().strip()
break
link = "N/A"
link_el = element.query_selector("a[href*='/products/']")
if link_el:
href = link_el.get_attribute("href")
if href:
link = make_absolute_url(href, base_url)
image = "N/A"
img_el = element.query_selector("img")
if img_el:
src = img_el.get_attribute("src") or img_el.get_attribute("data-src")
if src:
image = make_absolute_url(src, base_url)
vendor_el = element.query_selector(".vendor, [data-vendor]")
vendor = vendor_el.text_content().strip() if vendor_el else "N/A"
if title != "N/A" and title.strip() and price != "N/A":
products.append({
"product_id": product_id,
"product_handle": product_handle,
"title": title,
"price": price,
"link": link,
"image": image,
"vendor": vendor
})
except Exception as e:
print(f"Warning: Error extracting one product: {e}")
continue
return productsStep 5: Handle multiple pages
This method supports traditional pagination-based Shopify collections. Infinite-scroll or cursor-based pagination is not covered.
def check_next_page(page, current_page):
"""Check if there's a next page available."""
try:
next_selectors = [
'a[rel="next"]',
'.pagination__next',
'.next',
'.page-next',
'a:has-text("Next")'
]
for selector in next_selectors:
next_btn = page.query_selector(selector)
if next_btn:
is_disabled = (
next_btn.get_attribute("disabled") is not None or
next_btn.get_attribute("aria-disabled") == "true" or
"disabled" in (next_btn.get_attribute("class") or "").lower()
)
if not is_disabled:
return True
page_numbers = page.query_selector_all('.pagination__page, .page')
for elem in page_numbers:
try:
if int(elem.text_content()) > current_page:
return True
except:
continue
except Exception as e:
print(f"Error checking next page: {e}")
return False
def is_duplicate_page(current_products, previous_products):
"""Check if the current page shows the same products as the previous page."""
if not previous_products or not current_products:
return False
compare_count = min(3, len(current_products), len(previous_products))
if compare_count == 0:
return False
matches = 0
for i in range(compare_count):
if (current_products[i]["title"] == previous_products[i]["title"] and
current_products[i]["price"] == previous_products[i]["price"]):
matches += 1
return matches / compare_count >= 0.7
def scrape_all_pages(base_url, max_pages=10):
"""Scrape products from multiple pages of a Shopify collection."""
playwright, browser, context = create_browser()
all_products = []
current_page = 1
previous_page_products = []
try:
while current_page <= max_pages:
if current_page == 1:
url = base_url
else:
url = f"{base_url}?page={current_page}"
print(f"\nProcessing page {current_page}: {url}")
page = visit_store_page(context, url)
if not page:
print("Could not load page, stopping.")
break
products = extract_products(page)
if not products:
print("No products found on this page. Finished.")
page.close()
break
if previous_page_products and is_duplicate_page(products, previous_page_products):
print("Duplicate page detected. Finished.")
page.close()
break
all_products.extend(products)
previous_page_products = products
print(f"Page {current_page}: collected {len(products)} products.")
has_next_page = check_next_page(page, current_page)
page.close()
if not has_next_page:
print("No more pages available. Finished.")
break
current_page += 1
if current_page <= max_pages:
print("Waiting before next page...")
time.sleep(1.5)
finally:
browser.close()
playwright.stop()
return all_productsStep 6: Run the scraper and save results
def main():
# Use the collections page for best results
store_url = "https://example-store.myshopify.com/collections/all"
print("Starting headless browser scraper...")
products = scrape_all_pages(store_url, max_pages=10)
if products:
df = pd.DataFrame(products)
initial_count = len(df)
if 'product_id' in df.columns and df['product_id'].nunique() > 1:
df = df.drop_duplicates(subset=['product_id'], keep='first')
elif 'product_handle' in df.columns and df['product_handle'].nunique() > 1:
df = df.drop_duplicates(subset=['product_handle'], keep='first')
else:
df = df.drop_duplicates(subset=['title', 'price'], keep='first')
from datetime import datetime
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
filename = f"shopify_products_{timestamp}.csv"
df.to_csv(filename, index=False, encoding="utf-8")
print(f"\nDone! Saved {len(df)} unique products to {filename}")
print(f"Removed {initial_count - len(df)} duplicate entries")
if len(df) > 0:
print("\nSample of collected data:")
print(df[['title', 'price', 'vendor']].head().to_string(index=False))
else:
print("No products collected.")
if __name__ == "__main__":
main()The main advantage of headless browser scraping is its flexibility. It can access stores where JSON endpoints are blocked and extract data from pages that rely on JavaScript for rendering. This makes it suitable for many modern Shopify storefronts that cannot be handled with simple HTTP requests.
At the same time, there are clear trade-offs. Headless browsers consume more memory and CPU than standard requests. They also run slower because each page must be fully rendered before data can be collected. For large-scale projects that target many stores, this approach often requires stronger infrastructure or the use of cloud-based browser services.
Scraping a Shopify store is not always a smooth process. You might encounter technical blocks or complex website structures along the way. Here are the most common obstacles and practical ways to solve them.
Dynamic Content Loading & JSON Blocks
Many modern Shopify stores use JavaScript to load products or pricing, which cannot be read by simple requests. Furthermore, some store owners specifically block the default /products.json endpoint for security reasons. In both situations, the standard HTTP request method will fail or provide incomplete data.
Solution: Use headless browsers like Playwright. These tools render the entire page, execute all necessary JavaScript, and allow you to extract data from the final, visible HTML elements.
IP Blocking
Websites track visitor activity very closely. If one IP address views thousands of pages very quickly, it is flagged as unusual traffic. This rapid speed looks robotic, not human. Also, bots often lack the random delays or natural browser details a real shopper has. This strange and fast behavior causes security systems to block the IP address right away.
Solution: You need to rotate your digital identity constantly. Using high-quality rotating residential proxies is essential for such a case. Services like IPcook offer these reliable IP addresses, which make your scraper look exactly like a genuine shopper to Shopify's security. This constant rotation prevents any single IP from getting flagged, ensuring continuous data collection.
👍 Key Features of IPcook:
Global Reach: Over 55 million real, residential IP addresses worldwide
Automatic Rotation: IPs change constantly for seamless, non-stop data collection.
Session Control: Option to keep the same IP for up to 24 hours when needed.
High Uptime: Guaranteed 99.99% reliability for operations.
Cost-Effective: Starts from ＄0.5/GB, ideal for frequent web scraping tasks.
CAPTCHAs and Advanced Bot Detection
Even with a good IP, systems like Cloudflare sometimes can still challenge you with CAPTCHAs if they detect non-human browser behavior. If a challenge screen appears, a basic code script will fail.
Solution: Implement stealth techniques (masking the navigator.webdriver flag) in your headless browser to fix browser fingerprints.
Rate Limits and 429 Errors
Servers protect themselves from being overwhelmed. If you send too many requests in a short time, the server will block you with a 429 Error.
Solution: Add a delay between your requests (e.g., using time.sleep()). Scrape at a slow and predictable speed, typically waiting 1.5 to 3 seconds between page loads. This helps avoid site performance issues.
Now you have a clear roadmap for scraping Shopify stores. Choosing the right method really depends on your needs. For a quick look at a store's products, the simple JSON method is perfect. When you need more control and automation, a custom Python script offers greater flexibility. And for those complex, JavaScript-heavy sites that block simpler techniques, a headless browser is your most reliable tool.
However, no matter which approach you choose, you will face a common challenge: IP blocks. Websites are designed to detect and stop automated traffic from a single source. Therefore, using a high-quality proxy service is necessary. For this, we recommend using rotating residential proxies from IPcook. Its network makes your requests appear as if they are coming from genuine shoppers, which can dramatically reduce the chance of being blocked. Plus, its service is incredibly affordable, with plans starting from just $0.5/GB.
With the right method, a reliable proxy, and a commitment to ethical scraping, you are well-equipped to unlock the market insights you need to succeed.
