How to Crawl Shopee Data: A Step-by-Step Guide

This guide explains how to plan an authorized Shopee data collection workflow, validate product records and export a usable file. The runnable example uses clearly labelled fictional HTML so you can test the code without making requests to Shopee. Live implementation still requires permission, market-specific selectors or an approved interface, and validation against the actual source.

Table of Contents

[Open][Close]

1. Introduction to Web Scraping and Data Crawling
2. Why Crawl Shopee Data? Business Use Cases
3. Ethical and Legal Considerations in Crawling Shopee
4. Setting Up the Environment and Tools
5. Step-by-Step Guide to Crawl Shopee Data
6. Advanced Techniques and Optimizations
7. Handling Legal and Technical Challenges
8. Best Practices for Data Management
- Data Cleaning and Consistency
- Database Storage
9. Leveraging the Data: Analysis and Visualization
- Visualization Tools
- Advanced Analysis Techniques
10. Conclusion: Responsible Crawling and Future Directions

1. Introduction to Web Scraping and Data Crawling

intro-to-web-scraping-and-crawling — **Introduction to Web Scraping and Data Crawling**

Understanding Web Scraping

Web scraping extracts selected fields from a page. It does not guarantee access, freshness or complete marketplace coverage. Start by checking whether an authorized export or API provides the fields you need; distinguish the source’s update frequency from your collection schedule.

Data Crawling Defined

Data crawling discovers and visits URLs within a defined scope; extraction turns the retrieved content into records. Define the country site, allowed paths, product/variant identifiers, pagination limits and stop conditions before collecting. See our crawling versus scraping comparison for the distinction.

Key Tools for Web Scraping

To crawl and scrape Shopee data effectively, you’ll likely use a mix of tools and frameworks, each offering unique advantages depending on the nature of your data and the complexity of Shopee’s website. Common tools include:

Python Libraries:
- BeautifulSoup: For parsing HTML, ideal for simpler scraping tasks.
- Scrapy: A powerful framework for large-scale, structured data extraction.
- Selenium: For handling dynamic content or pages that require user interaction.
JavaScript Libraries:
- Puppeteer: Often used for scraping JavaScript-heavy sites where the content is rendered dynamically.
Network configuration: use only the routing permitted for your project. Proxies do not grant access rights, guarantee availability or replace rate limits.

Tool choice does not grant permission or guarantee compliance. Confirm the intended source, allowed collection method and reuse conditions before a live run.

2. Why Crawl Shopee Data? Business Use Cases

Crawling Shopee data provides actionable insights for a variety of business applications:

Market Research and Trend Analysis

A repeatable category sample can help compare observed prices and product availability over time. Record its market, category, capture time and selection method; a partial sample cannot establish total marketplace demand or sales.

Competitor Monitoring

Keeping tabs on competitor product listings, prices, and promotional strategies allows businesses to refine their own pricing models. By observing competitors’ promotional patterns or seasonal discounts, a business can adjust its listings to stay competitive.

Dynamic Pricing and Price Optimization

Price monitoring can support pricing decisions when observations are comparable. Keep currency, variant, promotion eligibility and shipping treatment explicit. The listed price alone does not establish the final checkout price or a profitable selling price.

Customer Sentiment Analysis

Where collection and reuse are permitted, reviews can help identify recurring themes. Record the language, date range and sampling limits; reviewers are not a representative sample of all customers. Avoid retaining unnecessary names or other personal information.

3. Ethical and Legal Considerations in Crawling Shopee

When crawling Shopee data, it’s essential to operate within legal and ethical boundaries to avoid penalties and protect consumer privacy.

Adhere to Shopee’s robots.txt File

Check the /robots.txt file on the exact regional hostname you intend to access. The Robots Exclusion Protocol communicates crawler rules; it is not access authorization. Review the relevant market’s terms and your permission separately.

Rate Limiting and Respecting Server Load

Use the source’s permitted request rate and concurrency, cache reusable responses, and bound retries. Stop on refusals or rate limits rather than repeatedly sending the same request. A delay between requests does not guarantee continued access.

Comply with Data Protection Regulations

Collect only fields necessary for the stated purpose, define retention and access controls, and assess applicable privacy obligations for the relevant people and markets. Public visibility, encryption or removing names does not by itself establish that a collection is compliant.

4. Setting Up the Environment and Tools

Select the Right Tool for Your Project

Choosing the right tool depends on your project’s scope and complexity:

BeautifulSoup and Requests for small, simple projects.
Scrapy for complex, large-scale crawling.
Selenium for dynamic sites where pages load additional content on interaction.

Proxy and User-Agent Configuration

Use a consistent, accurate client identity where appropriate and document any authorized proxy routing. Rotating user agents or addresses is not a substitute for permission and does not guarantee access. Treat a login screen, CAPTCHA or refusal as a stop-and-review condition.

Setting Up Python Libraries

Run the following commands in a terminal using the Python environment where you will run the script. The main example needs Requests and Beautiful Soup; Selenium is optional. A notebook-only !pip command is not valid in a normal Python file.


Copied!python -m pip install beautifulsoup4 requests
# Optional browser-rendering example only:
python -m pip install selenium
python -m pip install beautifulsoup4 requests
# Optional browser-rendering example only:
python -m pip install selenium

Install a WebDriver for Selenium

Current Selenium bindings normally use Selenium Manager to discover or obtain a compatible driver. Restricted networks may require a separately configured driver. Install only from trusted sources and keep browser and driver compatibility in mind.

5. Step-by-Step Guide to Crawl Shopee Data

The following three Python blocks form one script, crawl_example.py. They demonstrate retrieval boundaries, validated extraction and CSV export. Run them in order: the default fixture produces two fictional product records with titles, prices and one optional rating. They are not a tested live Shopee crawler or a real dataset.

Step 1: Inspect Shopee’s HTML Structure

For an authorized live source, inspect both the returned HTML and rendered page. Verify stable product identifiers, title/price selectors, currency and optional rating fields. A successful HTTP response may still be a login or challenge page. The article.product selectors below describe only our fixture; do not assume they exist on Shopee.

Step 2: Writing the Basic Code

Start with the local fixture below. The optional fetch function sets explicit timeouts, rejects redirects and unexpected response types, and stops on common refusal/rate-limit statuses. These checks follow the Requests timeout and error-handling guidance; they do not verify permission or prove that a response contains products.


Copied!from datetime import datetime, timezone
from urllib.parse import urlsplit
import requests

# Fictional local fixture: these are NOT Shopee selectors or market observations.
html_text = """<main>
<article class="product" data-product-id="demo-1">
  <h2 class="title">Example storage box</h2>
  <span class="price" data-currency="VND">125000</span>
  <span class="rating">4.5</span>
</article>
<article class="product" data-product-id="demo-2">
  <h2 class="title">Example notebook</h2>
  <span class="price" data-currency="VND">39000</span>
</article>
</main>"""
source_url = "local:fictional-product-fixture"
observed_at = datetime.now(timezone.utc).isoformat()

def fetch_permitted_html(url, expected_host, *, authorized=False):
    """One request; call only for a specifically authorized HTTPS source."""
    parsed = urlsplit(url)
    if (not authorized or parsed.scheme != "https"
            or parsed.hostname != expected_host or parsed.username
            or parsed.password or parsed.port not in (None, 443)):
        raise ValueError("Require authorization and an exact HTTPS hostname")
    # No redirects or retry loop: review refusals and destination changes manually.
    with requests.Session() as session:
        response = session.get(url, timeout=(5, 20), allow_redirects=False)
    if response.status_code in (401, 403, 429):
        raise RuntimeError(f"Stop and review access/limits: HTTP {response.status_code}")
    if response.status_code != 200:
        raise RuntimeError(f"Unexpected HTTP status: {response.status_code}")
    if "text/html" not in response.headers.get("Content-Type", "").lower():
        raise ValueError("Expected HTML, not an API response or download")
    return response.text

# Default: use the local fixture above; no request is made.
# For permitted live collection, replace html_text/source_url only after
# verifying the exact URL, robots rules, permission and actual page selectors.
from datetime import datetime, timezone
from urllib.parse import urlsplit
import requests

# Fictional local fixture: these are NOT Shopee selectors or market observations.
html_text = """<main>
<article class="product" data-product-id="demo-1">
  <h2 class="title">Example storage box</h2>
  <span class="price" data-currency="VND">125000</span>
  <span class="rating">4.5</span>
</article>
<article class="product" data-product-id="demo-2">
  <h2 class="title">Example notebook</h2>
  <span class="price" data-currency="VND">39000</span>
</article>
</main>"""
source_url = "local:fictional-product-fixture"
observed_at = datetime.now(timezone.utc).isoformat()

def fetch_permitted_html(url, expected_host, *, authorized=False):
    """One request; call only for a specifically authorized HTTPS source."""
    parsed = urlsplit(url)
    if (not authorized or parsed.scheme != "https"
            or parsed.hostname != expected_host or parsed.username
            or parsed.password or parsed.port not in (None, 443)):
        raise ValueError("Require authorization and an exact HTTPS hostname")
    # No redirects or retry loop: review refusals and destination changes manually.
    with requests.Session() as session:
        response = session.get(url, timeout=(5, 20), allow_redirects=False)
    if response.status_code in (401, 403, 429):
        raise RuntimeError(f"Stop and review access/limits: HTTP {response.status_code}")
    if response.status_code != 200:
        raise RuntimeError(f"Unexpected HTTP status: {response.status_code}")
    if "text/html" not in response.headers.get("Content-Type", "").lower():
        raise ValueError("Expected HTML, not an API response or download")
    return response.text

# Default: use the local fixture above; no request is made.
# For permitted live collection, replace html_text/source_url only after
# verifying the exact URL, robots rules, permission and actual page selectors.

Step 3: Parse HTML Data with BeautifulSoup

Append this parser to the same script. Required fields and duplicate IDs fail visibly; a missing rating stays missing rather than becoming zero. The decimal-price format is deliberately strict and applies only to this fixture. For a live market, explicitly handle its separators, ranges, variants and currency rules before parsing.


Copied!from bs4 import BeautifulSoup
from decimal import Decimal, InvalidOperation
import re

def parse_products(document, source, timestamp):
    soup = BeautifulSoup(document, "html.parser")
    products = soup.select("article.product")
    if not products:
        raise ValueError("No product cards: check rendering, selectors or a challenge page")
    rows, seen = [], set()
    for product in products:
        product_id = (product.get("data-product-id") or "").strip()
        title = product.select_one(".title")
        price = product.select_one(".price")
        rating = product.select_one(".rating")
        if not product_id or product_id in seen:
            raise ValueError("Missing or duplicate product ID; do not silently merge")
        if title is None or not title.get_text(strip=True) or price is None:
            raise ValueError(f"Missing required field: {product_id}")
        currency = (price.get("data-currency") or "").strip().upper()
        price_text = price.get_text(strip=True)
        # Fixture format only: no currency symbols, thousands separators or ranges.
        if not re.fullmatch(r"[A-Z]{3}", currency):
            raise ValueError("Missing/invalid currency code; verify the source currency")
        if not re.fullmatch(r"[0-9]+(?:\.[0-9]+)?", price_text):
            raise ValueError("Ambiguous price format; use a market-specific parser")
        amount = Decimal(price_text)
        rating_value = None
        if rating is not None and rating.get_text(strip=True):
            try:
                rating_value = Decimal(rating.get_text(strip=True))
            except InvalidOperation as error:
                raise ValueError("Invalid rating") from error
            if not rating_value.is_finite() or not 0 <= rating_value <= 5:
                raise ValueError("Rating must be a finite number between 0 and 5")
        seen.add(product_id)
        rows.append({
            "product_id": product_id,
            "title": title.get_text(" ", strip=True),
            "price": str(amount),
            "currency": currency,
            "rating": str(rating_value) if rating_value is not None else None,
            "source_url": source,
            "observed_at": timestamp,
        })
    return rows

rows = parse_products(html_text, source_url, observed_at)
print(f"Validated {len(rows)} fictional fixture rows")
from bs4 import BeautifulSoup
from decimal import Decimal, InvalidOperation
import re

def parse_products(document, source, timestamp):
    soup = BeautifulSoup(document, "html.parser")
    products = soup.select("article.product")
    if not products:
        raise ValueError("No product cards: check rendering, selectors or a challenge page")
    rows, seen = [], set()
    for product in products:
        product_id = (product.get("data-product-id") or "").strip()
        title = product.select_one(".title")
        price = product.select_one(".price")
        rating = product.select_one(".rating")
        if not product_id or product_id in seen:
            raise ValueError("Missing or duplicate product ID; do not silently merge")
        if title is None or not title.get_text(strip=True) or price is None:
            raise ValueError(f"Missing required field: {product_id}")
        currency = (price.get("data-currency") or "").strip().upper()
        price_text = price.get_text(strip=True)
        # Fixture format only: no currency symbols, thousands separators or ranges.
        if not re.fullmatch(r"[A-Z]{3}", currency):
            raise ValueError("Missing/invalid currency code; verify the source currency")
        if not re.fullmatch(r"[0-9]+(?:\.[0-9]+)?", price_text):
            raise ValueError("Ambiguous price format; use a market-specific parser")
        amount = Decimal(price_text)
        rating_value = None
        if rating is not None and rating.get_text(strip=True):
            try:
                rating_value = Decimal(rating.get_text(strip=True))
            except InvalidOperation as error:
                raise ValueError("Invalid rating") from error
            if not rating_value.is_finite() or not 0 <= rating_value <= 5:
                raise ValueError("Rating must be a finite number between 0 and 5")
        seen.add(product_id)
        rows.append({
            "product_id": product_id,
            "title": title.get_text(" ", strip=True),
            "price": str(amount),
            "currency": currency,
            "rating": str(rating_value) if rating_value is not None else None,
            "source_url": source,
            "observed_at": timestamp,
        })
    return rows

rows = parse_products(html_text, source_url, observed_at)
print(f"Validated {len(rows)} fictional fixture rows")

Step 4: Store Data in CSV Format

Append this export block, then run python crawl_example.py in a writable working folder. It creates product_fixture.csv with two rows; the second rating is blank. Existing files are not overwritten. The spreadsheet-facing export prefixes formula-like text with an apostrophe; keep an unmodified source copy separately and import untrusted CSV fields as text.


Copied!import csv
from pathlib import Path

FIELDS = ["product_id", "title", "price", "currency", "rating",
          "source_url", "observed_at"]

def spreadsheet_text(value):
    text = "" if value is None else str(value)
    # CSV quoting alone does not prevent spreadsheet formula interpretation.
    if text.lstrip().startswith(("=", "+", "-", "@")) or text.startswith(("\t", "\r", "\n")):
        return "'" + text
    return text

def write_csv(records, path):
    # Exclusive creation protects an existing export from accidental overwrite.
    with Path(path).open("x", newline="", encoding="utf-8-sig") as handle:
        writer = csv.DictWriter(handle, fieldnames=FIELDS)
        writer.writeheader()
        for row in records:
            writer.writerow({key: spreadsheet_text(row[key]) for key in FIELDS})

write_csv(rows, "product_fixture.csv")
print("Saved product_fixture.csv; this is demonstration data, not a Shopee dataset")
import csv
from pathlib import Path

FIELDS = ["product_id", "title", "price", "currency", "rating",
          "source_url", "observed_at"]

def spreadsheet_text(value):
    text = "" if value is None else str(value)
    # CSV quoting alone does not prevent spreadsheet formula interpretation.
    if text.lstrip().startswith(("=", "+", "-", "@")) or text.startswith(("\t", "\r", "\n")):
        return "'" + text
    return text

def write_csv(records, path):
    # Exclusive creation protects an existing export from accidental overwrite.
    with Path(path).open("x", newline="", encoding="utf-8-sig") as handle:
        writer = csv.DictWriter(handle, fieldnames=FIELDS)
        writer.writeheader()
        for row in records:
            writer.writerow({key: spreadsheet_text(row[key]) for key in FIELDS})

write_csv(rows, "product_fixture.csv")
print("Saved product_fixture.csv; this is demonstration data, not a Shopee dataset")

Step 5: Automate with Scheduling

Schedule only after a permitted live collection passes validation. Use a run identifier, prevent overlapping runs, retain error counts and record the source, capture time and stop reason. Use a fresh output filename per run. The demonstration’s fixed filename intentionally fails on a second execution instead of replacing an earlier export.

6. Advanced Techniques and Optimizations

Handle Infinite Scrolling with Selenium

Some authorized pages need browser rendering. This optional function follows Selenium’s explicit-wait approach with a card-count wait, a scroll limit, a minimum interval and finally cleanup. The selector must be verified on the actual page. A timeout or stable card count is a stopping condition, not proof of complete coverage; virtualized lists require collecting records during each scroll.


Copied!from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from urllib.parse import urlsplit
import time

def rendered_snapshot(url, expected_host, card_selector, *, authorized=False,
                      max_scrolls=5, min_interval=2):
    """Bounded final-DOM snapshot, not proof that every listing was collected."""
    parsed = urlsplit(url)
    if (not authorized or parsed.scheme != "https"
            or parsed.hostname != expected_host or parsed.username
            or parsed.password or parsed.port not in (None, 443)):
        raise ValueError("Require authorization and an exact HTTPS hostname")
    if not 1 <= max_scrolls <= 10 or min_interval < 1:
        raise ValueError("Use bounded scrolling and the source's permitted interval")
    driver = webdriver.Chrome()
    try:
        driver.set_page_load_timeout(30)
        driver.get(url)
        def card_count(browser):
            current = urlsplit(browser.current_url)
            if current.scheme != "https" or current.hostname != expected_host:
                raise RuntimeError("Unexpected navigation; stop collection")
            return len(browser.find_elements(By.CSS_SELECTOR, card_selector))
        WebDriverWait(driver, 10).until(lambda browser: card_count(browser) > 0)
        reason = "scroll_limit_reached"
        for _ in range(max_scrolls):
            previous_count = card_count(driver)
            driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
            time.sleep(min_interval)
            try:
                WebDriverWait(driver, 10).until(
                    lambda browser: card_count(browser) > previous_count
                )
            except TimeoutException:
                reason = "no_new_cards_within_timeout"
                break
        card_count(driver)  # Recheck origin before taking the snapshot.
        return driver.page_source, reason
    finally:
        driver.quit()

# Define only: no browser is started until you explicitly call this function.
# Use a verified selector on an authorized page; virtualized lists need
# per-scroll record persistence, which this final-DOM example does not provide.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from urllib.parse import urlsplit
import time

def rendered_snapshot(url, expected_host, card_selector, *, authorized=False,
                      max_scrolls=5, min_interval=2):
    """Bounded final-DOM snapshot, not proof that every listing was collected."""
    parsed = urlsplit(url)
    if (not authorized or parsed.scheme != "https"
            or parsed.hostname != expected_host or parsed.username
            or parsed.password or parsed.port not in (None, 443)):
        raise ValueError("Require authorization and an exact HTTPS hostname")
    if not 1 <= max_scrolls <= 10 or min_interval < 1:
        raise ValueError("Use bounded scrolling and the source's permitted interval")
    driver = webdriver.Chrome()
    try:
        driver.set_page_load_timeout(30)
        driver.get(url)
        def card_count(browser):
            current = urlsplit(browser.current_url)
            if current.scheme != "https" or current.hostname != expected_host:
                raise RuntimeError("Unexpected navigation; stop collection")
            return len(browser.find_elements(By.CSS_SELECTOR, card_selector))
        WebDriverWait(driver, 10).until(lambda browser: card_count(browser) > 0)
        reason = "scroll_limit_reached"
        for _ in range(max_scrolls):
            previous_count = card_count(driver)
            driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
            time.sleep(min_interval)
            try:
                WebDriverWait(driver, 10).until(
                    lambda browser: card_count(browser) > previous_count
                )
            except TimeoutException:
                reason = "no_new_cards_within_timeout"
                break
        card_count(driver)  # Recheck origin before taking the snapshot.
        return driver.page_source, reason
    finally:
        driver.quit()

# Define only: no browser is started until you explicitly call this function.
# Use a verified selector on an authorized page; virtualized lists need
# per-scroll record persistence, which this final-DOM example does not provide.

Proxy Rotation for Large-Scale Crawling

Before scaling, measure the authorized sample’s error rate, missing fields and duplicate records. Increase workload only within the agreed limits. Proxy routing may serve legitimate network requirements, but rotation does not guarantee access or justify continuing after a refusal.

CAPTCHA Solutions

If a CAPTCHA or access challenge appears, stop the automated run and record the failure. Review the permitted access method with the source owner. Do not treat a CAPTCHA-solving service as evidence that collection is authorized or reliable.

7. Handling Legal and Technical Challenges

IP Ban Avoidance

Log HTTP status, response type and failed validation separately. Stop on a block, reduce load where required and retry only errors covered by an explicit bounded policy. Do not silently record an empty result as a successful category crawl. See our scraping failure and validation guide.

Understanding Shopee’s Policies

Check the terms for the particular Shopee market. For example, section 3.1 of Shopee Thailand’s Terms of Service requires prior written consent for automated monitoring or copying, with a stated search-engine exception. This regional example is not a blanket legal conclusion for every market. Prefer approved exports or interfaces that meet your requirements.

Data Protection and Compliance

Restrict access to collected records, avoid unnecessary personal fields, and define retention and deletion procedures. Aggregation or pseudonymization can reduce exposure but should not be described as guaranteed anonymization. Review the obligations relevant to the collection and intended reuse.

8. Best Practices for Data Management

Data Cleaning and Consistency

Deduplicate by a documented key such as market, shop, product, variant and observation time—not title alone. Keep missing values distinct from zero, preserve raw price text, and record transformations. Do not invent missing values merely to make a table complete.

Database Storage

For large datasets, use a database such as MySQL or MongoDB. Databases support quick access, complex queries, and are ideal for handling big data.

9. Leveraging the Data: Analysis and Visualization

Visualization Tools

Reporting tools such as Tableau and Power BI, or Python charts, can summarize the validated records. Label sample scope, currency, observation dates and missing-data counts alongside each chart; a chart cannot correct a biased or incomplete sample.

Advanced Analysis Techniques

With pandas and Matplotlib, compare equivalent observations across dates. Separate price changes from currency, variant, voucher and sample changes. Product listings and review counts alone do not establish transaction-level sales or customer preferences.

10. Conclusion: Responsible Crawling and Future Directions

A useful collection starts with a specific decision and a permitted source, then validates each record before analysis. This example gives you a reproducible two-record fixture, explicit failures and a CSV output. Adapting it to Shopee requires independently verified access, rendering and field mappings; the sample does not promise live availability or full-market coverage.

For a project brief, specify the market, allowed source, fields, variant rules, collection frequency, freshness requirements and delivery format. Our Shopee data service explains the collection scope to discuss, and the product extraction guide covers a separate structured-data approach. Before scheduling a larger collection, use the Shopee collection scope and coverage checks to define accepted records and report failed or incomplete observations separately.

Ready to start scraping Shopee data? Visit easydata.io.vn to learn more about our data scraping solutions and book a demo today!

BOOK A DEMO