Web scraping

Keep prices, listings and content from being harvested.

10 minutes

Scrapers copy prices, listings and content at machine speed. Spot the automated browsers and the visitors reading far more than anyone does.

What you will build: a check on the action you protect that allows normal users, asks doubtful ones to verify, and blocks the clear cases. See the web scraping overview for the attack in pictures.

Before you start

Finish one Web quickstart (for example JavaScript or React) and one Server quickstart (for example Node.js). The decision code below is TypeScript; every server SDK returns the same event.

How the attack works

  • Price monitoring. Competitors pulling every price, every hour.
  • Listing harvests. Whole catalogues copied in one night.
  • Hosted fleets. Headless browsers on cloud servers.

What to look at

FieldWhy it matters here
botHeadless and automated browsers.
velocityEvents per visitor and visitors per IP.
datacenterIpTraffic from cloud and hosting ranges.
  1. 1

    Identify on the action

    Call the browser SDK when the user performs the action, not on page load, and send the eventId with the request. Pass your own user id as linkedId whenever you have one: the account signals count by it.

    TypeScript
    const { eventId } = await trace.get({ linkedId: user?.id, tags: { action: 'web-scraping' } });
  2. 2

    Verify on your server

    import { verifyEvent } from '@trace-id/sdk-node';
    
    const event = await verifyEvent(eventId, { secretKey: process.env.TRACE_SECRET_KEY!, endpoint, timeoutMs: 1500 }).catch(() => null);

    Events can be verified for 60 minutes. If verification fails, decide whether to fail open or closed; for most of this flow, fail open.

  3. 3

    Decide

    TypeScript
    const { signals } = await trace.verify(eventId);
    
    if (signals.bot.detected) return block();
    if (signals.velocity.detected || signals.datacenterIp.detected) return slowDown();
    return renderPage();

    trace.verify() is the embedded call; with a server SDK, use the event from step 2 instead. The thresholds are yours. Trace scores and explains; event.reasons lists which signals added how much, so log it with every decision.

Example decisions

Product page views, one hour, illustrative:

VisitorPagesDecisionWhy
Xk3p…9Wc14 pagesAllowNormal browsing
Qp8w…kYa60 pagesAllowHeavy, but a real browser
Mn2r…e8Q900 pagesVerifyvelocity + datacenterIp
Zt7c…1Lb4,000 pagesBlockbot + highActivity

Roll it out

  1. Ship steps 1 to 3 with logging only: record the decision, block nothing.
  2. After a week, look at who would have been blocked or asked to verify. Shared computers, offices and privacy tools are the usual false positives.
  3. Turn on Verify first, then Block for the clearest cases.
Suggest an edit

Spotted something wrong or unclear? Tell us on GitHub and we will fix it.