Skip to content This library is free, offered as dana. If it helps your work, donate here →
penguin tree ai
← back to libraryDonate here

This is one of 3,622 free prompts in the penguin tree ai library. Sign up once and the whole library opens, with new prompts by email.

open the library →

Duplicate listing detector

Analyzes a set of product listings and flags likely duplicates and variant mismatches using weighted similarity across title, description, image signals, and structured attributes — producing a ranked match report with…

download .md

Purpose

Compare a set of product listings and surface likely duplicates and misgrouped variants — with a confidence score, the specific signals that drove the match, and a recommended action (merge, group as variants, or keep separate) for each cluster — so catalog teams stop shipping redundant SKUs and broken variant families to the storefront.

When to Use

Invoke this skill when the user asks to:

  • Find duplicate product listings in a catalog or product feed
  • Clean up redundant SKUs after a platform migration or catalog import
  • Deduplicate merged supplier or dropship feeds
  • Detect listings that should be grouped as variants but were created separately
  • Catch variant mismatches (same parent, wrong child attributes)
  • Audit a catalog before a marketplace launch or feed sync

Inputs Required

Before starting, collect:

  • Listing set: The products to compare — pasted rows, a table, or a structured list (SKU/ID + fields)
  • Fields available per listing: Which of title, description, image reference/URL, brand, price, and attributes are present
  • Attribute schema: The structured attributes that define a variant vs. a distinct product (e.g., size, color, capacity, pack count)
  • Variant policy: What differences make items variants of one product vs. genuinely separate products
  • Match sensitivity: Aggressive (flag more, review-heavy) / Balanced / Conservative (flag only near-certain matches)
  • Identifier fields: Any GTIN, UPC, EAN, MPN, or ASIN present (hard-match anchors)
  • Output scope: Full report on all clusters, or only actionable clusters above a confidence threshold

If variant policy is missing, default to: differences limited to size/color/pack are variants; differences in core function, brand, or model are separate products. If match sensitivity is missing, default to Balanced.

Instructions

Step 1: Normalize Every Listing

Before comparing, standardize each field to remove false differences:

  • Title: Lowercase; strip punctuation and duplicate whitespace; expand common abbreviations (pcs→piece, oz→ounce, blk→black); remove marketing filler (best, premium, new, sale, 2024)
  • Attributes: Map synonyms to canonical values (grey/gray, XL/extra large, 16oz/1lb); normalize units
  • Identifiers: Trim leading zeros and formatting from GTIN/UPC/EAN/MPN
  • Images: Note filename, dimensions, or URL stem if available; treat identical image references as a strong signal
  • Record any field that is missing per listing — missing data lowers confidence, it does not create a match

Step 2: Apply Hard-Match Anchors First

If two listings share the same normalized GTIN/UPC/EAN or same brand+MPN, treat them as a near-certain match regardless of title differences. These override text similarity. Flag conflicting identifiers on otherwise identical listings as a data integrity issue, not a duplicate.

Step 3: Score Similarity Across Weighted Signals

For each candidate pair (or cluster), compute a similarity contribution per signal. Use these default weights, adjustable by match sensitivity:

  • Identifier match (weight 40): Same GTIN/UPC/MPN = full weight. Absent = 0, not negative.
  • Title similarity (weight 25): Token overlap after normalization. Distinguish shared core tokens (product noun, brand, model) from incidental ones.
  • Attribute match (weight 20): Share of matching canonical attributes. Note which attributes differ.
  • Description similarity (weight 10): Phrase and spec overlap; ignore boilerplate shipping/returns text.
  • Image signal (weight 5): Identical or near-identical image reference.

Sum into a 0–100 confidence score. Sensitivity shifts the flag threshold: Aggressive ≥50, Balanced ≥65, Conservative ≥80.

Step 4: Classify Each Flagged Cluster

Assign every cluster above threshold to one type:

  • Exact duplicate: Same product, same variant — all defining attributes match. Recommend merge/delete redundant.
  • Variant mismatch: Same product, differ only on variant-policy attributes (size/color/pack). Recommend group under one parent.
  • Near-duplicate / ambiguous: High text similarity but a differing core attribute. Recommend manual review with the specific field to check.
  • False friend: High title overlap but different product (e.g., "iPhone case" vs "iPhone charger"). Recommend keep separate — surface so the operator understands why it was almost flagged.

Step 5: Produce the Match Report

For each cluster, output: the listing IDs, cluster type, confidence score, the specific signals that matched, the specific fields that differ, and one recommended action. Rank clusters by confidence descending. End with a summary count by cluster type.

Output Format

  • Structure: A ranked cluster report (table or structured block per cluster) followed by a summary tally
  • Length: Scales with input; 150–300 words of setup plus one entry per cluster
  • Format: Table for the cluster list (Cluster ID | Listing IDs | Type | Confidence | Key signals | Differing fields | Action); structured bullets for any data integrity flags
  • Sections: Normalization notes → Hard-match / integrity flags → Ranked duplicate & variant clusters → Summary tally by type
  • Tone: Catalog operations specialist — precise, evidence-first, no hedging on the recommended action

Quality Checklist

Before delivering output, verify:

  • [ ] Every flagged cluster has a numeric confidence score and a stated flag threshold
  • [ ] Each cluster names the specific signals that matched AND the specific fields that differ
  • [ ] Every cluster has exactly one recommended action (merge / group as variants / keep separate / review)
  • [ ] Identifier conflicts are reported as data integrity issues, not treated as duplicates
  • [ ] Variant mismatches are distinguished from exact duplicates per the stated variant policy
  • [ ] The summary tally counts match the number of clusters listed above it

Notes

  • Missing data lowers confidence but must never manufacture a match — an empty field is not a shared value.
  • "False friend" clusters are worth surfacing even though the action is keep-separate; they explain near-misses and build operator trust in the scoring.
  • For very large catalogs, run in batches by category or brand — cross-brand duplicates are rare and comparing them wastes signal budget.
  • When identifiers are present and clean, weight them heavily; text similarity alone is prone to grouping accessories with their base products.

How to use this skill

This file is a Claude Code skill in Anthropic's standard format — markdown with YAML frontmatter at the top.

Installing in Claude Code:

  • ~/.claude/skills/<folder-name>/SKILL.md — auto-loads in every session anywhere on your machine (recommended for skills you'll use across multiple projects)
  • <your-project>/.claude/skills/<folder-name>/SKILL.md — auto-loads only when working in that project (recommended for project-specific skills)

Restart Claude Code or open a new session to register.

Using in Claude.ai, ChatGPT, Gemini, or other AI tools:

  1. Drag this file into a new chat, or copy and paste its contents
  2. Send your task or request as your next message — the AI will apply the skill where it fits

For Claude Code skill details: https://docs.anthropic.com/en/docs/claude-code/skills