Tabalyst
ENFR
Get started
ENFR

Tabalyst ToolsAvailable

Every repeated row, exactly.

Tabalyst Dedup finds the rows that occur more than once in a CSV file and writes the distinct rows, the repeated rows and every occurrence of them to new CSV files, with a JSON report whose counts always add up. The source is never modified.

Get startedTabalyst Dedup docs
ExactOne command, one report
tabalyst dedup customers.csv
Source rows:  6
Distinct:     3
Groups:       2
Extra copies: 3
Duplicate records found.
  customers.unique.csv
  customers.duplicates.csv
  customers.duplicates_all.csv
Report: customers.dedup.json

Real output of tabalyst 0.7.1, on a small example.

The tool

Exact or failed, never approximated

  • 01 / SAME ROW

    Every cell identical

    Two rows are the same only when every cell is identical: case, spaces and leading zeros count.

  • 02 / EXACT

    Complete or failed with a reason

    Nothing is ever approximated. The rows are kept in memory up to a budget, then on disk, and a file that cannot be processed fails with a reason.

  • 03 / SAFE

    The source is never modified

    Dedup reads CSV files only and asks no question. A file that is already processed and unchanged is skipped.

Get started

Find the duplicates

  1. 1 · Install
    pip install tabalyst

    # Also updates it: Tabalyst changes often

  2. 2 · One file
    tabalyst dedup customers.csv

    # Writes the CSV files and the report beside the source

  3. 3 · A batch
    tabalyst dedup "*.csv" -d deduped

    # Also writes dedup.summary.json in deduped/

Real output

What it writes

Beside the source, or in the folder given with -d, Dedup writes customers.unique.csv (every distinct row once, at its first place), customers.duplicates.csv (each repeated row once) and, when a row occurs three times or more, customers.duplicates_all.csv (every occurrence of the repeated rows). customers.dedup.json is the report. A file without duplicate gets its report only.

Scope, not hype

What to know before you rely on it

  • Dedup reads CSV files only.
  • A file may hold 10,000,000 records by default (--max-rows), and the memory budget is 512 MB by default (--memory-budget-mb).
  • Rows that differ by a single character, a space or a leading zero are different rows.

The exactness, memory and limits ↗ page gives the details.

Next step

Where to go next