Set Notation for AI Datasets: Membership, Splits, and Leakage

Suppose an image classifier scores 96% on its validation data. Then someone notices that img-04 was used for training and validation. The score may look precise, but one of its examples was no longer unseen.

Sets give us a compact way to state and test that failure. A set records which objects belong to a collection. From there, intersection exposes overlap, union checks coverage, and a partition describes a clean assignment of every example to exactly one dataset split.

By the end of this lesson, you will be able to read membership and set-builder notation, calculate common set operations, and implement a first-line audit for training, validation, and test IDs. This advances the data side of the variables, models, and learning picture introduced in Lesson 1.

Membership turns a collection into a yes-or-no claim

A set is a collection whose membership is unambiguous. If $D$ is the set of all example IDs in a small dataset, we might write

The braces mean “the set containing these elements.” The statement

reads “img-02 is an element of $D$.” The symbol $\notin$ negates that claim, so $\text{img-09}\notin D$.

The number of distinct elements in a finite set is its cardinality, written $|D|$. In the example above, $|D|=4$.

Set-builder notation describes a rule instead of a long list

Listing every element becomes impractical for a real dataset. Set-builder notation defines membership by a condition:

Read this from left to right: “$D_{\text{cat}}$ is the set of examples $x$ in $D$ such that the label of $x$ is cat.” The colon means “such that.” A vertical bar, $|$, is also commonly used in the same position.

The condition after the colon acts like a filter. For

if img-01 and img-04 have the cat label, then

This notation separates the universe being considered—here, $D$—from the membership rule. Without $x\in D$, “all cat images” could mean every cat image that exists rather than the examples available to this project.

Three operations audit a dataset split

Let $T$, $V$, and $E$ denote the training, validation, and test ID sets. Three operations answer different engineering questions.

The empty set, written $\varnothing$, contains no elements. Therefore, $T\cap V=\varnothing$ states that training and validation share no IDs. Such sets are called disjoint.

A clean split is a partition

Suppose the complete dataset is $D$. Training, validation, and test sets form a partition of $D$ when two requirements hold.

First, their union covers the whole dataset:

Second, every pair is disjoint:

Coverage prevents an example from disappearing. Pairwise disjointness prevents an example from occupying two splits. Together they say every element of $D$ belongs to exactly one part.

The initial state covers all seven IDs, yet it fails the partition test because img-04 belongs to both $T$ and $V$. Uncheck either of those two memberships to produce a valid partition. Then try leaving an ID unchecked everywhere: the overlap is gone, but the coverage condition fails.

Python sets mirror the audit

Python’s official documentation defines its set type as an unordered collection of distinct hashable objects. It supports the same union, intersection, difference, membership, and disjointness operations used above.

all_ids = {f"img-{i:02d}" for i in range(1, 8)}
train_ids = {"img-01", "img-02", "img-03", "img-04"}
validation_ids = {"img-04", "img-05"}
test_ids = {"img-06", "img-07"}

overlap = (
    (train_ids & validation_ids)
    | (train_ids & test_ids)
    | (validation_ids & test_ids)
)
covered = train_ids | validation_ids | test_ids
missing = all_ids - covered

is_partition = not overlap and not missing and covered == all_ids

print(sorted(overlap))
print(sorted(missing))
print(is_partition)

The verified output is:

['img-04']
[]
False

The operator & computes intersection, | computes union, and - computes difference. not overlap is true only when the overlap set is empty.

For a reusable check, make the conditions explicit:

def is_partition(universe, *parts):
    covered = set().union(*parts)
    total_memberships = sum(len(part) for part in parts)
    no_overlap = total_memberships == len(covered)
    return no_overlap and covered == universe

If any ID occurs in two parts, the sum of the separate sizes exceeds the size of their union. If an ID is missing, the union differs from the universe.

Disjoint IDs are necessary, not sufficient, for leakage safety

The opening overlap is a form of data leakage: information from outside the training boundary influences model construction or evaluation. Scikit-learn’s leakage guidance explains that such leakage can make evaluation look overly optimistic and recommends separating data before fitting preprocessing steps.

An empty ID intersection catches only exact identifier reuse. It does not prove that the evaluation set is genuinely independent. Two different IDs may point to resized copies of the same photograph. Frames from one video may be split across training and validation. Records from the same patient, customer, or future time period may carry shared information even when their row IDs differ.

There is another boundary between mathematics and implementation: Python sets do not retain sequence positions and discard duplicate values. Keep the ordered sample list for training. Derive a set of stable IDs for membership checks.

Lesson 5 will add indices and subscripts, letting us distinguish an example’s identity from its position inside a batch or tensor. That distinction is much easier to state once the dataset itself has a clear membership boundary.

Check your understanding

no_overlap = not (
    (train & validation)
    or (train & test)
    or (validation & test)
)
covered = (train | validation | test) == all_ids
return no_overlap and covered