Data Transformations: Normalization, Discretization, Sampling & Compression
GATE 2027 DA core: data transformation
“Choose, calculate, and validate the transformation named in a data-preparation problem.”
1.Prerequisites and the transformation pipeline
§ 9.1.Prerequisites: relational attributes and basic descriptive statistics. See [Relational Algebra and Calculus](/notes/dbms/relational-algebra-and-calculus) and [Probability and Statistics](/notes/probability-and-statistics/combinatorics-probability-axioms-and-independence).
A data transformation maps a source representation into one that supports analysis while controlling which information is retained. First identify the task: numerical scale normalization, discretization into categories, sampling rows, or compression of a representation. Record the input domain, output domain, fitted parameters, and any information loss.
Normalization here means feature scaling, such as min-max scaling or z-score standardization. It is not relational database normalization into 1NF/2NF/3NF, and it does not mean statistical normalization to a probability distribution unless the question says so. A constant feature has no min-max range; a zero standard deviation makes z-scores undefined, so the pipeline needs a declared policy.
For train/test evaluation, estimate minima, maxima, means, standard deviations, and bin boundaries from the training split and apply those frozen values to validation/test records. Fitting on all data leaks information about held-out observations.
Choose the operation by its goal
| Feature / Aspect | Transformation | What it does |
|---|---|---|
| Normalization / scaling | Rescales numeric values | Target interval or mean/standard deviation; training-only fit |
| Discretization | Maps numeric values to bins/categories | Equal-width, equal-frequency, or domain thresholds |
| Sampling | Selects rows for estimation or computation | Replacement, sampling frame, and design |
| Compression | Encodes data with fewer bits/bytes | Lossless vs lossy and codec assumptions |
Worked calculations across the four transformations
Using min=10 and max=30, x'=(20-10)/(30-10)=10/20=0.5.
Bins [0,10), [10,20), [20,30), … place 24 in [20,30). State the endpoint convention because values on a boundary can otherwise be ambiguous.
A simple random sample of 2 distinct rows from 10 has C(10,2)=45 possible unordered samples; each row has inclusion probability 2/10=0.2.
Compression ratio = 800/200 = 4:1. If compression is lossless, decompression recovers all 800 original bytes.
4.Traps to catch
§ 9.4.Relational normal forms constrain redundancy and dependencies; feature scaling changes numeric units.
Fit parameters on training data, then reuse them unchanged.
Lossy compression deliberately discards detail to reduce size.
Practice: choose and calculate
Training minimum and maximum are 5 and 25. Under min-max scaling to [0,1], what is the transformed value of 15?
A constant feature has min=max=7. What is the ordinary min-max formula's denominator?
A lossless codec reduces a 1200-byte file to 300 bytes. What is the compression ratio?
One-minute revision
“Match the transformation to the goal; preserve assumptions and account for information loss and leakage.”
- •Min-max: a+(x-min)(b-a)/(max-min)
- •Z-score: (x-μ)/σ
- •Compression ratio: original/compressed size
- •Simple random sample inclusion probability: n/N
- •Zero range or zero variance
- •Training/test leakage
- •Wrong bin boundary convention
- •With versus without replacement
- •Lossy versus lossless
- •Database normal forms versus feature scaling
Primary Academic References & Exam Sources
Every concept in Notes From a B.Tech Brain is synthesized from authoritative academic texts, seminal peer-reviewed research, verified exam archives, and renowned university lectures.
Curated for clarity, rigorous numerical solving, and semester excellence:
GATE-focused numerical problem-solving with clear worked examples.
Fast conceptual coverage with step-by-step GATE numericals.
Systematic textbook-style explanations ideal for first-time learners.
Deep theoretical rigor on normalization and transaction serializability.