Augmentation, differential privacy and pseudonymization

The snippets on this page start from this setup:

import os

import pandas as pd

from avatars.manager import Manager

manager = Manager(api_key=os.environ["AVATAR_API_KEY"])
df = pd.read_csv("fixtures/wbcd.csv")

runner = manager.create_runner("wbcd_processing")
runner.add_table("wbcd", df)
runner.advise_parameters("wbcd")

How to augment your dataset

Data augmentation generates more avatars than there are original rows, with the same statistical patterns. Use it for example to train a model on a larger dataset. Set data_augmentation_strategy to a factor:

runner.update_parameters("wbcd", data_augmentation_strategy=3.0)  # 3 times more rows
runner.run()
avatars_df = runner.shuffled("wbcd")

In the web app: Expert mode → Data Augmentation → Factor.

How to rebalance a minority class

from avatar_yaml.models.parameters import AugmentationStrategy

runner.update_parameters(
    "wbcd",
    data_augmentation_strategy=AugmentationStrategy.minority,
    data_augmentation_target_column="Class",
)
# minority     : the smallest class grows to the size of the second smallest class
# not_majority : all the classes grow to the size of the largest class

The PlotKind.CLASS_PROJECTION_2D plot shows the result (see Plots and reports).

In the web app: Expert mode → Imbalanced Data.

How to augment each modality with its own ratio

runner.update_parameters(
    "wbcd",
    data_augmentation_strategy={"2": 1.0, "4": 2.0},  # ratio for each modality of the target
    data_augmentation_target_column="Class",
)

How to augment and keep the original rows (sensitive)

By default, the original rows are anonymized too. With data_augmentation_should_anonymize_original_table=False, the real rows stay in the output, and only the new rows are synthetic.

runner.update_parameters(
    "wbcd",
    data_augmentation_strategy=2.0,
    data_augmentation_should_anonymize_original_table=False,
)

Warning

The output then contains real data: it is not anonymous. Use it only inside your organization, for example to enlarge a training set that does not leave it.

How to use differential privacy (FastDP)

FastDP applies differential privacy mechanisms in the latent space. A smaller epsilon gives a stronger protection and a lower utility. k and an epsilon cannot be used together.

# From scratch
runner.set_parameters("wbcd", fast_dp_epsilon=5)

# After advise_parameters (which sets a k), remove k explicitly
runner.update_parameters("wbcd", k=None, fast_dp_epsilon=5)

In the web app: use case Avatar DP Anonymization.

Warning

update_parameters("wbcd", fast_dp_epsilon=5) after advise_parameters raises ValueError: Expected either k or fast_dp_epsilon to be set, not both. Pass k=None.

How to use differential privacy (OpenDP, MST)

runner.set_parameters("wbcd", open_dp_epsilon=1.0)

The same rules apply: no k with open_dp_epsilon, and pass k=None with update_parameters.

How to pseudonymize PII columns (Beta)

⚠️ Beta Feature — Pseudonymization is currently in beta. The API and behavior may change in future releases.

Pseudonymization replaces direct identifiers (PII) with artificial values. It works with avatarization: the PII columns are set aside before the anonymization, are not modelled, and come back with their pseudonymized values in the output.

from avatars import (
    ConstantStrategy,
    FakeDataStrategy,
    HashSha256Strategy,
    PiiType,
    SpecificIdStrategy,
)

people = pd.read_csv("fixtures/pseudo_data.csv")

runner = manager.create_runner("pseudonymization_example")
runner.add_table("people", people)
runner.advise_parameters("people")
runner.update_parameters(
    "people",
    pseudonymized_columns={
        "first_name": FakeDataStrategy(pii_type=PiiType.FIRST_NAME),
        "last_name": FakeDataStrategy(pii_type=PiiType.LAST_NAME),
        "email": FakeDataStrategy(pii_type=PiiType.EMAIL, consistent=False),
        "ssn": SpecificIdStrategy(pattern="###-##-####"),
        "phone": HashSha256Strategy(),
        "address": ConstantStrategy(value="REDACTED"),
    },
)
runner.run()
result = runner.shuffled("people")

In the web app: step Configure tables → Transformation = Pseudonymization.

Available strategies

The strategies can be imported from avatars or from avatar_yaml.

  • FakeDataStrategy(pii_type=PiiType.EMAIL) — Generates realistic fake values. Requires a pii_type. By default (consistent=True), the same source value always gives the same fake value. Pass consistent=False for a new value on each row. high_variability=True generates a unique value for each row (slower, no collisions).

  • HashSha256Strategy() — Replaces values with their SHA-256 hex digest. Always deterministic: the same input gives the same hash in all runs. Good to link datasets; avoid it on low-entropy values (short codes, integers), which are open to dictionary attacks.

  • Uuid4Strategy() — Replaces each unique value with a random UUID v4. Pass consistent=False for a new UUID on each row.

  • ConstantStrategy(value="REDACTED") — Replaces every value with a fixed string. Use it when a column must be fully suppressed.

  • IntegerStrategy() — Maps each unique value to a unique pseudonymous integer in random order. Pass consistent=False for an independent integer on each row.

  • SpecificIdStrategy(pattern="EMP-####") — Generates structured IDs from a pattern. Placeholders: # (digit), ? (letter), ^ (alphanumeric), {{col}} (value of another column). Set the letter case with letter_case=SpecificIdLetterCase.UPPER, LOWER or BOTH.

Supported PII types

FakeDataStrategy needs a PiiType: EMAIL, FIRST_NAME, LAST_NAME, FULL_NAME, PHONE, SSN, ADDRESS, FREE_TEXT.

Pseudonymize identifiers in several tables

Pseudonymize the primary key of the parent table. The foreign keys that point to it in the child tables get the same mapping automatically: do not configure them.

from avatars import Uuid4Strategy

patient = pd.read_csv("fixtures/patient.csv")
visit = pd.read_csv("fixtures/visit.csv").drop(columns=["doctor_id"])

runner = manager.create_runner("pseudonymization_multitable")
runner.add_table("patient", patient, primary_key="p_id", individual_level=True)
runner.add_table(
    "visit", visit, primary_key="visit_id", foreign_keys=["patient_id"], individual_level=False
)
runner.add_link("patient", "p_id", "visit", "patient_id")
runner.advise_parameters()
runner.update_parameters("patient", pseudonymized_columns={"p_id": Uuid4Strategy()})
runner.run()