Avatarization parameters

The snippets on this page start from this setup:

import os

import pandas as pd

from avatars.manager import Manager

manager = Manager(api_key=os.environ["AVATAR_API_KEY"])
df = pd.read_csv("fixtures/wbcd.csv")

runner = manager.create_runner("wbcd_parameters")
runner.add_table("wbcd", df)
runner.advise_parameters("wbcd")

set_parameters or update_parameters?

  • update_parameters keeps the current parameters (for example the recommended ones) and changes only the parameters you pass.

  • set_parameters replaces all the parameters of the table, including the recommended ones.

runner.advise_parameters("wbcd")

# Keep the recommendation, change only k
runner.update_parameters("wbcd", k=20)

# Replace everything: ncp, imputation, ... from the recommendation are lost
runner.set_parameters("wbcd", k=20)

Warning

  • update_parameters raises ValueError: No existing parameters found if you did not call advise_parameters or set_parameters before.

  • set_parameters without k (or a DP epsilon) creates no avatarization parameters: only the metrics are configured. This is what you want to evaluate an existing synthetic dataset, not to anonymize.

To remove parameters, use runner.delete_parameters("wbcd").

How to set k (privacy / utility trade-off)

k is the number of neighbours used to generate each avatar. It is the parameter with the largest effect on the result. Change it step by step.

runner.update_parameters("wbcd", k=30)
# small k (e.g. 5)  -> avatars close to the originals: more utility, less privacy
# large k (e.g. 50) -> smoother avatars: more privacy, less utility

k=2 is the minimum and emits a warning. A k larger than the number of rows makes the job fail.

In the web app: step Configure job → k.

How to set ncp (number of components)

ncp is the number of dimensions of the projection used to find the neighbours.

runner.update_parameters("wbcd", ncp=5)
# low ncp  -> neighbours are found on the axes with the most variance
# high ncp -> more dimensions are used, including the "secondary" variables

In the web app: Expert mode → ncp.

How to give more weight to a variable

A larger weight separates the individuals more along this variable. Use it to preserve the variable of interest of your study better. The default weight is

runner.update_parameters("wbcd", column_weights={"Class": 3.0, "Bare_Nuclei": 2.0})

In the web app: step Configure tables → Column weight.

How to handle categorical variables with many modalities

runner.update_parameters("wbcd", use_categorical_reduction=True)

The categorical variables are encoded as numeric vectors before the projection. Use it when the total number of modalities is large (more than about 20). It can reduce the utility. See also How to group rare modalities.

In the web app: Expert mode → Categorical reduction for variables with many modalities.

How to keep a column out of anonymization

Choose by what you want in the output:

You want the column…

Use

removed from the output

columns_to_drop in add_table (Data handling)

in the output, not modelled, reattached by row

exclude_variable_names + ROW_ORDER (freeze)

in the output, reattached by similarity

exclude_variable_names + COORDINATE_SIMILARITY

in the output, replaced by fake values (PII)

pseudonymized_columns (Augmentation, differential privacy and pseudonymization)

Freeze a column (reattach by row order)

The column does not enter the projection. Its values are reattached to the avatars by row order.

from avatar_yaml.models.parameters import ExcludeVariablesMethod

runner.update_parameters(
    "wbcd",
    exclude_variable_names=["Mitoses"],
    exclude_variable_method=ExcludeVariablesMethod.ROW_ORDER,  # default
)

In the web app: step Configure tables → Transformation = Freeze.

Exclude a column and reattach it by similarity

Recommended for a categorical variable with very many modalities: the variable is excluded from the projection, then each avatar gets the value of its closest original.

from avatar_yaml.models.parameters import ExcludeVariablesMethod

runner.update_parameters(
    "wbcd",
    exclude_variable_names=["Mitoses"],
    exclude_variable_method=ExcludeVariablesMethod.COORDINATE_SIMILARITY,
    use_excluded_variables_in_metrics=True,  # measure the risk of this variable too
)

Warning

  • By default, an excluded variable is also excluded from the metrics: its risk is not measured. Set use_excluded_variables_in_metrics=True to measure it.

  • ROW_ORDER keeps more identifying information than COORDINATE_SIMILARITY.

  • Do not exclude primary keys or foreign keys. If the column contains PII, pseudonymize it instead.

How to choose the imputation of missing values

Missing values are modelled: the server adds an indicator, fills the gaps for the computation, then puts the missing values back in the avatars, in the same proportions as in the original data.

from avatar_yaml.models.parameters import ImputeMethod

runner.update_parameters(
    "wbcd",
    imputation_method=ImputeMethod.KNN,  # FAST_KNN, KNN, MEAN, MEDIAN, MODE
    imputation_k=5,  # neighbours for KNN and FAST_KNN
    imputation_training_fraction=0.5,  # fraction of the data used to fit the imputation
)

In the web app: Expert mode → Imputation for missing data.

How to get avatars without missing values

By default, the avatars have missing values in the same proportions as the original data. With imputation_return_data_imputed=True, you get complete avatars.

from avatar_yaml.models.parameters import ImputeMethod

runner.update_parameters(
    "wbcd",
    imputation_method=ImputeMethod.FAST_KNN,  # required, see the warning below
    imputation_return_data_imputed=True,
)

In the web app: Expert mode → Return imputed data.

Warning

imputation_return_data_imputed=True is ignored, with no error, when no imputation parameter is set (for example set_parameters("wbcd", k=10, imputation_return_data_imputed=True)). Always pass imputation_method with it. Check the result with runner.print_parameters("wbcd").

How to reproduce a run exactly

The same seed, the same data and the same parameters give the same avatars. Set the seed when you create the runner:

runner = manager.create_runner("wbcd_seed_42", seed=42)
runner.add_table("wbcd", df)
runner.advise_parameters("wbcd")
runner.run()

To reuse the exact configuration of a past run, export it as YAML (see Reuse past runs and configurations).