Metrics and evaluation

Each run computes privacy metrics (how hard it is to re-identify someone) and signal metrics (how well the avatars keep the statistics of the original data). The definitions and the targets are in the public documentation.

The snippets on this page start from this setup:

import os

import pandas as pd

from avatars.manager import Manager

manager = Manager(api_key=os.environ["AVATAR_API_KEY"])
df = pd.read_csv("fixtures/wbcd.csv")

runner = manager.create_runner("wbcd_metrics")
runner.add_table("wbcd", df)
runner.advise_parameters("wbcd")

How to define the attack scenario

Tell which variables an attacker can plausibly know (known_variables) and which sensitive variable they want to find (target). Without them, the linkability and attribute inference metrics are not computed for this realistic scenario.

runner.update_parameters(
    "wbcd",
    known_variables=["Clump_Thickness", "Uniformity_of_Cell_Size"],
    target="Class",
)

In the web app: step Configure tables → Probably known.

How to adjust the closeness threshold

quantile_threshold sets the distance under which an avatar is considered “too close” to an original in the distance-based metrics (closest rate). It is a percentile of the reference distances; the default is 5.

runner.update_parameters("wbcd", quantile_threshold=5)

How to read the metrics

runner.run()

privacy = runner.privacy_metrics("wbcd")[0]
signal = runner.signal_metrics("wbcd")[0]

summary = pd.Series({**privacy, **signal}).drop("metadata", errors="ignore")
print(summary)

The targets depend on the context of the run (pia_data_recipient, …, see Getting started): they are stricter for open data than for an internal use. The technical report (see Plots and reports) shows each metric next to its target.

To find the variables that are reproduced the least well:

hellinger = signal.get("hellinger_dict")
if hellinger:
    print(pd.Series(hellinger).sort_values(ascending=False).head())

How to read the metrics of a multi-table run

For a multi-table run, get one summary line for each table:

runner.metrics_summary()  # DataFrame: privacy and signal meta-metrics for each table

runner.render_privacy_metrics_summary() and runner.render_signal_metrics_summary() return the same values as dictionaries.

How to evaluate an existing synthetic dataset

You can compute the metrics of a dataset generated elsewhere (with another tool, or an older version) without running an avatarization. Pass the synthetic data with avatar_data, set the parameters without k, and run only the metrics jobs.

from avatars.models import JobKind

evaluation = manager.create_runner("evaluate_existing_dataset")
evaluation.add_table(
    "iris",
    data="fixtures/iris.csv",
    avatar_data="fixtures/iris_avatarized.csv",  # same row order as the original data
)
evaluation.set_parameters("iris", known_variables=["sepal.length", "sepal.width"], target="variety")
evaluation.run(jobs_to_run=[JobKind.privacy_metrics, JobKind.signal_metrics])

evaluation.privacy_metrics("iris")

In the web app: use case Privacy Assessment.

Warning

The synthetic data must keep the row order of the original data: row i of the synthetic data must be generated from row i of the original data. Otherwise, the row-by-row metrics have no meaning.