# Metrics and evaluation

Each run computes privacy metrics (how hard it is to re-identify someone) and
signal metrics (how well the avatars keep the statistics of the original
data). The definitions and the targets are in the
[public documentation](https://docs.octopize.io/docs/principles/metrics/).

The snippets on this page start from this setup:

```python
import os

import pandas as pd

from avatars.manager import Manager

manager = Manager(api_key=os.environ["AVATAR_API_KEY"])
df = pd.read_csv("fixtures/wbcd.csv")

runner = manager.create_runner("wbcd_metrics")
runner.add_table("wbcd", df)
runner.advise_parameters("wbcd")
```

## How to define the attack scenario

Tell which variables an attacker can plausibly know (`known_variables`) and
which sensitive variable they want to find (`target`). Without them, the
linkability and attribute inference metrics are not computed for this
realistic scenario.

```python
runner.update_parameters(
    "wbcd",
    known_variables=["Clump_Thickness", "Uniformity_of_Cell_Size"],
    target="Class",
)
```

**In the web app:** step **Configure tables → Probably known**.

## How to adjust the closeness threshold

`quantile_threshold` sets the distance under which an avatar is considered
"too close" to an original in the distance-based metrics (closest rate). It
is a percentile of the reference distances; the default is 5.

```python
runner.update_parameters("wbcd", quantile_threshold=5)
```

## How to read the metrics

```python
runner.run()

privacy = runner.privacy_metrics("wbcd")[0]
signal = runner.signal_metrics("wbcd")[0]

summary = pd.Series({**privacy, **signal}).drop("metadata", errors="ignore")
print(summary)
```

The targets depend on the context of the run (`pia_data_recipient`, ...,
see {doc}`getting_started`): they are stricter for open data than for an
internal use. The technical report (see {doc}`reporting`) shows each metric
next to its target.

To find the variables that are reproduced the least well:

```python
hellinger = signal.get("hellinger_dict")
if hellinger:
    print(pd.Series(hellinger).sort_values(ascending=False).head())
```

(multi-table-metrics)=

## How to read the metrics of a multi-table run

For a multi-table run, get one summary line for each table:

```python
runner.metrics_summary()  # DataFrame: privacy and signal meta-metrics for each table
```

`runner.render_privacy_metrics_summary()` and
`runner.render_signal_metrics_summary()` return the same values as
dictionaries.

(evaluate-existing-dataset)=

## How to evaluate an existing synthetic dataset

You can compute the metrics of a dataset generated elsewhere (with another
tool, or an older version) without running an avatarization. Pass the
synthetic data with `avatar_data`, set the parameters without `k`, and run
only the metrics jobs.

```python
from avatars.models import JobKind

evaluation = manager.create_runner("evaluate_existing_dataset")
evaluation.add_table(
    "iris",
    data="fixtures/iris.csv",
    avatar_data="fixtures/iris_avatarized.csv",  # same row order as the original data
)
evaluation.set_parameters("iris", known_variables=["sepal.length", "sepal.width"], target="variety")
evaluation.run(jobs_to_run=[JobKind.privacy_metrics, JobKind.signal_metrics])

evaluation.privacy_metrics("iris")
```

**In the web app:** use case **Privacy Assessment**.

:::{warning}
The synthetic data must keep the row order of the original data: row `i` of
the synthetic data must be generated from row `i` of the original data.
Otherwise, the row-by-row metrics have no meaning.
:::