Data handling

The snippets on this page start from a manager (see Getting started):

import os

import pandas as pd

from avatars.manager import Manager

manager = Manager(api_key=os.environ["AVATAR_API_KEY"])

How to add a table from a DataFrame or a file

runner.add_table() accepts a pandas DataFrame or the path of a .csv or .parquet file.

runner = manager.create_runner("wbcd_upload")

# From a file on disk: the server detects the separator and the encoding
runner.add_table("wbcd", "fixtures/wbcd.csv")

# Or from a DataFrame: the column types come from the pandas dtypes
# df = pd.read_csv("fixtures/wbcd.csv")
# runner.add_table("wbcd", df)

Use a DataFrame when you want to inspect, validate or preprocess the data before you send it (for example with a client-side processor). Otherwise, a file path is simpler.

In the web app: step Load datasets.

Note

A DataFrame is uploaded as Parquet (wbcd.parquet); a file keeps its extension (wbcd.csv). This matters when you replay a YAML configuration: see Reuse past runs and configurations.

How to reuse a file already in my storage

A file uploaded once stays in your storage until its retention window ends. manager.list_input_files() lists these files, newest first. Give one to runner.add_table() in place of the data: the table reads it from the storage and nothing is uploaded again.

An upload is stored under the table name: runner.add_table("wbcd", df) writes wbcd.parquet, and a path keeps its extension. A new upload under a name already in your storage overwrites the stored file. Reuse the stored file for the same data; upload again to replace it.

files = manager.list_input_files()
for file in files.values():
    print(file.name, file.size, file.uploaded_at, file.expires_at)

runner = manager.create_runner("wbcd_reuse")
runner.add_table("wbcd", files["wbcd.csv"])

When you know the name of the file, InputFile("wbcd.csv") names it without a listing. Either way, add_table() checks that the file is in your storage, and a name that is not there raises a ValueError that lists the stored files. A stored file gives no column types to the client: declare them with types= or let runner.advise_parameters() detect them, as for a file path.

In the web app: step Load datasets, source My storage.

How to change the type of a column

Force a type with types= when the inferred type is not the right one. For example, a numeric variable with a few distinct values is often better modelled as a category, and an identifier made only of digits must not be read as a number.

from avatar_yaml.models.schema import ColumnType

runner = manager.create_runner("wbcd_types")
runner.add_table(
    "wbcd",
    "fixtures/wbcd.csv",
    types={"Clump_Thickness": ColumnType.CATEGORY},
)
# Available types: INT, NUMERIC, CATEGORY, BOOL, DATETIME, DATETIME_TZ

You can also change the dtype in pandas before you add the table:

df = pd.read_csv("fixtures/wbcd.csv")
df["Clump_Thickness"] = df["Clump_Thickness"].astype("string")
runner.add_table("wbcd", df)

Dates in ISO 8601 format are detected automatically.

In the web app: step Configure tables.

How to drop a column

Use columns_to_drop to remove a column before the avatarization. The column is not used and is not in the avatar table.

runner = manager.create_runner("wbcd_drop")
runner.add_table("wbcd", "fixtures/wbcd.csv", columns_to_drop=["Mitoses"])

In the web app: step Configure tables → Transformation = Drop.

To keep a column in the output without modelling it, see How to keep a column out of anonymization.

How to see what the server detected in my data

After advise_parameters, table_summary shows the types, the number of modalities and the missing values that the server detected:

runner = manager.create_runner("wbcd_summary")
runner.add_table("wbcd", "fixtures/wbcd.csv")
runner.advise_parameters("wbcd")
runner.table_summary("wbcd")

How to handle a large dataset

The server limits the number of rows and the number of dimensions.

Many rows

A large dataset is split into chunks automatically. Each chunk is anonymized on its own, then the chunks are merged back. You do not need to change your code.

Many dimensions

The number of dimensions is the number of continuous variables plus the number of modalities of the categorical variables. Categorical variables with many modalities (high cardinality) are the usual cause. You can:

How to anonymize several linked tables (multi-table)

Declare the primary key and the foreign keys of each table, then link each child table to its parent.

  • Set individual_level=True on a table where one row is one individual (a patient, a doctor), and individual_level=False on a table where an individual can have several rows (the visits of a patient). At least one table must be at the individual level.

  • A child table declares one foreign key for each parent.

patient = pd.read_csv("fixtures/patient.csv")
doctor = pd.read_csv("fixtures/doctor.csv")
visit = pd.read_csv("fixtures/visit.csv")

runner = manager.create_runner("multitable_example")
runner.add_table("patient", patient, primary_key="p_id", individual_level=True)
runner.add_table("doctor", doctor, primary_key="d_id", individual_level=True)
runner.add_table(
    "visit",
    visit,
    primary_key="visit_id",
    foreign_keys=["patient_id", "doctor_id"],
    individual_level=False,
)

# Let the server propose the links from the declared keys
links_advice = runner.advise_links()
for link in links_advice.links:
    print(f"{link.parent_table} -> {link.child_table} ({link.confidence})")
for key in links_advice.unlinked_foreign_keys:
    print(f"Not linked: {key.table}.{key.field}, candidates: {key.candidate_tables}")

runner.advise_parameters()  # all the tables
runner.run()

To set a link yourself, or to change its method, use add_link (and delete_link to replace an advised link):

from avatar_yaml.models.schema import LinkMethod

runner.delete_link(parent_table_name="doctor", child_table_name="visit")
runner.add_link(
    parent_table_name="doctor",
    parent_field="d_id",
    child_table_name="visit",
    child_field="doctor_id",
    method=LinkMethod.MINIMUM_DISTANCE_ASSIGNMENT,
)

Link methods:

  • LINEAR_SUM_ASSIGNMENT (default): good privacy and utility trade-off, uses more resources.

  • MINIMUM_DISTANCE_ASSIGNMENT: acceptable trade-off, uses fewer resources. Use it for large datasets.

  • TIME_SERIES: for time series tables (see below).

In the web app: step Link datasets (shown when there are two tables or more).

Warning

Do not use LinkMethod.SENSITIVE_ORIGINAL_ORDER_ASSIGNMENT: it keeps the original order of the rows, which is a privacy leak.

For multi-table metrics, see How to read the metrics of a multi-table run.

How to anonymize time series

A time series is a child table linked to its individuals with LinkMethod.TIME_SERIES. Give the time column with time_series_time.

from avatar_yaml.models.parameters import ProjectionType
from avatar_yaml.models.schema import LinkMethod

individuals = pd.read_csv("fixtures/sensors_vanilla.csv")
sensor = pd.read_csv("fixtures/sensors_timeseries1.csv")

runner = manager.create_runner("time_series_example")
runner.add_table("individuals", individuals, primary_key="id", individual_level=True)
runner.add_table(
    "sensor", sensor, primary_key="primary_key", foreign_keys=["id"], time_series_time="t"
)
runner.add_link("individuals", "id", "sensor", "id", method=LinkMethod.TIME_SERIES)

runner.set_parameters("individuals", k=5)
runner.set_parameters(
    "sensor",
    time_series_projection_type=ProjectionType.FLATTEN,  # or ProjectionType.FPCA (default)
    time_series_nf=10,  # number of components kept for each series
)
runner.run()

Other time series options: time_series_nb_points and time_series_method=AlignmentMethod.MEAN | MAX | MIN | SPECIFIED (from avatar_yaml.models.parameters import AlignmentMethod).