# Data handling

The snippets on this page start from a `manager` (see {doc}`getting_started`):

```python
import os

import pandas as pd

from avatars.manager import Manager

manager = Manager(api_key=os.environ["AVATAR_API_KEY"])
```

## How to add a table from a DataFrame or a file

`runner.add_table()` accepts a pandas DataFrame or the path of a `.csv` or
`.parquet` file.

```python
runner = manager.create_runner("wbcd_upload")

# From a file on disk: the server detects the separator and the encoding
runner.add_table("wbcd", "fixtures/wbcd.csv")

# Or from a DataFrame: the column types come from the pandas dtypes
# df = pd.read_csv("fixtures/wbcd.csv")
# runner.add_table("wbcd", df)
```

Use a DataFrame when you want to inspect, validate or preprocess the data
before you send it (for example with a {doc}`client-side processor <processors>`).
Otherwise, a file path is simpler.

**In the web app:** step **Load datasets**.

:::{note}
A DataFrame is uploaded as Parquet (`wbcd.parquet`); a file keeps its
extension (`wbcd.csv`). This matters when you replay a YAML configuration:
see {doc}`advanced`.
:::

## How to reuse a file already in my storage

A file uploaded once stays in your storage until its retention window ends.
`manager.list_input_files()` lists these files, newest first. Give one to
`runner.add_table()` in place of the data: the table reads it from the storage and
nothing is uploaded again.

An upload is stored under the table name: `runner.add_table("wbcd", df)` writes
`wbcd.parquet`, and a path keeps its extension. A new upload under a name already in
your storage overwrites the stored file. Reuse the stored file for the same data;
upload again to replace it.

```python
files = manager.list_input_files()
for file in files.values():
    print(file.name, file.size, file.uploaded_at, file.expires_at)

runner = manager.create_runner("wbcd_reuse")
runner.add_table("wbcd", files["wbcd.csv"])
```

When you know the name of the file, `InputFile("wbcd.csv")` names it without a listing.
Either way, `add_table()` checks that the file is in your storage, and a name that is
not there raises a `ValueError` that lists the stored files.
A stored file gives no column types to the client: declare them with `types=` or let
`runner.advise_parameters()` detect them, as for a file path.

**In the web app:** step **Load datasets**, source **My storage**.

## How to change the type of a column

Force a type with `types=` when the inferred type is not the right one. For
example, a numeric variable with a few distinct values is often better
modelled as a category, and an identifier made only of digits must not be
read as a number.

```python
from avatar_yaml.models.schema import ColumnType

runner = manager.create_runner("wbcd_types")
runner.add_table(
    "wbcd",
    "fixtures/wbcd.csv",
    types={"Clump_Thickness": ColumnType.CATEGORY},
)
# Available types: INT, NUMERIC, CATEGORY, BOOL, DATETIME, DATETIME_TZ
```

You can also change the dtype in pandas before you add the table:

```python
df = pd.read_csv("fixtures/wbcd.csv")
df["Clump_Thickness"] = df["Clump_Thickness"].astype("string")
runner.add_table("wbcd", df)
```

Dates in ISO 8601 format are detected automatically.

**In the web app:** step **Configure tables**.

## How to drop a column

Use `columns_to_drop` to remove a column before the avatarization. The
column is not used and is not in the avatar table.

```python
runner = manager.create_runner("wbcd_drop")
runner.add_table("wbcd", "fixtures/wbcd.csv", columns_to_drop=["Mitoses"])
```

**In the web app:** step **Configure tables → Transformation = Drop**.

To keep a column in the output without modelling it, see
{ref}`keep-a-column-out`.

## How to see what the server detected in my data

After `advise_parameters`, `table_summary` shows the types, the number of
modalities and the missing values that the server detected:

```python
runner = manager.create_runner("wbcd_summary")
runner.add_table("wbcd", "fixtures/wbcd.csv")
runner.advise_parameters("wbcd")
runner.table_summary("wbcd")
```

## How to handle a large dataset

The server limits the number of rows and the number of dimensions.

### Many rows

A large dataset is split into chunks automatically. Each chunk is
anonymized on its own, then the chunks are merged back. You do not need to
change your code.

### Many dimensions

The number of dimensions is the number of continuous variables plus the
number of modalities of the categorical variables. Categorical variables
with many modalities (high cardinality) are the usual cause. You can:

- set `use_categorical_reduction=True`: the categorical variables are encoded
  as numeric vectors (cat2vec) before the projection. This can reduce the
  utility. See {doc}`parameters`;
- group the rare modalities with the
  {ref}`GroupModalitiesProcessor <group-rare-modalities>`;
- exclude the variable and reattach it by similarity
  ({ref}`keep-a-column-out`);
- encode the variable as a continuous variable yourself (frequency or target
  encoding).

## How to anonymize several linked tables (multi-table)

Declare the primary key and the foreign keys of each table, then link each
child table to its parent.

- Set `individual_level=True` on a table where one row is one individual
  (a patient, a doctor), and `individual_level=False` on a table where an
  individual can have several rows (the visits of a patient). At least one
  table must be at the individual level.
- A child table declares one foreign key for each parent.

```python
patient = pd.read_csv("fixtures/patient.csv")
doctor = pd.read_csv("fixtures/doctor.csv")
visit = pd.read_csv("fixtures/visit.csv")

runner = manager.create_runner("multitable_example")
runner.add_table("patient", patient, primary_key="p_id", individual_level=True)
runner.add_table("doctor", doctor, primary_key="d_id", individual_level=True)
runner.add_table(
    "visit",
    visit,
    primary_key="visit_id",
    foreign_keys=["patient_id", "doctor_id"],
    individual_level=False,
)

# Let the server propose the links from the declared keys
links_advice = runner.advise_links()
for link in links_advice.links:
    print(f"{link.parent_table} -> {link.child_table} ({link.confidence})")
for key in links_advice.unlinked_foreign_keys:
    print(f"Not linked: {key.table}.{key.field}, candidates: {key.candidate_tables}")

runner.advise_parameters()  # all the tables
runner.run()
```

To set a link yourself, or to change its method, use `add_link` (and
`delete_link` to replace an advised link):

```python
from avatar_yaml.models.schema import LinkMethod

runner.delete_link(parent_table_name="doctor", child_table_name="visit")
runner.add_link(
    parent_table_name="doctor",
    parent_field="d_id",
    child_table_name="visit",
    child_field="doctor_id",
    method=LinkMethod.MINIMUM_DISTANCE_ASSIGNMENT,
)
```

Link methods:

- `LINEAR_SUM_ASSIGNMENT` (default): good privacy and utility trade-off, uses
  more resources.
- `MINIMUM_DISTANCE_ASSIGNMENT`: acceptable trade-off, uses fewer resources.
  Use it for large datasets.
- `TIME_SERIES`: for time series tables (see below).

**In the web app:** step **Link datasets** (shown when there are two tables or
more).

:::{warning}
Do not use `LinkMethod.SENSITIVE_ORIGINAL_ORDER_ASSIGNMENT`: it keeps the
original order of the rows, which is a privacy leak.
:::

For multi-table metrics, see {ref}`multi-table-metrics`.

## How to anonymize time series

A time series is a child table linked to its individuals with
`LinkMethod.TIME_SERIES`. Give the time column with `time_series_time`.

```python
from avatar_yaml.models.parameters import ProjectionType
from avatar_yaml.models.schema import LinkMethod

individuals = pd.read_csv("fixtures/sensors_vanilla.csv")
sensor = pd.read_csv("fixtures/sensors_timeseries1.csv")

runner = manager.create_runner("time_series_example")
runner.add_table("individuals", individuals, primary_key="id", individual_level=True)
runner.add_table(
    "sensor", sensor, primary_key="primary_key", foreign_keys=["id"], time_series_time="t"
)
runner.add_link("individuals", "id", "sensor", "id", method=LinkMethod.TIME_SERIES)

runner.set_parameters("individuals", k=5)
runner.set_parameters(
    "sensor",
    time_series_projection_type=ProjectionType.FLATTEN,  # or ProjectionType.FPCA (default)
    time_series_nf=10,  # number of components kept for each series
)
runner.run()
```

Other time series options: `time_series_nb_points` and
`time_series_method=AlignmentMethod.MEAN | MAX | MIN | SPECIFIED`
(`from avatar_yaml.models.parameters import AlignmentMethod`).