# Processors

Processors transform the data to keep a business rule or to help the
modelling. There are two kinds:

- **Server-side processors** are parameters: you pass them with
  `processors=[...]` in `set_parameters` / `update_parameters`. The server
  applies them before and after the avatarization. The output has the same
  columns as the input. When you pass several processors, their order in the
  list is the order of the preprocessing (reversed for the postprocessing).
- **Client-side processors** run on your machine, from `avatars.processors`:
  call `preprocess` on the original data before `add_table`, then
  `postprocess` on the avatars. The metrics are computed on the preprocessed
  data.

The snippets on this page start from this setup:

```python
import os

import pandas as pd

from avatars.manager import Manager

manager = Manager(api_key=os.environ["AVATAR_API_KEY"])
df = pd.read_csv("fixtures/wbcd.csv")
```

## Server-side processors

### How to keep a total equal to the sum of its parts

Use `RelativeDifferenceParameters` when a variable is the sum (or an upper
bound) of other variables: total hours and hours per activity, total revenue
and revenue per category, ...

```python
from avatar_yaml.models.parameters import RelativeDifferenceParameters

players = pd.DataFrame(
    {
        "minutes_forward": [300, 120, 0, 50],
        "minutes_midfielder": [200, 400, 100, 0],
        "minutes_defender": [0, 80, 600, 450],
        "total_minutes": [500, 600, 700, 500],
    }
)

runner = manager.create_runner("relative_difference_example")
runner.add_table("players", players)
runner.set_parameters(
    "players",
    k=2,
    processors=[
        RelativeDifferenceParameters(
            target="total_minutes",
            references=["minutes_forward", "minutes_midfielder", "minutes_defender"],
            # scaling_unit=100,  # allow a small tolerance for rounding
        )
    ],
)
runner.run()
```

The same processor keeps a constraint `A ≤ B` (for example a number of
visits lower than or equal to a number of documents): use `B` as the target
and `A` as the reference.

### How to keep start and end dates coherent across records

`InterRecordRangeDifferenceParameters` keeps `start < end` and the sequence
of the periods of the same individual (stays of a patient, contracts of an
employee, phases of a project).

```python
from avatar_yaml.models.parameters import InterRecordRangeDifferenceParameters

stays = pd.DataFrame(
    {
        "patient_id": [1, 1, 2, 2, 3, 3],
        "start": [0, 10, 3, 20, 5, 9],
        "end": [5, 12, 9, 25, 7, 15],
        "age": [60, 60, 45, 45, 52, 52],
    }
)

runner = manager.create_runner("range_difference_example")
runner.add_table("stays", stays)
runner.set_parameters(
    "stays",
    k=2,
    processors=[
        InterRecordRangeDifferenceParameters(
            id_variable="patient_id",  # rows of the same individual
            target_start_variable="start",
            target_end_variable="end",
        )
    ],
)
runner.run()
```

## Client-side processors

(group-rare-modalities)=

### How to group rare modalities

`GroupModalitiesProcessor` merges the rare modalities of a categorical
variable into one modality. It reduces the number of dimensions. It cannot be
undone, so there is no `postprocess`.

```python
from avatars.processors import GroupModalitiesProcessor

adult = pd.read_csv("fixtures/adult_with_missing.csv")

# In the variables with 10 modalities or more, merge the modalities with
# fewer than 25 individuals into "other"
processor = GroupModalitiesProcessor(min_unique=10, global_threshold=25, new_category="other")
adult_pre = processor.preprocess(adult)

runner = manager.create_runner("group_modalities_example")
runner.add_table("adult", adult_pre)
```

You can also give a threshold for each variable with
`variable_thresholds={"native-country": 50}`.

### How to keep a discrete numeric variable discrete

`ToCategoricalProcessor` models a numeric variable with few distinct values
as a category. The avatars then have only values that exist (no 2.4
children, no score of 3.7 on an integer scale).

```python
from avatars.processors import ToCategoricalProcessor

processor = ToCategoricalProcessor(to_categorical_threshold=20)  # <= 20 values -> category
df_pre = processor.preprocess(df)

runner = manager.create_runner("to_categorical_example")
runner.add_table("wbcd", df_pre)
runner.advise_parameters("wbcd")
runner.run()

avatars_df = processor.postprocess(df, runner.shuffled("wbcd"))  # back to numeric
```

### How to keep parts that sum to a total

`ProportionProcessor` expresses variables as proportions of a reference
variable, so that the parts still sum to the total in the avatars.

```python
from avatars.processors import ProportionProcessor

budget = pd.DataFrame(
    {"total": [10, 20, 30], "a": [5, 5, 10], "b": [3, 10, 10], "c": [2, 5, 10]}
)
processor = ProportionProcessor(
    variable_names=["a", "b", "c"], reference="total", sum_to_one=True, decimal_count=0
)
budget_pre = processor.preprocess(budget)
# runner.add_table("budget", budget_pre) ... runner.run() ...
# budget_avatars = processor.postprocess(budget, runner.shuffled("budget"))
```

### How to convert dates myself

Dates in ISO 8601 format are detected automatically. Use
`DatetimeProcessor` only when you want to control the conversion yourself:
it converts the dates to numeric timestamps, and back.

```python
from avatars.processors import DatetimeProcessor

events = pd.DataFrame(
    {"admission": pd.to_datetime(["2023-01-02", "2023-05-10", "2023-09-30"])}
)
processor = DatetimeProcessor()
events_pre = processor.preprocess(events)  # dates -> timestamps
events_post = processor.postprocess(events, events_pre)  # timestamps -> dates
```

### How to keep the mean and the standard deviation

`ExpectedMeanProcessor` moves the avatar values so that their mean (and
optionally their standard deviation) matches the original data, globally or
by group.

```python
from avatars.processors import ExpectedMeanProcessor

processor = ExpectedMeanProcessor(
    target_variables=["Bare_Nuclei"], groupby_variables=["Class"], same_std=True
)
df_pre = processor.preprocess(df)

runner = manager.create_runner("expected_mean_example")
runner.add_table("wbcd", df_pre)
runner.advise_parameters("wbcd")
runner.run()

avatars_df = processor.postprocess(df, runner.shuffled("wbcd"))
```

:::{warning}
This improves the univariate statistics, sometimes at the cost of the
multivariate relations and of privacy. Compute the metrics again on the
postprocessed avatars.
:::

### How to keep a cumulative value increasing

`InterRecordCumulatedDifferenceProcessor` makes sure that a cumulative
value never decreases for an individual. Do not use it when the identifier or
the target variable has missing values.

```python
from avatars.processors import InterRecordCumulatedDifferenceProcessor

visits = pd.DataFrame({"pid": [1, 1, 1, 2, 2], "cumulated_docs": [2, 5, 9, 1, 4]})
processor = InterRecordCumulatedDifferenceProcessor(
    id_variable="pid",
    target_variable="cumulated_docs",
    new_first_variable_name="docs_first",
    new_difference_variable_name="docs_diff",
)
visits_pre = processor.preprocess(visits)  # cumulated value -> first value + differences
visits_post = processor.postprocess(visits, visits_pre)
```

Other client-side processors are listed in the {doc}`API reference
<../processors>`.

## How to write my own processor

A processor is any object with `preprocess(df)` and
`postprocess(source, dest)`. This example handles a very skewed count (for
example a number of documents per person): log transform and scaling before,
exact inverse and rounding after.

```python
import numpy as np


class LogCountProcessor:
    """log1p + standardization before; exact inverse + rounding after."""

    def __init__(self, columns: list[str]) -> None:
        self.columns = columns
        self.params: dict[str, tuple[float, float]] = {}

    def preprocess(self, df: pd.DataFrame) -> pd.DataFrame:
        df = df.copy()
        for col in self.columns:
            z = np.log1p(df[col].astype(float))
            mean, std = z.mean(), z.std(ddof=0) or 1.0
            self.params[col] = (mean, std)
            df[col] = (z - mean) / std
        return df

    def postprocess(self, source: pd.DataFrame, dest: pd.DataFrame) -> pd.DataFrame:
        dest = dest.copy()
        for col in self.columns:
            mean, std = self.params[col]
            dest[col] = np.clip(np.rint(np.expm1(dest[col] * std + mean)), 0, None).astype("int64")
        return dest


counts = pd.DataFrame({"nb_documents": np.random.default_rng(0).negative_binomial(1, 0.02, 500)})
processor = LogCountProcessor(["nb_documents"])
counts_pre = processor.preprocess(counts)
# runner.add_table("counts", counts_pre) ... runner.run() ...
# counts_avatars = processor.postprocess(counts, runner.shuffled("counts"))
```

The metrics are computed in the transformed space. See
[Tutorial 5](https://github.com/octopize/avatar-python/tree/main/notebooks)
for more examples.