Processors

Processors transform the data to keep a business rule or to help the modelling. There are two kinds:

• Server-side processors are parameters: you pass them with processors=[...] in set_parameters / update_parameters. The server applies them before and after the avatarization. The output has the same columns as the input. When you pass several processors, their order in the list is the order of the preprocessing (reversed for the postprocessing).

• Client-side processors run on your machine, from avatars.processors: call preprocess on the original data before add_table, then postprocess on the avatars. The metrics are computed on the preprocessed data.

The snippets on this page start from this setup:

import os

import pandas as pd

from avatars.manager import Manager

manager = Manager(api_key=os.environ["AVATAR_API_KEY"])
df = pd.read_csv("fixtures/wbcd.csv")

Server-side processors

How to keep a total equal to the sum of its parts

Use RelativeDifferenceParameters when a variable is the sum (or an upper bound) of other variables: total hours and hours per activity, total revenue and revenue per category, …

from avatar_yaml.models.parameters import RelativeDifferenceParameters

players = pd.DataFrame(
    {
        "minutes_forward": [300, 120, 0, 50],
        "minutes_midfielder": [200, 400, 100, 0],
        "minutes_defender": [0, 80, 600, 450],
        "total_minutes": [500, 600, 700, 500],
    }
)

runner = manager.create_runner("relative_difference_example")
runner.add_table("players", players)
runner.set_parameters(
    "players",
    k=2,
    processors=[
        RelativeDifferenceParameters(
            target="total_minutes",
            references=["minutes_forward", "minutes_midfielder", "minutes_defender"],
            # scaling_unit=100,  # allow a small tolerance for rounding
        )
    ],
)
runner.run()

The same processor keeps a constraint A ≤ B (for example a number of visits lower than or equal to a number of documents): use B as the target and A as the reference.

How to keep start and end dates coherent across records

InterRecordRangeDifferenceParameters keeps start < end and the sequence of the periods of the same individual (stays of a patient, contracts of an employee, phases of a project).

from avatar_yaml.models.parameters import InterRecordRangeDifferenceParameters

stays = pd.DataFrame(
    {
        "patient_id": [1, 1, 2, 2, 3, 3],
        "start": [0, 10, 3, 20, 5, 9],
        "end": [5, 12, 9, 25, 7, 15],
        "age": [60, 60, 45, 45, 52, 52],
    }
)

runner = manager.create_runner("range_difference_example")
runner.add_table("stays", stays)
runner.set_parameters(
    "stays",
    k=2,
    processors=[
        InterRecordRangeDifferenceParameters(
            id_variable="patient_id",  # rows of the same individual
            target_start_variable="start",
            target_end_variable="end",
        )
    ],
)
runner.run()

Client-side processors

How to group rare modalities

GroupModalitiesProcessor merges the rare modalities of a categorical variable into one modality. It reduces the number of dimensions. It cannot be undone, so there is no postprocess.

from avatars.processors import GroupModalitiesProcessor

adult = pd.read_csv("fixtures/adult_with_missing.csv")

# In the variables with 10 modalities or more, merge the modalities with
# fewer than 25 individuals into "other"
processor = GroupModalitiesProcessor(min_unique=10, global_threshold=25, new_category="other")
adult_pre = processor.preprocess(adult)

runner = manager.create_runner("group_modalities_example")
runner.add_table("adult", adult_pre)

You can also give a threshold for each variable with variable_thresholds={"native-country": 50}.

How to keep a discrete numeric variable discrete

ToCategoricalProcessor models a numeric variable with few distinct values as a category. The avatars then have only values that exist (no 2.4 children, no score of 3.7 on an integer scale).

from avatars.processors import ToCategoricalProcessor

processor = ToCategoricalProcessor(to_categorical_threshold=20)  # <= 20 values -> category
df_pre = processor.preprocess(df)

runner = manager.create_runner("to_categorical_example")
runner.add_table("wbcd", df_pre)
runner.advise_parameters("wbcd")
runner.run()

avatars_df = processor.postprocess(df, runner.shuffled("wbcd"))  # back to numeric

How to keep parts that sum to a total

ProportionProcessor expresses variables as proportions of a reference variable, so that the parts still sum to the total in the avatars.

from avatars.processors import ProportionProcessor

budget = pd.DataFrame(
    {"total": [10, 20, 30], "a": [5, 5, 10], "b": [3, 10, 10], "c": [2, 5, 10]}
)
processor = ProportionProcessor(
    variable_names=["a", "b", "c"], reference="total", sum_to_one=True, decimal_count=0
)
budget_pre = processor.preprocess(budget)
# runner.add_table("budget", budget_pre) ... runner.run() ...
# budget_avatars = processor.postprocess(budget, runner.shuffled("budget"))

How to convert dates myself

Dates in ISO 8601 format are detected automatically. Use DatetimeProcessor only when you want to control the conversion yourself: it converts the dates to numeric timestamps, and back.

from avatars.processors import DatetimeProcessor

events = pd.DataFrame(
    {"admission": pd.to_datetime(["2023-01-02", "2023-05-10", "2023-09-30"])}
)
processor = DatetimeProcessor()
events_pre = processor.preprocess(events)  # dates -> timestamps
events_post = processor.postprocess(events, events_pre)  # timestamps -> dates

How to keep the mean and the standard deviation

ExpectedMeanProcessor moves the avatar values so that their mean (and optionally their standard deviation) matches the original data, globally or by group.

from avatars.processors import ExpectedMeanProcessor

processor = ExpectedMeanProcessor(
    target_variables=["Bare_Nuclei"], groupby_variables=["Class"], same_std=True
)
df_pre = processor.preprocess(df)

runner = manager.create_runner("expected_mean_example")
runner.add_table("wbcd", df_pre)
runner.advise_parameters("wbcd")
runner.run()

avatars_df = processor.postprocess(df, runner.shuffled("wbcd"))

Warning

This improves the univariate statistics, sometimes at the cost of the multivariate relations and of privacy. Compute the metrics again on the postprocessed avatars.

How to keep a cumulative value increasing

InterRecordCumulatedDifferenceProcessor makes sure that a cumulative value never decreases for an individual. Do not use it when the identifier or the target variable has missing values.

from avatars.processors import InterRecordCumulatedDifferenceProcessor

visits = pd.DataFrame({"pid": [1, 1, 1, 2, 2], "cumulated_docs": [2, 5, 9, 1, 4]})
processor = InterRecordCumulatedDifferenceProcessor(
    id_variable="pid",
    target_variable="cumulated_docs",
    new_first_variable_name="docs_first",
    new_difference_variable_name="docs_diff",
)
visits_pre = processor.preprocess(visits)  # cumulated value -> first value + differences
visits_post = processor.postprocess(visits, visits_pre)

Other client-side processors are listed in the API reference.

How to write my own processor

A processor is any object with preprocess(df) and postprocess(source, dest). This example handles a very skewed count (for example a number of documents per person): log transform and scaling before, exact inverse and rounding after.

import numpy as np


class LogCountProcessor:
    """log1p + standardization before; exact inverse + rounding after."""

    def __init__(self, columns: list[str]) -> None:
        self.columns = columns
        self.params: dict[str, tuple[float, float]] = {}

    def preprocess(self, df: pd.DataFrame) -> pd.DataFrame:
        df = df.copy()
        for col in self.columns:
            z = np.log1p(df[col].astype(float))
            mean, std = z.mean(), z.std(ddof=0) or 1.0
            self.params[col] = (mean, std)
            df[col] = (z - mean) / std
        return df

    def postprocess(self, source: pd.DataFrame, dest: pd.DataFrame) -> pd.DataFrame:
        dest = dest.copy()
        for col in self.columns:
            mean, std = self.params[col]
            dest[col] = np.clip(np.rint(np.expm1(dest[col] * std + mean)), 0, None).astype("int64")
        return dest


counts = pd.DataFrame({"nb_documents": np.random.default_rng(0).negative_binomial(1, 0.02, 500)})
processor = LogCountProcessor(["nb_documents"])
counts_pre = processor.preprocess(counts)
# runner.add_table("counts", counts_pre) ... runner.run() ...
# counts_avatars = processor.postprocess(counts, runner.shuffled("counts"))

The metrics are computed in the transformed space. See Tutorial 5 for more examples.