User Experience Optimization Experiment Simulations

# The `uxo_sim` Package

A package for simulations of data matching industry UX optimization experiments, as discussed in:

```
@article{van_adelsberg_modeling_2019,
    title = {Modeling {A}/{B} {Test} {Data} is {Hard}: {Effects} of {Overdispersion}, {RandomWalks}, and {Cointegration}},
    language = {en},
    journal = {NeurIPS 2019 Workshop on Robust  AI in Financial Services: Data, Fairness, Explainability, Trustworthiness, and Privacy},
    author = {van Adelsberg, Matthew and Sweeney, Mackenzie},
    month = dec,
    year = {2019}
}
```

The code for running the simulations is included, along with figures and CSV files for each of three specific simulation runs that are used in a publication currently under review for ICML 2020.

## Packages:

1. `data`: code for running the simulations to produce datasets
2. `viz`: code for visualizing the simulation outputs

## Scripts:

1. `save_datasets`: save CSV for each simulated dataset in the `inventory`
2. `save_figs`: save PNG figure for each simulated dataset in `plots`

## Simulation Datasets:

### `fixed_effects_od_20_21_seed27`

Data is simulated from a beta-binomial distribution with overdispersion parameter `gamma=0.01` for each of the two treatments and rates `theta=0.20` and `theta=0.21`. This corresponds to beta distribution parameters `alpha, beta = 19.8, 79.2` and `alpha, beta = 20.79, 78.21`.

### `drift_down_then_up`

Data is simulated from a beta-binomial distribution with overdispersion parameter `gamma=0.01` for each of the two treatments. The rates start at `theta=0.20` and `theta=0.21` and then:

1. decrease by 0.005 each day for 20 days
2. increase by 0.005 each day for 30 days
3. stay constant for 10 days

The corresponding beta distribution parameters on each day are a function of `theta, gamma` and can be obtained via this function (implemented in `uxo_sims.data.simulations`:
```python
def alpha_beta_from_gamma_theta(gamma, theta):
    virtual_sample_size = 1 / gamma - 1
    alpha = theta * virtual_sample_size
    beta = virtual_sample_size - alpha
    return alpha, beta
```

### `arm_addition`

Data is simulated from a beta-binomial distribution with overdispersion parameter `gamma=0.01` for each of the two treatments. The rates start at `theta=0.10` and `theta=0.11` and increase by 0.005 each day for 40 days. The corresponding beta distribution parameters can be obtained with the same function as indicated in `drift_down_then_up`.