EDI-GRAPHIC: A TOOL TO STUDY PARAMETER DISCRIMINATION AND CONFIRM IDENTIFIABILITY IN BLACK-BOX MODELS, AND TO SELECT DATA-GENERATING MACHINES

In a Data-Generating Experiment (DGE), the data, X, is often obtained from a Black-Box and is approximated with a learning machine/sampler, f(Y,θ);θ∈Θ, Y is random, f is known. When X has unknown c.d.f., Fθ, non-identifiability of θ cannot be confirmed and may limit the predictive accuracy of the learned model, f(Y,θ̂);θ̂ estimate of θ. Using properties of the Expected P-value for the Kolmogorov-Smirnov test, the Empirical Discrimination Index (EDI) and the Proportion of P-Values Index (PPVI) are introduced: i) to confirm almost surely, discrimination of θ from θ*, i.e. Fθ≠Fθ*, ii) to confirm with EDI-graphics identifiability of θ(∈Θ) by repeating i) for θ* in a fine sieve of Θ, and iii) to compare EDI-graphics and PPVIs of DGEs and select to use the DGE with the greater parameter discrimination and the smaller number of θ* violating identifiability of θ. Among the applications, EDI and PPVI explain why the g-estimate in Tukey’s g-and-h model is better than that for the g-and-k model, unless the sample size is extremely large; h=h0=k. EDI-graphics indicate that Normal learning machines have better parameter discrimination than Sigmoid learning machines and their parameters are non-identifiable.