EDI-Graphic: A Tool To Study Parameter Discrimination and Confirm Identifiability in Black-Box Models, and to Select Data-Generating Machines

In a Data-Generating Experiment (DGE), the data, X, is often obtained from a Black-Box and is approximated with a learning machine/sampler, f(Y,θ);θ∈Θ, Y is random, f is known. When X has unknown cdf, Fθ, nonidentifiability of θ cannot be confirmed and may limit the predictive accuracy of the learned model, f(Y,θ̂);θ̂ estimate of θ. Using properties of the Expected p-value for the Kolmogorov-Smirnov test, the Empirical Discrimination Index (EDI) and the Proportion of p-Values Index (PPVI) are introduced: (i) to confirm almost surely, discrimination of θ from θ*,that is,Fθ≠Fθ*, (ii) to confirm with EDI-graphics identifiability of θ(∈Θ) by repeating (i) for θ* in a fine sieve of Θ, and (iii) to compare EDI-graphics and PPVIs of DGEs and select to use the DGE with the greater parameter discrimination and the smaller number of θ* violating identifiability of θ. Among the applications, EDI and PPVI explain why the g-estimate in Tukey’s g-and-h model is better than that for the g-and-k model, unless the sample size is extremely large; h=h0=k. EDI-graphics indicate that Normal learning machines have better parameter discrimination than Sigmoid learning machines and their parameters are nonidentifiable. Supplementary materials for this article are available online.