EDI-Graphic: A Tool To Study Parameter Discrimination and Confirm Identifiability in Black-Box Models, and to Select Data-Generating Machines

Abstract

In a Data-Generating Experiment (DGE), the data, $\mathbf{X}$, is often obtained from a Black-Box and is approximated with a learning machine/sampler, $f(\mathbf{Y},\theta );\theta \in \Theta ,$ $\mathbf{Y}$ is random, f is known. When $\mathbf{X}$ has unknown cdf, $F_{\theta },$ nonidentifiability of θ cannot be confirmed and may limit the predictive accuracy of the learned model, $f(\mathbf{Y},\widehat{\theta });\widehat{\theta }$ estimate of $\theta .$ Using properties of the Expected p-value for the Kolmogorov-Smirnov test, the Empirical Discrimination Index (EDI) and the Proportion of p-Values Index (PPVI) are introduced: (i) to confirm almost surely, discrimination of θ from ${\theta }^{*},\text{that is,}{F}_{\theta }\ne F_{{\theta }^{*}},$ (ii) to confirm with EDI-graphics identifiability of $\theta (\in \Theta )$ by repeating (i) for ${\theta }^{*}$ in a fine sieve of $\Theta ,$ and (iii) to compare EDI-graphics and PPVIs of DGEs and select to use the DGE with the greater parameter discrimination and the smaller number of ${\theta }^{*}$ violating identifiability of $\theta .$ Among the applications, EDI and PPVI explain why the g-estimate in Tukey’s g-and-h model is better than that for the g-and-k model, unless the sample size is extremely large; $h=h_0=k.$ EDI-graphics indicate that Normal learning machines have better parameter discrimination than Sigmoid learning machines and their parameters are nonidentifiable. Supplementary materials for this article are available online.

KEYWORDS:

• Breiman’s Algorithmic model
• Black-box
• Data-generating experiment
• EDI-graphics
• Empirical Discrimination Index (EDI)
• g-and-k model
• Identifiability
• Intractable models
• Learning machines
• Proportion of p-values Index (PPVI)
• p-values
• Tukey’s g-and-h model

Supplementary Materials

Proofs and R-functions used in Examples 4.1–4.6.

Acknowledgments

Many thanks are due to Professor Faming Liang and Professor Galin Jones, Editors, who have handled, respectively, the original submission and the revisions. Thanks are due to the referees, for their comments that improved the presentation of the paper, and to Mr. Yongzhen Feng, Tsinghua University, for the suggestions to improve readability.

Disclosure Statement

The authors report there are no competing interests to declare.

Notes

1 When $\Theta =R^p,$ without preliminary estimation, ${\Theta }^{*}$ has countably infinite elements.

2 Since one DGE is studied.