EDI-Graphic: A Tool To Study Parameter Discrimination and Confirm Identifiability in Black-Box Models, and to Select Data-Generating Machines

Abstract

In a Data-Generating Experiment (DGE), the data, $\mathbf{X}$, is often obtained from a Black-Box and is approximated with a learning machine/sampler, $f(\mathbf{Y},\theta );\theta \in \Theta ,$ $\mathbf{Y}$ is random, f is known. When $\mathbf{X}$ has unknown cdf, $F_{\theta },$ nonidentifiability of θ cannot be confirmed and may limit the predictive accuracy of the learned model, $f(\mathbf{Y},\widehat{\theta });\widehat{\theta }$ estimate of $\theta .$ Using properties of the Expected p-value for the Kolmogorov-Smirnov test, the Empirical Discrimination Index (EDI) and the Proportion of p-Values Index (PPVI) are introduced: (i) to confirm almost surely, discrimination of θ from ${\theta }^{*},\text{that is,}{F}_{\theta }\ne F_{{\theta }^{*}},$ (ii) to confirm with EDI-graphics identifiability of $\theta (\in \Theta )$ by repeating (i) for ${\theta }^{*}$ in a fine sieve of $\Theta ,$ and (iii) to compare EDI-graphics and PPVIs of DGEs and select to use the DGE with the greater parameter discrimination and the smaller number of ${\theta }^{*}$ violating identifiability of $\theta .$ Among the applications, EDI and PPVI explain why the g-estimate in Tukey’s g-and-h model is better than that for the g-and-k model, unless the sample size is extremely large; $h=h_0=k.$ EDI-graphics indicate that Normal learning machines have better parameter discrimination than Sigmoid learning machines and their parameters are nonidentifiable. Supplementary materials for this article are available online.

KEYWORDS:

• Breiman’s Algorithmic model
• Black-box
• Data-generating experiment
• EDI-graphics
• Empirical Discrimination Index (EDI)
• g-and-k model
• Identifiability
• Intractable models
• Learning machines
• Proportion of p-values Index (PPVI)
• p-values
• Tukey’s g-and-h model

1 Introduction

Statistical modeling is used in numerous fields, from Economics and Psychology to Biology, Engineering, and Machine Learning, among others. Often, it is assumed the sample, $\mathbf{X}=\{X_1,\dots ,X_n\},$ consists of observations with known cdf, $F_{\theta },$ we can do calculations with, that is, $F_{\theta }$ is tractable; θ is unknown, element of metric space $(\Theta ,\rho ).$ Recently, X_i is obtained either from a Sampler/Quantile function, $f(Y_i,\theta ),$ or from a Black-Box and is approximated with a learning machine, $f(Y_i,\theta ),i=1,\dots ,n;$ f is known. The cdf $F_{\theta }$ is either unknown or intractable, and we use in the sequel “unknown” to denote both and “sampler $f(\theta )$” assuming the Y-input is in the background.

Breiman (2001) called the Black-Box model algorithmic, observed that statisticians rarely adopt it, and commented: “If our goal as a field is to use data to solve problems, then we need to move away from exclusive dependence on data models and adopt a more diverse set of tools.” Earlier, Tukey (1962, p. 60) wrote: “Procedures of diagnosis, and procedures to extract indications rather than extract conclusions, will have to play a large part in the future of data analyses and graphical techniques offer great possibilities in both areas.” Both suggestions have been widely adopted nowadays in Data Science. Such tools and their theoretical justifications are presented for Black-Box models in this work and in Yatracos (2020, 2021, 2022). Among the discussants in Breiman (2001), Cox and Efron looked at the problem as prediction of X, ignoring the statistical inference aspect for θ in the X-model. Consequently, Mathematical statisticians did not follow Breiman’s suggestion. Computer scientists study the same problem with f as learning machine.

Modeling goals include estimation of $\theta ,$ and for the Black-Box in addition the accurate and reproducible prediction of future outputs of f using the estimate, $\widehat{\theta },$ of $\theta .$ These goals require identifiability of $\theta ,$ which depends on $\mathcal{F}=\{F_{{\theta }^{*}};{\theta }^{*}\in \Theta \}.$ Recall that the parameters in $\mathcal{F}$ are identifiable when for each θ and ${\theta }^{*}$ in $\Theta ,$ $\theta \ne {\theta }^{*}$ implies $F_{\theta }\ne F_{{\theta }^{*}}.$ Identifiability is confirmed so far only when cdfs in $\mathcal{F}$ are known and tractable, usually via the Fisher Information Matrix which is not available for unknown models.

Another problem not frequently studied for unknown models, $\mathcal{F},$ is that often, due to the shapes of $F_{{\theta }^{*}},{\theta }^{*}\in \Theta ,$ the sample size, n, needed for small estimation error may be excessively large and the statistician may not be aware about it. There is no direct data-tool measuring discrimination of θ from ${\theta }^{*}$ by evaluating the distance between $F_{\theta }$ and $F_{{\theta }^{*}},$ and also confirm θ-identifiability. For example, using Tukey’s g-and-h model (see (2)) and the g-and-k model (see (3)), the difficulty in the discrimination (and estimation) of parameters has been studied via Maximum Likelihood estimates (Rayner and MacGillivray, 2002).

In Machine Learning, nonidentifiability of θ is ubiquitous. The extent of nonidentifiability, that is, the number of ${\theta }^{*}$ making θ nonidentifiable, and the level of discrimination of parameters in $\mathcal{F}$ are criteria for the choice of the data-generating/learning machine, f, used. For tractable models and the Kolmogorov distance, $d_K,$ the larger $d_K(F_{\theta },F_{{\theta }^{*}})$ is, the greater the discrimination between θ and ${\theta }^{*}$ is. θ is identifiable when $d_K(F_{\theta },F_{{\theta }^{*}})>0$ for any ${\theta }^{*}\ne \theta ,$ which implies also $F_{{\theta }^{*}}\ne F_{\theta }.$ These suggest for unknown models $F_{\theta }$ and $F_{{\theta }^{*}},$ estimating their Kolmogorov distance and are leading to the use of tools from Kolmogorov-Smirnov test to study discrimination and identifiability of $\theta ,$ as described in Section 3.

Answers in the arising questions are provided by introducing:

(a) the Empirical Discrimination Index (EDI) of θ and ${\theta }^{*},$ used to study discrimination and identifiability; EDI is based on repeated samples and estimates the expected p-value for the Kolmogorov-Smirnov test between $F_{\theta }$ and $F_{{\theta }^{*}}$ under both hypotheses, $F_{\theta }=F_{{\theta }^{*}}$ and $F_{\theta }\ne F_{{\theta }^{*}},$ and (b) the Proportion of p-Values Index (PPVI) complementing EDI, and used to compare “locally” different samplers f₁, f₂ with the same $\mathbf{Y}$-input, regarding the estimation difficulty of a parameter-value, $\theta ={\theta }_0,$ with respect to parameter ${\theta }^{*}$ at distance $ϵ(>0).$

The motivation for this work was that Matching Estimates of $\theta =(g,h)$ were satisfactory at g₀ for Tukey’s g-and-h model/sampler but not for the g-and-k model/sampler for the same values of h, k (Yatracos 2020, 2021). The questions were “Why?” and “How this problem could be predicted in an Algorithmic model?” EDI and PPVI are used to answer these questions, confirming in particular that Tukey’s g-and-h model has greater g-discrimination than the g-and-k model, unless the sample size is extremely large.

In Example 4.1, EDI-graphics are presented for the normal and Cauchy models to compare the discrimination of their parameters, and to observe the form of EDI-graphics for identifiable parameters. In Examples 4.2 and 4.3, nonidentifiability is confirmed with EDI-graphics, respectively, for the parameters a, b in a normal model with a + b the mean and known variance, and for the mixture of two normal distributions with known variance. In Examples 4.4 and 4.5, Tukey’s g-and-h model and the g-and-k model are compared using EDI and PPVI, to confirm that the former has better discrimination and is to be preferred as data-generating or learning machine if estimation is the goal, unless the sample size, n, is extremely large. In Example 4.6, EDI-graphics are depicted for Normal and Sigmoid data-generating machines, showing that both have unidentifiable parameters but the former has better parameter discrimination.

EDI is not used herein as a statistical test, even though tools from hypotheses testing are used to motivate it without reference to significance level. EDI-graphics are simply observed for various sample sizes. There is no other method available to confirm identifiability and discrimination of θ when the underlying data model is unknown, thus, EDI is not compared with another method in the applications. A referee suggested to use EDI with the fashionable Wasserstein distance, $W_p,p\ge 1,$ instead of the Kolmogorov distance, $d_K,$ but this is not supported for Black-Box models which may have heavy tails (Yatracos 2022), and also because $W_p({\widehat{\mu }}_n,{\mu }_{\theta })$ does not necessarily converge to zero in probability as n increases, which is needed in Proposition 3.1 (b) for independent observations; ${\widehat{\mu }}_n$ is the empirical distribution/measure, and ${\mu }_{\theta }$ is the underlying distribution/probability measure of the data. The same referee had the concern “EDI is used with Data Generators, or when data models are known and tractable. However, for real data, the underlying generating data models are unknown and untractable. This means EDI is not applicable in many real applications.” To study identifiability and discrimination, either the cdf, $F_{\theta },$ has to be known and tractable, or, $F_{\theta }$ has to be “learned from samples obtained for various ${\theta }^{*}$”, that is, either from a sampler, or a quantile function as with Tukey’s g-and-h model, or simply via the inverse of $F_{\theta }$ and uniform r.vs on [0,1]. If none of this is possible, as in the referee’s example, one cannot study identifiability and discrimination. Note that in several statistics papers with intractable cdfs it is assumed that data can be generated for various ${\theta }^{*}$ values. Thus, with real data, rather than proposing an intractable cdf indexed by $\theta ,$ one can propose instead a function, $f(\mathbf{Y},\theta ),$ generating the data; $\mathbf{Y}$ random.

Rothenberg (1971) established conditions for parameter identifiability using $F_{\theta }$’s Fisher Information Matrix for tractable $F_{\theta },\theta \in \Theta .$ Nonidentifiable statistical models include mixture models (Hartigan 1985), autoregressive moving averages (Veres 1987) and change point problem (Csörgo and Horvath 1997). In Statistical Machine Learning, nonidentifiability of θ is ubiquitous and has deep influence in particular on the output, $f(\theta ),$ of the Learning Machine, f. The estimate, $\widehat{\theta },$ affects the capability of the learned model (or representation), $f(\widehat{\theta }),$ to predict future data (Ran and Hu 2017). For example, in Deep Neural Networks it is preferable that when a network relearns with data from the same model, the obtained learned representation is nearly similar to $f(\widehat{\theta }).$ This wish motivated studying linear identifiability in function space (Roeder, Metz, and Kingma 2021) for tractable models with general exponential form, extending results on distinct models (Hyvärinen and Morioka 2016; Hyvärinen, Sasaki, and Turner 2018). Other tools include the Fisher Information matrix, the asymptotic order of the Likelihood Ratio test statistic of the MLE and the Kullback-Leibler divergence of tractable models, used among others by Fukumizu (2003), Watanabe (2001), Fukumizu and Amari (2000), and Ran and Hu (2014). A detailed review is presented in Ran and Hu (2017), endorsing in the “Summary and Perspective” (p. 1196) the view in Breiman (2001) for algorithmic models and especially the tools needed to address identifiability, by adding: “This will become one of the most important issues for machines in the future.” EDI and PPVI clarify these issues and their advantage over the previously described methods is that there is no need to know the underlying model of the data to check for parameter identifiability and parameter discrimination.

Dempster and Schatzoff (1965) treated p-value as random and used its expected value, the Expected Significance Level, as sensibility index for comparing several multivariate tests. The Expected Significance Level was viewed as “reasonable compromise” to the Neyman-Pearson theory that uses the power of the test which depends on the α-level. Sackrowitz and Samuel-Cahn (1999) used Expected p-Value (EPV) instead of Expected Significance Level, to stress that p-value is random, suggested EPV as test’s performance measure when it is difficult to evaluate the power function, and examined the p-values under the alternative for location and scale models. Since then, EPV and its estimates have been used to study tests for parametric tractable models, but not for studying identifiability or discrimination of parameters in unknown models using the Kolmogorov-Smirnov test.

Empirical discrimination indices for θ based on tests’ comparisons, provide useful information in estimation problems since in the elementary estimation problem, Θ consists of two parameters, θ and ${\theta }^{*},$ and can be solved by testing. Stein (1964) showed inadmissibility of the usual estimator for the variance of a normal distribution with unknown mean using a test of hypotheses to obtain the improved estimate. LeCam (1973) and Birgé (2006) used successfully multiple, simultaneous tests of simple hypotheses for estimation with infinite dimensional parameter space, $\Theta .$

In Section 2, previous results related indirectly to parameter discrimination are presented. In Section 3, EPVs properties are presented and EDI and PPVI are introduced along with the use of EDI-graphics. Applications follow in Section 4. Proofs are included in the supplementary material, along with the R-programs used. The reader may proceed directly to Figures 1–8 that depict EDI-graphics with informative captions.

Fig. 1 EDI-graphics indicate identifiability for the Normal model, since one EDI-value remains near 0.5 as n increases. The graphics will be compared for parameter discrimination with those of the Cauchy model in Figure 2. Smaller EDI values for a model indicate greater discrimination, and therefore better estimate of θ for the same sample size, n.

Fig. 2 EDI-graphics indicate identifiability for the Cauchy model, since one EDI-value remains near 0.5 as n increases. Comparison with Figure 1 indicates EDI-values of the Cauchy model are larger than or equal to those of the Normal model, for the same distance from θ and n-value, indicating better parameter discrimination for the latter.

Fig. 3 EDI-graphics indicate nonidentifiable parameters for the normal model, $N(a+b,1),$ with $\theta =(a,b)=(1.2,1.6).$ Several EDI-values near 0.5 as n increases, indicate nonidentifiability, with ${\theta }^{*}=(a^{*},b^{*}),a^{*}+b^{*}=\mathrm{2.8.}$ When $n=1000,$ EDI-values near 0.2 correspond to nonidentifiable ${\theta }^{*}$ with $a^{*}+b^{*}=\mathrm{2.7.}$

Fig. 4 Nonidentifiability of the normal mixture, $0.25N(0.4,1)+0.75N(1.2,1),$ is confirmed by the two circles with y-coordinates near 0.5 for all sample sizes n. Circles with similar y-coordinates far from 0, as n increases, indicate nonidentifiable parameters.

Fig. 5 Nonidentifiability of the normal mixture, $0.25N(0.35,1)+0.75N(1.25,1),$ with the parameter $\theta =(0.25,0.35,1.25)$ not included in the sieve, is confirmed by the two circles with the larger y-coordinates, which decrease as n increases. Circles with similar y-coordinates far from 0, as n increases, indicate nonidentifiable parameters, as in Figure 4.

Fig. 6 EDI-graphics for comparison of data generating machines: Tukey’s g- and- h (denoted TGH) and the g- and- k (denoted GK) models. Tukey’s g- and- h has better parameter discrimination for moderate sample size, reconfirming Example 4.4.

Fig. 7 EDI-graphics showing nonidentifiability for data, X, from a Normal learning machine, $X=f(Z,\theta )=p\varphi (Z-{\mu }_1)+(1-p)\varphi (Z-{\mu }_2),\theta =(p,{\mu }_1,{\mu }_2)=(0.25,0.4,1.2)={\theta }_{46}^{*};Z$ is a standard Normal $r.v.$ with density $\varphi .$

Fig. 8 EDI-graphics showing nonidentifiability for data, X, from a Sigmoid learning machine, $X=f(Z,\theta )=\mathit{\text{ps}}(Z-{\mu }_1)+(1-p)s(Z-{\mu }_2),\theta =(p,{\mu }_1,{\mu }_2)=(0.25,0.4,1.2)={\theta }_{46}^{*};Z$ is a standard Normal $r.v.,$ $s(u)={(1+e^{-u})}^{-1},u\in R.$ Comparison with Figure 7 favors the Normal learning machine which shows better discrimination.

2 Model Shapes and Parameter Discrimination

Tukey (1962, 1977) used the g-and-h model to better fit sample, $\mathbf{X},$ with heavy tails and skewness, and the λ-distribution to fit both symmetric and asymmetric data. The idea is to model the quantiles of $\mathbf{X}$ directly and not via a density (Yan and Genton 2019). Thus, $\mathbf{X}$ is seen as modification of data, $\mathbf{Y},$ using known data-generator f, with unknown parameter $\theta (\in R^p),$ for example, $\theta =(g,h),$ (1) $$X_i=f(Y_i,\theta ),i=1,\dots ,n;$$(1)

$p\ge 1,\mathbf{Y}=\{Y_1,\dots .Y_n\}$ is observed from a known model with known parameter $\tilde{\eta }.$

Tukey’s quantile-modeling approach has been used since then in several fields, (1) evolved and has led to an abundance of models’ shapes that would fit better $\mathbf{X}.$ However, this abundance of model shapes near the underlying distribution increases the difficulty in estimating $\theta ,$ and f in (1) should be examined before its use as potential data-generator.

Tukey’s (1977) $(a,b,g,h)$-model accommodates data from non-Gaussian distribution, with g real-valued controlling skewness, nonnegative h controlling tail heaviness and with location, $a\in R,$ and scale, $b>0.$A vector of independent standard normal random variables, $\mathbf{Z}=\{Z_1,\dots ,Z_n\},$ and parameter values $a,b,g,h$ are used to obtain (2) $$X_{1,i}(g,h)=a+b\frac{e^{g{Z}_i}-1}{g}{e}^{.5h{Z}_i^2},i=1,\dots ,n;$$(2) $${\mathbf{X}}_1(g,h)=(X_{1,1}(g,h),\dots ,X_{1,n}(g,h)).$$

The $(a,b,g,k)$-model (Haynes, MacGillivray, and Mengersen 1997) includes distributions with more negative kurtosis than the normal distribution and some bimodal distributions (Rayner and MacGillivray 2002, p. 58). Standard normal $Z_1,\dots ,Z_n$ and parameter values $a,b,g,k$ are used to obtain (3) $$X_{2,i}(g,k)=a+b[1+c·\frac{1-e^{-g{Z}_i}}{1+e^{-g{Z}_i}}]{(1+Z_i^2)}^k{Z}_i,i=1,\dots ,n;$$(3)

${\mathbf{X}}_2(g,k)=(X_{2,1}(g,k),\dots ,X_{2,n}(g,k)),c$ is a parameter used to make the sample correspond to a density and usually $c=.8.$

With abuse of notation, we use in (2) and (3) $\mathbf{X}(g,u)$ instead of $\mathbf{X}(a,b,g,u),u=h,k$ and the same for their coordinates. Since g-and-k and g-and-h use both the same $\mathbf{Y}=\mathbf{Z},$ EDI and PPVI are used in the Examples 4.4 and 4.5.

For the g-and-k model (Haynes, MacGillivray, and Mengersen 1997) and the generalized g-and-h models, Rayner and MacGillivray (2002) confirmed the difficulty of the MLE to discriminate distributional shapes and parameters’ values with small and moderate $n:$ “… computational Maximum Likelihood procedures are very good for very large sample sizes, but they should not necessarily be assumed to be safe for even moderately large sample sizes” (p. 58); also, “… with moderately large positive (i.e., to the right) skewness, the MLE method fitting to the g-and-k distribution cannot efficiently discriminate between moderate positive values and small negative values of the kurtosis parameter.” (p. 64).

For Tukey’s asymmetric λ-distributions, with wider variety of distributional shapes than g-and-h and using the Moments estimation method it is observed: “An additional difficulty with the use of this distribution when fitting through moments, is that of nonuniqueness, where more than one member of the family may be realized when matching the first four moments …” (Ramberg et al. 1979; Rayner and MacGillivray 2002, p. 58).

These findings suggest the search for a data dependent tool to evaluate the discrimination and identifiability of θ-values.

3 Discrimination and Identifiability of Parameters in Algorithmic Models with EDI and PPVI

3.1 Data Generating Experiment-Definitions- Assumptions on f

The elements of a statistical problem with data obtained from known models are included in Le Cam’s Statistical Experiment, $(\mathcal{X},\mathcal{A},\{F_s,s\in \Theta \}),$ with $\mathcal{X}$ sample space, $\mathcal{A}$ its σ-field and F_s probability (or cdf) indexed by s in parameter space $\Theta ;$ see, for example, LeCam and Yang (1990). The parallel notion of Data-Generating Experiment is introduced for data obtained from Breiman’s Algorithmic models and Samplers.

Definition 3.1.

A Data-Generating Experiment (DGE), $(\mathcal{X},f,\Theta ),$ consists of the sample space, $\mathcal{X},$ and the data-generating mechanism (or sampler), f, with inputs parameter θ ($\in \Theta$) and sample, $\mathbf{Y},$ that may be latent or not.

The findings in Section 2 indicate, for DGEs with unknown models, the need to study the discrimination of parameters using tools independent from estimation methods. Examples with DGEs appear in the Applications section and include in particular Tukey’s g-and-h model and the g-and-k model.

In a DGE, sample $\mathbf{X}$ with size n is drawn using an unknown parameter, $\theta (\in \Theta ),$ in the sampler, f. The observations in $\mathbf{X}$ take values in $R^d,$ have unknown cdf, $F_{\theta },$ and the aim is to estimate $\theta ;d\ge 1.$ Herein, Θ is a subset of $R^p,p\ge 1,$ equipped with distance, $\rho .$ For a statistical inference problem to be meaningful, every θ in Θ should be identifiable. DGE includes any underlying structure needed, for example, a distance for cdfs, a prior for θ and σ-field(s). The user can select ${\theta }^{*}\in \Theta$ to draw one or more ${\mathbf{X}}^{*}$ via f. With abuse of notation we use $f({\theta }^{*})$ instead of $f({\theta }^{*},\mathbf{Y}).$

The sampler, f, in a DGE replaces succesfully the uknown underlying models $\{F_s;s\in \Theta \}$ in the Statistical Experiment, when repeated samples are drawn for selected ${\theta }^{*}$ in Θ providing adequate information for $F_{{\theta }^{*}}$ (Yatracos 2020).

Definition 3.2.

For any two distribution functions F, G in $R^d,d\ge 1,$ their Kolmogorov distance (4) $$d_K(F,G)=\mathrm{\text{sup}}\left\{\right|F(y)-G(y)|;y\in R^d\}.$$(4)

Definition 3.3.

For any sample $\mathbf{U}=(U_1,\dots ,U_n)$ of random vectors in $R^d,n{\widehat{F}}_{\mathbf{U}}(u)$ denotes the number of $U_i$’s with all their components smaller or equal to the corresponding components of $u(\in R^d).{\widehat{F}}_{\mathbf{U}}$ is the empirical cdf of $\mathbf{U}.$

Definition 3.4.

θ is discriminated from ${\theta }^{*}$ when $F_{\theta }\ne F_{{\theta }^{*}},$ or equivalently if $d_K(F_{\theta },F_{{\theta }^{*}})>0.$

Definition 3.5.

θ is identifiable when for any ${\theta }^{*}\in \Theta ,{\theta }^{*}\ne \theta ,$ it holds that $F_{\theta }\ne F_{{\theta }^{*}}.$

The larger the distance $d_K(F_{\theta },F_{{\theta }^{*}})$ is, the greater the discrimination between θ and ${\theta }^{*}$ is. If θ is discriminated from all ${\theta }^{*}\in \Theta ,{\theta }^{*}\ne \theta ,$ then θ is identifiable. However, in a DGE, $d_K(F_{\theta },F_{{\theta }^{*}})$ can be only estimated using f due to the unknown cdfs, $F_{\theta }$ and $F_{{\theta }^{*}}.$

The assumption from “Looking inside the Black-Box (Breiman 2002, p. 3)” is used in A1 for the DGE.

A1: In DGE $(R^d,f,\Theta ),$ the data, $\mathbf{X},$ are generated by independent draws via f.

Assumption A1 is used in Proposition 3.1, (b), (c). However, (b) can be obtained with dependent draws via f, under the weaker assumption A2.

A2: In DGE $(R^d,f,\Theta ),$ for the data, $\mathbf{X},$ with size n and underlying cdf of its components, $F_{\theta },\underset{n\to \infty }{\mathrm{\text{lim}}}{d}_K({\widehat{F}}_{\mathbf{X}},F_{\theta })=0$ in probability.

3.2 Motivation and Description of the EDI-Approach

In practice, θ is not distinguished locally from ${\theta }^{*}$-values in an open ρ-ball, $N_{ϵ}(\theta ),$ centered at θ with small radius $ϵ.$ Violation of θ-identifiability due to ${\theta }^{*}\in N_{ϵ}(\theta )$ for ϵ small, is not expected to have significant effect in learning machines but also in probability calculations. Thus, we restrict attention to identifiability of θ with respect to ${\theta }^{*}$ in $N_{ϵ}^c(\theta );$ A^c denotes the complement of set A.

For a DGE, the goals herein are: (i) to study the discrimination of θ and ${\theta }^{*}$ at ρ-distance greater than or equal to small $ϵ,$ (ii) to confirm identifiability of θ using (i) for all ${\theta }^{*}$ in an ϵ-dense subset/sieve, ${\Theta }^{*},$ of $\Theta ,$ and (iii) to select among several data-generating machines which one to use for better estimation of parameters, taking into consideration the sample size, and preferring less nonidentifiability, that is, a smaller number of ${\theta }^{*}$ making θ nonidentifiable, and more/greater discrimination, namely larger $d_K(F_{\theta },F_{{\theta }^{*}}).$

For $i),$ the main tool to discriminate θ from ${\theta }^{*}$ is the empirical discrimination index, ${\text{EDI}}_M,$ which estimates the expected p-value, $\text{EPV}(\theta ,{\theta }^{*};n),$ for the Kolmogorov-Smirnov test of hypotheses $F_{\theta }=F_{{\theta }^{*}}$ against $F_{\theta }\ne F_{{\theta }^{*}}$ and has special properties (Proposition 3.1). Samples $\mathbf{X}$ and ${\mathbf{X}}^{*}$ of size n are drawn using $f(\theta )$ and $f({\theta }^{*}),$ respectively, and $d_K(F_{\theta },F_{{\theta }^{*}})$ is estimated by $d_K({\widehat{F}}_{\mathbf{X}},{\widehat{F}}_{{\mathbf{X}}^{*}}).$ To evaluate how large $d_K(F_{\theta },F_{{\theta }^{*}})$ is, since $d_K({\widehat{F}}_{\mathbf{X}},{\widehat{F}}_{{\mathbf{X}}^{*}})$ is random, one could rely on its p-value exceeding the observed $t^{*}=d_K({\widehat{F}}_{\mathbf{x}},{\widehat{F}}_{{\mathbf{x}}^{*}}).$ Having the luxury of the f-availability, M such samples with $t^{*}$ and p-values are obtained, and the average, ${\text{EDI}}_M,$ of these p-values is used. Smaller value of ${\text{EDI}}_M$ (or, simply EDI) indicates greater discrimination, as smaller p-value does.

For (ii), $\Theta (\subseteq R^p)$ is assumed to be compact which holds after a preliminary estimation of $\theta (\in R^p);$ see, for example, Yatracos (2020). Let ${\theta }_1^{*}=\theta$ and form a partition of $N_{ϵ}^c(\theta ),$ namely, $N_{ϵ}({\theta }_2^{*}),\dots ,N_{ϵ}({\theta }_m^{*}).$ ϵ-identifiability of θ is confirmed almost surely by confirming with ${\text{EDI}}_M$ its discrimination from each ${\theta }^{*}$ in ${\Theta }^{*}-\left\{{\theta }_1^{*}\right\},$ (5) $${\Theta }^{*}=\{{\theta }_1^{*},\dots ,{\theta }_m^{*}\}.$$(5) ¹

For (iii), EDI-graphics can be compared for several samplers, as in Figures 6–8.

For practical purposes, ϵ-identifiability of θ will imply identifiability for all the parameters in $N_{ϵ}(\theta ).$ Global identifiability in Θ can be confirmed establishing ϵ-identifiability for $\theta ={\theta }_i^{*},i>1.$ Alternatively, EDI-graphics for one θ only, with ${\theta }^{*}$ in sieve, ${\Theta }^{*},$ and various sample sizes, n, may be enough as observed in Example 4.1 for which θ-identifiability implies $\tilde{\theta }$-identifiability for every $\tilde{\theta }$ in $\Theta .$ This is expected to hold for location models.

3.3 The EDI-Details

In a DGE $(R^d,f,\Theta ),$ with underlying unknown cdfs, $\mathcal{F}=\{F_s:s\in \Theta \},$ let (6) $$\eta =d_K(F_{\theta },F_{{\theta }^{*}}),0\le \eta \le 1,$$(6) and consider the hypotheses (7) $$H:\eta =0(i.e.,F_{\theta }=F_{{\theta }^{*}})\text{against}{H}^{*}:\eta ={\eta }^{*}>0(i.e.,F_{\theta }\ne F_{{\theta }^{*}}).$$(7)

For the obtained samples $\mathbf{X}$ and ${\mathbf{X}}^{*}$ of size n, respectively, with unknown cdfs, $F_{\theta }$ and $F_{{\theta }^{*}},$ let (8) $$T_n=d_K({\widehat{F}}_{\mathbf{X}},{\widehat{F}}_{{\mathbf{X}}^{*}}).$$(8)

Under H, let G₀ denote T_n’s cdf, and P₀ the corresponding probability. H is rejected if T_n is large, $T_n>t^{*}=d_K({\widehat{F}}_{\mathbf{x}},{\widehat{F}}_{{\mathbf{x}}^{*}}),$ or instead if (9) $$\text{PV}=\text{PV}(\text{DGE})=P_0(T_n>t^{*})=1-G_0(t^{*})$$(9) is small. The p-value (9) is calculated under H, and is random since it depends on the observed $t^{*}$-value. Its expected value, EPV, is going to be calculated under both H and $H^{*},$ and we write (10) $$\begin{array}{c}\text{EPV}{(\theta ,{\theta }^{*};n)}^2=\text{EPV}(\theta ,{\theta }^{*};n,\text{DGE})\\ ={\int }_0^1{P}_0(T_n>t^{*})d{G}_{{\eta }^{*}}(t^{*})\\ =1-E_{{\eta }^{*}}{G}_0(T_n^{*}),{\eta }^{*}\ge 0.\end{array}$$(10) ²

In the right side of (10), $T_n^{*}$ is the random variable with observed value $t^{*}$ obtained when ${\eta }^{*}=d_K(F_{\theta },F_{{\theta }^{*}})\ge 0.$

Definition 3.6

(EDI). In a $\text{DGE}(R^d,f,\Theta ),$ let ${\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_M$ be samples of size n drawn with unknown cdf $F_{\theta },$ and let ${\mathbf{X}}_1^{*},\dots ,{\mathbf{X}}_M^{*}$ be samples of size n from $F_{{\theta }^{*}}.$ Let PV_i be the p-value (9) for the two sided Kolmogorov-Smirnov test (7) using ${\mathbf{X}}_i,{\mathbf{X}}_i^{*},i=1,\dots ,M.$ The Empirical Discrimination Index (EDI) of θ and ${\theta }^{*}$ is (11) $${\text{EDI}}_M(\theta ,{\theta }^{*};n)={\text{EDI}}_M(\theta ,{\theta }^{*};n,\text{DGE})=\frac{1}{M}\sum _{i=1}^{M}P{V}_i.$$(11)

The smaller the ${\text{EDI}}_M$-value is in (11), the greater/better the discrimination of θ and ${\theta }^{*}$ is. EDI is also used instead of ${\text{EDI}}_M$ in the sequel.

Proposition 3.1.

In $\text{DGE},(R^d,f,\Theta ),$ let T_n and $T_n^{*}$ be as previously described.

1. For every θ in Θ and for every sample size n, (12) $$\text{EPV}(\theta ,\theta ;n)=E_0[1-G_0(T_n^{*})]=.5.$$(12)

2. Under either A1 or A2, for every ${\eta }^{*}>0,$ (13) $$\underset{n\to \infty }{\mathrm{\text{lim}}}\text{EPV}(\theta ,{\theta }^{*};n)=\underset{n\to \infty }{\mathrm{\text{lim}}}{E}_{{\eta }^{*}}[1-G_0(T_n^{*})]=0.$$(13)

3. Under A1, if ${\eta }^{*}=d_K(F_{\theta },F_{{\theta }^{*}})<{\eta }^{**}=d_K(F_{\theta },F_{{\theta }^{**}}),$ and $T_n^{**}$ is defined as T_n in (8) using ${\mathbf{X}}^{**}$ from $F_{{\theta }^{**}},$ then for large n, (14) $$\begin{array}{c}\text{EPV}(\theta ,{\theta }^{*};n)=E_{{\eta }^{*}}[1-G_0(T_n^{*})]\ge E_{{\eta }^{**}}[1-G_0(T_n^{**})]\\ =\text{EPV}(\theta ,{\theta }^{**};n).\end{array}$$(14)

From Proposition 3.1 (a), (b), discrimination of θ and ${\theta }^{*}$ and identifiability of θ are confirmed when, for large n-values, as M increases ${\text{EDI}}_M$ converges almost surely to its expected value. This always holds under A1 and may hold also under A2 for dependent data. Thus, when ${\text{EDI}}_M(\theta ,{\theta }^{*};n,\text{DGE}),$ $\theta \ne {\theta }^{*},$ stays near 0.5 as n increases, θ is not identifiable. Also, if ${\text{EDI}}_M(\theta ,{\theta }^{*};n,\text{DGE})>{\text{EDI}}_M(\theta ,{\theta }^{**};n,\text{DGE}),$ then θ has greater discrimination from ${\theta }^{**}$ than from ${\theta }^{*}$ for sample size n.

Remark 3.1.

In Dempster and Schatzoff (1965) and Sackrowitz and Samuel-Cahn (1999), property (a) in Proposition 3.1 has been used as well as that, when T_n and $T_n^{*}$ are independent, from (10) (15) $$\begin{array}{c}P(T_n>T_n^{*})={\int }_0^{1}P(T_n>t^{*}|T_n^{*}=t^{*})d{G}_{{\eta }^{*}}(t^{*})\\ =\text{EPV}(\theta ,{\theta }^{*};n)=1-E_{{\eta }^{*}}{G}_0(T_n^{*}),\end{array}$$(15) with T_n and $T_n^{*}$ obtained, respectively, under H and $H^{*},$ or under H and H.

3.4 DGE Selection with EDI

EDI in (11) can be used to compare two or more DGEs and choose one if estimation is the goal. Assume that DGE1 and DGE2 are indexed by a parameter $\theta \in \Theta$ (usually of similar nature, for example, location) and that samples as in Definition 3.6 are obtained. Then, the discrimination of θ from ${\theta }^{*}$ is greater in DGE1 than DGE2 if (16) $${\text{EDI}}_M(\theta ,{\theta }^{*};n,\text{DGE}1)<{\text{EDI}}_M(\theta ,{\theta }^{*};n,\text{DGE}2).$$(16)

When (16) holds for θ and ${\theta }^{*}$ in $\Theta ,$ then DGE1 is preferred if both DGE1 and DGE2 could be used for modeling. The reason is that what holds for a choice of θ and ${\theta }^{*},$ is expected to hold for all the parameters in $\Theta .$ The EDI comparison can be used for DGE1 and DGE2 with parameters, respectively $\theta ,{\theta }^{*}$ in $\Theta ,$ $\zeta ,{\zeta }^{*}$ in $\mathcal{Z}$ and $\rho (\theta ,{\theta }^{*})={\rho }_{\mathcal{Z}}(\zeta ,{\zeta }^{*});{\rho }_{\mathcal{Z}}$ is distance in $\mathcal{Z}.$

3.5 The Proportion of p-values Index, PPVI

Another tool, complementing EDI, is introduced for a particular type of DGEs.

Definition 3.7.

When DGE1 and DGE2 use each quantile model (1) with variables $\mathbf{Y},{\mathbf{Y}}^{*}$ from the same model, for each of the M tests providing (11), $\mathbf{Y}(={\mathbf{Y}}^{*})$ can be used to generate the data and p-values are compared. The proportion of p-values index is (17) $$\begin{array}{c}{\text{PPVI}}_M(\theta ,{\theta }^{*};n,\text{DGE}1,\text{DGE}2)\\ \hspace{1em}=\frac{\#\{P{V}_j(\text{DGE}2)<P{V}_j(\text{DGE}1),j=1,\dots ,M\}}{M}.\end{array}$$(17)

${\text{PPVI}}_M$ is an additional tool to extract indications for discrimination. PPVI is also used instead of ${\mathit{\text{PPVI}}}_M.$

DGE Selection with the PPVI Discrimination Criterion

For DGE1 and DGE2 as in Definition 3.7, the discrimination of θ from ${\theta }^{*}$ for sample size n is greater/easier in DGE1 than DGE2 if (18) $$\begin{array}{c}{\text{PPVI}}_M(\theta ,{\theta }^{*};n,\text{DGE}1,\text{DGE}2)<.5\\ {\text{PPVI}}_M(\theta ,{\theta }^{*};n,\text{DGE}2,\text{DGE}1)>.5.\end{array}$$(18)

Because of the strict inequality, “<”, used in (17), ${\text{PPVI}}_M(\theta ,{\theta }^{*};n,\text{DGE}2,\text{DGE}1)$ is also included in (18). In some cases, when every θ is identifiable and (18) holds, a better estimate is obtained for DGE1 unless the sample size is very large; compare in Table 2, PPVIs for Tukey’s g-and-h model (DGE1) and the g-and-k model (DGE2) for $200\le n\le 35000.$ Similar are the findings is Table 2 comparing EDI (g-and-h) with EDI (g-and-k).

Table 2 Model parameters: g = 5, $g^{*}=3,h=k=2.5$. EDIs and PPVIs for g provide similar indications and are based on M = 1000 repeats.

g-Local Discrimination: Tukey’s g-and-h and g-and-k models
n	EDI (g-and-h)	EDI (g-and-k)	PPVI (g-and-h, g-and-k)	PPVI (g-and-k, g-and-h)
50	5.05 e-01	5.31e-01	0.1540	0.2980
100	4.47e-01	4.92e-01	0.1762	0.4287
200	3.61e-01	4.38e-01	0.1864	0.5459
500	2.00e-01	3.15 e-01	0.1683	0.6845
1000	8.12e-02	1.836 e-01	0.1379	0.7807
1500	3.56e-02	1.083e-01	0.1150	0.8327
2500	6.21e-03	3.19e-02	0.0988	0.8745
5000	1.08e-04	1.86e-03	0.0680	0.9205
10000	7.31e-08	5.37e-06	0.0388	0.9379
35000	0	2.22e-20	0	1e-04
40000	0	0	0	0

3.6 Implementation

In DGE $(R^d,f,\Theta ),$ Θ is a compact subset (rectangle) of R^p with distance, $\rho (\theta ,{\theta }^{*}),$ the L₂-distance; $d\ge 1,p\ge 1.$ For $\theta \in \Theta$ and $ϵ(>0)$ small enough such that parameters in a neighborhood, $N_{ϵ}(\theta ),$ are “indistinguishable” for each coordinate, let ${\Theta }^{*}=\{{\theta }_1^{*},\dots ,{\theta }_m^{*}\}$ be the sieve in (5), with $\theta \in {\Theta }^{*}$ but not necessarily equal to ${\theta }_1^{*}.$ The construction of $N_{ϵ}(\theta )$ and the sieve, ${\Theta }^{*},$ is described in the examples. The reader may prefer to use a different sieve than the ${\Theta }^{*}$ used, for example, in R based on midpoints of intervals instead of the end-points of intervals for each θ-coordinate. In the EDI-graphic for $\theta ,$ the y-axis is used for EDI-values and the x-axis for the Euclidean L₂-distance between θ and the elements of sieve, ${\Theta }^{*}.$

When $d=1,$ the R-function ks.test provides the p-value for the Kolmogorov-Smirnov two-sample test of equality for θ and ${\theta }^{*},$ with $\mathbf{X}$ and ${\mathbf{X}}^{*},$ respectively, from $F_{\theta }$ and $F_{{\theta }^{*}}.$ For $d\ge 2,$ the approaches in Peacock (1983) and Polonik (1999) can be used to obtain p-values The theory in the latter was implemented by Glazer, Lindenbaoum, and Markovitch (2012), who estimated high-density regions directly instead of using density estimates.

Use of EDI-graphics

EDI-graphics of ${\text{EDI}}_M\left(\theta ,{\theta }_k^{*};n\right)$ against $\rho (\theta ,{\theta }_k^{*}),k=1,\dots ,m,$ for various $n:$

1. show nonidentifiability of θ (and other parameters) when, as n increases, at least ${\text{EDI}}_M(\theta ,{\theta }_i^{*}=\theta ;n)$ and ${\text{EDI}}_M(\theta ,{\theta }_j^{*};n)$ have values near 0.5, $i\ne j,$

2. indicate greater discrimination of θ from ${\theta }_1^{*}$ than from ${\theta }_2^{*}$ when $\text{EDI}(\theta ,{\theta }_1^{*};n)<\text{EDI}(\theta ,{\theta }_2^{*};n),$ and

3. allow comparing data-generating machines with their EDI-graphics using (A) and (B), preferring greater discrimination and less nonidentifiability when estimation is the goal.

4 Applications

EDI-graphics for $\theta ,$ with ${\theta }^{*}$ in the sieve ${\Theta }^{*},$ are presented for various sample sizes, for known models and also DGEs with identifiable and nonidentifiable parameters to see their differences when using the graphics. For the interpretation of EDI-graphics follow (A)–(C) at the end of the previous section and/or read the Figures’ captions. Figures 1–8 indicate that EDI-graphics for one θ and for ${\theta }^{*}$ in ${\Theta }^{*},$ and for moderate and large sample size, n, are sufficient for checking identifiability in Θ and parameter discrimination, thus, confirming EDI’s usefulness.

EDI-graphics for statistical models are first presented.

Example 4.1.

EDI-graphics for Normal (N) and Cauchy (C) models are depicted in Figures 1 and 2, respectively, for studying discrimination and confirming identifiability of θ=$(\mu ,\sigma )=(1.2,1.6).$ The assumed parameter spaces for μ and σ are, respectively, $[0,2]$ and $[0.4,2.4],$ and $\Theta =[0,2]\times [0.4,2.4].$ For $ϵ=0.4,$ consider in each parameter space, respectively, sieve ${\mu }_j^{*}=.4(j-1),1\le j\le 6$ and ${\sigma }_k^{*}=0.4+0.4(k-1),1\le k\le 6,$ providing Θ-sieve, ${\Theta }^{*},$ with elements ${\theta }_i^{*}=({\mu }_j^{*},{\sigma }_k^{*}),i=1,\dots ,36,$ that includes $\theta .$

M = 100 independent samples $\mathbf{X}$ and ${\mathbf{X}}^{*},$ each of size n, are obtained with parameters, respectively, θ and ${\theta }_i^{*}$ in both models, and the corresponding EDIs are calculated, $i=1,\dots ,36,$ for $n=100,300,600,1000.$ Identifiability of θ is indicated in Figures 1 and 2 since $\text{EDI}(\theta ,\theta ;n)$ takes value near 0.5 and the other EDI values, $\text{EDI}(\theta ,{\theta }^{*};n),$ decrease as $d_K(F_{\theta },F_{{\theta }^{*}})$ increases and, in this example, the Euclidean distance of θ and ${\theta }^{*}$ increases. All the EDI-values eventually decrease to zero as n increases except for $\text{EDI}(\theta ,\theta ;n).$ The simulation results are expected from Proposition 3.1 since ${\text{EDI}}_M$ converges to the EPV.

Indicative $\text{EDI}(\theta ,{\theta }^{*};n)$-values are provided in Table 1, for ${\theta }_{22}^{*}=\theta =(1.2,1.6)$ and ${\theta }_{23}^{*}=(1.2,2),$ to observe better parameter discrimination for the Normal model.

Table 1 EDI-values in column ${\theta }_{23}^{*}(N)$ converge faster to zero than those in column ${\theta }_{23}^{*}(C),$ indicating greater discrimination of θ with the Normal model than the Cauchy model.

EDI $(\theta ,{\theta }^{*})$ FOR NORMAL(N) AND CAUCHY(C)
n	${\theta }_{22}^{*}(N)=\theta$	${\theta }_{23}^{*}(N)$	${\theta }_{22}^{*}(C)=\theta$	${\theta }_{23}^{*}(C)$
100	0.53	0.41	0.53	0.48
300	0.56	0.20	0.49	0.35
600	0.56	0.05	0.51	0.21
1000	0.47	0.02	0.53	0.10

Example 4.2.

EDI-graphics for Normal distributions of the form $N(a+b,1)$ are depicted in Figure 3 to confirm nonidentifiability of the parameter $\theta =(a,b)=(1.2,1.6).$ The assumed parameter spaces for a and b are, respectively, $[0,2]$ and $[0.4,2.4],$ and $\Theta =[0,2]\times [0.4,2.4].$ For $ϵ=0.1,$ in each parameter space, consider $a_j^{*}=.1(j-1),1\le j\le 21$ and $b_k^{*}=.4+.1(k-1),1\le k\le 21,$ providing sieve ${\theta }_i^{*}=(a_j^{*},b_k^{*})\in \Theta ,i=1,\dots ,{21}^2.$

M = 100 independent samples $\mathbf{X}$ and ${\mathbf{X}}^{*}$ each of size n are obtained with parameters, respectively, θ and ${\theta }_i^{*},$ and the corresponding EDIs are calculated, $i=1,\dots ,{21}^2,$ for $n=100,1000,30,000,100,000.$ For $n=100,$ several circles in the EDI-graphic (in Figure 3) “jump” and have y-values near 0.5, as the distance on the x-axis from $\theta =(1.2,1.6)$ increases. For $n=1000,$ the ${\theta }^{*}$-values that indicate nonidentifiability of θ have EDI values near 0.5 and the sum of the ${\theta }^{*}$ parameters is $2.8,$ as with $\theta .$ The ${\theta }^{*}$ with EDI-values near 0.2 indicate additional nonidentifiable parameters and have sum of parameters 2.7.

Example 4.3.

Let $N(\mu ,{\sigma }^2)$ denote Normal cdf with mean, $\mu ,$ and variance, ${\sigma }^2.$ Consider the normal mixture model, $\mathcal{F},$ with known variance, ${\sigma }^2=1,$ and parameter $\theta =(p,{\mu }_1,{\mu }_2),$ which is nonidentifiable, $$\begin{array}{c}\mathcal{F}=\{\mathit{\text{pN}}({\mu }_1,1)+(1-p)N({\mu }_2,1),(p,{\mu }_1,{\mu }_2)\in \Theta \\ =[0,1]\times [0,2]\times [0,2]\}.\end{array}$$

EDI is used to confirm algorithmically nonidentifiability of $\theta =(0.25,0.4,1.2).$ $S_p=\{0,0.25,0.5,0.75,1\}$ is a sieve for $[0,1]$ and $S_{{\mu }_1}=S_{{\mu }_2}=\{0,0.4,0.8,1.2,1.6,2\}$ are sieves for $[0,2],$ and (19) $${\Theta }^{*}=S_p\times S_{{\mu }_1}\times S_{{\mu }_2}=\{{\theta }_i^{*}:1\le i\le 180\}$$(19) is sieve for $\Theta ,$ with $\theta ={\theta }_{46}^{*}.$ ${\theta }_{128}^{*}=(0.75,1.2,0.4)$ makes θ nonidentifiable. Using $M=100,$ EDI-graphics for $n=100,500,15,000,30,000,$ appear in Figure 4. When $n=100,$ the EDI-values are spread in $[0,0.5],$ but as n increases, for example, when $n=500,$ several EDI-values are near zero. When $n=15,000,$ there are 6 EDI-values far from 0. Those with similar values correspond to nonidentifiable ${\theta }_{46}^{*}$ and ${\theta }_{128}^{*},$ ${\theta }_{63}^{*}=(0.25,1.6,0.8)$ and ${\theta }_{125}^{*}=(0.75,0.8,1.6),{\theta }_{88}^{*}=(0.5,0.8,1.2)$ and ${\theta }_{93}^{*}=(0.5,1.2,0.8).$ When $n=30,000,$ the EDI-values of the last two are nearly zero, those corresponding to indices 63 and 125 decrease, while those for indices 46 and 128 remain near 0.5, as expected.

In Figure 5, simulations are repeated when the means’ coordinates in $\theta =(0.25,0.35,1.25)$ are not in ${\Theta }^{*}.$ θ’s closest element in the sieve is ${\theta }_{46}^{*}=(0.25,0.4,1.2).$ The same pattern with Figure 4 is observed in Figure 5, except for the two larger EDI-values which decrease slower than the other EDI-values towards zero as n increases. Nonidentifiable ${\theta }_{63}^{*},{\theta }_{125}^{*}$ and ${\theta }_{88}^{*},{\theta }_{93}^{*}$ have similar EDI-values, with those of the last two vanishing for $n\ge 15,000.$

EDI-graphics are used also for studying discrimination and identifiability of the parameters in data-generating machines which are compared. Such examples follow.

Example 4.4.

Tukey’s g-and-h model (DGE1) and the g-and-k model (DGE2) are compared, g- locally with EDI and PPVI. Normal sample, Z is used to obtain ${\mathbf{X}}_1(g,h)$ and ${\mathbf{X}}_2(g,k)$, respectively, from (2) and (3) with $g=5,h=k=2.5,a=0,b=1,c=\mathrm{0.8.}$ Normal sample, ${\mathbf{Z}}^{*},$ also of size n, is used to obtain similarly ${\mathbf{X}}_1^{*}(g^{*},h),{\mathbf{X}}_2^{*}(g^{*},k)$ with different $g^{*}=3.$ The p-value for the Kolmogorov-Smirnov test is obtained for both DGE1 and DGE2 models. Both experiments are repeated M = 1000 times for $n=50,100,200,500,1000,1500,2500,5000,10,000,35,000,40,000.$ For each n, EDIs for DGE1 and DGE2 are computed and counters measure the number of times out of M the p-value for DGE2 is smaller than, or larger than, that of DGE1. The results appear in Table 2. Comparison of EDIs indicate that for the g and $g^{*}$ used and $n<40,000,$ Tukey’s g-and-h model has greater g-discrimination than the g-and-k model. For example, for 5% significant difference, n = 1500 is needed for the g-and-h model, and n = 2500 is needed for the g-and-k model. Comparison of PPVIs indicate that for $200\le n\le 10,000$ Tukey’s g-and-h model has greater g-discrimination than the g-and-k model, with $\text{PPVI}(g\text{‐and‐}h,g\text{‐and‐}k)$ decreasing to zero and $\text{PPVI}(g\text{‐and‐}k,g\text{‐and‐}h)$ increasing to one. A sample of size $n=40,000$ is needed for the parameters g = 5 and $g^{*}=3$ of the g-and-k model to be discriminated as in Tukey’s g-and-h model.

Remark 4.1.

The results in Example 4.4 for the g-and-k model suggested comparing also smooth histograms for this model. Visually, there is no discrimination between overlayed g-and-k smooth histograms with parameters, respectively, g = 5 and $g^{*}=3.5$ for various sample sizes less than or equal to $n=10,000$ and $\mathbf{Z}={\mathbf{Z}}^{*}.$ We did not use $n>10,000.$

From the results in Example 4.4, Tukey’s g-and-h model has better local discrimination than the g-and-k model, unless the sample size is very large. The findings in Table 2 extend those in Rayner and MacGillivray (2002), based on the data and not on a particular estimate, for example, the MLE. The results are confirmed g- globally below, with EDI-graphics.

Example 4.5.

Tukey’s g- and-h and the g- and-k models are examined as Learning Machines with same parameters $a=0,b=1,\theta =(g,h)=(g,k)=(5,2.5).$ It is assumed g is unknown, but $h=k=2.5$ is known. To use the same R programs we considered parameter spaces for g and $h=k,$ respectively, $[2,5]$ and $[2.5,2.5].$ The sieve for the first parameter space, with $ϵ=0.4,$ is $g_j^{*}=2+0.6(j-1),1\le j\le 6,$ and $h_k^{*}=2.5,1\le k\le 6,$ therefore the Θ-sieve consists of ${\theta }_i^{*},i=1,\dots ,36.$

M = 100 independent samples $\mathbf{X}$ and ${\mathbf{X}}^{*}$ each of size n are obtained with parameters, respectively, θ and ${\theta }_i^{*}$ in both models, and the corresponding EDIs are calculated, $i=1,\dots ,36,$ for $n=200,1000.$ EDI-graphics for both models appear in Figure 6 indicating identifiability. Comparison of the EDI-graphics for the same n value, indicates that Tukey’s g- and-h model has better parameter discrimination than the g-and-k model. The findings confirm graphically the results in Example 4.4.

Example 4.6.

Similar set-up as in Example 4.3 is used, with only difference that data, X, is obtained from the Normal learning machine that is convex combination of normal densities, (20) $$X=f(Z,\theta )=p\varphi (Z-{\mu }_1)+(1-p)\varphi (Z-{\mu }_2).$$(20)

$(p,{\mu }_1,{\mu }_2)$ is element of $\Theta =[0,1]\times [0,2]\times [0,2],$ with the same sieve, ${\Theta }^{*}.$ The model parameter $\theta =(0.4,1.2,1.6)={\theta }_{46}^{*},$ and Z is a standard Normal random variable with density $\phi$. M = 100 learning samples are used for EDI and $n=100,500,15000,30000.$A similar experiment is examined for a Sigmoid learning machine, with $\varphi$ in (19) replaced by $$s(u)=\frac{1}{1+e^{-u}},u\in R.$$

Figures 7 and 8 indicate nonidentifiability and that the parameters are better discriminated with the data from the Normal learning machine.

5 Conclusion

For Black-Box models, parameter identifiability cannot be confirmed. For learning machines, nonidentifiability is ubiquitous and the resulting difficulty in the estimation of the parameters and the reproducibility of the learned models are not yet quantified. The Empirical Discrimination Index, $\text{EDI}(\theta ,{\theta }^{*};n),$ is used for ${\theta }^{*}$ in a sieve of $\Theta ,$ to confirm with EDI-graphics almost surely identifiability of all parameters θ in $\Theta ,$ or nonidentifiability. EDI and the Proportion of p-values Index (PPVI) are also useful tools for identifying among samplers and learning machines those that have greater parameter discrimination for a given sample size, n, thus, leading to more efficient estimates.

Supplementary Materials

Proofs and R-functions used in Examples 4.1–4.6.

Acknowledgments

Many thanks are due to Professor Faming Liang and Professor Galin Jones, Editors, who have handled, respectively, the original submission and the revisions. Thanks are due to the referees, for their comments that improved the presentation of the paper, and to Mr. Yongzhen Feng, Tsinghua University, for the suggestions to improve readability.

Disclosure Statement

The authors report there are no competing interests to declare.

Notes

1 When $\Theta =R^p,$ without preliminary estimation, ${\Theta }^{*}$ has countably infinite elements.

2 Since one DGE is studied.