Contents

Qualitative variation: Difference between revisions

Revision as of 10:47, 29 May 2013

There are various indices of qualitative variation; a number are summarized and devised by Wilcox (Wilcox 1967), (Wilcox 1973), who requires the following standardization properties to be satisfied:

• Variation varies between 0 and 1.
• Variation is 0 if and only if all cases belong to a single category.
• Variation is 1 if and only if cases are evenly divided across all category.^[1]

In particular, the value of these standardized indices does not depend on the number of categories or number of samples.

For any index, the closer to uniform the distribution, the larger the variance, and the larger the differences in frequencies across categories, the smaller the variance.

Indices of qualitative variation are in this sense complementary to information entropy, which is maximized when all cases belong to a single category and minimized in a uniform distribution, but they are not complementary in the sense of a particular IQV equaling 1 minus entropy. Indeed, information entropy can be used as an index of qualitative variation.

One characterization of a particular index of qualitative variation (IQV) is as a ratio of observed differences to maximum differences.

Formulae

Wilcox gives a number of formulae for various indices of QV (Wilcox 1973), the first, which he designates DM for "Deviation from the Mode", is a standardized form of the variation ratio, and is analogous to variance as deviation from the mean.

The formula for this is derived as follows:

${\displaystyle M=\sum _{i=1}^{K}(f_{m}-f_{i})}$

where f_m is the modal frequency, K is the number of catagories and f_i is the frequency of the i^th group.

This can be simplified to

${\displaystyle M=Kf_{m}-N}$

where N is the total size of the sample.

Freeman's index is^[2]

${\displaystyle v=1-{\frac {f_{m}}{N}}}$

This is related to M as follows:

${\displaystyle {\frac {({\frac {f_{m}}{N}})-{\frac {1}{K}}}{{\frac {N}{K}}{\frac {(K-1)}{N}}}}={\frac {M}{N(K-1)}}}$

The ModVR is then defined as

${\displaystyle ModVR=1-{\frac {Kf_{m}-N}{N(K-1)}}}$

Low values of ModVR correspond to small amount of variation and high values to larger amounts of variation.

One formula for IQV,^[3] given as M2 in (Gibbs 1975, p. 472) harv error: no target: CITEREFGibbs1975 (help) is:

${\displaystyle {\text{IQV}}:={\frac {K}{K-1}}\left(1-\sum _{i=1}^{K}p_{i}^{2}\right)}$

where K is the number of categories, and ${\displaystyle p_{i}=f_{i}/N}$ is the proportion of observations that fall in a given category i. The factor of ${\displaystyle {\frac {K}{K-1}}}$ is for standardization.

The unstandardized index, ${\displaystyle \left(1-\sum _{i=1}^{K}p_{i}^{2}\right)}$, denoted as M1 (Gibbs 1975, p. 471) harv error: no target: CITEREFGibbs1975 (help), can be interpreted as the likelihood that a random pair of samples will belong to the same category (Lieberson 1969, p. 851), so this formula for IQV is a standardized likelihood of a random pair falling in the same category. M1 and M2 can be interpreted in terms of variance of a multinomial distribution (Swanson 1976) (there called an "expanded binomial model").

Related indices

The sum

${\displaystyle \sum _{i=1}^{K}p_{i}^{2}}$

has also found application. This is known as the Simpson index in ecology and as the Herfindahl index or the Herfindahl-Hirschman index (HHI) in economics. A variant of this is known as the Hunter-Gaston index in microbiology^[4]

M1

The M1 statistic defined above has been proposed several times in a number of different settings under a variety of names. These include Gini's index of mutability,^[5] Simpson's measure of diversity,^[6] Bachi's index of linguistic homogeneity^[7], Mueller and Schuessler's index of qualitative variation,^[8] Gibbs and Martin's index of industry diversification,^[9] Lieberson's index.^[10] and Blau's index index in sociology, psychology and management studies.^[11] The formulation of all these indices are identical.

Simpson's D is defined as

${\displaystyle D=1-\sum {\frac {n_{i}(n_{i}-1)}{n(n-1)}}}$

where n is the total sample size and n_i is the number of items in the i^th category.

For large n we have

${\displaystyle u\sim 1-\sum _{i=1}^{K}p_{i}^{2}}$

Another statistic that has been proposed is the coefficient of unalikeability which ranges between 0 and 1.^[12]

${\displaystyle u={\frac {c(x,y)}{n^{2}-n}}}$

where n is the sample size and c(x,y) = 1 if x and y are alike and 0 otherwise.

For large n we have

${\displaystyle u\sim 1-\sum _{i=1}^{K}p_{i}^{2}}$

where K is the number of categories.

M2

Greenberg's monolingual non weighted index of linguistic diversity,^[13] is the M2 statistic defined above.

Other measures

Berger-Parker index

The Berger-Parker index equals the maximum ${\displaystyle p_{i}}$ value in the dataset, i.e. the proportional abundance of the most abundant type. This corresponds to the weighted generalized mean of the ${\displaystyle p_{i}}$ values when q approaches infinity, and hence equals the inverse of true diversity of order infinity (1/^∞D).

Rényi entropy

The Rényi entropy is a generalization of the Shannon entropy to other values of q than unity. It can be expressed:

${\displaystyle {}^{q}H={\frac {1}{1-q}}\;\ln \left(\sum _{i=1}^{K}p_{i}^{q}\right)}$

which equals

${\displaystyle {}^{q}H=\ln \left({1 \over {\sqrt[{q-1}]{\sum _{i=1}^{K}p_{i}p_{i}^{q-1}}}}\right)=\ln({}^{q}\!D)}$

This means that taking the logarithm of true diversity based on any value of q gives the Rényi entropy corresponding to the same value of q.

The value of ${\displaystyle {}^{q}\!D}$ is also known as the Hill number.^[14]

Evaluation of indices

Different indices give different values of variation, and may be used for different purposes: several are used and critiqued in the sociology literature especially.

If one wishes to simply make ordinal comparisons between samples (is one sample more or less varied than another), the choice of IQV is relatively less important, as they will often give the same ordering.

In some cases it is useful to not standardize an index to run from 0 to 1, regardless of number of categories or samples (Wilcox 1973, pp. 338), but one generally so standardizes it.

See also

• Statistical dispersion
• Diversity index
• Variation ratio
• Information entropy

Notes

References

• Gibbs, Jack P.; Poston, Jr., Dudley L. (1975), "The Division of Labor: Conceptualization and Related Measures", Social Forces, 53 (3): 468–476, doi:10.2307/2576589, JSTOR 2576589 {{citation}}: Unknown parameter |month= ignored (help)

• Lieberson, Stanley (1969), "Measuring Population Diversity", American Sociological Review, 34 (6): 850–862, doi:10.2307/2095977, JSTOR 2095977 {{citation}}: Unknown parameter |month= ignored (help)

• Swanson, David A. (1976), "A Sampling Distribution and Significance Test for Differences in Qualitative Variation", Social Forces, 55 (1): 182–184, doi:10.2307/2577102, JSTOR 2577102 {{citation}}: Unknown parameter |month= ignored (help)

• Wilcox, Allen R. (1967), Indices of qualitative variation (PDF)

• Wilcox, Allen R. (1973), "Indices of Qualitative Variation and Political Measurement", The Western Political Quarterly, 26 (2): 325–343, doi:10.2307/446831, JSTOR 446831 {{citation}}: Unknown parameter |month= ignored (help)