clusterstopping.pl

- NAME
- SYNOPSIS
- USAGE
- INPUT

- Required Arguments:

- INPUTFILE

- Optional Arguments:

- -prefix PRE
- -measure MSR
- -space SP
- -delta INT
- -clmethod CL
- -crfun CR
- -rowmodel RMOD
- -threspk1 NUM
- -precision NUM

- Adapted Gap Statistic related options:

- -B NUM
- -typeref TYP
- -seed NUM
- -percentage NUM

- Other Options :

- -help
- -version
- -verbose

- OUTPUT
- DETAILS ABOUT THE CLUSTER-STOPPING MEASURES

- Formulation for PK1:
- Formulation for PK2:
- Formulation for PK3:
- Adapted Gap Statistic:

- DETAILS ABOUT THE INPUT MATRIX FORMAT
- AUTHOR
- COPYRIGHT

# NAME

clusterstopping.pl

# SYNOPSIS

Predicts the optimal number of clusters for the given data.

# USAGE

clusterstopping.pl [OPTIONS] INPUTFILE

This script tries to find the optimal number of clusters for the given INPUTFILE.

# INPUT

## Required Arguments:

### INPUTFILE

Matrix file containing either: * context vectors (in dense or sparse format) * similarity values between contexts

## Optional Arguments:

#### --prefix PRE

Specify a prefix to be used for the output filenames.

If prefix is not specified then prefix is created by concatenating time stamp to the string ``expr''.

#### --measure MSR

Specify the cluster stopping measure to be used.

(Further details about the measures given below under the Section ``DETAILS ABOUT THE CLUSTER-STOPPING MEASURES'')

The possible option values: pk1 - PK1 measure pk2 - PK2 measure pk3 - PK3 measure [Default] gap - Adapted Gap Statistic pk - All the three PK measures all - All the four measures - PK1, PK2, PK3 and Gap

#### --space SP

Specify if the clustering should be performed in vector or similarity space.

The possible option values: vector [Default] similarity

#### --delta INT

NOTE: Delta value can only be a positive integer value.

Specify 0 to stop the iterating clustering process when two consecutive crfun values are exactly equal. This is the default setting when the crfun values are integer/whole numbers.

Specify non-zero positive integer to stop the iterating clustering process when the diffference between two consecutive crfun values is less than or equal to this value. However, note that the integer value specified is internally shifted to capture the difference in the least significant digit of the crfun values when these crfun values are fractional. For example: For crfun = 1.23e-02 & delta = 1 will be transformed to 0.0001 For crfun = 2.45e-01 & delta = 5 will be transformed to 0.005 The default delta value when the crfun values are fractional is 1.

However if the crfun values are integer/whole numbers (exponent >= 2) then the specified delta value is internally shifted only until the least significant digit in the scientific notation. For example: For crfun = 1.23e+04 & delta = 2 will be transformed to 200 For crfun = 2.45e+02 & delta = 5 will be transformed to 5 For crfun = 1.44e+03 & delta = 1 will be transformed to 10

#### --clmethod CL

Specifies the clustering method.

The possible option values: rb - Repeated Bisections [Default] rbr - Repeated Bisections for by k-way refinement direct - Direct k-way clustering agglo - Agglomerative clustering bagglo - Partitional biased Agglomerative clustering [Only in vector space]

For large amount of data, 'rb', 'rbr' or 'direct' are recommended.

#### --crfun CR

Selects the criteria function for Clustering. The meanings of these criteria functions are explained in Cluto's manual.

The possible option values: i1 - I1 Criterion function i2 - I2 Criterion function [Default] h1 - H1 Criterion function h2 - H2 Criterion function e1 - E1 Criterion function

    =head4 --sim SIM

Specifies the similarity measure to be used

The possible option values: cos - Cosine [Default] corr - Correlation Coefficient

    NOTE: This option can be used only in vector space.

#### --rowmodel RMOD

The option is used to specify the model to be used to scale every column of each row. (For further details please refer Cluto manual)

The possible option values: none - no scaling is performed [Default] maxtf - post scaling the values are between 0.5 and 1.0 sqrt - square-root of actual values log - log of actual values

    =head4 --colmodel CMOD

The option is used to specify the model to be used to (globally) scale each column across all rows. (For further details please refer Cluto manual)

The possible option values: none - no scaling is performed [Default] idf - scaling according to inverse-document-frequency

#### --threspk1 NUM

The threshold value that should be used by the PK1 measure to predict the k value.

Default = -0.7

#### --precision NUM

Specifies the precision to be used.

Default: 4

### Adapted Gap Statistic related options:

#### --B NUM

The number of replicates/references to be generated.

Default: 1

#### --typeref TYP

Specifies whether to generate B replicates from a reference or to generate B references.

The possible option values: rep - replicates [Default] ref - references

#### --seed NUM

The seed to be used with the random number generator.

Default: No seed is set.

#### --percentage NUM

Specifies the percentage confidence to be reported in the log file. Since Statistics::Gap uses parametric bootstrap method for reference distribution generation, it is critical to understand the interval around the sample mean that could contain the population (``true'') mean and with what certainty.

Default: 90

### Other Options :

#### --help

Displays the quick summary of program options.

#### --version

Displays the version information.

#### --verbose

Displays to STDERR the current program status.

# OUTPUT

- prefix.pk1

     Contains the crfun values, PK1 values and the predicted k.

- prefix.pk2

     Contains the crfun values, PK2 values and the predicted k.

- prefix.pk3

     Contains the crfun values, PK3 values and the predicted k.

- prefix.gap

     Contains the crfun values, delta values and the predicted k.

- prefix.gap.log Contains a table of values for Gap(k), Obs(crfun(k)), Exp(crfun(k)), sd(k), `s(k)` and Confidence interval. Also contains individual crfun values for each of the B replicates/references for every value of K.

The following files (*.dat) files are created to facilitate creation of plots if required.

- prefix.cr.dat

     Contains the crfun values.

- prefix.pk1.dat

     Contains the PK1 values.

- prefix.pk2.dat

     Contains the PK2 values.

- prefix.pk3.dat

     Contains the PK3 values.

- prefix.exp.dat

     Contains the Exp(crfun(k)) values generated for Gap Statistics.

- prefix.gap.dat

     Contains the gap(k) values.

# DETAILS ABOUT THE CLUSTER-STOPPING MEASURES

Each of the four cluster-stopping measure PK1, PK2, PK3 and Gap try to predict the optimal number of clusters for the provided data (represented in the form of vectors or similarity values).

Following is a description about each of the measures and how they select a k value.

## Formulation for PK1:

               crfun[m] - mean(crfun[1...deltaM])
     PK1[m] = -----------------------------------
                    std(crfun[1...deltaM])

     where m = 2,...deltaM
     deltaM is a m value beyond which the crfun value does not change much.

k selection strategy: If m is the first value for which the PK1 score is greater than the threshold value then m-1 value is selected as the k value when using one of the maximization criterion functions like I2, H2 etc. While using minimization criterion functions like E1, if m the first value for which the PK1 score is less than the threshold value then m-1 is selected as the k value.

## Formulation for PK2:

                crfun[m]
     PK2[m] = ------------
               crfun[m-1]

where m = 2,...deltaM

k selection strategy: One standard deviation interval for the set of PK2[m] values id calculated and the m value for which the PK2 score is outside this interval but is still closest to the interval (from outside) is selected as the k value.

## Formulation for PK3:

                    2 * crfun[m]
     PK3[m] = -------------------------
               crfun[m-1] + crfun[m+1]

    where m = 2,...deltaM

k selection strategy: Similar to that of PK2. One standard deviation of the PK3[m] scores is computed and the m value for which the PK3 score is outside this interval but is still closest to the interval (from outside) is selected as the k value.

## Adapted Gap Statistic:

Adapted Gap tries to predict the optimal number of clusters by comparing the criterion function values that one gets for the observed/given data with the criterion function values that one gets for a random data. It uses hypothesis testing in this process where the null hypothesis is that the optimal number of clusters for the given data is 1 and the aim is to refute this null hypothesis if the given data exhibits enough pattern.

The plot of crfun values for given data is compared with the plot of crfun values for random or reference data. The m value corresponding to the first point of maximum difference between the two plots is chosen as the predicted k when using the maximization crfuns. While for the minimization crfuns, the m value corresponding to the first point of minimum difference between the two plots is choosen as the predicted k value.

We provide two methods for the generation of reference data:

1. One can choose to generate a random dataset over the observed distribution by holding the row and the column marginals fixed and then generating B replicates from this random dataset using Monte Carlo sampling.

    2. Or to generate B random datasets over the observed distribution by holding
    the row and the column marginals fixed.

Please refer to http://search.cpan.org/dist/Algorithm-RandomMatrixGeneration/ to learn more about generating random dataset over the observed distribution by holding the row and the column marginals fixed.

# DETAILS ABOUT THE INPUT MATRIX FORMAT

The input matrix can be in either dense or sparse format. The cell values can be integer or real. Depending upon the value specified for the space parameter the header in the input file (first line) changes.

Example of input matrix in dense format when space=vector: The first line specifies the dimensions - #rows #cols From the second line the actual matrix follows.

     6 5
     1.3       2       0       0       3
     2.1       0       4       2.7     0
     1.3       2       0       0       3
     2.1       0       4       2.7     0
     1.3       2       0       0       3
     2.1       0       4       2.7     0

Example of input matrix in dense format when space=similarity: The matrix, when in similarity space, is square and symmetric. The first line specifies the dimensions - #rows/#cols From the second line the actual matrix follows.

     5
     1.0000   0.3179   0.5544   0.2541   0.4431
     0.3179   1.0000   0.1386   0.4599   0.5413
     0.5544   0.1386   1.0000   0.5143   0.2186
     0.2541   0.4599   0.5143   1.0000   0.5148
     0.4431   0.5413   0.2186   0.5148   1.0000

Example of input matrix in sparse format when space=vector: The first line specifies the dimensions & number of non-zero elements - #rows #cols #nonzeroElem

From the second line the matrix contents follow. Only non-zero elements are specified. Thus the elements are specified as pairs of - #col elem. The row number is implied by the (line number-1).

     8 10 41
     1 3 4 2 8 2 10 1
     1 1 2 5 3 1 5 2 7 1 9 2
     1 3 4 2 8 2 10 1
     1 1 2 5 3 1 5 2 7 1 9 2
     1 3 4 2 8 2 10 1
     1 1 2 5 3 1 5 2 7 1 9 2
     2 4 3 1 4 2 5 5 7 1 9 1
     2 4 3 1 4 2 5 5 7 1 9 1

Example of input matrix in sparse format when space=similarity: The first line specifies the dimensions & number of non-zero elements - #rows/#cols #nonzeroElem The matrix format is same as explained above.

     5 15
     1 1.0000 3 0.5544 5 0.4431
     2 1.0000 3 0.1386 4 0.4599 5 0.5413
     1 0.5544 2 0.1386 3 1.0000
     2 0.4599 4 1.0000
     1 0.4431 2 0.5413 5 1.0000

# AUTHOR

Anagha Kulkarni, University of Minnesota, Duluth.

Ted Pedersen, University of Minnesota, Duluth.

# COPYRIGHT

Copyright (c) 2006-2008,

     Ted Pedersen, University of Minnesota, Duluth.
     tpederse@d.umn.edu

     Anagha Kulkarni, University of Minnesota, Duluth.
     kulka020@d.umn.edu

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to

     The Free Software Foundation, Inc.,
     59 Temple Place - Suite 330,
     Boston, MA  02111-1307, USA.