User Documentation
 All Files Functions Groups
Multinomial Logistic Regression
+ Collaboration diagram for Multinomial Logistic Regression:
About:

Multinomial logistic regression is a widely used regression analysis tool that models the outcomes of categorical dependent random variables (denoted \( Y \in \{ 0,1,2 \ldots k \} \)). The models assumes that the conditional mean of the dependant categorical variables is the logistic function of an affine combination of independent variables (usually denoted \( \boldsymbol x \)). That is,

\[ E[Y \mid \boldsymbol x] = \sigma(\boldsymbol c^T \boldsymbol x) \]

for some unknown vector of coefficients \( \boldsymbol c \) and where \( \sigma(x) = \frac{1}{1 + \exp(-x)} \) is the logistic function. Multinomial Logistic regression finds the vector of coefficients \( \boldsymbol c \) that maximizes the likelihood of the observations.

Let

By definition,

\[ P[Y = y_i | \boldsymbol x_i] = \sigma((-1)^{y_i} \cdot \boldsymbol c^T \boldsymbol x_i) \,. \]

Maximizing the likelihood \( \prod_{i=1}^n \Pr(Y = y_i \mid \boldsymbol x_i) \) is equivalent to maximizing the log-likelihood \( \sum_{i=1}^n \log \Pr(Y = y_i \mid \boldsymbol x_i) \), which simplifies to

\[ l(\boldsymbol c) = -\sum_{i=1}^n \log(1 + \exp((-1)^{y_i} \cdot \boldsymbol c^T \boldsymbol x_i)) \,. \]

The Hessian of this objective is \( H = -X^T A X \) where \( A = \text{diag}(a_1, \dots, a_n) \) is the diagonal matrix with \( a_i = \sigma(\boldsymbol c^T \boldsymbol x) \cdot \sigma(-\boldsymbol c^T \boldsymbol x) \,. \) Since \( H \) is non-positive definite, \( l(\boldsymbol c) \) is convex. There are many techniques for solving convex optimization problems. Currently, logistic regression in MADlib can use:

We estimate the standard error for coefficient \( i \) as

\[ \mathit{se}(c_i) = \left( (X^T A X)^{-1} \right)_{ii} \,. \]

The Wald z-statistic is

\[ z_i = \frac{c_i}{\mathit{se}(c_i)} \,. \]

The Wald \( p \)-value for coefficient \( i \) gives the probability (under the assumptions inherent in the Wald test) of seeing a value at least as extreme as the one observed, provided that the null hypothesis ( \( c_i = 0 \)) is true. Letting \( F \) denote the cumulative density function of a standard normal distribution, the Wald \( p \)-value for coefficient \( i \) is therefore

\[ p_i = \Pr(|Z| \geq |z_i|) = 2 \cdot (1 - F( |z_i| )) \]

where \( Z \) is a standard normally distributed random variable.

The odds ratio for coefficient \( i \) is estimated as \( \exp(c_i) \).

The condition number is computed as \( \kappa(X^T A X) \) during the iteration immediately preceding convergence (i.e., \( A \) is computed using the coefficients of the previous iteration). A large condition number (say, more than 1000) indicates the presence of significant multicollinearity.

The multinomial logistic regression uses a default reference category of zero, and the regression coefficients in the output are in the order described below. For a problem with \( K \) dependent variables \( (1, ..., K) \) and \( J \) categories \( (0, ..., J-1) \), let \( {m_{k,j}} \) denote the coefficient for dependent variable \( k \) and category \( j \). The output is \( {m_{k_1, j_0}, m_{k_1, j_1} \ldots m_{k_1, j_{J-1}}, m_{k_2, j_0}, m_{k_2, j_1}, \ldots m_{k_2, j_{J-1}} \ldots m_{k_K, j_{J-1}}} \). The order is NOT CONSISTENT with the multinomial regression marginal effect calculation with function marginal_mlogregr. This is deliberate because the interfaces of all multinomial regressions (robust, clustered, ...) will be moved to match that used in marginal.

Input:

The training data is expected to be of the following form:

{TABLE|VIEW} sourceName (
    ...
    dependentVariable INTEGER,
    independentVariables FLOAT8[],
    ...
)
Usage:
  • The number of independent variables cannot exceed 65535.
  • The reference category ranges from [0, numCategories-1]. The default reference category is zero.
  • Get vector of coefficients \( \boldsymbol c \) and all diagnostic statistics:
    SELECT * FROM mlogregr(
        'sourceName',
        'dependentVariable',
        'independentVariables' [,
        numberOfIterations [,
        'optimizer' [,
        precision, [,
        ref_category]]]]
    );
    Output:
    ref_category | coef | log_likelihood | std_err | z_stats | p_values | odds_ratios | condition_no | num_iterations
    ----+------+----------------+---------+---------+----------+-------------+--------------+--------------- ...
  • Get vector of coefficients \( \boldsymbol c \):
    SELECT (mlogregr('sourceName', 'dependentVariable', 'independentVariables')).coef;
  • Get a subset of the output columns, e.g., only the array of coefficients \( \boldsymbol c \), the log-likelihood of determination \( l(\boldsymbol c) \), and the array of p-values \( \boldsymbol p \):
    SELECT coef, log_likelihood, p_values
    FROM mlogregr('sourceName', 'dependentVariable', 'independentVariables');

Note that the categories are encoded as integers with values from {0, 1, 2,..., numCategories-1}

Examples:
  1. Create the sample data set:
    sql> SELECT * FROM data;
                      r1                      | val
    ---------------------------------------------+-----
     {1,3.01789340097457,0.454183579888195}   | 1
     {1,-2.59380532894284,0.602678326424211}  | 0
     {1,-1.30643094424158,0.151587064377964}  | 1
     {1,3.60722299199551,0.963550757616758}   | 1
     {1,-1.52197745628655,0.0782248834148049} | 1
     {1,-4.8746574902907,0.345104880165309}   | 0
    ...
    
  2. Run the multi-logistic regression function:
    sql> \x on
    Expanded display is off.
    sql> SELECT * FROM mlogregr('data', 'val', '2', 'r1', 100, 'irls', 0.001);
    -[ RECORD 1 ]--+--------------------------------------------------------------
    coef           | {5.59049410898112,2.11077546770772,-0.237276684606453}
    log_likelihood | -467.214718489873
    std_err        | {0.318943457652178,0.101518723785383,0.294509929481773}
    z_stats        | {17.5281667482197,20.7919819024719,-0.805666162169712}
    p_values       | {8.73403463417837e-69,5.11539430631541e-96,0.420435365338518}
    odds_ratios    | {267.867942976278,8.2546400100702,0.788773016471171}
    condition_no   | 179.186118573205
    num_iterations | 9
Literature:

A collection of nice write-ups, with valuable pointers into further literature:

[1] Annette J . Dobson: An Introduction to Generalized Linear Models, Second Edition. Nov 2001

[2] Cosma Shalizi: Statistics 36-350: Data Mining, Lecture Notes, 18 November 2009, http://www.stat.cmu.edu/~cshalizi/350/lectures/26/lecture-26.pdf

[3] Srikrishna Sridhar, Mark Wellons, Caleb Welton: Multilogistic Regression: Notes and References, Jul 12 2012, http://www.cs.wisc.edu/~srikris/mlogit.pdf

[4] Scott A. Czepiel: Maximum Likelihood Estimation of Logistic Regression Models: Theory and Implementation, Retrieved Jul 12 2012, http://czep.net/stat/mlelr.pdf

See Also
File multilogistic.sql_in (documenting the SQL functions)