MADlib
1.0 A newer version is available
User Documentation
|
Multinomial logistic regression is a widely used regression analysis tool that models the outcomes of categorical dependent random variables (denoted \( Y \in \{ 0,1,2 \ldots k \} \)). The models assumes that the conditional mean of the dependant categorical variables is the logistic function of an affine combination of independent variables (usually denoted \( \boldsymbol x \)). That is,
\[ E[Y \mid \boldsymbol x] = \sigma(\boldsymbol c^T \boldsymbol x) \]
for some unknown vector of coefficients \( \boldsymbol c \) and where \( \sigma(x) = \frac{1}{1 + \exp(-x)} \) is the logistic function. Multinomial Logistic regression finds the vector of coefficients \( \boldsymbol c \) that maximizes the likelihood of the observations.
Let
By definition,
\[ P[Y = y_i | \boldsymbol x_i] = \sigma((-1)^{y_i} \cdot \boldsymbol c^T \boldsymbol x_i) \,. \]
Maximizing the likelihood \( \prod_{i=1}^n \Pr(Y = y_i \mid \boldsymbol x_i) \) is equivalent to maximizing the log-likelihood \( \sum_{i=1}^n \log \Pr(Y = y_i \mid \boldsymbol x_i) \), which simplifies to
\[ l(\boldsymbol c) = -\sum_{i=1}^n \log(1 + \exp((-1)^{y_i} \cdot \boldsymbol c^T \boldsymbol x_i)) \,. \]
The Hessian of this objective is \( H = -X^T A X \) where \( A = \text{diag}(a_1, \dots, a_n) \) is the diagonal matrix with \( a_i = \sigma(\boldsymbol c^T \boldsymbol x) \cdot \sigma(-\boldsymbol c^T \boldsymbol x) \,. \) Since \( H \) is non-positive definite, \( l(\boldsymbol c) \) is convex. There are many techniques for solving convex optimization problems. Currently, logistic regression in MADlib can use:
We estimate the standard error for coefficient \( i \) as
\[ \mathit{se}(c_i) = \left( (X^T A X)^{-1} \right)_{ii} \,. \]
The Wald z-statistic is
\[ z_i = \frac{c_i}{\mathit{se}(c_i)} \,. \]
The Wald \( p \)-value for coefficient \( i \) gives the probability (under the assumptions inherent in the Wald test) of seeing a value at least as extreme as the one observed, provided that the null hypothesis ( \( c_i = 0 \)) is true. Letting \( F \) denote the cumulative density function of a standard normal distribution, the Wald \( p \)-value for coefficient \( i \) is therefore
\[ p_i = \Pr(|Z| \geq |z_i|) = 2 \cdot (1 - F( |z_i| )) \]
where \( Z \) is a standard normally distributed random variable.
The odds ratio for coefficient \( i \) is estimated as \( \exp(c_i) \).
The condition number is computed as \( \kappa(X^T A X) \) during the iteration immediately preceding convergence (i.e., \( A \) is computed using the coefficients of the previous iteration). A large condition number (say, more than 1000) indicates the presence of significant multicollinearity.
The multinomial logistic regression uses a default reference category of zero, and the regression coefficients in the output are in the order described below. For a problem with \( K \) dependent variables \( (1, ..., K) \) and \( J \) categories \( (0, ..., J-1) \), let \( {m_{k,j}} \) denote the coefficient for dependent variable \( k \) and category \( j \). The output is \( {m_{k_1, j_0}, m_{k_1, j_1} \ldots m_{k_1, j_{J-1}}, m_{k_2, j_0}, m_{k_2, j_1}, \ldots m_{k_2, j_{J-1}} \ldots m_{k_K, j_{J-1}}} \). The order is NOT CONSISTENT with the multinomial regression marginal effect calculation with function marginal_mlogregr. This is deliberate because the interfaces of all multinomial regressions (robust, clustered, ...) will be moved to match that used in marginal.
The training data is expected to be of the following form:
{TABLE|VIEW} sourceName ( ... dependentVariable INTEGER, independentVariables FLOAT8[], ... )
SELECT * FROM mlogregr( 'sourceName', 'dependentVariable', 'independentVariables' [, numberOfIterations [, 'optimizer' [, precision, [, ref_category]]]] );Output:
ref_category | coef | log_likelihood | std_err | z_stats | p_values | odds_ratios | condition_no | num_iterations
----+------+----------------+---------+---------+----------+-------------+--------------+--------------- ...
SELECT (mlogregr('sourceName', 'dependentVariable', 'independentVariables')).coef;
SELECT coef, log_likelihood, p_values FROM mlogregr('sourceName', 'dependentVariable', 'independentVariables');
Note that the categories are encoded as integers with values from {0, 1, 2,..., numCategories-1}
sql> SELECT * FROM data; r1 | val ---------------------------------------------+----- {1,3.01789340097457,0.454183579888195} | 1 {1,-2.59380532894284,0.602678326424211} | 0 {1,-1.30643094424158,0.151587064377964} | 1 {1,3.60722299199551,0.963550757616758} | 1 {1,-1.52197745628655,0.0782248834148049} | 1 {1,-4.8746574902907,0.345104880165309} | 0 ...
sql> \x on Expanded display is off. sql> SELECT * FROM mlogregr('data', 'val', '2', 'r1', 100, 'irls', 0.001); -[ RECORD 1 ]--+-------------------------------------------------------------- coef | {5.59049410898112,2.11077546770772,-0.237276684606453} log_likelihood | -467.214718489873 std_err | {0.318943457652178,0.101518723785383,0.294509929481773} z_stats | {17.5281667482197,20.7919819024719,-0.805666162169712} p_values | {8.73403463417837e-69,5.11539430631541e-96,0.420435365338518} odds_ratios | {267.867942976278,8.2546400100702,0.788773016471171} condition_no | 179.186118573205 num_iterations | 9
A collection of nice write-ups, with valuable pointers into further literature:
[1] Annette J . Dobson: An Introduction to Generalized Linear Models, Second Edition. Nov 2001
[2] Cosma Shalizi: Statistics 36-350: Data Mining, Lecture Notes, 18 November 2009, http://www.stat.cmu.edu/~cshalizi/350/lectures/26/lecture-26.pdf
[3] Srikrishna Sridhar, Mark Wellons, Caleb Welton: Multilogistic Regression: Notes and References, Jul 12 2012, http://www.cs.wisc.edu/~srikris/mlogit.pdf
[4] Scott A. Czepiel: Maximum Likelihood Estimation of Logistic Regression Models: Theory and Implementation, Retrieved Jul 12 2012, http://czep.net/stat/mlelr.pdf