Dimension Reduction Techniques for Conditional Quantiles in R: A Vignette

Eliana Christou

University of North Carolina at Charlotte

Abstract

While dimension reduction techniques have been well developed, they focus on the entire conditional distribution of the data. However, there are a lot of research areas where extremes are important and require the study of the conditional quantiles. quantdr is an R package that performs dimension reduction to conditional quantiles by determining the directions that span the central quantile subspace (CQS).

Introduction

Data visualization and non-parametric fitting can become very challenging when data is high-dimensional. This has led to the development of many dimension reduction techniques that focus on the entire conditional distribution of the data. However, when specific aspects of the conditional distribution are of interest, these methods can provide more directions than necessary.

Quantile regression (QR) has received a lot of attention since its inception from Koenker and Bassett (1978) and it has been an attractive tool for research areas where non-central parts of the data are important. Dimension reduction techniques for conditional quantiles have recently made an appearance and are still in development. For some references, see Wu et al. (2010), Kong and Xia (2012, 2014), Luo et al. (2014), and Christou and Akritas (2016).

More recently, Christou (2020) proposed a new technique for finding the fewest linear combinations of the predictor variable \(\mathbf{X}\) that contain all the information about the conditional quantile. To give further intuition, for a univariate response \(Y\) and a \(p\)-dimensional predictor \(\mathbf{X}\), the method focuses on finding a \(p \times d_{\tau}\) matrix \(\mathbf{B}_{\tau}\), for \(\tau \in (0, 1)\) and \(d_{\tau} \leq p\), such that \(Y\) and \(Q_{\tau}(Y|\mathbf{X})\) are conditionally independent given \(\mathbf{B}_{\tau}^{\top} \mathbf{X}\), where \(Q_{\tau}(Y|\mathbf{X})\) denotes the \(\tau\)th conditional quantile of \(Y\) given \(\mathbf{X}\). The space spanned by \(\mathbf{B}_{\tau}\) is called the \(\tau\)th quantile dimension reduction subspace for the regression of \(Y\) on \(\mathbf{X}\). The intersection of all \(\tau\)th dimension reduction subspaces is called the \(\tau\)th central quantile subspace (\(\tau\)-CQS) and is denoted by \(\mathcal{S}_{Q_{\tau}(Y|\mathbf{X})}\).

The purpose of this vignette is to demonstrate the implementation of the functions appearing in the quantdr package, along with some examples. The main function of the package is cqs which returns the estimated directions of the \(\tau\)-CQS.

Getting Started

To get started you need to first install the package using the command

and then use it during any R session by issuing the command

For an overview of the available help files of the package use

The package includes the following functions:

To get help on a specific function type help(function.name) or ?function.name for a convenient shorthand. Try

For further examples, refer to the coding examples listed in the documentation for each quantdr function.

Central quantile subspace

Overview

The overall goal of the cqs function is to identify the coefficients of the linear combination \(\mathbf{B}_{\tau}^{\top}\mathbf{X}\) in order to replace the \(p \times 1\) predictor vector \(\mathbf{X}\) with the low-dimensional \(d_{\tau} \times 1\) predictor vector \(\mathbf{B}_{\tau}^{\top}\mathbf{X}\). To do that, Christou (2020) proved two important results:

  • \(\boldsymbol{\beta}_{\tau} \in \mathcal{S}_{Q_{\tau}(Y|\mathbf{X})}\), where \(\boldsymbol{\beta}_{\tau}\) is the slope vector from regressing the conditional quantile on \(\mathbf{X}\), i.e., \[\begin{eqnarray*} (\alpha_{\tau}, \boldsymbol{\beta}_{\tau}) = \arg \min_{(a_{\tau}, \mathbf{b}_{\tau})} E \{ Q_{\tau}(Y| \mathbf{A}^{\top} \mathbf{X}) - a_{\tau} - \mathbf{b}_{\tau}^{\top}\mathbf{X}\}^2, \end{eqnarray*}\] and \(\mathbf{A}\) spans the central subspace (Li 1991). This implies an initial dimension reduction using \(\mathbf{A}\), making the nonparametric estimation of \(Q_{\tau}(Y|\mathbf{A}^{\top}\mathbf{X})\) tractable.

  • \(E\{Q_{\tau} (Y|U_{\tau})\mathbf{X}\} \in \mathcal{S}_{Q_{\tau}(Y|\mathbf{X})}\), where \(U_{\tau}\) is a measurable function of \(\mathbf{B}_{\tau}^{\top}\mathbf{X}\), provided that \(Q_{\tau}(Y|U_{\tau})\mathbf{X}\) is integrable.

The above two results imply the following:

  1. If the dimension of the \(\tau\)-CQS is known to be one, then we can fit a linear regression model of \(Q_{\tau}(Y|\mathbf{A}^{\top}\mathbf{X})\) on \(\mathbf{X}\) and report the slope vector as the basis vector.

  2. If the dimension of the \(\tau\)-CQS is known to be greater than one, then we can set \(\boldsymbol{\beta}_{\tau, 0}=\boldsymbol{\beta}_{\tau}\) as the initial vector and then create more vectors using \(\boldsymbol{\beta}_{\tau,j}=E\{Q_{\tau}(Y | \boldsymbol{\beta}_{\tau, j-1}^{\top}\mathbf{X})\mathbf{X}\}\), for \(j=1, \dots, p-1\). In order to obtain linearly independent vectors, we can perform an eigenvalue decomposition on \(\mathbf{V}_{\tau}\mathbf{V}_{\tau}^{\top}\), where \(\mathbf{V}_{\tau}=(\boldsymbol{\beta}_{\tau, 0}, \dots, \boldsymbol{\beta}_{\tau, p-1})\) and choose the eigenvectors corresponding to the \(d_{\tau}\) nonzero eigenvalues.

  3. If the dimension of the \(\tau\)-CQS is unknown, we can apply the procedure from 2. Then, we can estimate the dimension \(d_{\tau}\) using the eigenvalues resulting from the eigenvalue decomposition. Existing techniques such as the Cross-Validation (CV) criterion or the modified-BIC type criterion of Zhu et al. (2010) can be used to determine the structural dimension.

All of the above are incorporated in the cqs function.

The algorithm at a glance

The first step of the algorithm requires to fit a linear regression model of \(Q_{\tau}(Y|\mathbf{A}^{\top}\mathbf{X})\) on \(\mathbf{X}\), which implies the non-parametric estimation of the conditional quantile. The second step of the algorithm relies on estimating a conditional expectation and performing an eigenvalue decomposition. Specifically,

  1. We use a dimension reduction technique to estimate the basis matrix \(\mathbf{A}\) of the central subspace, denoted by \(\widehat{\mathbf{A}}\), and form the new sufficient predictors \(\widehat{\mathbf{A}}^{\top} \mathbf{X}_{i}\), \(i=1, \dots, n\). For this we use the function dr from the package dr.

  2. For each \(i=1, \dots, n\), we use the local linear conditional quantile estimation method of Guerre and Sabbah (2012) to estimate \(Q_{\tau}(Y|\widehat{\mathbf{A}}^{\top} \mathbf{X}_{i})\). For this we use the llqr of the presented package.

  3. We set \(\widehat{\boldsymbol{\beta}}_{\tau}\) to be \[\begin{eqnarray*} (\widehat{\alpha}_{\tau}, \widehat{\boldsymbol{\beta}}_{\tau}) = \arg \min_{(a_{\tau}, \mathbf{b}_{\tau})} \sum_{i=1}^{n} \{\widehat{Q}_{\tau}(Y|\widehat{\mathbf{A}}^{\top}\mathbf{X}_{i}) - a_{\tau} - \mathbf{b}_{\tau}^{\top} \mathbf{X}_{i}\}^2. \end{eqnarray*}\] For this we use the lm function of the stats package.

  4. If \(d_{\tau}=1\) we stop and report \(\widehat{\boldsymbol{\beta}}_{\tau}\) as the estimated basis vector for \(\mathcal{S}_{Q_{\tau}(Y|\mathbf{X})}\). Otherwise, we move to Step 5.

  5. We set \(\widehat{\boldsymbol{\beta}}_{\tau, 0}=\widehat{\boldsymbol{\beta}}_{\tau}\), and, for \(j=1, \dots, p-1\), we
    1. form the predictors \(\widehat{\boldsymbol{\beta}}_{\tau, j-1}^{\top} \mathbf{X}_{i}\), \(i=1, \dots, n\) and estimate \(Q_{\tau}(Y| \widehat{\boldsymbol{\beta}}_{\tau, j-1}^{\top} \mathbf{X}_{i})\) using the local linear conditional quantile estimation method. For this we use the llqr function of the presented paper.
    2. form \(\widehat{\boldsymbol{\beta}}_{\tau, j} = n^{-1} \sum_{i=1}^{n} \widehat{Q}_{\tau}(Y|\widehat{\boldsymbol{\beta}}_{\tau, j-1}^{\top} \mathbf{X}_{i}) \mathbf{X}_{i}\).
  6. We form the \(p \times p\) matrix \(\widehat{\mathbf{V}}_{\tau}=(\widehat{\boldsymbol{\beta}}_{\tau, 0}, \dots, \widehat{\boldsymbol{\beta}}_{\tau, p-1})\) and choose the \(d_{\tau}\) eigenvectors corresponding to the \(d_{\tau}\) largest eigenvalues of \(\widehat{\mathbf{V}}_{\tau} \widehat{\mathbf{V}}_{\tau}^{\top}\). For this we use the eigen function of the base package.

If \(d_{\tau}\) is unknown, a suggested structural dimension can be estimated using existing techniques such as the CV criterion or the modified-BIC type criterion of Zhu et al. (2010). In terms of the cqs function, the modified-BIC type criterion is utilized and returns a suggested dimension. Note that the user can extract the eigenvalues and apply them to any other existing technique.

Output of cqs

For the purpose of this section, we simulate data from the homoscedastic single-index model \[\begin{eqnarray*} Y=3X_{1}+X_{2}+\epsilon, \end{eqnarray*}\] where \(\mathbf{X}=(X_{1}, \dots, X_{10})^{\top}\) and the error \(\epsilon\) are generated according to a standard normal distribution. The \(\tau\)-CQS is spanned by \((3, 1, 0, \dots, 0)^{\top}\) for all \(\tau\). We focus on the estimation of the 0.5-CQS.

The cqs function proceeds by specifying an \(n \times p\) design matrix \(\mathbf{X}\), a vector of the response variable \(Y\), and a quantile level \(\tau\). The dimension of the \(\tau\)-CQS, i.e., \(d_{\tau}\), is optional. Note that for this example \(d_{\tau}=1\) for every quantile level \(\tau\).

We now discuss the output of the cqs function.

When the dimension of the \(\tau\)-CQS is known to be one, the algorithm reports the ordinary least-squares slope vector as the basis vector of the subspace. This is denoted by qvectors. Moreover, the function cqs outputs dtau, which in this case is specified by the user.

When the dimension of the \(\tau\)-CQS is unknown (or known to be greater than one), the algorithm continues, creates more vectors, and performs an eigenvalue decomposition. Therefore, the output includes one more element, the eigenvalues provided by the eigenvalue decomposition and denoted by qvalues.

out2 <- cqs(x, y, tau = tau)
out2
#> $qvectors
#>               [,1]        [,2]        [,3]        [,4]        [,5]        [,6]
#>  [1,]  0.943455116 -0.13476598 -0.17744563 -0.16367260 -0.03861975  0.16934165
#>  [2,]  0.294419772  0.24766667  0.03225656  0.72366990  0.33478583 -0.54329021
#>  [3,]  0.046107246  0.06529525  0.53266234  0.13123921 -0.88043015 -0.14159136
#>  [4,]  0.040122031  0.01460153  0.16795753 -0.39697720  0.29701816 -0.03795763
#>  [5,] -0.006834183 -0.10566656  0.11664105 -0.45930021  0.17583088 -0.40132163
#>  [6,] -0.031218720  0.35939077  0.26044391  0.06932040 -0.16570677  0.10427297
#>  [7,]  0.028571630 -0.02989090  0.19065394 -0.35496799 -0.17310236 -0.59695362
#>  [8,]  0.021185991  0.07911447 -0.23884205 -0.07758967 -0.23544416 -0.09464941
#>  [9,]  0.056994812  0.85417116 -0.50698277 -0.12425469 -0.56099590 -0.13595234
#> [10,]  0.067929808  0.64866301  0.54201199  0.18404654  0.30651407  0.16017927
#>              [,7]         [,8]        [,9]        [,10]
#>  [1,] -0.17073257 -0.109538030  0.13146841  0.106268958
#>  [2,]  0.03193966 -0.069063890  0.05991995  0.115250619
#>  [3,]  0.10978844 -0.118644404  0.01715956 -0.252735231
#>  [4,]  0.79902281  0.154914759 -0.08996129 -0.245829527
#>  [5,] -0.06793766 -0.606903933 -0.26799106  0.132027186
#>  [6,]  0.03549024  0.057995850 -0.77249367  0.641990938
#>  [7,] -0.33767033  0.681352883  0.10778790  0.287225672
#>  [8,] -0.28718156  0.280040749 -0.62495503 -0.680324968
#>  [9,] -0.05571900 -0.005263087  0.15985521  0.002217146
#> [10,] -0.32258107  0.069583317  0.11878564 -0.275808295
#> 
#> $qvalues
#>  [1] 1.012657e+02 6.500130e-02 7.391152e-04 2.215046e-04 1.807622e-05
#>  [6] 8.719940e-06 4.488049e-06 1.447917e-06 3.558377e-07 7.537092e-10
#> 
#> $dtau
#> [1] 1

Note that the dimension dtau is correctly estimated by the modified-BIC type criterion. Therefore, this output suggests to take the first eigenvector as the basis vector for the \(\tau\)-CQS. We can extract the direction using

If we want to measure the estimation error we can use the angle between the true and the estimated subspaces. This is performed using the subspace function of the pracma package. Note that the angle is measured in radians, and so we divide by \(\pi / 2\).

Note that the estimation error is slightly higher when the dimension of the subspace is unknown and needs to be estimated.

We can continue further and estimate the conditional quantile function. To do this, we first form the new sufficient predictor \(\widehat{\mathbf{B}}_{\tau}^{\top}\mathbf{X}\) using

We then estimate the conditional quantile using the llqr function of the presented package. The main arguments of the function are: the design matrix, which in this case is \(\widehat{\mathbf{B}}_{\tau}^{\top}\mathbf{X}\), the vector of the response variable \(Y\), and the quantile level \(\tau\). The rest of the arguments, i.e., the bandwidth \(h\), the method for the estimation of \(h\), and the single observation \(x_0\) for which to perform the estimation, are optional. If \(h\) is not specified, then it will be determined using either the rule-of-thumb bandwidth of Yu and Jones (1998) or the CV criterion; the default choice is the rule-of-thumb. However, the user needs to be careful about the bandwidth selection. When the dimension of the predictor variable is large compared to the sample size, local linear fitting meets the ‘curse of dimensionality’ problem. In situations like that, the bandwidth selected by the rule-of-thumb or the CV criterion might be small and cause the function to fail. For these cases, we advice the user to pre-specify the bandwidth in the function. Finally, if \(x_0\) is not specified, the estimation will be performed on the entire design matrix. For this example we use

The output consists of the estimation of the conditional quantile function at each point of the design matrix, i.e., \(\widehat{Q}_{\tau}(Y|\widehat{\mathbf{B}}_{\tau}^{\top} \mathbf{X}_{i})\), \(i=1, \dots, n\), and the estimation of the bandwidth using the rule-of-thumb. For illustration purposes, we repeat the above using the CV criterion for the bandwidth selection.

If \(h\) is pre-specified, then the output reports the value given by the user, i.e.,

To illustrate the median regression, compared with the original data, we can use

The same procedure can be performed for multiple quantile levels. For example, we estimate the \(\tau\)-CQS for \(\tau= 0.1, 0.25, 0.5, 0.75, 0.9\).

Similarly, we can plot the data along with the estimated conditional quantile functions for the various quantile levels.

Conclusion

This vignette provides a brief introduction to the R package quantdr and presents a tutorial on how to implement the basic function cqs. Performing dimension reduction techniques to conditional quantiles is an active research topic and therefore updates and new functions will be incorporated into forthcoming versions of the package. For this reason, this document will be updated accordingly and will be made available through the package. Suggestions and recommendations from the users will be highly appreciated and welcomed. This package is also available through GitHub. Feel free to contact the author at .

References

  1. Christou, E. (2020) Central quantile subspace. Statistics and Computing, 30, 677–695

  2. Christou, E. and Akritas, M. (2016) Single index quantile regression for heteroscedastic data. Journal of Multivariate Analysis, 150, 169-182

  3. Guerre, E., and Sabbah, C. (2012) Uniform bias study and Bahadur representation for local polynomial estimators of the conditional quantile function. Econometric Theory, 28, 87-129

  4. Koenker, R., and Bassett, G. (1978) Regression quantiles. Econometrica, 46, 33-50

  5. Kong, E., and Xia, Y. (2012) A single-index quantile regression model and its estimation. Econometric Theory, 28, 730-768

  6. Kong, E., and Xia, Y. (2014) An adaptive composite quantile approach to dimension reduction. Annals of Statistics, 42, 1657-1688

  7. Li, K.-C. (1991) Sliced inverse regression for dimension reduction. Journal of the American Statistical Association, 86, 316-327

  8. Luo, W., Li, B., and Yin, X. (2014) On efficient dimension reduction with respect to a statistical functional of interest. Annals of Statistics, 42, 382-412
  9. Wu, T. Z., Yu, K., and Yu, Y. (2010) Single index quantile regression. Journal of Multivariate Analysis, 101, 1607-1621

  10. Yu, K., and Jones, M. C. (1998) Local linear quantile regression. Journal of the American Statistical Association, 93, 228-238

  11. Zhu, L.-P., Zhu, L.-X., and Feng, Z.-H. (2010) Dimension reduction in regression through cumulative slicing estimation. Journal of the American Statistical Association, 105, 1455-1466