Tikrit Journal of Pure Science

I n linear regression model, the biased estimation is one of the most commonly used methods to reduce the effect of the multicollinearity. In this paper, a simulation study is performed to compare the relative efficiency of some kinds of biased estimators as well as for twelve proposed estimated ridge parameter (k) which are given in the literature. We propose some new adjustments to estimate the ridge parameter. Finally


Introduction
be the multiple linear regression model, where y is an (n  1) vector of responses, X is an (n  p) design matrix of the explanatory variables, p is the number of the explanatory variables , β is a (p  1) vector of unknown parameters of interest, ε is an (n × 1) vector of residuals that follow the standard assumptions, namely, ( ) 0 E   and ' 2 ( ) I is an identity matrix of order n.
The OLS of  is the best linear unbiased estimator (BLUE) which is given by  ̂OLS = (′) −1 ′ …(1. 2) The most important assumption in multiple linear regression model, the explanatory variables must be considered as independent of each other.But, practically, there are probably linear dependencies between these variable values.Mainly, this problem could appear in econometric data and it's called multicollinearity.Multicollinearity influences the regression analysis extremely and it is one of the main problems.The existence of multicollinearity makes the estimates of the correlation coefficients large and very large sampling variances of the OLS estimated Lukman et al. [1].To overcome this problem, there are various methods have been mentioned in literature and one of them is by using the biased estimators.The common biased estimation method is the ridge regression which was proposed by Hoerl and Kennard [2] and still the researchers working in this area like Kibria, and Banik [3].They suggested using the ordinary ridge regression (ORR) as bellow: where k is the ridge parameter and the value of 0 k  .The ORR estimator is biased to a certain value of k which is unknown and therefore it should be estimated from real data.A number of ways for obtaining biased estimates of β with smaller MSE have been developed.By extending Hoerl and Kennard's model, Crouse et al. [4] defined the unbiased ridge regression (URR) estimator as follows:  ̂(, ) = ( ′  +  p ) −1 ( ′  + ), … (1.4) where  is a random vector with ~(, ( 2 /)).Battah and Gore [5] proposed a modified unbiased ridge regression (MURR) estimator of β and still the researchers who work in this area like Lukman et al. [6]and Tarima et al. [7] which is denoted as below: the ORR and URR estimators have been combined to obtain the MURR which was driven from ORR by using URR rather than OLS .The two-parameter estimator (TPE) proposed by Ozkale and Kacıranlar [8] and still the researchers working in this area like Asar, and Genç [9].which is denoted as follows:  ̂(k , d) = ( ′  +  p ) −1 ( ′  +  ̂OLS ) =  kd  ̂OLS …. (1.6) where  kd = ( ′  +  p ) −1 ( ′  + ), 0 k  and d is shrinkage parametar such that 0 1. d  To simplify the considerations about the linear model, the canonical form is often used.Therefore, a symmetric matrix S = X′X has an eigenvalueeigenvector decomposition of the form S = TΛT′, where T is an orthogonal matrix and Λ is a real diagonal matrix.The diagonal elements of Λ are the eigenvalues of S and the column vectors of T are the eigenvectors of S. The orthogonal version of the regression model in (1-1) is  =  ′  +  =  +  …(1.7)where Z = XT, γ = T ′β and Z′Z = Λ=dig( 12 , ,..., p    ).The OLS estimator of γ is given by The goal of this paper is to compare the different biased estimators as well as with different estimated value of k using the MSE as a measure of goodness of fit.The paper is organized as follows.In Section 2, we present the methodology of different estimators of k and propose some new estimators.A Monte Carlo simulation has been given in Section 3. The discussions of the results of the simulation are given in Section 4. Finally, in Section 5, a real data set as an application of this study is given.

Estimation of Ridge Parameter
Hoerl and Kennard [2] showed the properties of ORR in detail.They concluded that the total variance decreases and the squared bias increases as k increases.The variance function is monotonically decreasing and the squared bias function is monotonically increasing.That means, there is a chance that some k exists such that the MSE for ORR is less than MSE for the OLS.It is well known that k is unknown and estimated from the sample of the study.For this reason, there are many articles proposed different ridge parameters in the literature using different techniques.Recently, many researchers studied this area and proposed different estimates of k.We review available methods in literatures to estimate the value of k as follows: -Hoerl and Kennard [2] suggested k to be (denoted here by ˆHK k 3) -Hocking et al. [12] suggested k to be(denoted here by( ˆHSL where ˆi OLS  is the i th element of ˆOLS  -Nomura [13] suggested k to be (denoted by ˆHMO k where i  is the i th eigenvalues. -Kibria [14] proposed the following estimators for k based on arithmetic mean (AM), geometric mean (GM), and median of 2 2   ˆi  .These are defined as follows: The estimator based on AM (denoted by ˆAM k The estimator based on GM (denoted by ˆGM k The estimator based on median (denoted by ˆMED k

ˆ( )
where max  is the maximum eigenvalue of the matrix X'X.

A simulation study
The aim of the current study is to perform a comparison of different biased estimators for variate estimates of ridge parameter which are given in (2.1-2.17) and identify some good estimators for practitioners.We conduct a simulation study using Matlab.This simulation has been designed depends on specific factors that are expected to influence the properties of estimators which be subjected to a statistical investigation Lukman et al. [17].Since the degree of the collinearity among several explanatory variables (Xs) is very essential, Kibria [14] was followed to generate X's using the following equation: where the z ij independent standard normal pseudorandom numbers and  represents the correlation between any two X's.These various are standardized so that X'X is being in correlation forms.The response variable y is considered by ... , i=1,2,...,n , where the e i is i.i.d.N(0, σ 2 ).Therefore, zero intercept for (3.2) will be assumed.Also the number of explanatory variables 5 p  , while the values of  are chose as (1,5,10,20).The correlation φ will choose as (0.75, 0.85, 0.90, 0.95) and sample size n=(50, 100, 150) .The coefficients β 1 , β 2, …, β p are selected as the eigenvectors corresponding to the largest eigenvalue of the matrix X'X subject to constraint β'β = 1.Thus, for n, p, β, λ, φ, and σ, sets of Xs are created.Then the experiment was repreformed 10000 times by creating new error terms.The estimated MSE for each estimator is calculated as follows: where *  would be any of the estimators (OLS, ORR, MURR, or TPE).

The discussion of simulation results
In this section we present the results of our Monte Carlo experiment concerning the properties of the different methods used to choose the ridge parameter K, when multicollinearity among the columns of the design matrix of the explanatory variables exist.The simulation results are presented in Tables 1-12 and we will discuss the results by dividing the results in three parts:

4-2 The simulation results according to the different estimated ridge parameter
In order to know the preference of the estimated ridge parameter that mentioned in this paper, Tables (4-2 to 4-5) show an explanation that, where we can observe the following: 1-By increasing the sample size, we observe others estimated of ridge parameter which gives lowest MSE and still MED, HKB, and LW give well performance as we observed in Table (4-2).
2-From Tables( (4-3 ) to (4-5)) and Tables (1-12), the proposed estimated ridge parameter (MU1-MU5) are working well compared to other estimated ridge parameter, especially with MURR estimator and this is the case for all situations as well as it compared with OLS estimator.3-From Tables( (4-3) to (4-5)) in general we observe that all estimated ridge parameter working well with MURR estimator which is the best estimator according to this study, that means we can use any one of them to find the MURR estimator. 134

Real Life Application
In order to give more explanation for the study, we consider the data set in economics on total national research and development expenditures as a percent of gross national product originally due to Gruber [18] and later by Akdeniz and Erol [19], among others.This reflects the relationship between the dependent Y variable the percentage expended by the United States and the other four independent X1 , X2, X3, and X4 variables.The vector X1 reflects the amount that France spent, X2 that West Germany spent, X3 that Japan spent, and X4 that the former Soviet Union spent on.
The goal is to compare the traces of the estimated MSE matrices of (ORR), (MURR) and (TPE).The trace of the MSE matrix of the (ORR) is given by 137 mse( ˆR the trace of the MSE matrix of the (MURR) is given by mse( ˆ() J k  )=tr(MSE( ˆ( ), J k the trace of the MSE matrix of the (TPE) is given by mse( ˆ( , ) kd (( 1) )  , we can observe that, the minimum mse for the ORR estimator will be got if k is estimated by HKB.Also the minimum mse for the MURR estimator will be given by estimating k by HKB while the minimum mse for the TPE estimator will be given by estimating k by MU2.The performance of the estimated k that given in this study is showing that under moderate degree of multicollinearity, the most of them give minimum mse if they used in the MURR estimator except (AM, MU3 and MU4) where the OLS estimator is better than of them.Therefore, not all proposed ridge parameter can be used to get minimum mse when the degree of multicollinearity is moderate.Finally, we can say that, this study gives us a broad view on the behaviour of the estimators and when they can be used to give a good performance compared to the other suggested estimators.
al.[10] proposed k to be (denoted here by ˆHKB k

Table 4 -2 The simulation results according to the different estimated ridge parameter
We can observe that the variables in  ′  matrix suffer for high correlations among them and this is the one advantage of standardizing the X matrix where it can be seen which variables are highly correlated.Another method for diagnosing multicollinearity in linear regression, is the Condition Index (C.I.) which is defined as follows: