When we perform a regression fit of a straight line to a set of (x,y) data points we typically minimize the sum of squares of the "vertical" distance between the data points and the line. In other words, taking x as the independent variable, we minimize the sum of squares of the errors in the dependent variable y. However, this isn't the only possible approach. For example, we might choose to optimize the "horizontal" distances from the points to the line (i.e., the errors in the x variable), or the "perpendicular" distances to the line. If we regard each data point (x,y) as a sample, and if we assume the sample is taken at the precise value of the independent variable x, then it is sensible to regard each data point as being at the exactly correct x coordinate, and all the error is in the sampled value of the dependent coordinate y. On the other hand, if there is some uncertainty in the value of x for each sample, then conceptually it could make sense to take this into account when performing the regression to get the "best" fit. If the distribution of errors in both x and y are random (e.g., normally distributed) then one might think we could just sweep up the error in x as just one more contribution to the measured error in y, so the fitted line should be the same. However, this is not generally the case, as can be seen by considering the simple example of three (x,y) data points (0,0), (10,4), (10,8). To minimize the sum of squares of the errors in the y variable, the line must clearly pass through (0,0) and (10,6), whereas to minimize the sum of squares of the errors in the x variable the optimum line must be tilted more steeply and not pass through (0,0). Similarly if we minimize the sum of squares of the "perpendicular" distance to the line, we will get still a different line. However, the meaning of "perpendicular" is ambiguous because in general the units of x and y may be different, and so the "angles" of lines in the abstract "xy plane" do not have any absolute significance. For example, if x is time, and y is intensity, we can plot the data points with different scalings, so there is no unique notion of "perpendicular" in the time-intensity plane. In order to make the best fit, we need to scale the plot axes (conceptually) such that the variances of the errors in the x and y variables are numerically equal. Once we have done this, it makes sense to treat the results as geometrical points and find the line that minimizes the sum of squares of the perpendicular distances between the points and the line. Of course, this requires us to know the variances of the error distributions. If we don't, then the "best" line will be ambiguous. This is presumably why is it common practice to simply fit the dependent variable, since we don't have sufficient information to know, a priori, how the variances of the x and y errors are related. If we are given a set of (x,y) data points, and we somehow have sufficient information to scale them so the distributions of errors in the x and y variables have equal variances, then we can proceed to fit a line using "perpendicular" regression. One way of approaching this is to find the "principle directions" of the data points. Let's say we have the suitably scaled (x,y) coordinates of n data points. To make it simple, let's first compute the average of the x values, and the average of the y values, calling them X and Y respectively. The point (X,Y) is the centroid of the set of points. Then we can subtract X from each of the x values, and Y from each of the y values, so now we have a list of n data points whose centroid is (0,0). To find the principle directions, imagine rotating the entire set of points about the origin through an angle q. This sends the point (x,y) to the point (x',y') where x' = x cos(q) + y sin(q) y' = -x sin(q) + y cos(q) Now, for any fixed angle q, the sum of the squares of the vertical heights of the n transformed data points is S = SUM [y']^2, and we want to find the angle q that minimizes this. (We can look at this as rotating the regression line so the perpendicular corresponds to the vertical.) To do this, we take the derivative with respect to q and set it equal to zero. The derivative of [y']^2 is 2y'(dy'/dq), so we have dS/dq = 2 SUM [-x sin(q)+y cos(q)][-x cos(q)-y sin(q)] We set this to zero, so we can immediately divide out the factor of 2. Then, expanding out the product and collecting terms into separate summations gives [SUM xy] sin(q)^2 + [SUM (x^2 - y^2)] sin(q)cos(q) - [SUM xy] cos(q)^2 = 0 Dividing through by cos(q)^2, we get a quadratic equation in tan(q): {xy}tan(q)^2 + {x^2 - y^2}tan(q) - {xy} = 0 where the "curly braces" indicate that we take the sum of the contents over all n data points (x,y). Dividing through by the sum {xy} gives tan(q)^2 + A tan(q) - 1 = 0 where A = {x^2-y^2}/{xy}. Solving this quadratic for tan(q) gives two solutions, which correspond to the "principle directions", i.e., the directions in which the "scatter" is maximum and minimum. We want the minimum. Just to illustrate on a trivial example, suppose we have three data points (5,5), (6,6), and (7,7). First compute the centroid, which is (6,6), and then subtract this from each point to give the new set of points (-1,-1), (0,0), and (1,1). Then we can tabulate the sums: x y x^2 - y^2 xy --- --- --------- ---- -1 -1 0 1 0 0 0 0 1 1 0 1 ----- ----- 0 2 In this simple example we have {x^2-y^2} = 0 and {xy} = 2, which means that A = 0, so our equation for the principle directions is simply tan(q)^2 - 1 = 0 Thus the two roots are tan(q)=1 and tan(q)=-1, which corresponds to the angles +45 degrees and -45 degrees. This makes sense, because our original data points make a 45 degree line, so if we rotate them 45 degrees clockwise they are flat, whereas if we rotate them 45 degrees the other way they are vertically arranged. These are the two principle directions of this set of 3 points. The "best" fit through the original three points is a 45 degree line through the centroid - which is obvious in this trivial example, but the method works in general with arbitrary sets of points. For another example, suppose we have four data points (2,6), (4,2), (16,8), and (14,12). The centroid of these points is (9,7), so we can subtract this from each point to give the new set of points (-7,-1), (-5,-5), (7,1), and (5,5). Then we can tabulate the sums: x y xy x^2 - y^2 --- --- ---- --------- -7 -1 7 48 -5 -5 25 0 7 1 7 48 5 5 25 0 ---- ---- sums: 64 96 In this case we have {xy} = 64 and {x^2-y^2} = 96, which gives A = 3/2, so our equation for the principle directions is tan(q)^2 + (3/2)tan(q) - 1 = 0 The two roots are tan(q) = 1/2 and -2, which correspond to the angles +26.565 degrees and -63.434 degrees. This is consistent with the fact that our original four data points are the vertices of a rectangle whose edges have the slopes 1/2 and -2. The "best" fit through these four points is a line through the centroid with a slope of 1/2. (It's interesting that the two quantities which characterize the points, namely xy and x^2 - y^2, are both hyperbolic conic forms, and they constitute the invariants of Minkowski spacetime when expressed in terms of null coordinates and spatio-temporal coordinates, respectively.)

Return to MathPages Main Menu