From the previous posts on Linear Regression (using Batch Gradient descent, Stochastic Gradient Descent, Closed form solution), we discussed couple of different ways to estimate the  parameter vector in the least square error sense for the given training set. However, how does the least square error criterion work when the training set is corrupted by noise? In this post, let us discuss the case where training set is corrupted by Gaussian noise.

For the  training set, the system model is :

,

where,

is the input sequence,

is the output  sequence,

is the parameter vector and

is the noise in the observations.

Let us assume that the noise term are independent and identically distributed following a Gaussian probability having mean 0 and variance .

The probability density function of noise term can be written as,
.

This means that probability of the output sequence  given and parameterised by is,

.

Let us write the likelihood of , given all the observations of input sequence and output as,

.

Given that all the observations are independent, the likelihood of  is,

.

Taking logarithm on both sides, the log-likelihood function is,

.

From the above expression, we can see that maximizing the likelihood function is same as minimizing

Recall: This is same cost function which was minimized in the Least Squares solution.

Summarizing:

a) When the observations are corrupted by independent Gaussian Noise, the least squares solution is the Maximum Likelihood estimate of the parameter vector .

b) The term is not a playing a role in this minimization. However if the noise variance of each observation is different, this needs to get factored in. We will discuss this in another post.

References

CS229 Lecture notes1, Chapter 3 Probabilistic Interpretation, Prof. Andrew Ng

Related posts:

1. Straight line fit using least squares estimate
2. Batch Gradient Descent
3. Stochastic Gradient Descent

Some of us would have used Newton’s method (also known as Newton-Raphson method) in some form or other. The method has quite a bit of history,  starting with the Babylonian way of finding the square root and later over centuries reaching the present recursive way of finding the solution. In this post, we will describe Newton’s method and apply it to find the square root and the inverse of a number.

Geometrical interpretation

We know that the derivative of a function at  is the slope of the tangent (red line) at  i.e.,

.

Rearranging, the intercept of the tangent at x-axis is,

.

From the figure above, can see that the tangent (red line) intercepts the x-axis at  which is closer to  the  where compared to . Keep on doing this operation recursively, and it converges to the zero of the function OR in another words the root of the function.

In general for iteration, the equation is :

.

Finding square root

Let us, for example try to use this method for finding the square root of D=100. The function to zero out in the Newton’s method frame work  is,

, where .

The first derivative is

.

The recursive equation is,

.

Matlab code snippet

clear ; close all;
D = 100; % number to find the square root
x = 1; % initial value
for ii = 1:10
	fx = x.^2 - D;
	f1x = 2*x;
	x = x - fx/f1x;
	x_v(ii) = x;
end
x_v' =
   50.5000000000000
   26.2400990099010
   15.0255301199868
   10.8404346730269
   10.0325785109606
   10.0000528956427
   10.0000000001399
   10.0000000000000
   10.0000000000000
   10.0000000000000

We can see that the it converges within around 8 iterations. Further, playing around with the initial value,

a) if we start with initial value of x = -1, then we will converge to -10.

b) if we start with initial value of x = 0, then we will not converge

and so on…

Finding inverse (division)

Newton’s method can be used to find the inverse of a variable D. One way to write the function to zero out is, but we soon realize that this does not work as we need know in the first place.

Alternatively the function to zero out can be written as,

.

The first derivative is,

.

The equation in the recursive form is,

.

Matlab code snippet

clear ; close all;
D = .1; % number to find the square root
x = [.1:.2:1]; % initial value
for ii = 1:10
	fx = (1./x) - D;
	f1x = -1./x.^2;
	x = x - fx./f1x;
	x_v(:,ii) = x;
end
plot(x_v');
legend('0.1', '0.3', '0.5', '0.7', '0.9');
grid on; xlabel('number of iterations'); ylabel('inverse');
title('finding inverse newton''s method');

The following plot shows the convergence of inverse computation to the right value for different values of for this example matlab code snippet.

Figure : convergence of inverse computation

Finding the minima of a function

To find the minima of a function, we to find where the derivative of the function becomes zero i.e.  .

Using Newton’s method, the recursive equation becomes :

.

Thoughts

We have  briefly gone through the Newton’s method and its applications to find the roots of a function, inverse, minima etc. However, there are quite a few aspects which we did not go over, like :

a) Impact of the initial value on the convergence of the function

b) Rate of the convergence

c) Error bounds of the converged result

d) Conditions where the convergence does not happen

and so on…

Hoping to discuss those in another post…

References

Wiki on Newton’s method

Related posts:

1. Chi Square Random Variable
2. Signal to quantization noise in quantized sinusoidal
3. Closed form solution for linear regression

In the previous post on Batch Gradient Descent and Stochastic Gradient Descent, we looked at two iterative methods for finding the parameter vector  which minimizes the square of the error between the predicted value  and the actual output  for all  values in the training set.

A closed form solution for finding the parameter vector  is possible, and in this post let us explore that. Ofcourse, I thank Prof. Andrew Ng for putting all these material available on public domain (Lecture Notes 1).

Notations

Let’s revisit the notations.

 be the number of training set (in our case top 50 articles),

 be the input sequence (the page index),

 be the output sequence (the page views for each page index)

 be the number of features/parameters (=2 for our example).

The value of  corresponds to the  training set

The predicted the number of page views for a given page index using a hypothesis  defined as :

where,

 is the page index,

.

Formulating in matrix notations:

The input sequence is,

 of dimension [m x n]

The measured values are,

 of dimension [m x 1].

The parameter vector is,

 of dimension [n x 1]

The hypothesis term is,

of dimension [m x 1].

From the above,

.

Recall :

Our goal is to find the parameter vector  which minimizes the square of the error between the predicted value  and the actual output  for all  values in the training set i.e.

.

From matrix algebra, we know that

.

So we can now go about to define the cost function  as,

.

To find the value of  which minimizes , we can differentiate  with respect to .

To find the value of  which minimizes ,  we set

,

.

Solving,

Note : (Update 7th Dec 2011)

As pointed by Mr. Andre KrishKevich, the above solution is same as the formula for liner least squares fit (linear least squares, least square in wiki)

Matlab/Octave code snippet

clear ;
close all;
x = [1:50].';
y = [4554 3014 2171 1891 1593 1532 1416 1326 1297 1266 ...
	1248 1052 951 936 918 797 743 665 662 652 ...
	629 609 596 590 582 547 486 471 462 435 ...
	424 403 400 386 386 384 384 383 370 365 ...
	360 358 354 347 320 319 318 311 307 290 ].';

m = length(y); % store the number of training examples
x = [ ones(m,1) x]; % Add a column of ones to x
n = size(x,2); % number of features
theta_vec = inv(x'*x)*x'*y;

The computed  values are

.

Note :

a)

(Refer: Matrix calculus notes - University of Colorado)

b)

(Refer : Matrix Calculus Wiki)

References

An Application of Supervised Learning – Autonomous Deriving (Video Lecture, Class2)

CS 229 Machine Learning Course Materials

Refer: Matrix calculus notes - University of Colorado

Matrix Calculus Wiki

Related posts:

1. Batch Gradient Descent
2. Stochastic Gradient Descent
3. Linear to log conversion

For curve fitting using linear regression, there exists a minor variant of Batch Gradient Descent algorithm, called Stochastic Gradient Descent.

In the Batch Gradient Descent, the parameter vector  is updated as,

.

(loop over all elements of training set in one iteration)

For Stochastic Gradient Descent, the vector gets updated as, at each iteration the algorithm goes over only one among  training set, i.e.

.

When the training set is large, Stochastic Gradient Descent can be useful (as we need not go over the full data to get the first set of the parameter vector )

For the same Matlab example used in the previous post, we can see that both batch and stochastic gradient descent converged to reasonably close values.

Matlab/Octave code snippet

clear ;
close all;
x = [1:50].';
y = [4554 3014 2171 1891 1593 1532 1416 1326 1297 1266 ...
	1248 1052 951 936 918 797 743 665 662 652 ...
	629 609 596 590 582 547 486 471 462 435 ...
	424 403 400 386 386 384 384 383 370 365 ...
	360 358 354 347 320 319 318 311 307 290 ].';
%
m = length(y); % store the number of training examples
x = [ ones(m,1) x]; % Add a column of ones to x
n = size(x,2); % number of features
theta_batch_vec = [0 0]';
theta_stoch_vec = [0 0]';
alpha = 0.002;
err = [0 0]';
theta_batch_vec_v = zeros(10000,2);
theta_stoch_vec_v = zeros(50*10000,2);
for kk = 1:10000
	% batch gradient descent - loop over all training set
	h_theta_batch = (x*theta_batch_vec);
	h_theta_batch_v = h_theta_batch*ones(1,n);
	y_v = y*ones(1,n);
	theta_batch_vec = theta_batch_vec - alpha*1/m*sum((h_theta_batch_v - y_v).*x).';
	theta_batch_vec_v(kk,:) = theta_batch_vec;
	%j_theta_batch(kk) = 1/(2*m)*sum((h_theta_batch - y).^2);

	% stochastic gradient descent - loop over one training set at a time
	for (jj = 1:50)
		h_theta_stoch = (x(jj,:)*theta_stoch_vec);
		h_theta_stoch_v = h_theta_stoch*ones(1,n);
		y_v = y(jj,:)*ones(1,n);
		theta_stoch_vec = theta_stoch_vec - alpha*1/m*((h_theta_stoch_v - y_v).*x(jj,:)).';
		%j_theta_stoch(kk,jj) = 1/(2*m)*sum((h_theta_stoch - y).^2);
		theta_stoch_vec_v(50*(kk-1)+jj,:) = theta_stoch_vec;
	end
end

figure;
plot(x(:,2),y,'bs-');
hold on
plot(x(:,2),x*theta_batch_vec,'md-');
plot(x(:,2),x*theta_stoch_vec,'rp-');
legend('measured', 'predicted-batch','predicted-stochastic');
grid on;
xlabel('Page index, x');
ylabel('Page views, y');
title('Measured and predicted page views');

j_theta = zeros(250, 250);   % initialize j_theta
theta0_vals = linspace(-2500, 2500, 250);
theta1_vals = linspace(-50, 50, 250);
for i = 1:length(theta0_vals)
	  for j = 1:length(theta1_vals)
		theta_val_vec = [theta0_vals(i) theta1_vals(j)]';
		h_theta = (x*theta_val_vec);
		j_theta(i,j) = 1/(2*m)*sum((h_theta - y).^2);
    end
end
figure;
contour(theta0_vals,theta1_vals,10*log10(j_theta.'))
xlabel('theta_0'); ylabel('theta_1')
title('Cost function J(theta)');
hold on;
plot(theta_stoch_vec_v(:,1),theta_stoch_vec_v(:,2),'rs.');
plot(theta_batch_vec_v(:,1),theta_batch_vec_v(:,2),'kx.');

The converged values are :

.

From the below plot, we can see that Batch Gradient Descent (black line) goes straight down to the minima, whereas Stochastic Gradient Descent (red line) keeps hovering around (thick red line) before going down to the minima.

References

An Application of Supervised Learning – Autonomous Deriving (Video Lecture, Class2)

CS 229 Machine Learning Course Materials

Related posts:

1. Least Squares in Gaussian Noise – Maximum Likelihood
2. Batch Gradient Descent
3. Closed form solution for linear regression

I happened to stumble on Prof. Andrew Ng’s Machine Learning classes which are available online as part of Stanford Center for Professional Development. The first lecture in the series discuss the topic of fitting parameters for a given data set using linear regression.  For understanding this concept, I chose to take data from the top 50 articles of this blog based on the pageviews in the month of September 2011.

Notations

Let

be the number of training set (in our case top 50 articles),

be the input sequence (the page index),

be the output sequence (the page views for each page index)

be the number of features/parameters (=2 for our example).

The value of corresponds to the training set

Let us try to predict the number of page views for a given page index using a hypothesis, where is defined as :

where,

is the page index,

.

Linear regression using gradient descent

Given the above hypothesis, let us try to figure out the parameter which minimizes the square of the error between the predicted value and the actual output for all values in the training set i.e.

Let us define the cost function as,

.

The scaling by fraction is just for notational convenience.

Let us start with some parameter vector , and keep changing the to reduce the cost function , i.e.

.

The parameter vector after algorithm convergence can be used for prediction.

Note :

1. For each update of the parameter vector , the algorithm process the full training set. This algorithm is called Batch Gradient Descent.

2. For the given example with 50 training sets, the going over the full training set is computationally feasible. However when the training set is very large, we need to use a slight variant of this scheme, called Stochastic Gradient Descent. We will discuss that in another post.

3. The proof of the derivation of  involving differential with will be of interest. We will discuss that in another post.

Matlab/Octave code snippet

clear ;
close all;
x = [1:50].';
y = [4554 3014 2171 1891 1593 1532 1416 1326 1297 1266 ...
	1248 1052 951 936 918 797 743 665 662 652 ...
	629 609 596 590 582 547 486 471 462 435 ...
	424 403 400 386 386 384 384 383 370 365 ...
	360 358 354 347 320 319 318 311 307 290 ].';

m = length(y); % store the number of training examples
x = [ ones(m,1) x]; % Add a column of ones to x
n = size(x,2); % number of features
theta_vec = [0 0]';
alpha = 0.002;
err = [0 0]';
for kk = 1:10000
	h_theta = (x*theta_vec);
	h_theta_v = h_theta*ones(1,n);
	y_v = y*ones(1,n);
	theta_vec = theta_vec - alpha*1/m*sum((h_theta_v - y_v).*x).';
	err(:,kk) = 1/m*sum((h_theta_v - y_v).*x).';
end

figure;
plot(x(:,2),y,'bs-');
hold on
plot(x(:,2),x*theta_vec,'rp-');
legend('measured', 'predicted');
grid on;
xlabel('Page index, x');
ylabel('Page views, y');
title('Measured and predicted page views');

The computed values are

.

With this hypotheses, the predicted page views is shown in the red curve (in the below plot).

In matlab code snippet, kept the number of step of gradient descent blindly as 10000. One can probably stop the gradient descent when the cost function is small and/or when rate of change of is small.

Couple of things to note :

1. Given that the measured values are showing an exponential trend, trying to fit a straight line does not seem like a good idea. Anyhow, given this is the first post in this series, I let it pass.

2. The value of controls the rate of convergence of the algorithm. If is very small, the algorithm takes small steps and takes longer time to converge. Higher value of causes the algorithm to take large steps, and may cause algorithm to diverge.

3. Have not figured how to select value suitable (fast convergence) for the data set under consideration. Will figure that out later.

Plotting the variation of for different values of

clear;
j_theta = zeros(250, 250);   % initialize j_theta
theta0_vals = linspace(-5000, 5000, 250);
theta1_vals = linspace(-200, 200, 250);
for i = 1:length(theta0_vals)
	  for j = 1:length(theta1_vals)
		theta_val_vec = [theta0_vals(i) theta1_vals(j)]';
		h_theta = (x*theta_val_vec);
		j_theta(i,j) = 1/(2*m)*sum((h_theta - y).^2);
    end
end
figure;
surf(theta0_vals, theta1_vals,10*log10(j_theta.'));
xlabel('theta_0'); ylabel('theta_1');zlabel('10*log10(Jtheta)');
title('Cost function J(theta)');
figure;
contour(theta0_vals,theta1_vals,10*log10(j_theta.'))
xlabel('theta_0'); ylabel('theta_1')
title('Cost function J(theta)');

Given that the surface() plot is bit unwieldy in my relatively slow desktop, using contour() plot seems to be a much better choice. Can see that the minima of this cost function lies near the computed values of

.

References

An Application of Supervised Learning – Autonomous Deriving

Related posts:

1. Stochastic Gradient Descent
2. Closed form solution for linear regression
3. Least Squares in Gaussian Noise – Maximum Likelihood

In this post, let us discuss a simple implementation friendly scheme for computing the absolute value of a complex number . The technique called (alpha Max + beta Min) algorithm is discussed in Chapter 13.2 of Understanding Digital Signal Processing, Richard Lyons and is also available online at Digital Signal Processing Tricks – High-speed vector magnitude approximation

The magnitude of a complex number is

.

The simplified computation of the absolute value is

where

.

The values of and can be tried out to understand the performance. For analysis we can use a complex number with magnitude 1 and phase from 0 to 180 degrees.

Option#1

= 1, = 1/2,

Option#2

= 1, = 1/4,

Option#3

= 1, = 3/8

Option#4

= 7/8, = 7/16

Option#5

= 15/16, = 15/32

Simulation Model

The script performs the following.

(a) Generate a complex number with phase varying from 0 to 180 degrees.

(b) Find the absolute value using the above 5 options

(c) For each option, find the maximum error, average error and root mean square error

Click here to download Matlab/Octave script for computing the approximate value of magnitude of a complex number

Figure: Plot of approximate value of magnitude of a complex number

Table: Error in the approximate value computation with various values of ,

Observations

1. The chosen values of , facilitates simple multiplier-less implementation of approximate computation (can be implemented using only bit shift and addition).

2. For Options (1),  (3) the maximum error is more than the expected value. Hence we need to allocate extra bits for the output to prevent overflow.

3. The error in the approximate magnitude computation repeats every 90 degrees.

Reference

Chapter 13.2 of Understanding Digital Signal Processing, Richard Lyons

Related posts:

1. Using CORDIC for phase and magnitude computation
2. BER for BPSK in Rayleigh channel
3. CORDIC for phase rotation

In a previous post (here), we looked at using CORDIC (Co-ordinate Rotation by DIgital Computer) for understanding how a complex number can be rotated by an angle without using actual multipliers. Let us know try to understand how we can use CORDIC for finding the phase and magnitude of a complex number.

Basics

The CORDIC algorithm is built on successively multiplying the complex number , by . As can be noticed, as the elements of can be represented in powers of 2, the multiplication can be achieved by using the appropriate ‘bit shift’. For further details, please refer to the previous post (CORDIC for phase rotation).

Finding the magnitude and phase

It is reasonably obvious that the multiplying a complex number by does not change the magnitude of .

Given so, if phase rotation of results in , and the imaginary component of is 0, then the real part of stores the magnitude of .

To put in equations, if

, where ,

then,

(real part of is the magnitude of )

(the rotation angle is the negative of the phase of )

This is the fundamental idea behind finding the magnitude and phase of a complex number using CORDIC.

The sequence of events is as shown below:

(a) The input complex number is subject to a series of phase rotations.

(b) The sign of the phase rotation is the negative of the sign of imaginary component of .

(c) After multiple iterations, imaginary component of tends to zero.

(d) Then, the real part of the new complex vector represents the magnitude and the cumulative phase value represents the negative of the phase of .

Figure: Flow chart for the operations involved in using CORDIC for computing phase and magnitude

The reference phase (phase of ).

The scaling factor of 1.64676025786545 is to remove the ‘gain’ following successive rotations by . Please look at the previous post on CORDIC (here) for details.

Note:

If the input complex number lies in the second or third quadrant, it needs to be first shifted to the first/fourth quadrant before we start the sequence of events shown in the figure above (as the CORDIC range is limited to around +/-90 degrees). This can be achieved by multiplication by or , as appropriate.

Simulation model

Simple Matlab/Octave script for computing the phase and magnitude of a complex number using the CORDIC approach. Quick comparison indicate that the computed and acutal values are closely matching.

Click here to dowload
Reference

[DSPGURU-CORDIC] CORDIC FAQ in dspGuru(TM)

Hope this helps.

Krishna

Related posts:

1. CORDIC for phase rotation
2. Approximate Vector Magnitude Computation
3. First order digital PLL for tracking constant phase offset