http://arxiv.org/pdf/1201.6035v1.pdf

If I am correct that HH^T has to be invertible, then it follows that HPH^T is invertible.

I don’t see physically why your stepping equation should be insoluble when m > n. In fact I think it should not be insoluble even if m < n, as the Kalman filter is just giving you the best estimate.

The sum of two PSD matrices is always PSD (the positive semidefinite matrices form a cone in the vector space of all symmetric matrices; cones are closed under sum). So if P and W are PSD, then so is HPH’ + W. If W is not singular, or if H and P are not singular, then HPH’ + W is invertible.

@ Warwick You are correct about my meaning for m > n. As per your comment on invertibility, I don’t know that I have a sum of invertibles. HH’ is definitely not invertible, am not sure about HPH’. The current method is to get a least-squares fit with a pseudoinverse.

—–

I hate to clog the comments section like this, so bear with me. The following excerpt bears directly on my application. It covers my implementation question I started off asking, but I still don’t see how the matrix sum is guaranteed to be symmetric, positive-definite. Any thoughts?

From _Kalman Filtering; Theory and Practice Using MATLAB_, second edition, by Grewal and Andrews:

Cholesky decomposition provides an effcient and numerically stable method for solving equations of the form AX = Y when A is a symmetric, positive-definite matrix. The modified Cholesky decomposition is even better, because it avoids taking scalar square roots. It is the recommended method for forming the term

(HPH’ + R)^-1 H

in the conventional Kalman filter without explicitly inverting a matrix. That is, if one
decomposes HPH’ + R as UDU’ , then

(UDU’)(HPH’ + R)^-1 H = H.

It then suffices to solve

UDU’X = H

for X.

I’d say that this answers the implementation question I posed. There’s still the matter of invertibility of

HPH’ + W

but this page is for alternatives to matrix inversion so I’ll leave off posting any more questions here. Thanks Matt and Warwick!

If your equations do not have a solution (overdetermined), then you can get the least-squares fit with a pseudoinverse, and I guess there are then similar decompositions for that. I don’t know whether a Kalman filter should lead to insoluble equations.

When you say m > n, I am taking it that means your observation y is an m-vector but your underlying process x is an n-vector. Correct?

S = (HPH’ + W)

is not guaranteed to be invertible even if H, P, and W are. I don’t understand how this update equation can be used then (I’m not saying it can’t, I just don’t see how one is to use an update with an inversion that may not be possible at every step).

(HPH’ + W)^-1

where H is my output map, P is the estimate of the error covariance, and W is the covariance matrix for the measurement noise. I gather that there’s no way to be sure of the invertibility of this sum (besides, though I guess that P and W are invertible I know that H for my case does not have full row rank, as m > n). Here is the entire equation for the state update.

x_{k+1|k+1} = x_{k|k} + PH(HPH’ + W)^-1(y – Hx_{k+1|k})

Anyway, that’s the background for this comment; I thought my matrix was not guaranteed to be invertible. I’m now looking for tips on how to implement this equation. Comment #28 states that “implementations [of the Kalman filter] never use the inverse,” but yet the first one I looked at does (http://code.google.com/p/efficient-java-matrix-library/wiki/KalmanFilterExamples). Other comments above mention Cholesky decomposition for multiplication tasks, so I’m guessing that might be the approach to take. Though now, after reading a Wikipedia entry, I’m guessing this can’t be the right approach since I don’t think the sum will necessarily be Hermitian, positive definite. What do I use instead of the inverse here?

As for resources, I’d recommend any of the linear algebra books Ward Cheney has written. His latest book is very expensive, but you could look for a used copy or an earlier edition.

Thanks. I missed your comment (!) but in the end I ended up doing exactly this, using the band-diagonal LU algorithm in Press et al (not sure to whom they credited it). This for a genuine quintic spline curve not a weighted average. This then worked in practice about 10x slower than just taking derivatives using local quadratic approx, but the improvement was worthwhile in certain respects.
Cheers,

Warwick

Instead I am writing LU by hand, and trying my best to exploit the special property involved. Viz, the polynomial takes the form a0 + a1(x-xmid)+a2(x-xmid)^2+..+a5(x-xmid)^5 where if the points are x0,…,x5 then xmid is the average of x2 and x3. Therefore where d corresponds to distance from xmid, one row is 1,d2,d2*d2,d2^3,d2^4,d2^5, and one row is 1,-d2,d2*d2,-d2^3,d2^4,-d2^5. And the first column is all 1’s. Other than that I’m not sure there is much to exploit.
I wouldn’t prefer to write an LU routine (a la Press et al) because of the pivoting business – trying to keep this really straightforward as well as fast.

I just wondered, since this specific problem is obviously a very common one, does anyone have a standard, numerically stable algorithm for doing it. (This reinventing the wheel is time-consuming, and unfortunately I have to go to work so can only work on this voluntary project at the weekend.)

To learn about matrix factorizations, Matrix Computations by Golub and Van Loan is a classic.

I take it the discussion here is about very large matrices? Or does it apply to small 4×4 matrices as well? For example, I have an iduction motor simulation in matab which is a 4 by 4 matrix and I was wondering if thie factorisation could speed up the simulation? And if so, where can I learn about factorisation of matrices?

Cheers,

Using the singular value decomposition, A = U S V’, with U’U = V’V = I, S diagonal,


A'A = (USV')'USV' = VSU'USV' = VSSV'
inv(A'A) = inv(VSSV') = inv(V') inv(SS) inv(V) = V inv(SS)' V'


Since S is diagonal,

inv(A'A) = V inv(S) inv(S)' V' = (V inv(S)) (V inv(S))'

For any matrix X, diag(XX’) is the sum of the square of the rows of X or the columns of X’. With S diagonal, the inverse is just element-wise 1/S, and you can compute the diagonal entries of inv(A’A) by summing the squares of the columns of V’ and dividing the result by the corresponding diagonal entry in S.

Paul