Lawrence -
        Median and related nonlinear filters really are the right way
     to deal with long-tailed distribution noise problems. But, as with
     everything, you really need to understand your signal. If a spike
     lasts for more than N samples in a 2N+1 point median filter, it will
     not remove it. I presume from your comments that the spikes are
     longer than this. If you can decrease the sampling frequency to
     the point where you meet this criteria, it will solve your problem.
     I would imagine that a 5 point median filter would work beautifully
     with the right timing. Even a three point one (very easy to code)
     might well be sufficient. The point here is that you really have
     to look at your data to know what's going on.
        Another post mentioned dropping min and max values, and then
     averaging the rest. This is actually closely related to median
     filtering. There is an amazing amount of (somewhat irrelevant)
     theory behind these filters that comes out of robust estimation
     theory. If your curious, check out the Kluwer book by Pitas and
     Venetsanopoulas (sp?).
        Yet another cheap and dirty way to track the median is to
     build a delta modulator. At each moment in time you have an
     estimate of the value (this is the filter output) which you
     compare to the input. If the input is greater, you add delta
     to your estimate. If it is smaller, you subtract delta. The tradeoff
     is to make delta big enough to track changes reasonably quickly,
     yet make it small enough to effectively filter your noise. Why does
     this track the median? Because the estimate will tend to go to the
     point where 50% of the time it is going up, and 50% of the time
     it is going down. That is the median. If it the estimate was
     some place else, say at a point where you're going up 80% of the
     time, it would tend to move up. Of course, this is not a true
     median filter, but it is VERY easy to code and may work well
     for your application. Note that there is no reason why delta
     can't be smaller than the resolution of your input. This
     might help you choose a good compromise delta...
        Anyhow, I hope this helps. It sure beats going postal...

     --- Paul Dietz


______________________________ Reply Separator _________________________________
Subject: Re: Averaging Algorithm
Author:  PICLIST@MITVMA.MIT.EDU at WDI-INTERNET
Date:    9/25/97 2:47 PM


Paul Dietz Wrote:

>      There is a much better way! It is called a median filter.
>      Basically, this is just like a moving average (or more
>      generically, an FIR) filter, except that you take the median of
>      the data instead of the average. You do this by sorting the
>      data in the window by value, and then outputing the value which
>      ends up in the middle.
>

Alas!  I have spent a couple days messing around with median filters.
 The first one I wrote pushed large values off the end of the sort
stack, into lala land.  It would never climb up.  The next one pushed
them off the bottom of the sort stack.  Would never climb down.
Finally I got one that read in 32 analog values, sorted them, and
took the median as the setpoint.  Code works. BUT the  Relay fires
whenever I hit the lamp on my bench.  Noise immunity is still out of
reach after a lot of hard work.  Oh well, I know how to sort now.   I
fear the noise immunity is harder to buy than this.

I see three avenues:

1. Go postal.

2. Try a longer time delay between measurements, until I get around
the noise.

3. give up on median filters as being too complex, and going back to
averaging.

Another day, another $0.50 US.

Best Regards,

Lawrence Lile