On Wed, 4 Nov 1998, Stefan Sczekalla-Waldschmidt wrote:

> Hello Scott,
>
>
> While I still have to look at your Web-Page, discussing with two friends
> in the amateur-radio-club yesterday evening gave also some new Ideas to
> me.
>
> While the dataset I4ll look at is rather small we came up with the Idea
> to first
> filter the data for extremes, using a kind of modifiable window below
> and over the expected center data. the size and position of the window
> is modified by monitoring how often per time the window borders are
> missed. The values passing this "filter" will
> be stored in the well known circular buffer. because of having this data
> filtered in fornt of storing, there should be no need for sorting - just
> average the data.
>
> What do you think about this idea - beside I dont know if this way would
> have been useful for you.

If you know the range before hand, then this would make more sense.
However, a median filter isn't necessarily filtering just out-of-range
data. As Lawrence and others have said before, it's intended to instead
filter that occasional noise spike that is not representative of the data
(or at least representative of that portion of data for which you're
interested).

Or to say it differently, the median filter works by truncating the
extremes of the last N samples. So for example, suppose you have 'normal
ranging' data and then you get a 'spike'. The median filter would remove
this spike. But suppose that the 'spike' is really a step function - the
new trend in the data. Well eventually, the last N samples will all be at
the same level as the 'spike' and the median filter will pass this new
value. So the point I'm trying to make, is that the thresholding is in the
indices of the sorted data and not in the values of the data.

If you want to protect against invalid, out-or-range data, then threshold
the values as they're obtained. If you want to remove the occasional (and
possible expected, but undesirable) spike, then use a median filter. In
either case, you may wish to add a subsequent linear filter (e.g. FIR/IIR
type which could be as simple as finding their average). However, placing
a median threshold around the current average (as you discuss above) may
prevent the algorithm from tracking a step function.

Does this make sense?


Scott