Vitaliy wrote:

> Gerhard Fiedler wrote:
>>>> While it's true that you can process data "on the fly", I don't
>>>> like the coupling it creates. I like it when the responsibilities
>>>> of each module are clearly defined. Processing data is not the
>>>> comm module's responsibility, its job is to assemble a packet and
>>>> pass it onto the next level.
>>> 
>>> Ah, you're still stuck in the big system mindset.  Just because the
>>> data is handled as a stream doesn't mean protocol layering and
>>> module boundaries need to be broken.  Consider the comm mondule's
>>> job as presenting a byte stream to the next layer up instead of a
>>> packet. Let's say data is arriving via a UART.  The comm module
>>> might handle flow control, escape codes, validating packets, and
>>> extracting the payload in those packets.  It presents a GET_BYTE
>>> routine to the next level up.  As one packet is exhausted, it
>>> automatically switches to the payload from the next. 
>> 
>> This is actually not only a small-system concern. If you're dealing
>> with lots of data, like gigabytes, each layer of buffering costs a
>> lot, even on today's typical server machines. If you do the typical
>> small-data decoupling between layers, which consists of a lot of
>> copying of data from one layer's object structure to the next
>> layer's object structure, you can easily end up with a dozen of
>> objects where data is copied without need. 
> 
> Have you guys ever heard of passing data by reference?

Short answer: No... can you please explain?

Longer answer: You know enough about my background that it could have
occurred to you that I do have heard about this, and that probably there
is a reason why I didn't consider it in the scenario I was talking
about. Of course a number of these layers do something to the data,
that's usually (in part) the reason for their existence... so just
passing the same data through the layers doesn't really apply here. Your
typical data base end user GUI wouldn't be happy with a reference to
gigabytes of raw stream data from disk when the user only asked about
the fifteen numbers that are the totals by country and their sum, for
example. /Somewhere/ this data has to be processed, and not just passed
through by reference. 

If you're dealing with lots of data ("lots" in relation to the capacity
of the system, no matter whether big or small), streaming the data
through the processing stages without buffering it, as much as possible,
is generally the way to go, efficiency-wise. Check out e.g.
http://www.vectorwise.com (there is a link to a white paper about the
technology). This and the concerns addressed here are of course way off
the normal desktop computer programming -- but that's what I was talking
about.

Gerhard
-- 
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist