Hi

Lately (and some time before that) there has been discussion on streams vs.
producers, and shortcomings of the stream API and the producer/consumer API.
See: ...


Additionally there have been some questions on how to make producers and
consumers work. I think the main reason for this is that there are a few
blanks that one has to fill out oneself when working with producer/consumer.
Both in the documentation and API.

This presents a suggestion for an improved producer/consumer API (in a 
somewhat unstructured manner).

I have not fully investigated the critisism of streams, and neither the
shortcomings of the existing producer/consumer API. I have however used
both the stream and producer/consumer API, and my suggestion is based on
my experience from these:



Revamped Twisted Producer Consumer API:

Goals:

Make it possible to create producers without knowing consumers in advance
 (this is often possible, but not always, see protocols.basic.FileSender)
This has the side effect that producers and consumers can be created in any
order and then paired up.

Make it straightforward to pair up a producer and consumer.

Have a standard method for getting information on completeness/errors
from the process.

Make it possible to have a non event driven producer-consumer pair

Don't change everything.


Changes:

- make all consumer have finish()

Consumers who do not need it, can just have a method definition
and a pass statement. This simplifies creating producers as they
do not have to check if the consumer adheres to IFinishableConsumer
before calling finish() (has this ever been done?/is finish() REALLY
necessary)


- add a beginProducing(consumer)

This method returns a deferred, which will fire when the producer does
not have any more data to produce or stopProducing is called. In case
of any errors, the deferred errback. Setting data source, range, etc,
must be done in the constructor. See protocols.basic.FileSender, for
how not to do it (this construction makes it impossible to create the
producer before creating the consumer).

This method should call consumer.registerProducing, to create a pair.

This makes it possible to use producers in the web2 api, as there is now
a standard method for starting to produce data.


- Add a class called something like ConsumerPullHelper

This wraps a consumer, and must be paired combined with a pullproducer.  
When started, the helper will start pulling data from the producer and
write it to the consumer. This makes it possible to write consumers which
can does not have to do pulling (and hence also can be used for push
producers). Of course if your consumer is event-driven it will have to its
own polling.

The functionality of this is basically to call producer.resumeProducing()
repetively. This is not particularly nice or anything, but sometimes it is
necessary. It will also decrease performance somewhat in certain cases
(I think).



Okay.... just read:
http://twistedmatrix.com/trac/browser/sandbox/exarkun/consumer-sketch.py?rev=17753

There are some similarities. The produceTo is pretty much the same as
beginProducing.

finish is gone - good/bad? I think there should be only one consumer interface,
so this is better. Is finish() used anywhere?


The consumer helper could still be built on top. It might also be viewed as
producer helper.

How do push producers work in the new model?

Not sure what the whole transport / buffer thing is about (besides some
notification of changes, which someone probably has a use case for).


Otherwise it looks ok. There is just the matter of getting things made :-)




Questions / Constructive critisism welcome.