pyspark.streaming.StreamingContext.queueStream

StreamingContext.queueStream(rdds: List[pyspark.rdd.RDD[T]], oneAtATime: bool = True, default: Optional[pyspark.rdd.RDD[T]] = None) → pyspark.streaming.dstream.DStream[T]

Create an input stream from a queue of RDDs or list. In each batch, it will process either one or all of the RDDs returned by the queue.

Parameters

rddslist

Queue of RDDs

oneAtATimebool, optional

pick one rdd each time or pick all of them once.

defaultpyspark.RDD, optional

The default rdd if no more in rdds

Notes

Changes to the queue after the stream is created will not be recognized.

previous

pyspark.streaming.StreamingContext.binaryRecordsStream

next

pyspark.streaming.StreamingContext.socketTextStream