Man pages sections > man2 > diststream

idiststream, odiststream - distributed interface for large data

diststream(2rheolef) rheolef-6.7 diststream(2rheolef)

NAME

idiststream, odiststream - distributed interface for large data streams

DESCRIPTION

This class provides a parallel and distributed stream interface for large data management. File decompresion is assumed using gzip and a recursive seach in a directory list is provided for input.
 
     odiststream foo("NAME", "suffix");
 
is similar
 
     ofstream foo("NAME.suffix").
 
The main difference is that writting to ofstream is executed on all processes while odiststream manage nicely the multi-process environment. For convenience, the standard streams cin, cout and cerr are extended to din, dout and derr. Notice that, as orheostream (see rheostream(2)), if NAME does not end with ` .suffix', then ` .suffix' is automatically added. By default, compression is performed on the fly with gzip, adding an additional ` .gz' suffix. The flush action is nicely handled in compression mode:
 
     foo.flush();
 
This feature allows intermediate results to be available during long computations. The compression can be deactivated while opening a file by an optional argument:
 
     odiststream foo("NAME", "suffix", io::nogz);
 
An existing compressed file can be reopen in append mode: new results will be appended at the end of an existing file:
 
     odiststream foo("NAME", "suffix", io::app);
 
Conversely,
 
        irheostream foo("NAME","suffix");
 
is similar to
 
        ifstream foo("NAME.suffix").
 
Also idiststream manage nicely the multi-process environment. However, we look at a search path environment variable RHEOPATH in order to find NAME while suffix is assumed. Moreover, gzip compressed files, ending with the ` .gz' suffix is assumed, and decompression is done.

IMPLEMENTATION

class odiststream {
public:
  typedef std::size_t size_type;
// allocators/deallocators:
odiststream(); odiststream (std::string filename, std::string suffix = "", io::mode_type mode = io::out, const communicator& comm = communicator()); odiststream (std::string filename, io::mode_type mode, const communicator& comm = communicator()); odiststream (std::string filename, std::string suffix, const communicator& comm); odiststream (std::string filename, const communicator& comm); odiststream(std::ostream& os, const communicator& comm = communicator()); ~odiststream();
// modifiers:
void open (std::string filename, std::string suffix = "", io::mode_type mode = io::out, const communicator& comm = communicator()); void open (std::string filename, io::mode_type mode, const communicator& comm = communicator()); void open (std::string filename, std::string suffix, const communicator& comm); void open (std::string filename, const communicator& comm); void flush(); void close();
// accessors:
const communicator& comm() const { return _comm; } bool good() const; operator bool() const { return good(); } static size_type io_proc();
// internals:
std::ostream& os(); bool nop();
protected: // data: std::ostream* _ptr_os; bool _use_alloc; communicator _comm; };
 

IMPLEMENTATION

class idiststream {
public:
  typedef std::size_t size_type;
// allocators/deallocators:
idiststream(); idiststream (std::istream& is, const communicator& comm = communicator()); idiststream (std::string filename, std::string suffix = "", const communicator& comm = communicator()); ~idiststream();
// modifiers:
void open (std::string filename, std::string suffix = "", const communicator& comm = communicator()); void close();
// accessors:
const communicator& comm() const { return _comm; } bool good() const; operator bool() const { return good(); } static size_type io_proc();
// internals:
std::istream& is(); bool nop(); bool do_load();
protected: // data: std::istream* _ptr_is; bool _use_alloc; communicator _comm; };
 
 

SEE ALSO

rheostream(2)
rheolef-6.7 rheolef-6.7