This article is formatted for a 1024x768 fullscreen browser

the iostream interface

The C++ stream library provides an extensible mechanism for delivering text to and from the user.  Due to the disdain many programmers feel towards the cout and cin objects (I empathize - those were gloomy days indeed), they dismiss the C++ iostream as incapable of playing any important role in advanced programs.  To this I say, "Halt, code monkeys!"  The iostream is not the simplistic and inconsequential entity you suspect...  It is a highly extensible, highly powerful, highly obfuscated class hierarchy!  Follow me on this iostream expedition - we will derive the infamous stream buffer and pump iostream data where ever we want!  What could possibly be more entertaining?

First, let's examine the superficial aspects of the iostream.  Residing in the <istream> header is the basic_istream<> template class.  The basic_istream<> class definition starts like this:

The template arguments define the character type (_E is generally a char or wchar_t) and the traits of that character type.  The default value char_traits<_E> is a set of static functions for performing basic operations on characters such as comparisons, casting, and finding lengths.  Consult the <iosfwd> header for more details.

To make the functionality of basic_istream<> available to inexperienced programmers, the designers of the C++ standard library have defined the following type definitions:

All iostream objects that expose input functionality implement basic_istream<>.  Output functionality is a similar story.  Found in the <ostream> header, the template class basic_ostream<> also has character type and character trait template arguments:

The standards committee has been consistent in its use of type definitions:

Iostream objects that expose output functionality implement basic_ostream<>.

So what mechanisms are used to deliver this input and output functionality that we are all too familiar with?  Class methods and extractors/inserters.  The methods of basic_istream<> and basic_ostream<> define the required functionality of the objects that implement them.   Let's look at the public methods of these two classes:

These methods define the mandatory functionality for the objects that implement them.  While these two classes are templates and will accept any data type, each of these methods must be defined for that particular data type.  For example, passing a template argument of wchar_t (the UNICODE data type) when instantiating a basic_ostream<> object requires that all of basic_ostream<>'s methods be defined for wchar_t in order for the code to compile.  The first thing you need to ask is "does it make sense?"  Does basic_ostream<wchar_t>& basic_ostream<wchar_t>::operator<<(int n) make sense?  Can you serialize an integer into a stream of wchar_ts?  What about basic_ostream<wchar_t>& basic_ostream<wchar_t>::seekp(pos_type pos) - can you seek for a particular character in a stream?  Yes on both counts!  Examining the list of basic_ostream<> methods one by one, we see that they all "make sense" for the wchar_t data type.  In fact, the char_traits<> template structure has already been specialized for wchar_t (look in the <iosfwd> header), so the work required to instantiate such an object is really minimal. 

Now let's consider instantiating a basic_ostream<> object that uses a class Cow data type.  Are there any methods in basic_ostream<> that are illogical for such a data type?  basic_ostream<Cow>& basic_ostream<Cow>::seekp(pos_type pos) is perfectly reasonable - you can seek out a particular cow from a stream (or pasture).  What about basic_ostream<Cow>& basic_ostream<Cow>::flush()?  This is legitimate too - a few years ago Texas flooded and all the cows got flushed.  Finally, let's consider basic_ostream<Cow>& basic_ostream<Cow>::operator<<(void * n).  Is it possible to serialize a pointer to a stream of cows?  For this we consulted cow expert Oprah Winfrey, who answered with a resounding "no."  Because all of the methods of basic_ostream<Cow> cannot be defined, class Cow does not make a good data type for the iostream.  Only wchar_t and char  have the supporting class framework already defined for them.  If you wish to use some custom character data type with the iostream you are free to do so; however, defining the supporting class framework will not be fun.

You've seen how the methods of basic_istream<> and basic_ostream<> expose some of the functionality of the iostream.  This functionality isn't extensible though.  The designers of the iostream have devised an elegant and simple architecture to give both the input stream and output stream extra kick: the extractor and the inserter.  These two entities leverage C++'s operator overloading capabilities to provide the programmer with a very consistent mechanism for adding types that can accept a text stream and be serialized into a text stream.

First, let's look at the extractor.  As its name suggests, this mechanism extracts text from the input stream and serializes it into a type.  The <istream> header defines a number of standard extractors, including the popular extractor to read data into a C character string:

Look back at the list of public methods for basic_istream<>.  Nowhere will you find basic_istream<_E>& basic_istream<_E>::operator>>(_E* s), the method whose signature matches the above snippet of code.  Instead, this functionality is delivered via an extractor.  The C++ iostream designers discovered that such a function didn't require access to the private parts of basic_istream<>, so they implemented it as a static function.  The prototype is simple:

template<class _E, class _Tr> basic_istream<_E, _Tr>&
        operator>>(basic_istream<_E, _Tr>&, _E* s).

All extractors have a similar prototype.  See the <istream> header for the set of standard extractors.

The output stream can be extended with the extractor's twin brother, the inserter.  Like extractors, inserters are static functions that overload the bitshift operators to extend the functionality of the iostream.  This small example program shows a simple inserter in action.  Notice that the inserter functions are friends of class boo; this allows them to access boo's private data.  I have defined both an ANSI character and wide character version of the inserter to allow it to work with both data types:

The output is exactly as intended:

This system of user-defined extractors and inserters is more than just a syntactical convience to the programmer; as we'll explore in the next section, these seemingly innocent functions contribute to a very flexible framework of streaming text to and from any source.

iostream polymorphism

The C++ standard library defines three types of instantiatable iostream objects: the file stream, the standard string stream, and the C string stream.  The class definitions are found in the headers <fstream>, <sstream>, and <strstream>, respectively.  Each of the types comes in three flavors: input stream (implementing basic_istream<>), output stream (implementing basic_ostream<>), and input and output stream (implementing both basic_istream<> and basic_ostream<>).  All together we have nine C++ classes:

It's hard not to notice that the C string classes aren't templated.   They manage only C character strings.  The templated classes from <sstream> are prefered over the more primitive classes of <strstream>. 

Let's take a peek at an example program that reveals how all output stream objects are polymorphic.  This program defines a function BlastString that takes a reference to a basic_ostream<char> object and feeds text through it using both an inserter (for the C character array) and a basic_ostream<char> method (for outputting the single character - the target name).  The program creates a basic_ostringstream<char> object called osA which streams all output into an std::string which can be accessed through the basic_ostringstream::str() method.  osB is an instance of the ostrstream class.  This object streams text into a character array.  The method ostrstream::str() returns a pointer to that character array.

And the output is:

Ahh, terrific.  So much to analyze here.  We pass each of our three stream objects to BlastString, which works with their basic_ostream<char> slices.  This function uses the character array inserter to blast text through each output stream.  Recall that inserters and extractors are static functions; they are not class methods.  Therefore, they cannot be virtual.  If the inserter isn't virtual, how can it operate on objects with much different functionality?  Both the ostringstream and ostrstream objects write to internal strings rather than the console.  To demonstrate that BlastString really does work on all basic_ostream<>-derived types regardless of data target, we spit their strings out to the console.  How does the basic_ostream<> support this polymorphism?  You may not suspect magic, but after this next example, you should: 

Two lines of output is all she wrote:

Now let's follow through the program's execution.  Just like the previous example, our three output stream objects are created.  Only object osC will write to the console; the other two write to strings.  I perform some iostream magic which leaves the output stream objects confused and disoriented.  In their code-induced daze, each object attempts to recognize its fragile identity by streaming some text.  Now much to our dismay, osA, the ostringstream object, writes not to its string but to the console!  Also, osB writes to osA's string and osC writes to osB's string.  From this odd behavior we conclude something very important: the functionality of an iostream object is not specified by deriving basic_ostream<> or basic_istream<> and redefining their methods.  Why then would anyone derive classes from these two bases?  The answers are surprisingly intuitive.  Number one: by implementing a derived class, the programmer exposes new methods that offer new functionality to the client (for example basic_ostringstream<>::str() gives the client programmer a copy of the object's output string).   Number two: construction and destruction of the iostream object can be simplified through the use of a derived constructor and virtual destructor.  So what is so important about iostream object construction and destruction that requires instantiation of a derived class?  Let's take a look at the constructors of basic_ifstream<> (any other class derived from basic_ostream<> or basic_istream<> will prove the point just as well):

It is helpful to review the order in which slices of a class hierarchy are constructed.  Let's take the default constructor apart first.   When an instance of basic_ifstream<> is created using the default constructor, basic_ifstream<>::basic_ifstream() passes the address of the _Fb file buffer to the base class constructor basic_istream<>::basic_istream(basic_streambuf<_E, _Tr>*, bool), which in turn invokes the trivial, protected default constructor of its base class basic_ios<> (and ultimately the protected constructor of the root of the inheritence chain, ios_base).  In other words, constructors are called in order of increasing abstraction. 

After the constructors of basic_ifstream<>'s base classes have finished executing, the object's own data members are constructed.  In this case there is only a single data member:  basic_filebuf<> _Fb.  After this file buffer has finished constructing itself, the code of basic_ifstream<>'s constructor is finally executed.  The default constructor executes no code, so our object is constructed and ready for use.  If the client programmer has invoked the two-argument constructor, the open(const char*, ios_base::openmode) method is invoked on the file buffer object and a failbit is set accordingly. 

So what's the point of deriving a type from basic_istream<> and defining constructors that merely delegate to the base classes?  The answer is a product of keen design: a class's constructor implicitly invokes the constructors of all the type's data members.  Additionally, the presence of a destructor implicitly destroys the type's data members.  A key concept in object oriented design is that the scope of outer objects define the lifetimes of inner objects.  Without deriving basic_istream<> or basic_ostream<>, there is no safe mechanism for governing the scope of the object's stream buffer. 

Now it's time to move past this irritating intro material and really scrutinize our friend the stream buffer.  basic_filebuf<> is, of course, derived from basic_streambuf<>.  As we whip out our coveted printed Standard Library source code and turn to header <streambuf>, we are surprised to see our friend, the virtual keyword, modifying method declarations.  With the exception of the virtual destructor, our seven-letter friend has been absent from the basic_istream<> and basic_ostream<> hierarchies.   Could the stream buffer's virtual functions be the source of the mysterious polymorphism demonstrated above?  An enticing conjecture indeed...

the C++ stream buffer

The introduction is over; now it's time to dig into the details of the stream buffer and derive it all the way to the Mexican border.  At the most fundamental level a stream buffer supports two operations.  You can put data to a stream buffer (think output stream) and get data out of it (this provides the input stream functionality).  There are a host of basic_streambuf<> protected methods that control how the put area and get area are accessed:

The six stream pointers of basic_streambuf<> control how data is stored to and read from the buffer.  Let's take a look at the trio of get pointers.  The pointer returned from basic_streambuf<>::eback() points to the start of the get buffer.  This get buffer stores the data sought by the methods of basic_istream<> and its extractors.  The method egptr() defines the buffer's size by pointing to one element past the end of the get buffer.  This is consistent with the convention employed in the STL containers' end() methods.  In addition to the beginning and end pointers of the buffer, the get area defines a current position with basic_streambuf<>::gptr().  The gbump(), _Gninc(), and _Gndec() methods move the current position pointer by the appropriate amount after a read or write to the get buffer.

The put area exhibits similar behavior and ways of access.  The pbase() method of basic_streambuf<> returns the beginning of the output buffer.  epptr() is the semantic equivalent of egptr() - it returns one past the end of the put buffer.  As expected, pptr() retrieves the current position pointer into the put area.  pbump() and _Pninc() advance the pptr() pointer after inserting data into the put area.  Unlike the get buffer, the put area has no decrement current position method.  Why is this?  The Standard Library implementors didn't want to encourage programmers to decrement the put current position pointer and re-write over data, potentially corrupting the stream.  Reading old data is a harmless activity, but writing over data before it has been flushed may lead to unexpected program behavior.

Stream buffers come in two flavors: buffered and unbuffered.   Both types will achieve the same results, but there are many scenarios in which a buffered system is more efficient.  So what are the real differences between the two?   For convience BSB will mean "buffered stream buffer" and USB will mean "unbuffered stream buffer."  The BSB allocates an array of elements for the put buffer and/or the get buffer.  It is perfectly acceptable to maintain a buffered put buffer and an unbuffered get buffer, or vice versa.  If the appropriate current position pointer (pptr() for the output buffer and gptr() for the input buffer) returns zero, the stream is unbuffered.  Otherwise, the stream is buffered and the beginning of buffer and end of buffer pointers are also defined.  A BSB can be written to when the pptr() < epptr().  Similarly, a BSB can be read from when gptr() < egptr().  Finally, when the get area is a BSB and gptr() > eback(), you can put back the last read character by invoking the _Gdec() method.  Notice that you aren't required to copy the data to this decremented position - the data was never erased.   Let's make a list of these rules:

• If gptr() is null, the get area is unbuffered.  Otherwise the get area is buffered and the pointers egptr() and eback() both point into the get stream.
• If pptr() is null, the put area is unbuffered.  Otherwise the put area is buffered and the pointers epptr() and ebase() both point into the put stream.
• If the get area is buffered (i.e. gptr() is non-null) and gptr() < egptr(), you can read an element from the stream.  Be sure to invoke _Gninc() after each read to advance the get pointer to the next element.
• If the put area is buffered (i.e. pptr() is non-null) and pptr() < epptr(), you can write an element to the stream.  Be sure to invoke _Pninc() after each write to avoid writing over elements.
• If the get area is buffered and gptr() > eback(), invoking the _Gndec() method has the effect of putting the last read character back into the get buffer.

You can set the get and put buffers by calling one of the _Init() methods.  The default method sets all six pointers to zero, defining both the put and get areas as unbuffered.  The six parameter _Init() method directly sets the six buffer pointers.  You can set one area to buffered and the other to unbuffered by passing the addresses of zero integers and zero pointers as appropriate.  Note that the third and sixth parameters are pointers to integers.  These integers define the length of the buffer.  The epptr() and egptr() methods perform a simple arithmetic operation to return the address of the element one past the end of the buffer.

We now understand how the get and put buffers are defined and accessed.  What we don't know is how buffered data is read from a source or sent to a target.  Unbuffered i/o is also still a mystery.  These behaviors are polymorphic and are responsible for the variety of behaviors amongst output and input streams.  The following methods of basic_streambuf<> allow for tremendous flexibility and are the subject of this article:

Don't worry; it's simpler than it looks.  The public methods I've listed have something in common: they all delegate to one or more of basic_streambuf<>'s protected virtual members.  These public methods are accessed by the methods of basic_ostream<>, basic_istream<>, and their inserters and extractors.  Because our aim is to derive a basic_streambuf<> class, we will only study the virtual methods.  It is unlikely that you will ever deal directly with the public methods.

Let's break down the virtual functions into categories.  Each method deals with either the get area (input) or the put area (output).   Additionally, each of these areas can be buffered or unbuffered.  Here's the breakdown:

The most common method for reading in characters is the basic_streambuf<>::xsgetn() method.  This method is invoked directly from the public method basic_streambuf<>::sgetn().  xsgetn() copies _N elements starting at gptr() to the parameter array _S.  If the operation has read up to the end of the get buffer and still requires more input or if the system is unbuffered, the xsgetn() implementation could invoke the method basic_streambuf<>::uflow().  Calling uflow()is a request for the next character.  An USB will attempt to return this character from its source through whatever means possible.  If it cannot, it returns _Tr::eof(), signaling an error.  If the system is buffered, uflow() may invoke the basic_streambuf<>::underflow() method.  underflow() will attempt to refresh the get buffer.  Popular techniques for this include disposing of the existing data and refilling the buffer by querying the input source.  The get buffer can also be extended, and this new space filled with new input.  Letting a vector<> or similar datatype provide the get buffer simplifies the process; the vector<>::resize() and push_back() methods will automatically extend the buffer.  After the input buffer is filled with data, the three get pointers are adjusted to reflect the changes.  The next character is returned from underflow(), but the _Gninc() method is not called.  If underflow() cannot grab new input, return _Tr::eof().  Before forwarding underflow()'s return valler to the caller, uflow() needs to increment the get pointer.  Study the default implementations of these methods before reading on.

The put buffer methods are consistent with those just discussed.  The primary output function is basic_streambuf<>::xsputn(), and is publically accessed through a call to basic_streambuf<>::sputn().  xsputn() copies _N elements from the parameter array _S to the put buffer.  If the put area is unbuffered or the pptr() pointer equals the epptr(), xstpun() passes  basic_streambuf<>::overflow() the next character to be output.  The implementor of overflow() has many options.  For a BSB, overflow() can send the put buffer to the output device, write the character that was passed to overflow() to the location eback(), and reset the gptr() to eback() + 1.  The put buffer can also be extended (again, letting a vector<> provide the put and get buffers makes extending the buffer as simple as a call to vector<>::resize()), and the argument character written to the position pptr().  If this operation succeeds, _Pninc() should be invoked and _Tr::not_eof() returned.  For an USB, overflow() must either send the parameter character to the output device or return the error code _Tr::eof(). 

This leaves but one unanswered question: how do you get buffered output from the put area to the output device?  basic_ostream<> provides a public method called flush().  flush() is simplicity itself - it merely delegates to basic_ostream<>::rdbuf()->pubsync().  basic_streambuf<>::pubsync() just returns basic_streambuf<>::sync() - a virtual function.  sync() is also an appropriate place to flush the get buffer.  So does that mean you need to continually call flush() on your output and input stream objects to keep their devices synched?   That would really be inconvenient.  If you examine any of the iostream output functions, you'll notice that they all build a basic_ostream<>::sentry before working with the stream buffer.  When the stream buffer work is done the stack unwinds and the sentry object is destroyed.  This class's destructor calls a method osfx(), which compares a bit unitbuf then delegates to flush().  unitbuf is one of iostream's many format flags.  Unless the stream buffer should not be flushed after every output for performance reasons, be sure to leave the unitbuf flag on. 

basic_istream<> exhibits similar pattern of flushing.  basic_istream<>::sentry (which is built and destroyed with every input) does its work not in the destructor, but in the constructor.  An input stream is said to be "tied" with an output stream if they share the same device.  Imagine what would happen if you foolishly turned off the unitbuf bit on the cout object.  You attempt to great the user with a friendly "Hello World!" but the "World!" part is left in the put buffer.   You've turned off automatic flushing but are confident that "World!" will be synched with the following output operation.  However, your next statement is a cin>> myString; operation.  This is a blocking call, so the program can't continue until the user types a word in.  The user sees this apparent crash and types in "Stinkin'" to express his anger.  This is just the input that the program required!  It then resumes with output, and eventually empties the rest of the put buffer to the screen.   So what does the user ultimately see?  "Hello Stinkin'World!"   Not a good way to start your day.  We could prevent these potential problems if basic_istream<> flushed the tied output stream before grabbing any input from its source.  This is exactly what the sentry constructor does; it invokes flush on the tied output stream.  Oh, happy day.

It is not mandatory that you implement your basic_streambuf<>-derived class by defining the virtual methods in exactly the manner described above.  If your output or input device requires a solution that is not consistent with the above recommendations, that's just fine.  Feel free to implement the virtual functions in whichever style you deem appropriate.  However if you do follow these standard conventions, you can safely fall back on the default implementations of many methods.  For example, the basic_streambuf<>-derived class basic_filebuf<> (used not only for file i/o but also for console i/o) does not override the methods xsputn() and xsgetn().  The class  follows the standard conventions, so the default implementation of these two important functions was sufficient. 

A word of warning: implementing unbuffered input is not as easy as it sounds.  If you are having trouble getting unbuffered input to work, consider implementing buffered input instead.  The stream buffer is designed for maximum flexibility and is somewhat overengineered for simple applications.  Without really studying the guts of the system, implementing unbuffered io may seem a bit mysterious.

There are many virtual methods of basic_streambuf<> that we didn't cover.  A typical user of the iostream will rarely, if ever, need these methods.  The default implementations will return the customary _Tr::eof() error code, so you don't need to worry that not implementing these functions will lead to unexpected behavior.

Now brace yourself.  I'm going to put all of this garbage to work and derive the stream buffer.  If all goes well (and it had damn well better), we'll end up with an iostream class that lets us instantiate multiple console windows of arbitrary dimensions and color, all of which live in a common thread!

endgame: deriving the C++ stream buffer

We will attack this problem in two distinct steps.  First we'll build our console class.  This class has no particular affinity to the iostream - it will support only a small set of methods.  Although this console class will let us write and read to its window, it is a terribly unelegant solution.  For brevity (and oh how I do love brevity), we will not equip this class with the ability to backspace.   This is one of the many improvements I'll unload on the reader.

Aside from the basic_streambuf<char, char_traits<char> > implementation, the most interesting aspect of this project is its reliance on a single UI thread.  This class maintains a static reference count.  When the reference count is zero and a new console window is being constructed, a UI thread is created.  All future console windows will live in this thread.  When each window is destroyed, the static reference count is decremented.  When this reference count reaches zero, the UI thread is destroyed. 

Each console window can be instantiated in any foreground and background color combination.  The Courier New 16point font is standard, but the number of rows and columns of text can be determined by the client code at runtime.   Let's look at the console class definition to surmise what we're up against:

This class has a fair number of data members, but only two public methods.  _winzooconsole::drawcharacter() simply renders the parameter character to the appropriate position on the console window.  _winzooconsole::getinput() reads in a character from the window and returns it to the caller.  The _winzooconsole::params structure holds relevent rendering information.  Now let's take a look at our class's constructor.  Note that we register our window class in the static definition.   This is a convienence; if we had registered the window class in _winzooconsole's constructor we'd have had to device some goofy mechanism to ensure that we didn't re-register the class.

This is just good old fashioned Windows API.  Note the call to ::CreateThread() at the beginning of the constructor.  We've put this call inside an event to prevent nasty race conditions.   Let me draw your attention to the ::Sleep(0) statement immediately following the CreateThread().  The created UI thread needs to call ::GetMessage() once to initialize its message pump.  If you post a message to this thread before it has had a timeslice to execute GetMessage(), the message will be lost.  Towards the bottom of the constructor we make a call to ::PostThreadMessage(), specifying the WM_APP_CREATEWINDOW message.  We've filled out the _params structure and are passing its address as the lParam of our WM_APP_CREATEWINDOW message.  Alright - that was simple enough.  Bring on the console's destructor and thread function:

This code looks rather mean at first glance.  However, notice the real absence of C++ keywords.  There is very little logic involved with this program - it's mostly just API calls.  The console's destructor waits for the _initordestroy event to make sure that the UI thread isn't being tampered with by another console constructor or destructor.  We delete our in-memory rendering context, free its GDI objects, and finally destroy our console window.  After decrementing the reference count, we check to see if there are any consoles left.  If there aren't, we post a WM_QUIT message to our UI thread.  GetMessage() will return false after pulling WM_QUIT of the queue.  This breaks the message pump and the thread function returns.  If a thread is going unused, why keep it around?

The message pump code is straight forward.  We check the queue to see if the current message is WM_APP_CREATEWINDOW.  If it is, we clap our heels in glee and cast the message's lParam to grab a reference to our rendering parameters structure.  A window is created of the requested size, figuring in the system metrics dimensions of the thin frame and caption bar.  Notice that we do not ask that the window be made immediately visible.  Instead we save the address of the console object in the window's GWL_USERDATA slot.  Now the window and its C++ object can communicate with each other.  You must be getting really anxious to see the stream buffer code, so I'll just dump the rest of our console class:

That's it for our console class!  Keep in mind that there are only two public methods: one for input and one for output.  It's barebones as far as convenience is concerned.  We will leverage the ungodly power of the iostream hierarchy to turn this unelegant solution into a paramount of usability.  Our stream buffer will support