April / Monthly / Tutorials / Software / Home

Files provide long term, high capacity data storage that is essential to all large scale computational undertakings. Since file input/output (file I/O) is necessarily a cornerstone of all ISIP software, the ISIP Foundation Classes (IFC's) require a standardized, efficient, and easy to use file I/O interface. Without such an interface, tremendous amounts of time would be wasted rewriting code to parse files for individual applications. ISIP's solution to this is the Signal Object File (Sof) class. This tutorial gives an overview of the Sof class as well as of four other IFC's involved in file I/O.

Signal Object File (Sof) class

Sophisticated file formats are an essential part of speech research since we often augment raw speech data with auxiliary information such as the recording conditions, annotations, etc. Even more importantly, as the native file I/O format for all ISIP classes Sof files provide a standardized way for all IFC's to write themselves to, and read themselves from, files. The standard interface makes file parsing for new applications simple and straightforward to write.

An Sof file is nothing more than an index, which provides information about the location of each object stored in the file, and the corresponding object data. Sof transparently supports two basic storage formats: text (useful for building human readable files such as parameter files) and binary (useful for sampled data). Sof also transparently converts binary data between different architectures by performing the appropriate byte transformations as needed. This prevents users from having to concern themselves with the low level details.

Example 1: 

      @ Sof v1.0 @
  
  @ Long 0 @
  value = 13;
  
  @ Long 1 @
  value = 27;

  @ Long 32 @
  value = -2812;
    An example of a very simple Sof file is shown to the right. The first line is the Sof file header. For a text file, it consists solely of the keyword Sof and a version number wrapped in the user-specified delimiter character. Whatever delimiter character is specified in the header will be used throughout the file.

    The third line of the file is an example of an object header. An object header has two components: a class name and an integral tag. The class name must be a single word - no spaces are allowed. The tag following the class name provides an ability to have multiple instances of the same object in a file. Every object written to an Sof file can be uniquely addressed by a (name, tag) pair.

    The first data in the file is an instance of the Long Scalar class. The data space for (Long,0) begins immediately on the line following the object header and ends at the second newline character. This extra line of space is not necessary, but helps to make the file more readable. Sof itself does not deal at all with this data space, it simply maintains the index of pointers and positions the file pointer to assist the higher level classes to read and write themselves from disk.
Example 2:
    Holding to the signal processing model of programming, one can easily imagine a case where every instance of some object (say speech signals) is operated on in turn. This can be simplified for example to a program that reads in and prints to the screen all long integers found in that file. The core parts of such a routine are as follows: 
      01  // file: $isip/doc/examples/class/io/io_example_00/example.cc
  02  // version: $Id: index.html,v 1.5 2006/06/30 22:40:01 ewt16 Exp $
  03  //
  04  
  05  // isip include files
  06  //
  07  #include <File.h>
  08  #include <Sof.h>
  09  #include <Long.h>
  10  #include <Console.h>
  11  
  12  // main program starts here:
  13  //  this program reads long integer entries from a text Sof file and prints
  14  //  each one found
  15  //
  16  int main(int argc, const char **argv) {
  17  
  18    // declare an Sof file object
  19    //
  20    Sof sof1;
  21    
  22    // open a file in read only mode:
  23    //  note that the Sof object determines whether the input file is text or
  24    //  binary automatically. in this example, it happens to be text.
  25    //
  26    String filename(L"./file.sof");
  27    sof1.open(filename, File::READ_ONLY);
  28  
  29    // declare a Long object used to read from the Sof file
  30    //
  31    Long j;
  32    
  33    // loop through all Long objects in the file, starting with the first
  34    // and ending when we have visited all objects with the given name
  35    //  note that the sof1 object is looking up the object based on its name
  36    //  which, in this case, is "Long". one could uniquely determine each Long
  37    //  object in the file by assigning each a different name and using that
  38    //  name to read in the object rather than the default name.
  39    //
  40    long tag = sof1.first(j.name());
  41    while (tag != Sof::NO_TAG) {
  42      
  43      // have the object read itself:
  44      //  this calls the Long::read method. each object in the math library
  45      //  and above knows how to read itself from an Sof file
  46      //
  47      j.read(sof1, tag, j.name());
  48      
  49      // output the object to the console
  50      //
  51      String output;
  52      output.assign(j);
  53      output.insert(L"I found the value " , 0);
  54      Console::put(output);
  55      
  56      // go to the next object
  57      //
  58      tag = sof1.next(j.name(), tag);
  59    }
  60  
  61    // close the input file
  62    //
  63    sof1.close();
  64  
  65    // exit gracefully
  66    //
  67    Integral::exit();
  68  }
One subtlety of this code example is that it also works on binary files with absolutely no changes! This demonstrates the degree to which the details of file I/O are abstracted from the user. The Long::read() method branches on the mode of the file, switching between formatted text and direct binary input. 

  @ Sof v1.0 @
  
  @ Long @
  value = 13;
  
  @ Long @
  value = 27;

  @ Long 32 @
  value = -2812;
In the previous example, the tag numbers were not at all useful to the program. While tags can be very useful for grouping together and ordering data (say parallel arrays of floating point numbers and integers), often they are not needed. To this end, text Sof files can be simplified even further by omitting the tag numbers. For the purpose of an iterative program such as the print routine above, the Sof file shown at right will behave exactly the same as the first example. Upon reading an object header with no tag, Sof will implicitly assign this object a unique tag. These tags are from a special range that will not be written back out to the file. The price of this feature is that tags less than - 2^30 (< - 1.07 billion) cannot be specified by the user.

From a high-level programmer's perspective, we have now covered the meat of Sof. The file is opened by passing a filename to an Sof::open() method. The optional arguments for the overloaded open methods specify the file access mode and a file type. The file access mode should be File::READ_ONLY, File::WRITE_ONLY, File::READ_PLUS, or File::WRITE_PLUS. The first two are self explanatory, File::READ_PLUS allows for reading and writing to an existing file, and File::WRITE_PLUS creates a new file with the option of reading data back out. The file type is either File::TEXT or File::BINARY, useful only for newly created files (File::WRITE_ONLY or File::WRITE_PLUS), as existing files already have a specified type. When a file is opened in a write mode a lock will automatically be obtained. Once a Sof file is open, the program needs only navigate the (name, tag) pairs before informing an object to read itself and calculate such tags to ask an object to write itself. The functions first() and last() return the first and last tags of a specified class name in the file. Additionally, next() and prev() allow for iteration through items of the same name, and the number of instances can be found with number(). Objects can be deleted from a file either one at a time, a named class at a time, or all at once with the three delete() methods. The entire file can be deleted from disk with delete_file(). Hooks are also available to copy data from one Sof file to another, changing the byte mode for binary files. Every other aspect of I/O is left to the objects.

File class

The File class abstracts file manipulations, which are operating system specific, and provides a general interface that all IFC's use internally to access files. The File class capabilities include, but are not limited to:
  • Reading binary or text from files or from stdin

  • Writing binary or text to files or to stdout

  • Creation and removal of directories

  • Creation, tracking, and automatic removal of temporary files

  • Automatic handling of system-specific byte-order modes

The file class handles most low level file operations for the IFC's. It is not intended for direct interfacing with the user.

Filename class

The Filename class can possesses a few file manipulation capabilities. Rather than building one directory at a time, as does the File class, the Filename class can build entire paths at once. It can also retrieve the file's extension, directory, or operating system of origin.

AudioFile class

The AudioFile class inherits the File class, which is to say it possesses all of the capabilities of the File class, and some added abilities of it's own. The AudioFile class was created to handle external (non-IFC) audio data formats. It can read and write raw, Microsoft wav, and NIST SPHERE formats in addition to the IFC standard Sof format. It can be modified to allow other data formats. Another useful feature is the ability to read data from a required sample index or sample time range from a file.

FeatureFile class

The FeatureFile class inherits the File class. The FeatureFile class is another IFC which must work with external data formats; it is used to manage feature data, obtained either in Sof form from files produced by isip_transform or from other sources in raw binary or raw text formats.

Refer to the online manual pages for complete descriptions of the above mentioned classes. To see detailed examples of the usage of these classes, refer to any of the utilities provided in our software distribution.