Formats

 FORMATS

Introduction

Formats allow accessing a bitstream in a human-readable way; without a format, data are only “undifferentiated strings of bits”, which cannot be “interpreted and rendered in human-sensible form”  [1].

Preservation over long-term involves more than just accessing and interpreting data now; one has to be sure that access will be possible in the future. Like all components of digital information, formats (and the softwares and hardwares needed to read them) evolve at a rapid pace, become obsolete. If one cannot understand the way they are encoded, one cannot migrate them to new formats, nor emulate the environments in which they are readable – and the whole content is definitely lost.

Not only the sustainability of a format is essential, but also its ability to preserve content without any information loss. This is a key requirement for archives, whose mission is to preserve faithful testimonies, and ensure the communicability and comprehension over time. Loss, even often acceptable for current use, may be unacceptable for future use.

In this chapter, we will try to define a digital video format suitable for the long-term preservation of the essences. Of course, our conclusions will only apply to the master archives, and not to the access copies (e.g. DVD sold to the public, with lossy compression; or future enhanced documents for diffusion in auditorium), which are not in the field of what we call “archive” (documents aimed at long-term preservation). The preservation of the descriptive metadata (the database) will be discussed in chapter on metadata.

We will first define some key concepts, then present the quality and sustainability factors, and finally discuss the main advantages and disadvantages of existing video formats.

Some definitions

Compression

Compression allows saving costs in storage and transmission by reducing the amount of data. One can distinguish between lossless compression (the output from the decompressor is bit-for-bit identical with the original input to the compressor; one usually speaks about mathematically lossless compression, or reversible compression), and lossy compression (there is irreversible information loss, and the difference is perceptible at various levels). Some codecs use pseudo lossless compression, which means that there is visually no difference, but information is however irreversibly lost.

Of course, only uncompressed or losslessly compressed bitstreams are acceptable for archives, because the key requirement is information preservation: keeping the entire image in a pristine way (uncompressed or losslessly compressed) is after all the best possible insurance for maintaining the long term asset value.

In the field of digital video, there are only a few lossless codecs: some proprietary codecs (like HuffYUV), and only one open standard: JPEG 2000, which is considered as the only real lossless codec suitable for archives [8]. But some wrappers support uncompressed bitstreams, like AVI, QuickTime, or MXF.

We should also note that datatapes (LTO, AIT; etc.) may offer lossless compression, with a 2:1 ratio for LTO. But this compression is not optimized for video, so that the compressed bitstream may take more space than the uncompressed one [8]. On the opposite, compression algorithms used in digital video tape formats (like Digital Betacam) are all lossy.

Of course, lossy compression is very useful for diffusion, through the Internet or on DVD, for example. But only the access copies, not the master archives, should be compressed this way. For this purpose, a format scalable to various qualities is advantageous, because it allows to create various copies with one single original. JPEG 2000 is able to do that.

Wrappers

Wrappers, or containers formats, are file formats encapsulating the encoded bitstream and information about the data (relationships between the streams, representation information, preservation information, descriptive metadata, etc.). In the multimedia and video field, all formats are containers, because they have to store the image itself, eventually the sound, and the relationships between all those elements in order to create a sequence.

The OAIS model recommendation [5] advocates for the use of “packages” for the transmission and conservation of data objects; wrappers encapsulating essence and metadata are a way to implement this recommendation. A contrario, converting a video sequence into TIFF (for example) sequences, and combine them in a bundling format (like JAR, or TAR) is not a good solution, because audio would be completely separate.

[14] distinguishes between two categories of wrappers: codec wrappers, and rich metadata-supporting wrappers.

Codec wrappers, like Motion JPEG 2000, QuickTime or AVI, encapsulate video, audio and other bitstreams (the essence), compressed using any codec (for AVI) or a specific codec (for Motion JPEG 2000), together with basic technical and structural metadata, and sometimes bibliographic metadata. Some wrappers support uncompressed bitstreams (AVI and QuickTime, e.g.), others not (MPEG-4, for example). The aim of such wrappers is only to facilitate the playback of the files, but they may be suitable for long term preservation.

Rich metadata-supporting file wrappers, like MXF and AAF, contain in themselves all the means to display/read the essence; they are self-describing, and so self-containing, bundling formats [12] or packages. The OAIS reference model [5] recommends the use of such formats which haven’t any external dependencies (to format registries, for example), and which could theoretically last “forever” (it is possible to recreate a software from the metadata – Representation Information – even when all documentation would be lost). However, they are more intendended for what OAIS calls SIP (Submission Information Package) and DIP (Dissemination Information Package), so for exchange (MXF means Material eXchange Format), rather than for the preservation of an homogeneous collection, intended for very restricted internal use, like MJF is. On the opposite, one could wonder if simple codec wrappers are able to embed as many metadata as the OAIS model recommends, including reference metadata (see “self-documentation” below), and so if MXF could be the format for the AIP (Archived Information Package) too.

Preservation strategies

There are several strategies to assure preservation of essences throughout time, each with its risks and advantages [9]:

  • Technological preservation, which means maintaining the original equipment needed to render the information. It is not recommended to use this strategy for long-term preservation, because of the high costs for maintenance.
  • Technological emulation, which consists in recreating the original software environment. This strategy involves great technological efforts, and is maybe more adequate for complex formats, whose “look and feel” has to be preserved, than for basic file formats (such as videos).
  • Capturing with “flat formats”, which consists in converting the format in a much simpler one. This strategy induces lost of functionalities and of quality, which is not acceptable for video formats.
  • Migration to newer formats. The main problem with migration is the possible lost of information. To prevent it, one should migrate to standard formats, without lost (see the factors below), or to migrate only on request: an original input is preserved, and a software to create readable outputs is updated as one goes along with evolution [17] [18].
  • Modification of the file format for preservation, by adding metadata.
  • Encapsulation in wrappers like the Universal Preservation Format. The different elements of the file are preserved in their original form, combined with metadata. This strategy is more focused on preservation than on access. One should not get this strategy and the “self-documentation” factor (see below) mixed up: encapsulation doesn’t allow any evolution/migration of the format, while the “self-documentation” factor just refers to the ability of a format to contain the means of rendering the bitstream.
  • Filming, which is a strategy for complex file formats, consuming huge resource, and not at all adapted to video formats.

The best strategy for video preservation is probably migration (recommended by the OAIS reference model). In any case, whatever strategy is chosen, selecting a format that fits the requirements presented below allows reducing the future costs and the complexity of operations.

Selection criteria

One can find several approaches in the literature regarding the choice of a preservation file format: [12], [2], [4], or [11]. The only differences between them is in the way things are expressed, but the main principles are the same. In this synthesis, we will particularly lean on the criteria used by [12], because they are the most inclusive and the best structured ones.

Quality and functionality factors

The factors of quality and functionality designate the ability of a format to “represent the significant characteristics required or expected by current and future users of a given content item” [12].

This doesn’t mean defining the resolution requirements: digitization (or re-encoding) will be made at the “fullest resolution feasible with the current state of the art” [7], not only for archival purpose, but also for quality of diffusion; and the born-digital materials will be kept as they were created, or decoded and saved as lossless/uncompressed [15]. The question here is about determining whether a file format is able to support this high resolution or not.

Rendering

This term refers to a “baseline for the behavior of content when presented to users” [12]. In other words, the format must be adapted for current and future use. For video, the format should allow playback through a software that allows:

  • Single playback of the image with sound mono or stereo
  • Control over picture elements (brightness, hue, contrast), sound elements (volume, tone, balance) and navigation (rewind, fast forward, go to segment, etc.)
  • Analysis and excerpting of picture and sound

 

This normal rendering should not be limited to any specific hardware or device.

Most of currently publicly used video formats support this baseline through desktop PC applications, like Windows Media Player or QuickTime. But specialized formats, used in professional environments like archives or post-production, cannot reach directly this normal rendering. For example, the MXF format is not intended for PC applications, and requires a professional equipment. However, it is possible to “re-wrap” the streams to more readable format, or to transcode the file.

The signal processing research activities which will be conducted on the archives will require some specific applications going beyond the normal rendering (content indexing, HDTV output, etc.). Those requirements must be defined by the researchers themselves; but complex, over-specified formats are difficult to preserve (see “transparency” factor below) [2].

Clarity and fidelity

The term of clarity designates the support for high image resolution. It includes the “factors that will influence a careful (even expert) viewing experience” [12]. Following [14], the copy must capture the essence of [recordings] in such a way that the copy is essentially unchanged from the original”, in order to allow detailed analysis, HDTV (or any other high quality resolution) output, content-based retrieval, and other features. In other words, it should be able to “represent the complexity and fullness of meaning of the underlying content” [1].

The term of fidelity designates the support for high audio resolution. It includes the “factors that will influence a careful (even expert) listening experience” [12].

Clarity and fidelity are of course dependent from the quality of the original and from the way it is digitized; but to restitute it in the best possible way, the selected format should be able to support:

  • Larger picture size (full screen resolution).
  • Sampling rate: for image, 4:2:2 subsampling is acceptable, even if there is information loss. 4:4:4 sampling is better (but cannot be stored on video tapes). For audio, the recommended sampling rate is 96 KHz.
  • Higher bit rate. For audio, 24 bits is recommended
  • Extended dynamic range
  • Codec: if possible, lossless or uncompressed. It is recommended to avoid temporally compressed bitstreams, so to maintain the frame integrity of individual frames in a stream.

 

Sustainability factors

As a general factor, any preservation file format should conform to the OAIS reference model [4]. The OAIS model doesn’t give any technical requirements; nor there is any specific chapter dedicated to the preservation format of digital material; however, the structure and components of the Information Packages, as described in the model [5](p. 4-37), give some general advice upon what information a format should contain beside of the data themselves (but not how to contain it). This leads to self-containing formats (image, sound and metadata in one file), easier to maintain and migrate, as wrappers do (AVI, QuickTime, MXF, etc.).

[12] defines seven general (common to any kind of document) sustainability factors. It is not about formats that could last forever and be accessed in hundred years (which is impossible in the actual state of technology), but about formats which allow easy and lossless future migration (or emulation), independently from any quality factor. In addition to those factors, we added one (stability) from [2]. Factors relating to complex formats (documents with hyper-links, or databases) are not discussed here, because they have specific needs (preservation of the “look and feel”, for example).

Disclosure

Following the OAIS model, the conservation of digital information is only possible through the conservation of the data content (the bits) and of the documentation upon how to make those bits human readable: “Data interpreted using its Representation Information yields Information” [5] (p. 2-4). This Representation Information, a kind of technical metadata, has to be available, before being conserved (see below “self-documentation” factor).

In general, open standards offer better access to the specifications; however, proprietary formats may also have complete documentation fully available. What is important is the existence and preservation of this documentation.

Adoption

Widely adopted formats are economically viable for the manufacturers; so future developments are more probable, and the format will less likely become quickly obsolete. Furthermore, there is more probability to easily find support, migration or emulation tools, and a wide variety of tools to read the files (interoperability).

To assess the degree to which a format is adopted, one should see how it is implemented in digital world: PC tools, native support in web browsers, etc. If it has been reviewed and accepted by another institution as a preferred format for preservation, the format can be considered as “widely adopted”. For example, Library of Congress’ list of preferred formats is a good resource [12].

Transparency

Formats representing information in a simple, rather than a complex way, are easier to migrate. Emulation and rendering tools will be designed also more easily.

So a format is suitable for preservation if it uses direct forms of encoding, without encryption nor compression; in case where uncompressed video is not feasible, lossless compression should be preferred. And the two above factors (“disclosure” and “adoption”) limit the transparency inhibiting effects of compression.

Self documentation

This factor refers to the ability of a format to embed metadata. Here we don’t speak about detailed descriptive metadata, but only about metadata allowing the proper rendering of the essence, and the understanding of its context. The preservation of those metadata together with the data allows ensuring the authenticity of the data over time.

The OAIS reference model [5] recommends to create Information Packages, containing the essence (the data) themselves, with:

  • Representation Information (to allow the data to be rendered and used as information. See “disclosure” above).
  • Preservation Description Information, containing:
    • reference (to identify and describe the content),
    • context (e.g. to document the purpose of the content’s creation),
    • fixity (to permit checks on the integrity of the content data),
    • provenance (to document the chain of custody and any changes since document was originally created).

 

Embedding metadata (instead of storing them separately) allows easier management over time: the format contains the data and the way to display it without external mean, through, for example, recreating the software. So the format exists on its own.

But this is not incompatible with separate metadata stores, that would contain metadata extracted from the file itself, and enriched with other content-based metadata, in order to create retrieval tools (catalogs, indexes, etc.).

We should also notice that there are some existing OAIS compliant file formats online registries, allowing to share Representation Information (specifications), like the Representation Information Registry Repository [6], GDFR [10], or PRONOM [16]. But the quality of information is often “mediocre” [17].

External dependencies

If a format is dependent on a hardware, software or operating systems, maintaining it will also mean maintain the hardware, software or operating system which it is dependent on. In order to facilitate the migration from one technical environment to another, it is essential to be able to use and render the data independently from any external device. In other words, the format should be platform-independent, or widely interoperable.

Impact of patents

Patents make the development of the format slower, increase the cost of transcoding for migration purposes by constraining to use fee-based softwares, and make future costs higher and unpredictable if there are royalties based on the use of the format. E.g. MPEG-4 implies high costs to create encode/decode softwares.

Technical protection mechanisms

In a preservation environment, access, security and preservation copies must be possible. So one should avoid formats bundled to a specific medium (like DigiBeta), or any other that constraint to the use of a particular dispositive, or prevent the establishment of backup procedures and disaster recovery operations. DRM should also be avoided on preservation masters.

Stability

[2] adds another factor to those proposed by [12]. It is the stability factor, that allows less migration operations. A stable format is characterized by non frequent changes (or not backward compatible changes) in the specifications. For example, proprietary formats may evolve more rapidly, following the market needs and the laws of supply and demand.

Discussion

The number of digital video formats is limited; it is even more limited when speaking about preservation formats. Moreover, we will not discuss about codecs, because the situation is very clear: one shouldn’t choose lossy compression, and JPEG2000 is the only lossless – and fully documented – compression algorithm.

We can limit our analysis to 5 wrapper-formats, which are not only preferred by [12], but also by [1], [14] and [13]: MXF, AAF, Motion JPEG 2000, AVI and QuickTime.

The MPEG family is out of scope for preservation, for it doesn’t allow lossless compression of a whole video sequence, as well as formats like RealMedia. Some open-standards have the same functionalities than AVI or QuickTime, but are not so widely adopted (for now): e.g. Matroska, Nut. For this reason, they are also not discussed here. Finally, we won’t speak about DCDM or DPX, which are formats more intended to film materials and playback in theaters.

All of the selected formats are able to wrap uncompressed or losslessly compressed bitstreams, also fitting the quality requirements mentioned above. For lossless compression, we will speak only about JPEG 2000.

Here we will shortly introduce them, and present their main strengths and weaknesses regarding sustainability. For complete descriptions and links to specifications, see [12].

MXF – AAF

MXF (Material Exchange Format) is a subtype of AAF (Advanced Authoring Format). Both are open standards, cross-platform and vendor-independent. They are self-contained formats, without external dependencies. They are not compression formats, but can encapsulate any uncompressed or lossless compressed bitstreams (audio and video) with metadata.

While AAF is intended to production data interchange (useful to manage workflows), MXF is more adapted to finished products, and is preferred by archivists [14], often called the “digital equivalent to videotape”.

The quality may be excellent, given the fact that it can encapsulate uncompressed or compressed with JPEG 2000 bitstreams, and that 4:4:4 sampling is possible. So when encapsulating uncompressed or JPEG 2000 compressed data, MXF fits all the sustainability factors: open-standard, well documented, widely adopted in preservation institutions. But the main advantage of MXF over other wrappers is its ability to embed detailed preservation metadata.

For all those reasons, MXF is considered as a preferred preservation format by [12] and [14], fitting all the sustainability requirements.

Motion JPEG 2000

Motion JPEG 2000 is a part of the JPEG 2000 standard. It is a codec allowing to produce different derivatives, with different levels of compression, from uncompressed (4:4:4 sampling is possible), to lossless compressed, and to lossy compressed, and all this from one single file; JPEG 2000 is fully scalable, which is a big advantage. The Motion JPEG 2000 codec only uses intraframe compression, and not temporal compression.

We mention it in this section, because Motion JPEG 2000 can be considered as a wrapper, since he can contain sound (LPCM is possible) and a large extent of metadata. However, it is more usual to use JPEG 2000 compression scheme, wrapped in a MXF container [3].

Motion JPEG 2000 has a growing use, and is widely adopted. It is an open format, with publicly accessible specifications. It is also platform-independent (no specific hardware, software or operating system). Finally, it is self-documenting (descriptive and preservation metadata). For all those reason, Motion JPEG 2000 is a good archival format.

The main advantage of Motion JPEG 2000 is that it is the only existing lossless compression format. Moreover, the support for rich metadata is greater than in AVI or QuickTime.

AVI – QuickTime

AVI (Audio and Video Interleaved) and QuickTime (the .mov extension) are the most widely adopted video formats; though they are proprietary, they can be considered as de facto standards, and their specifications are publicly available. Both can contain uncompressed video streams, and the 4:4:4 sampling is supported by QuickTime.

The main advantage of those formats is their adoption, which implies a wide variety of encoding/decoding tools, which is not the case of MXF and Motion JPEG 2000. They are well suited for long term preservation.

The main disadvantages of AVI and QuickTime are the absence of a lossless compression codec, which can prevent important storage and transmission saves, and the absence of rich metadata support.

Conclusion

MXF seems to be the most suitable format for long term preservation. Following [14], MXF with JPEG 2000 encoding is the implementation that can best guarantee access to digital video over long term, for its capability to embed rich metadata.

However, given the fact that MXF is able to encapsulate any other format, and that metadata will be discussed in chapter on metadata, it is sufficient for the moment to say that:

  • If storage space is privileged, Motion JPEG 2000 with lossless compression should be chosen.
  • If not, AVI and QuickTime (uncompressed) are acceptable preservation file formats.

Bibliography

All resources accessed on December 16, 2008

 

[1] ABRAMS, Stephen. File Formats. In: DCC Digital Curation Manual [online]. October 2007. 53 p. http://www.dcc.ac.uk/resource/curation-manual/chapters/file-formats/file-formats.pdf

[2] BROWN, Adrian. Selecting File Formats for Long-Term Preservation [online]. [London]: The National Archives, 2003 (Digital Preservation Guidance Note 1). 8 p. http://www.nationalarchives.gov.uk/documents/selecting_file_formats.pdf

[2bis] BROWN, Adrian. Selecting File Formats for Long-Term Preservation [online]. [London]: The National Archives, 2008 (Digital Preservation Guidance Note 1, Issue 2). http://www.nationalarchives.gov.uk/documents/selecting-file-formats.pdf (note: issue 2, not used to write this document)

[3] BUCKLEY, Robert. JPEG 2000: A Practical Digital Preservation Standard? [online]. York: Digital Preservation Coalition, 2008 (DPC Technology Watch Series Report 08-01). 28 p.  http://www.dpconline.org/docs/reports/dpctw08-01.pdf

[4] CHRISTENSEN, Steen S. Archival Data Format Requirements [online]. Copenhagen: The Royal Library, 2004. 17 p. http://netarkivet.dk/publikationer/Archival_format_requirements-2004.pdf

[5] CONSULTATION COMMITTEE FOR SPACE DATA SYSTEMS (CCSDS). Reference Model for and Open Archival Information System (OAIS) [online]. Washington, DC: CCSDS, 2002. 148 p.  http://public.ccsds.org/publications/archive/650x0b1.pdf

[6] DCC. CASPAR. FREEBXML. Representation Information Registry Repository [online]. http://registry.dcc.ac.uk/omar/

[7] EPFL. Montreux Jazz Festival Digital Archive Project: A unique and first of a kind high resolution digital archive of the Montreux Jazz Festival. October 2008

[8] GILMOUR, Ian. DAVILA, Justin R. Lossless Video Compression for Archives: Motion JPEG2k and Other Options [online]. Media Matters, 2006. 8 p. http://www.media-matters.net/docs/WhitePapers/WPMJ2k.pdf

[9] GUERCIO, Maria. CAPPIELLO, Cinzia. File Formats Typology and Registries for Digital Preservation: Delos Report [online]. Universita’ degli Studi di Urbino, 2004. 54 p. http://www.dpc.delos.info/private/output/DELOS_WP6_d631_finalv2(5)_urbino.pdf

[10] HARVARD COLLEGE. GDFR: Global Digital Format Registry [online]. http://hul.harvard.edu/gdfr/about.html

[11] INTERNATIONAL COUNCIL ON ARCHIVES (ICA), COMMITTEE ON CURRENT RECORDS IN AN ELECTRONIC ENVIRONMENT. Electronic Records: A Workbook for Archivists [online]. ICA, 2005. 80 p (ICA Study, 16). http://www.ica.org/sites/default/files/Study16ENG_5_2.pdf

[12] LIBRARY OF CONGRESS. Sustainability of Digital Formats: Planning for Library of Congress Collections [online]. Last update 05/21/2007. http://www.digitalpreservation.gov/formats/index.shtml

[13] McDONOUGH, Jerome P. Preservation-Worthy Digital Video, or How to Drive Your Library into Chapter 11: Presented at the Electronic Media Group Annual Meeting of the American Institute for Conservation of Historic and Artistic Works, Portland, Oregon, June 13, 2004 [online]. 2004. 12 p. http://aic.stanford.edu/sg/emg/library/pdf/mcdonough/McDonough-EMG2004.pdf

[14] MEDIA MATTERS. Digital Video Preservation Reformatting Project: A Report [online]. Media Matters, 2004. 110 p. http://www.danceheritage.org/preservation/DigitalVideoPreservation1.pdf

[15] PRESTOSPACE. Preservation Guide : Roadmap [online]. Last modified on June 30 2008. http://wiki.prestospace.org/pmwiki.php?n=Main.Roadmap

[16] THE NATIONAL ARCHIVES (UK). PRONOM [online]. http://www.nationalarchives.gov.uk/PRONOM/default.htm

[17] UNIVERSITY OF LEEDS. Representation and Rendering Project [online]. 2003. http://www.leeds.ac.uk/reprend/

[18] UNIVERSITY OF MICHIGAN. UNIVERSITY OF LEEDS. CAMiLEON Project [online]. 2003. http://www.si.umich.edu/CAMILEON/