Process multimedia with the JMF API

REF: http://www.builderau.com.au/program/java/soa/Process-multimedia-with-the-Java-Media-Framework-API/0,339024620,339288502,00.htm

The Java Media Framework (JMF) API allows developers to process media in many different ways. It deals with real-time multimedia presentations and effects processing.

JMF handles time-based media (ie, media that changes with respect to time). Examples of this are: video from a television source and audio from a raw-audio format file and animations.

Data processing

JMF abstracts the media it works with into DataSource objects (for media being read into JMF) and DataSink objects (for data being exported out). Developers are not given significant access to the particulars of any given format. Media is represented as sources (which are obtained from URLs) that can be read in and played, processed, and exported (though not all codecs support processing and transcoding). A Manager class offers static methods that are the primary point-of-contact with JMF for applications.

There are three stages of data processing in JMF architecture: input, processing, and output.

|> Input includes working with capture devices, reading data from files on a hard drive, and all sorts of network data input. During the input stage, data is read from a source and passed in buffers to the processing stage. The input stage may consist of reading data from a local capture device (such as a webcam or TV capture card), a file on a disk, or stream from the network.
|> Processing consists of a number of codecs and effects designed to modify the data stream to one suitable for output. These codecs may perform functions such as compressing or decompressing the audio to a different format, adding a watermark, cleaning up noise, or applying an effect to the stream (such as echo to the audio). Once the processing stage applies its transformations to the stream, it passes the information to the output stage.
|> Output includes the use of a video renderer, saving output to disk, and saving output to the network. The output stage may take the stream and pass it to a file on disk, output it to the local video display, or transmit it over the network. For example, a JMF system may read input from a TV capture card from the local system that is capturing input from a VCR in the input stage. It may then pass it to the processing stage to add a watermark in the corner of each frame and then broadcast it over the local intranet in the output stage.

Component architecture

JMF is built around a component architecture. The components are organised into these main categories: media handlers, data sources, codecs and effects, renderers, and mux/demuxes.

|> Media handlers are registered for each type of file that JMF must be able to handle. To support new file formats, you can create a new MediaHandler.
|> A data source handler manages source streams from various inputs. These can be for network protocols, such as http or ftp, or for simple input from disks.
|> Codecs and effects are components that take an input stream, apply a transformation to it, and output it. Codecs may have different input and output formats, while effects are simple transformations of a single input format to an output stream of the same format.
|> A renderer is similar to a codec, but the final output is somewhere other than another stream. A VideoRenderer outputs the final data to the screen, but another kind of renderer could output to different hardware, such as a TV out card.
|> Mux/demuxes (which stands for Multiplexers and Demultiplexers) are used to combine multiple streams into a single stream or vice versa, respectively. They are useful for creating and reading a package of audio and video for saving to disk as a single file or transmitting over a network.

Presenting data

JMF provides a number of pre-built classes that handle the reading, processing, and displaying of data. Using the Player class, you can easily incorporate media into any graphical application (AWT or Swing). The Processor class allows you to control the encoding or decoding process at a more granular level than the Player class, such as adding a custom codec or effect between the input and output stages.

The Player class automatically handles the set-up of the file handler, video and audio decoders, and media renderers. It is possible to embed the Player in a Swing application, but you must be careful because it is a heavyweight component. Below is an example of how to use the Player with AWT:

import java.applet.*;
import java.awt.*;
import java.net.*;
import javax.media.*;

public class PlayerApplet extends Applet {
Player player = null;
public void init() {
setLayout( new BorderLayout() );
String mediaFile = getParameter( "FILE" );
try {
URL mediaURL = new URL( getDocumentBase(), mediaFile );
player = Manager.createRealizedPlayer( mediaURL );
if (player.getVisualComponent() != null)
add("Center", player.getVisualComponent());
if (player.getControlPanelComponent() != null)
add("South", player.getControlPanelComponent());
}
catch (Exception e) {
System.err.println( "Got exception " + e );
}
}
public void start() {
player.start();
}

public void stop() {
player.stop();
player.deallocate();
}
public void destroy() {
player.close();
}
}In this case, I use the static createRealizedPlayer() function of the Manager class to create the Player object. This ensures that the visual and control panel components are created before it gets added to the window. It is also possible to create an unrealised player and implement the ControllerEventHandler interface. The window then waits for the controllerUpdate event to fire and adds the components to the layout as they are realised:

public synchronized void controllerUpdate( ControllerEvent event ) {
         if ( event instanceof RealizeCompleteEvent ) {
                 Component comp;
                 if ( (comp = player.getVisualComponent()) != null )
                          add ( "Center", comp );
                 if ( (comp = player.getControlPanelComponent()) != null )
                          add ( "South", comp );
                 validate();
         }
}A more modern Swing-based application can use the Player class in a similar way. The following code creates a Swing-based TV capture program with the video output displayed in the entire window:

import javax.media.*;
import javax.swing.*;
import java.awt.*;
import java.net.*;
import java.awt.event.*;
import javax.swing.event.*;

public class JMFTest extends JFrame {
    Player _player;
    JMFTest() {
        addWindowListener( new WindowAdapter() {
            public void windowClosing( WindowEvent e ) {
                _player.stop();
                _player.deallocate();
                _player.close();
                System.exit( 0 );
            }
        });
          setExtent( 0, 0, 320, 260 );
        JPanel panel = (JPanel)getContentPane();
        panel.setLayout( new BorderLayout() );
        String mediaFile = "vfw://1";
        try {
            MediaLocator mlr = new MediaLocator( mediaFile );
            _player = Manager.createRealizedPlayer( mlr );
            if (_player.getVisualComponent() != null)
            panel.add("Center", _player.getVisualComponent());
            if (_player.getControlPanelComponent() != null)
            panel.add("South", _player.getControlPanelComponent());
        }
        catch (Exception e) {
            System.err.println( "Got exception " + e );
        }
    }

    public static void main(String[] args) {
        JMFTest jmfTest = new JMFTest();
        jmfTest.show();
    }
}Real-time capturing

You can capture video and audio data in real-time from input sources and stream it to files on the local filesystem. To capture audio, you must specify the sampling frequency, the sample size, and the number of channels. JMF will attempt to locate any devices that will support this format and return a list of any that match:

CaptureDeviceInfo di = null;
Vector deviceList = CaptureDeviceManager.getDeviceList(
new AudioFormat( "linear", 44100, 16, 2 ) );
if ( deviceList.size() > 0 )
        di = (CaptureDeviceInfo)deviceList.firstElement();
Processor p = Manager.createRealizedProcessor(di.getLocator());
DataSource source = p.getDataOutput();The source object returned from the Processor class can be turned into a Player object by calling Manager.createPlayer(). To capture it to an audio file, a DataSink can take the data:

DataSink sink;
MediaLocator dest = new MediaLocator("file://output.wav");
try {
        sink = Manager.createDataSink(source, dest);
        sink.open();
        sink.start();
} catch (Exception e) { }Capturing video is identical to capturing audio. Most video sources have an accompanying audio track that must be encoded as well, so I must create a compound destination file:

Format formats[] = new Format[2];
formats[0] = new AudioFormat( "linear", 44100, 16, 2 );
formats[1] = new VideoFormat( "cvid "); // Cinepak video compressor
Processor p;
try {
    p = Manager.createRealizedProcessor( new ProcessorModel( formats, null ) );
} catch ( Exception e ) { }

DataSource source = p.getDataOutput();
MediaLocator dest = new MediaLocator( "file://output.mov" );
DataSink filewriter = null;
try {
    filewriter = Manager.createDataSink( source, dest );
    filewriter.open();
    filewriter.start();
} catch ( Exception e ) { }
p.start();This source will create a QuickTime-format file called output.mov with an audio track encoded raw and a video track encoded with the Cinepak compressor.

Criticism

Many JMF developers complain that the API only supports a few codecs and formats in modern use. For example, its all-Java version cannot play MPEG-2, MPEG-4, Windows Media, RealMedia, most QuickTime movies, Flash content newer than Flash 2, and needs a plug-in to play the MP3 format. While the performance packs offer the ability to use the native platform's media library, they're only offered for Windows and Solaris. Windows-based JMF developers can unwittingly think JMF provides support for more formats than it does, and be surprised when their application is unable to play those formats on other platforms. These are the primary reasons for criticising JMF by some developers.

Conclusion

JMF scales across different media data types, protocols, and delivery mechanisms. JMF provides a plug-in architecture that allows JMF to be customised and extended. Technology providers can extend JMF to support additional media formats. High performance custom implementation of media players (or codecs possibly using hardware accelerators) can be defined and integrated with JMF.

Additional resources about JMF

|> JMF 2.1.1 set-up instructions
|> Full JMF 2.x guide
|> List of supported formats in JMF 2.1.1(the current version)
|> JMF code samples from Sun
|> MP3 plug-in for JMF