Xuggler 3.0: "Ben"

Announcing Xuggler 3.0: Get It Here.

Table of Contents

Summary

Since 2.0 was a major release, the original plan was to have 2.1 be a "only bug fixes" release.

Screw that!

Here comes Xuggler 3.0! Here's some of the new features you can try out:

  • A simpler API for decoding and encoding: MediaTool. See the new tutorials, but we think it rocks.

  • New MemoryModel technology that can speed up Xuggler programs by over 40%.

  • Improved scaling for multi-threaded programs, including the ability to interrupt blocking calls.

  • Seamless Java IO support (e.g. InputStream, and OutputStream).

  • A simpler API for getting access to video, audio and packet memory.

  • The ability to query which formats and codecs Xuggler supports.

  • ... and more!

As usual, read on for the details.

Release Notes

MediaTool API

We've consistently heard from people that decoding and encoding files is too hard. So we've taken the MediaReader object from Xuggler 2.0 and expanded it into a much much simpler API for decoding, encoding and viewing audio.

How simple? This opens a flash media file and encodes it to Quicktime.

// open a reader
IMediaReader reader = MediaTool.makeReader("inputfile.flv");

// write a new file as we read it
reader.addListener(MediaTool.makeWriter("outputfile.mov", reader));

// and GO!
while (reader.readPacket() == null)
  /* watch my video */ ;

We provide some Youtube videos showing how to use the API. Here's how to decode and encode with the MediaWriter tool:

And here's how to use the MediaWriter to modify or encode media from scratch:

For more, read the tutorials, or see the com.xuggle.mediatool documentation.

Memory Models: Tunable Memory-Management

Xuggler 2.0 had a standard memory model that made all Xuggler objects work like Java objects, even though they are really low-level native objects. This meant users could use Xuggler without having to worry about when native memory was released. But this safety comes at a slight performance cost, and some people wanted the ability to run a little faster.

With Xuggler 3.0, we still support our original model (it's still the default), but we also let those Java Ninjas among you use one of FOUR new additional memory models, which can make your Xuggler code up to 40% faster!

These new memory models come at a development cost -- Under those models Xuggler is much less forgiving if you forget to release a resource or of if you overallocate objects. But if you're writing in the core Xuggler API and want to try different approaches to memory management, please go ahead.

Or, use MediaTool, which has been designed to work smoothly under any of the memory models.

To experiment with the new memory model support, see the com.xuggle.ferry.JNIMemoryManager Object

Multi-Threading Support

Xuggler 2.0 worked well in client apps and single threaded applications, but could block in uninterruptable ways when reading from network sockets, and had some issues under heavy multi-threaded scenarios (i.e. multiple threads decoding or encoding).

With 3.0 we've changed a lot of the underpinning code to address these issues, done major testing and profiling to validate those changes, and more than doubled the number of Xuggler tests to 570(!!) so hammer the living-day-lights out of it everytime we do a build.

The end result is:

  • Blocking Xuggler calls will interrupt and return quickly if you interrupt the thread they are running on.

  • Several Java (yes, we confirmed with Sun) bugs have been worked around.

  • We ensure FFMPEG locks for the minimum amount of time using a new feature of FFMPEG. Major thanks to Andreas Oman for adding it.

  • We've confirmed in testing on a 4GB, Intel Core Duo 2.1 KHz machine that we can run over 100 simulaneous decoders, or 50 simultaneous encoders using the Sun's Server JVM on Linux.

See the com.xuggle.xuggler.IContainer documentation for which methods are potentially blocking (and therefore interruptable). Only IContainer calls may potentially block -- all others Xuggler calls will return in deterministic time.

Java I/O

This is an oft-requested feature from the community; you can now open IContainer objects with standard Java InputStreams OutputStreams, DataInput, DataOutput, RandomAccessFile, ReadableByteChannel or WritableByteChannel objects.

That means you can decode and encode directly to thousands of different java libraries. And then some.

Here's an example:

IContainer container = IContainer.make();
RandomAccessFile file = new RandomAccessFile("myfile.flv", "r");
container.open(file, Type.READ, null);

To get started, see the com.xuggle.xuggler.IContainer#open() methods documentation.

We've also made it much easier to plug in your IO handlers for Xuggler in the event the ones above don't work for you. Just implement your own com.xuggle.xuggler.io.IURLProtocolHandler and pass it directly to a com.xuggle.xuggler.IContainer 

IURLProtocolHandler handler = new CustomURLProtocolHandler();
container.open(handler, Type.READ, null);

Easier Access To Memory Contents

In Xuggler 3.0 we've introduced some new methods on the com.xuggle.ferry.IBuffer and com.xuggle.xuggler.IMediaData interfaces. You can now call get and put to easily copy data into and out of buffers, without getting your hards dirty with Direct NIO ByteBuffers.

Here's an example of putting some audio samples into a set of audio samples: 

  IAudioSamples samples = ...;
  short[] mySamples = new short[]{ 0x387e, 0x002C, 0x1814, 0x327F };
  samples.put(mySamples, 0, 0, mySamples.length);
And here's getting the payload out of a packet and modifying it: 
  IPacket packet = ...;
  byte[] bufferToFill = new byte[packet.getSize()];
  packet.get(0, bufferToFill, 0, packet.getSize());
  // change the first byte
  bufferToFill[0] = 0x37;
  // and put it back
  packet.put(bufferToFill, 0, 0, packet.getSize());

What's even better is using these methods can be more efficient that getting the underlying IBuffer and ByteBuffer objects yourself and copying to and from Java arrays.

Dynamic Querying of IContainerFormat

Turns out Adobe's FLV container cannot hold FLAC audio. That's just the way it is. In fact, every container format has some codecs they support and some codecs they can't.

But in prior versions of Xuggler you just had to know this yourself, and wouldn't find out there was a problem until you tried to write a container header.

In Xuggler 3.0 you can now query a com.xuggle.xuggler.IContainerFormat object to find out what codecs can be decoded from or encoded to it, and even which codecs it prefers!

Here's an example of finding out all the codecs that FLV supports:

IContainerFormat format = IContainerFormat.make();
format.setOutputFormat("flv", null, null);
for(ICodec.ID id : format.getOutputCodecsSupported())
  System.out.println("Codec supported: " + id);

To get started, see:

Dynamic Querying of Installed Codecs

You can now query Xuggler for the full set of Codecs installed on the system, and then indivdually query each one to find out what capabilities it can support. This enables you to write programs that present drop-down boxes of Codecs for a user to select.

We've also added new ICodec methods that will tell you what pixel formats are supported and which frame rates are supported, among others.

Here's some sample code showing how to get a list of all codecs on the system, and then prints out any pixel formats they claim to support: 

for(ICodec codec : ICodec.getInstalledCodecs())
{
  if (codec.getType() == ICodec.Type.CODEC_TYPE_VIDEO) {
    System.out.print("Codec " + codec.getLongName() + "supports: ");
  
    if (codec.getNumSupportedVideoFrameRates() > 0)
      for(IPixelFormat.Type type : codec.getSupportedVideoPixelFormats())
        System.out.print(type + "; ");
    else
      System.out.print("unspecified;");
    System.out.print("\n");
  }
}

To get started, see the com.xuggle.xuggler.ICodec documentation.

Improved Options & Settings Viewing

We've introduced a new object called com.xuggle.xuggler.Configuration that can be used to find out what codecs, containers, and options are supported by Xuggler, and what the default option settings are. To see the output run this on Windows: 
java -cp %XUGGLE_HOME%\share\java\jars\xuggle-xuggler.jar com.xuggle.xuggler.Configuration

or this on Linux or Mac: 

java -cp $XUGGLE_HOME/share/java/jars/xuggle-xuggler.jar com.xuggle.xuggler.Configuration

toString() support

Ever wondered what's inside that opaque Xuggler object you're passing around? Wonder no longer. Just call toString() to get a summary of what we think we're passing around. Or if you're using Eclipse to debug, just click on the variable in the memory viewing window, and you'll see it there.

Here's some example code of this in action:

IContainer container = IContainer.make();
container.open("testfile.flv", IContainer.Type.READ, null);
System.out.println(container);

And here's what gets printed as a result (we're wrapping for this web-page, but the actual output is one line):

com.xuggle.xuggler.IContainer@2093804144[url:testfile.flv;type:READ;
  format:com.xuggle.xuggler.IContainerFormat@137989432[iname:flv;
  ilongname:FLV format;];]

JavaDoc Diagrams

Our Javadoc now includes some diagrams of inheritance for those of you more pictorially inclined. Thanks to UMLGraph for that.

For a nice example, see the com.xuggle.mediatool package documentation:

Exhaustive Test Infrastructure

Our build system now separates simple unit tests from long-running Exhaustive Tests, and we've changed several of our long-running tests to be Exhaustive Tests. Our standard install steps on Linux and Mac end up faster as a result.

If you want to see how your system performs with the full load test, you can do: 

ant run-tests-all

Warning: these command wil test how Xuggler performs under hard failure situations (e.g. no more OS virtual memory), and as a result will likely render your machine somewhat unusable and swapping a lot while they run. This is especially true on 64-bit machines.

If you want to add new tests, just make sure your test class name ends with "ExhaustiveTest.java". For example: com.xuggle.mediatools.MediaToolExhaustiveTest.

Build System now under MIT License

Our build system, in all it's craziness, has been released under the MIT license. It'll build javadoc with diagrams. It'll build native code if you have it. It'll run tests and collate results. It'll be incredibly lacking in documentation. But if you're interested, it's the contents of the mk/buildtools directory and we use it on over 10 different projects.

To get started, see the mk/buildtools/buildhelper.xml file in your Xuggler source code tree.

Warnings

You must un-install prior versions of Xuggler before installing this version.

This release of Xuggler is backwards compatible with the last release with the following exceptions:

  • com.xuggle.xuggler.MediaReader has been removed and replaced with com.xuggle.mediatool.IMediaReader.

  • In prior versions of Xuggler if you opened a com.xuggle.xuggler.IContainer object for writing, wrote a header, and then forgot to either write a trailer or close the container, we automatically wrote the trailer when we collected the object. We no longer do this as writeTrailer() can potentially block, and the object can potentially be collected on the Finalizer thread, leading to a Java crash. Work arounds: either call IContainer.close() or IContainer.writeTrailer() yourself.

  • com.xuggle.ferry.JNIWeakReference, an internal only object, has been replaced with com.xuggle.ferry.JNIReference which is now a public object.

Known Issues

Critical

There are no known critical issues in this release.

Major

Users of this version of Xuggler should be aware of the following limitations (which will be addressed in a future Xuggler release):

  • MediaWriter does not resample audio. To work around ensure the audio sample rate and channels you ask MediaWriter to encode is the same as what you set up your audio streams with. Tracking issue.
  • MediaViewer is experimental and not very robust. It's included only to get feedback, but you should not use it in production environments. Tracking issue.

Minor

There are two bugs in the current version of FFMPEG that may impact you. Both have simple workarounds:

  • Issue #92: Doing resize and changing color space from YUV to either BGR24 or ARGB AT THE SAME TIME produces erroneous black pixels. Workaround: do resize operations separately from colorspace conversions.
  • Issue #93: YUV to ARGB Converter causes every 2nd line of pixels to be offset by one. Work around: use BGR24 converter.

Issues Fixed

  • Defect #151: MediaViewer threads should be Daemon threads
  • Defect #148: MediaWriter unable to encode to mpeg1 or mpeg2 video
  • Defect #144: DecodeAndPlayAudioAndVideo demo is busted
  • Defect #140: MediaViewer should not start threads if it doesn't need to
  • Defect #139: MediaViewer close race condition
  • Defect #138: EstablishOutputCodec implementation needs tests and fixes
  • Defect #136: Encoding audio with no timestamps fails
  • Defect #132: MediaWriterTest fails on CreateAudio if MediaViewer is enabled
  • Defect #130: Writing trailers while closing a dangling media container on BLOCKABLE media could cause JVM deadlock
  • Defect #128: DecodeAndPlayVideo doesn't just play video and doesn't exit
  • Defect #122: OutOfMemoryErrors on Windows can cause the Java virtual machine to exit
  • Defect #119: IStreamCoder.encodeVideo will cause memory corruption if passed in frame has wrong dimensions
  • Defect #118: Xuggler sometimes hits deadlock when running MultiThreadedReadingExhaustiveTest
  • Defect #117: OutOfHeapMemory error could core dump JVM
  • Defect #116: Xuggler can complain about insufficient thread locking
  • Defect #115: MediaWriter core-dumps when transcoding MKV test file to MPG
  • Defect #111: [ERROR] [Transcoder[RC3]] org.ffmpeg - [flv @ 0x4c7e760] Error, Invalid timestam p=368, last=368
  • Defect #107: Adding streams to an output container after writing a header can cause ffmpeg to crash
  • Defect #105: cannot open MPEG2VIDEO codec : YUV420 versus YUV420P.
  • Defect #104: Xuggler incorrectly calculates audio timestamps for audio streams with a non-zero start duration
  • Defect #102: Xuggler doesn't set packet timebases when demuxing
  • Defect #101: Red5 Tip of Tree (r3590) and Xuggler 2.0 demos appear broken together
  • Defect #100: The red5 scope in the sample app is application-level only, should be changed to match the incoming connection
  • Defect #99: 64-bit Xuggler Container Test failing
  • Defect #97: Xuggler crashes if you attempt call IContainer.writeHeader() with no streams set up.
  • Defect #96: Ffmpeg has deprecated av_decode_audio2 and av_decode_video
  • Defect #95: libxuggle-ferry fails to build in xuggler-2.0 on Ubuntu 8.10 locale es_CL
  • Defect #94: ARGB Tests started failing with latest FFMPEG release
  • Defect #37: Fix ivy to install last version
  • Defect #19: javax.imageio.ImageIO.write() produces dark JPEG files.

Enhancements Added

What's Next?

Who knows? It'll depend on what the community reaction to Xuggler 3.0 is, but if we were guessing, here's top of our list (things we wanted to get in Xuggler 3.0 but didn't make the deadline):

  • MediaWriter should resample audio for users behind the scenes if needed, and should take javax.sound format specifiers.
  • The video conversion API should be simpler to use without sacrificing performance for those who need it.
  • MediaViewer should be more stable -- FYI we're not planning on building a video player (we're leaving that for the community right now), but having a MediaViewer helps with debugging other things.
  • There are a variety of community-requested fixes and features that we didn't get to for 3.0 that we're still working on.

Warning: At some point the FFMPEG is going to move to GIT for source control. If you're in the habit of building Xuggler from our tip of tree, you'll want to install GIT. We'll likely be forced to require GIT and SVN to build Xuggler once FFMPEG makes the move. No ETA on this yet, but there is a lot of chatter on the FFMPEG lists about it. By the way, the advantage of this is we should also then be able to auto-integrate the latest libx264 as well.

Warning: We will likely have to move our SVN repository from Googlecode to some other site (still TBD). We'll try to keep disruption to a minimum if we need to do that.

As for when is the next version? Frankly, we don't know. Stay tuned.

Build Information

Tests

  • Java Tests: 570 (+316)
  • Native Tests: +Gobs

Fine Print

By reading these release notes, you authorize us to go through your thrash and look for discarded Pez dispensers. Really we need them.

Source Control Versions

Xuggler 3.0.660.FINAL

Hudson Good Build http://build.xuggle.com/job/xuggler_jdk5_stable/34/

Dynamic Revisions
* svn://svn.mplayerhq.hu/ffmpeg/trunk : 19118
* svn://svn.ffmpeg.org/mplayer/trunk/libswscale : 29350
* http://theyard.googlecode.com/svn/theyard/code/trunk/share/gnu/cxxtest/cxxtest-3.10.1 : 170
* http://xuggle.googlecode.com/svn/trunk/build/java : 657
* http://xuggle.googlecode.com/svn/trunk/java/xuggle-xuggler : 660

- Static Dependencies
* libx264:
 ftp://ftp.videolan.org/pub/videolan/x264/snapshots/x264-snapshot-20090125-2245.tar.bz2
* libmp3lame: http://downloads.sourceforge.net/lame/lame-398.tar.gz: 3.98
* libspeex: http://downloads.xiph.org/releases/speex/speex-1.2rc1.tar.gz: 1.2 RC1
* libfaac: http://downloads.sourceforge.net/faac/faac-1.28.zip: 1.28

On Windows, here's what we built our installer with:
    * Mingw gcc/g++ 4.2.4: http://www.tdragon.net/recentgcc/
    * Msys 1.0.11
    * mingw-runtime 3.15.1
    * w32api 3.13
    * gmake 3.81
    * binutils 2.91
    * yasm 0.7.2.2153
    * Msys Perl 5.6
    * Ant 1.7
    * Java JDK 1.5
    * MSFT C++ Visual Studio 2005
    * NullSoft NSIS Installer  2.42
    * SWIG 1.3.36 modified with doc gen capabilities
    * Subversion 1.5