spam: the other white meat

Media Mover can be pretty CPU intensive- transcoding with FFmpeg can be a substantial hit on your server resources. If you couple this with a large amount of batch processing, you are looking at high loads over a long period of time. And of course, this does not even begin to look at what is going on for your users on the site.

Drupal employs the Throttle module to handle high load situations. It looks at traffic patterns on the site to determine if a throttle condition should be employed. In considering how to limit the activity of Media Mover on a server this is an obvious consideration. One of the things that makes Media Mover (for most use cases) different than the needs of modules effected by Throttle is that it is not bound by the traffic occuring on the site- it may well be a factor, but it is not the only consideration. If a Media Mover configuration is using FFmpeg to transcode files, it is almost exclusively CPU bound. Of course there is the Apache process that kicks off cron, some degree of Drupal processing, and perhaps a MySQL hit or two, but largely FFmpeg wants to make use of the server CPU. Media Mover will cycle through sets of files that it is loading from MySQL, but again, in terms of its total impact, mostly we are looking at FFmpeg hitting the CPU.

In this case, if we want to limit Media Mover from running when a server is starting to get overloaded on its CPU we need to check the CPU load. I’ve written a fairly basic utility that does this on Linux / Unix based servers on Media Mover’s cron processing script. It allows other scripts to still access the API without being throttled, but Cron is stopped if load creeps beyond what you set it for.

You can set the threshold where Media Mover should stop, and select which average you want to use as the threshold- the one, five or 15 minute load averages. Once the load drops below the threshold, the Media Mover Cron runs will start up again.

Asset is the obvious choice if you need your users to be able to easily upload content and place it within WYWISYG editor. Media Mover is probably on the other side of the spectrum- tones of options, lots of arcane settings that will not make sense if you are not familiar with the ins and outs of rich media- further if you just want to get your transcoded media on the page.

The problem is that Media Mover is more of an API than it is an interface. In fact, you’d be hard pressed to really call it an interface if you’re an end user. A while back, I released the Auto Run module for Media Mover which runs Media Mover configurations when a node is created or updated. This really set the groundwork for starting to move toward running Media Mover configurations on specific user interactions- the ones that I think a majority of people are looking for. This approach- the transcode on create/update content- while workable, still does not give the end user control of how page layout works. For some, this is a fine middle ground approach, for others, unworkable. From my perspective, Asset provides a solid way for users to add content to their nodes. Of course WYSISYG editors are problematic, but they are not going anywhere, and hopefully, their HTML will just get better. Asset goes along way toward improving the integration between WYSIWYG editors and Drupal. Media Mover can remain a work horse for rich media, but it can also play nice when it comes to offering up its services.

From what I’ve heard from other Drupal admins and people who use the web in general, they do not want to be bothered by file formats, complex preferences, or sorting out what formats are playable and what you need for the non-playable ones. Asset, Media Mover, and FLVMediaPlayer combined offer a step forward in addressing this- users can upload “video” and get “video” on the web, without a tremendous amount of knowledge of what is actually going on. This gives the admin the opportunity to fine tune what options are available, while users get to have the fun of having a full transcoding system at their finger tips. Hence a union between Asset and Media Mover seems a perfect thing.

I did a little screen cast to show how this works. It shows the process of enabling the modules, making sure that there are profiles in the FLVMediaPlayer, setting up Asset, and the process of actually uploading and inserting with Asset. If you are not familiar with these modules, should still make some sense, though it will probably be more meaningful if you are familiar with Asset.

I’ve just added the new Auto Run module to Media Mover’s 1.0alpha3 release. This module is a straight forward attempt to bridge the gap between people who need the complex functionality that Media Mover offers and people who need their data to be converted on demand. This module will run specified Media Mover configurations on node inserts and updates- meaning that when a user submits content or updates content, the specific Media Mover configurations will be run. In addition, this functionality leverages some earlier updates that let Media Mover run only on the specific node in question. This helps keep processing overhead down.

This screen shot shows the listing of the Media Mover configurations that are available to run. Note that the administrator should choose configurations that are harvesting from nodes- otherwise it is just wasted cycles.

Using a similar system that I built for Media Mover, I implemented an import / export system in the FLV Media Player module which allows an administrator to move configurations from one site to another. This is great if you’re working on your development site and want to port the new configuration to your live site.

Once the 1.0 release is stabilized for both, I’ll write some functions that allow modules to define FLV Media Player profiles and Media Mover configurations in custom modules so that these can be kept under version control, making it easier to deploy changes via version control.

Thanks to heyrocker for the photo.

I just realized that my talk on Media Mover at the Seattle Drupal Users’ Group from this past June was posted on Blip. I guess I’m a bit behind the times! This is similar to the talk that I did in NYC with less command line and more talking. Interestingly, some of the items that are “future plans” have actually been completed already and are in the alpha releases of Media Mover 1.

The Seattle group is really impressive- super well organized, really well attended weekend with great talks and great people. Both Robin and Gregory really knocked this one out of the park!

The power and flexibility of FFmpeg is a strength and a weakness. While it provides a huge number of options, it’s easy to create a configuration that actually doesn’t work. Case in point has been my experience with Media Mover- the most common thing that people want to use it for is creating FLV files from a variety of source material. Unfortunately, setting the codec or bit rate right can be somewhat challenging, to the point where people create configurations that will not convert the files they have.

I’ve setup FFmpeg wrapper to use configuration files. For each output format, a format file can be create which defines what kinds of options are available (codecs, bit rates, sample rates, etc). These files are loaded via ajax and any form that conforms to a set of rules can be updated on the fly as the user selects different options.

While this currently only works for Media Mover’s FFmpeg processing module, it would work for any module that wanted to implement the setup. Further, people who have experience with specific kinds of output formats can contribute back configuration files that help improve the output from FFmpeg.

The screencast below shows this process in action- I’m adding a new configuration to Media Mover and then selecting different output options for FFmpeg. While I only have one configuration complete at this time (FLV) switching between other options grabs the default (ie: all) options. It also turns the advanced configurations off when you select an option that uses the default- this way FFmpeg will take it’s best guess at converting unless you specifically tell it to.

On top of this, I’ve added support in mm_ffmpeg to enter in your own command- while this is potentially dangerous, it gives people with advanced needs of FFmpeg the ability to run what they need.

I put together a module for Media Mover which allows some basic support for harvesting files from the local file system. It uses the same form that other Media Modules do- provides an absolute path to a directory and has a file extension filter. It currently does not work recursively, but it should be enough to provide the initial functionality.

Here you can see the options that the mm_dir module gives on a configuration screen:

The code is in CVS and will be released with the next Media Mover release.

See the discussion here: http://drupal.org/node/258956#comment-900776
UPDATE: now also supports storing to the filesystem as well.