spam: the other white meat

When Media Mover was first under development, it was started on an already obsolete version of Drupal. Now with RC1 due any day now (next week, I swear) I’ve realized that I can’t wait any longer to start moving toward Drupal Six. So I’ve started the porting process and of course it is not an easy road.

I used deadwood to generate the menus system for me- mostly worked, but I’ve still had to do some cleanup, probably because my own menus weren’t that well designed. I tried having deadwood convert the full module, but kept running into errors. I think it has a greater difficulty with complex modules as would be expected. At anyrate, there is a functional patch for Drupal-5 to upgrade to six here: http://drupal.org/node/269126

Media Mover can be pretty CPU intensive- transcoding with FFmpeg can be a substantial hit on your server resources. If you couple this with a large amount of batch processing, you are looking at high loads over a long period of time. And of course, this does not even begin to look at what is going on for your users on the site.

Drupal employs the Throttle module to handle high load situations. It looks at traffic patterns on the site to determine if a throttle condition should be employed. In considering how to limit the activity of Media Mover on a server this is an obvious consideration. One of the things that makes Media Mover (for most use cases) different than the needs of modules effected by Throttle is that it is not bound by the traffic occuring on the site- it may well be a factor, but it is not the only consideration. If a Media Mover configuration is using FFmpeg to transcode files, it is almost exclusively CPU bound. Of course there is the Apache process that kicks off cron, some degree of Drupal processing, and perhaps a MySQL hit or two, but largely FFmpeg wants to make use of the server CPU. Media Mover will cycle through sets of files that it is loading from MySQL, but again, in terms of its total impact, mostly we are looking at FFmpeg hitting the CPU.

In this case, if we want to limit Media Mover from running when a server is starting to get overloaded on its CPU we need to check the CPU load. I’ve written a fairly basic utility that does this on Linux / Unix based servers on Media Mover’s cron processing script. It allows other scripts to still access the API without being throttled, but Cron is stopped if load creeps beyond what you set it for.

You can set the threshold where Media Mover should stop, and select which average you want to use as the threshold- the one, five or 15 minute load averages. Once the load drops below the threshold, the Media Mover Cron runs will start up again.

Asset is the obvious choice if you need your users to be able to easily upload content and place it within WYWISYG editor. Media Mover is probably on the other side of the spectrum- tones of options, lots of arcane settings that will not make sense if you are not familiar with the ins and outs of rich media- further if you just want to get your transcoded media on the page.

The problem is that Media Mover is more of an API than it is an interface. In fact, you’d be hard pressed to really call it an interface if you’re an end user. A while back, I released the Auto Run module for Media Mover which runs Media Mover configurations when a node is created or updated. This really set the groundwork for starting to move toward running Media Mover configurations on specific user interactions- the ones that I think a majority of people are looking for. This approach- the transcode on create/update content- while workable, still does not give the end user control of how page layout works. For some, this is a fine middle ground approach, for others, unworkable. From my perspective, Asset provides a solid way for users to add content to their nodes. Of course WYSISYG editors are problematic, but they are not going anywhere, and hopefully, their HTML will just get better. Asset goes along way toward improving the integration between WYSIWYG editors and Drupal. Media Mover can remain a work horse for rich media, but it can also play nice when it comes to offering up its services.

From what I’ve heard from other Drupal admins and people who use the web in general, they do not want to be bothered by file formats, complex preferences, or sorting out what formats are playable and what you need for the non-playable ones. Asset, Media Mover, and FLVMediaPlayer combined offer a step forward in addressing this- users can upload “video” and get “video” on the web, without a tremendous amount of knowledge of what is actually going on. This gives the admin the opportunity to fine tune what options are available, while users get to have the fun of having a full transcoding system at their finger tips. Hence a union between Asset and Media Mover seems a perfect thing.

I did a little screen cast to show how this works. It shows the process of enabling the modules, making sure that there are profiles in the FLVMediaPlayer, setting up Asset, and the process of actually uploading and inserting with Asset. If you are not familiar with these modules, should still make some sense, though it will probably be more meaningful if you are familiar with Asset.

I’ve just added the new Auto Run module to Media Mover’s 1.0alpha3 release. This module is a straight forward attempt to bridge the gap between people who need the complex functionality that Media Mover offers and people who need their data to be converted on demand. This module will run specified Media Mover configurations on node inserts and updates- meaning that when a user submits content or updates content, the specific Media Mover configurations will be run. In addition, this functionality leverages some earlier updates that let Media Mover run only on the specific node in question. This helps keep processing overhead down.

This screen shot shows the listing of the Media Mover configurations that are available to run. Note that the administrator should choose configurations that are harvesting from nodes- otherwise it is just wasted cycles.

Thanks to heyrocker for the photo.

I just realized that my talk on Media Mover at the Seattle Drupal Users’ Group from this past June was posted on Blip. I guess I’m a bit behind the times! This is similar to the talk that I did in NYC with less command line and more talking. Interestingly, some of the items that are “future plans” have actually been completed already and are in the alpha releases of Media Mover 1.

The Seattle group is really impressive- super well organized, really well attended weekend with great talks and great people. Both Robin and Gregory really knocked this one out of the park!

The power and flexibility of FFmpeg is a strength and a weakness. While it provides a huge number of options, it’s easy to create a configuration that actually doesn’t work. Case in point has been my experience with Media Mover- the most common thing that people want to use it for is creating FLV files from a variety of source material. Unfortunately, setting the codec or bit rate right can be somewhat challenging, to the point where people create configurations that will not convert the files they have.

I’ve setup FFmpeg wrapper to use configuration files. For each output format, a format file can be create which defines what kinds of options are available (codecs, bit rates, sample rates, etc). These files are loaded via ajax and any form that conforms to a set of rules can be updated on the fly as the user selects different options.

While this currently only works for Media Mover’s FFmpeg processing module, it would work for any module that wanted to implement the setup. Further, people who have experience with specific kinds of output formats can contribute back configuration files that help improve the output from FFmpeg.

The screencast below shows this process in action- I’m adding a new configuration to Media Mover and then selecting different output options for FFmpeg. While I only have one configuration complete at this time (FLV) switching between other options grabs the default (ie: all) options. It also turns the advanced configurations off when you select an option that uses the default- this way FFmpeg will take it’s best guess at converting unless you specifically tell it to.

On top of this, I’ve added support in mm_ffmpeg to enter in your own command- while this is potentially dangerous, it gives people with advanced needs of FFmpeg the ability to run what they need.

Doug Green just did a great presentation for us on SimpleTest. It is really inspiring to see how straight forwarding adding testing is to modules. I spent a few minutes to add the frame work to Media Mover and very very… well, simple test. This first one is not much but for the 1.0 release, I will start adding more…