Spam: The other white meat

update 12/9/06:

I've done quite a bit of working pulling together and administrative system. It's by no means polished, but the database is storing each set of configurations (for harvest, process, and storage) and can pull each of these out for each run of of the configuration. This means that it's possible to have multiple runs of say a video conversion with different conversion rates.

The media_mover hook is almost stable at this point, the major piece left is the "save" function which keeps track of how files were moved around and created.

----

For a few reasons, I've decided to announce the media mover module long before it's ready for complete release. One is that it's clear other people in the drupal community are really looking to tackle media handling, particularly video processing, another is that it's high time that I get the code that I did for flunk arnold project out into the world in a more public way.

So first the disclaimer: this code is raw. It's not been baked yet at all. That being said, the rough concept is in place.

What it is
The Media Mover module is a module which allows admins to setup a media move process which gathers files, processes them and stores them. Media Mover does little on its own- it calls a set of modules which implement the media_mover hook and plugs them together in to media mover scripts.

What it does
Media mover is a handler module for a set of driver modules. It basically runs the three types of actions that any one media move can have: harvest, process, storage.

harvest- a mechanism to gather the files that drupal will process
process- a mechanism to process the files
storage- a mechanism to store files.

Currently I have a harvest process which uses the drupal files table, a process process which uses ffmpeg to convert video files, and an amazon s3 module which stores the completed files on amazon. I have to build the administrative side of things, but I have it running now so that you can pass in something like:


$config->harvest->module = "media_mover";
$config->harvest->action = "db";
$config->process->module = "mm_ffmpeg";
$config->process->action = "create thumbnail from video";
$config->process->configuration = "";
$config->storage->module = "mm_s3";
$config->storage->action = "move file to amazon s3 service";
$configurations[] = $config;


What's kind of fun about this is that you can stuff these configs into an array, and process the media again to do something different, eg, duplicating the above and changing the process action to:


$config->process->action = "convert video";


Which would go back through the same files and process the uploaded files and turn them into videos.

I still have a fair bit of abstracting, code clean up and error handling to implement, but I created a hook media_mover which allows you to create your own handler:


function media_mover_media_mover($op, &$file = array()) {
switch ($op) {
case 'name':
return "Media Mover module";
break;case 'configuration':break;case 'admin':
return _media_mover_admin();
break;case 'type':
return array(1 => "harvest");
break;

case 'actions':
return array(1 => "select drupal uploaded files");
break;

case 'directories':
return array('converted', 'converted/thumbnails');
break;

case 'harvest':
return _mm_get_list_of_files();
break;

case 'process':
break;

case 'storage':
break;

default:
return;
break;

}
return;
}

There is a large amount of work to go, particularly writing a configuration system that is allows admins to sets of configurations, however, I think things are exciting enough to start imaging what kind of modules could be built.... let's see:

harvest: fetch files from a ftp/scp server
harvest: email inbox (grab cell phone video)
harvest: get files from the drupal files db (already written!)
harvest: collect files at a particular action/workflow state
harvest: get youtube videos from a particular userprocess: create video teaser
process: create audio teaser (podcasting?)
process: use ffmpeg to convert files (already written!)
process: create different bit rates of video/audio (pro downloads vs. anon downloads)storage: ftp/sftp storage
storage: amazon s3 storage (already written!)
storage: backupstorage: backup

Since it's now possible to use it, I think I'm getting close to wanting to solicit comments, however, until I build the configuration builder, it's only really useful for admins.
I'll be posting updates as things progress. This code is going to be public available and will be through out the development process.

Please note, this tarball includes the Storage3 amazon library from http://blog.apokalyptik.com/Storage3/ and some pear libraries. Don't expect this to work if you're downloading it. Do expect to look at the media_mover hook and think about how 1) it could be better 2) how to make some amazing module that could plug into this

Download the prelim code