File API

So there is quite a bit of duplicated code and convoluted interfaces for manipulating files. especially images...

imagecache, image, watermark, imageexact, etc, etc are all relatively simple 'filters' or 'processors' for files. I've got different versions of imagecache with various gd filters, image2ascii conversion, mosaic affects, etc floating around, along with some tools for processing quicktime files and front-ends for ffmpeg I'd like to make available to the community, but coding a module for each one would be a real pain, and would make for a rather module heavy drupal...

imagecache is really a proof of concept work for generating images on the fly, but I had to throw the 'preset' and 'action' management interface into it so it could do its good work. I think there is a strong need to sort this kind of processing into its own little library. Actions.module or filter.module would be a good model for a UI I think.

Sorry, I would've posted this to the What do you want in a file API? thread, but unfortunately you can't attach files to comments.

These are some use cases that Shane Peery, one of our clients, wrote up for AMBuzz, an intranet site for a large media distribution company. They have some fairly complex needs, so I figured I'd post them here in the hopes they might inspire some of the folks working on this stuff, and also in case anyone had any bright implementation ideas. :)

I've also attached some general observations/comments he made regarding Drupal's file API as someone with experience in programming, but new to Drupal. Some of the stuff he mentions is relatively easy to sort out (for example, the security stuff is a matter of turning on private downloads and moving the files directory outside of public_html), but I think there are some interesting ideas here anyway.

I'm cross posting an issue here to let you guys know of this way:

Private and Public File Access

Banner.module isn't really guilty here. There are a lot more modules that do not work with private file access.

Hey there - I work for the Electronic Frontier Foundation (EFF), and we are migrating our huge old legacy site into Drupal. We are trying to get some help developing some additions to the file upload module, and Zack suggested I post our specs here for feedback from you all.

Here's the email I sent him with the specs of our dream file upload module:

EFF needs a file upload and organization module that interacts with files stored on the filesystem. There are some contributed modules with similar functionality, but none that do exactly what we need. It's my belief that adding functionality to the existing 'upload' module that comes with Drupal would be a fine approach.

The handling and publishing of files is one of Drupals weakest points. My site was largely based on me publishing mp3 files and code. I wanted to upload the files with FTP and publish them from an upload directory on the server. At that point I made my own "filepublishing module". And from that point of I started to ponder on what could be done with the way Drupal handles files. And now I have a rough idea of how I would like to do it.
I probably won't have the time to do any time soon, so I'm sharing it here to make shure the idea is not lost when I start to think about other things.

In another comment (Multisite with separate files/ per vhost) I discuss a problem that I am having with moving databases between multisite installations when they each have their own files/ directory. It concludes:

A solution is to not store the filepath prepended to every filename in files.filepath, but rather to prepend the filepath when needed. Not sure where to begin (maybe file_create_path) but I thought I'd float the idea to see what others think of it before I actually attempt to patch core - and probably some modules...

My primary concerns regarding file handling are

• Ensuring only authorized users can download files
• Making it easy to add a file field to a node form

I've posted more file handling ideas here. Please take a look. I'd like feedback.

Some people suggested that every module should handle files by itself. I am for a central storage of file information, though. This has several advantages such as being able to get a quick overview on all files, re-use files elsewhere and so on.

So, one of the big things with static file serving is that it requires an entire Drupal bootstrap, which sucks up a lot of system resources. We have work to do in Drupal/the File API to support some interesting scalability options, remote files, etc. etc.

But, I think there are some interesting models for doing highly scalable static file serving outside of Drupal (and Amazon's S3 is cool).

I was hoping to have some of these questions answered in the poll, but most people have chosen to just cast a vote and not comment.

What dopry wants...
-- specific processing based on rulesets(mimetype, extension, nodetype, userroles, etc).
-- queued processing for long time functions (ex. transcoding media files)...
-- finegrained access control.
-- themable rendering.

I have other stuff kicking the back of my head, but its mostly architectrual stuff to make it more accessible.

There is a horrible line between file.inc and upload.module. File.inc supposed provides hooks like hook_file_download but these get hijacked by upload.module. Also, file.inc does not define or use any database tables (which is great, as using a database for files should remain optional imho), but upload.module does and confusingly names the database table 'files' rather than 'uploads'.

Its important that we make a definite separation between the core file.inc file and the optional upload.module.

• File.inc should provide hooks and functions for use with files which should not restrict people to having to make a file a node or store any details of a file in the database.