| Database Storage System [message #10741] |
Wed, 23 August 2006 01:50  |
 erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
Hi everyone,
I just posted a Wiki article that offers a system for storing files in a database that uses a caching system. It also addresses the problem of the dot in the query string which seems to pop up on the forum a lot.
If anyone has a chance, could you download the project and make sure that it works for you?
http://www.symfony-project.com/trac/wiki/DatabaseFileStorage System
Thanks.
Jim Podroskey [www.pbtg.com]
|
|
|
|
|
|
| Re: Database Storage System [message #10863 is a reply to message #10764 ] |
Thu, 24 August 2006 16:56  |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
| joesimms wrote on Wed, 23 August 2006 13:56 |
Hi
I just wondered if there is any reason for having the file info in a seperate table?
|
Yeah... The reason is for performance issues. We keep the information about the file seperate so we don't need to pull large amounts of Blob data just when we're listing files or when its in the cache. We access the file_data ONLY when we need to.
Jim Podroskey [www.pbtg.com]
|
|
|
|
|
|
| Re: Database Storage System [message #10902 is a reply to message #10896 ] |
Fri, 25 August 2006 14:44 |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
| smallbeer wrote on Fri, 25 August 2006 12:21 |
This method really does seem like having the best of both worlds.
Could this method be used for media files? specifically MP3s? Or would you say the file size involved really shouldn't be stored in the db?
|
Well, configuration files are one of the limiting factors. The default packet size for mysql defaults to 1 meg, and reading the file info memory causes a hit on what PHP is storing.
I personally haven't tested this method with anything bigger than 4 or 5 meg, but that should be good for MP3s. And yes, this method should work independent of file types. Next week I'll do some tests and see when things start getting sluggish.
Jim Podroskey [www.pbtg.com]
|
|
|
| Re: Database Storage System [message #13543 is a reply to message #10741 ] |
Mon, 09 October 2006 23:35 |
forkmantis
Messages: 37
Registered: November 2005
Location: Texas |
Member |
|
|
Thanks Erestar. This was a great article.
I needed it for a project I'm working on, and found it very easy to implement.
I ran into some trouble with large files, and started looking into different methods of addressing the issue. Google lead me to this page: http://www.dreamwerx.net/phpforum/?id=1
I borrowed the idea of splitting the file into chunks, and made an adaption of your code. I've attached the modified code to this message.
I've implemented a few additional details in my production version that I'd be willing to share if there is interest.
- saving file by filename rather than id
- cleaning file names to be URL safe
- generating subdirectories 3 levels deep to prevent too many files from residing in a single directory
- checking for duplicate file name before commit
- using checksums to check for duplicate file before commit
- allow for a new version of an existing file to be uploaded to the same permalink. Removes old version of the file from cache
will killian
|
|
|
|
|
| Re: Database Storage System [message #13613 is a reply to message #10741 ] |
Tue, 10 October 2006 17:15 |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
That's awesome forkmantis. I'll take a look at the code this evening.
I'm in the process of implementing this as a new plugin. Its actually just about down, but I'll probably grab your shecksum and splitting method. I'd already done the other things you mentioned.
With your work hopefully I'll finish the plugin system soon and get it uploaded for everyone's review.
Thanks!!
Jim Podroskey [www.pbtg.com]
|
|
|
| Re: Database Storage System [message #13615 is a reply to message #13613 ] |
Tue, 10 October 2006 17:21 |
forkmantis
Messages: 37
Registered: November 2005
Location: Texas |
Member |
|
|
Hmmm... I was just preparing to implement the other features. So do you have them all covered? Is there any point in me doing anything else with the original code at this point? Is there any way I can contribute to the plugin?
will killian
|
|
|
| Re: Database Storage System [message #13616 is a reply to message #13615 ] |
Tue, 10 October 2006 17:24 |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
Oh, sorry, I misread you!
Tell you what... I'll get the plugin up into the wiki and source and you can take a look. It requires sfGuard to work, and needs some extra interfacey stuff. However, I'd be more than happy to share the workload on getting this up to production quality.
I'm also working on a Photo Gallery plugin that is based on the file storage plugin, so if there's any interest in that I'll post it as well.
Jim Podroskey [www.pbtg.com]
|
|
|
| Re: Database Storage System [message #13617 is a reply to message #13616 ] |
Tue, 10 October 2006 17:30 |
forkmantis
Messages: 37
Registered: November 2005
Location: Texas |
Member |
|
|
Sounds good.
Would it be worth putting up a subversion repository? I'd be willing to host, if need be.
will killian
|
|
|
| Re: Database Storage System [message #13618 is a reply to message #13617 ] |
Tue, 10 October 2006 17:35 |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
Well, I was gunna try to use symfony's. Otherwise I already have it in my companies repo.
I need to talk to my manager one last time just to make sure that he's still on-board with open-sourcing our CMS application (which is what all of these plugins are for). Shouldn't be a problem though!
Jim Podroskey [www.pbtg.com]
|
|
|
| Re: Database Storage System [message #13636 is a reply to message #10741 ] |
Tue, 10 October 2006 22:48 |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
Still waiting to hear back from Fabien and for my server admin to open up the subversion port.
Until there, here's the plugin.
You'll notice that the bulk of the code from the action has been moved to a utility class, and there's also a utility to unzip compressed files and create database objects for all of them.
Forkmantis,
If you're willing to temporarily setup a repo for this that'd be great. I'm going to continue working on this and my photo gallery tonight, and was hoping you'd have a chance to implement some of your changes into this new plugin architecture.
Thanks!
Jim Podroskey [www.pbtg.com]
|
|
|
| Re: Database Storage System [message #13668 is a reply to message #13543 ] |
Wed, 11 October 2006 16:18 |
Draven
Messages: 760
Registered: January 2006 |
Faithful Member |
|
|
| forkmantis wrote on Mon, 09 October 2006 23:35 |
- saving file by filename rather than id
- cleaning file names to be URL safe
- generating subdirectories 3 levels deep to prevent too many files from residing in a single directory
- checking for duplicate file name before commit
- using checksums to check for duplicate file before commit
- allow for a new version of an existing file to be uploaded to the same permalink. Removes old version of the file from cache
|
I have a need for this plugin so I'm going to implement most of the above today.
Let me know if you have an SVN going or I'll post the changes back to this post.
A few other options I need to add is.
- Retrieve a list of files by name and show previews in thumbnail size for use in an article/content system.
- clean up admin interface a bit and turn this more into a media manager than just a file storage.
- Files will be stored in same upload cache dir except the first letter of file name will be used to store in alpha-numeric subfolders to organize files.
If you would rather I branch this into a media manager I can do so. Let me know if what I mentioned above doesn't fit your original needs.
[Updated on: Wed, 11 October 2006 16:23]
|
|
|
|
|
|
| Re: Database Storage System [message #13672 is a reply to message #13670 ] |
Wed, 11 October 2006 16:39 |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
Hi guys,
I'd like to keep the file storage plugin kind of seperate from other plugins. The main purpose of this is to make it work with other plugins, like the media storage.
The file plugin should just provide a means for other data objects to transparently access files.
In addition to the file plugin, I also have a gallery plugin in the works.
The gallery plugin offers the ability to associate both a full sized image and a thumbnail with a photo object, which contains tags, description, photographer, etc. I'm in the process of making a very full featured admin section for this to easily allow tag and description changes.
The file plugin allows users to upload zip files, and the gallery provides plugin options.
These plugins are currently at svn://madhatter.pbtg.com/cms/plugins
Checkouts can be anonymous, so if you'd like commit access please just email me at erestar@gmail.com and I'll get you set up.
On some more annoying news, my home internet was out this morning and I had a few more things to commit, but the file plugin (sfPropelFileStoragePlugin) should be stable enough to use.
I look forward on working on these with you. Let me know if you want commit access to the repo.
Jim Podroskey [www.pbtg.com]
|
|
|
|
| Re: Database Storage System [message #13677 is a reply to message #13676 ] |
Wed, 11 October 2006 17:35 |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
Sounds good Draven. I was initially going in the exact same direction as you, but after thinking about it for a while decided against putting too much into this system.
You're more than free to do whatever you like but the design of this plugin lends its self really well to being used in whatever manner you want. If you want to discuss how your media plugin can simply make use of this plugin I'd be more than happy to talk to you. My only concern is you'll be incorporating this plugin into an existing plugin, which kind of defeats the purpose of plugins, IMHO.
Anyway, let me know how you want to work this!
[Edit]
Oh, and I just wanted to be clear that I'm not trying to be a credit hog here... I'm more concerned about having specific plugins for specific things. The whole point of sfFileStorage is to be used as a plugin in other plugins just like what you're describing. Maybe just take a look at how the sfGalleryPlugin makes use of the sfPropelFileStorage plugin at the SVN link provided previously.
[Updated on: Wed, 11 October 2006 17:37]
Jim Podroskey [www.pbtg.com]
|
|
|
| Re: Database Storage System [message #13678 is a reply to message #13677 ] |
Wed, 11 October 2006 17:38 |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
Oh, also, I'd be really interested in your new thumbnail plugin, as I've already had to hack the current one several times to get it working how I want. Please keep me posted on its progress.
Thanks!
Jim Podroskey [www.pbtg.com]
|
|
|
| Re: Database Storage System [message #13679 is a reply to message #10741 ] |
Wed, 11 October 2006 17:56 |
Draven
Messages: 760
Registered: January 2006 |
Faithful Member |
|
|
This doesn't really defeat the purpose of plugins, as a media manager is still a plugin, just a more powerful one.
I don't personally like the idea of having 5 plugins to do the job one plugin could do. To me it's like splitting the sfGuard plugin into sfUser, sfCredentials, sfGroups and sfAuth (think plsitting it's modules into plugins, which is basically the same argument). This seems silly which is why it's just sfGuard.  The same idea for a media plugin, rather than having sfFileStorage, sfThumbNail, sfImageFileSystem, etc etc, you just have sfMediaManager with methods to use whichever part is needed in other plugins.
Again, we just have different views on the situation, which is alright. That's the beauty of opensource
I had already planned to build this plugin for a project I have, you just helped jump start one part... thanks again for posting it.
[Updated on: Wed, 11 October 2006 17:59]
|
|
|
| Re: Database Storage System [message #13688 is a reply to message #13679 ] |
Wed, 11 October 2006 18:48 |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
I still disagree with your approach but whatever floats your boat! Good luck!
Jim Podroskey [www.pbtg.com]
|
|
|
|
| Re: Database Storage System [message #18032 is a reply to message #10741 ] |
Sun, 17 December 2006 05:08 |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
Glad to hear it! If you have any feature requests feel free to post them here or ask me or Forkmantis in the IRC channel.
Jim Podroskey [www.pbtg.com]
|
|
|
| Re: Database Storage System [message #18546 is a reply to message #18032 ] |
Thu, 28 December 2006 04:29 |
infect
Messages: 98
Registered: January 2006 |
Member |
|
|
I think there's a small bug in the schema if you're attempting to use this plugin with sfGuard. I'm running symfony 1.0-beta2
The current schema that is used (if you use the "symfony file-storage-set-user-manager sfGuard" task):
<foreign-key foreignTable="sf_guard_user" phpName="user" >
<reference local="sf_guard_user_id" foreign="id" />
</foreign-key>
This should be:
<foreign-key foreignTable="sf_guard_user">
<reference local="user_id" foreign="id" />
</foreign-key>
... since there is no "sf_guard_user_id" column in the file_info table (it's called "user_id").
Also, in the documentation for the plugin, under the section for modifying the .htaccess file, there's a small typo... it should be:
RewriteCond %{REQUEST_FILENAME} !-f
instead of:
Cond %{REQUEST_FILENAME} !-f
[Updated on: Thu, 28 December 2006 04:36]
Mark Quezada
MirthLab
|
|
|
|
| Re: Database Storage System [message #21959 is a reply to message #18652 ] |
Mon, 19 February 2007 11:13 |
ksn135
Messages: 22
Registered: January 2007
Location: Moscow, Russia |
Junior Member |
|
|
For correct handling non-english filenames add 4 lines marked as "++" to serve() function in
\plugins\sfPropelFileStoragePlugin\modules\sfPropelFileStora ge\actions\actions.class.php
<?php
class sfPropelFileStorageActions extends sfActions
{
...
protected function serve($file_info)
{
...
$this->getResponse()->setHttpHeader('content-type', $file_info->getMimeType(), true);
$this->getResponse()->setHttpHeader('content-length', $file_info->getSize(), true);
++ $this->getResponse()->setHttpHeader('content-disposition',
++ sprintf( '%s; filename="%s"',
++ (strstr($_SERVER['HTTP_USER_AGENT'],'MSIE')) ?
++ 'inline' : 'attachment', $file_info->getName() ));
return $file_data;
}
...
}
?>
And in your template DON'T forget to add urlencode function call:
<?php
...
$file = $object->getsfPropelFileStorageInfo();
$name = $file->getName();
echo link_to($name,'@download_by_file_name?name='.urlencode($name));
...
?>
--
WBR,
ksn135.
|
|
|
| Re: Database Storage System [message #21960 is a reply to message #21959 ] |
Mon, 19 February 2007 11:15 |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
I'll make those changes tomorrow when I get into the office. Thanks for the tip!
Jim Podroskey [www.pbtg.com]
|
|
|
| Re: Database Storage System [message #22080 is a reply to message #21959 ] |
Tue, 20 February 2007 17:32 |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
Me and Sporkmantis are working out the best way to make the response headers configurable. Hopefully we'll have something by the end of the day!
Jim Podroskey [www.pbtg.com]
|
|
|
| Re: Database Storage System [message #22225 is a reply to message #21959 ] |
Wed, 21 February 2007 21:45 |
forkmantis
Messages: 37
Registered: November 2005
Location: Texas |
Member |
|
|
| ksn135 wrote on Mon, 19 February 2007 04:13 |
For correct handling non-english filenames add 4 lines marked as "++" to serve() function in
\plugins\sfPropelFileStoragePlugin\modules\sfPropelFileStora ge\actions\actions.class.php
...
|
ksn135,
Could you give us an example of a file you are having trouble w/?
will killian
|
|
|
|
| Re: Database Storage System [message #22323 is a reply to message #22261 ] |
Thu, 22 February 2007 20:40 |
infect
Messages: 98
Registered: January 2006 |
Member |
|
|
Erestar,
Here's an issue I've come across that I wanted to let you know about. After using this plugin for a while (it works pretty well) I noticed that there's really no reason to store the cache status in the database. In this file "modules/sfPropelFileStorage/actions/action.class.php" there's this code:
<?php
if($file_info->getIsCached() && $cache->has($file_info->getId(), 'uploaded_files'))
{
//Ok! We have a cached copy of the file! We're going to created a throw-away FileData object to
//store our cached file in.
$file_data = new sfPropelFileStorageData();
$file_data->setBinaryData($cache->get($file_info->getId(), 'uploaded_files'));
}
?>
...the "$file_info->getIsCached()" check can pretty much be omitted as checking if the cache itself has the file is sufficient. In fact, you can remove all instances of this check (as well as removing the field from the database) since checking the actual cache will always be enough to know if you should grab the file from the database. Although this seems to work fine the way it is for smaller things, once this application gets distributed across many servers, the database will get updated with an "isCached = true" status if one of the frontend servers has cached the file, but since the other frontend servers have not cached the file, they'll need to go grab it from the database as well, which basically makes the isCached checking and setting not only useless but it will also execute unnecessary queries on the database (since everytime a file is grabbed from the database, it will update the isCached field).
Let me know if that makes sense or if you'd like more samples.
Thanks again for the great work on this plugin!
Mark Quezada
MirthLab
|
|
|
| Re: Database Storage System [message #22325 is a reply to message #22323 ] |
Thu, 22 February 2007 21:26 |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
That's a really good point and I'll chalk that up to an oversite on my part. Fix forthcoming.
Jim Podroskey [www.pbtg.com]
|
|
|
| Re: Database Storage System [message #22327 is a reply to message #22325 ] |
Thu, 22 February 2007 21:56 |
erestar
Messages: 114
Registered: November 2005 |
Senior Member |
|
|
Actually, we always need to make a call to the database, because the database should verify that the file exists. Otherwise, in a multiserver environment, the cache file could remain if the file was removed from the DB on another server.
However, discussions with Fork brought up the fairly obvious point that, also in a multiserver env, the first time a file is changed it'll recache on that server, have it's is_cached flag set to true, then continue serving up the old file on other servers.
I'm not positive how I feel about _changing_ the contents of the file to preserve the ID (since it seems to make more sense to just upload a new file and preserve the name) but I'm thinking about implementing timestamps on the cache files and comparing with the created at date.
Anyone have any thoughts?
Jim Podroskey [www.pbtg.com]
|
|
|
Members
Search
FAQ
Register
Login
Home
