Friday, April 24, 2009

Amazon S3 Sub-Directories SOLVED!

Today was 'S3 DAY'. A number of ups and downs for success and frustration on this project.

I was frustrated with the inability to have sub-directories or sub-folders in S3.

However, thanks to these posts: http://developer.amazonwebservices.com/connect/message.jspa?messageID=36250, it's a snap!

All you have to do is put slashes in the file name and S3 assumes the rest. No fuss like there is with Buckets.
For example: [BucketName]/projectid_00000001/fileid_0000000001/ActualFileName.txt
where you tell S3 that the file name is "projectid_00000001/fileid_0000000001/ActualFileName.txt".


Progress on the RiaForge S3.cfc
I have made quite a number of custom modifications on the RiaForge s3.cfc project. I have stripped it down to remove all bucket support, fixed the issues raised from differences using Railo on Linux, and re-programmed a number of variables to work from the Application scope.

I was also able to design it to maintain a project sub-directory schema using the project ID numbers. S3 seemed to get fussy with subfolders as numbers, so I put some characters in front of the ID number (like 'projectid_00000001').

I was previously using a sub-sub-directory schema to maintain file versions. The file names are stored in the database and the record Primary Key ID padded with zeros to ten characters in length enabled multiple files in the same directory with the same file name.

The S3 schema ends up like this:

[BucketName]
projectid_00000001
fileid_0000000001
ActualFileName.txt
fileid_0000000002
ActualFileNumber2.txt
fileid_0000000003
ActualFileName.txt    (This is actually a newer version of fileid_0000000001 by storing the 'parentid' DB field if '1', code handles the rest)
projectid_00000002
fileid_0000000004
WhateverFileName.txt     (Again, here is the original file, newer versions follow)
fileid_0000000005
WhateverFileName.txt     (This is a newer version of fileid_0000000004, handled by code, parentid=4, sorted by createdate for all children)
fileid_0000000006
WhateverFileName.txt     (This is the newest version of fileid_0000000004, handled by code, parentid=4, sorted by createdate for all children)



Hmm, the parent file ID schema is a little bit different and more complicated than the above example, but you get the idea. There is a directory for each file. The directories are each unique allowing the file within the directory to be identical to other files in the same project. The code is doing all the heavy lifting, all required file info for the display is stored in the database, including Category IDs, Taggings, or whatever. Authenticated links are generated for instant download form Amazon S3 so the entire bucket can be private and secure.

It's a sweet setup.