s3 rest

Back in the ‘burgh at Carnegie Mellon’s radio station WRCT, a really awesome poster with a baby crawling to its death in a bucket reminded you of how dangerous buckets can really be. Especially when DJing techno.  I can’t work with Amazon S3 without wishing all my baby resources and folders and files would crawl in to an S3 death bucket and vanish. OMG AMAZON!

So anyhoo, to create folders instead of just files (or what appears to be folders in the awesome S3 firefox organizer), do a PUT with:

•header content_type to binary/octet-stream
•append _$folder$ to the end of the path name (as the extension)
•set the data to blank
This bitty rails code demonstrates this using the S3 plugin:


  S3_FOLDER_EXT = '_$folder$'
  def exists_in_S3?(file_name, bucket_name, options ={})
    connect_to_S3

    file_search_name = file_name.dup
    file_search_name << S3_FOLDER_EXT if options[:folder]

    AWS::S3::Bucket.objects(bucket_name, :prefix => file_search_name).any?

  end

  def create_S3_folder(file_name, bucket_name, options={})
    connect_to_S3
    folder = AWS::S3::S3Object.store(file_name + S3_FOLDER_EXT, '', bucket_name,
      {:access => :public_read,
       :content_type => "binary/octet-stream",
       :cache_control => 'max-age=3600,post-check=900,pre-check=3600' }.update(options))
  end

  def connect_to_S3
    unless AWS::S3::Base.connected?
      AWS::S3::Base.establish_connection!(:access_key_id => 'blah', :secret_access_key => 'giraffe!')
    end
  endSo your request path would look like:
/baby_bucket?prefix=folder1/folder2_$folder$
And the output:

<ListBucketResult xmlns=”http://s3.amazonaws.com/doc/2006-03-01/”>
<Name>baby_bucket</Name>
<Prefix>folder1/folder2_$folder$</Prefix>
<Marker></Marker>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
</ListBucketResult>


At this point, you should ask yourself why you want to create folders in the first place. Because I want to stick my head in a bucket right now.


=========================================================
how can i create a folder under a bucket using boto library for amazon s3, i followed the manual, and create keys with contents with permission, metadata etc, but no where in the boto's documentation say how to create folders under bucket, or create folder under folders in bucket.
amazon-s3 boto
link|flag edited Dec 21 '09 at 12:11
skaffman
62.7k642100 asked Dec 21 '09 at 12:04
vito huang
1348

86% accept rate

  it seems amazon s3 doesn't have concept of folder, some suggest creating key with name like "folder/test.txt' to get around it. i tried using firefox s3 plugin to create folder, and list all keys in boto, it outputs the folder i just created as "<Key: vitoshares,everyone_$folder$>], so how can i view/add/modify content to/from this folder? – vito huang Dec 21 '09 at 12:29

2 Answersactive
newest
votes 
up vote
3

down vote
accepted  Yes, thers no folder concept in S3, you can create file names like "abc/xys/uvw/123.jpg" which many S3 access tools like S3Fox show like a directory structure. But actually the its a single file in a bucket.

Amey Kanade
link|flag answered Jan 26 at 18:03
user240469
514

  thanks for the answer, so i guess if i want to see the contents of a particular folder, i will need to traverse lots other unnecessary files? – vito huang Jan 27 at 23:50
The S3 API lets you query a bucket for keys with a given prefix. So you should be able to construct a request that only returns objects in your logical folder. See: docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/… – James Cooper Jul 22 at 19:58



up vote
1

down vote  Hey,

Amazon S3 indeed doesn't have a concept of the folders. As you correctly mentioned you need to create keys with slash symbol to mimic that. You can use CloudBerry Explorer freeware to create folders separated by '/' rather than with $folder$ suffix.


==================================================
Create a directory on S3 - how?
Posted by: Luis Batista - Dom Digital
Posted on: Mar 9, 2010 9:23 AM     Reply

 

This question is not answered. Helpful answers available: 2. Correct answers available: 1. 

Hello,

A newbie question, how create a directory (folder) on a bucket ?

I use this code:
--------------------------------
foreach (FileInfo FI in FileList)
{
  AmazonS3 clientS3 = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKeyId, secretAccessKey);
    try
    {
      PutObjectRequest request = new PutObjectRequest();
      request.WithFilePath(FI.DirectoryName + "
" + FI.Name)
      .WithBucketName(BucketName)
      .WithKey(MyDirectory + "/" + FI.Name)
      .WithCannedACL(S3CannedACL.PublicRead);
      S3Response response = clientS3.PutObject(request);
      response.Dispose();
   }
   catch (AmazonS3Exception ex)
   {
      if (ex.ErrorCode \!= null && (ex.ErrorCode.Equals("InvalidAccessKeyId") || ex.ErrorCode.Equals("InvalidSecurity")))
      {
         Console.WriteLine("Please check the provided AWS Credentials.");
         Console.WriteLine("If you haven't signed up for Amazon S3, please visit http://aws.amazon.com/s3");
      }
      else
      {
         Console.WriteLine("An error occurred with the message '{0}' when writing an object", ex.Message);
      }
   }       
}   

---------------------------
In "MyDirectory" variable I put the name of folder I want on bucket.
I use the firefox pluglin "S3 Organizer" to manage files on Amazon S3, if I delete the file, the directory is removed, is like the name of file include the directory name!

Any help with this situation?

Regards,
DD




  Replies: 3 | Pages: 1 - Last Post: Mar 12, 2010 7:48 AM by: Colin Rhodes 


Replies 

Re: Create a directory on S3 - how?
Posted by:  Colin Rhodes
Posted on: Mar 9, 2010 11:12 AM
in response to: Luis Batista - Dom Digital     Reply

 

There is no such thing as directories on S3. Many S3 clients simulate folders by detecting '/' characters in the keys of the objects on S3.

Some clients will allow you to create empty objects with keys that end in a '/'. Some clients use these objects as a folder placeholder. 



Re: Create a directory on S3 - how?
Posted by: Luis Batista - Dom Digital
Posted on: Mar 11, 2010 4:56 AM
in response to: Colin Rhodes     Reply

 

Hello,

You can put a example code how create a empty object with "/" and then put objects in this "placeholder" ?


Regards,
DD


Message was edited by: Luis Batista - Dom Digital 



Re: Create a directory on S3 - how?
Posted by:  Colin Rhodes
Posted on: Mar 12, 2010 7:48 AM
in response to: Luis Batista - Dom Digital     Reply

 

When I use S3, I don't create placeholder "folder" objects because I have no use for them. My main data bucket on S3 has the data broken down into sets of files associated with an ID, where the ID is the "folder".

When I want to write a data object, I simply write to "{id}/{filename}". I have no need to create a placeholder "folder" object.

The important thing to understand is that the placeholder "folder" doesn't do anything except when visualizing the structure. You don't add objects to a folder -- you just create them with the desired prefix.

That said, if you really want to create a placeholder, you could write an empty string object to the key 'myfolder/'. If an empty string doesn't work, add a single character. 



==========================================================
Using .NET AWS SDK to create folder in S3 bucket
Posted by: gopinatht
Posted on: Aug 30, 2010 12:00 PM     Reply

 

Hi All,

I apologize if this question has been asked before but I did a search and I could not find a good response.

I am trying to create a folder in a bucket in S3 using rhe AWSSDK .NET API. Here is my code:

PutObjectRequest putReq = new PutObjectRequest();
putReq.BucketName = "MyUniqueBucket";
putReq.WithKey("FromApi/");
putReq.WithContentType("binary/octet-stream");
s3Client.PutObject(putReq);

When I do this, I get the following exception:

An unhandled exception of type 'System.ArgumentException' occurred in AWSSDK.dll

Additional information: Please specify either a Filename, provide a FileStream or provide a ContentBody to PUT an object into S3.

What am I doing wrong? How can I create the folder?

Thanks in advance.

Best Regards
Gopinath



  Replies: 1 | Pages: 1 - Last Post: Aug 31, 2010 12:05 AM by: Norm@AWS 


Replies 

Re: Using .NET AWS SDK to create folder in S3 bucket
Posted by:  Norm@AWS
Posted on: Aug 31, 2010 12:05 AM
in response to: gopinatht     Reply

 

S3 doesn't support folders in a bucket.  Many tools simulate folders by adding a 0 byte object when the folder is first being "created" and then removing the 0 byte file once the first object is added to the simulated folder. 

In your example you create the folder by putting a 0 byte object with an object key "FromAPI_$folder$".  Then later objects would be added with the object key set to something like "FromAPI/FirstObject". 

Norm