[Amazon S3] Reading File content from S3 bucket in Java

In continuation to last post on listing bucket contents, in this post we shall see how to read file content from a S3 bucket programatically in Java. The ground work of setting the pom.xml is explained in this post

Lets jump to the code. The piece of code is specific to reading a character oriented file, as we have used BufferedReader here, we shall see how to get binary file in a moment.

Printing File content on Console

public void readFromS3(String bucketName, String key) throws IOException {
    S3Object s3object = s3.getObject(new GetObjectRequest(
            bucketName, key));
    System.out.println(s3object.getObjectMetadata().getContentType());
    System.out.println(s3object.getObjectMetadata().getContentLength());

    BufferedReader reader = new BufferedReader(new InputStreamReader(s3object.getObjectContent()));
    String line;
    while((line = reader.readLine()) != null) {
      // can copy the content locally as well
      // using a buffered writer
      System.out.println(line);
    }
  }

We create a GetObjectRequest instance, passing in the bucket name and the file name. Passing the bucket name is where a lot of issues may come, so please specify it correctly.

Next steps are same as reading a normal file. We read line by line and print the content on Console. If you need to save the content in a local file, you can create a BufferedWriter and instead of printing write to it (Don't forget to add new line after writing to buffer).

S3 Object metadata has some interesting information about the object. Do take time to print it.

Calculating size of file

What if we need to calculate the file size. S3 Object Metadata does contain the file length. Let's see how we can calculate it. We shall modify the above snippet a bit.

public void readS3ObjectUsingByteArray(String bucketName, String key) throws IOException {
    S3Object s3object = s3.getObject(new GetObjectRequest(
            bucketName, key));

    InputStream stream = s3object.getObjectContent();
    byte[] content = new byte[BUFFER_SIZE];

    int totalSize = 0;

    int bytesRead;
    while ((bytesRead = stream.read(content)) != -1) {
      System.out.println(String.format("%d bytes read from stream", bytesRead));
      totalSize += bytesRead;
    }
    System.out.println("Total Size of file in bytes = "+totalSize);
  }

This is the snippet modified to read any file. Here we operate a byte level. We create a byte buffer and keep on reading from the stream. I have chosen a buffer of 64K, try with different setting. It's interesting to see how the number of bytes read differ. Have fun with this.

Copying the S3 File locally

Let's modify the above example to write the file locally.

public void readS3ObjectUsingByteArray(String bucketName, String key, String outputFileName) throws IOException {
    S3Object s3object = s3.getObject(new GetObjectRequest(
            bucketName, key));

    InputStream stream = s3object.getObjectContent();
    byte[] content = new byte[BUFFER_SIZE];

    BufferedOutputStream outputStream = new BufferedOutputStream(new FileOutputStream(outputFileName));
    int totalSize = 0;
    int bytesRead;
    while ((bytesRead = stream.read(content)) != -1) {
      System.out.println(String.format("%d bytes read from stream", bytesRead));
      outputStream.write(content, 0, bytesRead);
      totalSize += bytesRead;
    }
    System.out.println("Total Size of file in bytes = "+totalSize);
    // close resource even during exception
    outputStream.close();
  }

The code remains same, we just start writing the byte array content to the file. Watch out while writing to the OutputStream, ensure to write only the array portion which has content i.e. number of bytes read from stream. We have the code ready. If you have worked on creating S3 browser, you can add the functionality to download the file now.

Bringing everything together

The complete code can be found at S3Reader.java

2 thoughts on “[Amazon S3] Reading File content from S3 bucket in Java

  1. when i am accessing bucket, getting exception.

    Error Message: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 6C54B6E926338CBF)
    HTTP Status Code: 403
    AWS Error Code: AccessDenied
    Error Type: Client
    Request ID: 6C54B6E926338CBF

    • Because your file is not public , make sure you are reading it using your app id and app secret after craeting IAM user with power access. or making the file publicly readable.

Leave a Reply

Your email address will not be published. Required fields are marked *