Google Drive File Upload - Pre Generated File Id for Uploads

This is a one of the part of some of articles I wanted to publish for the things I learned or had less of web content when we were integrating a feature for Google Drive Folder Sync. Basic goal was to connect a Google Drive Folder and upload files from our app to Google and vice-versa. In this article we want to focus mainly on the uploads from our application.

Why do we need the Pre Generated Id for Files?

We are using Google Drive API V3 for this feature - https://developers.google.com/drive/v3/web/about-sdk. The core functionality of Drive apps is to download and upload files in Google Drive. In this case for some of the files we were uploading - request did not succeeded due to some indeterminate server error or timeout. Although different request for file upload for same file worked previously or later. And on checking the Google Folder - we found that the file was uploaded multiple times.

To avoid such duplicate file uploads - we needed some kind of a metadata which we can pass on when uploading the file and later on lookup(using that same metadata) if that file exist and if not then only retry the upload. And to solve this problem we found the pre-generated id solution which google provides in Google Drive API was apt for our needs.

Advantaged with Pre-Generated ID even if you retry the upload where the previous request for upload did not return an adequate response but the file upload was successful - Google won't retry that upload and will return a HTTP 409 error instead of creating duplicate files.

How do we get the Pre Generated Id for Files?

There is a short documentation on Drive API V3 page - https://developers.google.com/drive/v3/web/manage-uploads under the section - Uploading using a pregenerated ID which states - The Drive API allows you to retrieve a list of pregenerated file IDs that can be used for uploading and creating resources.

I will add the code snippet written in Ruby Language to fetch pregenerated ids -
https://gist.github.com/abhijitsinha/fb4c6490d610c694262066af8838d790

module Services
  module GoogleDriveSync
    class GoogleGeneratedFileIds
      include HTTParty
      base_uri 'https://www.googleapis.com/drive/v3'

      attr_accessor :access_token, :number_of_ids

       def initialize(access_token, options = {})
         self.access_token  = access_token
        self.number_of_ids = options[:number_of_ids].presence || 1
      end

      def headers
        { 'Authorization' => "Bearer #{@access_token}" }
      end

      def fetch
        response = self.class.get("/files/generateIds?count=#{number_of_ids}", headers: headers)
        JSON.parse(response.body)['ids']
      end
    end
  end
end

We are making a GET request to the API endpoint ‘/files/generateIds’. We need to pass on the google access token (for the user you are uploading files) in the header and specify count(optional, default 10) which is number of ids you need.

Documentation - https://developers.google.com/drive/v3/reference/files/generateIds

How to use the Pre Generated File Id for uploading Files?

Upload and file creation requests can then include these pre generated IDs by setting the id fields in the file metadata. We are using a ruby gem google drive (https://github.com/gimite/google-drive-ruby) to upload files to Google - internally this gem uses the gem google-api-client (https://github.com/google/google-api-ruby-client) which has all the Google API endpoints covered but the implementation is staggering given the vast code-base and API endpoints it is covering, with google_drive you will find better readable methods, better abstraction and as the name suggest more specific to google drive related features.

But the gem google drive do not have the ability to use pre generated file id - so we have forked the gem and made minor changes to support pre generated id for the file uploads(we also plan to setup a small PR for this gem to have the changes incorporated in the original gem) - https://github.com/abhijitsinha/google-drive-ruby/blob/master/lib/google_drive/session.rb#L510

Besides setting pre generated id for the file we also set the description and app_properties(app metadata) for the file to help files identify better and for other purposes specific to our app.

Hope this article helps you with some insights when uploading files to Google and saves your time implementing it. Thank you for reading.