We are aware of the issue with the badge emails resending to everyone, we apologise for the inconvenience - learn more here.

Forum Discussion

DavidM27's avatar
DavidM27
Explorer | Level 4
5 years ago

Issue UploadSession and UploadSessionBatch with big files

Hello, 

 

I'm working with the dropbox API in TypeScript. I do want to be able to able all kind of files (small, large, multiple files...)

For that I have made a method to uplaod a single file, and a method to uploadMultiple file. The fisrt one is using the uploadSessionFinish() method provided by the sdk, the other one is using uploadSessionFinishBatch(). To handle small and large file I also have made a method that handle the file and return a filesUploadSessionFinishArg so I can use it in my uploadFile and uploadMultipleFiles method. 
This is working well for a single file either small or large. But as soon as I upload multiple files containing at least one big file it fails... 

Here is what the code looks like : 

Upload single file ->

  async uploadFile( file: ArrayBuffer, path: string): Promise<MyFile> {
    const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg = await 
    this.prepareUploadFile({ file, path, })
    const _file: DropboxTypes.files.FileMetadata = await 
    this.dropBox.filesUploadSessionFinish(finishSessionArg)
    return dropBoxMapper.convertDropboxFileToFile(_file)
  }

Upload multiple file -> 

async uploadMultipleFiles(files: Array<ArrayBuffer>, filesPath: Array<string>) : Promise<Array<MyFile>> {
    const finishUploadBatchArg: 
     DropboxTypes.files.UploadSessionFinishBatchArg = {
      entries: [],
    }
    // For each file I get the finishArg
    for (const [index, file,] of files.entries()) {
      const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg = 
      await this.prepareUploadFile({ file:file, path:filesPath[index], })
      finishUploadBatchArg.entries.push(finishSessionArg)
    }

    const response: DropboxTypes.files.UploadSessionFinishBatchLaunch = 
    await this.dropBox.filesUploadSessionFinishBatch(finishUploadBatchArg)

    //Check the batch status from here until it is complete
    const checkArg : DropboxTypes.async.PollArg = {
      async_job_id: (<DropboxTypes.async.LaunchResultBaseAsyncJobId> 
       response).async_job_id,
    }

    return this.checkUploadSessionBatch(checkArg)
  }

prepareUpload method -> 

  async prepareUploadFile(file: ArrayBuffer, path: string) : Promise<DropboxTypes.files.UploadSessionFinishArg> {
    const UPLOAD_FILE_SIZE_LIMIT: number = 8 * 1024 * 1024 //For testing purpose max size for a single upload is 8Mb
    const CHUNK_SIZE: number = 4 * 1024 * 1024 // Each chunk is 4Mb 
   // If smaller than 8Mb single upload
    if (UPLOAD_FILE_SIZE_LIMIT > file.byteLength) {
      const uploadSessionStartArg : DropboxTypes.files.UploadSessionStartArg = {
        contents: file,
        close: true,
      }
      const response: DropboxTypes.files.UploadSessionStartResult = await this.dropBox.filesUploadSessionStart(uploadSessionStartArg)
      const sessionId: string = response.session_id
      const cursor: DropboxTypes.files.UploadSessionCursor = {
        contents: file,
        offset:file.byteLength,
        session_id: sessionId,
      }
      const commitInfo: DropboxTypes.files.CommitInfo = {
        contents: file,
        path: path,
        autorename: true,
      }

      const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg= {
        contents : file,
        cursor: cursor,
        commit: commitInfo,
      }
      // Workaround to fix issues explained here : https://www.dropboxforum.com/t5/API-Support-Feedback/Uploading-multiple-files-with-UploadSession-with-TypeScript/m-p/399797/thread-id/21878#M21888
      delete finishSessionArg.cursor.contents
      delete finishSessionArg.commit.contents
      delete finishSessionArg.contents

      return finishSessionArg
    } else { //If bigger than 8Mb divide the file in multiple 4Mb chunks
      const fileChunks: Array<ArrayBuffer> = []
      let offset: number = 0

      while (offset < file.byteLength) {
        var chunkSize = Math.min(CHUNK_SIZE, file.byteLength - offset)
        fileChunks.push(file.slice(offset, offset + chunkSize))
        offset += chunkSize
      }

      let sessionId: string = ''
      const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg= {
        cursor: { contents: {}, session_id: sessionId, offset: 0, },
        commit: { contents: {}, path: path, },
        contents: {},
      }
      for (const [index,chunk,] of fileChunks.entries()) {
        if (0 === index) {
          const uploadSessionStartArg : DropboxTypes.files.UploadSessionStartArg = {
            contents: chunk,
            close: false,
          }
          const response = await this.dropBox.filesUploadSessionStart(uploadSessionStartArg)
          sessionId = response.session_id
        } else if (index < fileChunks.length-1) {
          const cursor: DropboxTypes.files.UploadSessionCursor = {
            contents: chunk,
            offset: index * CHUNK_SIZE,
            session_id: sessionId,
          }
          const uploadSessionAppendArg : DropboxTypes.files.UploadSessionAppendArg = {
            cursor: cursor,
            contents: chunk,
            close:false,
          }
          delete cursor.contents

          await this.dropBox.filesUploadSessionAppendV2(uploadSessionAppendArg)
        } else {
          const cursor: DropboxTypes.files.UploadSessionCursor = {
            contents: chunk,
            offset: index * CHUNK_SIZE,
            session_id: sessionId,
          }
          const commitInfo: DropboxTypes.files.CommitInfo = {
            contents: chunk,
            path: path,
            autorename: true,
          }
          finishSessionArg.commit = commitInfo
          finishSessionArg.cursor = cursor
          finishSessionArg.contents = chunk
        }
      }

      // Workaround to fix issues explained here : https://www.dropboxforum.com/t5/API-Support-Feedback/Uploading-multiple-files-with-UploadSession-with-TypeScript/m-p/399797/thread-id/21878#M21888
      delete finishSessionArg.cursor.contents
      delete finishSessionArg.commit.contents
      delete finishSessionArg.contents

      return finishSessionArg
    }
  }

So to explained what is wrong when I use that code, it tries to upload every file properly, each chunks seems to be uplad with the appendV2 method. But once it is done all the small files are properly uploaded but the large ones are getting this status : 

{".tag": "failure", "failure": {".tag": "lookup_failed", "lookup_failed": {".tag": "not_closed"}}}

Do you see any reason for this to happen ? 

Thank you for your help.

  • When using filesUploadSessionFinishBatch, "UploadSessionStartArg.close or UploadSessionAppendArg.close needs to be true for the last upload_session/start or upload_session/append_v2 call". That is to say, since filesUploadSessionFinishBatch can't itself take any further file data, you need to make sure you've already finished sending all of the file data for each upload session, and have indicated that to the Dropbox API on the previous call for that upload session, by setting 'close' to 'true'.

     

    The 'not_closed' error you're getting is indicating that you didn't close the upload session before calling filesUploadSessionFinishBatch. So, you'll need to update your code to set 'close:true' on UploadSessionAppendArg for the the last call to filesUploadSessionAppendV2 for each upload session that you're then sending to filesUploadSessionFinishBatch.

  • Greg-DB's avatar
    Greg-DB
    Icon for Dropbox Staff rankDropbox Staff

    When using filesUploadSessionFinishBatch, "UploadSessionStartArg.close or UploadSessionAppendArg.close needs to be true for the last upload_session/start or upload_session/append_v2 call". That is to say, since filesUploadSessionFinishBatch can't itself take any further file data, you need to make sure you've already finished sending all of the file data for each upload session, and have indicated that to the Dropbox API on the previous call for that upload session, by setting 'close' to 'true'.

     

    The 'not_closed' error you're getting is indicating that you didn't close the upload session before calling filesUploadSessionFinishBatch. So, you'll need to update your code to set 'close:true' on UploadSessionAppendArg for the the last call to filesUploadSessionAppendV2 for each upload session that you're then sending to filesUploadSessionFinishBatch.

    • DavidM27's avatar
      DavidM27
      Explorer | Level 4

      Greg, thank you for your reply !

      So if I understand properly, correct me if I'm wrong, in case of uplaoding a big files within a batch session, I should not stop the AppendArgV2() at the chunkList-1 but until all the chunk are uploaded and then I build my finishArg. But in the other case I should stick with what I've done.

      What if I always append all chunks and close on the last chunk ? Does it sound possible to you ?

      • DavidM27's avatar
        DavidM27
        Explorer | Level 4

        I'm answering my own question, it seems to work when I always close the fileUploadSession with the appendArg on the last chunk event when I'm not using the batch.

        Thank you for your help ! If you have any comments or suggestion on the code I sent you above please let me know I would be glad to have your feedback on it.