We are aware of the issue with the badge emails resending to everyone, we apologise for the inconvenience - learn more here.

Forum Discussion

not_oppenheimer's avatar
not_oppenheimer
New member | Level 2
6 years ago

Dropbox API search vs. recursive list_folder with filtered results

I'm trying to utilize the API to scan my directories for .XLS files and read them. I've attempted this in two ways:

using '/search'

 

url = "https://api.dropboxapi.com/2/files/search"
data = {
    "path": "/",
    "query": ".XLS",
    "start": 0,
    "max_results": 1000,
    "mode": {".tag":"filename"}
}
response = requests.post(url, headers=headers, data=json.dumps(data))
matches, more, start = response.json().values()
paths = [x['metadata']['path_display'] for x in matches]
while response.json()['more']:
    data["start"] = start
    response = requests.post(url, headers=headers, data=json.dumps(data))
    matches, more, start = response.json().values()
    paths.extend([x['metadata']['path_display'] for x in matches])
else:
    return set(filter(lambda x: '.XLS' in x, paths))

and using '/list_folder' and '/list_folder/continue'

data = {"path": "/", "recursive": True}
url = "https://api.dropboxapi.com/2/files/list_folder"
response = requests.post(url, headers=headers, data=json.dumps(data))
entries, cursor, has_more = response.json().values()

paths = [x['path_display'] for x in filter(lambda x: '.XLS' in x['path_display'], entries)]
data  = {
    "cursor": cursor,
}
while has_more:
    response = requests.post(url+'/continue', headers=headers, data=json.dumps(data))
    print(path, len(paths))
    entries, data['cursor'], has_more = response.json().values()
    paths.extend([x['path_display'] for x in filter(lambda x: '.XLS' in x['path_display'], entries)])
else:
    return set(paths)

For some reason that I can't discern based on diffs of the results, the search option is not exhaustive. The majority of the files it's missing are "recent" files in that they were inserted into the directory (or a subfolder) within the last six months, but that doesn't hold for all of the files and doesn't seem to be a known issue with the API. I can much more easily parallelize '/search' and would prefer to use it, but I need to know that it is exhaustive. 

 

  • The results returned by the /2/files/search endpoint are not technically exhaustive. For search queries that have a very large number results, all of them may not be returned. Specifically, there is a max value limit of 9,999 for the 'start' parameter, so if there are more than 10,000 matches, you won't be able to retrieve everything.

    For use cases where you have that many entries to retrieve, please use /2/files/list_folder[/continue] instead.

    If that doesn't seem to be the issue here though, please open an API ticket with details on the missing search results so we can look into it for you.

  • Greg-DB's avatar
    Greg-DB
    Icon for Dropbox Staff rankDropbox Staff

    The results returned by the /2/files/search endpoint are not technically exhaustive. For search queries that have a very large number results, all of them may not be returned. Specifically, there is a max value limit of 9,999 for the 'start' parameter, so if there are more than 10,000 matches, you won't be able to retrieve everything.

    For use cases where you have that many entries to retrieve, please use /2/files/list_folder[/continue] instead.

    If that doesn't seem to be the issue here though, please open an API ticket with details on the missing search results so we can look into it for you.

    • not_oppenheimer's avatar
      not_oppenheimer
      New member | Level 2

      Thanks for your response. You're absolutely correct that I'm butting up against the 9,999 maximum. 

      Looking through the documentation there seems to be no built-in method to circumvent or increase that limit. I'll have to be more clever on my end it seem. 

About Dropbox API Support & Feedback

Node avatar for Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

5,877 PostsLatest Activity: 12 months ago
325 Following

If you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X or Facebook.

For more info on available support options for your Dropbox plan, see this article.

If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!