Custom search issues

[bkanuka]

First part of this I figured out on my own. In lib/searc/index.js line 55 needs to be:

    if(!options || !options.command) {

because options is config.search and so options.search is undefined and you never get to use a custom command!

Second part I can't figure out. I have a dummy search program that prints "./output.pdf" to console rigth now but I can't get it to work. Error is: Path must be a string. Received [ '/output.pdf' ]. Full output below

Sorry if the line numbers don't up - I added some debug lines on my own trying to figure this out. I'm new to node.js so...thanks for being helpful and patient with me!

  explorer:search Data [ '/output.pdf' ] +28ms
  explorer:tree Tree for path /output.pdf and options { search: '', sort: '', order: 'asc', page: 1, root: '/nas', path: '/nas', parent: '/nas', buildUrl: [Function: buildUrl], extend: [Function: extend], urlOptions: { limit: undefined, order: 'asc'
, sort: undefined, page: 1 }, hooks: { upload: { menu: [Function: menu] }, archive: { action: [Function: action] }, move: { action: [Function: action], directory: [Function: directory] } }, notifications: { num: 0, move: [], archive: [], upload: [] 
}, user: { username: 'admin', home: '/nas', key: 'ic1XNtmjNPWyiWFQKfS0h7B5j', admin: 1, readonly: 0, ignore: '', trash: '', archive: '', upload: '', getCookie: [Function: getCookie], sanitize: [Function: sanitize], isValid: [Function: isValid], valu
eToIntegerBool: [Function: valueToIntegerBool], crypt: [Function: crypt], generateKey: [Function: generateKey], update: [Function: update], toString: [Function: toString] }, upload: { concurrency: 10, maxSize: '100mb', maxCount: 10, path: '/root/.co
nfig/explorer/upload' }, messages: { info: [], error: [] }, app_root: '/', cacheTTL: 86400, cache: { time: MemoryCache { namespace: 'tree:time' }, size: MemoryCache { namespace: 'tree:size' } }, concurrency: 100, maxDepth: 10, limit: 100, remove: { 
method: 'mv', path: '/root/.config/explorer/trash' }, archive: { path: '/root/.config/explorer/tmp' }, filters: [ [Function: noDotFiles] ] } +9ms
  explorer:tree paths function path:/output.pdf +1ms
  explorer:tree path: /output.pdf +3ms
  explorer:tree f: /output.pdf +1ms
TypeError: Path must be a string. Received [ '/output.pdf' ]
    at assertPath (path.js:28:11)
    at Object.join (path.js:1246:7)
    at /opt/explorer/lib/tree.js:269:15
    at tryCatcher (/opt/explorer/node_modules/bluebird/js/main/util.js:26:23)
    at MappingPromiseArray._promiseFulfilled (/opt/explorer/node_modules/bluebird/js/main/map.js:56:38)
    at MappingPromiseArray.init (/opt/explorer/node_modules/bluebird/js/main/promise_array.js:92:18)
    at MappingPromiseArray.init (/opt/explorer/node_modules/bluebird/js/main/map.js:29:23)
    at Async._drainQueue (/opt/explorer/node_modules/bluebird/js/main/async.js:128:12)
    at Async._drainQueues (/opt/explorer/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues (/opt/explorer/node_modules/bluebird/js/main/async.js:15:14)
    at runCallback (timers.js:789:20)
    at tryOnImmediate (timers.js:751:5)
    at processImmediate [as _immediateCallback] (timers.js:722:5)
2018-01-19 22:29:41 Error: A server error occur, if this happens again please contact the administrator: Path must be a string. Received [ '/output.pdf' ]
    at new HTTPError (/opt/explorer/lib/HTTPError.js:12:17)
    at Spawner.<anonymous> (/opt/explorer/lib/utils.js:182:18)
    at tryCatch1 (/opt/explorer/node_modules/promise-spawner/node_modules/bluebird/js/main/util.js:63:19)
    at Promise$_callHandler [as _callHandler] (/opt/explorer/node_modules/promise-spawner/node_modules/bluebird/js/main/promise.js:695:13)
    at Promise$_settlePromiseFromHandler [as _settlePromiseFromHandler] (/opt/explorer/node_modules/promise-spawner/node_modules/bluebird/js/main/promise.js:711:18)
    at Promise$_settlePromiseAt [as _settlePromiseAt] (/opt/explorer/node_modules/promise-spawner/node_modules/bluebird/js/main/promise.js:868:14)
    at Promise$_settlePromises [as _settlePromises] (/opt/explorer/node_modules/promise-spawner/node_modules/bluebird/js/main/promise.js:1006:14)
    at Promise$_rejectPromises [as _rejectPromises] (/opt/explorer/node_modules/promise-spawner/node_modules/bluebird/js/main/promise.js:999:10)
    at Async$_consumeFunctionBuffer [as _consumeFunctionBuffer] (/opt/explorer/node_modules/promise-spawner/node_modules/bluebird/js/main/async.js:74:12)
    at Async$consumeFunctionBuffer (/opt/explorer/node_modules/promise-spawner/node_modules/bluebird/js/main/async.js:37:14)
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)
GET /search?search=&path=%2Fnas 302 124.843 ms - 88

Side note: I'm working on a docker image that integrates your explorer with recoll search so that I can search inside files. I'll put it up on github soon.

[bkanuka]

oh and lib/search/index.js line 59 needs to be:

return (search, path) => spawner.spawn(options.command.replace('${search}', search), {cwd: path})

[bkanuka]

maybe the second issue could be solved by changing lines in lib/tree.js to:

  var pages = 0
  var concurrency = {concurrency: options.concurrency}
  var num = 0
  var totalSize = 0
  var directorySize = getDirectorySize(options)

  var pathsArray = paths(path, options)

  if(Array.isArray(path)) {
   path = ''
  }

  return pathsArray.map(function(f) {
    var v = p.join(path, f)
    // debug('Map stat', f)

and then search tools need to be modified to return the whole path. This works for me, but might break native search (and some others).

[soyuka]

I almost removed this feature cause felt too dangerous. May you share your custom search? Why do you need one?

[bkanuka]

I'm using recoll as my search tool. Basically I'm using your tool to show a bunch of PDFs, and recoll can search the content of the PDFs. Recoll can search inside tons of documents - it just happens to be mostly PDF in my case.

The external search tool is a major reason I'm using your project. There is a recoll web ui but its gone a little stale - and frankly it's not as pretty.

I have this all as a docker image. Its working but still needs some work. https://github.com/bkanuka/recoll-explorer-docker

[soyuka]

That's a nice tool! Can you give me an example of how:

• do you use this in CLI
• the string you use a custom search
• an example (few lines) of the response given by recoll

Thanks!

[bkanuka]

recoll comes with a cli tool recollq but it's output is a little hard to parse, so I wrote a python script that uses recoll's API. It's only a few lines: https://github.com/bkanuka/recoll-explorer-docker/blob/master/recollsearch

My recollsearch expects the last command line argument to be the path, so i modified your code to replace ${path} with the path:

    return (search, path) => spawner.spawn(options.command.replace('${search}', search).replace('${path}', path), {cwd: path})

Then my custom search command is just:

'recollsearch ${search} "${path}"'

Output is the full path from /. But recollsearch limits results to only files under the directory you are currently in (recursively). Example:

/home/bkanuka/documents/personal/house.pdf
/home/bkanuka/documents/work/workstuff.pdf

That said, I wrote recollsearch so it could really return anything, including partial paths. For what its worth, recoll can also return highlighted document snippets (the area around the search term) but this would require a gui to display it.

[bkanuka]

BTW this is working with the changes I noted above and my PR. You may have a better way though. Let me know what you're thinking!

edit: nope, something not working perfectly. I think it has to do with whitespace. I'm getting an error from explorer: No file entry (file does not exist ?!). Or well...more hacking