Mango query, but with keys=[]

[RussellMcOrmond]

We have many scenarios where we need to be able to pass in a set of keys, something we can do with a /_view/ . We would also like to use other features of the Mango interface, such as only returning specific fields rather than the all-or-nothing include_docs

Is there any method to combine these? When I do an /_explain/ I can see the "mrargs", but don't see any method to pass these in and give Mango some additional hints.

Related: Are there plans to allow the Mango interface to be used against a /_view/ ? Merging some of the features of views and indexes would enhance functionality considerably.

Thank you in advance for any thoughts.

[wohali]

I must be not understanding -- why can't you use a big $and array of keys in your selector?

Mango is using the exact same mrview file format on the backend, but the functionality is intentionally limited to be only what is possible with declarative statements. This provides predictable performance out of Mango, which can't be guaranteed with an open-ended system like JS views.

So, you're not going to see future Mango functionality that supports if/then/else clauses, custom reduces, etc. But future Mango functionality might be more flexible, and CouchDB is always open to future Mango enhancement.

[SaschaAdler]

Hi there, I'm a colleague of Russell's.

We're essentially running into this problem: #1853

We'd like to query a view, which can either be represented in JavaScript as

{
  map: function(doc) {
    if ("slug" in doc) { emit(doc.slug, null); }
  }
}

or in Mango as

{
   "index": {
      "fields": [
         "slug"
      ]
   }
}

We'd like to query multiple slugs at once, and then filter the fields returned by the response, since many results have a large array field we can ignore for the purposes of this lookup that we'd rather not have to transmit through HTTP. We can perform the lookup effectively with the JavaScript view by sending our list of slugs as the keys query parameter, but we only have a binary choice between returning all of the found document, or none of it. On the other hand, the Mango query can specify which fields to return in its results, but it can't perform the lookup well. (This code seems to understand that this would be theoretically possible, but sadly it's not implemented yet, and since I've never written a line of Erlang I doubt I could help out.) We could fashion an $or array to write this query, but not an $and one.

For now we'll presumably have to deal with the performance hit of sending lots of large local HTTP requests. We're comfortable ignoring Mango completely and working with JavaScript views. What I find curious is that the very useful fields filtering that Mango querying offers isn't available when either retrieving a document with GET /db/doc or as a parameter to pass to a view query along with include_docs=true. It might be worth extending Couch's basic functionality to handle that use case, instead of figuring out how to predict whether or not it would be safe to pass keys into mrview. (Again, really, I wish I had time to learn Erlang to help with that.)

[SaschaAdler]

Given documents that look like:

{
  "_id": "some-opaque-identifier",
  "slug": "some-human-readable-identifier",
  "label": "The name of the thing",
  "members": [
    // a really long list of things
  ]
}

when we know we don't need the members list in the responses to our query, with a Mango request we can specify

{
  "selector": {
    "slug": {
      "$in": ["some-human-readable-identifier", "some-other-one"]
    }
  },
  "fields": ["_id", "slug", "label"]
}

There's no analog to fields when doing this lookup with a JavaScript view, as far as we can tell, and so any JS view query where we want to retrieve certain fields from the result's documents requires us to request the entire document, including the large members array that we don't need to retrieve at this point.

[wohali]

So yes, that looks like what I'd expect in Mango. How is this unsuitable at present? Is there an enhancement you would prefer?

Speaking unofficially, from what I see in development, it is much more likely that Mango-based queries will be supported going forward, vs. enhancements to JS-based views. I would get comfortable with doing queries this way. In other words, do not expect fields to show up in the /_view interface anytime soon.

[SaschaAdler]

The problem with that Mango lookup is that it has to do a full scan of the index we've set up on slug, and so with a database with thousands of documents that lookup takes multiple seconds, The analogous lookup with JS views doesn't scan the full index and runs in only a few milliseconds. We could send individual Mango queries for each slug, but we've been able to overwhelm the Couch HTTP endpoint with requests before, and we'd rather not risk doing so when we're looking up hundreds of slugs.

A Mango query enhancement would involve determining whether or not the query can be expressed in mrview using keys instead of startkey and endkey, or collapsing $or and $in selectors in ways that make them a union of indexable searches.

[wohali]

This sounds like a good enhancement request, I'll try and make sure the appropriate people see it.

[janl]

The JS-View version of Mango’s fields is not using null as the second argument to emit:

{
  map: function(doc) {
    if ("slug" in doc) { emit(doc.slug, { slug: doc.slug, label: doc.label }); }
  }
}

Also, I’m not sure why a list of (admittedly unwieldy) $or operations on the same field slug would not result in a fully indexed Mango query?