Uri too long (retrieve multiple)

[Notaduck]

Hi Again, so I gave v2 a shot and so far I really enjoy to work with the Library! I was wondering if you have a good idea here.
I'm trying to retrieve multiple contacts at once, however the emailList seems to be to long and I get the following error Invalid URI: The Uri string is too long. is there a build in feature to split the list to a more appropriate size? Or do you happen do know how to figure out what the max length is?

  async findManyDynamicsContactByEmail(emails: string[]) {

    const emailList = `(${emails.map((email) => `'${email}'`).join(', ')})`;


    const request: RetrieveMultipleRequest = {
      collection: DynamicsCollections.CONTACT,
      filter: `emailaddress1 in ${emailList}`,
      count: true,
    };

    const response =
      await this.dynamicsWebApi.retrieveMultiple<DynamicsContact>(request);

    return response.value;
  }

[AleksandrRogov]

hi @Notaduck , do you happen to know the full length of the generated URL? Also, is the error coming from Dynamics (if it comes from Dynamics, the error should have a status code and stack trace and etc)? or is it coming from the library?

The library converts long URLs into a batch operation. But I wonder if you are hitting the Dynamics 365 url limitation.

Also, are you using it in the node.js application?

[Notaduck]

Thanks for the reply @AleksandrRogov, I actually dont know how long it is, but is is LOOOONG. I believe there was about 40K email adresses in the filter.

This is the full error coming from a node.js/NestJS application.

Error: Invalid URI: The Uri string is too long.
    at Function.handleHttpError (/Users/daniel/Projects/send-it/node_modules/dynamics-web-api/dist/cjs/dynamics-web-api.js:318:23)
    at IncomingMessage.<anonymous> (/Users/daniel/Projects/send-it/node_modules/dynamics-web-api/dist/cjs/dynamics-web-api.js:762:25)
    at IncomingMessage.emit (node:events:525:35)
    at endReadableNT (node:internal/streams/readable:1358:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

Hmmmm, I just tried another approach

    const request: RetrieveMultipleRequest = {
      collection: DynamicsCollections.CONTACT,
      filter: `emailaddress1 in ('daniel.xxxx@xxxxx.com')`,
      count: true,
    };

Which throws the following error

Error: The method or operation is not implemented.
    at Function.handleHttpError (/Users/daniel/Projects/send-it/node_modules/dynamics-web-api/dist/cjs/dynamics-web-api.js:318:23)
    at IncomingMessage.<anonymous> (/Users/daniel/Projects/send-it/node_modules/dynamics-web-api/dist/cjs/dynamics-web-api.js:762:25)
    at IncomingMessage.emit (node:events:525:35)
    at endReadableNT (node:internal/streams/readable:1358:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

So as a quick fix I have made this as a solution, but I don't nessesary think it is the correct one in regards to performance.

  async findManyDynamicsContactByEmail(emails: string[]) {
    const chunkSize = 50;
    let emailsChunkarr = [];

    for (let i = 0; i < emails.length; i += chunkSize) {
      emailsChunkarr.push(
        `(${emails
          .slice(i, i + chunkSize)
          .map((email) => `emailaddress1 eq '${email}'`)
          .join(' or ')})`,
      );
    }

    let result: Array<DynamicsContact> = [];

    await Promise.all(
      emailsChunkarr.map(async (emails) => {
        const request: RetrieveMultipleRequest = {
          collection: DynamicsCollections.CONTACT,
          filter: emails,
          count: true,
        };

        const response =
          await this.dynamicsWebApi.retrieveMultiple<DynamicsContact>(request);

        result.push(...response.value);
      }),
    );

    return result.filter((e) => Object.keys(e).length > 0);
  }

I also tried to use the batch funcion like this, which resulted in the URI being to long again.

    const fileterQuery = `${emails
      .map((email) => `emailaddress1 eq ${email}`)
      .join(` or `)} `;

    const request: RetrieveMultipleRequest = {
      collection: DynamicsCollections.CONTACT,
      filter: fileterQuery,
      count: true,
    };

    this.dynamicsWebApi.startBatch();

    await this.dynamicsWebApi.retrieveMultiple<DynamicsContact>(request);

    const response = await this.dynamicsWebApi.executeBatch();

    return response[0].value;

[AleksandrRogov]

I will try to reproduce this error. It looks like it is coming from Dynamics, there is probably a limitation on the url length.
Your workaround is correct. Though, I would make a batch operation, instead of executing all the requests in a loop. Like so:

this.dynamicsWebApi.startBatch();

//make sure there are no more than 1000 chunks - it is batch operation limitation in Dynamics
//if that happens you can gradually increase the chunk size
if (emailsChunkarr.lengh > 1000) throw new Error("Too many chunks");

emailsChunkarr.map(async (emails) => {
  const request: RetrieveMultipleRequest = {
    collection: DynamicsCollections.CONTACT,
    filter: emails,
    count: true,
  };

  this.dynamicsWebApi.retrieveMultiple<DynamicsContact>(request);
});

const results = await dynamicsWebApi.executeBatch();

const responses =[];
results.forEach(r => responses.push(r.value));

Concerning "in" operator. You have to use this function instead. Here's an example:

const response = await dynamicsWebApi.retrieveMultipleRequest({
    collection: `accounts`,
    filter: "Microsoft.Dynamics.CRM.In(PropertyName=@p1,PropertyValues=@p2)",
    queryParams: ["@p1='lastname'", '@p2=["Doe", "2nd Doe"]']
});

// the request will be made to the following url:
// https://<server>?$filter=Microsoft.Dynamics.CRM.In(PropertyName=@p1,PropertyValues=@p2)&@p1=\'lastname\'&@p2=["Doe", "2nd Doe"]

[AleksandrRogov]

@Notaduck forgot to tag you in the reply above, not sure if you got the notification.

But I also found this: dotnet/runtime#1857
I did not find a specific issue for .NET Framework, that one is for .NET Core but it looks like the overall length of the URI cannot exceed 65520 characters. So it's not really a Dynamics limitation, it is a limitation of the .NET Framework. And it actually looks like the error is coming from the new URI constructor in .NET. So, you need to keep that in mind.