JsonMappingException: Could not read test result file allure-results

[EvgenyMarchuk]

Describe the Bug

Getting JsonMappingException using allure serve when parameters value is more than maximum allowed. But report is generated

Steps to Reproduce

1. Try to get json file with params value more than maximum allowed(20000000)
2. open cmd
3. run allure serve

Expected Behaviour

There is no error

Screenshots or Additional Context

Could not read test result file allure-results\065735e26ab94a27b7912691bf93fc82-result.json
com.fasterxml.jackson.databind.JsonMappingException: String value length (20051112) exceeds the maximum allowed (20000000, from `StreamReadConstraints.getMaxStringLength()`) (through reference chain: io.qameta.allure.model.TestResult["parameters"]->java.util.ArrayList[0]->io.qameta.allure.model.Parameter["value"])

What Language are you using?

C#

What Framework/Allure Integration you are using?

nunit 4

What version of Allure Integration you are using?

2.12

What version of Allure Report you are using?

2.27

Code of Conduct

• I agree to follow this project's Code of Conduct

[EvgenyMarchuk]

I found the same defect opened in last year. IDK why allure(integration) stores so much information(

[delatrie]

Hi, @EvgenyMarchuk !

If a test has complicated parameters (objects with lots of transitive references, large collections, ORM entities. etc), there might be issues like that.
Could you please narrow the issue to a single test and describe its parameters' types? That will help us develop better constraints to impose on our conversion algorithm. That will also help me to come up with a workaround for you.

[TonEnfer]

Hi @delatrie
I get a similar error when passing Mock objects (based on the Moq library - https://github.com/devlooped/moq) for my services or repositories as parameters in my test.
Serialization of such parameters leads to the fact that the test report (json) can be more than 700 MB in size, and the resulting html in single-file mode takes up more than 800 MB and does not open in the browser.

[delatrie]

@TonEnfer
Could you please show how you parametrize your tests (just one example, with personal data stripped)?

[TonEnfer]

@delatrie
It's quite difficult for me to prepare an example so that it does not contain personal data, but later I will try to prepare an example with which it is possible to reproduce the problem

[delatrie]

Thank you. It will help me better understand some edgy use cases when our serialization algorithm performs poorly.

At the moment, you may use the type formatters API to workaround the issue:

using Allure.Net.Commons;

class ToStringFormatter<T> : TypeFormatter<T>
{
    public string Format(T? value) => value?.ToString() ?? "null";
}

using Allure.Net.Commons;
using NUnit.Framework;
using Moq;

[SetUpFixture]
class AllureSetup // Should be in the root namespace of the test assembly or without a namespace at all
{
    [OneTimeSetUp]
    public void SetupAllure()
    {
        AllureLifecycle.Instance.AddTypeFormatter(new ToStringFormatter<Mock<IMyRepository>>());
        /* Formatters for other mocks and types that cause troubles */
    }
}

[TonEnfer]

@delatrie
Here is the minimal code with which I can reproduce the problem.
Running this test produces a report of 138 MB in size. The parameter string is very (VERY!) long. Unfortunately, my computer is not powerful enough to analyze it (or I’m using the wrong tools, but VSCode and Notepad++ can’t handle it)
If the Setup call is removed from the body of the Test method, the report file takes on an acceptable size.
Calls to the Bar and VerifyAll methods do not affect , but are left for the overall picture
.csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Allure.Xunit" Version="2.12.0" />
    <PackageReference Include="AutoFixture" Version="4.18.1" />
    <PackageReference Include="AutoFixture.AutoMoq" Version="4.18.1" />
    <PackageReference Include="AutoFixture.Xunit2" Version="4.18.1" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.9.0" />
    <PackageReference Include="Moq" Version="4.20.70" />
    <PackageReference Include="xunit" Version="2.7.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.5.7">
        <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
        <PrivateAssets>all</PrivateAssets>
    </PackageReference>
  </ItemGroup>
</Project>

Class1.cs;

using AutoFixture;
using AutoFixture.AutoMoq;
using AutoFixture.Xunit2;
using Moq;
using Xunit;

namespace Allure482;

public class Class1
{
	[
		Theory,
		AutoMockData
	]
	public void Test(
		[Frozen] Mock<IFoo> testInterface
	)
	{
		testInterface.Setup(x => x.Bar());

		testInterface.Object.Bar();

		testInterface.VerifyAll();
	}
}

public interface IFoo
{
	void Bar();
}

public class AutoMockDataAttribute(): AutoDataAttribute(() => new Fixture().Customize(new AutoMoqCustomization()));

allure-results.zip

[TonEnfer]

A much easier way to reproduce the problem:

    public static IEnumerable<object[]> GetMembers => [[() => { }]];

    [Theory, MemberData(nameof(GetMembers))]
    public void Test(Action checkResultAction)
    {
        checkResultAction();
    }

This code produces a JSON report slightly larger than 12.5 MB.
This does not cause the exception described in the Issue, but it does a good job of showing what is causing the problem.

It seems that Allure should include formatters for at least Func and Action by default.
However this will not solve the problem when Func or Action is a property on an object that is serialized to JSON so it seems that a JSON converter for these types is required

To resolve the issue, you can specify the following ContractResolver as the ContractResolver in the SerializerSettings of the FormatFunctions class:

    private class NewtonsoftContractResolver: DefaultContractResolver
    {
        protected override JsonContract CreateContract(Type objectType)
        {
            if (typeof(Delegate).IsAssignableFrom(objectType))
            {
                return base.CreateStringContract(objectType);
            }

            return base.CreateContract(objectType);
        }
    }

This solution will serialize all delegates as a string.

P.S.
There is one more nuance that I would like to mention, but which is not directly related to this problem - using the current version of Allure significantly slows down the execution of tests - in my CI/CD the test execution time (about 1000 different tests) increased by about 1.5 times. I didn't investigate the problem because I don't have time for it, but I can assume that this may be due to JSON serialization in general and the not very high performance of Newtonsoft.Json in particular.

P.S.S.
I was very excited when I first saw Allure, and extremely disappointed when I first tried it.
I am very upset by allure-csharp's long release cycle, as well as the lack of testing, which leads to new bugs after each release.
Because of these problems, I have not been able to use Allure for many months.

[EvgenyMarchuk]

Hello! When is it going to be fixed?
Thanks!

[delatrie]

The problem is that given

[Test]
public void Test(T a) { /* ... */ }

we can't decide whether to include a in the report or not for all possible values of T. The same goes for nested properties/fields of type T. Sometimes, people put unpredictably complicated data in test parameters. While I personally see it more as an antipattern (Allure aside, that makes reporting of any sort much more complex), we still need a solution that covers all cases.

We can do some heuristics to exclude some types that obviously don't help us in the report and only blow up its size. As suggested by @TonEnfer, we can safely exclude any delegate types (including, but not limited to, Action, Func<T>, and variants with args). If someone wants to contribute on that matter, I will gladly accept those PRs.

However, heuristics only partially solves the problem. From my perspective, we need the following (in addition to heuristics):

• Limit parameter values displayed in the report (a digest of the original value could be added as a new hidden attribute to help Allure put apart test results). That will help reduce the size but doesn't fix performance issues.
• Provide flexible opt-in/opt-out mechanics for tests, fixtures, and parameters that can be tuned via the configuration/API to give users more control over what they want to see in the report. That needs to be designed appropriately as, to the best of my knowledge, we don't have that in other adapters either.
• Allure event listeners for scenarios that can't be implemented with the above mechanisms. That's something we already have in allure-java and will eventually be implemented here as well.

Sorry for not giving you any specific dates. I want you to please be sure that this issue will be my top priority as soon as I have some time for allure-csharp.

[rohitvipin]

Xunit:

[ModuleInitializer]
public static void ConfigureAllure()
    => AllureLifecycle.Instance.AddTypeFormatter(new CompactStringTypeFormatter<CreateRequest>());

Custom logic to reduce size

public class CompactStringTypeFormatter<T> : TypeFormatter<T>
{
    public override string Format(T value)
    {
        var type = typeof(T);

        Debug.WriteLine(JsonConvert.SerializeObject(value));

        var values = type.GetProperties()
            .Where(x => x.CanRead && (!x.PropertyType.IsClass || x.PropertyType.IsPrimitive || x.PropertyType == typeof(string)))          
            .Select(property => new { property.Name, Value = property.GetValue(value) })
            .Where(x => x.Value != null)
            .Take(5)
            .Select(x => $"{x.Name}:{x.Value}");

        return string.Join(',', values);
    }
}