Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export benchmark results #8

Merged
merged 46 commits into from
May 30, 2024
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
d18243d
Add skeleton project structure.
atmoos Mar 24, 2024
82c8c4b
Map the summery to GlassView types.
atmoos Mar 25, 2024
afa106a
Add statistics & gc stats to model.
atmoos Mar 25, 2024
e58e9c9
Complete first version of DTO impls.
atmoos Mar 26, 2024
7b8b9d2
separate types out into dedicated files.
atmoos Mar 26, 2024
aacdfd9
Add exporter.
atmoos Mar 26, 2024
7da20dc
First export attempts.
atmoos Mar 27, 2024
8ac0722
Full serialization of benchmark data.
atmoos Mar 27, 2024
2fb9186
complete serialization
atmoos Mar 27, 2024
29462b8
Minor cleanup
atmoos Mar 27, 2024
4532290
Use map functions to build the benchmark summary export.
atmoos Apr 8, 2024
8591bb2
Apply simple changes from PR.
atmoos May 4, 2024
cc85e04
Move serialization to core.
atmoos May 4, 2024
90add55
Add (failing) test for serialization of benchmarks.
atmoos May 4, 2024
fd9a0f0
Have deserialization succeed.
atmoos May 4, 2024
ea6804b
Change all our models to records.
atmoos May 4, 2024
9c6aafd
Have all models be immutable.
atmoos May 4, 2024
2956804
Enable the use of app settings.
atmoos May 10, 2024
c150840
Fix naming
atmoos May 10, 2024
fe62e68
Optimise directory export impl.
atmoos May 10, 2024
e7110e3
Use buildprops everywhere.
atmoos May 16, 2024
1bd261f
Mention issue in mappings.
atmoos May 16, 2024
37c563b
Use Atmoos prefix for root namespace.
atmoos May 16, 2024
2542b27
Fix failing unit tests that fail due to line endings.
atmoos May 16, 2024
6109c28
Add json formatting to configuration options.
atmoos May 16, 2024
4682927
Fix failing unit test.
atmoos May 16, 2024
f9689c7
sanitize path in glass view tests.
atmoos May 16, 2024
9d37142
Add default export & emit log statements.
atmoos May 16, 2024
f684a70
Fix windows issue.
atmoos May 16, 2024
fa62f0d
Ensure benchmark results are exported to sensible directory.
atmoos May 16, 2024
e37433c
Final touches.
atmoos May 16, 2024
e408147
Add initial design document.
atmoos May 16, 2024
efbaaea
Add hardware intrinsics & parameters.
atmoos May 16, 2024
19b42cf
Clean up handling of json formatting options.
atmoos May 17, 2024
72dede4
Incorporate review findings
atmoos May 17, 2024
3d84e95
Have testable exporters & better log output.
atmoos May 17, 2024
bad339d
Test that configured directory extists.
atmoos May 20, 2024
84ff09d
Update README.md
atmoos May 20, 2024
6a7afbc
Clean-up benchmark scenarios.
atmoos May 21, 2024
579a933
Make the benchmark 'categories' obvious.
atmoos May 21, 2024
903c3b1
Remove last remaining assembly attribute.
atmoos May 24, 2024
e779c6e
Move Parameter into dedicated file.
atmoos May 24, 2024
1cc405a
Greatly simplify the benchmark serializer.
atmoos May 28, 2024
7c99197
Improve readability of serialization tests.
atmoos May 28, 2024
6c826e1
Clarify the record comparison situation.
atmoos May 29, 2024
6c15e4f
Improve wording in README.
atmoos May 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions source/GlassView.Benchmark/GlassView.Benchmark.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
atmoos marked this conversation as resolved.
Show resolved Hide resolved
<OutputType>Exe</OutputType>
</PropertyGroup>
<PropertyGroup>
<PlatformTarget>AnyCPU</PlatformTarget>
<DebugType>pdbonly</DebugType>
<DebugSymbols>true</DebugSymbols>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<Optimize>true</Optimize>
<Configuration>Release</Configuration>
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="BenchmarkDotNet" Version="0.13.12" />
<PackageReference Include="BenchmarkDotNet.Diagnostics.Windows" Version="0.13.12" Condition="'$(OS)' == 'Windows_NT'" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\GlassView.Export\GlassView.Export.csproj" />
<ProjectReference Include="..\GlassView.Core\GlassView.Core.csproj" />
</ItemGroup>
</Project>
16 changes: 16 additions & 0 deletions source/GlassView.Benchmark/Program.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
using System.IO;
using System.Threading.Tasks;
using BenchmarkDotNet.Columns;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Running;
using GlassView.Export;

using static BenchmarkDotNet.Columns.StatisticColumn;

IExporter exporter = new Exporter(Directory.CreateDirectory("test"));
// dotnet run -c Release --project Quantities.Benchmark/
var config = DefaultConfig.Instance.HideColumns(StdDev, Median, Kurtosis, BaselineRatioColumn.RatioStdDev);

var summary = BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args, config);

await exporter.Export(summary).ConfigureAwait(ConfigureAwaitOptions.None);
21 changes: 21 additions & 0 deletions source/GlassView.Benchmark/TestBenchmark.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
using System;
using System.Linq;
using BenchmarkDotNet.Attributes;

namespace GlassView.Benchmark;

[ShortRunJob]
[IterationCount(7)]
[MemoryDiagnoser]
public class TestBenchmark
{
private const Int32 count = 8096;

[Benchmark(Baseline = true), BenchmarkCategory("SomeCat")]
public Int32 SumIntegers() => Integers(count).Sum();

[Benchmark, BenchmarkCategory("OtherCat")]
public Double SumDoubles() => Doubles(count).Sum();
private static Double[] Doubles(Int32 count) => Enumerable.Range(0, count).Select(i => i - count / 2d).ToArray();
private static Int32[] Integers(Int32 count) => Enumerable.Range(0, count).Select(i => i - count / 2).ToArray();
}
30 changes: 30 additions & 0 deletions source/GlassView.Core.Test/GlassView.Core.Test.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<RootNamespace>GlassView.Core.Test</RootNamespace>

<IsPackable>false</IsPackable>
<IsTestProject>true</IsTestProject>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.9.0" />
<PackageReference Include="xunit" Version="2.7.0" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.5.7">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="coverlet.collector" Version="6.0.2">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\GlassView.Core\GlassView.Core.csproj" />
</ItemGroup>

</Project>
1 change: 1 addition & 0 deletions source/GlassView.Core.Test/GlobalUsings.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
global using Xunit;
10 changes: 10 additions & 0 deletions source/GlassView.Core.Test/UnitTest1.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
namespace GlassView.Core.Test;

public class UnitTest1
atmoos marked this conversation as resolved.
Show resolved Hide resolved
{
[Fact]
public void Test1()
{

}
}
10 changes: 10 additions & 0 deletions source/GlassView.Core/GlassView.Core.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<RootNamespace>GlassView.Core</RootNamespace>
</PropertyGroup>

</Project>
9 changes: 9 additions & 0 deletions source/GlassView.Core/ICountable.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
using System.Collections;

namespace GlassView.Core;

public interface ICountable<T> : IEnumerable<T>
sereneowl marked this conversation as resolved.
Show resolved Hide resolved
{
Int32 Count { get; }
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}
7 changes: 7 additions & 0 deletions source/GlassView.Core/ICreate.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
namespace GlassView.Core;

public interface ICreate<out TSelf, in TParameter>
atmoos marked this conversation as resolved.
Show resolved Hide resolved
where TSelf : ICreate<TSelf, TParameter>
{
public abstract static TSelf Create(TParameter argument);
}
6 changes: 6 additions & 0 deletions source/GlassView.Core/IName.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
namespace GlassView.Core;

public interface IName
{
String Name { get; }
}
9 changes: 9 additions & 0 deletions source/GlassView.Core/Models/AllocationInfo.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
namespace GlassView.Core.Models;

public sealed class AllocationInfo
sereneowl marked this conversation as resolved.
Show resolved Hide resolved
{
public required Int32 Gen0Collections { get; set; }
atmoos marked this conversation as resolved.
Show resolved Hide resolved
public required Int32 Gen1Collections { get; set; }
public required Int32 Gen2Collections { get; set; }
public required Int64 AllocatedBytes { get; set; }
}
10 changes: 10 additions & 0 deletions source/GlassView.Core/Models/BenchmarkCase.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
namespace GlassView.Core.Models;

public sealed class BenchmarkCase : IName
{
public required String Name { get; set; }
public required Boolean IsBaseline { get; set; }
atmoos marked this conversation as resolved.
Show resolved Hide resolved
public required String[] Categories { get; set; }
public required StatisticsInfo Statistics { get; set; }
public required AllocationInfo Allocation { get; set; }
}
12 changes: 12 additions & 0 deletions source/GlassView.Core/Models/BenchmarkSummary.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
namespace GlassView.Core.Models;

public sealed class BenchmarkSummary(IEnumerable<BenchmarkCase> cases) : IName, ICountable<BenchmarkCase>
sereneowl marked this conversation as resolved.
Show resolved Hide resolved
{
public required String Name { get; set; }
public required Int32 Count { get; set; }
public required String Namespace { get; set; }
public required DateTime Timestamp { get; set; }
public required TimeSpan Duration { get; set; }
public required EnvironmentInfo Environment { get; set; }
public IEnumerator<BenchmarkCase> GetEnumerator() => cases.GetEnumerator();
}
9 changes: 9 additions & 0 deletions source/GlassView.Core/Models/DotnetInfo.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
namespace GlassView.Core.Models;

public sealed class DotnetInfo
{
public required Boolean HasRyuJit { get; set; }
public required String BuildConfig { get; set; }
public required String DotNetVersion { get; set; }
public required Boolean HasAttachedDebugger { get; set; }
}
8 changes: 8 additions & 0 deletions source/GlassView.Core/Models/EnvironmentInfo.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
namespace GlassView.Core.Models;

public sealed class EnvironmentInfo
{
public required String OsVersion { get; set; }
public required DotnetInfo Dotnet { get; set; }
public required ProcessorInfo Processor { get; set; }
}
11 changes: 11 additions & 0 deletions source/GlassView.Core/Models/ProcessorInfo.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
namespace GlassView.Core.Models;

public sealed class ProcessorInfo : IName
{
public required String Name { get; set; }
public Int32 Count { get; set; }
public required String Architecture { get; set; }
public required String HardwareIntrinsics { get; set; }
public Int32 PhysicalCoreCount { get; set; }
public Int32 LogicalCoreCount { get; set; }
}
10 changes: 10 additions & 0 deletions source/GlassView.Core/Models/StatisticsInfo.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
namespace GlassView.Core.Models;

public sealed class StatisticsInfo
{
public required Double Mean { get; set; }
public required Double Median { get; set; }
public Double StandardError => StandardDeviation / Math.Sqrt(SampleSize);
public required Double StandardDeviation { get; set; }
public required Int32 SampleSize { get; set; }
}
9 changes: 9 additions & 0 deletions source/GlassView.Core/readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# GlassView Core

Contains abstract business logic for GlassView.
sereneowl marked this conversation as resolved.
Show resolved Hide resolved

- interfaces
- models
- tooling

Ideally, there is no dependency to [Benchmark.Net](https://benchmarkdotnet.org/).
13 changes: 13 additions & 0 deletions source/GlassView.Export.Test/ExportTest.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
using BenchmarkDotNet.Reports;

namespace GlassView.Export.Test;

public class ExportTest
{
private const String report = @"Quantities.Benchmark.DeserializationBenchmark-report.json";
atmoos marked this conversation as resolved.
Show resolved Hide resolved
[Fact]
public void Test1()
{

}
}
31 changes: 31 additions & 0 deletions source/GlassView.Export.Test/GlassView.Export.Test.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<RootNamespace>GlassView.Export.Test</RootNamespace>

<IsPackable>false</IsPackable>
<IsTestProject>true</IsTestProject>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.9.0" />
<PackageReference Include="xunit" Version="2.7.0" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.5.7">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="coverlet.collector" Version="6.0.2">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\GlassView.Core\GlassView.Core.csproj" />
<ProjectReference Include="..\GlassView.Export\GlassView.Export.csproj" />
</ItemGroup>

</Project>
1 change: 1 addition & 0 deletions source/GlassView.Export.Test/GlobalUsings.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
global using Xunit;
Loading