Consider revising the SESANS data format read by SasView (Trac #743)

[sasview-bot]

After discussion with some of you, especially Joachim, I’m getting more convinced that we should redefine the input format for SESANS measurements in !SasView. SANS the users provide their data in absolute units to the data-analysis software. With SESANS we can do the same thing by using ln(P(z))/t*(lambda2) as a function of the spin-echo length.

There are several advantages of using the SESANS in absolute units in the data format:

· This way we can already directly compare data measured at different instruments with different thickness samples.

· In the software we don’t have to worry about monochromatic of TOF measurements.

· In the software only two columns are needed P and z (in analogy with SANS I and q). This makes it much easier to work with the same software. This is something we meet right now with the architecture of the !SasView-code.

· The responsibility of the conversion into absolute units is with the user in the data-reduction, just like with SANS. This avoids any ambiguity in parameter passing in the data-format to !SasView or !SasFit.

· With fitting the contributions of components become linear, which is an enormous advantage if the analytical SESANS equations are available. This will make it possible to reuse many SANS parts of coding with multiple contributions and polydispersity.

· The calculation of the Jacobian in the chi2 calculation is more effective.

This idea is different from the idea we started to follow at the code camp in Feb 2015 in Copenhagen, where we thought it would be best to have the spin-echo length, the wavelength and the polarisation in the data-files.

There are some drawbacks to the format with the SESANS in absolute units.

· You present something further away from the primary data. It makes it more difficult to evaluate by eye the quality of fits. This can be circumvented with some extra work by a back transformation of the fit.

· It could be more tricky to take into account the spread in wavelength for the amount of scattering. However, I have the feeling it won’t be to difficult mathematically to do this in the data reduction before the fitting.

· We would have to change some of the earlier coding in !SasView and !SasFit. However, these changes are not difficult and will make further coding much simpler, more robust and reliable.

Please let me know if you have any strong opinions about the SESANS data format. Untill Tuesday morning we are working on the code with all the experts of !SasView around the table. If you are positive about providing SESANS data in absolute units to the analysis software, then I’ll come with a new proposal for the format based on my discussions with you.

Warm collegial regards,

Wim

Migrated from http://trac.sasview.org/ticket/743

{
    "status": "closed",
    "changetime": "2016-10-12T08:40:53",
    "_ts": "2016-10-12 08:40:53.732281+00:00",
    "description": "After discussion with some of you, especially Joachim, I\u2019m getting more convinced that we should redefine the input format for SESANS measurements in !SasView. SANS the users provide their data in absolute units to the data-analysis software. With SESANS we can do the same thing by using ln(P(z))/t*(lambda2) as a function of the spin-echo length.\n\nThere are several advantages of using the SESANS in absolute units in the data format:\n\n\u00b7        This way we can already directly compare data measured at different instruments with different thickness samples.\n\n\u00b7        In the software we don\u2019t have to worry about monochromatic of TOF measurements.\n\n\u00b7        In the software only two columns are needed P and z (in analogy with SANS I and q). This makes it much easier to work with the same software. This is something we meet right now with the architecture of the !SasView-code.\n\n\u00b7        The responsibility of the conversion into absolute units is with the user in the data-reduction, just like with SANS. This avoids any ambiguity in parameter passing in the data-format to !SasView or !SasFit.\n\n\u00b7        With fitting the contributions of components become linear, which is an enormous advantage if the analytical SESANS equations are available. This will make it possible to reuse many SANS parts of coding with multiple contributions and polydispersity.\n\n\u00b7        The calculation of the Jacobian in the chi2 calculation is more effective.\n\nThis idea is different from the idea we started to follow at the code camp in Feb 2015 in Copenhagen, where we thought it would be best to have the spin-echo length, the wavelength and the polarisation in the data-files.\n\nThere are some drawbacks to the format with the SESANS in absolute units.\n\n\u00b7        You present something further away from the primary data. It makes it more difficult to evaluate by eye the quality of fits. This can be circumvented with some extra work by a back transformation of the fit.\n\n\u00b7        It could be more tricky to take into account the spread in wavelength for the amount of scattering. However, I have the feeling it won\u2019t be to difficult mathematically to do this in the data reduction before the fitting.\n\n\u00b7        We would have to change some of the earlier coding in !SasView and !SasFit. However, these changes are not difficult and will make further coding much simpler, more robust and reliable.\n\nPlease let me know if you have any strong opinions about the SESANS data format. Untill Tuesday morning we are working on the code with all the experts of !SasView around the table. If you are positive about providing SESANS data in absolute units to the analysis software, then I\u2019ll come with a new proposal for the format based on my discussions with you.\n\n\nWarm collegial regards,\n\nWim",
    "reporter": "wimbouwman",
    "cc": "",
    "resolution": "fixed",
    "workpackage": "SESANS Integration",
    "time": "2016-10-09T14:40:23",
    "component": "SasView",
    "summary": "Consider revising the SESANS data format read by SasView",
    "priority": "major",
    "keywords": "",
    "milestone": "SasView 4.1.0",
    "owner": "wimbouwman",
    "type": "enhancement"
}

[smk78]

Trac update at 2016/10/09 14:40:53:

• smk78 changed owner from "" to "wimbouwman"
• smk78 changed status from "new" to "assigned"

[smk78]

Trac update at 2016/10/09 14:41:58: smk78 commented:

I agree with you. If we were talking about data files generated by the experiment, I would want to keep some of the other information such as wavelength etc, but if we are talking about what SASView takes as input, I think having input similar to SANS is a good idea and the choice you suggest is the obvious one (I assume by the way that P(z) is normalized to P0(z)). Presumably it is up to the user to provide the error bars on ln(P)/t*lambda2.

Best

roger

[smk78]

Trac update at 2016/10/09 14:42:56: smk78 commented:

I have not had any TOF SESANS data available. However, I would assume, that they should have in principle for each spin echo length a separate Q_xmax and Q_zmax. In your data from Delft you had only one Qxmax and Qzmax in your format. My question would be here, if one should instead supply Theta_xmax and Thata_zmax. If you than keep the wavelength column one can calculated for each spin echo length the maximum Q value of the setup which will be needed in case for a truncated Hankel transform. The truncated Hankel transform has also the advantage, that it does not diverge for slow decaying form factor like the polymer form factor. When I played around with the data I also tested the effect of the wave length resolution. The effect is of 5% relative resolution normally still neglectable. Instead of removing information I would rather suggest to include the transformation from P(z) to G(z) in the data format. A wavelength resolution is easier to add for P(z). Otherwise one either ignores the wavelength resolution or need to desmear the data before the transformation to G(z). In the end the number of columns do not matter, they just need to be well documented as already the columns in the existing format.

My point for using G(z) in the analysis is mainly its property of being a linear combination of the individual contribution of a multicomponent system, so that size distribution, different species etc can be handled as for SAS analysis. P(z) also has the disadvantage, that in case of wrongs starting values the exponential function of large negative values becomes zero and the fitting routine fails to adjust for example the number density or contrast parameters, when too large starting values were chosen.

You mentioned to me that you like to fit simultaneously SAS and SESANS data. The implementation I made was quick and should rather supply you with a familiar GUI with minimal change. At the Moment I just set a flag for an additional operator, which can be the identity or Hankel operator. However, this flag is not a flag for each scattering contribution but rather for all. But the reason for that is the internal data structure from SASfit which would be needed to be extended, which I thought I could do later.

I would suggest for the data format to be consistent in units, i.e. only nm or Angstrom and their reciprocal units and either always absolute or relative errors which is not the case at the moment. A “G(z)-G0” column would be nice. In principle this can be done by the reading routine for ses-data files, but I think users would like to find the data, which are fitted directly in them.

If you like to include resolution in the analysis, is there something else necessary next to the limitation in Q due to the detector size and the wavelength spread? Does the error in spin echo length already includes the wavelength spread? If somebody already worked on the resolution function I would be interested in a reference to it.

Best

Joachim

[sasview-bot]

Trac update at 2016/10/09 14:58:15: wimbouwman commented:

The ticket has the following subtasks:

• Rewrite documentation on the data format
• Rewrite documentation on SESANS in !SasView
• Change the labels in the data loader for SESANS
• Consider if the wavelength column can be ignored in the dataleader without having consequences for the data-type recognition further in !SasView
• Remove the exponential function and multipication with the wavelength in the Hankel and cosine transformations
• Remove the exponential function and multipication with the wavelength in the direct analytical SESANS sphere model

[sasview-bot]

Trac update at 2016/10/12 08:40:53:

• wimbouwman changed resolution from "" to "fixed"
• wimbouwman changed status from "assigned" to "closed"