New output types

[dwysocki]

We've discussed over both Gmail chat and GitHub issues, a number of new output types we need to support. I've decided to make a checklist here. Feel free to edit this post by either (un)checking items, or adding new ones. I've separated them into tabular and plot outputs.

• Table
  • light curve (2576355)
    • remove the phased light curve columns from the stdout table, and instead output them as photometry tables: (phase, mag, err)
  • residual
    • output a table of residuals for every input data point, including outliers, with an extra column of T/F to flag inliers/outliers: (time, mag, inlier?)
    • partially finished, currently outputs phase instead of time (blocked by Remove unnecessary phasing of data #77)
  • periodogram (6e8cbc8)
    • output a table describing the periodogram: (frequency or period, power, p-value)
    • choose between frequency or period with the --output-periodogram-form={frequency,period} option
  • fourier coeffs (6819788)
    • output a table listing the Fourier fit coefficients, in place of those written to stdout
    • the formatting will be made with multi-periodic fits in mind, and for now will take the form: (k, A, A_err, Phi, Phi_err)
      • with multi-periodic fitting in place, output will take the form: (k_1, ..., k_n, A, A_err, Phi, Phi_err)
  • fourier ratios (6819788)
    • output a table listing the Fourier ratios (e.g. R21 and Phi21), in place of those written to stdout
    • note: this will be tricky when we do multiple periods, so we may want to disable it in that case, at least initially
• Plot
  • light curve
    • output a plot of the light curve, as we've been doing
    • Engines:
      • mpl
      • TikZ
  • residual
    • output a plot of the residuals between the fitted light curve and the input data with a dotted line at 0
    • Engines:
      • mpl (a37e8a3)
      • TikZ
  • periodogram
    • output a plot of the periodogram: (frequency, power) with a dotted line indicating the significance threshold
    • Engines:
      • mpl (a37e8a3)
      • TikZ

[dwysocki]

I've added the architecture for outputting residuals, so you can make the actual plotting code now, Earl.

[dwysocki]

Now that I think about it, what exactly are we going to plot for the periodogram? We actually do two periodograms, one with coarse and one with fine precision, so which do we plot? Both? Do we just insert the fine periodogram into the coarse one?

[earlbellinger]

Yes, we can insert them. A periodogram (ordinarily) has frequencies or periods on the abscissa and an estimate of the spectral density on the ordinate. So basically we can plot (freqs, pgram) from here. That means we'll probably want to start returning the entire periodogram instead of just the max value. But it's no problem to have a more dense region of frequencies in some places of the plot.

[dwysocki]

Just wanted to add to this comment chain what Earl observed recently.

The periodogram text files can end up being enormous, Earl had one reach ~50MB in size. How do we deal with this? This makes --output-all a bit more complicated, as we don't want to accidentally make somebody run out of disc space.

[dwysocki]

In order to properly implement the residual table, we will need to have access to the times associated with the magnitudes. As we currently do pointless phasing, this is nigh impossible, so we need to resolve issue #77. This might be a painful process, as a lot of things are coupled to phased data >.>