Demeter/Athena/Artemis To Do List

Table of Contents

1 top priorities

1.1 small things

1.2 big things

  • [ ] From Joel Ullom: In our data, the error bar in each energy bin of the EXAFS spectrum is known from Poisson statistics since the photons are (laboriously) counted one by one. This creates the opportunity to propagate these errors through the Artemis fitting process. Potentially, the resulting error bars on the fit parameters are more meaningful. This software capability would be interesting to explore. This is a Larch chore
  • [X] From Joel Ullom: In the rebinning functionality of Athena, all rebinnings reduced the amplitude of the white line feature in our ferrioxalate data. You explained well why rebinning is seldom useful when using a conventional EXAFS energy grid. However, we only reproduce the conventional energy grid through rebinning so I think rebinning is still relevant for us. It would be nice to be able to rebin without reducing spectral detail in the XANES region. – What is the width of the box?
  • [X] 5 and 6 legged paths
  • [ ] PCA improvements
  • [ ] feff8.5L
  • [X] persistance of analysis methods in Athena This can be implemented in the same way I implemented XDI persistance – for each group, serialize a blessed scalar and write that to the record

2 larch chores

  • [ ] l versions of the A,A,&H need path wrangling on Windows that includes PATH info for larch
  • [X] XDI headers and saving current with larch
  • [X] finsh Athena templates, including recently added ones (e.g. MEE templates, pixel)
  • [ ] start work on Artemis-related templates
  • [ ] kaiser vs. kaiser-bessel
  • [ ] use of stddev array in LCF is slightly incorrect, needs to be sensitive to the sapce in which the merge was made.
  • [ ] command line in command buffer in A&A & capture feedback from server
  • [X] testing of Larch + XDI & of Larch - XDI
  • [ ] empirical standard generation in Athena

2.1 Larch v. Ifeffit problem areas

  • Xray/Absorption/ – this needs to be refactored
  • t/ tests need to be tested with Larch
  • in particular, need a t/001_larch.t test (with an appropriate SKIP in case of no server)
  • “nergy”: XDAC vs. Ifeffit. Larch will see “energy”.
    • Perhaps a larch or ifeffit configuration attribute
    • ack-ing for ’\bnergy\b’ yields:

      54:  #my @energy    = $mc->get_array('nergy');
      53:    my @energy = $mc->get_array('nergy');
      324:becomes "nergy".  Go figure.
  • Need to add Larch copyright statement throughout

    ##             Larch is copyright (c) 2013 Matt Newville, Tom Trainer

3 perl modules to evaluate

  • Path:Tiny to replace :Spec
  • Test::Class and Test::Class::Moose

4 things that need doing

4.1 Moose issues

4.2 wx issues

4.3 installation issues [2/2]

  • [X] Relocation needs testing on Windows, need to make sure that the executables can all be found,
  • [X] also need to be able to find all of Ifeffit CL and CAPS files.

4.4 Demeter bugs, shortcomings, missing features [5/21] [23%]

  • [ ] in Demeter::Data::IO::save_many, why doesn’t updating in n specifying “background” not work to bring data up to date? (fft, i.e. doing autobk, does work)
  • [ ] tie multiple groups together (e.g. MED data with reference)
  • [ ] config param updated AND that param is currently showing in the Prefs tool – need to update display in Prefs tool
  • [ ] add a record to an existing Athena project file
  • [ ] clnorm + Julia and Jingen’s MBACK replacement
  • [ ] Data object should note that datatype is xanes and not compute fbkg since the bkg array does not exist
  • [ ] cannot specify title for stddev and variance plots
  • [ ] Demeter command backend needs lots of polishing
  • [ ] 3rd derivative
  • [ ] SDK: better documentation for freeze/thaw//serialize/deserialize
  • [ ] error check deserialization for all objects, verify that it is a yaml or whatnot, return 0 if not
  • [ ] OpenBabel interface, tied into Feff
  • [ ] MRU lists and unicode, unicode-y file paths fail the -e test at Demeter::MRU line 77
  • [ ] Feff8 integration
  • [ ] Use PDL in pathfinder
  • [ ] background groups for AXAFS
  • [X] Add tests to Path ranking framework (Karine Provost’s idea, implemented in Demeter)
  • [X] serialize VPath, put VPath into dpj and fpj files
  • [X] set E0 to peak of white line
  • [X] need to disable indicators during the Rk plot. Indicators (as implemented) are not consistent with multiplot
  • [X] Plucking from gnuplot really sucks

4.5 Athena

4.5.1 needs testing [/]

4.5.2 Tools [22/25] [88%]

  1. [X] main
  2. [X] calibrate
  3. [X] align
  4. [X] deglitch
  5. [X] smooth with Savitsky-Golay filter Matlab routine for SG filter; smooth with boxcar average See this
  6. [X] truncate
  7. [X] rebin
  8. [X] convolution and noise
  9. [ ] deconvolution
  10. [X] self absorption
  11. [X] copy series
  12. [X] data summation
  13. [ ] dispersive
  14. [X] multi-electron removal
  15. [X] LCF
  16. [X] PCA PCA from PDL mail archives
  17. [X] Peak fit
  18. [X] log ratio
  19. [X] difference
  20. [ ] explain FT
  21. [X] disk watcher
  22. [X] XDI
  23. [X] journal
  24. [X] plugins
  25. [X] preference
  1. Requires PDL
    1. smoothing by S-G (done with Larch)
    2. deconvolution
    3. PCA
    4. boxcar average

4.5.3 needs doing [5/15] [33%]

  • [ ] beter error handling in plugins – specifically need to do a better job of trapping errors in the fix method. Perhaps fix should always be called within an eval?
  • [X] report to a text dialog on values of a particular parameter for all/marked groups
  • [ ] implement megamerge in Athena
  • [X] links to documentation on each tool
  • [X] Main window [3/3] [100%]
    • [X] SpinCtrl for use with Rbkg, event handler triggered twice on first use (found work-around)
    • [X] SpinCtrl for use with bkg_spl1/2 (I find this to be a much worse idea) (not implemented)
    • [X] push bkg_e0 algorithm onto marked/all groups and recalculate
  • [ ] Convolve/noise tool
    • add noise to chi data
  • [ ] Difference tool [0/1]
    • [ ] difference in k
  • [-] LCF tool [1/5]
    • [X] persistance of LCF results – use serialized blessed scalar, like XDI
    • [ ] satisfying both inclusive and unity can be trouble for poorly normalized data and/or poor standard selection (see email from Jack Hitch)
    • [ ] compute R-factor in k or R using fit from energy. (see mail from Scott 7 January 2012)
    • [ ] boundary on E0 shift. See email from Dean Hesterburg from 30 May, 2014. This is a Larch chore.
    • [ ] plot in norm(E) for this fit in chi(k) or plot in chi(k) for this fit in norm(E)
  • [-] PCA tool [3/11]
    • [ ] test setection against analysis space. do the right thing with xanes data and chi(k) analysis or chi(k) data and energy analysis
    • [X] persistance – what’s the value of persistance for PCA?
    • [ ] manual entry of component coefficients (??)
    • [ ] cluster analysis, select two components and plot the associations of each data with those two in a scatter plot.
    • [ ] worry about “def” line being too long in tt template
    • [ ] plotting in pgplot
    • [ ] error analysis, RET, SPOIL, etc, determination of number of components, validity of TT, etc
    • [ ] specialized file headers for reconstruction and tt save files
    • [ ] pluck buttons for analysis range don’t work
    • [X] save eigenvectors to a file
    • [X] if the marked groups are changed after performing the PCA but before reconstructing data, the plot will fail with and exception. need to save the list of data that went into the PCA and check against that rather than the current list of marked groups
  • [-] Peakfit tool [8/12]
    • [X] persistance – use serialized blessed scalar, like XDI
    • [ ] verify that fit is sensible (ndata/ninfo vs nparam)
    • [ ] broken using XES
    • [ ] explicitly state ninfo and epsilon
    • [ ] for Larch, bounds on parameters
    • [X] implement pseudovoight
    • [X] verify that all centroids are within (or near) fit range
    • [X] behavior when changing data
    • [X] pluck fit ranges
    • [X] ndata is full data range rather than fit range with Fityk backend (igoring this)
    • [X] explicitly set title of plot (i can live with this)
    • [X] sequence over marked groups
  • [ ] Data watcher
  • [-] MEE tool [2/3]
    • [X] arctan removal
    • [ ] other algorithms from the literature
    • [X] good guesses for parameters – at least a pluck button for shift
  • [ ] Plot menu
    • zoom/unzoom/cursor for pgplot
  • [X] metadata tool
  • [X] Data summer
    • push_values method needs to update menus to reflect changes in group list

4.6 raw data and plugins

  • Need to expand the filetype system by examining data from all the XAS beamlines in the world. Yes … all of them.
  • solicit help from the facility representatives

4.7 Windows issues [4/9] [44%]

  • [X] parameter group context menus don’t get posted (see
  • [ ] the frickin’ Gnuplot error logs seem to remain open and locked on Windows when a crash happens
  • [ ] status bar does not get color for wait or error messages this may be unfixable, see
  • [ ] The atoms panel background color is too light. Which window needs its BackgroundColour set to wxNullColour?
  • [ ] clampbox does not get enabled/disabled explicitly, is this another aspect of StaticBox that is different on Windows? (see link above)
  • [ ] Need to test that paths with (parens|commas|quotes) get followed correctly in all situations
  • [X] Initial initialization of gnuplot and feff executable locations in the situation where the package has been moved or reinstalled such that and old demeter.ini still exists
  • [X] relocation of Strawberry leaves Ifeffit unable to find phase shift and CL tables (use an ENV variable?)
  • [X] The Strawberry+Demeter package does not coexist at all with ActivePerl. Best solution is to generate ppd for Demeter armed with all dependencies. Would need to compile wrapper and somehow get gnuplot on the machine. Solved by explicitly calling Strawberry in the .bat files

4.8 Artemis

4.8.1 Artemis bugs and missing features [1/8] [12%]

  • [X] More work on history, see Anthony Ardizzi’s issue
  • [ ] Reorganize lists, move individual items up and down, move blocks up and down, Path list, Plot list
  • [ ] do SSPaths get serialized and deserialized with the pointers to the feff calculation set correctly and no additional folders being created in stash (as was the case for FSPath)?
  • [ ] implementing derivative of phase plotting would require proper handling of this signal in the bkg, residual, and running R-factor plots. Also probably want to disallow it for R123 plots.
  • [ ] better way of moving an empirical standard from Athena to Artemis – should be able to import it directly from the athena project file.
  • [ ] Display of S02 glyph on some Windows is broken, breaking S02 context menu
  • [ ] Eric says: “if u use the automatic choice of parameters for the paths, in combination with a few quick first shell models and an imported feff.inp the whole thing runs in trouble. I don’t really expect name collisions, but it seems to loose track between all the feff runs.” This is a bit unclear…
  • [ ] Path drag and drop [0/3]
    • [ ] DND of an SSPath does not work correctly
    • [ ] DND of FPath also broken
    • [ ] DND of selection (currently only one at a time)
  1. Project [2/4]
    • [X] VPaths to/from project file
    • [X] SSPaths not saved/restored properly?
    • [ ] Indicators to/from project file
    • [ ] GDS objects get created with same groupname over and over as they are used in fits in the history
  2. Advanced fitting
    1. MFC [0/1]
      • [ ] Balance interstitial energies for MFC fits
    2. MDS & Fit Sequence [0/4]
      • [ ] Import mutiple data sets from an Athena project file
      • [ ] feffit.inp import: needs testing; MDS that is not merely MKW
      • [ ] Clone data sets such that the path list gets replicated efficiently (i.e. for MDS fits)
      • [ ] Clone all/marked paths between data sets

4.8.2 Histograms [12/14] [85%]

  • [X] Triangle object
    • yields a DS path and a TS path
    • by R and theta
    • by a trio of Cartesian coordinates
  • [ ] triangle histogram
  • [-] Error checking in Artemis, e.g. check that there is at least one bin in the supplied range(s)
    • [X] SS
    • [ ] ncl
    • [ ] thru

4.9 Hephaestus

More standards!!

4.10 Other object types [0/3]

  • [ ] Structural Units
    • Extension of VPath
    • Store GDS, feff, and path objects in a zip file.
    • On import, mark GDS parameters as merge if in conflict
  • [ ] MSPaths
    • Much like SSPath, make an arbitrary n-legged path
  • [ ] Nearly collinear paths, ThreeBody
    • Extension of VPath
    • need to worry about load order in
    • Define a three body configuration, generate its 4-legged path and a sequence of three-legged paths along with a mixing parameter.
    • It will take a single set of path parameters that are pushed onto the generated Path objects, except for the amplitude, which will be computed from the mixing parameter.
    • This is a single object for the user to interact with which expands into 2 or 3 3-legged paths and a single 4-legged path

5 Atoms and Feff

5.1 CIF [0/3] [0%]

  • [ ] CIF issue: CIF file with “_eof” token at end of file, as in this cif file
  • [ ] Handle CIF import problems more gracefully (See Shelly’s other email from 17 June 2011)
  • [ ] CIF errors are not handled gracefully (e.g. multiple occupancy)

5.2 Atoms [7/10] [70%]

  • [ ] Parsing an atoms.inp file for the space group is fragile. It currently grabs the 13 characters following the equals sign and white space. 13 is the longest standard HM symbol with a modifier – an example is ’I 41/a m d :1’ (see as an example). However, if additional spaces are present or if another keyword follows in fewer than 13 characters, the parser will do the wrong thing.
  • [X] Setting of Ba2Co9O14 (Chanapa Kongmark), also some problem with ZnFe2O4
  • [X] should be a way to insert tabulated shift vector, if it exists
  • [X] handle “: setting” notation: :R, :H, :1, :2 (for example)
    • :1 and :2 seem to have to do with whether the shift vector should be used, see ZnFe2O4.cif and use the shift vector
    • :R, :H specify rhombohedral or tri/hexagonal setting
  • [X] Lepidocrocite issue (amam gets changed to cmcm)
  • [ ] 2 sites at the same position with occupancies <1. see file above for an example
  • [ ] George Sterbinsky’s mailing list post that turned out to be about atoms’ sphere and rhomboid in a non-orthogonal group
  • [ ] very confusing error message when core is not a tag
  • [X] Shell tags in the feff.inp ATOMS list
  • [X]

    Rhombic groups seem not be handled properly. This example fails to generate a subshell of 3 atoms at ~1.9A

    title name:     Fe2O3  hematite
    space  R -3 c
    a    = 5.0380   b    = 5.0380   c    = 13.7720
    rmax = 6.00     core = Fe1
      Fe     0.00000   0.00000   0.35530  Fe1
      O      0.30590   0.00000   0.25000  O1
  • [X]

    This input data fails

    title formula:  LaCoO3
    title refer1:  PRB V. 66 P. 094408 (2002)
    title notes:   T = 300 K
    space  r -3 c
    a = 5.44864       c = 13.1035
    rmax = 6.00       core = Co1
      Co     0.00000   0.00000   0.00000  Co1
      La     0.00000   0.00000   0.25000  La1
      O      0.55032   0.00000   0.25000  O1

5.3 Feff

5.3.1 Feff8 is unsupported except as an Atoms output type

5.3.2 Polarization and ellipticity

5.3.3 Five and six legged paths

  1. This CIF file is an example of a structure that has five and six legged nearly collinear scattering paths at around 4 Angstroms. It has metal atoms bridged by cyanide (CN), like Prussian Blue (see for instance
  2. Another example: 1-4 dibromo benzene has Br scatterers across a benzene ring which can be seen at about 6 Angstroms, along with 3s, 4s, 5s, and 6s:

          /   \
    Br - C     C - Br
          \   /
  3. In fact, any benzene ligand has important 5s and 6s going around the ring. There are various Abs-C-C 3s, 4s, 5s, and 6s.

           /   \
    Abs - C     C 
           \   /
  4. Metals out to 8 Angstroms require 5s and 6s

6 Ifeffit

In file src/lib/iff_show.f, change line 431 from

character*(*) s, t , messg*256


character*(*) s, t , messg*1024

Also need to specify locations of CL data and phase shift tables via an ENV variable

7 Weird stuff I’d prefer not to implement unless demanded

  1. xfit output (only used by women who glow and men who plunder)
  2. csv and text report (excel is implemented)
  3. point finder (this was Shelly’s request)
  4. session defaults (did anyone but me actually use these?)
  5. set to standard (i.e. the one that is marked) – confusing and little used
  6. tie relative energy value to changes in E0 (this was something Jeremy requested originally)
  7. set e0 by algorithm for all and marked – also confusing and little used (implemented)
  8. plot margin lines for deglitching, deglitch many points (this was something that was most useful for a timing problem at 10ID that no longer exists) (implemented)
  9. preprocessing truncation and deglitching (truncation might be worth implementing)

Author: Bruce Ravel