1. Introduction

FEFF and IFEFFIT are amazing tools. Together and over the years, they have enabled the analysis and interpretation of many thousands of EXAFS experiments spanning an impressively broad range of scientific disciplines.

Unfortunately, each of these tools is quite difficult to use to its full capabilities. The lack of readily accessible, flexible, highly-capable high-level tools1 has, to some extent, limited the use of FEFF and IFEFFIT to relatively simple problem. Most examples of their use in the literature are restricted to fairly simple parameterization of a model structure that is calculated using FEFF in the most straight-forward manner.

Certainly examples exist of highly sophisticated used of FEFF and IFEFFIT. For instance, we see the clever use of multiple FEFF calculations on crystalline materials as analogs for metallo-organic environments by S. Kelly, et al. As another example, we find a very complete accounting of the effects of anti-site disorder in a crystal by S. Calvin, et al. Both of these examples, however, are the work of particularly dedicated and talented experts in the application of FEFF and IFEFFIT and both represent many months of painstaking effort to develop idiosyncratic analytical methodology.

  • S. Calvin, E. E. Carpenter, B. Ravel, V. G. Harris, and S. A. Morrison. Multiedge refinement of extended x-ray-absorption fine structure of manganese zinc ferrite nanoparticles. Phys. Rev. B, 66:224405, Dec 2002. doi:10.1103/PhysRevB.66.224405.
  • S. D. Kelly, K. M. Kemner, J. B. Fein, D. A. Fowle, M. I. Boyanov, B. A. Bunker, and N. Yee. X-ray absorption fine structure determination of pH-dependent U-bacterial cell wall interactions. Geochimica et Cosmochimica Acta, 66(22):3855 – 3871, 2002. doi:10.1016/S0016-7037(02)00947-X.
  • B. Ravel, C. Scorzato, D. P. Siddons, S. D. Kelly, and S. R. Bare. Simultaneous XAFS measurements of multiple samples. Journal of Synchrotron Radiation, 17(3):380–385, May 2010. doi:10.1107/S0909049510006230.

There has to be a better way.

DEMETER is my attempt to provide a set of tools to enable the sort of high-level approach to EXAFS data analysis that has always been the province of the dedicated few. It is also the result of more than a decade of writing software specifically intended to enable the use of FEFF and IFEFFIT.

Many people reading this wil be familiar with my programs ATHENA and ARTEMIS. They have been in wide use for many years throughout the XAS community and around the world. As graphical interfaces to XAS data management and the use of FEFF and IFEFFIT, each has proven successful to a certain degree. The original versions of the programs suffered from a major flaw (and any number of minor ones!). There was no easy way to write a small, personal program that replicated the exact behavior of the GUI programs.

As convenient as ATHENA is for processing modest amounts of data at the beamline or upon returning home from a beam run, it is not really the right tool for managing huge volumes of data. These days, at a beamline with quick-XAS capabilities, it is quite common to generate many hundreds, even many thousands, of XAS scans. The interactivity that makes ATHENA so appealing when handling small amounts of data becomes a tedious, repetitive nightmare when processing large amounts of data. Unfortunately, there used to be no way of separating the data processing capabilities of ATHENA from the graphical interface. This is not an inherent flaw of any of the tools used to create ATHENA – it is entirely due to my own inexperience when I began writing ATHENA.

This problem is where DEMETER started. DEMETER is “middleware” – it is a system of software tools that sits above FEFF and IFEFFIT and below the program that the user actually interacts with. DEMETER is not, by itself, a computer program. Rather it is the tool from which computer programs for XAS data processing and analysis are built.

The package containing the DEMETER libraries actually does include various kinds of interface tools. For example, ARTEMIS is written using DEMETER. However, this version of ARTEMIS does not add any functionality related to XAS data management not already present in the DEMETER libraries. It is merely a graphical shell layered on top of DEMETER's capabilities. As a result, it is possible – indeed, often quite easy – to write a small program which performs a fit to EXAFS data in exactly the same manner as ARTEMIS.

Armed with this middleware layer, it is much easier to consider implementing tools for automating large quantities of data or writing tools for special, one-off data processing chores. And DEMETER offers tools for easily passing data between ATHENA and ARTEMIS and your own DEMETER-using programs.

Tools for automation and for easy access to the capabilities available in the GUI programs would be benefit enough to merit the creation of a software library. DEMETER, however, offers quite a bit more than that. It has capabilities already written or under development for very sophisticated uses of FEFF. Inspired by the articles cited above and by other interesting uses of XAS theory, DEMETER offers easy access to a variety of ways of manipulating the output of FEFF that had previously required a deep understanding of FEFF's inner workings.

So there you have it. DEMETER is a software tool for making easy XAS chores very easy and for making difficult XAS chores tractable. This document is full of code samples which demonstrate the DEMETER way of solving XAS data management and analysis problems. In many cases, you should be able to cut-and-paste examples into your own programs, modifying them slightly to suite your particular problem. Hopefully, DEMETER has enough flexibility that you can begin working on problems that have not even crossed my mind.

1.1. The technology behind Demeter

DEMETER uses perl. This is, I suppose, an unsexy choice these days. All the cool kids, after all, use python. I like perl. I can think in perl. And I can code quickly and fluently in perl. What's more, perl has CPAN, the worlds largest repository of language extensions. CPAN means that I have far fewer wheels to recreate (and probably get wrong). Virtually any language extension I need in pursuit of making DEMETER awesome probably already exists.

DEMETER uses Moose. This is, on the balance, a very good thing, indeed. Moose brings many powerful capabilities to the programming table. When I was about halfway through writing DEMETER, I paused for a bit less than a month to rewrite everything I had thus far created to use Moose. This left me with about 2/3 as many lines of code and a code base that was more robust and more featureful. Neat-o!

For the nerdly, Moose is an implementation of a meta-object protocol. This interesting and powerful tool allows for the semantics of the object system to be modified at either compile or run time. The problem of adding features and functionality to the object system is therefore pushed downstream from the developers of the language to the users of the language. In good CPAN fashion, a healthy and robust ecosystem has evolved around Moose producing a whole host of useful extensions.

Moose offers lots of great features, including an extremely powerful attribute system, a type attribute system, method modifiers, an ability to mix object and aspect orientation, and a wonderfully deep set of automated tests. I am confident that simply by using Moose, my code is better code and, because Moose testing is so deep, I am confident that any bugs in DEMETER are my fault and not the fault of the people whose work I depend on.

For all the wonderfulness of Moose, it does have one big wart that I need to be up-front about. Moose is slow at start-up. Since DEMETER is big and Moose starts slowly, any program using DEMETER will take about 2 extra second to start. For a long-running program like a complicated fitting script or a GUI, an additional couple of seconds at start-up is no big deal. For quick-n-dirty or one-off application, that may a bit annoying. The Moose folk claim to be working on start-up issues. I am keeping my fingers crossed. Until then, I live with the slow start-up, confident that the rest of DEMETER is worth the wait.

1.2. Some vocabulary

Throughout this document, I use the language of object systems to describe DEMETER. I don't expect than everyone using DEMETER should know much about object oriented programming. indeed, my hope is that the examples in this document can be followed and adapted by anyone with a basic grounding in the use of perl. To help introduce that person to the prospect of coding with DEMETER, I'll define a few terms. (More specifically, I'll relate my understanding of those terms. I have no formal training in computer science and am probably wrong a lot about this stuff. DEMETER seems to work nonetheless.) For a bit more Moose-specific information, see the concepts page from the Moose manual.

Object
Think of an object as a box containing data. This box comes with a set of instructions for how to manipulate the stuff in the box. The data are called the attributes of the object and tend to be things like numbers, strings, booleans, or other data structures. The set of instructions are called methods and are very similar to subroutines in a non-object-oriented perl script with the caveat that there is an important syntax relation between an object and any of its methods.
Attribute
An attribute is a piece of information about the object that is either set by the user or computed as a result of some interaction with the object. You can always query an object about an attribute value. Some attributes are read-write, which means that they can be set in your program. Others are read-only, which means they have a value that is set when the object is created and cannot be further modified in your program. See the attributes page from the Moose manual for more details.
Accessor
An accessor is a kind of method whose specific job is to query an object for an attribute value or to set the attribute to a new value. DEMETER uses a feature of Moose whereby an attribute is given a name and this name is used as the name of the accessor method. When the accessor is called without an argument, it gets the value of the attribute. When it is called with a value, it attempts to set the attribute to that value. DEMETER defines two additional accessor functions, set and get which are used to access multiple attributes with a single method call.
Type constraints
Moose offers a flexible type constraint system, which means that an attribute can be restricted to have a value that meets a defined criterion. For example, the value for the kmin parameter of the Fourier transform can be defined to be a non-negative number. Attempting to set it to a negative value triggers an error. In this way, sanity checking of parameters is built deeply into DEMETER. See the types page from the Moose manual for more details.
Method
A method is a thing that can be done to an object. For example, there is an object type in DEMETER that is used to contain an XAS spectrum. That object has a method called plot whose purpose is to prepare and display a plot of the data. Another object type is used to define an EXAFS fitting model. That object has a method called fit which is used to actually perform the fit and store the resulting statistical parameters. Many other methods serve rather more mundane chores. In every case, though, the method is something that is done to or with a particular object. Each object has a set of methods that can be called. This set of methods defines the complete behavior of the object.
Trigger
A trigger is something that happens what an accessor is used to set an attribute value. It is an action that takes place as a result of changing an attribute value. For example, when the value of Rbkg is changed for an object associated with XAS data, the trigger is used to set a flag that assures that the background removal is performed anew the next time it is necessary to do a Fourier transform of those data. Triggers are used to control much of the high-level functionality in DEMETER.

I will, on occassion in this document, point the reader to web sites where more of the programming details can be found.

Footnotes

[1]This isn't really true now that LARCH exists. When I started writing DEMETER – and first write this document, it was most certainly true.


DEMETER is copyright © 2009-2016 Bruce Ravel – This document is copyright © 2016 Bruce Ravel

This document is licensed under The Creative Commons Attribution-ShareAlike License.

If DEMETER and this document are useful to you, please consider supporting The Creative Commons.