FEFF and IFEFFIT are amazing tools. Together and over the years, they have enabled the analysis and interpretation of many thousands of EXAFS experiments spanning an impressive broad range of scientific disciplines.
Unfortunately, each of these tools is quite difficult to use to its full capabilities. The lack of readily accessible, flexible, highly-capable high-level tools has, for the most part, limited the use of FEFF and IFEFFIT to relatively simple problem. Most examples of their use in the literature are restricted to fairly simple parameterization of a model structure that is calculated using FEFF in the most straight-forward manner.
Certainly examples exist of highly sophisticated used of FEFF and IFEFFIT. For instance, we see the clever use of multiple FEFF calculations on crystalline materials as analogs for metallo-organic environments in S. Kelly, et al., Geochem. Cosmo. Acta, 66:22, (2002) p. 3875. As another example, we find a very complete accounting of the effects of anti-site disorder in a crystal in S. Calvin, et al., Phys Rev B, 66:22, (2002) p. 224405. Both of these examples, however, are the work of particularly dedicated and talented experts in the application of FEFF and IFEFFIT and both represent many months of painstaking effort to develop idiosyncratic analytical methodology.
There has to be a better way.
DEMETER is my attempt to provide a set of tools to enable the sort of high-level approach to EXAFS data analysis that has always been the province of the dedicated few. It is also the result of more than a decade of writing software specifically intended to enable the use of FEFF and IFEFFIT.
Many people reading this wil be familiar with my programs ATHENA and ARTEMIS. They are in wide use throughout the XAS community and around the world. As graphical interfaces to XAS data management and the use of FEFF and IFEFFIT, each has proven successful to a certain degree. Each one, however, suffers from one major flaw (and any number of minor flaws). There is no easy way to write a small, personal program that replicates the exact behavior of the GUI programs.
As convenient as ATHENA is for processing modest amounts of data at the beamline or upon returning home from a beam run, it is not really the right tool for managing huge volumes of data. In several days at a beamline with quick-XAS capabilities, it is quite common to generate many hundreds, even many thousands, of XAS scans. The interactivity that makes ATHENA so appealing when handling small amounts of data becomes a tedious, repetitive nightmare when processing large amounts of data. Unfortunately, there is no way of separating the data processing capabilities of ATHENA from the graphical interface. (This is not an inherent flaw of any of the tools used to create ATHENA -- it is entirely due to my own inexperience when I began writing ATHENA.)
This problem is where DEMETER starts. DEMETER is “middleware” -- it is a system of software tools that sit between FEFF and IFEFFIT and the program that the user actually interacts with. DEMETER is not, by itself, a computer program. Rather it is the tool from which computer programs for XAS data processing and analysis are built.
The package containing the DEMETER libraries actually does include various kinds of interface tools. For example, there is a version of ARTEMIS written using DEMETER. However, this version of ARTEMIS does not add any functionality related to XAS data management not already present in the DEMETER libraries. It is merely a graphical shell layered on top of DEMETER's capabilities. As a result, it is possible -- indeed, often quite easy -- to write a small program which performs a fit to EXAFS data in exactly the same manner as ARTEMIS.
Armed with this middleware layer, it is much easier to consider implementing tools for automating large quantities of data or writing tools for special, one-off data processing chores. And DEMETER offers tools for easily passing data between ATHENA and ARTEMIS and your own DEMETER-using programs.
Tools for automation and for easy access to the capabilities available in the GUI programs would be benefit enough to merit the creation of a software library. DEMETER, however, offers quite a bit more than that. It has capabilities already written or under development for very sophisticated uses of FEFF. Inspired by the articles cited above and by other interesting uses of XAS theory, DEMETER offers easy access to a variety of ways of manipulating the output of FEFF that had previously required a deep understanding of FEFF's inner workings.
So there you have it. DEMETER is a software tool for making easy XAS chores very easy and for making difficult XAS chores tractable. This document is full of code samples which demonstrate the DEMETER way of solving XAS data management and analysis problems. In many cases, you should be able to cut-and-paste examples into your own programs, modifying them slightly to suite your particular problem. Hopefully, DEMETER has enough flexibility that you can begin working on problems that have not even crossed my mind.
DEMETER uses perl. This is, I suppose, an unsexy choice these days. All the cool kids are, after all, using python or ruby. I like perl. I can think in perl. And I can code quickly and fluently in perl. What's more, perl has CPAN, the worlds largest repository of language extensions. CPAN means that I have far fewer wheels to recreate (and probably get wrong). Virtually any language extension I need in pursuit of making DEMETER awesome probably already exists.
DEMETER uses Moose. This is, on the balance, a very good thing, indeed. Moose brings many powerful capabilities to the programming table. When I was about halfway through writing DEMETER, I paused for a bit less than a month to rewrite everything I had thus far created to use Moose. This left me with about 2/3 as many lines of code and a code base that was more robust and more featureful. Neat-o!
For the nerdly, Moose is an implementation of a meta-object protocol. This interesting and powerful tool allows for the semantics of the object system to be modified at either compile or run time. The problem of adding features and functionality to the object system is therefore pushed downstream from the developers of the language to the users of the language. In good CPAN fashion, a healthy and robust ecosystem has evolved around Moose producing a whole host of useful extensions.
Moose offers lots of great features, including an extremely powerful attribute system, a type attribute system, method modifiers, an ability to mix object and aspect orientation, and a wonderfully deep set of automated tests. I am confident that simply by using Moose, my code is better code and, because Moose testing is so deep, I am confident that any bugs in DEMETER are my fault and not the fault of the people whose work I depend on.
For all the wonderfulness of Moose, it does have one big wart that I need to be up-front about. Moose is slow at start-up. Since DEMETER is big and Moose starts slowly, any program using DEMETER will take about 2 extra second to start. For a long-running program like a complicated fitting script or a GUI, an additional couple of seconds at start-up is no big deal. For quick-n-dirty or one-off application, that may a bit annoying. The Moose folk claim to be working on start-up issues. I am keeping my fingers crossed. Until then, I live with the slow start-up, confident that the rest of DEMETER is worth the wait.
Throughout this document, I use the language of object systems to describe DEMETER. I don't expect than everyone using DEMETER should know much about object oriented programming. indeed, my hope is that the examples in this document can be followed and adapted by anyone with a basic grounding in the use of perl. To help introduce that person to the prospect of coding with DEMETER, I'll define a few terms. (More specifically, I'll relate my understanding of those terms. I have no formal training in computer science and am probably wrong a lot about this stuff. DEMETER seems to work nonetheless.) For a bit more Moose-specific information, see the concepts page from the Moose manual.
I will, on occassion in this document, point the reader to web sites where more of the programming details can be found.