It's called XML datablock in libccp4. Search cmtzlib.c for xml. But I suppose the Fortran API is missing.

It's stored after MTZENDOFHEADERS, so it doesn't affect history and other records.

It was proposed and added in 2010. Here is the proposal:

Date: 2010年5月24日 09:34:10 +0100
From: Kevin Cowtan <cowtan@ysbl.york.ac.uk>
To: CCP4-dev <ccp4-dev@dl.ac.uk>
Here's new versions of mtzdata.h and cmtzlib.h/c, starting from Martyn's
version with the spacegroup confidence code. This version includes
column groups, column provenance, an XML datablock, and (as an
afterthought) extensibilty.
The XML datablock just appears after the end of the exiting headers.
It's just a string. It can't contain '0円', otherwise it can be used any
way you like.

Thanks for the clarification: not having a Fortran API in the core CCP4 library is probably ok (one could always write that missing wrapper oneself if needed).

As far as I can see, there is nothing XML-like at all in the way cmtzlib handles any dangling lines after MTZENDOFHEADERS. So one could definitely wrap any appendix into something slightly more structured as mentioned above, right?

I usually see the typical workflow often as

 synchrotron/beamline/instrument
 -> raw data
 -> processed data
 -> MTZ file
 -> structure solution
 -> refinement
 -> deposition

with a distinct break between the first part (up to processed data) and the second part (from structure solution to deposition). This can be a break in location (beamline vs home lab), computer system used (synchrotron compute and storage cluster vs lab fileserver or local storage) or just access (not all data/files from original data processing are still accessible at the time the deposition is started).

Since MTZ is very often the working format of choice for reflection data, it would be great if it could provide a bridge/connection between these two parts: not necessarily by providing the same full set of meta data, but some kind of (provenance) tracking information. If (1) some identifier (DOI?) can (a) uniquely describe the first part and (b) provide access to the full set of data including the raw diffraction images, and if (2) this information could travel unaltered as part of the reflection data right down to the deposition stage, then it would allow us to hook the deposition preparation into the original data collection/processing information in the same way we might connect our refined model with a LIMS or sequence database.

mmCIF has the advantage of being a very rich and general deposition format (and well documented), while MTZ has the advantage of being highly tuned towards fast access to specific types of reflection data and a small set of essential meta data (and being well documented). Together they are quite powerful while a single one could always give problems at certain stages.

Thanks, I can see the point of inserting a "fingerprint" in the MTZ that matches that in the mmCIF, so that these can be matched together as coming from the same source later on, notwithstanding the breaks you describe.

My assumption is that this is the desired flowchart

 -> raw data
 -> processed data
 -> unmerged reflns mmCIF --╮
 -> MTZ file |
 -> structure solution |
 -> refinement |
 -> deposition <------╯

That is, the unmerged reflns mmCIF forms part of the deposition package rather than the information being pulled from an unmerged MTZ file.

Can the mmCIF file from DIALS have all this information (beamline name, collection date, collection temperature)?

We have to distinguish between (1) the information that is (or should) be present in the raw diffraction data and that the processing software/system will have access to and (2) the information that is in the data acquisition part but doesn't necessarily make it into the raw diffraction data software.

For a data processing system that is triggered by (or close to) the data acquisition system there is the possibility of fetching relevant information from either. When the starting point is purely the raw diffraction data we might not have all information (but there could be a way of fetching e.g. ISPyB information via a web-based API even if processing at home). Ideally, any information that is currently not part of the raw diffraction meta data but should be present (in order to provide a unique fingerprint) would need to be added to that meta data in order to allow the same level of identification no matter where the raw diffraction data is processed ("transferability").

Maybe as a note how we've been approaching this in autoPROC over the years (after reading plenty of meta data out of a multitude of different diffraction data and image headers etc): initially we decided to collect any known deviations from some perceived standard on a public wiki, partially in the hope this was going to become obsolete with the advent of imCIF, full CBF and HDF5. However, this became a time-consuming and cumbersome task - both to us as maintainers (often detectives) and also to users. So we bit the bullet and went for an automatic system that would remove one of the aspects (annoying to users) ... while still leaving the maintenance work to us ;-) This is similar to the generate_XDS.INP tool from the XDS developers or the specific format readers in dxtbx as far as I understand them.

Anyway, we had to make a decision what would uniquely define a specific instrument. What we came up with is the combination of detector serial number and data collection date (beamline names are a fuzzy concept and temperature is often some hardwired value of 100K). And yes, some instruments don't write one or both of these. And yes, some instruments have incorrect values ... but they still seem to be the most reliable items in the raw diffraction meta data. Of course, we might be in trouble if a detector moves between beamlines and across timezones ... we'll get there once a synchrotron is built on Fiji ;-)

@CV-GPhL, yes your automatic system seems similar to dxtbx in that respect. DIALS (via dxtbx) will recognise images from specific beamlines around the world based on matching serial numbers etc., so we could capture some of the data acquisition metadata even after-the-fact. In practice we don't currently keep this information: after import of the images, the format class that was used to interpret them is not recorded as part of the data processing metadata. We could add this, and there are other good reasons to store the format class (such as when plugins are used that might override those supplied in the standard dxtbx library). However, this would not work everywhere, as you point out. There are no doubt many beamlines across the world that get a generic format base class rather than an instrument-specific match. For Diamond, auto-processing by xia2 will of course have access to the truth, by virtue of being triggered by data acquisition.

That's a great news!

I don't know what to do in case of multiple beamlines. When the user manually selects beamline during deposition, I think they can select only one. And having one beamline is simpler. OTOH it's possible to have multiple values for any mmCIF category:

loop_
_diffrn_source.source 
_diffrn_source.pdbx_synchrotron_site 
_diffrn_source.pdbx_synchrotron_beamline 
_diffrn_source.pdbx_wavelength_list
SYNCHROTRON 'Light Source X' 'Beamline A' 0.9792
SYNCHROTRON 'Light Source X' 'Beamline B' 0.9999

I'm pretty sure that the appendix wouldn't get copied together with test flags. If ccp4 cad is used, it ignores such appendices. Perhaps there are tools that would copy it, but I suppose it'd be from the file with data.

If CCP4 CAD ignores such appendices it might get dropped a lot of times (CAD is such a workhorse). I would try and keep the content of the appendix as simple as possible: as long as it provides /some/ way of finding the original deposition-ready mmCIF file from processing/scaling it will already be a huge benefit.

On the other hand, if it carries many more values but we can't trust that this is accurate (because info from N>1 datasets was lost, wrong info copied through test-set flag transfer etc) we'll end up in another situation where stuff makes it into a deposition just because it populates some fields ... a bit like all those wwPDB entries that are clearly cut-n-paste jobs from a common template with all detail lost.

I'd rather have a NULL than an unreliable value. The NULL value might get corrected/replaced very easily later on while non-NULL values tend to stay forever.

Clemens: the assumption here is that you have a GUI or a workflow that keeps track of the provenance of files. You start from an MTZ file from a synchrotron and this file doesn't get lost in the project.

Generally, providing a way to find metadata in another file would be more complex than directly providing metadata in a file.
Also, „original deposition-ready mmCIF file from processing/scaling" is not possible – you need merged data for deposition.

Are there other items, apart from the ones I listed in the example, that could be included?
Mmcif has tags for describing monochromator, collimation, goniometer and other things. I'm wondering what is useful and possible to determine by the synchrotron automation software.
Possible additions are in categories radiation, source, detector, measurement.