The maps submitted for the map challenge feature different sizes, sampling and orientations, making comparisons non-trivial. I decided to pose all maps in the same orientation, size and sampling to allow comparison with a reference and using the same mask for all. I developed a method to scale and orient the maps to a reference, aiming at introducing a minimum of interpolation artifacts. While not error-free, it presents a fair relative assessment of the maps. So far this allowed me to identify good maps, as well as maps with potentially serious problems.

The key point of the proposed method is that it may be used for sorting EM maps in a way that the sorting process is placed in an statistical framework. The aim is to detect those maps that are significantly different (better or worse) than the rest.

The goal of our assessment plan is to develop an unbiased method of evaluating the map quality, not based in the reported resolution (FSC graph), but in it's ability to correctly guide an atomic model prediction software (ROSETTA).

The planned method is divided in three stages:

1st Stage
The method starts by identifying a suitable section of the map where the protein structure is known. The selected section of the map is then extracted from the original map. Utilizing the protein sequence and the extracted map section the Rosetta protocol denovo_density can be used to obtain an initial atomic model.

2nd Stage
The initial atomic model, the protein sequence and the extracted map section can be used to produced several thousand atomic model using the Rosetta function, rosetta_scripts.

3rd Stage
Lastly, all generated atomic model are compared to the native atomic model. The resulting root mean square (rms) of the comparison is plotted in a scattered plot to visualized the rms spread. Good quality maps will consistently guide the modelling to an atomic model reconstruction similar to the native structure, while low quality maps will poorly guide the modelling. In other words, good quality maps will show a small rms values and small spread while low quality maps will show higher rms values with a larger spread.

Unfiltered maps will be used. One of the maps will be used as the reference map to align (rigid-body alignment) all other maps and the maps will be re-sampled on the grid of the reference map (the alignment and re-sampling of the maps will be done in Chimera). The re-sampled maps will have the size (in voxels) and the voxel size of the reference map. Pearson correlation coefficients (CC) will then be computed (in Spider) among the maps and among their approximations with 3D Gaussian functions (Gaussian-based approximations of the maps will be computed in Xmipp, using the same standard deviation of Gaussian functions and the same desired Gaussian-based approximation error for each map of a target). The dissimilarities (distances) among maps and among their Gaussian-based approximations (1-CC) will be projected onto a low-dimensional (2D or 3D) space. The maps and their Gaussian-based approximations will be visualized in this low-dimensional space of distances, which should help understand how different these maps are actually.

We only evaluated TRPV1 maps. We re-calculating FSC from half maps using the same spherical mask, to evaluate the nominal resolution of each submitted map. We then use EMRinger to assess side chain quality of each maps, and rank the scores. Last we visually compare all maps to see how specific features are resolved, in particular the density contributed by specifically bound lipid molecules. We finally summarize evaluation of all maps in this entry and compare them. Here, the first two comparison will be quantitative, but the last one is more qualitative.

The goal of the study is to characterize the submitted maps based on several scores, aiming to correlate scores to the resolution of the map. Scores include Map-Map & Map-Model FSC plots, Atom Inclusion, SSE & Rotameric Z-scores.