To investigate whether progress is being made in the field of medical informatics, we compared the annual proceedings of a conference on medical informatics published since 1977, looking for markers that could serve as guideposts to maturation. The markers we had in mind were whether the systems described and whether they cited previous work in the medical informatics fields. Results: By several indicators, the quality of the research reports submitted to a large conference on computer medical applications in medical care is improving. More studies are being conducted and reported, more data are being presented in a more complex fashion, and more authors are including references in their manuscripts... During the study period, less than 5% of reports claimed that a program benefited patients, and even fewer indicated that a program worked outside the authors' institution. This highlights a serious deficiency in our field, the lack of a clear demonstration of reproducible benefit. These findings did not vary with either the application or the technical area addressed.