Reliable Insights

What can you do to pin down metrics-parsing related scrape errors in Prometheus?

Not all scrape errors are due to network or HTTP issues. There's also a variety of parsing errors that are possible including "INVALID" is not a valid start token, strconv.ParseFloat: parsing "...": invalid syntax, expected timestamp or new record, got "MNAME" and expected label name, got "INVALID". There's a number of common causes of these such as having hyphens or periods in metric names, numbers at the start of label or metric names, and missing the newline on the last line of the output.

While spotting these by hand is possible, there's an easier way to figure out which line is the issue. promtool which comes with Prometheus has a subcommand to check metrics syntax. Let's download it and try it out: