I don't know if it's relevant but on OpenBSD, if the strtol() is too complicated to manage, there is strtonum() alternative which is meant to be simpler
is an OpenBSD extension

OpenBSD's strtol not so complicated with compare to glibc's variant. Just some edge case you can to filter by simple check. "0x" syntax disallowed generally by tor_inet_pton so no more additional side effects.

My reading of C99 is that strtol("0xquux", &next, 16) must return zero and next must point to the x. The optionality in paragraph 3 is for the input data, not the implementation.

Under Linux produce:
l:0 rest:xquux

Under OpenBSD produce:
l:0 rest:0xquux

The question is to know if the Tor code is good enough and OpenBSD need to fix something or OpenBSD is sufficiently conformant and the Tor code need to adapt.

I believe the OpenBSD result is correct according to the C99 text and contradicts catalyst's statement. Because paragraph 7.20.1.4.7 states

If the subject sequence is empty or does not have the expected form, no conversion is performed; the value of nptr is stored in the object pointed to by endptr, provided that endptr is not a null pointer.

With the example code, nptr does not have the expected form, so no conversion is performed therefore nptr == *endptr. Furthermore, paragraph 7.20.1.4.8 states

The strtol, strtoll, strtoul, and strtoull functions return the converted value, if any. If no conversion could be performed, zero is returned. If the correct value is outside the range of representable values, LONG_MIN, LONG_MAX, LLONG_MIN, LLONG_MAX, ULONG_MAX, or ULLONG_MAX is returned (according to the return type and sign of the value, if any), and the value of the macro ERANGE is stored in errno.

With the example code, nptr does not have the expected form, so no conversion is performed therefore the return value is zero.

Combining these two together means that when the subject sequence is empty or does not have the expected form, no conversion is performed and the return value is zero and nptr == *endptr.

Nothing in C99 7.20.1.4 explicitly says that a string starting with 0x should result in nptr == *endptr when base == 16.

It might be ambiguous about whether 0x is an expected subject sequence for strtol with base == 16. I think the ambiguity is whether the subject sequence is 0 vs 0x, rather than empty. 7.20.1.4p7 says nptr == *endptr if the subject sequence "is empty or does not have the expected form", but 7.20.1.4p4 defines the subject sequence as "the longest initial subsequence of the input string, starting with the first non-white-space character, that is of the expected form", so "not have the expected form" is redundant because that is impossible for a subject sequence as defined.

It goes on to say that the subject sequence is empty if "the input string is empty or consists entirely of white space, or if the first non-white-space character is other than a sign or a permissible letter or digit". Neither of these is true for a string starting with 0x, because 0 is a permissible digit for base == 16.

This also brings up the question of whether a subject string can consist of only a sign prefix. I think it can be, but the reference to 6.4.4.1 implies that is not true, at least for base == 0, because an integer-constant cannot be empty.

If anybody wants to ask the OpenBSD Libc maintainers what they think about the standard here, they should feel free to do so politely. Apparently OpenBSD inherited the code from NetBSD, which may also have the same behavior. FreeBSD appears to have patched itself to have the behavior that Tor expects.

I'll refrain from any arguments about whether the standard permits this behavior; even if it doesn't, the behavior apparently exists in the wild, so Tor has to work around it.

If anybody wants to ask the OpenBSD Libc maintainers what they think about the standard here, they should feel free to do so politely. Apparently OpenBSD inherited the code from NetBSD, which may also have the same behavior. FreeBSD appears to have patched itself to have the behavior that Tor expects.