vfscanf() may use the variable 'alloc' uninitialized when taking the branch introduced by commit b287cd745c2243f8e5114331763a5a9813b5f6ee. Spotted by clang.
as reported/analyzed by Pascal Cuoq, the shlim and shcnt macros/functions are called by the scanf core (vfscanf) with f->rpos potentially null (if the FILE is not yet activated for reading at the time of the call). in this case, they compute differences between a null pointer (f->rpos) and a non-null one (f->buf), resulting in undefined behavior. it's unlikely that any observably wrong behavior occurred in practice, at least without LTO, due to limits on what's visible to the compiler from translation unit boundaries, but this has not been checked. fix is simply ensuring that the FILE is activated for read mode before entering the main scanf loop, and erroring out early if it can't be.
this is mandated by C and POSIX standards and is in accordance with glibc behavior.
this glibc abi compatibility function was missed when the scanf aliases were added.
invalid format strings invoke undefined behavior, so this is not a conformance issue, but it's nicer for scanf to report the error safely instead of calling free on a potentially-uninitialized pointer or a pointer to memory belonging to the caller.
for conversion specifiers, alloc is always set when the specifier is parsed. however, if scanf stops due to mismatching literal text, either an uninitialized (if no conversions have been performed yet) or stale (from the previous conversion) of the flag will be used, possibly causing an invalid pointer to be passed to free when the function returns.
this seems to have been a regression from the refactoring which added the 'm' modifier.
this commit only covers the byte-based scanf-family functions. the wide functions still lack support for the 'm' modifier.