It's not that the behavior was defined by the C standard, but that you could confidently predict what a given compiler would generate, for a given platform, so it was quite normal to write programs which made productive use of officially-undefined behavior when that was the behavior you wanted. (It may well have been defined by that particular compiler's documentation.)
Nowadays, people are used to thinking entirely inside the abstraction provided by the language spec, but that was not the case in the '80s and early '90s. The boundary between the C language model and the underlying machine architecture was porous. You would include snippets of assembly language in your C code, if you wanted to do something more quickly than you thought the compiler could do it; and your C code would take full advantage of your knowledge about the underlying memory layouts, calling conventions, register usage, and so forth.
It did not really matter that there were gaps in the "C machine" abstraction because nobody was really programming against the "C machine"; they were programming against their actual hardware, using C as a tool for generating machine code.
you could confidently predict what a given compiler would generate, for a given platform
But then you're no longer writing Standard C. You're writing compiler-flavoured C, which is another source of footguns for projects that outlive the compiler (or version) they were originally written for. Which is fine, as you say, for embedded-style projects that only target one processor model and one compiler version. But I don't think that applies to many of today's projects.
Well, of course not - it didn't exist yet! Or, if it existed, your compiler didn't support it yet; or, even if you had upgraded to a newer compiler which did support it (not a given back then), your codebase and development style preceded the standard, so you continued in the existing style.
Not defined as part of the standard, but before compilers got as smart about their optimizations, it was easier to have behavior that was technically undefined but could be reasoned about in practice.
Now that compilers know cleverer optimizations, undefined behavior is often impossible to reason about because the compiler can change your logic into something else that is more optimal and is equivalent to your logic only in well-defined cases.
You can read the C89 rationale here[0] but in general the point of undefined behavior was (and maybe still is, i didn't check other rationales) partly to let implementations not bother with catching hard-to-catch errors for things that could actually happen and partly to allow for implementation extensions for things they didn't want to or couldn't define.
In addition the entire idea of introducing undefined, unspecified and implementation-defined behaviors was to let existing implementations do, for the most part, whatever they were already doing while still being standards conformant (ok, the rationale's exact words is to "allow a certain variety among implementations", but in practice C compilers already existed in 1989 and the companies behind them most likely wanted to call them as "C89 conformant" without having to make significant changes).
C89 didn't define undefined behavior because that wouldn't make sense, but it did define what it means and going by the C89 rationale about what it was meant to be used for, clearly the idea wasn't the extremist "breaking your code at the slight whiff of UB because optimizations" but "letting you do things that we can't or don't want to define while keeping our own hands clear".
The "letting you" bit is important which is why they have the distinction between "strictly conforming program" and "conforming program" (i.e. minus the "strict") - which essentially has the former only be for "maximally portable" programs and the latter being "whatever conforming implementations accept", with conforming implementations being any C implementation that can compile strictly conforming programs - regardless of any extensions the implementation may have as long as these do not affect the strictly conforming programs.
In other words it was C89 Committee's way of saying "a (conforming) C program is basically anything a C compiler compiles as long as said C compiler also compiles C programs that adhere to the strict conformance we defined here" - which BTW flies in the face of the entire idea that introducing a single instance of "undefined behavior" makes the entire program not "valid C" anymore (after all program with undefined behavior can still be a conforming program as long as it is accepted by a compiler that also accepts strictly conforming programs).
This is the sort of circular self-referencing logic you get when committees try to standardize something that already has a bunch of not necessarily compatible implementations while also trying to not ruffle the feathers of the companies behind them too much.
It'd be an amusing tale if only some people (who you can ignore anyway) didn't get into fights about what page x, paragraph y, verse z of the Standard[1] say and decades later funneling that logic into compilers (which are somewhat harder to ignore) that break existing working code while Bible thumping their standards book whenever someone goes "WTF, this thing used to work before i upgraded the compiler"[3]
