lemme talk about tokenization and spelling for a minute. we easily could set up tokenization to have one english word per token, and old school NLP (word2vec etc) did exactly that. modern tokenization techniques specifically do not. this won't and shouldn't be done, good tokens aren't "decided", they're learned as a separate process, and well optimized tokens are critical in the performance of LLMs. beyond that we don't *want* LLMs to be incapable of "misspellings", that would make them far less flexible. one example as to why is that u get things like natural capability for translation between languages. other simpler ones include stuff like being able to produce pig latin, invent new words or names, and generally play with language w the same fidelity as a human. smart models *dont* make misspellings by mistake to be clear. chatgpt doesn't just "misspell". but u can prompt and fine tune models to hit latent spaces where misspellings are *wanted*. truth terminal misspells because it's fine tuned and prompted on being a twitter shitposter, and real twitter shitposters misspell words both by accident and purposely for effect. we *want* it to do that, it's a better poster for it.
@Andrew76984254So is a job that has to be done manually tokenizing the english language by dividing words into parts with meaning (i.e. "n't" as a token to negate certain words) and assigning them with vectors of meaning?