vault backup: 2024-06-18 19:20:19

content/education/omscs/cs-6200.md

@@ -10,6 +10,10 @@ I loved the class. It's probably my favorite class I've taken in anything, ever.
 
 In GIOS, the cool thing is that the projects are open-ended. All you have to do is pass a test suite they provided. For Project 1, I did it strictly by the book. For Project 3, I did something very weird and zany to the point where the Autograder actually got confused by my project. Project 4 was my least favorite: it uses gRPC and C++, and I ended up disliking both by the end of it (mostly the C++ API of gRPC, I've used the protocol in other languages successfully).
 
+In all honesty, I wasn't really interested in the class material. It's relatively outdated for modern OS development. Which is fine, I'm not expecting much for how little I'm paying for this education. Just something to keep in mind.
+
+The projects don't really reflect "OS development". It's more, "you're developing an application using OS constructs". I didn't really mind it to be clear, it just that what you're tested on and study, and what you do are very different.
+
 ### Notes I wrote because of this class
 
 [[sending-fds-through-uds]]: I utilized this in Project 3.

content/education/omscs/cs-7674.md

@@ -8,5 +8,5 @@ My second class at OMSCS was cs-7674, Machine Learning for Trading (**ML4T**). I
 
 I'm probably killing my career by saying ML isn't really my cup of tea. Honestly, stocks aren't either, I'm a believer in the Efficient Market Hypothesis and [[stocks|buy $VTSAX and chill]]. I'm more interested in ML from a mathematical standpoint, this class is mostly Applied ML. It'll certainly help you get a job, if that's what you're looking for.
 
-I spent maybe 10 hours per week on the class. I didn't feel really engaged during the class at all.
+I spent maybe 10 hours per week on the class. I didn't feel really engaged during the class at all. I just thought it was kinda... 
 

content/machine-learning/llms/tokenization-is-all-you-need.md

@@ -0,0 +1,24 @@
+---
+tags:
+  - thoughts
+title: Tokenization is all you need
+---
+A LLM, at the end of the day, is a fancy autocomplete. A fancy autocomplete that ingests a bunch of tokens, embeds them to "understand" the semantic context of said tokens, and then outputs a bunch of tokens based on statistical modeling.
+
+I've been thinking a lot recently about how we can make LLMs really start to do interesting things. The current meta (pun not intended) is to make the LLMs output structured info. Say we want the LLM to call a function, we simply tell the LLM "hey, here's my registry of functions" and we expect it to output JSON that describes how it wants to utilize these functions. Of course, is very buggy as LLM's do not understand what "JSON" is and instead understand the statistical nature of what JSON should look like, leading to the development of many fault-tolerant JSON parsers. 
+
+This sucks. In the context of "function calling", the LLM performs actions with the functions, and then it receives more inputs that it uses to form its next response. It's slow, inefficient, and it's not at all how humans operate. I mean, maybe we do, we operate on this world and then observe the side effects and continue operating. 
+
+But what if, instead of teaching the LLM how to output its actions in a tokenized form, we allow it to output the actions itself in its embedded space?
+
+For example, if we give the LLM 2 tokens — one represents "raise left hand", and the other represents "raise right hand", the LLM will output tokens that represent that reality. Then, some external system can interpret these tokens and do the appropriate action — after all, even the brain needs a body to actually execute actions — but the point being is that the LLM is consciously executing those actions, not describing *how* it wants to execute those actions.
+
+Isn't that the same for what humans do? We're constantly ingesting information, whether it be sound, heat, visual, etc. I don't think we convert into numerical embeddings in the technical sense, but I do think we do something similar. When we receive stimulus, we act. Our supercomputer bodies are able to take all that information and react to it by converting it into a "process our body understands" involving complex chemistry, that enables us to do actions and interact with the world, continuously feeding feedback and acting. We're also really advanced prediction models.
+
+For an example: most "text-to-speech" and "speech-to-text" AI systems that enable you to have an AI waifu operate on a two step process. First, you convert the speech into text, you tokenize the text, you feed it to the LLM, and it outputs some text, which is put back into words. It's costly, inefficient, and most importantly, it sucks. When your words are put into text, context is lost. Your tone of voice, your enunciation, every single subtlety is lost. And vice versa, the text being converted into speech loses the context for the tone of voice — after all, how does the speech model know what the AI "meant"?
+
+The only problem becomes, "how do we turn voices into a numerical representation?" I don't have a good answer to this, but I do think  there are a bunch of smart people working and thinking about this. "how do tell an AI that if feels like it should lift its arm in response to stimulus, how can we represent that mathematically?" I don't know, but if you do, you've created Ultron.
+
+I personally believe the next step towards our AI utopia is having LLMs understand more than words. We need them to understand *everything*. And to understand everything, we simply need to teach it how it can represent what we know as *everything* in terms of numerical values.
+
+So tokenization is all you need. We need to find a way to break down every single stimulus in the world in terms of 1s and 0s and then force feed it down the throats of LLMs to make them understand.