Embedding
- This converts words into vector (array of floating point numbers)
- Embeddings will have whole vocabulary of words, these are not words but we refer them as tokens
- LLMS accept tokens (input tokens) and generate tokens (output tokens)
- Pricing of hosted llms is based on tokens
- Try the below prompt
Give me model pricing of all popular llms for api usage in a tabular form
Transformers
-
Refer Here for visualization
-
Transformers take sentence, the token are captured with postions and then the embedding converts token into floating point
- Transformer output will be a probability distribution
- llm picks a token and this choice can be influenced by temperature (0-2)
- llms continue generation till the
<EOS>token is selected or max limit is reached.
Give me max input and output tokens of all popular llms in a tabular form
Well known training datasets
- Common crawl dataset
- Wikipedia
- books
- Github Opensource code bases
-
Arxiv
-
Models can be trained by raw datasets or distillation
-
Models will also go through training on how to respond to humans
- Give me name of model which knows how to follow instructionss and model which doesn’t
Give me most popular models in instruct and base form to run on my hardware
Exercises
- To run a transformer on my system which frameworks are required?
- Give me 5 popular open source transformers which are not llms which i can run
