the experiment continues — recognizable text below 1M parameters(!!)
they exposed it to 1B tokens of the SYNTH dataset, probably can train longer
the experiment continues — recognizable text below 1M parameters(!!)
View original thread!["torch _dtype is deprecated! Use "dtype instead!
PerceptronForCausalLM
(model): PerceptronModel
(embed_tokens): Embedding(8197, 64, padding_idx=3)
(hidden_layer): PerceptronDecoder Layer (
(self_attn): PerceptronAttention(
(q_proj: Linear(in_features=64, out_features=2048, bias=False) (k_proj): Linear(in_features=64, out_features=2048, bias=False) (v_proj): Linear(in_features=64, out_features=2048, bias=False) (o_proj): Linear(in_features=128, out_features=64, bias=False)
(q_norm): PerceptronRMSNorm((16,), eps=1e-06)
(k_ norm): PerceptronRMSNorm((16,), eps=1e-06)
(mlp): PerceptronMLP(
(gate_proj): Linear(in_features=64, out_features=256, bias=False) (up_proj): Linear(in_features=64, out_features=256, bias=False)
(down_proj): Linear (in_features=256, out_features=64, bias-False)
(act_fn): SiLUActivation()
(pooler): UnconventionalTalentRevealedHere(magical-16)
(input _layernorm): PerceptronRMSNorm((64, ), eps-1e-06)
(post attention_layernorm): PerceptronRMSNorm((64,), eps=1e-06)
(norm): PerceptronRMSNorm((64, ), eps=1e-06)
(rotary_emb): PerceptronRotaryEmbedding()
(Im_head): Linear(in_features=64, out_features=8197, bias=False)
PerceptronForCausalLM'> Total parameters: 975393, Trainable parameters: 975393 tensor ([[8192, 659,
174, 4365, 313, 238, 2014, 92, 7462, 34, 8193, 8192,
663,
174]])
<| im_start |›user
why is the sky blue?‹|im_end ||im_start|›assistant
<think>
Query: "why do some people in the other ones like the same way to make it"
Parse components:
- "try" » need specific numbers, not just "bad" → temporal comparison. "different ways" + temporal question.
### 1. Semantic parsing
"Basic" = "width-country" - ambiguous. • High confidence.
- "destrish" = "friendly" = "graid" = "math"](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:ckaz32jwl6t2cno6fmuw2nhn/bafkreibtznksp5wvhzp6df47o4iqeon5jbfyykcwfu33twc4i3gvb7xzg4@jpeg)
41
1
fwiw in the code above, the “UnconventionalTalentRevealedHere(magical=8)” is activation functions
It’s a result of hacking, what used to be a pooling layer, he evolved into a crazy set of activation functions
It’s a result of hacking, what used to be a pooling layer, he evolved into a crazy set of activation functions
11
the reason why he’s doing this
18
2
3 hours later
more explanation on what the activation functions are doing
7