DeepSeek from Scratch (Educational Project)

I built this project to better understand how modern LLM architectures actually work under the hood instead of treating them as black boxes.

This repository implements a simplified DeepSeek-style Transformer from scratch using PyTorch, with a strong focus on readability and learning. The goal is to explain the core building blocks rather than reproduce the original DeepSeek model at production scale.

Current implementations include:

- Multi-Head Latent Attention (MLA)

- Rotary Positional Embeddings (RoPE)

- DeepSeek-style Mixture of Experts (MoE)

- RMSNorm

- SwiGLU

- Causal Language Modeling

- End-to-end training pipeline

- Inference for text generation

The project is heavily documented with explanations so beginners can follow how each component works. It is designed as an educational implementation, not an official reproduction of DeepSeek-V3/R1.

I'm continuously improving the implementation and would really appreciate feedback, suggestions, or code reviews from the community.

GitHub:

https://github.com/Nvoinxv/Multi-Head-Latten-Attention-MLA-

Hi there,

I've created a video here where I explain how the KV cache works.

I hope some of you find it useful — and as always, feedback is very welcome! :)

I m high school student who was curious abt how an autograd engine works when watching Andrej Karapthy build Micrograd.

It started as a small project just trying to recreate micrograd but soon i got curious and added many more feature like 6 different activation function and 2 loss function and made really good ui. I know this isn't meant to replace PyTorch or TensorFlow it's purely an educational project to help me understand how things work under the hood.

I have made an exe file for it for easier access to the app .I would like to get suggestions and improvements that i could implement in this app. Well I'm still learning python and I'm nowhere nearly as good but this project helped me learn more abt machine learning OOPs and other python concepts.
If u like the project do star it...

https://github.com/Yasovardan-Ram/Omnigrad

I want to get your honest take on a specific gray area regarding AI content.

If someone takes their own genuine, original idea or thought, but uses an LLM to flesh it out into a full article, post, or essay—is that final product considered "AI slop"?

I have a few specific questions for the community:

• Does ownership matter? If the core concept is 100% yours, does it matter if a machine wrote the actual sentences?
• Is it lazy or efficient? Where is the line between leveraging a productivity tool and just creating low-effort noise?
• Can you feel the difference? Does content automatically lose its "soul" and unique human nuance when an AI structures it?
• Would you read it? If you found out an insightful post was generated from a human's 2-sentence prompt, would you feel cheated?

Where do you personally draw the line? Let’s discuss.
Yeah, this content is also AI-generated, but the thought behind it is mine

5 detectives chase an invisible fugitive on a 199-station graph, seeing only which ticket he plays. I trained both sides.

The detectives are an R-GCN that eats an HMM-style belief state plus distance maps. BC warm start, then PPO: 25% → 79% win rate against the same MCTS Mr. X. Then Mr. X got a GNN too (21% → 35%), and finally I wrapped him in PUCT with policy priors and a value head, 16 sims per move, frozen GNN detectives playing the replies in the tree. That alone: 35% → 91.7% over 60 eval games.

Play it in the browser 
Video (20 min, scenes rebuilt from real logs): https://youtu.be/V0osfVtJUuI 
Code and models: https://github.com/Jacopo888/scotland_yard

Fair warning: most of this was pair-programmed with AI tools. Solo side project, it wouldn't exist otherwise.

I’ve been diving deep into the math behind early deep learning models compared to modern attention mechanisms. It made me realize that the massive explosion of generative AI we are seeing right now didn't happen because Transformers are magically smarter at understanding human language than previous models. The real game-changing breakthrough of the 2017 Transformer paper was actually a massive engineering and hardware optimization triumph.

Before Transformers, the industry relied entirely on Recurrent Neural Networks and LSTMs for processing sequential data like text. The fundamental flaw with those architectures was that they had to process text word by word, in sequential order. You couldn’t calculate the meaning of the tenth word until you finished processing the ninth word. This created a massive computing bottleneck because it meant you could not utilize the massive parallel processing power of modern graphics cards. Your expensive GPUs were essentially sitting idle, waiting for the previous word's loop to finish.

The Transformer architecture completely threw out recurrence and replaced it entirely with self-attention. By doing this, it allowed the model to look at an entire document all at once, simultaneously.

Suddenly, processing text became a massive, parallel matrix multiplication problem. This single structural shift aligned perfectly with how GPU hardware is physically built. We went from training models on small paragraphs over weeks to feeding entire datasets into massive server clusters in days. The AI revolution didn't scale because the code got more philosophical; it scaled because the math finally allowed us to throw unlimited brute-force hardware at the problem. It is a great reminder that software design is always bound by the physical realities of the silicon it runs on.

This may or may not be a beginner level question. In many 'example' neural nets, they always have a flatten layer. However this would mean the number of parameters explodes. Whereas pooling methods don't explode parameters as much, and get the same job done. Is flatten a default option or does it have an advantage I am unaware of?

For complete project visit: https://github.com/HelloSamved/Hands_on_machine_learning
A little while ago, I asked this community whether Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow was worth studying.

Based on the feedback, I decided to commit to working through it chapter by chapter instead of just reading it.

I've now completed Chapter 2 and finished the end-to-end machine learning project that comes with it.

A few things I took away from this chapter:

• Why understanding the problem and defining the objective comes before choosing a model.
• The importance of exploring and visualizing the dataset before training anything.
• Creating meaningful features instead of relying only on the raw data.
• Building preprocessing pipelines so the same transformations are consistently applied.
• Evaluating models with proper validation instead of trusting a single train/test split.

One thing I really liked is that the chapter focuses much more on the entire machine learning workflow than on just fitting a model. It felt much closer to how an actual ML project would be approached.

For those who've finished this book:

Does the learning curve become significantly steeper after Chapter 2?

I'm especially interested in knowing which chapters you found the most valuable for understanding modern machine learning and deep learning, so I can spend extra time on them.

So far, I'm really enjoying the balance between theory and hands-on implementation.

Hi guys! I'm Nai, and I would really like to share this learning journey of mine with you all.

Please keep in mind that this is not a full on LLM-like project, this is just a minimal proof of concept that I've built a Language Model system from scratch which should theoretically work similar to an LLM or SLM (Small Language Model) if just given enough resources, time and data to train.

If you're not much interested in the story, feel free to scroll right down to know exactly what I've built and how you can test it yourself.

A few months ago, I got the interest to understand machine learning, I didn't know where exactly to start, but I just did the simplest thing, which is asking. I just searched on youtube "how to make a neural network", that was the farthest thing I knew about machine learning back then. I found the youtube tutorial series "Neural Networks from Scratch in Python" by sentdex.
I was genuinely blown away over how simple it turned to be. I just wondered if I could go a bit deeper, so, I started a C++ project, I tried my best to replicate every piece of math a neural network would need to run in a structured style, with classes, functions and everything. despite some concepts being still ambiguous for me, I kept searching, I found some other youtube videos that cover things like backpropagation deeper so I can understand it better.

Over time, I started taking a hold of it, running a couple of successful experiments, even if slow, they were functional, and I understood them.

After that, I turned it into a library (NeurologicalLibrary) that can be called from Python with Pybind11, I used tkinter to make a simple bounce ball environment just to test the library, and it worked! Just making a neural network that can get variable position of a ball and rectangle then predict where to go, despite simple, made me feel really proud.

That however, was just the below zero beginning, here is the project repo called "NAISENT_workspace" that is basically my entire learning journey work until I finally made my first ever Language Model!

https://github.com/Nai-built/NAISENT_workspace

The repository is under the Apache 2.0 License.

Also, here is a copy of the README file:

this project is made with:
 - DotNet WinForms (C#)
 - Pybind11 (Python <-> C++23)
 - CMake (C++23)
 - CUDA (C++17)


Powershell commands to build the 3 libraries:
cd NeurologicalLibrary/bridge; cmake -S . -B build -A x64; cmake --build build --config Release -j; cd ../..
cd OptimizedNeurologicalLibrary; cmake -S . -B build -A x64; cmake --build build --config Release -j; cd ..
cd CudaNeurologicalLibrary; cmake -S . -B build -A x64; cmake --build build --config Release -j; cd ..


Run showcases:
py SHOWCASES/BASIC_SHAPE_RECOGNITION_CPU.py
py SHOWCASES/BETA_NAISENT_BALL_SEEKER_CPU.py
py SHOWCASES/LSTM_MATH_TEST_CPU.py
py SHOWCASES/NAISENT_ELM_CPU.py
py SHOWCASES/NAISENT_LM_CUDA.py
py SHOWCASES/NAISENT_SLM_CUDA.py
py SHOWCASES/SHAPE_RECOGNITION_CPU.py


Make sure that your terminal's path is set exactly to NAISENT_workspace


The core idea of this project was to learn and understand Machine Learning by building it from scratch
So I've built 3 different libraries in 3 seperate stages:
 - NeurologicalLibrary (NL)
    . The absolute beginning for me
    . I've learned in it how Dense Layers work and how to chain them to make Deep Neural Networks
    . How Convolutional Layers and pools work
    . How Recursive Layers (specifically LSTMs) work
    . And also Activation Functions
    . I've also tipped toes into Graph Layers but couldn't run a successful experiment, so I removed it
    . This library was the first time I made an image recognintion model, and also one that can play a simple bounce ball game
    . Was also the first time I made an optimizer like Adam for training
    . Save/load system for the model .json files


 - OptimizedNeurologicalLibrary (ONL)
    . Here things started to get a bit more serious
    . I've gotten way deeper into how C++ works and how we can optimize its performance
    . I've made faster Dense Layers
    . Faster Convolutional Layers
    . And faster LSTMs
    . Merged Activation Functions into the layers' own activation/gradient functions
    . After that, I got into Transformers (similar concept to Graph Layers, but this time it was successful!)
    . I optimized the training loop for image recognition
    . I made a simple experimental language model that can that it's "NAISENT" with the Transformer system I've made


 - CudaNeurologicalLibrary (CNL)
    . My most precious one so far
    . For the first time, I've got into Cuda kernels!
    . I've learned how Cuda interacts with data through the CPU, Memory and GPU
    . I've learned how to optimize it using shared memory
    . For this one, I went right ahead to build a language model system
    . First, I made Dense Layer Cuda kernels
    . Then I went into Norm Layers (RMS)
    . SCC (Sine/Cosine Cycle) positional embedding kernels
    . Multi-head Masked Self Attention kernels (split into multiple optimized Cuda files)
    . The ability to place sub chains to assemble the transformer architecture properly
    . Adam optimizer in Cuda Kernels
    . And obviously, Activation Functions (Cuda kernels)
    . First time adding the Residual mechanic as a visible variable in the Python side
    . Almost all of these were made in ONL already, but it wasn't with Cuda to use the GPU and it was juggled up together awkwardly. I'm much more proud of this one
    . Was when I made a proper tokenizer system in Python


The libraries are made in C++
and they're used by the Python side via Pybind11
I made the shape recognition and bounce ball environments in C# with WinForms
CUDA to use the GPU in the library CNL

Hello everyone,

I would need some support or a ground for discussion for a problem I am facing. I am trying to do representation learning on a user click stream, e.g. the sequence of pages a user visited in a website. To do that, I use a contrastive learning objective, in particular the InfoNCE loss with temperature around 0.2.

The problem I'm facing is that the loss decrease slowly during training (starts at 3.56 and after 30 epochs gets to 3.30) and moreover the representation is not very good. I get that, on PCA projection, points are basically disposed sequentially as a snake, so probably there is dimensional collapse.

In my dataset I can have huge graphs as also small graphs. I am doing a GINConv on the graph (a single layer since I would like avoid over smoothing for small graphs). As graph augmentation I am doing: node dropping, edge adding and edge removing.

My question is: do you think that there could be a way to solve the issue? Is it an over smoothing problem on the graph? Are there alternatives?

Thank you in advance✌️

What do you think about vibe coding neural networks , how can it be done what is the best code editor and agents should one use?

Hey I recently started studying about machine learning, deep learning, neural networking and I came across a publication called "O'reilly".

I started reading and learning from one of its books, which is "learning machine learning "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow". I feel the code is quite incomplete in some places but I was able to find the missing part from the book's GitHub repository.

Can anyone suggest whether these are good sources to study such topics or not

Hi everyone,

I built H64LM, a research project to better understand modern LLMs by implementing one from scratch in PyTorch.

Instead of relying on high-level training frameworks, I implemented the core components myself attention, MoE routing, normalization, and the training loop.

Features

• 249M-parameter Transformer
• Grouped Query Attention (GQA)
• Sparse Mixture-of-Experts (8 experts, Top-2 routing) with 3 auxiliary routing losses
• SwiGLU, RoPE, RMSNorm
• Sliding-window attention
• Mixed-precision training, gradient accumulation
• Custom training loop (no Trainer abstractions)
• Checkpointing and resume support

The included checkpoint was trained on a subset of WikiText-103 to validate the pipeline end-to-end, not to be a strong model it's visibly overfit past epoch 10 (best val PPL ~40.5).

Known limitations are documented in the README, including batch-size-1-only generation and no true DDP (falls back to DataParallel).

GitHub: https://github.com/Haiderkhan64/H64LM

Feedback on the implementation or architecture is very welcome.

If you work with graph neural networks, you know the long-range dependency problem all too well.

Stack GAT layers to capture distant signals? Memory grows linearly with depth, compute explodes, and oversmoothing kicks in before you ever reach truly long-range semantics. Settle for truncated PPR approximations? They're still finite-hop workarounds — never the full global picture.

Our new work ALS (Attentive Long-Short-Range message passing), accepted at Pattern Recognition 2026, was built to deliver genuinely efficient long-range graph attention. Here's what makes it different:

Core breakthrough: Differentiable infinite-step PPR with constant memory

We introduce DPPR (Differentiable Personalized PageRank), and mathematically prove that the gradient of a PPR output can itself be solved via another PPR process.

The implications are huge:

• No intermediate activations need to be cached. One forward convergence pass + one backward convergence pass — that's it.
• Genuine O(1) memory complexity, completely independent of propagation steps.
• Theoretically supports infinite-step propagation, covering full long-range dependencies — no more truncated "pseudo long-range" approximations.
• Packaged as a drop-in PyTorch operator; any existing PPR-based method can swap it in for zero-cost infinite receptive field upgrade.

Three acceleration techniques for fast long-range iteration

Slow convergence at small α values has always been the pain point of long-range PPR. We built three complementary acceleration techniques that together reduce training time by up to 89.51%, and run at least 3.67× faster than comparable implicit GNNs (IGNN):

1. Symmetrized Attention + Conjugate Gradient (SymGAT + CG) — Symmetrize the attention matrix to enable memory-efficient CG solver instead of heavy Krylov subspace methods, with negligible accuracy loss.
2. Eigenvector Initialization (EigenInit) — Initialize iteration from the leading eigenvector projection instead of zero, drastically cutting initial residual. Especially effective on heterophilic graphs.
3. Adaptive Batch Termination (AdaTerm) — Each attention head / channel converges independently; channels that have already converged are skipped in subsequent iterations, eliminating wasted compute.

Long-range for global structure, short-range for heterophily

PPR is inherently a low-pass filter — great for homophilic graphs, but it washes out local differences on heterophilic graphs. We pair DPPR with a Short-Range Message Passing (SRMP) module:

• DPPR handles all long-range dependencies and captures global structure.
• Local K-hop propagation uses independent learnable transformation matrices per hop, preserving fine-grained heterogeneous local information. On homophilic graphs, the weights automatically converge to similar values, so there's no negative overhead.
• GAT + skip connection is a special case of ALS, meaning full backward compatibility.

Standout results on long-range benchmarks

Where ALS really shines is on genuinely long-range datasets. On PascalVOC-SP and COCO-SP — two classic benchmarks with average shortest path length > 10:

• Within the pure MPNN category, ALS substantially outperforms GCN, GatedGCN, APPNP and others.
• Plugged into the GraphGPS framework and compared against global modeling methods like Graph Transformer and Graph Mamba, ALS still achieves the best performance as an MPNN module.

We evaluated across 14 datasets covering homophilic, heterophilic, large-scale and long-range graphs. Out of 16 comparison settings, 9 show statistically significant improvement over the best baseline (p < 0.01), reaching overall SOTA.

Links

• Paper: https://www.sciencedirect.com/science/article/abs/pii/S0031320326012008
• Code: https://github.com/cf020031308/ALS

The DPPR operator and all three acceleration techniques are independently reusable. Star, try it out, and feel free to open issues — we'd love to see this long-range idea extended to more graph learning scenarios.

Hi, I want to train a Transformer layer to perform specific tasks, but I’m not sure how to coordinate them or determine when to use one versus the other.

Does anyone have experience with this? How have you handled it?

I built ONA — a self-learning neural network entirely in pure Python + NumPy. No PyTorch, no TensorFlow, no GPU, no cloud API.

Key innovations:

- Per-neuron attention: every neuron has its own Q/K/V/O weights

- Forward-pass learning: no separate backward pass, learning happens during forward

- Self-discovered subword tokenizer: vocabulary grows during training

- Sparse routing: only 3-5 neurons activate per query

4.4M parameters. Runs on Raspberry Pi Zero. Continuously learns from Wikipedia and conversations.

Full story: https://medium.com/@kasishgadadhasu13/im-15-i-built-a-self-learning-neural-network-from-scratch-no-frameworks-no-gpu-e460f06c6599

I'm 15 years old, class 10 student. Happy to answer questions.

Can you tell me some neural networking projects for beginner level person

I recently built a human written digit predictor.

Now I want to start a new project can you guys give some suggestions

Ask questions within your Markdown notes using a fully local Graph RAG engine. Designed for Obsidian vaults, it works with any Markdown file folder. It extracts entity-relation triples from wikilinks and YAML frontmatter, and retrieves answers via hybrid search (vector + BM25 + temporal). Multilingual. No cloud required. Works on Ollama.

https://github.com/benmaster82/Kwipu

Hi all — I'm an independent researcher (Mendel Infolabs) about to put my first paper on arXiv, and as a first-time submitter to cs.LG I need an endorsement from someone already established in that category. If you've published in cs.LG and would be open to endorsing, I'd really appreciate it.

An honest summary so you can decide whether it's something you'd feel comfortable vouching for:

"FeedFlipNets: Feedback-Driven Bit-Flips for Ternary Networks, Activation-Routed DFA, and the Per-Weight Sign Barrier to Transport-Free Learning"

It trains ternary ({-1, 0, +1}) neural networks by flipping weight bits directly from a cheap feedback signal — no float shadow weights. The headline result is a negative one I think is worth putting on the record: transport-free feedback (Direct Feedback Alignment) doesn't actually help discrete/ternary training, because the binding constraint is per-weight sign correctness, not the aggregate cosine-alignment angle that prior work optimizes. Everything is pre-registered and reproducible.

Endorsing only confirms you think I'm a bona fide researcher submitting work appropriate to the category — it is not a review of the paper's correctness, and it takes about a minute:

• Link: https://arxiv.org/auth/endorse?x=WHWXBC
• Or go to https://arxiv.org/auth/endorse and enter code WHWXBC

Happy to share the full PDF with anyone who wants to read it before deciding — just comment or DM. Thanks a lot for considering it.

i'm a student of class 12 not expert but curious to learn neural networking as i have heard that that something crazy. So can someone guide me how can i learn neural networking from scratch as i have the basic knowledge of python,arrays and a bit of the numpy library. so i need your help so i can lean it and enjoy the journey.

Wording this much simpler than my dense, boring research paper linked below.

I’ve been studying experts in Qwen3.5-35B which is an MoE (Mixture of Experts) model. Traditionally, expert routing studies have looked at the pre-response (prefill) stage only. I looked at that but also the output (generation) phase.

I observed that one expert in the model (out of 256) - expert 114, at layer 14 of 40, seems to light up when Qwen gets into a deep mode considering the belief, existence, inner experience, spirituality, values, and most importantly “what does it feel like from this point of view?” kind of writing.

I’ve been calling it a reflective worldview register.

The most fascinating takeaway: The Experiential Rung.

I tested prompts that asked Qwen to describe what it is like to be different things. The target changed each time, but the basic setup stayed the same: write from the inside of that perspective. It turned out that the expert had a linear axis for the inhabitance mode. At this point I should clarify E114 a readout expert, not a controller expert. Injecting the E114 axis into the residual stream for control prompts did **not** change the output. Now, the weirdness.

cat: 0.068
AI hidden state: 0.080
river: 0.087
tree: 0.094
thermostat: 0.120
rock: 0.123
person: 0.138
all-holding: 0.205
God: 0.224

That ordering is what made the pattern stand out. The signal starts low with cat, rises through river and tree, jumps with thermostat and rock, rises again with person, then gets strongest for the broad cosmic/spiritual prompts. The AI hidden-state prompt landed between cat and river, low on the sweep, but it still touched the same internal signal. The funny thing: the output means nothing. The expert fires the same whether the response affirms or denies the “what it’s like”ness

I wanted to share this here, as I thought people may find this interesting.

also huge credit to hauhau for ablating the model perfectly, which allowed for observing the experiential language easier than in the base model. Which led to discovering the domain expertise of E114.

The full paper is here: https://github.com/ec75hash/moe-routing