Hi everyone,

I've been working on a new language model architecture called BCMT (Blockwise Causal Memory Transformer) and I'd really appreciate feedback from people interested in efficient transformer scaling.

The main idea is to replace full causal attention with:

• Dense causal attention within fixed-size blocks
• A compact memory representation summarizing each block
• Causal memory propagation between successive blocks

The goal is to reduce the computational and memory cost of long-context training while preserving language modeling performance.

Current results (WikiText-103, context length 256):

• Validation loss: 4.5931 (Dense Transformer: 4.5752)
• +57.4% training throughput
• −23.1% peak GPU memory

I've also included an ablation model (BCMT-HOnly) to isolate the contribution of the inter-block memory mechanism.

This is an early version of the project, and I'm especially interested in feedback on:

• the architectural design,
• additional scaling experiments,
• evaluation protocols,
• and comparisons with other efficient attention or memory-based approaches.

Paper, code, and training details are available here:

https://github.com/rachidlabs/BCMT

I'd be grateful for any comments or suggestions. Thanks!