Hi everyone,

I've been working on a new language model architecture called BCMT (Blockwise Causal Memory Transformer) and I'd really appreciate feedback from people interested in efficient transformer scaling.

The main idea is to replace full causal attention with:

• Dense causal attention within fixed-size blocks
• A compact memory representation summarizing each block
• Causal memory propagation between successive blocks

The goal is to reduce the computational and memory cost of long-context training while preserving language modeling performance.

Current results (WikiText-103, context length 256):

• Validation loss: 4.5931 (Dense Transformer: 4.5752)
• +57.4% training throughput
• −23.1% peak GPU memory

I've also included an ablation model (BCMT-HOnly) to isolate the contribution of the inter-block memory mechanism.

This is an early version of the project, and I'm especially interested in feedback on:

• the architectural design,
• additional scaling experiments,
• evaluation protocols,
• and comparisons with other efficient attention or memory-based approaches.

Paper, code, and training details are available here:

https://github.com/rachidlabs/BCMT

I'd be grateful for any comments or suggestions. Thanks!

GPU cloud operators: how do you decided which customers get capacity when you’re supply constrained? Is this manual or automated?

[https://arxiv.org/pdf/2607.13511](https://arxiv.org/pdf/2607.13511))

the core idea is, we cannot have ternary PTQ with fixed matrix size, trying to do that is dead end. so i tried decomposing the matrix to 2 ternary matrices and inner diagonal scaling matrix. now that the inner rank can be arbitrarily large the accuracy can be arbiratily small. and its not that it has to be very large too i also showed that it does take only slightly more vram then current quantisation methods. the slight more vram is worth it if we abuse the ternary math.

A gigantic new model from Moonshot - the biggest open-source LLM by a large margin (but note that it's more sparse than before).

We have also scaled up Mixture of Experts (MoE) sparsity, effectively activating 16 out of 896 experts when paired with a Stable LatentMoE framework. Together with refined training and data recipes, these structural changes yield an approximate 2.5× improvement in overall scaling efficiency compared to Kimi K2, allowing the model to convert compute into intelligence more effectively.

It looks a bit better than Opus 4.8/GPT 5.5 but a bit worse than Fable/GPT 5.6.

I am struck by the fact that they made so little progress on Humanity's Last Exam (58.7%, vs 54% for Kimi K-2.6, which was released in February). LLMs appear to be stalling out at around 60% on Humanity's Last Exam (note that Grok 4 Heavy scored 50.7% over a year ago) despite making rapid progress in other benchmarks.

To be honest, I am now pretty suspicious of that benchmark - particularly after FrontierMath and SWE-Bench Pro were found to have lots of unanswerable/unscorable questions.

Welcome to A Tale of Two Nations, a contamination-proof, cross-domain adversarial stress-testing suite designed to push frontier large language models (LLMs) to their absolute logical limits.

Unlike traditional low-context benchmarks that suffer from data contamination, this ecosystem uses a highly intricate, multi-layered systemic scenario to evaluate an AI's long-horizon reasoning, context-gating integrity, and synthetic logic capabilities under zero-shot conditions.

Will your local model pass? Open-sourcing with scoring framework~✨

Links to the project:

• GitHub Repository: https://github.com/SMahjuba/A-Tale-of-Two-Nations-AI-Benchmark
• Hugging Face Dataset: https://huggingface.co/datasets/SMahjuba/A-Tale-of-Two-Nations-AI-Benchmark