NVIDIA and Princeton Unveil ReplaySSM to Accelerate Hybrid-Model Inference

ReplaySSM, a joint project by NVIDIA and Princeton's Dao AI Lab, reduces state memory traffic in hybrid language models by caching recent inputs rather than full state matrices, delivering up to 1.80× kernel speedups at serving batch sizes. (Read More)