Compressed in 2 and 4 bits without retrain, the Kroko transducer processes 1.28 s of audio in 0.90 s on a chip without a neural accelerator.
Billal Boumedine carried Kroko, a continuous English voice recognition model (Zipformer2 transducer with 65.7 million parameters), on ESP32-P4, a microcontroller with two RISC-V 400 MHz cores, a 128-bit SIMD extension and 32 MB SRAM, but without a neuronal accelerator.
2 or 4 bits depending on the sensitivity of each matrix
Without any re-training, the 290 encoder matrices are stored on 2 or 4 bits per weight, the accuracy of each being chosen according to its measured sensitivity, with a prior correction and then a rounded GPTQ. The 71 MB int8 published fall to 26.65 MB. A C engine with hand-written cores processes each 1.28 second audio slice in 0.90 seconds on both cores, including a beam search. As optimizations progressed, the encoding time per slice increased from 51.3 s to 0.62 s.
The error rate on LibriSpeech was 4.34% (clean test) and 10.26% (test-other), compared with 3.94% and 9.17% for the original model. The author also documents his failures, including an attempt to descend to 16 MB stopped at 19.63 MB, and specifies that he did not measure energy consumption. Engine, firmware and weight are published.
Read the full article on the original source
Go to the source →