Fine-Tuning Nemotron 3.5 ASR for Regional Accents

NVIDIA outlines how to adapt its open speech model to local dialects in under five hours without degrading existing language skills.

Commercial speech recognition models handle standard languages reasonably well, but they often stumble on regional accents, colloquial slang, and local phrasing. To address this friction, NVIDIA released a technical guide and open source code detailing how to execute fine-tuning on Nemotron 3.5 ASR for underserved dialects. Using Saudi Arabic as an empirical test case, the company adapted the 600-million-parameter streaming model in roughly four and a half hours, cutting word error rates by nearly half in laboratory runs.

Fine-Tuning Nemotron 3.5 ASR Without Forgetting

The project drew on 133.7 hours of filtered audio from the SADA 2022 corpus, compiling 103,559 recordings centered on the Najdi and Hijazi dialects. Before training, the baseline model showed a word error rate (WER, measuring the share of mistranscribed or omitted words) of 55.05%. After fine-tuning, that error rate fell to 29.96% on the dialect test split and from 58.84% to 35.61% across the entire SADA test set, according to figures released by NVIDIA.

A primary obstacle in this kind of specialization is catastrophic forgetting, where an artificial neural network grasps new vocabulary but discards prior linguistic capabilities. To prevent that drift, the engineering team added a replay stream to the optimization loop. The training mix combined 90% regional dialect audio with 10% reference samples from the FLEURS dataset (split into 7% English and 3% Modern Standard Arabic). That balance preserved core recognition and slightly reduced English transcription errors from 11.04% to 10.42%.

Fast Training and Diarization Support

The underlying system used in the tests was Nemotron 3.5 ASR Streaming, designed for real-time transcription across 40 languages with native punctuation and capitalization. The training run lasted 12,000 steps and finished in about 4.5 hours across two GPUs, using duration-based bucketing across 30 buckets to maximize memory efficiency.

The workflow also links with Nemotron 3 Diarization, an open-weights model that distinguishes and tracks up to eight speakers in a single audio channel with timestamp alignment. NVIDIA packaged the full pipeline into an interactive Jupyter notebook hosted on GitHub in the NVIDIA Riva tutorials repository.

Practical Limits and Impact for Teams

These results warrant measured expectations. The performance numbers come straight from NVIDIA’s internal engineering benchmarks and lack external independent verification. Furthermore, a word error rate hovering near 30% still demands human supervision in mission-critical settings such as medical charting or legal transcription.

Even so, the walkthrough provides a functional blueprint for technical teams handling linguistic regions overlooked by large cloud providers. Instead of paying recurring per-minute API fees to proprietary vendors that fail on colloquial terms, engineering groups can deploy compact open models on private hardware and specialize them with modest datasets.