How a Pronunciation Assessment API Works: From Audio to Multi-Dimensional Scores

Sentence evaluation demo: multi-dimensional scores returned immediately after reading

When you say "good morning" in a language-learning app and the screen instantly shows a pronunciation score of 98—highlighting exactly which word and which phoneme held you back—what just happened?

For developers and technical decision-makers, understanding how a pronunciation assessment API produces its scores may matter more than simply getting the integration working: it determines whether you can explain a given score to your users, and which metrics deserve attention during vendor selection.

This article opens up the complete pipeline of the DolphinSOE pronunciation assessment engine, following one piece of audio all the way to a set of multi-dimensional scores.

What Is Pronunciation Assessment, and How Does It Differ from Speech Recognition?

At its core, a pronunciation assessment service converts a user's speech input into an explainable set of pronunciation scores.

It typically does four things:

1. Receives audio and a reference text
2. Recognizes what the user actually read
3. Aligns the recognition result with the target text word by word and phoneme by phoneme
4. Returns the scores

The biggest difference between pronunciation assessment and speech recognition lies in their goals:

In short, speech recognition cares about what you said, while pronunciation assessment cares about how well you said it. To explore how inputs and outputs differ across question types, see the question type documentation.

What Happens During an Assessment Request?

The pronunciation assessment API takes the user's recorded audio and a reference text, and automatically returns multi-dimensional pronunciation scores with corrective feedback. Taking English sentence evaluation as an example, a typical request can be broken into four stages:

1. Capture: the client records the user's audio at a 16 kHz sample rate (e.g., mp3 / wav) and submits it together with the reference text.
2. Recognition & alignment: an acoustic model turns the waveform into text and aligns it with the reference text word by word and phoneme by phoneme.
3. Scoring: the aligned result enters the scoring and error-detection modules, which classify error types and produce the overall and per-dimension scores.
4. Feedback: structured JSON returns to the client, which renders scores, waveform highlights, and corrective hints.

The processing flow of one assessment request

How Is the Assessment Engine Implemented Under the Hood?

To understand where the scores come from, it helps to look at how the "recognition" step has evolved.

The traditional approach uses a GMM-HMM hybrid architecture: a Gaussian Mixture Model (GMM) serves as the acoustic model to obtain phonemes, a pronunciation dictionary stitches phonemes into words, and a language model polishes them into a sentence; the recognized sentence is then compared with the target sentence to compute accuracy and fluency. In such a cascaded structure, each module is optimized on its own, which makes global optimality hard to guarantee—and errors accumulate from one module to the next.

The DolphinSOE engine takes a more advanced end-to-end deep learning approach: raw audio goes in, assessment results come out, and the neural network in between is trained as a single unit under one unified objective. The acoustic model is a dual-head LSTM:

• Feature extraction: the audio goes through pre-emphasis, framing, windowing, Short-Time Fourier Transform (STFT), mel filtering, and mean removal to produce fbank features.
• Dual-head LSTM acoustic model: stacked LSTMs take the fbank features and feed two output branches trained with multi-task learning—one branch predicts text, the other predicts phonemes. Both branches share the same LSTM representation and produce text and phoneme alignments simultaneously in a single forward pass. The text branch typically uses CTC to align audio frames with the character sequence and applies beam search to obtain candidate word sequences; the phoneme branch produces frame-level phoneme alignments.
• Decoding and fusion: candidates from the text branch are refined with a language model into the full recognition result hypothesis; the phoneme branch's output feeds phoneme-level scoring directly. Coming from the same trunk and produced at the same time, this is why the API can return both the text result hypothesis and the fine-grained phoneme result phonemes in one response.

For developers, the significance of this architecture is that you get not just a final score, but also finer-grained results for visualization and corrective feedback.

What Dimensions Does the Result Contain?

Assessment results are multi-dimensional, covering phonemes, intonation, fluency, pausing, completeness, and more. However, the dimensions differ across languages and question types, depending on the characteristics of the language and the design of the question type. For DolphinSOE's English question types, the most-watched dimensions are:

In addition, after alignment, the engine tags every word in the reference text with a type label describing what happened in this particular utterance. Insertions, deletions, and misreadings are identified right here:

If you are designing a product, these fields determine whether you can turn "scores" into "understandable learning feedback." To see how these dimensions appear in a user interface, try our demo page.

A Real Response Example

Below is a real response from the English word evaluation mode (reference text "good morning"), trimmed for brevity. You can clearly see how the scores drill down from the overall score to individual phonemes:

{
  "overall": 98.86,
  "accuracy": 92.73,
  "refText": "good morning",
  "hypothesis": "good morning",
  "wordInfo": [
    {
      "refText": "good",
      "hypothesis": "good",
      "type": "normal",
      "score": 96.04,
      "accuracy": 97.71,
      "syllables": [
        {
          "syllable": "g_uh_d",
          "score": 97.71,
          "phonemes": [
            { "phoneme": "g", "score": 94.93, "startTime": 340, "endTime": 500 },
            { "phoneme": "uh", "score": 99.58, "startTime": 500, "endTime": 570 },
            { "phoneme": "d", "score": 99.54, "startTime": 570, "endTime": 640 }
          ]
        }
      ]
    },
    {
      "refText": "morning",
      "type": "normal",
      "score": 99.41,
      "accuracy": 99.29,
      "syllables": [
        {
          "syllable": "m_ao_r",
          "score": 99.26,
          "phonemes": [
            { "phoneme": "m", "score": 98.77, "startTime": 640, "endTime": 700 },
            { "phoneme": "ao", "score": 99.75, "startTime": 700, "endTime": 830 },
            { "phoneme": "ao", "score": 99.75, "startTime": 830, "endTime": 920 }
          ]
        },
        {
          "syllable": "n_ih_ng",
          "score": 99.30,
          "phonemes": [
            { "phoneme": "n", "score": 98.06, "startTime": 920, "endTime": 1010 },
            { "phoneme": "ih", "score": 99.99, "startTime": 1010, "endTime": 1140 },
            { "phoneme": "ng", "score": 99.86, "startTime": 1140, "endTime": 1420 }
          ]
        }
      ]
    }
  ]
}

Note the hierarchy: overall / accuracy sit at the top level → each wordInfo carries a word-level score and accuracy → drill down to syllables → down to the smallest unit, phonemes.

Response hierarchy: overall → word → syllable → phoneme

This "overall → word → syllable → phoneme" hierarchy is the core of DolphinSOE's multi-level fine-grained feedback.

For the full field reference of the English word question type, see the API documentation.

Can the Score Range and Leniency Be Adjusted?

The engine provides a set of tunable parameters for different user scenarios:

• scale: the scoring scale (1–100), standardizing the output range.
• looseness: leniency (0–9); higher values are more forgiving.
• ratio: an adjustment coefficient (0.8–1.5) for fine-tuning overall scores.
• precision: scoring precision.

For practice scenarios you can raise looseness so users feel more comfortable speaking; for serious exams you can tighten it so scoring is stricter.

For parameter configuration, see the API documentation. If you would like to discuss tuning strategies for your specific scenario, feel free to contact us for support.

From a Developer's View, What Happens in One Call?

Chaining the steps above together, a typical API call looks like this:

1. The client records audio with the SDK, passing the language, question type, reference text, and other parameters.
2. The server first applies voice activity detection (VAD) to trim silence, then extracts features (fbank).
3. The acoustic model produces text and phoneme alignments in one forward pass; the language model refines the text branch.
4. The scoring modules produce per-dimension scores.
5. Structured JSON returns to the client, and the front end renders scores and highlights.

If you are already at the selection or PoC stage, an efficient path is usually: feel the effect on the demo page first, then complete technical validation against the API documentation.

Conclusion

The "magic" of a pronunciation assessment API is essentially an intelligent pipeline from audio to multi-dimensional scores: capture → recognition and alignment (end-to-end deep learning) → multi-dimensional scoring → feedback.

DolphinSOE pronunciation assessment has been running reliably in the Japanese market, currently serving about 100k calls per day, with customers ranging from language-learning apps to schools and cram schools. This is continued proof that the service has been verified under real high-concurrency, classroom-grade workloads, with stability and latency ready for production launch.

Beyond English, DolphinSOE also offers Japanese pronunciation assessment covering word, sentence, and paragraph question types, fitting the learning and assessment needs of Japanese-language education.

To learn more about integration, response formats, and question type support, see the API documentation. For pricing, solution advice, or business discussions, contact us.

Pronunciation assessment product UI example