How to evaluate an echo cancellation algorithm?
- On March 25, 2018
- Acoustic Echo Cancellation
When you encounter an echo problem in your application, network or devices it is obvious you need to introduce an echo canceller. Your next step is to explore at least two possible locations for performing the echo cancellation: cancelling the echo on the client side or doing it on the server side. The decision should be based on parameters like processing power, licensing, call capacity etc. Once you know where you want to place the echo canceller, you can find an echo cancellation software that looks suitable for your needs. Your next challenge is to evaluate the expected performance of the echo cancellation in your environment. In this post we will discuss evaluation methodologies that exists prior to actually integrating the echo cancellation in your system.
Off-line recorded files vs. real-time demo
In some cases, you would like to start with a stand-alone evaluation prior to integrating the echo canceller in your system. In order to do so there are usually two different approaches:
1. Running the AEC on audio files that were recorded from your system.
2. Using a stand-alone demo application.
At a first glance, the second approach seems to be more accurate since you are testing the echo cancellation in live calls. But, as a matter of fact the first approach is much more accurate and will provide you with results that are much closer to the ones you will eventually get in your live system. Why is that so?
The subtlety of echo cancellation
As we discussed in previous posts, an echo cancellation algorithm is not one size fit all and implementing an echo cancellation requires careful planning for different systems and usage scenarios. There are many factors that can influence the behavior of the echo even on the same hardware. Such factors might be:
- Different delays that are caused by different application processing.
- Different mic/spk synchronization that is caused by different application and network architectures.
- Different audio behavior caused by different audio stack APIs.
- Different echo behavior due to different usage scenarios, for example hands-free vs. handheld, multi-party call etc.
As a result, running echo cancellation on a stand-alone demo applications does not provide a good enough indication with regards to the behavior of the echo cancellation once integrated in your system. On the other hand, when you run the echo cancellation on files that you have recorded from your system – you get a much more accurate estimation of the final results. Now that we understand the importance of recorded files for the evaluation, the question is how to prepare good recordings?
Recorded audio files – dos and don‘ts
When preparing recorded files we want to have them capture real usage of your system and not some theoretical scenarios. For example, if your system usually serves two people talking then make sure to have two people talking when making the recordings. In addition please note the following:
- Make the recording in the location in which you plan to eventually implement the echo cancellation (e.g. on the server or on the client).
- If your system has different hardware and/or different usage scenarios – make sure to include them in the recorded files. You can generated different recorded files for each case.
- Make sure only one side in the call is generating echo, the other side can use headsets to prevent echo. When doing so we avoid howling that due to its nature cannot be addressed when dealing with offline recorded files. Don’t worry about howling since once you integrate a good echo canceller in your system there will be no howling in the live calls.
- Each recorded scenario should contain two audio files: one audio file contains the audio captured from the microphone (which also includes the echo) and the second audio file contains the audio that is going to be played on the speakers (and later be captured as echo in the microphone).
Contact us to discuss your requirements and our team will be happy to help you analyze your system and suggest the best approach to handle echo cancellation an improve audio quality in your system.