Neural Blind Source Separation and Diarization for Distant Speech Recognition

This is a repository of neural full-rank spatial covariance analysis with speaker activity (neural FCASA). Neural FCASA is a method for jointly separating and diarizing speech mixtures without supervision by isolated signals.

Installation

pip install git+https://github.com/b-sigpro/neural-fcasa.git

Inference

Using model pre-trained on the AMI corpus

python -m neural_fcasa.dereverberate input.wav input_derev.wav
python -m neural_fcasa.separate one hf://b-sigpro/neural-fcasa input_derev.wav output.wav

General usage

Dereverberation

python -m neural_fcasa.dereverberate input.wav output.wav

This is just a thin wrapper of nara_wpe. The options are as follows:

• --n_fft: Window length of STFT (default=512)
• --hop_length Hop length of STFT (default=160)
• --taps Tap length of WPE (default=10)
• --delay Delay of WPE (default=3)

Separation and diarization

python -m neural_fcasa.separate one /path/to/model/ input.wav output.wav

The options are as follows:

• --thresh: Threshold to obtain diarization result (default: 0.5)
• --out_ch: Output channel index for Wiener filtering (default: 0)
• --medfilt_size: Filter size of median postfiltering (default: 11)
• --dump_diar: Dump diarization results as a pickle file (default: false)
• --noi_snr: SNR for white noise added to the separated result. No noise is added with None. (default: None)
• --normalize: Perform normalization of the separated result (default: false)
• --device: device type (e.g., cuda and cpu) for inference. (default: cuda)

We used the following configuration for the evaluation in the paper:

python -m neural_fcasa.separate one hf://b-sigpro/neural-fcasa --dump_diar --noi_snr=40 --normalize input.wav output.wav

Limitations

The current inference script has the following limitations, which we are addressing to solve:

• The # mics. must be the same as that at the training (8).
• The input length must be less than 50 seconds due to the max. length of the positional encoding (5000).
• The performance will be maximized by making the input length the same as that at the training (10 seconds).

Training

The training script is compatible with PyTorch Lightning >= 2.2.3.

The training job script for AI Bridging Cloud Infrastructure (ABCI) is attached on recipes/neural-fcasa. We will add a detailed instruction for training soon.

Reference

@inproceedings{bando2023neural,
  title={Neural Blind Source Separation and Diarization for Distant Speech Recognition},
  author={Yoshiaki Bando and Tomohiko Nakamura and Shinji Watanabe},
  booktitle={accepted for INTERSPEECH},
  year={2024}
}

Acknowledgement

This work is based on results obtained from a project, Programs for Bridging the gap between R&D and the IDeal society (society 5.0) and Generating Economic and social value (BRIDGE)/Practical Global Research in the AI × Robotics Services, implemented by the Cabinet Office, Government of Japan.