is it supported with Batch size >1 ?

[QuPengfei]

Hi all,

is it supported with bs >1? found the following:

if (batch_size > 1)
MODEL_ASSERT(
("llama arch only supports continuous batching inference when giving multi prompts.", lctx.cont_batching));

thanks

[zhentaoyu]

Yes, it is supported, but only for a few model architectures. Please refer to https://github.com/intel/neural-speed/blob/main/docs/continuous_batching.md

[zhentaoyu]

Hi, @QuPengfei, if you have no other questions, we will close this issue. Thanks.