How to use multiple input parameters?

[joknjokn]

Hi,

Is it possible to use a model where I should input some additional parameters, in addition to the audio?

[Caldarie]

Hi @joknjokn

May I ask what the additional parameters are? I can implement this feature into the plugin, depending on its type

[joknjokn]

I'm trying BirdNet's bird song model. It benefits from getting latitude, longitude and week-number (turned into cosine or something). I actually think I got hardcoded lat/lng working, but not entirely sure it's correct... It's giving results however. Had to modify rawAudio to use multiInputsOutputs:

  final String model = 'assets/BirdNET_6K_GLOBAL_MODEL.tflite';
  final String label = 'assets/labels.txt';
  final String inputType = 'rawAudio';
  final int sampleRate = 48000;
  final int recordingLength = 144000;
  final int bufferSize = 2000;

TfliteAudioPlugin:
            case "rawAudio":
                // We need to feed in float values between -1.0 and 1.0, so divide the
                 // signed 16-bit inputs.
                for (int i = 0; i < recordingLength; ++i) {
                    floatInputBuffer[0][i] = inputBuffer[i] / 32767.0f;
                }
                float[][] floatInputBuffer2 = {new float[]{53.7958f, 13.43394f, -1, 1, 1, 0}};
                inputArray = new Object[]{floatInputBuffer, floatInputBuffer2};
                outputMap.put(0, floatOutputBuffer);
                tfLite.runForMultipleInputsOutputs(inputArray, outputMap);
                lastProcessingTimeMs = new Date().getTime() - startTime;

Also, I printed both input shapes:
Input shape: [1, 144000]
Input shape 2: [1, 6]

I know it's a mess. Just playing around at the moment. I'm entirely new at this.

[Caldarie]

Hmm, besides hard coding your inputs, I see no problem with it so far. As long as your model outputs something, then you’re doing something right.

Was this the answer you were looking for?

[joknjokn]

It also outputed before I added the multiInput. I think it might be more precise now, but difficult to say. It does recognize some common species.

However, I would like to make the lat/lng and week dynamic, so I guess the multiInput needs to be exposed to flutter. I think that would be a useful feature to your plugin. Maybe it could be made dynamic if the plugin lists the input shapes and takes the same amount of inputs? If that's possible.

[Caldarie]

Thank you for the suggestion. I will definitely take a shot at adding a dynamic input feature.

I will keep you updated.

[Caldarie]

If possible, could you share me your model to work with?

[joknjokn]

It's available from the source here (Creative Commons):
https://github.com/kahst/BirdNET-Lite

[Caldarie]

Many thanks!

[Caldarie]

hi @joknjokn,

I have experimented with the idea of dynamic inputs and have succeeded in implementing it on android. If you're interested in taking a look, you can find it on this branch here

So far, its limited to float values and one additional input; however, i do plan to make this dynamic as well.

Let me know your thoughts.

[joknjokn]

@Caldarie Sorry - forgot to respond and had my project on pause (because I also need to store recordings of the detections, and also can't find a way to create live spectrogram and spectrogram of the detections).

Either, I can soon try your solution, but is it possible to update this branch so it has all your other fixes/features as well? (Or is this dynamic functionality implemented in the main branch?).

[Caldarie]

Hi @joknjokn. No problems at all.

I have yet to implement multiple inputs on the master branch, but it is on my todo list. I do have an idea on how to tackle this problem, but may take time to do so. I will keep you updated if you’d like.

also, if you’re in interested, this plugin is compatible with spectrograms, though it’s yet to be tested. If you do have time to test this feature and make the necessary adjustments, I’d really appreciate it. Let me know if this up your alley.