Better handling of Pool and job reserve for multiple connections

[nesquena]

There is an issue with the pool concept and then with put and reserve. We should change beaneater_test with https://github.com/beanstalkd/beaneater/blob/master/test/beaneater_test.rb#L6 and test with multiple connections.

The issue is around how reserving works in this case:

@beanstalk = Beaneater::Pool.new(['127.0.0.1:11300', '127.0.0.1:11301'])
@tube = @beanstalk.tubes["foo"]
@tube.put("bar") # <-- put onto connection one
@tube.put("baz") # <-- put onto connection one

So now we have two jobs on connection one. If we now call

@tube.reserve

Reserve could randomly pick connection two in which case there are no jobs and it would hang forever. Also other variations of this problem when 2 jobs fall into one connection and one into another and calling reserve will eventually start to hang up even though the other connection still has jobs.

[nicotaing]

yeah i see, we should be more smart about it, i think we should stop using transmit_to_rand for reserving jobs.

Cycling through all connections it's neither the best approach, does threading sounds too crazy?

[zdzolton]

Perhaps an approach to try for the reserve method would be to look at the current-waiting counter (from the stats command) for each connection, distributing the reserve commands across connections. The goal being to avoid starving any particular queue for consumers.

[iros]

Is there any progress on this issue? I'm rather blocked by this implementation as well. I use on average 3 connections and on average about 1/3 of my jobs don't ever run. Seems like doing a round robin until we find a job in a queue or find no jobs is the right way to go on this one... no?

[nesquena]

Yeah I can think of a several ways to handle this better then we are handling it now. We will tackle this soon although if anyone else wants to take a stab at a pull request, that'd be great.

[nicotaing]

I'm aware it's not the best way, there's a workaround by setting up a timeout and eventually it will reserve the job on the right tube:

job = @beanstalk.tubes.reserve(5) # 5 seconds

We'll come up with a better way, but let me know if that workaround works for you