Invoker requires port :9000, or else will not work.

[gotoAndBliss]

I'm trying to migrate from POW, but lets give Invoker the benefit of the doubt and greenfield a new app rails create newapp && cd newapp. I did..

gem install invoker

My invoker.ini says..

[whisper]
directory = /Users/mycomputer/Sites/newapp
command = bundle exec rails s

I run sudo invoker setup

I run the app with .. invoker start invoker.ini

Looks like it's booting up!

whisper : => Booting Thin
whisper : => Rails 4.0.13 application starting in development on http://0.0.0.0:9000
whisper : => Run `rails server -h` for more startup options
whisper : => Ctrl-C to shutdown server
whisper : Thin web server (v1.6.3 codename Protein Powder)
whisper : Maximum connections set to 1024
whisper : Listening on 0.0.0.0:9000, CTRL+C to stop

But... curl whisper.dev -> Operation Timeout.
Strange. https://0.0.0.0:9000 -> This site can’t provide a secure connection 0.0.0.0 sent an invalid response.
Weird. http://0.0.0.0:9000/ -> Boots up my homepage without SSL support. So anything CORS fails. Once I try and click on a subdomain. It's a big fail.

[gnufied]

You can't retrieve https://0.0.0.0:9000 because obviously thin doesn't do https. So that part is correct.

However whisper.dev should work. Which OS is this? Since you said Pow, I am assuming this is OSX. Also which version of OSX?

@iffyuva @kgrz can you double check the behaviour? I don't use OSX anymore..

[gotoAndBliss]

OSX 10.12.5 (16F73) - Sierra

Good point about Thin. I didn't think of that. Out of curiosity, what was POW booting up that could have otherwise made it SSL compatible? Or what would I need to configure within Invoker?

[gotoAndBliss]

I updated my command as so

[whisper]
directory = /Users/peanut_butter/Sites/whisper.me
command = bash -lc "rvm ruby-2.1.5@whisper do bundle exec thin start --ssl"

Seems to try and boot an SSL version, but alas, no signed SSL so Chrome displays its warning page, and I can only access it by going directly to 0.0.0.0:3000 . No subdomains work. No whisper.dev.

[kgrz]

@gotoAndBliss The CORS errors are expected when you request AJAX resources from a different domain (in this case, whisper.dev), depending on the server IP and port. Say if you have your API server proxied to whisper.dev/api, you might not get any CORS errors.

I'm trying to understand if this will be a problem of invoker or not. Up till this point, it doesn't seem so.

[gotoAndBliss]

@kgrz Ha, I'm six hours deep into this. I would do anything to even have it boot under whisper.dev . It seems to only work as http://0.0.0.0:9000, and then the rest of the site is inaccessible because it's all subdomains that need a whisper.dev domain.

[kgrz]

Hmm, so two problems here. You can't access the page on whisper.dev. This either could mean that the page shows a "invoker can't find a process" error, or that it's making a request to your rails server, but all the JS AJAX calls are failing for some reason. What among these two best describes the issue you're facing?

[kgrz]

Also, when you mentioned this:

Seems to try and boot an SSL version, but alas, no signed SSL so Chrome displays its warning page

Did you click on the "ADVANCED" link on that warning, and hit "proceed"? I'm trying to map the exact issue you're facing.

[gotoAndBliss]

Thank you @kgrz for your attention.

I'm using this as my config now

[whisper]
directory = /Users/peanut_butter/Sites/whisper.me
command = bash -lc "rvm ruby-2.1.5@whisper do bundle exec thin start --ssl"

Which boots up as so

Starting process - whisper waiting for 1 seconds...
whisper : Warning! PATH is not properly set up, '/Users/peanut_butter/.rvm/gems/ruby-2.1.5@whisper/bin' is not at first place.
whisper :          Usually this is caused by shell initialization files. Search for 'PATH=...' entries.
whisper :          You can also re-add RVM to your profile by running: 'rvm get stable --auto-dotfiles'.
whisper :          To fix it temporarily in this shell session run: 'rvm use ruby-2.1.5@whisper'.
whisper :          To ignore this error add rvm_silence_path_mismatch_check_flag=1 to your ~/.rvmrc file.
whisper : Using rack adapter
whisper : DEPRECATION WARNING: Support for Rails < 4.1.0 will be dropped. (called from <top (required)> at /Users/peanut_butter/Sites/whisper.me/config/application.rb:5)
whisper : Thin web server (v1.6.3 codename Protein Powder)
whisper : Maximum connections set to 1024
whisper : Listening on 0.0.0.0:3000, CTRL+C to stop

If I go to whisper.dev or https://whisper.dev , I see in Chrome..

This site can’t be reached
This site on the company, organization or school intranet has the same URL as an external website. 
Try contacting your system administrator.

This either could mean that the page shows a "invoker can't find a process" error, or that it's making a request to your rails server, but all the JS AJAX calls are failing for some reason. What among these two best describes the issue you're facing?

I do not seem to be getting any invoker errors. The Invoker server does not get hit when I go to whisper.dev .

If I run scutil --dns

I see no .dev extensions so that tells me that maybe Invoker is not creating the DNS at all.

Did you click on the "ADVANCED" link on that warning, and hit "proceed"? I'm trying to map the exact issue you're facing.

Yes, eventually I went through which brings me to a broken SSL homepage. Again, this only works if I go to https://0.0.0.0:9000 , and from there my site is inaccessible because I work on subdomains within it. For example subdomain.whisper.dev. Since the TLD doesnt' register, the subdomains won't either.

[iffyuva]

@gotoAndBliss not having .dev resolver explains it. Im successfully using invoker with https and subdomain support. I need following info:

Do you have these files present? If so, what are their contents?

• /etc/resolver/dev
• ~/.invoker/config
• /Library/LaunchDaemons/com.codemancers.invoker.firewall.plist

[iffyuva]

@gotoAndBliss simply put, we need to validate the setup according to this file https://github.com/code-mancers/invoker/blob/master/lib/invoker/power/setup/osx_setup.rb Also, please turn off and turn on your wifi once. I wen't M$ Windowz way, and restarted my machine once just to be sure :D

[gotoAndBliss]

@iffyuva

After reinstalling, rebooting a few times, I was able to get dev back. Equally, I am able to get to an Invoker page that states

Application not running
Invoker did not get any response. Please check if the application is running.

And I can get that error page by accessing whisper.dev. Awesome!

/etc/resolver/dev

nameserver 127.0.0.1
port 23400

 cat ~/.invoker/config
---
:http_port: 23403
:https_port: 23404
:tld: dev
:dns_port: 23400

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.codemancers.invoker</string>
<key>ProgramArguments</key>
<array>
<string>sh</string>
<string>-c</string>
<string>echo "rdr pass on lo0 inet proto tcp from any to any port 80 -> 127.0.0.1 port 23403
rdr pass on lo0 inet proto tcp from any to any port 443 -> 127.0.0.1 port 23404" | pfctl -a 'com.apple/250.InvokerFirewall' -f - -E</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>UserName</key>
<string>root</string>
</dict>
</plist>

I went ahead and rebooted, and restarted WIFI as well. Same results.

Thanks @iffyuva

[kgrz]

Ah, so now the invoker error page is showing up, but requests are not going through? One thing I can point out is that since the port number (9000) is non-80, the invoker.ini should have a port config that is set to 9000.

Before you do that, try accessing https://whisper.dev:9000

[iffyuva]

In addition to above check, please paste info of invoker list

[gotoAndBliss]

@kgrz What do you mean by "non-80"? When I booted up thin on SSL it defaulted to 9000. So I started setting the port to that in the invoker.ini.

To keep things simple, but still produce the same current error ( going to braidio.dev produces the Application Not Running error ).. I removed the port call from invoker.ini so I have this now

[braidio]
directory = /Users/peanut_butter/Sites/whisper.me
command = bash -lc "rvm ruby-2.1.5@whisper do bundle exec thin start --ssl"

Which boots as ->

braidio : Warning! PATH is not properly set up, '/Users/peanut_butter/.rvm/gems/ruby-2.1.5@whisper/bin' is not at first place.
braidio :          Usually this is caused by shell initialization files. Search for 'PATH=...' entries.
braidio :          You can also re-add RVM to your profile by running: 'rvm get stable --auto-dotfiles'.
braidio :          To fix it temporarily in this shell session run: 'rvm use ruby-2.1.5@whisper'.
braidio :          To ignore this error add rvm_silence_path_mismatch_check_flag=1 to your ~/.rvmrc file.
braidio : Using rack adapter
braidio : DEPRECATION WARNING: Support for Rails < 4.1.0 will be dropped. (called from <top (required)> at /Users/peanut_butter/Sites/whisper.me/config/application.rb:5)
braidio : Thin web server (v1.6.3 codename Protein Powder)
braidio : Maximum connections set to 1024
braidio : Listening on 0.0.0.0:3000, CTRL+C to stop

And my invoker list ->

  +---------------------------------------+-------+------+--------------+---------------------------------------------+
  | dir                                   | pid   | port | process_name | shell_command                               |
  +---------------------------------------+-------+------+--------------+---------------------------------------------+
  | /Users/peanut_butter/Sites/whisper.me | 14267 |      | braidio      | bash -lc "rvm ruby-2.1.5@whisper do bundl.. |
  +---------------------------------------+-------+------+--------------+---------------------------------------------+

[iffyuva]

@gotoAndBliss command is wrong! It should have that -p $PORT stuff. port cannot be empty in invoker list. If its empty, invoker doesn't know where to route the request.

[gotoAndBliss]

@iffyuva Ah ok so I added the port like so

[braidio]
directory = /Users/peanut_butter/Sites/whisper.me
command = bash -lc "rvm ruby-2.1.5@whisper do bundle exec thin start --ssl -p $PORT"

Then the 9000 gets added back

 +---------------------------------------+-------+------+--------------+---------------------------------------------+
  | /Users/peanut_butter/Sites/whisper.me | 14998 | 9000 | braidio      | bash -lc "rvm ruby-2.1.5@whisper do bundl.. |
  +---------------------------------------+-------+------+--------------+---------------------------------------------+

And the same error persists if I curl braidio.dev.

Also tried this ->

curl braidio.dev:9000
curl: (52) Empty reply from server

[iffyuva]

So, does curl whisper.dev work? Also, can you paste the error again? Probably screenshot.

[gotoAndBliss]

Ah! A major breakthrough.

So curl braidio.dev:9000 gets an empty reply from server. But if I use Chrome, it loads the site.

The only caveat at this point is that the SSL is not configured. But I'm guessing this is something I have to do myself via self-signing and Invoker doesn't do that part out-of-box?

[kgrz]

But I'm guessing this is something I have to do myself via self-signing and Invoker doesn't do that part out-of-box?

Yes :)

[gotoAndBliss]

THANK YOU EVERYONE!!

[iffyuva]

You don't have to use :9000, invoker takes care of forwarding request from braidio.dev to port 9000.

[gotoAndBliss]

@iffyuva That doesn't seem to be true. Application Not Found occurs unless I specify a port.

[kgrz]

You can either specify the command like bundle exec rails -p $PORT or you can set the port configuration in invoker.ini

[whisper]
command=your command here
directory=same directory you have
port = 9000

[gotoAndBliss]

@kgrz Yes I tried both those separately and together, but I must specify the port in the url or it will not find the application.

[kgrz]

Hmm, that looks odd. The whole point of the project is to avoid users specifying a port in the URL :(

[gnufied]

You don't need to specify ssl params with thin. If you want to use https then invoker can take care of that. The ssl connections will be terminated at Invoker and forwarded to thin. i.e - drop the ssl params from thin and try.

[gnufied]

So what I am saying is, removing --ssl from thin arguments and then if everything configured properly you should be able to access your website on https://braidio.dev

[gotoAndBliss]

@gnufied Thanks for the advise. I dropped all the additional SSL info, and it still remains an insecure untrusted connection.

Perhaps my browser is caching this sort of certificate? Is there a way to refresh it.

Also, there's no way that just https://braidio.dev works. The port always has to be included. :(

[gotoAndBliss]

Perhaps Invoker doesn't support thin?

[gnufied]

Invoker doesn't care about web servers in general - so yeah it does support Thin.

I suspect, what is happening is - there is some left over pf filter rules by Pow and Invoker isn't the one that is receiving the request when you access your service using https://braidio.dev. So there is another filter rule which is directing traffic on port 80 and port 443 to elsewhere. Make sure you uninstalled pow properly and restarted the network.

[iffyuva]

@swanandp let me try to reproduce the issue, and update you. Thanks.

[swanandp]

Let me know if you want any debugging information. Might help others

[iffyuva]

@swanandp sure. For now, please rollback to old version, while we debug this issue.

[gotoAndBliss]

@iffyuva How does one rollback? And what version would be last working?

[iffyuva]

I have invoker 1.5.1. Can you try that version @gotoAndBliss

You follow these steps

$ gem install invoker -v 1.5.1

$ invoker -v
1.5.3
$ invoker _1.5.1_ -v
1.5.1
$ invoker _1.5.1_ start <ini file>

[gotoAndBliss]

@iffyuva Ah thanks so much. Tried 1.5.1, and 1.4.1, still need to assign a port though.

[iffyuva]

@gotoAndBliss @swanandp I have pushed a branch for debugging em-proxy gem. Can you send me logs for your app(s). Steps:

> git clone [email protected]:code-mancers/invoker.git
> cd invoker
> git checkout issue-194-debug-em-proxy
> bundle
> DEBUG_EM_PROXY=1 ./bin/invoker start <your-ini-file-path-here>

You should see sample logs like this: https://gist.github.com/iffyuva/c6327944a998c83411e1b4d4f045304e

Please gist them, and post them here. Thanks.

EDIT: Updated branch name in instructions.

[swanandp]

@iffyuva I am pretty sure this is a networking setup issue, and not an Invoker issue. I just uninstalled Invoker and tried Pow, and that isn't working as well. ( This is from my co-working space ).

I will try from home setup, and if everything works, let you know.

[gotoAndBliss]

@iffyuva Here you are sir. Thank you so much!!

https://gist.github.com/gotoAndBliss/299c3c95d6008baac88c755fd79b3ad0

[swanandp]

So, I got this working after running this:

sudo pfctl -f /etc/pf.conf
sudo pfctl -e

Contents of my pf.conf

scrub-anchor "com.apple/*"
nat-anchor "com.apple/*"
rdr-anchor "com.apple/*"
dummynet-anchor "com.apple/*"
anchor "com.apple/*"
load anchor "com.apple" from "/etc/pf.anchors/com.apple"

[swanandp]

And before that, I tried enabling port-forwarding: sudo sysctl -w net.inet.ip.fw.enable=1,

from the Pow troubleshooting guide: https://github.com/basecamp/pow/wiki/troubleshooting

[iffyuva]

@swanandp nice that you have solved it yourself :D

[gotoAndBliss]

@swanandp

Interesting response. I tried sudo sysctl -w net.inet.ip.fw.enable=1, but got sysctl: unknown oid 'net.inet.ip.fw.enable'. I'm using Mac OS X Sierra 10.12.5.

So I started looking for any kind of port forwarding ( I admittedly have no idea what I'm doing ). But I found this.. /etc/sysctl.conf

And messed with this net.link.ether.inet.arp_unicast_lim=0, set it to 1 and ran sudo pfctl -f /etc/pf.conf; sudo pfctl -e but that didn't work.

Out of curiosity, would using RVM somehow affect this? For RVM with pow, they recommend making a .powrc file that writes this ..

if [ -f "$rvm_path/scripts/rvm" ]; then
  source "$rvm_path/scripts/rvm"
  rvm use .
fi

Note, for RVM I did do all the recommended Invoker RVM things..

rvmsudo invoker setup
command = bash -lc "rvm ruby-2.1.5@whisper do bundle exec thin start --ssl -p $PORT"

[swanandp]

@gotoAndBliss

. Interesting response. I tried sudo sysctl -w net.inet.ip.fw.enable=1, but got sysctl: unknown oid 'net.inet.ip.fw.enable'. I'm using Mac OS X Sierra 10.12.5.

So, oid work in groups as well, you can successively try:

sysctl net
sysctl net.inet
sysctl net.inet.ip

Be warned, there is a lot of output to sift through:

➜  brainy git:(master) sysctl net | wc -l
     431
➜  brainy git:(master) sysctl net.inet | wc -l
     205
➜  brainy git:(master) sysctl net.inet.ip | wc -l
      48
➜  brainy git:(master) sysctl net.inet.ip.forwarding | wc -l
       1
➜  brainy git:(master)

Out of curiosity, would using RVM somehow affect this?

I don't think so. RVM comes into play after the port forwarding has happened. It's further up the stack.

[gotoAndBliss]

@swanandp

I tried this

sudo sysctl -w net.inet.ip.forwarding=1
net.inet.ip.forwarding: 0 -> 1

then

sudo pfctl -f /etc/pf.conf; sudo pfctl -e

But alas, still need to designate the port. My other net.inet.ip 's

https://gist.github.com/gotoAndBliss/049fa7e5ac94b29f0976042f954f019c

[gotoAndBliss]

Today I went a bit further. Curious if anyone can see something blatantly incorrect about my approach here..

Started also forwarding ip6

sudo sysctl -w net.inet6.ip6.forwarding=1
net.inet6.ip6.forwarding: 0 -> 1

Created a new forwarding rule

sudo touch /private/etc/pf.anchors/org.user.forwarding

With

rdr pass on lo0 inet proto tcp from any to any port 80 -> 127.0.0.1 port 9000
rdr pass on en0 inet proto tcp from any to any port 80 -> 127.0.0.1 port 9000
rdr pass on en1 inet proto tcp from any to any port 80 -> 127.0.0.1 port 9000

Added it to /private/etc/pf.conf

Parse tested it sudo pfctl -vnf /etc/pf.anchors/org.user.forwarding . It was correct.

Rebooted, and ran sudo pfctl -f /etc/pf.conf; sudo pfctl -e .

Still need to include the port 9000 to access domain.

[wkrsz]

Could it be caused by #184?

[gotoAndBliss]

@WojtekKruszewski Thanks so much for the recommendation. I was previously on 2.1.5, and went to 2.3.0, but the problem still persisted.

[Arkham]

I have bumped into the same issue. I can access my app fine by visiting localhost:9000, but if I visit web.dev I'm greeted by the usual

Application not running
Invoker did not get any response. Please check if the application is running.

I'm using the latest version (1.5.4) and these are my configuration files:

$ cat /etc/resolver/dev
nameserver 127.0.0.1
port 23400

$ cat ~/.invoker/config
---
:http_port: 23403
:https_port: 23404
:tld: dev
:dns_port: 23400

$ cat /Library/LaunchDaemons/com.codemancers.invoker.firewall.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.codemancers.invoker</string>
<key>ProgramArguments</key>
<array>
<string>sh</string>
<string>-c</string>
<string>echo "rdr pass on lo0 inet proto tcp from any to any port 80 -> 127.0.0.1 port 23403
rdr pass on lo0 inet proto tcp from any to any port 443 -> 127.0.0.1 port 23404" | pfctl -a 'com.apple/250.InvokerFirewall' -f - -E</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>UserName</key>
<string>root</string>
</dict>
</plist>

$ head -1 Procfile
web: sleep 7 && zeus server -p $PORT

and here is the output of invoker list

$ invoker list
  +-----+-----+------+------------------+---------------------------------------------+
  | dir | pid | port | process_name     | shell_command                               |
  +-----+-----+------+------------------+---------------------------------------------+
  |     | 828 | 9000 | web              | sleep 7 && zeus server -p 9000              |
  +-----+-----+------+------------------+---------------------------------------------+

[swanandp]

@Arkham Does this work for you: #194 (comment) ( It did for me, but didn't for @gotoAndBliss )

[Arkham]

Hey @swanandp! I've tried every suggestion in the thread and the current situation is:

• localhost:9000 ✅
• web.dev 🔴 Invoker did not get any response. Please check if the application is running.
• web.dev:9000 🔴 This site can’t be reached web.dev refused to connect.

[Dan2552]

@Arkham I had the same and was resolved by running with binding 0.0.0.0 (e.g. rails s -b 0.0.0.0)

[Arkham]

@Dan2552 amazing, works for me too!

[gotoAndBliss]

Would there be something similar for running rails thin?

[gnufied]

thin also has a -a option that you can use to specifying binding address.

[gotoAndBliss]

@gnufied Ah ok, because I tried -a 0.0.0.0, but nothing happened. Should the address always be 0.0.0.0?

[gotoAndBliss]

Purchased a brand new macbook, fresh install. Same issue. 😭

[gotoAndBliss]

I discovered that if I greenfielded a new app, everything works as expected. No port required.

I tried stripping down the problem app to practically nothing ( still in the process of stripping ). And everytime I make the call I get this error

whisper : 4736210368:error:1408F09C:SSL routines:ssl3_get_record:http request:ssl/record/ssl3_record.c:322:

Not sure if that means anything to anyone on this thread.

[oleg-vinted]

Might be related: #248