Debugging JVM Programs on Heroku

2012-09-28 / All Blog posts

Sometimes there is no substitute for debugging a remote application. The Java virtual machine provides the JPDA facility for this. JPDA is flexible, and can be configured in a variety of ways. Two attachment mechanisms are supported for debugging remote applications: inbound connections, whereby a debugging process on your machine attaches to a remote process via designated port at a specific IP address, and outbound connections, whereby a debugging process on your local machine listens to a designated port for a remote process to attach to it. JPDA can be used by all JVM-based languages, such as Java, Scala, Groovy and Clojure.

Heroku only allows one incoming port, so because an incoming port is used by Heroku to connect to the hosted app, debug connections originating from your IDE will not succeed. Instead, you must set up your Heroku application to initiate an outbound connection for debugging. This cannot be done if your app uses more than one dyno, so you must scale your Heroku back to one dyno before you can remotely debug it, like this:
heroku scale web=1

A Heroku app can initiate an outbound connection for debugging with or without a proxy server. The examples below use the Java debugger (jdb) and IntelliJ IDEA, but you could equally well set up a debug configuration for Eclipse in a similar manner. The settings below were used with a Play 2 application with Java and Scala controller classes.

Tip: heroku restart will restart your Heroku app using your existing slug. This accomplishes the same thing as:
heroku scale web=0
heroku scale web=1

The other way you can restart your app is by building a new slug, which then automatically starts. You can do this by checking in a bogus file:

date>ignoreme.txt; git add ignoreme.txt; git push heroku

Warning: Your Heroku app will crash if there is no listening process. Use the heroku logs command to check for a crash.

$ heroku logs
2012-09-29T18:20:53+00:00 heroku[web.1]: Starting process with command `target/start -Dhttp.port=${PORT} ${JAVA_OPTS}`
2012-09-29T18:20:55+00:00 app[web.1]: ERROR: transport error 202: connect failed: Connection refused
2012-09-29T18:20:55+00:00 app[web.1]: ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
2012-09-29T18:20:55+00:00 app[web.1]: JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [../../../src/share/back/debugInit.c:741]
2012-09-29T18:20:55+00:00 app[web.1]: FATAL ERROR in native method: JDWP No transports initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)
2012-09-29T18:20:56+00:00 heroku[web.1]: Process exited with status 134
2012-09-29T18:20:56+00:00 heroku[web.1]: State changed from starting to crashed

Remote Debugging Without a Proxy Server

If your local machine is accessible from the Internet at an IP address (a domain such as blah.no-ip.info would work equally well):

jdb -listen 9999& # Do not type this line if you are using an IDE
# If using an IDE, start debugging now

# Assumes that eth0 accesses the Internet; only works if you are not behind NAT
IPADDR=`ifconfig eth0|grep "inet addr"|awk -F: '{print $2}'|awk '{print $1}'`

# If you are behind NAT, you will need to use a dynamic DNS and set IPADDR to the machine name instead:
IPADDR=mycomputer.no-ip.info # modify to suit

# This is a really long line. I wrapped it, but you should not:
heroku config:add JAVA_OPTS='-Xdebug
  -Xrunjdwp:transport=dt_socket,address=$IPADDR:9999
  -Xms512M -Xmx1024M -Xss1M -XX:+CMSClassUnloadingEnabled
  -XX:MaxPermSize=256M -XX:+UseCompressedOops'

heroku restart

Here is what the run configuration for IntelliJ IDEA looks like. Note that debugger mode is set to listen, which implies server="n". You need to launch this run configuration before restarting your Heroku app.

Bash Script Implementation

I put the bash scripts in a gist:

Remote Debugging With a Proxy Server

If your local machine is hidden from the Internet by a proxy server at blah.domain.com, open a tunnel to it from your local machine, and listen to it:

# Assumes that eth0 accesses the Internet; only works if you are not behind NAT
IPADDR=`ifconfig eth0|grep "inet addr"|awk -F: '{print $2}'|awk '{print $1}'`

# If you are behind NAT, you will need to use a dynamic DNS and set IPADDR to the machine name instead:
IPADDR=mycomputer.no-ip.info # modify to suit

ssh -NR *:9999:localhost:9999 $IPADDR&

jdb -listen 9999& # Do not type this line if you are using an IDE

If using an IDE, start debugging now.

# This is a really long line. I wrapped it, but you should not:
heroku config:add JAVA_OPTS='-Xdebug
  -Xrunjdwp:transport=dt_socket,address=$IPADDR:9999
  -Xms512M -Xmx1024M -Xss1M -XX:+CMSClassUnloadingEnabled
  -XX:MaxPermSize=256M -XX:+UseCompressedOops'

heroku restart

Disabling Remote Debugging

The JAVA_OPTS value set earlier will remain in effect across multiple restarts of your Heroku app until you change it. To disable remote debugging, redefine the environment variable without the -Xdebug and -Xrunjdwp options:

# This is a really long line. I wrapped it, but you should not:
heroku config:add JAVA_OPTS='-Xms512M -Xmx1024M -Xss1M
  -XX:+CMSClassUnloadingEnabled
  -XX:MaxPermSize=256M -XX:+UseCompressedOops'

heroku restart

Saving the Run Configuration

If you enable the Share checkbox in the IntelliJ IDEA run configuration dialog box, the definition will be written to .idea/runConfiguration/. Normally you would not check in the contents of .idea/ to your source code repository, but this subdirectory is an exception, and because this run configuration has no local dependencies it can be shared without modification amongst all of your developer team.

$ cat .idea/runConfigurations/Heroku_remote.xml 
<component name="ProjectRunConfigurationManager">
  <configuration default="false" name="Heroku remote" type="Remote" factoryName="Remote">
    <option name="USE_SOCKET_TRANSPORT" value="true" />
    <option name="SERVER_MODE" value="true" />
    <option name="SHMEM_ADDRESS" value="javadebug" />
    <option name="HOST" value="localhost" />
    <option name="PORT" value="9999" />
    <method />
  </configuration>
</component>

comments powered by Disqus