Deploying JVM in tiny containers, be carefull!

I`ve deployed my rohrli.gamlor.info service into the smallest possible triton container. Except me, noone is going to use it anyway ^-^. This container has 128Mbyte of memory, and 1/16th CPU time. The OpenJDK JVM is known for consuming tons of memory. And it is somewhat true. However, the JVM also runs in a small container. In this blog post I give some recommendations. The problems exists in small Docker and Triton containers. I show everything for both.

Chrome & JVM eat memory generously

Chrome & JVM eat memory generously

Web App Example

The web app is just a ‘Hello World’ app. For better explaination, it also consumes tons of memory. I created the example in Scala, but the other JVM languages have the same issue. The source code for the example is here.

/** 
 * Snippet, memory limits 
 */
// Limit the amount of items in memory for this example.
// So, max ~192*256k=48MBytes
val maxMemoryItems = 192
val memoryPerItem = 256 * 1024
val memoryWasting = new ArrayBlockingQueue[Array[Byte]](maxMemoryItems)

/** 
 * Snippet, request handling
 */
// On each request, consume some memory.
// And keep things in memory to simulate memory use
case "/" :: Nil => {
    // On each request, consume some memory.
    // And keep things in memory to simulate memory use
    val chunkOfMemory = Array.ofDim[Byte](memoryPerItem)
    var addedToQueue = memoryWasting.offer(chunkOfMemory)

    // Throw things away until we've space
    while (!addedToQueue) {
        memoryWasting.poll()
        addedToQueue = memoryWasting.offer(chunkOfMemory)
    }

    val createdSoFar = createdItemsCounter.incrementAndGet()
    response.getWriter.write(
        s"""
            |{
            |"title":"Hello World! Let's use some memory",
            |"created-overall":"$createdSoFar",
            |"queue-size":"${memoryWasting.size()}",
            |"":"Hello. I'm so memory hungry. kthxbye!"
            |}
        """.stripMargin)
}

We compile and package the program into a Docker container. The Docker file is simple:

The we create the image with `build` and run it with `run`. With regular Docker we restrict the memory with the ‘-m 128m’ option to 128Mbyte. And with ‘–cpus 0.065’ we only give very little CPU time.
When using Triton, the ‘-m 128m’ option will find a sufficient large compute instance. Alternativly, you also can use ‘–label com.joyent.package=g4-highcpu-128M’ to create a tiny instance.
Once we started out container, we create some load with the ‘watch’ and ‘curl’ commands. Then, let`s watch the logs.

First Issue: Program Dies

The Triton instance won`t expirience any issue. But the plain Docker instance (Linux 4.10, Docker 17.04) suddenly stops working. =( The program got killed.

[root@gamlor-manjaro useless-demo-service]# docker run -m 128m --cpus 0.065 -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:1
2017-04-20 12:58:20.248:INFO::main: Logging initialized @5900ms to org.eclipse.jetty.util.log.StdErrLog
2017-04-20 12:58:23.348:INFO:oejs.Server:main: jetty-9.4.z-SNAPSHOT
2017-04-20 12:58:30.650:INFO:oejs.AbstractConnector:main: Started ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2017-04-20 12:58:30.748:INFO:oejs.Server:main: Started @16602ms
Killed

Hmmm….Why did it got killed? Aha…the memory is small, so we might ran out of memory and got killed. We can, while the container runs, inspect the memory usage of the JVM with `jstat`:

The penguin and whale are dangerous. Be careful!

The penguin and whale are dangerous. Be careful!


jstat is a bit confusing. All numbers are in KByte. The JVM has a few memory pools and each memory pool has a minimum (**MN), maximum (**MX) and current size (**C).
So, inside the regular Docker container it create really big memory pools. On my computer (16GB, Linux 4.10, Docker 17.04), OGCMX (old generation heap) memory is about 2.5GByte. However, on the Triton instance it is about 40Mybte large. So, with a 2.5GByte pool, if it actually uses more than 128MByte, it will die. For now, Docker just kills processes which use too much memory. However, the JVM sees all the hosts memory, so it creates giant memory pools.
However, the Triton container lies more convincingly, so the JVM sees the containers memory sizes and creates a sufficient small memory pool.
Well, there a solution to this issue. We just configure a small memory size with the `-Xmx` option. Plus I configure the JVM such that when it runs out of memory, it dumps the memory and dies.

After doing this, the JVM runs in the Docker container without any issue.

So Many Threads

Now, we configured the memory properly. However, there maybe more things we can configure properly? So, let`s see what threads are runing in the container with `jstack`:

##
# Local docker run 
##
# Run second version of our container
sudo docker run -d -m 128m --cpus 0.065 --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:2
# List the threads running
sudo docker exec jvm-on-small-container sh -c 'jstack `pgrep java`' | grep os_prio 
"qtp791452441-23" #23 prio=5 os_prio=0 tid=0x000056473bc03800 nid=0x42 waiting on condition [0x00007fefe4931000]
"Attach Listener" #22 daemon prio=9 os_prio=0 tid=0x000056473bf72800 nid=0x41 waiting on condition [0x0000000000000000]
"DestroyJavaVM" #21 prio=5 os_prio=0 tid=0x000056473bcd1800 nid=0x8 waiting on condition [0x0000000000000000]
"qtp791452441-19-acceptor-0@27b0949f-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #19 prio=3 os_prio=0 tid=0x000056473bf75000 nid=0x24 runnable [0x00007fefe4b33000]
"qtp791452441-17" #17 prio=5 os_prio=0 tid=0x000056473bc2d800 nid=0x22 runnable [0x00007fefe8160000]
"qtp791452441-16" #16 prio=5 os_prio=0 tid=0x000056473bc2b000 nid=0x21 runnable [0x00007fefe8261000]
"qtp791452441-15-lowPrioritySelector" #15 prio=1 os_prio=0 tid=0x000056473bc29000 nid=0x20 waiting for monitor entry [0x00007fefe8363000]
"qtp791452441-14-lowPrioritySelector" #14 prio=1 os_prio=0 tid=0x000056473bc8c000 nid=0x1f waiting for monitor entry [0x00007fefe8464000]
"qtp791452441-13-lowPrioritySelector" #13 prio=1 os_prio=0 tid=0x000056473bc8a000 nid=0x1e waiting for monitor entry [0x00007fefe8565000]
"qtp791452441-12-lowPrioritySelector" #12 prio=1 os_prio=0 tid=0x000056473bc86800 nid=0x1d waiting for monitor entry [0x00007fefe8666000]
"qtp791452441-11" #11 prio=5 os_prio=0 tid=0x000056473bd8a800 nid=0x1c runnable [0x00007fefe8766000]
"qtp791452441-10" #10 prio=5 os_prio=0 tid=0x000056473bd84800 nid=0x1b runnable [0x00007fefe8867000]
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x000056473bc63800 nid=0x19 runnable [0x0000000000000000]
"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x000056473bc14000 nid=0x18 waiting on condition [0x0000000000000000]
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x000056473bbf9000 nid=0x17 waiting on condition [0x0000000000000000]
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x000056473bbea000 nid=0x16 waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x000056473bbe7000 nid=0x15 waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x000056473bbe4800 nid=0x14 runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x000056473bbb5000 nid=0x13 in Object.wait() [0x00007fefe92cb000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x000056473bbb2000 nid=0x12 in Object.wait() [0x00007fefe93cc000]
"VM Thread" os_prio=0 tid=0x000056473bba8000 nid=0x11 runnable 
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x000056473b9f0000 nid=0x9 runnable 
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x000056473b9f2000 nid=0xa runnable 
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x000056473b9f3800 nid=0xb runnable 
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x000056473b9f5800 nid=0xc runnable 
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x000056473b9f7000 nid=0xd runnable 
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x000056473b9f9000 nid=0xe runnable 
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x000056473b9fa800 nid=0xf runnable 
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x000056473b9fc800 nid=0x10 runnable 
"VM Periodic Task Thread" os_prio=0 tid=0x000056473bc59800 nid=0x1a waiting on condition

##
# Joyent Triton run
##
# Run second version of our container
eval "$(triton env)"
docker run -d -m 128m --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:2
# List the threads running
docker exec jvm-on-small-container sh -c 'jstack `pgrep java`' | grep os_prio 
"qtp791452441-44" #44 prio=5 os_prio=0 tid=0x000000000215e000 nid=0x12de4 runnable [0x00007fffe51f7000]
"qtp791452441-43" #43 prio=5 os_prio=0 tid=0x0000000000262000 nid=0x12aa4 waiting for monitor entry [0x00007fffe52fa000]
"qtp791452441-42" #42 prio=5 os_prio=0 tid=0x000000000068a000 nid=0x11808 waiting on condition [0x00007fffe7c7a000]
"Scheduler-1896277646" #41 prio=5 os_prio=0 tid=0x0000000000145000 nid=0x1176f waiting on condition [0x00007fffe4e34000]
"Attach Listener" #39 daemon prio=9 os_prio=0 tid=0x0000000000717800 nid=0x116e2 waiting on condition [0x0000000000000000]
"DestroyJavaVM" #33 prio=5 os_prio=0 tid=0x0000000000716000 nid=0x10796 waiting on condition [0x0000000000000000]
"qtp791452441-32-acceptor-3@6dbc83d8-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #32 prio=3 os_prio=0 tid=0x0000000000714800 nid=0x1082a runnable [0x00007fffe5cff000]
"qtp791452441-31-acceptor-2@1ac08473-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #31 prio=3 os_prio=0 tid=0x0000000000713000 nid=0x10829 waiting for monitor entry [0x00007fffe6000000]
"qtp791452441-30-acceptor-1@34228b2c-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #30 prio=3 os_prio=0 tid=0x000000000070d800 nid=0x10828 waiting for monitor entry [0x00007fffe67fe000]
"qtp791452441-29-acceptor-0@467e71fc-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #29 prio=3 os_prio=0 tid=0x000000000070b000 nid=0x10827 waiting for monitor entry [0x00007fffe6f72000]
"qtp791452441-28-lowPrioritySelector" #28 prio=1 os_prio=0 tid=0x0000000000695800 nid=0x10818 runnable [0x00007fffe72b3000]
"qtp791452441-27" #27 prio=5 os_prio=0 tid=0x000000000068d800 nid=0x10816 waiting on condition [0x00007fffe75f6000]
"qtp791452441-26-lowPrioritySelector" #26 prio=1 os_prio=0 tid=0x000000000068b800 nid=0x10815 runnable [0x00007fffe7937000]
"qtp791452441-24-lowPrioritySelector" #24 prio=1 os_prio=0 tid=0x0000000000687800 nid=0x10812 waiting for monitor entry [0x00007fffe7fbc000]
"qtp791452441-23" #23 prio=5 os_prio=0 tid=0x0000000000685800 nid=0x10810 waiting for monitor entry [0x00007fffe82fe000]
"qtp791452441-22-lowPrioritySelector" #22 prio=1 os_prio=0 tid=0x0000000000683800 nid=0x1080f waiting for monitor entry [0x00007fffe85ff000]
"qtp791452441-21" #21 prio=5 os_prio=0 tid=0x000000000067d000 nid=0x1080e runnable [0x00007fffe8b1d000]
"Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00000000001ee800 nid=0x107c5 runnable [0x0000000000000000]
"C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00000000001e1800 nid=0x107c3 waiting on condition [0x0000000000000000]
"C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00000000001d7000 nid=0x107c1 waiting on condition [0x0000000000000000]
"C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00000000001d4000 nid=0x107bf waiting on condition [0x0000000000000000]
"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00000000001b7800 nid=0x107bd waiting on condition [0x0000000000000000]
"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00000000001ad800 nid=0x107bc waiting on condition [0x0000000000000000]
"C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00000000001a2000 nid=0x107bb waiting on condition [0x0000000000000000]
"C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x0000000000197800 nid=0x107b9 waiting on condition [0x0000000000000000]
"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x0000000000172000 nid=0x107b8 waiting on condition [0x0000000000000000]
"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x0000000000186800 nid=0x107b7 waiting on condition [0x0000000000000000]
"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x000000000017c000 nid=0x107b6 waiting on condition [0x0000000000000000]
"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x0000000000165800 nid=0x107b5 waiting on condition [0x0000000000000000]
"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x000000000010f000 nid=0x107b3 waiting on condition [0x0000000000000000]
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x0000000000104800 nid=0x107b2 waiting on condition [0x0000000000000000]
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00000000000f5000 nid=0x107b1 waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00000000000f2800 nid=0x107b0 waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00000000000f0800 nid=0x107af runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00000000000c1000 nid=0x107a3 in Object.wait() [0x00007ffffd5fe000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00000000000be000 nid=0x107a1 in Object.wait() [0x00007ffffe9fe000]
"VM Thread" os_prio=0 tid=0x00000000000b4000 nid=0x1079d runnable 
"VM Periodic Task Thread" os_prio=0 tid=0x00000000001f3800 nid=0x107c6 waiting on condition 

Triton isn`t a fun guy!

Triton isn`t a fun guy!


Howly! More than 30 threads are running. On my computer the regular Docker container has 8 `GC task thread`s, and 4 `CompilerThread`s. On Triton there are 15! The JVM saw a lot of CPU cores and created many threads to take advantage of it. However, our container is restricted to 0.065 cores. Those many threads won`t help. Of course, we can configure this:

In this really small container the `ParallelGC` is not really needed and we can use the `SerialGC`. In case you use a bit larger container, you can use `-XX:-XX:+ParallelGCThreads=$CPU` for `ParallelGC`. Or `-XX:G1ConcRefinementThreads=$CPU` in case you use `-XX:+UseG1GC`.
And we can use `-XX:CICompilerCount=$CPU` to reduce the compiler threads. However, it needs to be at least 2.

Still Many Threads

We still have many ‘acceptor’ and ‘selector’ threads. On Triton we’ve 4 acceptor and 4 selector threads:

##
# Local docker run 
##
# Run third version of our container
sudo docker run -d -m 128m --cpus 0.065 --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:3
# List the threads running
sudo docker exec jvm-on-small-container sh -c 'jstack `pgrep java`' | grep os_prio 
"Attach Listener" #20 daemon prio=9 os_prio=0 tid=0x0000564e7cd17000 nid=0x37 waiting on condition [0x0000000000000000]
"qtp791452441-19" #19 prio=5 os_prio=0 tid=0x0000564e7cd15000 nid=0x36 waiting on condition [0x00007fdcc863f000]
"qtp791452441-17-acceptor-0@77fe3be4-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #17 prio=3 os_prio=0 tid=0x0000564e7cc25000 nid=0x34 runnable [0x00007fdcc8841000]
"DestroyJavaVM" #16 prio=5 os_prio=0 tid=0x0000564e7c922800 nid=0x8 waiting on condition [0x0000000000000000]
"qtp791452441-15-lowPrioritySelector" #15 prio=1 os_prio=0 tid=0x0000564e7ce14000 nid=0x31 waiting for monitor entry [0x00007fdccbd6e000]
"qtp791452441-14" #14 prio=5 os_prio=0 tid=0x0000564e7ce12000 nid=0x30 runnable [0x00007fdccbe6e000]
"qtp791452441-13-lowPrioritySelector" #13 prio=1 os_prio=0 tid=0x0000564e7ce0f800 nid=0x2f waiting for monitor entry [0x00007fdccbf70000]
"qtp791452441-12" #12 prio=5 os_prio=0 tid=0x0000564e7ce0e000 nid=0x2e runnable [0x00007fdccc070000]
"qtp791452441-11-lowPrioritySelector" #11 prio=1 os_prio=0 tid=0x0000564e7ce0c000 nid=0x2d waiting for monitor entry [0x00007fdccc172000]
"qtp791452441-10-lowPrioritySelector" #10 prio=1 os_prio=0 tid=0x0000564e7ce0a000 nid=0x2b waiting for monitor entry [0x00007fdccc273000]
"qtp791452441-9" #9 prio=5 os_prio=0 tid=0x0000564e7ce07800 nid=0x2a runnable [0x00007fdccc373000]
"qtp791452441-8" #8 prio=5 os_prio=0 tid=0x0000564e7ce01000 nid=0x29 runnable [0x00007fdccc474000]
"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x0000564e7c8c2800 nid=0xf runnable [0x0000000000000000]
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x0000564e7c8b7800 nid=0xe waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x0000564e7c8ac800 nid=0xd waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x0000564e7c8aa800 nid=0xc runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x0000564e7c87d800 nid=0xb in Object.wait() [0x00007fdccccd4000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x0000564e7c873000 nid=0xa in Object.wait() [0x00007fdcccdd5000]
"VM Thread" os_prio=0 tid=0x0000564e7c869000 nid=0x9 runnable 
"VM Periodic Task Thread" os_prio=0 tid=0x0000564e7c8c5800 nid=0x10 waiting on condition 


##
# Joyent Triton run
##
# Run third version of our container
eval "$(triton env)"
docker run -d -m 128m --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:3
# List the threads running
docker exec jvm-on-small-container sh -c 'jstack `pgrep java`' | grep os_prio 
"Attach Listener" #23 daemon prio=9 os_prio=0 tid=0x000000000028c000 nid=0x732a waiting on condition [0x0000000000000000]
"qtp791452441-22" #22 prio=5 os_prio=0 tid=0x00000000006eb000 nid=0x7178 waiting on condition [0x00007fffe81fe000]
"qtp791452441-21-acceptor-3@161ab1a8-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #21 prio=3 os_prio=0 tid=0x00000000006ea000 nid=0x7177 waiting for monitor entry [0x00007fffe84ff000]
"DestroyJavaVM" #20 prio=5 os_prio=0 tid=0x00000000001df000 nid=0x7109 waiting on condition [0x0000000000000000]
"qtp791452441-19-acceptor-2@70b4bcd9-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #19 prio=3 os_prio=0 tid=0x00000000006e7000 nid=0x7175 waiting for monitor entry [0x00007fffe8800000]
"qtp791452441-18-acceptor-1@57033676-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #18 prio=3 os_prio=0 tid=0x0000000000747800 nid=0x7174 waiting for monitor entry [0x00007fffe97f6000]
"qtp791452441-17-acceptor-0@67e0dd84-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #17 prio=3 os_prio=0 tid=0x00000000006e4800 nid=0x7173 runnable [0x00007fffe8ffe000]
"qtp791452441-15-lowPrioritySelector" #15 prio=1 os_prio=0 tid=0x000000000066b000 nid=0x7159 waiting for monitor entry [0x00007fffe9af7000]
"qtp791452441-14" #14 prio=5 os_prio=0 tid=0x000000000066c800 nid=0x7157 runnable [0x00007fffe9df7000]
"qtp791452441-13-lowPrioritySelector" #13 prio=1 os_prio=0 tid=0x0000000000668800 nid=0x7156 waiting for monitor entry [0x00007fffea0f9000]
"qtp791452441-12" #12 prio=5 os_prio=0 tid=0x0000000000666800 nid=0x7155 runnable [0x00007fffea3f9000]
"qtp791452441-11" #11 prio=5 os_prio=0 tid=0x0000000000664800 nid=0x7154 runnable [0x00007fffea6fa000]
"qtp791452441-10-lowPrioritySelector" #10 prio=1 os_prio=0 tid=0x0000000000662800 nid=0x7153 waiting for monitor entry [0x00007fffea9fc000]
"qtp791452441-9" #9 prio=5 os_prio=0 tid=0x0000000000660800 nid=0x7152 runnable [0x00007fffeacfc000]
"qtp791452441-8-lowPrioritySelector" #8 prio=1 os_prio=0 tid=0x0000000000659800 nid=0x7151 waiting for monitor entry [0x00007fffeaffe000]
"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x0000000000130800 nid=0x7120 runnable [0x0000000000000000]
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x0000000000104000 nid=0x711e waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00000000000f2800 nid=0x711d waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00000000000f0800 nid=0x711c runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00000000000c1000 nid=0x7115 in Object.wait() [0x00007ffffd5fe000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00000000000be000 nid=0x7113 in Object.wait() [0x00007ffffe9fe000]
"VM Thread" os_prio=0 tid=0x00000000000b4000 nid=0x710f runnable 
"VM Periodic Task Thread" os_prio=0 tid=0x0000000000128800 nid=0x7121 waiting on condition 


Those threads are created by the web server, so you have to configure the web server you are using. In this example Jetty is used, and we configure it via simple Java properties:

And also change the Docker file:

And now, it starts to look fine, right?

Be Still Carefull of Many Threads

Our example web app now runs fine. But keep an eye on the thread count. Other Java/programming language library might create a thread pool and many threads. However, you can configure these libraries too.

Update 21th June, 2017

Well, first the OpenJDK team is improving the situation. As of JDK 8u131, the JVM does limit the CPU’s in Docker. And with the experimental flags ‘-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap’ it does limit the memory.

Also, Matt Rasband has a good blog post on how to estimate the memory usage of the JVM and set the flags properly.

Have fun running the JVM in a tiny container =)

Tagged on: , , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *