{"id":3279,"date":"2016-07-29T13:50:01","date_gmt":"2016-07-29T12:50:01","guid":{"rendered":"http:\/\/www.gamlor.info\/wordpress\/?p=3279"},"modified":"2021-03-11T09:42:56","modified_gmt":"2021-03-11T08:42:56","slug":"joyents-triton-native-our-friend","status":"publish","type":"post","link":"https:\/\/www.gamlor.info\/wordpress\/2016\/07\/joyents-triton-native-our-friend\/","title":{"rendered":"Joyent&#8217;s Triton : \/native, our friend!"},"content":{"rendered":"<p>We deployed our Docker container on Triton and everything works well. Suddenly, today there a issue. Our application is slow or stopped working. And in our container there is not debugging tool. So, what should we do?<br \/>\n<div id=\"attachment_3313\" style=\"width: 710px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/rebooting.en_.jpg\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-3313\" src=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/rebooting.en_-863x1024.jpg\" alt=\"Rebooting is no solution\" width=\"700\" height=\"831\" class=\"size-large wp-image-3313\" srcset=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/rebooting.en_-863x1024.jpg 863w, https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/rebooting.en_-253x300.jpg 253w, https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/rebooting.en_-768x911.jpg 768w, https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/rebooting.en_.jpg 1200w\" sizes=\"(max-width: 700px) 100vw, 700px\" \/><\/a><p id=\"caption-attachment-3313\" class=\"wp-caption-text\">Rebooting is no solution<\/p><\/div><\/p>\n<h2>Explore Triton&#8217;s Container a bit<\/h2>\n<p>I started again a nginx container on Triton:<br \/>\n<script src=\"https:\/\/gist.github.com\/674e33d7e8db51040fc00869de0e272a.js?file=explore-start-ngnix.sh\"><\/script><noscript><pre><code class=\"language-shell shell\">#Switch to Triton\neval $(triton env)\n# Start nginx on triton\ndocker run -d --name test -p 80 nginx<\/code><\/pre><\/noscript><br \/>\nWhen the container has started, start a shell inside the container with &#8216;docker exec&#8217;. Then explore the file system a bit.<br \/>\n<script src=\"https:\/\/gist.github.com\/674e33d7e8db51040fc00869de0e272a.js?file=interactive-shell-on-container.sh\"><\/script><noscript><pre><code class=\"language-shell shell\">roman@minty ~ $ docker exec -ti test \/bin\/bash\nroot@ae35cee7759b:\/# ls -l\n<\/code><\/pre><\/noscript><br \/>\n<div id=\"attachment_3254\" style=\"width: 310px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/native.jpg\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-3254\" class=\"size-medium wp-image-3254\" src=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/native-300x195.jpg\" alt=\"\/native?\" width=\"300\" height=\"195\" srcset=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/native-300x195.jpg 300w, https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/native.jpg 549w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><p id=\"caption-attachment-3254\" class=\"wp-caption-text\">\/native?<\/p><\/div><br \/>\nWhat the hell is \/native? Look a around a bit more. Maybe try to find popular unix tools in there? Aha! From `awk` to `zcat`, everything is there. So \/native is another Unix? Right! \/native is a window into Triton&#8217;s native operating system. And there are popular unix tools there.<br \/>\n<div id=\"attachment_3259\" style=\"width: 310px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/explored.jpg\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-3259\" class=\"size-medium wp-image-3259\" src=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/explored-300x228.jpg\" alt=\"Explore!\" width=\"300\" height=\"228\" srcset=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/explored-300x228.jpg 300w, https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/explored.jpg 561w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><p id=\"caption-attachment-3259\" class=\"wp-caption-text\">Explore!<\/p><\/div><br \/>\n<div id=\"attachment_3314\" style=\"width: 710px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/slash-native.en_.jpg\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-3314\" src=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/slash-native.en_-1024x350.jpg\" alt=\"At \/native\/ old unix tools live.\" width=\"700\" height=\"239\" class=\"size-large wp-image-3314\" srcset=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/slash-native.en_-1024x350.jpg 1024w, https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/slash-native.en_-300x103.jpg 300w, https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/slash-native.en_-768x262.jpg 768w, https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/slash-native.en_.jpg 1200w\" sizes=\"(max-width: 700px) 100vw, 700px\" \/><\/a><p id=\"caption-attachment-3314\" class=\"wp-caption-text\">At \/native\/ old unix tools live.<\/p><\/div><\/p>\n<h2>Let&#8217;s use \/native<\/h2>\n<p>Now we know that \/native exists so let&#8217;s use it. First let&#8217;s deploy a bad container. I&#8217;ve prepared a &#8216;bad container&#8217; example. The <a href=\"https:\/\/gist.github.com\/gamlerhart\/5582bce62cfea2ea4aad57956df2a7b5\">code is here<\/a>. Let&#8217;s start that &#8216;gamlerhart\/waste-io&#8217; container:<br \/>\n<script src=\"https:\/\/gist.github.com\/674e33d7e8db51040fc00869de0e272a.js?file=start-bad-container.sh\"><\/script><noscript><pre><code class=\"language-shell shell\">docker run -d --name waste-io-example gamlerhart\/waste-io<\/code><\/pre><\/noscript><br \/>\nThis container has a issue. Let&#8217;s start finding out what the problem is. Again, we `docker exec` into the container. If we try to use `iostat`, nothing is there. However, we can use \/native. Add \/native to the PATH and we can use `iostat`.<br \/>\n<script src=\"https:\/\/gist.github.com\/674e33d7e8db51040fc00869de0e272a.js?file=iostat.sh\"><\/script><noscript><pre><code class=\"language-shell shell\">roman@minty ~ $ docker exec -ti waste-io-example \/bin\/bash\nroot@3bd53e4a7fe4:\/# \nroot@3bd53e4a7fe4:\/# iostat -M 1\nbash: iostat: command not found\n\n#Oh...damn...iostat would tell us if our IO is sane.\n#Ah...\/native to the rescue\nroot@3bd53e4a7fe4:\/# iostat -M 1\n   tty      ramdisk1        sd0           sd1           sd2            cpu\n tin tout Mps tps serv  Mps tps serv  Mps tps serv  Mps tps serv   us sy dt id\n   0  236   0   0    0    0   0    0    2 148    0    2  42    1    4  8  0 89\n   0 1214   0   0    0    0   0    0    7 3583    0    0   0    0    9 14  0 77\n   0 2100   0   0    0    0   0    0    7 3531    0    5 157    0    7 17  0 77\n   0  941   0   0    0    0   0    0    7 3640    0    0   0    0    7 14  0 79<\/code><\/pre><\/noscript>.<br \/>\nAha! Our container burns quite a bit of IO. However, we want to know more. Maybe some kind of tracing. Let&#8217;s dtrace. Yep, there is dtrace in \/native. I&#8217;m not a dtrace pro, there is tons of materials online. (<a href=\"http:\/\/dtrace.org\/\">dtrace.org<\/a>, <a href=\"http:\/\/dtrace.org\/blogs\/\">guide<\/a>, <a href=\"http:\/\/www.brendangregg.com\/DTrace\/dtrace_oneliners.txt\">examples<\/a>). We&#8217;ll use the &#8216;lx-syscall&#8217; probe (<strong>L<\/strong>inu<strong>X<\/strong>-syscall).`-ln lx-syscall:::` lists lx-syscall probes. -n `syscall::: { @num[execname,pid,probefunc] = count() }` starts the probe, and groups by program, process-id and probe name. Stop the probe with `Ctrl+C`:<br \/>\n<script src=\"https:\/\/gist.github.com\/674e33d7e8db51040fc00869de0e272a.js?file=dtrace-start.sh\"><\/script><noscript><pre><code class=\"language-shell shell\">#linux-syscalls aka lx-syscalls\nroot@3bd53e4a7fe4:\/# dtrace -ln &#039;lx-syscall:::&#039;\n   ID   PROVIDER            MODULE                          FUNCTION NAME\n 1549 lx-syscall             sys32                     inotify_init1 return\n 1550 lx-syscall             sys32                            preadv entry\n 1551 lx-syscall             sys32                            preadv return\n 1552 lx-syscall             sys32                           pwritev entry\n \n# Start tracing the syscalls. Grouped by execname, pid and syscallname (probefunc)\nroot@3bd53e4a7fe4:\/# dtrace -n &#039;syscall::: { @num[execname,pid,probefunc] = count() }&#039;\ndtrace: description &#039;lx-syscall::: &#039; matched 1352 probes\n^C\n\n  node                                                  23210  fsync                                                         11368\n  node                                                  23210  ioctl                                                         11368\n  node                                                  23210  open                                                          11368\n  node                                                  23210  pwrite64                                                      11368\n  node                                                  23210  close                                                         11370\n  node                                                  23210  unlink                                                        11370\n  node                                                  23210  epoll_wait                                                    30332\n  node                                                  23210  read                                                          30332\n  node                                                  23210  write                                                         30332\n  node                                                  23210  futex                                                        180019\n \n<\/code><\/pre><\/noscript><br \/>\nWe see our node process is doing tons of write, read, open unlink and futex system calls, so our node process is the issue. But what is this program doing? What files is it opening? Let&#8217;s do more dtrace probes. `x-syscall::open:entry` only shows the open syscall, `execname==\u201dnode\u201d` filters out node only. And we group by file with &#8220;@num[copyinstr(arg0)] = count()`.<br \/>\n<script src=\"https:\/\/gist.github.com\/674e33d7e8db51040fc00869de0e272a.js?file=what-file.sh\"><\/script><noscript><pre><code class=\"language-shell shell\">root@3bd53e4a7fe4:\/# dtrace -n &#039;lx-syscall::open:entry\/execname==&quot;node&quot;\/ {  @num[copyinstr(arg0)] = count() }&#039;\ndtrace: description &#039;lx-syscall::open:entry&#039; matched 2 probes\n^C\n\n  \/tmp\/io-file-0.00023033588805110305                               1\n  \/tmp\/io-file-0.00029973264092197915                               1\n  \/tmp\/io-file-0.001082331117725488                                 1\n  \/tmp\/io-file-0.0017028727466004145                                1\n  \/tmp\/io-file-0.00224423432995402                                  1\n  \/tmp\/io-file-0.002446706340221372                                 1\n  \/tmp\/io-file-0.002880551536430742                                 1<\/code><\/pre><\/noscript><br \/>\nAh, tons of temporary files. Hmm&#8230;well where in the node program do we create these temp files? With Dtrace&#8217;s ustack\/jstack we can the the programs stack trace:<br \/>\n<script src=\"https:\/\/gist.github.com\/674e33d7e8db51040fc00869de0e272a.js?file=stack-probe.sh\"><\/script><noscript><pre><code class=\"language-shell shell\">root@3bd53e4a7fe4:\/# dtrace -n &#039;lx-syscall::open:entry\/execname==&quot;node&quot;\/ {  @num[jstack()] = count() }&#039;\ndtrace: description &#039;lx-syscall::open:entry&#039; matched 2 probes\n^C\n\n\n              libc.so.6`open+0x10\n              libc.so.6`sysconf+0x54c\n              node`_ZN2v88internal20MarkCompactCollector31NumberOfParallelCompactionTasksEil+0x46\n              node`_ZN2v88internal20MarkCompactCollector23EvacuatePagesInParallelEv+0xea\n              node`_ZN2v88internal20MarkCompactCollector29EvacuateNewSpaceAndCandidatesEv+0x67\n              node`_ZN2v88internal20MarkCompactCollector14CollectGarbageEv+0x29\n              node`_ZN2v88internal4Heap11MarkCompactEv+0x6d\n              node`_ZN2v88internal4Heap24PerformGarbageCollectionENS0_16GarbageCollectorENS_15GCCallbackFlagsE+0x4b0\n              node`_ZN2v88internal4Heap14CollectGarbageENS0_16GarbageCollectorEPKcS4_NS_15GCCallbackFlagsE+0x142\n              node`_ZN2v88internal4Heap15HandleGCRequestEv+0x84\n              node`_ZN2v88internal10StackGuard16HandleInterruptsEv+0x31c\n              node`_ZN2v88internal18Runtime_StackGuardEiPPNS0_6ObjectEPNS0_7IsolateE+0x45\n              0x2bd84b30961b\n              0x2bd84b44e1f9\n              0x2bd84b4490eb\n              0x2bd84b44281b\n              0x2bd84b30d157\n              0x2bd84b442615\n              0x2bd84b337f84\n              0x2bd84b322922\n              node`_ZN2v88internal12_GLOBAL__N_16InvokeEPNS0_7IsolateEbNS0_6HandleINS0_6ObjectEEES6_iPS6_S6_+0xb3\n              node`_ZN2v88internal9Execution4CallEPNS0_7IsolateENS0_6HandleINS0_6ObjectEEES6_iPS6_+0x61\n              node`_ZN2v88Function4CallENS_5LocalINS_7ContextEEENS1_INS_5ValueEEEiPS5_+0x139\n              node`_ZN2v88Function4CallENS_5LocalINS_5ValueEEEiPS3_+0x41\n              node`_ZN4node9AsyncWrap12MakeCallbackEN2v85LocalINS1_8FunctionEEEiPNS2_INS1_5ValueEEE+0x149\n              node`_ZN4nodeL5AfterEP7uv_fs_s+0x146\n              node`uv__work_done+0xa5\n              node`uv__async_event+0xab\n              node`uv__async_io+0xa3\n              node`uv__io_poll+0x3a0\n              node`uv_run+0x156\n              node`_ZN4node5StartEiPPc+0x468\n              libc.so.6`__libc_start_main+0xf5\n                2\n\n              libpthread.so.0`open+0x2d\n              node`worker+0xa1\n              node`uv__thread_start+0x19\n              libpthread.so.0`start_thread+0xc4\n            62043\n<\/code><\/pre><\/noscript><br \/>\nAh&#8230;no luck. The open syscall is in the event loop (Looking into node via dtrace is another topic). However, we know know that the node program creates tons of tmp\/io-file files and with this information we hopefully can fix that bug.<br \/>\n<div id=\"attachment_3316\" style=\"width: 710px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/dtrace.en_.jpg\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-3316\" src=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/dtrace.en_-1024x526.jpg\" alt=\"Sherlock &#039;dtrace&#039; Holmes\" width=\"700\" height=\"360\" class=\"size-large wp-image-3316\" srcset=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/dtrace.en_-1024x526.jpg 1024w, https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/dtrace.en_-300x154.jpg 300w, https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2016\/07\/dtrace.en_-768x394.jpg 768w\" sizes=\"(max-width: 700px) 100vw, 700px\" \/><\/a><p id=\"caption-attachment-3316\" class=\"wp-caption-text\">Sherlock &#8216;dtrace&#8217; Holmes<\/p><\/div><\/p>\n<h2>You know DTrace well?<\/h2>\n<p>You know dtrace well and cannot find many probes? Because you can only dtrace within the Triton container you have less probes available. And we cannot go into the global zone of Joyent&#8217;s public cloud service. On a private Triton deployment we could. That&#8217;s a topic for another time and another blog post.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>We deployed our Docker container on Triton and everything works well. Suddenly, today there a issue. Our application is slow or stopped working. And in our container there is not&hellip; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[268,187],"tags":[300,301,302],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/posts\/3279"}],"collection":[{"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/comments?post=3279"}],"version-history":[{"count":17,"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/posts\/3279\/revisions"}],"predecessor-version":[{"id":3321,"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/posts\/3279\/revisions\/3321"}],"wp:attachment":[{"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/media?parent=3279"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/categories?post=3279"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/tags?post=3279"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}