{"id":2619,"date":"2012-05-17T14:38:11","date_gmt":"2012-05-17T13:38:11","guid":{"rendered":"http:\/\/www.gamlor.info\/wordpress\/?p=2619"},"modified":"2021-03-11T09:19:12","modified_gmt":"2021-03-11T08:19:12","slug":"async-sql-and-akka","status":"publish","type":"post","link":"https:\/\/www.gamlor.info\/wordpress\/2012\/05\/async-sql-and-akka\/","title":{"rendered":"Async SQL and Akka"},"content":{"rendered":"<p>I\u2019ve already written several times about Akka and async programming with it. For example doing access <a href=\"https:\/\/www.gamlor.info\/wordpress\/2012\/03\/async-file-io-with-akka-and-java-7\/\">files<\/a> and <a href=\"https:\/\/www.gamlor.info\/wordpress\/2012\/04\/async-http-client-and-akka\/\">webservices<\/a> with it. So async API stack is complete, right? We\u2019ve got async network and web access via NIO and frameworks like <a href=\"http:\/\/www.playframework.org\/\">Play 2.0<\/a>, <a href=\"http:\/\/spray.cc\">Spray<\/a>, <a href=\"https:\/\/github.com\/sonatype\/async-http-client\">HTTPClient<\/a> etc. And with NIO 2 we easily can access the file system in asynchronous way. But wait, what about your database? What if tons of your data sits in a good old relational database? Unfortunately the current JDBC model is blocking by definition. No asynchronous operation is allowed.<\/p>\n<p>The standard way of doing things for async SQL operations is to use some background threads and do things there. And for many applications this is good enough. But if you do more and more db operations you start to consume more and more threads, which will be waiting for blocking operations. Wouldn\u2019t it be nicer if the database operations\u00a0are truly asynchronous like other operations?<\/p>\n<div id=\"attachment_2627\" style=\"width: 310px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2012\/05\/JDBC-Model.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-2627\" class=\"size-medium wp-image-2627\" title=\"JDBC-Model\" alt=\"JDBC is nice, but let's you wait\" src=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2012\/05\/JDBC-Model-300x189.png\" width=\"300\" height=\"189\" srcset=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2012\/05\/JDBC-Model-300x189.png 300w, https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2012\/05\/JDBC-Model.png 900w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><p id=\"caption-attachment-2627\" class=\"wp-caption-text\">JDBC is nice, but let&#8217;s you wait<\/p><\/div>\n<p>So I&#8217;ve looked for an alternative to JDBC. I\u2019ve found two approaches to provide a asynchronous JDBC alternative. The <a href=\"https:\/\/github.com\/mheath\/adbcj\">async-mysql-connector<\/a> and <a href=\"https:\/\/github.com\/mheath\/adbcj\">ADBCJ<\/a>. Unfortunate both are not really actively maintained. Anyhow I decided to go ahead with ADBCJ. More about that later. Let\u2019s first take a look at how we to use everything.<\/p>\n<h2>Async SQL in Akka<\/h2>\n<p>Let\u2019s take a short look how to use that API in Akka. First we need to configure the database in our Akka configuration:<\/p>\n<script src=\"https:\/\/gist.github.com\/2719603.js?file=application.conf\"><\/script><noscript><pre><code class=\"language- \">myapp{\n     async-jdbc{\n        url = &quot;adbcj:mysql:\/\/localhost\/employees&quot;\n        username = &quot;root&quot;\n        password = &quot;&quot;\n     }\n}<\/code><\/pre><\/noscript>\n<p>After that we can use the Database Akka extension to create a connection. And with that connection we then can do our database operations. The API is very JDBC like, but with the difference that all operations run asynchronously. Since they return Akka futures, you can compose the operations like your used to <a href=\"http:\/\/doc.akka.io\/docs\/akka\/2.0.1\/scala\/futures.html\">with Akka futures<\/a>. Here an example:<\/p>\n<script src=\"https:\/\/gist.github.com\/2719603.js?file=FirstExample.scala\"><\/script><noscript><pre><code class=\"language-scala scala\">\/\/ grab the extension\nval dbSupport = Database(actorSystem)\n\nval result = for{  \n  \/\/ open the connection asynchronously\n  connection &lt;- dbSupport.connect()\n  \/\/ Execute a query quest\n  newestEmployee &lt;- connection.executeQuery(\n\t&quot;SELECT first_name,last_name,hire_date FROM employees &quot; +\n\t&quot;ORDER BY hire_date DESC &quot; +\n\t&quot;LIMIT 0,5&quot;)\n  \/\/ When the results arrived, close the connection\n   _ &lt;- connection.close()\n} yield newestEmployee\n\n\/\/ Process the results\nresult.onSuccess{\n  case resultSet =&gt;{\n\tresultSet.foreach{\n\t  row =&gt; println(row(&quot;first_name&quot;).getString\n\t\t  + &quot; &quot;+row(&quot;first_name&quot;).getString\n\t\t  + &quot; since &quot; + row(&quot;hire_date&quot;).getString )\n\t}\n  }\n}.onFailure{\n  case e:Exception =&gt;{\n\te.printStackTrace()\n  }\n}<\/code><\/pre><\/noscript>\n<p>A regular query returns a immutable ResultSet. Alternatively you can pass a event handler to the query method. That handler will be called as the data streams in. This way you can assemble the data yourself. For example it can build up a string:<\/p>\n<script src=\"https:\/\/gist.github.com\/2719603.js?file=HandleAsEvents.scala\"><\/script><noscript><pre><code class=\"language-scala scala\">val resultAsString = for {\n\/\/ open the connection asynchronously\n  connection &lt;- dbSupport.connect()\n  \/\/ Execute a query quest\n  dataAsString &lt;- connection.executeQuery(\n\t&quot;SELECT first_name,last_name,hire_date FROM employees &quot; +\n\t  &quot;ORDER BY hire_date DESC &quot; +\n\t  &quot;LIMIT 0,5&quot;, &quot;&quot;) {\n\t\t  \/\/ Instead of creating a result set, we also can directly react\n\t\t  \/\/ as the data streams in\n\t\t  case StartRow(jsonToBuild) =&gt; jsonToBuild + &quot;-&quot;\n\t\t  case AValue(value, jsonToBuild) =&gt; jsonToBuild + &quot;,&quot; + value.getString\n\t\t  case EndRow(jsonToBuild) =&gt; jsonToBuild + &quot;\\n&quot;\n  }\n  \/\/ When the results arrived, close the connection\n  _ &lt;- connection.close()\n} yield dataAsString\n\n\/\/ Process the results\nresultAsString.onSuccess {\n  case stringData =&gt; print(stringData)\n}.onFailure {\n  case e: Exception =&gt; {\n\te.printStackTrace()\n  }\n}<\/code><\/pre><\/noscript>\n<p>Of course transactions, updates and prepared statements are also supported:<\/p>\n<script src=\"https:\/\/gist.github.com\/2719603.js?file=MoreFeatures.scala\"><\/script><noscript><pre><code class=\"language-scala scala\">val result = for {\n  \/\/ open the connection asynchronously\n  connection &lt;- dbSupport.connect()\n  _ &lt;- connection.beginTransaction()\n  insertedInfo &lt;- connection.executeUpdate(\n\t&quot;INSERT INTO employees (emp_no, birth_date, first_name, last_name, gender, hire_date) &quot; +\n\t&quot;VALUES (42, '1986-05-07', 'Roman', 'Stoffel', 'M', '2012-05-17')&quot;)\n  statement &lt;- connection.prepareQuery(&quot;SELECT first_name FROM employees WHERE emp_no = ?&quot;)\n  queryResult &lt;- statement.execute(42)\n  _ &lt;- connection.rollback()\n  _ &lt;- connection.close()\n} yield queryResult\n<\/code><\/pre><\/noscript>\n<p>Of course it\u2019s annoying to manually close the connection every time. Therefore a .withConnection method exists. It will open a connection for you, run your code and close the connection afterwards. Everything is done asynchronously, therefore closure has to return a future, and the connection is closed when that future completes:<\/p>\n<script src=\"https:\/\/gist.github.com\/2719603.js?file=WithConnection.scala\"><\/script><noscript><pre><code class=\"language-scala scala\">val result = Database(actorSystem)\n  .withConnection{\n  connection =&gt;connection.executeQuery(\n\t&quot;SELECT first_name,last_name,hire_date FROM employees &quot; +\n\t&quot;ORDER BY hire_date DESC &quot; +\n\t&quot;LIMIT 0,5&quot;)\n}<\/code><\/pre><\/noscript>\n<p>And also a .withTransaction exists. It does the same for the transaction. It will commit it if the given closure finishes regular. It rolls the transaction back if the closure or the returned future fails.<\/p>\n<script src=\"https:\/\/gist.github.com\/2719603.js?file=WithTransaction.scala\"><\/script><noscript><pre><code class=\"language-scala scala\">connection.withTransaction{\n  txConn =&gt;\n\ttxConn.executeQuery(\n\t&quot;SELECT first_name,last_name,hire_date FROM employees &quot; +\n\t&quot;ORDER BY hire_date DESC &quot; +\n\t&quot;LIMIT 0,5&quot;)\n}<\/code><\/pre><\/noscript>\n<div id=\"attachment_2628\" style=\"width: 310px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2012\/05\/AJDBC-Model.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-2628\" class=\"size-medium wp-image-2628\" title=\"AJDBC-Model\" alt=\"ADBCJ calls you back\" src=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2012\/05\/AJDBC-Model-300x206.png\" width=\"300\" height=\"206\" srcset=\"https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2012\/05\/AJDBC-Model-300x206.png 300w, https:\/\/www.gamlor.info\/wordpress\/wp-content\/uploads\/2012\/05\/AJDBC-Model.png 900w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><p id=\"caption-attachment-2628\" class=\"wp-caption-text\">ADBCJ calls you back<\/p><\/div>\n<h2>The ADBCJ Stuff<\/h2>\n<p>Now the stuff above is just a thin wrapper around the ADBCJ API. It\u2019s basically a JDBC clone, where all operations are asynchronous. That also means that it requires a driver which implement the operations asynchronously. Originally three drivers where around. A MySQL driver, a PostgreeSQL driver and a Bridge to JDBC. Of course the JDBC bridge cannot remove the blocking calls. It just just executes them on worker threads. So it doesn\u2019t bring a real benefit.<\/p>\n<p>Now unfortunately the ADBCJ implementations were only proof of concept implementations. So I <a href=\"https:\/\/github.com\/gamlerhart\/adbcj\">forked it on Github<\/a>, fixed the worst issues and implemented a few missing features. But I did that only for MySQL. So what\u2019s the exact state of it?<\/p>\n<ul>\n<li>The\u00a0 MySQL and JDBC bridge are up to date. The basics work: Transactions, queries, prepared statements etc.<\/li>\n<li>The PostgreeSQL compiles, but I haven\u2019t maintained it. Didn\u2019t bother to run the tests. Prepared statements not implemented.<\/li>\n<li>Many data type implementations are missing.<\/li>\n<li>The test suite is not in a good shape. I&#8217;ve improved it a lot, but it is still very minimalistic. It just doesn\u2019t cover enough.<\/li>\n<li>Connection pool for ADBCJ connections is missing. (There was some implementation, but not tests. Removed it for now until I&#8217;ve test etc im Place.)<\/li>\n<li>It&#8217;s just far away from rock solid.<\/li>\n<\/ul>\n<p>So for a real world application, ADBCJ would need a lot of work. First to improve the MySQL implementation and second to support other databases. Unfortunately this is tedious database protocol implementation work, which isn\u2019t fun. So unless someone is really desperate, it will never be done.<\/p>\n<p>Hopefully, JDBC version 42.0 will include async operations, and async operations can be done easily ;).<\/p>\n<h2>Get the Stuff<\/h2>\n<p>The tiny \u2018wrapper\u2019 is on my github repo here: <a href=\"https:\/\/github.com\/gamlerhart\/akka-async-apis\">https:\/\/github.com\/gamlerhart\/akka-async-apis<\/a>. My fork ADBCJ\u00a0 is here: <a title=\"https:\/\/github.com\/gamlerhart\/adbcj\" href=\"https:\/\/github.com\/gamlerhart\/adbcj. Also\">https:\/\/github.com\/gamlerhart\/adbcj.<\/a> Also the stuff is on my github hosted Maven Snapshot repository. You can grab it via SBT or Maven:<\/p>\n<p>Repository for Maven: <a href=\"https:\/\/github.com\/gamlerhart\/gamlor-mvn\/raw\/master\/snapshots\">https:\/\/github.com\/gamlerhart\/gamlor-mvn\/raw\/master\/snapshots<\/a><br \/>\nGroupID: info.gamlor.akkaasync<br \/>\nArtifactID: akka-dbclient_2.9.1<br \/>\nVersion: 1.0-SNAPSHOT<\/p>\n<p>And you also need the MySQL driver:<\/p>\n<p>GroupID: org.adbcj<br \/>\nArtifactID: mysql-async-driver<br \/>\nVersion: 0.3-SNAPSHOT<\/p>\n<p>So via SBT:<br \/>\n<script src=\"https:\/\/gist.github.com\/2719603.js?file=build.sbt\"><\/script><noscript><pre><code class=\"language-scala scala\">resolvers += &quot;Gamlor-Repo&quot; at &quot;https:\/\/github.com\/gamlerhart\/gamlor-mvn\/raw\/master\/snapshots&quot;\n\nlibraryDependencies += &quot;com.typesafe.akka&quot; % &quot;akka-actor&quot; % &quot;2.0&quot;\nlibraryDependencies += &quot;info.gamlor.akkaasync&quot;  %% &quot;akka-dbclient&quot; % &quot;1.0-SNAPSHOT&quot;\nlibraryDependencies += &quot;org.adbcj&quot; % &quot;adbcj-api&quot; % &quot;0.3-SNAPSHOT&quot;\nlibraryDependencies += &quot;org.adbcj&quot; % &quot;mysql-async-driver&quot; % &quot;0.3-SNAPSHOT&quot;<\/code><\/pre><\/noscript><\/p>\n<h2>The Future<\/h2>\n<p>Well, most applications probably don\u2019t have a real need for an asynchronous JDBC replacement. Connection pools and running enough threads is good enough. And many applications can use enough caching to reduce the database operation pressure. So the niche for this stuff is small =).<\/p>\n<h2>Next Post<\/h2>\n<p>Next time I probably shine some more light on ADBCJ itself.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I\u2019ve already written several times about Akka and async programming with it. For example doing access files and webservices with it. So async API stack is complete, right? We\u2019ve got&hellip; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[15,17],"tags":[291,245,162,295,226],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/posts\/2619"}],"collection":[{"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/comments?post=2619"}],"version-history":[{"count":11,"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/posts\/2619\/revisions"}],"predecessor-version":[{"id":3758,"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/posts\/2619\/revisions\/3758"}],"wp:attachment":[{"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/media?parent=2619"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/categories?post=2619"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.gamlor.info\/wordpress\/wp-json\/wp\/v2\/tags?post=2619"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}