Reporting In NoSQL
Posted by gamlerhart in software-development on February 6, 2010
Rob Conery, creator of SubSonic, states his opinion about reporting in NoSQL database: Use the right tool for the right job. Consider splitting up your data in ‘domain-data’ and ‘reporting-data’. The domain-data is kept databases which fits best, like a document-store, object-database etc. The data for the reporting stored separately. So it’s a separation of data concerns.
First you may think: “Oh no, more work”. Two data-stores, that’s even more code, more libraries etc you have to maintain. But I’m not if that is really true. When I’m thinking about all the ‘pain’ you go through to map you domain into a normalized relational model. Especially when the relational-model is used for both, reporting and the application-domain.
For example I’ve once build a quite sophisticated survey application. Well I first designed my domain-model for representing a survey. Like domain-objects for a question, possible-answers, rating-criteria’s etc. I’ve build a nice model an then mapped it onto a relational representation. Now of course a survey-application also needs detailed reports. Well first the domain-model was easy to work with. But it wasn’t optimal for reporting. Things got worse because I tried to ‘hammer’ the reporting-’nails’ with a ORM-mapper. And it was a nightmare, because the ORM couldn’t handle complex queries appropriately . So I needed to tweak the domain-model to make it more reporting friendly. In summary, I spend a lot of time tweaking and changing the domain-model and ended with something, which was neither good for reporting nor a good domain-model.
When I think back, I could have saved a lot of trouble by separating the both parts completely. Use nice domain-model for the survey, preferably stored in object-database or maybe a document-database. This is data which is actively changed and updated. And then for reporting, additional data is stored In big, flat and demoralized tables.
Conclusion: I can imagine that separating data for different purposes can be a real benefit.
What Am I Working On?
Posted by gamlerhart in 42, java, projects, software-development on February 4, 2010
I’m currently extending the dSail System-software. The idea is to provide real-time data about the air flow on a sailing boat. This information can help you tweaking the sail position to reach the highest speed possible. In order to get this data sensor-nodes are attached to the sail. Each node has multiple sensors, like pressure, wind-speed etc. Each node send its data to an analysis software. I’m working on that analysis software. It parses the raw data, logs it and shows nice real-time charts.
Right now we’re adoption the software for other projects and purposes. The goal is to create a toolkit for parsing, storing and visualizing all kinds of sensor-data. Air-pressure, temperature, CO2-content, rotation and acceleration and so on and so forth. The sky it the limit.
Anyway, I really enjoy working on it =)
Throwing Checked Exceptions Like Unchecked Exceptions in Java
Posted by gamlerhart in java on February 3, 2010
Everyone who’s programming Java knows checked exceptions. You only can throw a checked exception when you’ve declare it. And you have to catch an checked exception, no matter whether it actually can occur.
First my opinion: I think check exception are a failure. The intention was good, but in practice they cause pain, boilerplate-code and in general cause more problems than they solve.
So in my code I avoid checked exceptions. Unfortunately the existing JAVA-API’s are littered with checked exceptions. So I still need to deal with them. The usual strategy looks like this:
try{ someObject.doOperation(); } catch(OperationException e){ throw new RuntimeException("See inner exception",e); }
This works fine, but fills up the stack-trace with unnecessary garbage. (grey= garbage, black= important). And because you normally have multiple such rethrow-blocks in your code, the cause is nested somewhere in a huge stack trace:
Exception in thread "main" java.lang.RuntimeException: See inner exception
at ch.gamlor.test.ExceptionBlogPost.methodWithoutCheckedException(ExceptionBlogPost.java:27)
at ch.gamlor.test.ExceptionBlogPost.openSimulation(ExceptionBlogPost.java:20)
at ch.gamlor.test.ExceptionBlogPost.demostrateStackTrace(ExceptionBlogPost.java:15)
at ch.gamlor.test.ExceptionBlogPost.main(ExceptionBlogPost.java:11)
Caused by: ch.gamlor.test.OperationFailedException: Could not find appropriate example
at ch.gamlor.test.ExceptionBlogPost.legacyMethodWithCheckedExceptions(ExceptionBlogPost.java:32)
at ch.gamlor.test.ExceptionBlogPost.methodWithoutCheckedException(ExceptionBlogPost.java:25)
... 3 more
The Hack
Now recently I’ve discovered this post on StackOverflow, about unknown ‘features’ of the Java language. There someone showed how to throw checked exception without declaring them. Basically he combines the exception with generics and misuses the type erasure of the compiler, so that the type of the exception is erased =). Btw. the post includes another surprise: You can declare a class in methods =).
Here’s the code and more details how it works:
public final class UncheckedThrow { private UncheckedThrow(){} public static void throwUnchecked(final Exception ex){ // Now we use the 'generic' method. Normally the type T is inferred // from the parameters. However you can specify the type also explicit! // Now we du just that! We use the RuntimeException as type! // That means the throwsUnchecked throws an unchecked exception! // Since the types are erased, no type-information is there to prevent this! UncheckedThrow.<RuntimeException>throwsUnchecked(ex); } /** * Remember, Generics are erased in Java. So this basically throws an Exception. The real * Type of T is lost during the compilation */ public static <T extends Exception> void throwsUnchecked(Exception toThrow) throws T{ // Since the type is erased, this cast actually does nothing!!! // we can throw any exception throw (T) toThrow; } }
Now you just can use this method to ‘rethrow’.:
try{ legacyMethodWithCheckedExceptions(); } catch (OperationFailedException e){ UncheckedThrow.throwUnchecked(e); }
And surprisingly this works! And even better, no rethrow-garbage, and also the throw location is right!
Exception in thread "main" ch.gamlor.test.OperationFailedException: Could not find appropriate exampleat ch.gamlor.test.ExceptionBlogPost.legacyMethodWithCheckedExceptions(ExceptionBlogPost.java:32) at ch.gamlor.test.ExceptionBlogPost.methodWithoutCheckedException(ExceptionBlogPost.java:25) at ch.gamlor.test.ExceptionBlogPost.openSimulation(ExceptionBlogPost.java:20) at ch.gamlor.test.ExceptionBlogPost.demostrateStackTrace(ExceptionBlogPost.java:15) at ch.gamlor.test.ExceptionBlogPost.main(ExceptionBlogPost.java:11)
Fine Tuning The Hack
Now we’re facing the next problem. Supposed we’ve a method which returns a value and we use out little hack. Then we’ve a problem. Since the compiler doesn’t see the ‘exception’, it requires you to return something or throw something. Well that’s easy to fix. Change the signature from void to RuntimeException, so that you can throw it. But the implementation never actually returns, but throws the suppress checked exception.
public final class UncheckedThrow { private UncheckedThrow(){} // Now this returns an exception, so that you can satisfy the compiler by throwing it. // But in reality we throw the given exception! public static RuntimeException throwUnchecked(final Exception ex){ // Now we use the 'generic' method. Normally the type T is inferred // from the parameters. However you can specify the type also explicit! // Now we du just that! We use the RuntimeException as type! // That means the throwsUnchecked throws an unchecked exception! // Since the types are erased, no type-information is there to prevent this! UncheckedThrow.<RuntimeException>throwsUnchecked(ex); // This is here is only to satisfy the compiler. It's actually unreachable code! throw new AssertionError("This code should be unreachable. Something went terrible wrong here!"); } /** * Remember, Generics are erased in Java. So this basically throws an Exception. The real * Type of T is lost during the compilation */ public static <T extends Exception> void throwsUnchecked(Exception toThrow) throws T{ // Since the type is erased, this cast actually does nothing!!! // we can throw any exception throw (T) toThrow; } }
Now you can use it anywhere!
private int methodWithoutCheckedException() { try{ return legacyMethodWithCheckedExceptions(); } catch (OperationFailedException e){ throw UncheckedThrow.throwUnchecked(e); } }
It’s not only limited to rethrows, you can throw a checked exception anywhere. However I think that’s a very bad idea!
private String demoJustThrow(){ throw UncheckedThrow.throwUnchecked(new IOException("See, I'm not declared")); }
Conclusion
I’ve demonstrated how to throw checked exception like unchecked ones. Yes, this is certainly a real evil hack. However I also think that avoiding unnecessary garbage in stack traces is a real benefit. So I’ve started to use this for rethrowing checked exceptions in my projects.
I’ve also to mention that this could be a weakness in the javac-compiler. I haven’t checked this against the language definition =).
The source-file is here. Try it yourself: UncheckedThrow.java
The Way of the Whiteboard, Modern Hardware, Writing DSLs with LINQ and Component Relationships
Posted by gamlerhart in .NET, 42, software-development on January 25, 2010
- The Way of the Whiteboard: Persuading with Pictures: A enlightening and wonderful talk about explaining stuff with pictures. The advices in this talk do not only apply to every area. Watch it. There’s actually a predecessor talk here, with very similar content. His book is on my ‘to-buy’-list. =)
- In the presentation ‘Not Your Father’s Von Neumann Machine: A Crash Course in Modern Hardware’ Cliff Click explains how modern hardware works. I recommend it to anyone who’s concerned about performance. The basic key message is that modern hardware is extremely complex so that it’s nearly impossible to reason about performance. And that cache-misses dominate the performance of the CPU.
- Nicholas Blumhardt shows a beautiful way to create small external domain specific language using LINQ-expression.
- And yet another post from Nicholas Blumhardt about different kinds of component-relations and Autofac 2.x.
db4o: Tools For Adhoc Querying And Modification
Posted by gamlerhart in .NET, db4o on January 15, 2010
Remember my very first post about db4o? There I’ve made the statement that the tool-support is very bad. Well I don’t revise my opinion here. The situation is still bad compared to the relational database world. Mostly that’s because relational databases have a giant user base, are well known and have some common interfaces (JDBC, ODBC, etc).
(All posts of this series: the basics, activation, object-identity, transactions, persistent classes, single container concurrency, Queries in Java, C# 2.0, client-server concurrency, transparent persistence, adhoc query tools)
Now enough complaining. Only complaining doesn’t help anyway! In this post I’ll introduce two tools and show briefly, what you can do with them. Let’s start:
ObjectManager Enterprise (OME)
This is a tool which comes bundled with db4o. In the Java-Version it’s a Eclipse-plugin. In the .NET-version it’s a Visual Studio add-on.
For the .NET-version there are installers in the ‘ome2005’ and ‘ome2008’-directory of the db4o-distribution. Pick the one which corresponds to your Visual Studio version and install it. After restarting Visual Studio, you’ve a new buttons in you’re toolbar and a new menu-point ‘Tools’->’ObjectManager Enterprise’.
For the Java-Version there’s a zip-file in the ‘ome’-directory of the db4o-distribution. Unzip it somewhere. Open Eclipse, go to ‘Help’->’Install New Software’. Click on ‘Add’, then ‘Local’ and choose the directory you’ve unzipped the zip-file before. The click just through all the confirm-dialogs etc. Then restart Eclipse. Now there’s a new menu-point ‘OME’ in Eclipse.
Now you can connect to a db4o-database. Either you can connect to a running db4o-server or open a db4o-database-file directly.
Queries with OME
Now in the OME you don’t use a query-language for querying the database. This is quite a surprise. So how do you build queries then? You build then by drag and drop! On the left the OME shows a list of all known classes. When you expand such a class, you see the fields. Fields which are no primitives can be extended further. To build a query you choose a object-type, lets say ‘Person’. Then you expand the ‘Person’-class on the left, choose the fields you want to query and drag them into the ‘Query-Builder’-window. After sticking the criterions together, you can run the query. Furthermore you also can update field of persisted data.
Here a little screen-cast, which shows this:
To get an overview of the OME-Features, you also might take a look at this screen-cast.
LINQPad
Now I’m not a fan of this visual query-building. I prefer a query-language. But db4o doesn’t provide a language like SQL, but rather utilized the language of the platform. Now .NET brings a wonderful query-language, LINQ. And together with the excellent LINQPad you can use it like a traditional SQL-console. However some preparation is required:
- We need to add the db4o-assemblies. LINQPad has no idea about db4o, therefore we need to include the db4o database engine. To achieve this, go to the ‘Query’->’Advanced Query Properties’. On the dialog, click on ‘Add…’, then on ‘Browse…’. Browse to the db4o-assemblies. Add the ‘Db4objects.Db4o.dll’, ‘Db4objects.Db4o.Linq.dll’, ‘Cecil.FlowAnalysis.dll’ and the ‘Mono.Cecil.dll’-assemblies. Keep the ‘Advanced Query Properties’-dialog open for the next step.
- In the ‘Advanced Query Properties’-dialog open again the ‘Ad…’-dialog. Add you’re assembly which contains the stored classes. We need the classes in order to write LINQ-Queries.
- In the ‘Advanced Query Properties’-dialog go the tab ‘Additional Namespace Imports’. Add there the db4o-namespaces:
Db4objects.Db4o.Linq
Db4objects.Db4o
Additional add the namespace of your stored classes, for example:
MyProject.DomainModel - Click on ‘Set as default for new queries’ so that you don’t need to repeat the three steps above next time. Then we’re done.
Queries with LINQPad
Now LINQPad is ready for db4o. Choose “C# statements” or “VB statement” in the language-chooser. Then you can write you’re query like this:
using(var db = Db4oEmbedded.OpenFile(@"C:\file-path\database-file.db4o")){ var result = // query!!! result.Dump(); }
For example:
using(var db = Db4oEmbedded.OpenFile(@"C:\Users\Gamlor\Desktop\test.db4o")){ var result = from Person p in db where p.FirstName.Contains("Roman") select new {p.FirstName, p.LivesAt.Street, p.LivesAt.City.Name}; result.Dump(); }
As you see, you just use normal C# code. So you can run any LINQ-Query on you’re database.
Update with LINQPad
Well updating is also possible. LINQPad is nothing else than a C#-Console. So you just get the objects, change then and store the changes. For example:
using(var db = Db4oEmbedded.OpenFile(@"C:\Users\Gamlor\Desktop\test.db4o")){ var update = (from Person p in db where p.FirstName.Contains("Roman") select p).First(); update.FirstName = "Romanovski"; db.Store(update); update.Dump(); }
Some LINQPad-Tips
Since LINQPad doesn’t know anything about db4o, but is only a interactive C#-console, you are responsible for db4o specific behavior. For example the activation-depth, update-depth etc also apply within LINQPad. So for example when all properties of a nested object are null, it is might not activated yet.
You might also write a little helper class library which you can use within LINQPad together with db4o.
Since LINQPad is a popular tool, you certainly find more resources on the internet.
Conclusion
I’ve introduced two tools for ad-hoc querying and modification of the db4o-database. The Object Manager with it’s graphical query builder and LINQPad. Both are useful for inspecting the database.
However I would like to see more powerful tools. When you know one, please tell me.
Autofac 2.x
Posted by gamlerhart in .NET on January 14, 2010
A new version of favorite dependency-injection-container Autofac is making progress. It follows the same principals, brings some new features etc. However they also refactored the API, so it isn’t compatible with the older versions.
This means, that migrating to the new version is some work. However there are useful new features.
Lots of them I’ve implemented on top of the old Autofac-version:
- Collection Support ("Resolve All"): I’ve implemented something like this. Basically it allows you to resolve multiple implementation of a service. Very useful for some simple plugin-mechanisms. (not every application needs MEF)
- Auto-Generated Factories: Allows you to create multiple instances of an service. The container is responsible for implementing a factory. This feature I’ve also implemented.
- Owned Instances: Autofac has extremely good support for managing resources. Now you can inject a Owned<T> instance, which creates a new disposable scope. When you call Owed<T>.Dispose(), the service T, with its resources and dependencies are disposed.
I’ve implemented something similar: All services exposing the IDisposable-interface create a own scope. As soon as you dispose the service the scope with its resources etc is disposed.
Autofac 2.1 already bring support for the .NET 4.0 stuff like Lazy<T> etc.
I keep an eye on Nicholas Blumhardt’s blog and as soon as Autofac is out of beta, I’ll migrate.
Trailers of Doctor Who, Being Human and Survivors
Posted by gamlerhart in tv and movie on January 6, 2010
Ok, lots of trailers of some of my favorite shows. It going to be a wonderful tv-year.
Doctor Who Series 5
First, my favorite sci-fi series Doctor Who. (blog post about season 1, 2, 3, 4, 4.5). Trailer for series 5:
Being Human Season 2
Then my favorite super-natural comedy-drama (blog post): Trailer for season 2:
Even a clip from the first episode…I could resist despite the spoiler-danger…=(
Survivor Season 2
And to end this post, a more serious drama: Survivors (blog post). Trailer for Season 2
Maven-Reference, Validation and Geek-Videos
Posted by gamlerhart in .NET, 42, channel 9, people on January 6, 2010
- I’ve stumbled over this Maven online book. It’s quite useful for everyone who’s maintaining Maven-builds.
- This post about “Magic” null argument testing describes how to implement an extremely elegant way to check arguments in C#. The check looks like this:
new { argument,anotherArgument }.CheckNotNull();Read the post to find out how it works.
- Do you know co- and contravariance in programming-languages? Well how about co- and contravariance in physics? Have they something in common? Well on geeky series on channel9 Brian Beckman and Erik Meijer try to find out. In the first video Brian introduces co- and contravariance in physics. After that, Brian and Erik try together to find the similarities, Brian representing the physicist, Erik the programming-language guy. The first and second part are available, however the question is still no answered. Therefore a third video will be available later.







(4/5)