77dev

9/28/2016

Imagine that you are a Java developer, and you’re about to start your next big project. You need to make the fundamental decisions that will stick with you for the rest of the project. You want to pick the best object-oriented abstraction of your flexible data model because you do not want to deal with plain SQL. You want to support all kinds of data, and ideally, support all sorts of databases.

The obvious answer is to just use Hibernate, right? 90% of Java developers would agree with you, but does that make it the right decision?

Let’s take a look at what can go wrong if you blindly use Hibernate just because it’s the accepted standard.

Consider Monica, a Java developer. Monica has recently been promoted to the role of architect and is now responsible for laying out the technology stack for a new product at her company. She knows that in the Java world there is only one good tool for handling database communication: Hibernate. Hibernate is a well known and supported JPA standard. However, it is always a good idea to check a few things before starting a project. Fortunately, her colleague, Ben, knows the right guy.

4 Years Ago, Hibernate Sounds Like A Silver Bullet

Ben - Hello Monica, I’d like to introduce John. He’s a Hibernate expert, and he’s going to help you.

Monica - Hey John, glad you found some time for me. So, we are building our Next Big Thing, you know. We are planning on becoming the next Facebook or Google. Busy days. It’s going to be huge. Absolutely fantastic! Everybody is so excited! I’ve been promoted to the role of an architect, so now I have to select the stack we will be using. The only missing part is persistence …

John - Hibernate!

Monica - Yes! Exactly! Just what I was thinking! It seems like a perfect match and the real deal for us. A true enterprise solution for a true enterprise problem, proven by the market and with a long history. I’ve heard so many positive experiences with it. However, I have an issue with one of our teammates; he is totally against it. He knows a lot about databases, and he’s afraid of adding another layer between our application and the database. He is super smart, and I need some really good arguments to convince him this is a good decision. Can you help me with that?

John - Of course! I will be glad to. Hibernate is, indeed, an outstanding tool. It’s widely used in big, true enterprise solutions, like banks. You can’t go wrong with it. Think persistence: Pick Hibernate. If you are writing in Java, this is absolutely the right choice, plus you have ports for other languages. See how many job descriptions require it!

Monica - I absolutely agree! I have the same feelings about it. In a previous project, we were using mostly SQL via plain old JDBC. Ridiculous! I know! But, here’s the thing: We have really smart SQL guys in the team and when they saw SQL generated by Hibernate they got nervous. It seemed ugly and unreadable; will this be a problem in future?

John - Look. DBA guys have a different perspective. They are afraid of Hibernate because it seems to replace their role in the project. Moreover, databases have built-in query optimizers so you don’t need to worry how those queries will actually look. The database will optimize it for you. It’s all about rapid development, which SQL can’t do.

Monica - Really?! No longer dealing with SQL? Amazing! Last time a DBA spent weeks trying to optimize some queries. Weeks! Oh, I feel so embarrassed telling you this, but did you know that we were using … stored procedures (laughing). Oh, it was such a mess. Can you believe the project is still using it? I feel so sorry for people out there. They still have to write this tedious code over and over again. I wonder if it’s still a Java or SQL project?

John - That’s exactly the difference between an object-oriented approach and the relational one. It’s a so-called object-oriented impedance mismatch. Hibernate can close this gap. Developers can focus on building business logic. Push features make stakeholders and the entire management happy. Do the things that matter most: Business! Lots of boilerplate code will disappear, and you’ll have a magical, invisible, but reliable, connection between the logic and the data.

Monica - Mutual cooperation. Full synergy. Like the database was part of the language from the very beginning. I am so happy I get to be a leader of this technological leap of faith. It’s like warp speed in the software trek.

John - Yep! You’ve got it!

Monica - Oh gosh, I’m so excited! Thank you, John! I’m ready!

3 Years Ago, Growing Pains With Non-Flexible Solutions

Monica - Hey John, remember the project we spoke about last year?

John - Sure. How’s it going?

Monica - We are going to production soon. Everything is fine, but some questions have popped up.

John - Sure, hit me.

Monica - Well, we can no longer generate our database schema from scratch. What’s the best way to support schema changes without losing data?

John - Well, first, Hibernate isn’t intended to be used as production migration tool. Use something like FlywayDB or Liquibase. It’s pretty simple. You write down migration scripts, then you update the entity model along with the Hibernate mappings, so it keeps in sync with the actual database structure.

Monica - Hmm, I see. We were using just plain SQL migration in the previous project.

John - That’s fine too. As long as you keep the entity model and schema in sync, do it how you like.

Monica - I see. There’s another thing. We’re always struggling with lazy/eager fetching problems. At one point, we decided to do everything eagerly, but it seems suboptimal, and besides, sometimes it’s not possible to access some fields because there is no session, or something like that. Is that normal?

John - You need to learn more about Hibernate. Mapping from the database is not straightforward. Basically, there are multiple ways of doing it. You just need to pick a way that works for you. Lazy fetching gives you the ability to load those objects on demand, but you need to operate within an active session.

Monica - We’re still struggling with which database engine to use for the final deployment. I thoughtHibernate was portable, but we have some native queries that use some MS SQL magic, and we’d actually like to go with MySQL in the production.

John - Hibernate gives you flexibility as long as you are using detached criteria or HQL; any native queries will just bind your solution to the database.

Monica - Seems like we have to stick to the MS SQL then. Last question: My teammate said that there is no “limit” keyword in HQL. I thought he was joking, but I couldn’t find it either. Sorry for the stupid question…

John - Indeed, there is no “limit” keyword in HQL. You can control this via query object since it’s database-vendor specific.

Monica - Seems weird that all the other elements are in HQL. Nevermind. Thanks for your time!

2 Years Ago, We’re Now Hacking Together Solutions In SQL Again

Monica - John, at the beginning we weren’t going to deal with SQL, but now it seems like we have to. Our needs are growing, and it seems like there’s no way around it. It feels wrong, but we’ve started using SQL again on a daily basis.

John - Well, it’s not wrong. You didn’t have to focus on the database at the very beginning. However, as the project grows, it’s good to use SQL and work on the performance optimization.

Monica - Sometimes we spend days looking for errors. It seems like we have to analyze Hibernate-generated SQL because we have no idea why it’s not working as expected and it’s producing unexpected results. We hit some problems that are well known in the Hibernate bug tracker. Additionally, it’s hard to write proper migrations while keeping the entity model in sync. It’s time-consuming since we need to learn a lot about Hibernate internals and predict how it’ll work.

John - There’s always a learning curve. You don’t have to write much, but you do need to know how it works.

Monica - Working with bigger datasets is also annoying. Recently, we did a massive import to the database, and it was painfully slow. Then we found out that we had to clear the session to make it faster. Even so, it’s still significantly slower, so we decided to rewrite it as plain SQL statements. What’s funny is that writing plain SQL was actually the fastest way of doing it, so we decided to do it as our last option.

John - Import is not an object-oriented process. Hibernate focuses on object-oriented design. Remember that you can always use native queries.

Monica - Can you help me understand how Hibernate cache works? I just don’t get it. There are some first/second level caches. What is this all about?

John - Sure. It’s a so-called transaction-level cache of persistent data. It’s possible to configure a cluster or JVM-level cache on a class-by-class and collection-by-collection basis. You can even plug in a clustered cache. But remember that caches aren’t aware of any changes made to the persistent store by another application. They can, however, be configured to delete expired cached data regularly.

Monica - Sorry, think I’m having a bad day. Can you explain this a bit more?

John - Sure. Whenever you pass an object to save, update, saveOrUpdate, or retrieve it via load,get, list, iterate or scroll, that object is added to the internal cache of the session. You can also remove the object and its collections from the first-level cache.

Monica - Er…

John - Additionally, you can control cache modes. You can use normal mode to read and write items to the second-level cache. Use get mode to read from the second level but you can’t write back. Use put, which is same as get but you can’t read from the second level. You can also userefresh mode, which is going to write to the second level, but not read from it and bypass the use minimal puts property, forcing a refresh of the second-level cache for all items read from the database.

2 Weeks Ago, Giving Up On Hibernate

Monica - John, I thought we were entering a new era of software development. I thought we were doing a light-year’s jump. But, after four years, it seems like we are still dealing with all the same problems, only from a different angle. I had to learn Hibernate architecture, configuration, logging, naming strategies, tuplizers, entity name resolvers, enhanced identifier generators, identifier generator optimization, union-subclasses, XDoclet markup, bidirectional associations with indexed collections, ternary associations, idbag, mixing implicit polymorphism with other inheritance mappings, replicating object between two different datastores, detached objects and automatic versioning, connection release modes, stateless session interface, taxonomy of collection persistence, cache levels, lazy or eager fetching and many, many more. Even with everything I know, it seems like we’ve failed badly. It’s a software fiasco! Ultimate failure! Disaster! Armageddon!

John - Wait! What happened?

Monica - We’ve reached a dead end. Our application performance is ridiculously slow! To get a report, we have to wait two days! Two days to actually generate a dashboard for a customer. It means every day we have to increase our calculation tail, while our dashboard gets more and more outdated. Our DBA expert has been working two months to optimize some queries, while our database structure is a complete mess. There are developers supporting him, but the problem is the DBA is thinking in SQL, and the developers are spending days trying to translate this into detached criteria or HQL format. We are trying to use native SQL as much as possible since performance is crucial at the moment. Anyway, we can’t do much since the database schema just seems to be wrong. It felt right from the object-oriented perspective, but it seems ridiculous from the relational one. I’m asking myself: How has this happened? The developers are telling us changing the entities structure is going to be a massive effort, so we can’t afford that. I remember in the previous project it was a mess, but we never wound up at such a critical point. We were able to write an entirely different application to work with the data. Now, it’s risky to modify those generated tables since it’s really hard to make sure the entity model will always behave properly. And this isn’t even the worst part! To increase performance, we have to solve not only database issues, but also issues with the entire layer between our database and the application. It’s overwhelming! We have these new guys, you know, consultants. They are trying to extract data, put it into some other storage and then perform calculations from the outside. It’s all taking too much time!

John - I don’t know what to say.

Monica - You see John; I don’t want to blame you. I picked Hibernate to solve all these problems, but now I’ve learned it’s not a silver bullet. The damage has been done, and it’s irreversible. Actually, I would like to ask you something: I spent the last four years of my career dealing with Hibernate stuff. It seems I do not have a future at my current company. Can you help me?

Today, So What’s The Lesson Learned?

John - Hey, Peter, let me introduce Monica.

Peter - Hey, Monica! We’re building our new next big thing you know. It is going to be huge! We want to be like Uber! Do you know maybe how persistence…

Monica - Not Hibernate!

Wrap Up

Monica is a Hibernate expert. However, Hibernate in this instance was a wrong decision. The moment she discovered that her solution turned into a bigger problem than the original, it was the largest threat to the whole project.

Data is the central purpose of the application and, like it or not, affects the entire architecture. As we learned from the story, do not use Hibernate just because your Java application is using a database or because of social proof. Pick a solution that embraces flexibility. There are plenty of options for robust JDBC wrappers, such as JdbcTemplate or Fluent JDBC Wrapper. Alternatively, there are other powerful solutions, such as jOOQ.

3/08/2016

Working on data set visualization with R and shiny for Facebook data insights, we've faced a serious problem with some, one would say "simple feature".

"Hovering on plot data, shall display popover (tooltip) with the user picture, active link, short description and some other data".
First we've tried ggvis library. It was very promising and the tooltips worked like a charm.
However, very soon we've found out that library is not mature enough and leads to problems with plot interactions i.e. zooming - to get rid of marque artifacts we would have to patch the library with some custom JS and other "fixes". This way looked "quick and dirty" so we've decided to explore the possibilities of well known and mature ggplot. If it's possible to display the data we need in "panel", let's fetch the panel with needed HTML data and play with it's position and look. The final solution is as follows:

Tooltips are managed by combination of shiny+ggplot hover functionality and CSS styles.
By setting hover argument of 'plotOutput' we can access hover data from the server side, as an ordinary input.

Hover input is a list with:

position of cursor ON the image;

domain - that is values of variables at the plotting area edges;

range - that is position of plotting area edges in pixels relative to whole image element.

Additionally for ggplot mappings are returned.

To create tooltip first we need to identify position of the cursor inside the image element. We do it by calculating distances from left and top edge of image element from hover data. Then we create tooltip, in this app it is 'wellPanel' with required informations inside, and 'position' property set to 'absolute' and set 'left' and 'top' properties to calculated values.

However, 'absolute' position is defined as relative to the nearest positioned ancestor. Because we want to position tooltip inside the image, we need to put both 'plotOutput' with image and 'uiOutput' with tooltip content inside additional 'div' element with 'position' property set to 'relative'.

We don't set top, left etc. for this element, so the actual position of the image doesn't change - it's edges are identical as previously, so we can use 'div' (for positioning tooltip) as substitute for image.

Share your point in comments.

7/15/2014

The problem

I ran into a problem couple weeks ago with my Tomcat. I deployed my webapp and it's been up and running for some time and one day it suddenly crashed. My customer reported that the application is not working, and he gets Service unavailable error from Apache.

I started browsing log files: system logs, Apache logs, Tomcat logs, my own application logs, I read everything, and there was no trace of any error, all I could find were these lines:

Nothing unusual, it looked like Tomcat was just being properly shut down, just like that. You may guess, that it wasn't me who did this :) After some googling, I found nothing more than "maybe you have System.exit() somewhere in your code?". Damn, no, I don't.

The culprit

A friend of mine looked it all up for me, and he agreed that nothing unusual happens except that Tomcat's process is being normally shut down. We started analyzing bash scripts I used for starting up the server (they're simplified for educational purposes):

The last line was the problem.

The solution

I used to log in to my VPS with PuTTY and execute my scripts, it usually worked fine. But from time to time it happened that after deploying the webapp, I didn't kill my script after it did his job. tail -f shows the logs continuously so the script doesn't finish it's execution automatically. When the ssh connection timed out while my script was still running, shell killed it WITH IT'S CHILD PROCESSES - a Tomcat server among others and that was the whole problem. Simple solution - DO NOT use tail -f in your bash scripts, or at least be aware of it and make sure you kill the script before closing the ssh connection.

7/06/2014

Jasper + Grails/Java

Recently, I've been struggling with making this couple work together. JasperReports might be very useful when it comes to rendering reports out of your data, it can read your database and transform it into tables and charts. It sounds very nice in theory, but when your client wants to not only see the data itself, but also he wants to see it looking good then you're in trouble. Mine did, and I had some hard time setting it all up.

At the time I was working on it, there was a Jasper plugin for Grails version 1.6 which used some ancient version of JasperReports library and hence it was not compatible with my report I created in JasperSoft Studio 5.5.0. I had to build 1.7-SNAPSHOT version of the plugin to fix this. Luckily, couple days later plugin got updated, and I could use the officially released one (currently, version 1.8 is out).

The battle rages on

Nevertheless, even though I had it all theoretically compatible, I still couldn't render any of my reports. The problem now were... the fonts. My client wanted the reports to look nice, there were designs for it and they even wanted it to be printable as an advertising materials. It meant that reports should share look-and-feel with the corresponding website, including graphics and fonts. And this is where the real battle begun...

Font not found, font not available

My website, and my reports created in JasperSoft Studio have been using Ubuntu font family and it all worked fine there, but when trying to render the same report through Grails web application, it said that:

net.sf.jasperreports.engine.util.JRFontNotFoundException:
Font 'Ubuntu'is not available to the JVM.
See the Javadoc for more details.

First I tried to add the font to my JVM in very brutal low-level way - by adding to JavaHome/lib/fontconfig.properties but it didn't help much. After a while of stack-overflowing I found that I have to create a special .jar file with definition of my fonts for the reports, and that easiest way to do it is using iReports, so I did. It helped a little because now I was able to render my reports in all the other file formats, however only .pptx were actually usable, the rest didn't look as I expected. Pdfs though still didn't cooperate but the error message now has changed a bit.

Could not load the following font

I stuck with new error message like this:

Class net.sf.jasperreports.engine.JRRuntimeException Message Could not load the following font : pdfFontName : Ubuntu pdfEncoding : UTF-8 isPdfEmbedded : false
I couldn't explain it because the same report was being rendered fine in iReports and JasperSoft Studio, and also it worked fine in Grails when rendered to .pptx format, but not the bloody PDF. After some time of debugging guts of jasper's rendering engine and changing random things in order to find any clue, I started to wonder where does it get the pdfEncoding : UTF-8 property from and that was a milestone. It turned out that jasper uses three system properties:

I started digging through JasperSoft Studio's options and I found that it sets some values to them, and that was a silver bullet.

Helvetica to the rescue

I don't know why, but this default.pdf.font.name by default was set to Helvetica. Even though I don't have a font like this anywhere in my OS (Windows 8), neither do I have it anywhere on my VPS (Ubuntu), it made the whole thing work.

These four lines of code placed in my BootStrap.groovy worked like a charm and now I was finally happy to see my reports in a pdf file.

76 KBHBase Apache HBase is the Hadoop database. Use it when you need
random, realtime read/write access to your Big Data.
This project's goal is the hosting of very large tables -- billions of rows X millions of columns -- atop clusters
of commodity hardware.

75 KBparent Parent POM.xml with most oftenly used pre-configured features

72 KBApache Isis Core Core framework, providing metamodel, runtime and core APIs.
Also defines standard build process, as well as
standard set of 3rd party dependencies (eg for testing and
logging frameworks).

64 KBTitan: Distributed Graph Database Titan is a distributed graph database optimized for processing massive-scale graphs represented over a machine
cluster.
Titan separates the concerns of graph processing and manipulation from storing the graph on disk,
delegating that concern to an extensible set of persistence solutions.

48 KBPatternTesting Parent PatternTesting-Parent is the parent of the PatternTesting family.
The goal of PatternTesting is to verify Architecture/Design recommendations
and to write better code more easily.
It uses AOP and AspectJ to perform this feat.

41 KBSciJava Projects SciJava aims to provide an overview of available Java libraries for scientific computing. This POM provides a parent from which participating projects can declare their build configurations. It ensures that projects all use a compatible build environment, including Java version, as well as versions of dependencies and plugins.
Projects wishing to use pom-scijava as a parent project need to override the , , , , , and the sections.

31 KBGroupDocs Java Viewer GroupDocs.Viewer is an online document viewer that lets you read documents in your browser, regardless of
whether you have the software that they were created in. You can view many types to word processing documents
(DOC, DOCX, TXT, RTF, ODT), presentations (PPT, PPTX), spreadsheets (XLS, XLSX), portable files (PDF), and image
files (JPG, BMP, GIF, TIFF). For each file, you get a high-fidelity rendering, showing the document just as it
would if you opened it in the software it was created in. Layout and formatting is retained and you see an exact
copy of the original.
GroupDocs.Viewer lets you really read the document. You can search text documents, copy text and even embed the
document РІР‚вЂњ GroupDocs.Viewer and all - in a web page. You can print or download the file from GroupDocs.Viewer if
you need to work with it offline.

30 KBCRaSH parent The CRaSH is a shell for Java Content Repository that comes bundled as a war file to deploy in eXo Portal 2.5 or GateIn

30 KBGroupDocs Java Annotation GroupDocs.Annotation lets you add notes to PDF and Word documents, as well as to image files – all directly from
a web browser. It is a convenient web-based tool that doesn’t require any software installation and allows you
and your colleagues to annotate documents online.
Moreover, with GroupDocs.Annotation, you can add your notes to a document and then send it for approval or
review, or share the document with others for online collaborative review in real-time. This way you get
feedback faster and can keep everyone’s notes and comments in a single file.

30 KBGroupDocs Java Annotation GroupDocs.Annotation lets you add notes to PDF and Word documents, as well as to image files – all directly from
a web browser. It is a convenient web-based tool that doesn’t require any software installation and allows you
and your colleagues to annotate documents online.
Moreover, with GroupDocs.Annotation, you can add your notes to a document and then send it for approval or
review, or share the document with others for online collaborative review in real-time. This way you get
feedback faster and can keep everyone’s notes and comments in a single file.

29 KBWicketStuff Core Parent WicketStuff Core Parent is the parent project for all of the core WicketStuff projects. It tries
to unify the WS projects into a common structure that shares a common build / release cycle. This
will hopefully keep WicketStuff projects more in sync with Wicket and make it easier on new users.

29 KBCRaSH Parent The CRaSH is a shell for Java Content Repository that comes bundled as a war file to deploy in eXo Portal 2.5 or GateIn

29 KBMozilla Rhino Rhino is an open-source implementation of JavaScript written entirely in
Java. It is typically embedded into Java applications to provide
scripting to end users.

29 KBApache Crunch Apache Crunch is a Java library for writing, testing, and running
Hadoop MapReduce pipelines, based on Google's FlumeJava. Its goal
is to make pipelines that are composed of many user-defined functions
simple to write, easy to test, and efficient to run.

28 KBThe Netty Project The Netty project is an effort to provide an asynchronous event-driven
network application framework and tools for rapid development of
maintainable high performance and high scalability protocol servers and
clients. In other words, Netty is a NIO client server framework which
enables quick and easy development of network applications such as protocol
servers and clients. It greatly simplifies and streamlines network
programming such as TCP and UDP socket server.

About authors & company

Attention to detail, leaving nothing to chance - This is the shortest definition of our spirit. We believe we can deliver highest quality software products and services following this simple rule. We know every process can be verified, controlled and monitored, but true quality is produced naturally by right people.