Software Development | Hrakaroo

In Defense of Obscure Names

Sun, 07 Nov 2021 00:00:00 +0000

In defense of obscure names

Naming things is hard

As the joke goes

There are only two hard things in Computer Science: cache invalidation and naming things. – Phil Karlton

and while this is funny (perhaps only to those in Computer Science) underlying the quote is a real issue, naming things is really hard.

The challenge is not just in giving something an appropriate name for right now, but in trying to predict how it may evolve and be used in the future.

Maybe you start with a database of bicycles, but as powered bicycles are introduced you start adding those but from there it’s a slippery slope to electric motorcycles and then gas powered motorcycles. Pretty soon your database of bicycles includes all motorcycles and perhaps those three wheeled things I’ve seen driving on the freeway. Unless you had amazing forethought and were able to predict exactly how the industry was going to evolve, chances are whichever name you select is going to be incorrect at some point in the future.

Renaming isn’t always easy

So, if the scope does evolve and the original name is no longer correct we should just rename it, right? Sometimes this is the answer. Tools for software development have come a long way and doing a code refactoring isn’t nearly as hard as it once was. So if we have a variable name, function name, or even a class/object name that is no longer accurate we usually have the ability to safely rename it across all dependencies to a new and more appropriate name.

But the coding level is often where this capability ends. The industry has not yet developed the capability to reliability refactor service names across a distributed environment and trying to do it manually is time consuming and often fraught with errors that can negatively impact uptime.

In defense of the obscure

So if changing the name of something is difficult and if naming things correctly now is hard, (and predicting how they may evolve is nearly impossible,) then maybe the best thing we can do is to purposely give them an obscure name.

When I was in college we were able to select our own email address. Like most people I knew, I opted for the straightforward and used my last name for my email address: gerth@lclark.edu. However, one of my classmates opted for something entirely different, she selected the email address of squidlips@lclark.edu. At first I thought she was just trying to rebel, but her reasoning was actually pretty interesting. Her name was something rather common, (think Jane Doe) and her argument was that while doe@lclark.edu runs the risk of being mistaken for someone else, squidlips@lclark.edu is almost guaranteed not to.

This proved to be more than just an academic observation for as it happened, I was not the only Gerth at Lewis & Clark College. There was one other one, my father, a tenured professor, and I would occasionally receive email intended for him. People had seen gerth@lclark.edu and had incorrectly assumed I was my father, and were sending me information clearly not intended for the students. (Nothing salacious I’m sad to say).

In this case the name (my email address) was even 100% correct and it was still getting used incorrectly. And while while no one may have guessed who Squid Lips was, she never got any email not intended for her.

Obscure is better than wrong

We make snap decisions all the time and if I was looking to get a list of motorcycles and came across a service named bicycles I would immediately cross it off my list. But, if instead the service was named x34a5a or even horsetail I would at least have to consider it. I may first go looking for something more specific, but at least I wouldn’t cross it off my list.

(I also prefer the name horsetail over an obscure id designation as it’s much easier to remember horsetail and distinguish it from pigfoot than trying to remember a20b23 and distinguish it from c23s40.)

So if a thing is going to exist for a while, is likely to evolve in scope, and changing it’s name is non trivial, then giving it an obscure name may actually be better and ultimately easier than trying to come up with the perfect name. So in some cases, naming things may not be all that hard after all.

Avoiding Read/Write Locks

Sat, 06 Nov 2021 00:00:00 +0000

Intuitively, read/write locks seem to be a ideal solution. They prevent the read/write and write/write race conditions while making sure multiple read routines don’t block one another. So as long as the write operations are optimized to be as fast as possible, the overall throughput should be high.

In most cases this is true, but there is a subtlety to read/write locks that can end up causing them to be considerably more expensive than they initially appear.

Consider the following:

Thread 1 requests read access to a shared read/write lock.

At this point in time no one else has requested the lock for either read or write access so a read lock is granted to Thread 1.

Thread 2 requests read access to the same read/write lock.

Since the lock is currently only read locked by Thread 1, Thread 2 is also given read access.

Thread 3 requests write access to the same read/write lock.

Since the lock has been locked for read access by both Thread 1 and Thread 2, Thread 3 is put on a wait list for write access until Thread 1 and Thread 2 release their read locks.

Thread 4 requests read access to the same read/write lock.

What happens here?

A first guess might be to give Thread 4 the read lock, along with Threads 1 and 2, since Thread 3 has not yet obtained the write lock, but lets consider the fall out if we let this happen. Imagine a system where every read operation took a minute to execute and requests are comming in every 30 seconds. If every read request is granted a read lock so long as no write thread had been given a write lock, new read requests would always come in before previous ones finished ensuring that the lock was never free (ie not read locked.) In this situation the write operation would be blocked from every achieving write access. aka, lock starvation.

In order to prevent lock starvation it is necessary that as soon as a write lock is requested every subsequent read lock request is blocked from obtaining their lock until the thread requesting the write lock has been given the write lock, executes, and then releases their lock.

This means that as soon as a write lock is requested any read threads currently running will block all new read requests. So a read/write lock is, worse case, as “expensive” as the time it takes to execute the write operation plus the worse case time it takes to execute a read operation.

Therefore, even if you optimize your system so the write operations are as fast as possible, your overall throughput may still be dependent on your slowest running read operation.

What read/write locks are good for.

Read/Write are useful when you need a memory barrier and every thread on your service should only see the state of some memory either before or after a write operation, but never but never both. Meaning that it is not allowable for some threads to see the value of the memory before the write operations and other threads running at the same time to see the memory value after the write operation.

But, as we saw above, this comes at a cost and in many cases it may not be needed.

For example, suppose we have a data structure that holds the account balance for an individual. Suppose also that this account receives a constant stream of read requests that, for what ever reason, take several minutes to calculate. If we were to use a traditional read/write lock here then every time there is a deposit or withdrawal (a write operation) all read operations will be blocked until all running read operations finish and the write executes.

Although the write operation may be quick, the long execution time on the read operation is going to cause the system to appear blocked while it waits for the write lock. But if we can tolerate concurrent threads seeing two different values then we can remove the block all together and achieve higher overall throughput.

To achieve this we first need to make the data structure immutable. Updates would instead be handled by the write thread copying the immutable data structure locally, modifying it (or modify on copy) and then using an atomic operation to swap in the new data structure.

Read operations running against the old data structure would continue to run and would report the old value, while any new read operation would run over the new data structure and report the new value. Eventually all old read operations would finish and the only view would be of the new data structure, but for a time it would be possible for different threads to report different values.

Destroying the old data structure can be handled either automatically by letting the garbage collector pick it up, or we can add read counters to objects which can tell us when an object is no longer being accessed and we can delete it manually.

We could also re-introduce a simple lock for just the writer operations to ensure that multiple write operations always happen in sequence so they can never overlap and they would never cause inconsistent results.

So while read/write locks often appear, at first blush, like a panacea, they can actually carry a substantial hidden cost. And in some cases the functionality provided by a read/write lock may not even be what you really need.

Go for Java Developers - Part 2

Tue, 01 Oct 2019 00:00:00 +0000

This is my second post on things I’m finding interesting about the Go language. For additional background on this series you might want to read Part 1.

Dependency Management

This time I’m going to focus on dependency management and how go approaches the diamond dependency issue.

First off, Go does not support pre-compiled libraries the way Java does. (Personally, I think compiled Jar files are from a by-gone time and these days cause more harm than good, but that’s not why Go doesn’t have them.) Go doesn’t have them because Go code compiles down into native assembly so it would be impossible to distribute anything precompiled by Go and expect it to work on more than one platform.

So instead, Go publishes libraries as source code with Git tags and avoids artifactory all together. When you want to use a library you create a dependency on a git repo with a specific tag version.

This doesn’t exactly remove the diamond dependency issue as it is entirely possible to have a transitive dependency on two different versions of a library. In this case, as with Java, you still have to select the version you want to use. However, there are two substantial changes.

Can’t ignore the problem

First, you don’t have the option of ignoring or miss the issue. This means that if your service builds in Go, you can be confident your code does not contain any “hidden” mismatch errors. This effectively removes an entire class of errors that you have with Java. (TBH, these don’t pop up too often, but the more common a library is the greater the chance for this to happen. I’ve seen this error triggered with the Guava libraries.) Being able to completely remove them as a class of problems would be nice.

Conflicts identified at the source code level

Second, conflicts are identified at the source code level and not at the version level. This means that PATCH version mismatches won’t break your build so long as they don’t change function signature (and PATCHES really shouldn’t).

I’m not saying that the way Go deals with this is perfect but it does move the problem into a first class issue and forces a resolution. Java could take this same approach if you copied all of your dependencies into your code base as source code. But I did find it as an interesting consequence on how Go handles dependency management.

Go for Java Developers - Part 1

Thu, 01 Aug 2019 00:00:00 +0000

For the last 20+ years I’ve been mostly focused on Java development. But recently I had an opportunity to join a team which was focused on Go and having spent the early part of my career writing C++ I was intrigued by some of the decisions Go made. So this is the first (of possibly several) posts on things I found interesting about the language and since my I have been doing mostly Java development I’m going to be comparing and contrasting from that perspective.

Go interfaces

For this first post I’m going to discuss one specific implication of how Go handles interfaces.

If you are unfamiliar with Go, it uses structural typing for interfaces (which is more generally called “duck typing” meaning “if it walks like a duck and talks like a duck … it’s a duck.”) This means that if a struct in Go has functions on it which match the methods of an interface, instances of that struct are types of that interface. This sounds more complicated than it actually is so lets look at a quick example:

Suppose you wanted to create a Person class and a Named interface in Java, that might look like:

interface Named {
	String getName();
}

class Person implements Named {
	String name;

	String getName() {
    	return name;
	}
}

Pretty straight forward. Now, if we wanted to do a similar thing in Go it would probably look something like:

type Named interface {
     getName() string
}

type Person struct {
     name string
}

func (p Person) getName() string {
     return p.name
}

Ignoring the syntax differences, what is interesting is that in Go we don’t declare Person as implementing Named. This is implied by the function on Person matching the method in the Named interface. The compiler figures this out for us and instances of a Person are automatically of type Named.

So while this isn’t necessarily a new thing (Python mostly lets you do the same thing), this was something new for a statically typed language.

When I first saw this I thought “meh, so you can leave off the ‘implements’ keyword, whatever.” But I think this feature has a really interesting implication.

The setup

Back in Java land, let’s suppose you have a function that takes an SQL PreparedStatement as one of its arguments. Something like:

class Db {
   void setValue(PreparedStatement stmt, String value) {
 	stmt.setString(1, value);
   }
}

You might use this like:

PreparedStatement stmt = conn.prepareStatement("SELECT …");
Db db = new Db();
db.setValue(stmt, "someValue");

Writing a unit test for this would probably look something like:

class DbTest {

	@Test
	public void setValueTest() {
    	Db db = new Db();
    	PreparedStatement stmt = ??
    	db.setValue(stmt, "dog");

    	// todo - verify the statement received the value of "dog"
	}
}

The challenge is creating the test PreparedStatement. You could use one of the mocking frameworks in Java (Mockito, PowerMock, …) to mock the PreparedStatement and test that setValue is called with the value “dog”. While this does work, most/all of the mocking frameworks are basically a nice user interface over a rats nest of reflection calls. (Used incorrectly reflection is a tool to move compile time errors back to the runtime.) However, it is also possible to test this without using a mocking framework by building our own class which implements PreparedStatement:

class TestPreparedStatement implements PreparedStatement {

	String value;

	void setString(int parameterIndex, String x) {
    	this.value = x;
	}

	...
}

An instance of our TestPreparedStatement can now be passed into our setValue method and we can later verify that the internal ‘value’ is set to “dog”.

Ahhh … but there is a devil hidden in these details. PreparedStatement is a massive interface with well over 50 methods. In order to stub out the one method you want (setString) you are going to need to also stub out all of the other ones as well. We don’t ever use them so they can all throw a RuntimeException (and thankfully most modern IDEs can automatically generate this code for you) … but you are still dealing with a lot of boilerplate code.

The switch

Now let’s take a look at this same problem in Go land. (For the sake of argument let’s assume that PreparedStatement both exists in Go and works much in the same way as its Java counterpart).

type Db struct {
}

func (db Db) setValue(stmt PreparedStatement, value string) {
    stmt.setString(1, value)
}

And again it is used like:

stmt := conn.prepareStatement("SELECT …")
db := Db{}
db.setValue(stmt, "someValue")

Now let’s write the unit test:

func SetValueTest(t* testing.T) {
    db := Db{}
    stmt := ??
    db.setValue(stmt, "dog")

    // todo - verify the statement received the value of 'dog
}

We’ve once again hit the same issue. As with Java, we can use the Go’s mock framework (which also uses reflection) or, as before, we can try to roll our own.

Approaching it directly we could build out a TestPreparedStatement as:

type TestPreparedStatement struct {
    value string
}

func (tps TestPreparedStatement) setString(parameterIndex int, x string) {
     tps.value = x
}

But then we are going to have the same issue as we had with Java where we also need to stub out all of the other 50+ methods in the PreparedStatement interface which is just as wasteful as it was in Java. But we do have one other option.

Duck Typing

What if we didn’t stub out all of the other methods and instead created a new interface which only contained the one method we are using? So something like:

type StringSetter interface {
    setString(parameterIndex int, x string)
}

Now, since our TestPreparedStatement implements this one method it is automatically a type of StringSetter. But guess what, the object returned by conn.prepareStatement is also now a type of StringSetter since it too implements setString (by virtue of it being a PreparedStatement which also requires the same method.)

Now, we only need to change the signature of our setValue method to:

func (db Db) setValue(stmt StringSetter, value string) {
    stmt.setString(1, value)
}

This is where the power of the duck typing comes into play as doing this in Java would require modifying the definition of PreparedStatement so it explicitly extend StringSetter, which is virtually impossible. Yet in Go this is trivial as we can assign interfaces to objects by simply copying over the methods we want into the interface.

I found this to be a surprising feature about Go and is not something I had considered before. It feels really powerful and, to be honest, I’m not yet sure if this is a good feature about the language or not, but it is an interesting one.

Three Questions to Ask When Writing a New API

Thu, 06 Jun 2019 00:00:00 +0000

The following is blog post that was written from a speech I presented about questions to consider when writing an API.

Writing a functional API is relatively easy, but writing a good one that’s functional and empowers your users takes planning and patience. Designing a good API is about creating a sense of clarity and simplicity—it’s the bridge between your intention and your users.

Like most software development, building an API is a creative process; it’s impossible to completely define a hard-and-fast set of rules that will work in all cases. Nevertheless, three key questions—derived from what I consider the key characteristics of a good API—can serve you well as functional guideposts as you design and write your API:

Is your API’s usage discoverable?
Is your API composable?
Is your API safe to use?

Let’s take a closer look at each question.

Is your API’s usage discoverable?

In his famous book, The Design of Everyday Things, Don Norman coined the term discoverability. “When we interact with a product,” Norman wrote, “we need to figure out how to work it. This means discovering what it does, how it works, and what operations are possible.”

Take doors, for example: We interact with these standard physical objects every day. Often, based on the presence of affordances like knobs, handles, and push bars, it’s pretty clear how to open or close a door. But on occasion, a door’s design will suggest the opposite of how it actually works, and, as a result, we require instructions before we can properly use it. Just think of how many times you pulled a handle that actually needed to be pushed.

When we use a door the wrong way, we feel silly and stupid, but it’s not our fault. Actually it’s the design that’s bad.

Something similar can happen with a poorly designed API.

Consider the last API you used. How did you learn to use it? Did you read all the documentation first, or did you just jump right in? Maybe you weren’t sure about all of the parameters, so you sent in null for a few values and guessed at others. Did the API throw an error message when you did something wrong, or did it fail silently without any feedback? Did the error message clearly define which parameters were optional and which were not? Did you just keep plugging away until you got it right?

This is how most users will learn your API.

Your users are going to learn just enough to bootstrap themselves, and then they’ll figure the rest out as they go. With this fact in mind, you can help them along the away by increasing your API’s discoverability. You can do this through documentation; adhering to conceptual models; and using concise, symmetrical language.

Assume your users won’t read the documentation—until they need to

Just because your users won’t read your documentation doesn’t mean that you don’t need to provide it. You definitely do. But don’t design your API with the assumption that everyone will read the docs before they use it.

Some users would rather experiment than look up an answer in the docs. Every time I use Java’s substring() method, for example, I can never remember if the second value is an offset or a length, so I just write a little program to try it out both ways. This is usually quicker for me, and more fun, than looking up the answer.

In many cases, users who’ve learned to distrust documentation won’t read the docs anyway, at least not until they get desperate. Documentation is notorious for being out of date or just wrong. Now, this obviously isn’t true of all documentation, but think of how many times you’ve consulted documentation—or a help system or knowledge base—and found that either it provided answers that were totally useless, or it didn’t provide any related answers at all. Plenty of documentation does a poor job of anticipating the questions users might ask or how they might ask them. Additionally, even if users have a sense of what task they want to achieve, they may lack the exact vocabulary or use different terms for that task than the docs, which can make searching difficult.

You should also provide plenty of examples in your documentation—because users want them. Typically, examples are the first things users look for when learning a new API. Only after they gain a little context will they go look at the rest of the documentation. Examples are how users come to understand your API as a whole.

Create a conceptual model of how your API works

Don Norman explains that a conceptual model is “an explanation, usually highly simplified, of how something works.” Conceptual models are not schematics, and they should relate to other known conceptual models.

A good example of a conceptual model is the file system structure used on personal computers. File systems, like those on Mac and Windows operating systems, were intentionally based on the concept of files and folders that we were already familiar with in the physical world. This made it easy for non-technical users to understand and discover how to copy, store, and retrieve files on their PCs.

Even today, Unix uses this conceptual model of files and folders anytime a user attaches a device (e.g. a phone or external hard drive) to an operating system, which has completely eliminated the need for users to “discover” a new API every time they attach a device.

“Objects” in object-oriented programming are another example of a conceptual model. They’re specifically called objects so that we think of them as self-defining entities. Just as a ball object on the computer might support a bounce method, as well as other methods like throw, a ball in real life, through its design, also supports bounce and throw operations. In data-oriented programming, however, you don’t get this conceptual model, so you’re more likely to have a bounce function that will throw an error if you send it anything other than a ball.

Another example of working within conceptual models is the use of “object” in object-oriented programming. In this programming model, objects represent physical objects from the real world, such as servers, databases, and load balancers, and developers create relationships between those objects via APIs.

Use clear, consistent, and symmetrical language

In addition to documenting your API, you should also develop and publish a terminology dictionary for your API—and then use it consistently. For example, I commonly see APIs use terms like host and hostName, and account and accountId, almost interchangeably. Forcing your users to guess what the right call might be, or constantly changing the language, does not promote discoverability.

Like conceptual models, symmetrical language helps users work with your API with certain expectations in place. If your language is symmetrical, an open operation will be balanced with a close, and an add operation will be balanced with a delete.

In Python, for example, you use pop to remove an element, so the expectation would be that you’d use push to add an element, as that’s how it works in most other languages. Instead, Python uses append… and there’s plenty of Google search results from people confused by this poor discoverability.

Is your API composable?

When you build a composable API, you are letting your users select components of the API and use them in whatever pattern they want.

Small and composable methods are easier to describe and document than larger methods that contain a long chain of steps and caveats. They’re also easier to run regression and end-to-end tests against.

Most importantly, though, employing composable components gives your users the tools they need to build their own workflows with your API. You can’t predict all your users’ needs, so don’t force them into one execution pattern. Instead, create composable components and then use your examples to show how to combine them into larger execution patterns.

For example, consider the following methods:

setName(firstName, lastName)

vs.

setFirstName(firstName)
setLastName(lastName)

The second option is more composable than the first, as the second method allows you to easily update the value for lastName. With the first method you would first have to fetch the value of firstName so you could send it back in with the new value for lastName.

The second option is also more extensible, as you can easily add a method to set the middle name:

setMiddleName(middleName)

Finally, the second option is also 100% backwards compatible with existing code. If you were to update the first method to

setName(firstName, middleName, lastName)

you’d break the existing code.

Both you and your users will undoubtedly enjoy the free backwards compatibility, as building from smaller, composable components makes it much easier to extend your API as it grows; and to continue supporting support old operations alongside new ones.

Is your API safe to use?

Ensuring that your API is safe to use—that it won’t behave differently than users expect or break their workflows— is related to the discoverability of an API. But safety is so important that I want to call out the topic separately. When you publish your API, you create a relationship with your users that should be based on trust and transparency. Here’s how to make that happen:

Practice the principle of least astonishment

The principle of least astonishment tells us that a component of a system should behave in a way that most users will expect it to behave. The behavior should not astonish or surprise users.

The setDate method in GNU’s Coreutils, for example, surprises me every time I use it because I expect a set method to set a value and not alter it. If you set the year to any value less than 68, it automatically adds 2000 to the value; and if you set any value between 68 and 100, it automatically adds 1900. Every time I use this method, I’m astonished and have to re-read the documentation to make sure I’m using it correctly.

Follow the contract

Don’t try to interpret what you think your user is trying to do. For example, if your API expects a number, and the user provides a string, don’t try to parse a number out of the string. You aren’t doing anyone any favors: What happens when users enter an empty string: Is that 0 or null?

Design your API so that it’s deterministic and strict.

Trust nothing and fail fast

Similarly, your API should verify everything that users send, and immediately fail on errors. More specifically, garbage-in should not equal garbage-out. Garbage-in should fail. If your users are calling your methods with incorrect values, they may be in discovery mode, intentionally testing the boundaries and trying to figure out what is possible.

Help them understand what’s possible and what isn’t.

Plan to version from the start and aggressively deprecate old versions If you change the signature or external behavior of your API, version it.

And when you do roll an API’s version forward, dedicate time and resources to aggressively migrate users. If that’s not possible, try to rewrite older versions so they proxy to the new implementation. These steps will help avoid creating technical debt—which, like financial debt, definitely accrues interest over time. The longer an outdated version of your API sits around, the more ingrained it becomes in your user base, and the harder it will be to move users off of it. Set a migration date, and make it happen.

If you release a version that is likely to change quickly, make that fact explicit by tagging it as “incubating,” “unstable,” or “beta.” This helps provide breathing room if you need to turn off old versions of your API as you release new ones.

Separate your API from your implementation

Finally, publish your API version separately from its implementation. The implementation is likely to change faster than the API, so don’t tie the two together.

When versioning a library, for example, the API and its implementation are in the same package, so you can’t help but release them at the same time. But you can at least use semantic versioning to make it clear which parts are backwards compatible.

For a service, though, you can publish an API separately from its implementation. In fact, there are plenty of tools, including Apache Thrift, FlatBuffers, and Swagger, that allow you to write your API separately. With these tools, you write your spec and then build your implementation so that it implements the spec.

Nail that first impression

Your API is often a user’s first impression of your system. Spend time on discoverability, composability, and safety to make sure that first impression is a good one. Proper planning and design is critical to the effectiveness and success of your API. Taking the time to think things through will help to make your API a first-class feature—not a mere afterthought or means to an end.

SQL is a terrible API

Thu, 01 Nov 2018 00:00:00 +0000

A long time ago…

In 1974 the first relational database was created and along with it a new user interface language was developed to make it easy to query the data. This language (or rather sub-language) was called Structured English Query Language or SEQUEL. SEQUEL would later evolved into SQL and become the preeminent language for querying relational databases. In the 45+ years since it’s introduction SQL has evolved slightly but it is still largely similar to the version introduced in the 70s. What has changed during that time is how we interact with the database. Today, SQL is used significantly more by applications querying the database than by end users. This usage pattern has created its own ecosystem of utilities and libraries which help applications build SQL queries. One of the more popular libraries for building SQL queries in Java is Hibernate’s Criteria Builder which uses an annotation pre-processor and a builder pattern to facilitate building SQL queries. (Java’s Persistence API was based heavily on Hibernate’s Criteria Builder). But, at the end of the day the builder still generates a basic SQL string which is sent to the target database, parsed and then executed. Hibernate is just one example, but there are hundreds of other libraries that exist across various languages which all seek to provide the same basic functionality. To provide an API which abstracts away the actual building of an SQL query.

So why do these all tools exist? Is an SQL query so complicated to construct that we need libraries and utilities to help us? In general, no, most SQL queries are relatively straightforward to construct. The problem is that SQL was designed for humans and not computers. The inclusion of the word ‘English’ in the original name was not by chance, SQL was intended to be similar enough to English that it would be self descriptive and would only require minimal transformation from the spoken question to the actual query. SQL was not written to make querying easy from other applications. In order to query a database from an application, the application needs to build the SQL query programmatically at run time, which means that all errors in the SQL query string are also going to be discovered at run time.

An Impedance Mismatch

Compiled languages offer an enormous advantage over interpreted languages in that you can be confident that, if the application compiles, then it does not contain any syntax errors. This assurance removes an entire class of errors that exist in pure scripting languages. Yet, by having an application generate SQL you are re-introducing the possibility of a run time syntax error. Furthermore, the errors which do get introduced are almost always related to the construction of the SQL query rather than actual column names or keywords in the query. Meaning the column names and keywords are not as likely to be the source of syntax errors as they don’t normally change based on the user request. For example, an application which allows the searching of available flights is less likely to return different a type of data depending on the destination city. You may get back more or less result, but it will usually be the same basic set of information. Syntax errors around the column names and keywords are often found through basic testing and fixed.

The more common (and harder to find) syntax errors are introduced in the construction of the query itself.

Did you remember a space after the ‘SELECT’ and ‘FROM’ keywords?
Did you join your select fields with a comma, but remember to not include a comma after the last one?
If this is the first filter then we need to add the ‘WHERE’ keyword, but if this is the second one we need to add the ‘AND’ keyword, and don’t forget about the parentheses.

These types of syntax errors are often introduced based on the users search criteria and can be significantly more difficult to find with basic testing due to the sheer number of permutations. This is the real advantage of using a helper library like Hibernate Criteria Builder. They provide an assurance that if you use their libraries, the generated SQL will be free of construction syntax errors. This assurance is most often achieved through massive test coverage and a responsive development team which quickly patches any errors that are found. It would be safe to say that in the 45+ years since the introduction of SQL, millions of lines of code have been written in various languages, all attempting to work around this same basic problem. It’s a huge waste of intellectual effort for what is essentially a self inflicted problem.

Where to go from here

SQL and similar English based DSLs (domain specific languages) are immensely powerful for building complex queries quickly. As an end user, nothing is more frustrating than trying to build a complex search criteria through a form based UI. First enter the subject, then select the predicate, then enter the object, then click the plus sign to add another filter, rinse and repeat. This becomes incredibly tedious very quickly and makes building complex groupings all but impossible.

This is where an English based DSL really shines. It allows a seasoned user to rapidly create complex queries as they are doing it interactively and have to do minimum mapping from what they are trying to search for to the actual query they are running (just as SQL was initially designed for). But that is where the DSL should remain, as a tool for the end user. Do not just create an API which takes the DSL directly as, at best, you will be recreating the same impedance mismatch and likely forcing your users to create the same set of “builder” libraries.

Instead, define a flexible structure which can be used to describe a query. JSON, XML or even one of the compressed protocols like Thrift or Flatbuffers could work. Users can then create their query programmatically and completely remove the risk of introducing a construction based error because they forgot a space or included a comma at the wrong location. Your API can either accept this structure directly for querying, or you can split it up and have different endpoints accept different categories of queries based on the structure of the return type. Either way, creating a well defined structure programmatically is straight forward and can easily be supported from a multitude of languages.

As for your UI, you are going to have to create a parser for your DSL anyhow which is going to build an intermediary representation of your query. If you merge your intermediary representation with your defined query structure it should be trivial to show the user what they would need to send to the API in order to reproduce their query in the API.

In this scenario your query language can continue to evolve separate from your query structure. You can even support other English based DSLs as long as they can all generate the same query structure. But what ever you do, keep the two domains separate. SQL is a terrible API.

Trying to solve Java's Gordian Knot

Fri, 01 Jun 2018 00:00:00 +0000

Java’s dependency management suffers from the dreaded diamond dependency issue. And while this issue is not unique to Java, it is, perhaps, more acute due to the precompiled nature of Java’s Jar files.

A Quick Refresher

If you are unfamiliar with this issue, here is a quick refresher:

Suppose there exists a library which contains a useful method that takes a string as its parameter:

class Common {
    void helpful(String a) {
        ...
    }
}

This library is published to an artifactory as

    com.hrakaroo : common : 1.0

Now, suppose there exists two other libraries which both use this helpful method

class Dog {
    void method1() {
        Common c = new Common();
        c.helpful("dog");
    }
}

and

class Cat {
    void method2() {
        Common c = new Common();
        c.helpful("cat");
    }
}

Each of these are also published to artifactory as:

    com.hrakaroo : dog : 1.0
    com.hrakaroo : cat : 1.0

And each of these have a transitive dependency on Common. Okay, now you decide to build your service which uses Dog and Cat so your dependency tree looks like

    com.hrakaroo : dog : 1.0
        com.hrakaroo : common : 1.0
    com.hrakaroo : cat : 1.0
        com.hrakaroo : common : 1.0

All is good. But now the folks who created Common come out with a new version which changes the signature of helpful and adds a boolean flag. So the new version looks like

class Common {
    void helpful(String a, boolean flag) {
        ...
    }
}

And knowing they are going to break some things publish this under a new version in artifactory

    com.hrakaroo : common : 2.0

The folks who made the Cat library realize this new flag will fix a bug they have had so they update to it as

class Cat {
    void method2() {
        Common c = new Common();
        c.helpful("cat", true);
    }
}

And publish it under

    com.hrakaroo : cat : 2.0

But the Dog library makers don’t need the new functionality so they don’t bother to update.

And finally, you decide to update your service to use the newest version of Cat which has the bug fix you need. This changes your dependency tree to:

    com.hrakaroo : dog : 1.0
        com.hrakaroo : common : 1.0
    com.hrakaroo : cat : 2.0
        com.hrakaroo : common : 2.0

In Java 8 you can not bring in the same dependency more than once with different versions. There are two common ways to deal with this, and they are both wrong.

First, you can not do anything. (This is probably the most popular solution.) In this situation gradle will pick one (usually the latest version) and use that as its version. In this case it will select com.hrakaroo : common : 2.0 which means that when your service calls Dog.method1 it will give you a runtime exception as the JVM will be unable to find the definition for helpful(String).

Or, if you are using gradle, you can use it’s force tag and force the version down to com.hrakaroo : common : 2.0 which means that when your service calls Cat.method2 it will give you a runtime exception as the JVM will be unable to find the definition for helpful(String, boolean)

The only “correct” solution here is to use gradle’s failOnVersionConfict() which will fail to compile your project unless both your dog and cat dependencies use the same version. This means you will be forced to fix the issue before your project can compile, but this may not be practical as a large project has lots of moving parts and compatible versions may not be available. Additionally, failOnVersionConflict() doesn’t understand semantic versioning so it will fail on PATCH level differences which often makes this a very painful and non-practical solution.

Most people just choose to go with the plug-and-pray approach where they just hope they never call a code path which encounters a definition which doesn’t exist.

Cutting the knot

As the two easy solutions are wrong and the one correct solution is impratical, the only real answer here is to avoid the problem altogether. When building a library, limit your dependencies.

Building a service requires a different approach from building a shared library. From the technologies you use to the way you version and test it are substantially different. Services are often just thin wiring together of different frameworks and libraries while libraries are more single tasked. And yet, they too often I don’t see people appreciate this difference. Instead they hack together libraries like they do services.

Stick to vanilla Java

I really like the Kotlin language, but I don’t think it has a place (yet) in shared libraries. Part of what makes Kotlin fun are all of the extension and infix libraries which are all packaged in the kotlin stdlib dependency. Any time you have more than two kotlin library dependencies you are just about assured to have a version conflict on the kotlin stdlib. Libraries should be written in Java to remove as many dependencies as possible.

Avoid huge common or utility libraries

Apache’s commons-lang3 is a fantastic library, but too often I’ve seen brought in so that the developer can use the StringUtils.join() method. Not only is this method trivial to write, but with Java 8 this can be done directly off the stream using .collect(Collectors.joining(...)).

The same can be said for Google’s Guava library, which is an enormous library that often makes non-compatible changes. In one library I reviewed the author had brought in a dependency on Guava so that they could use the Preconditions checks. While I think this type of defensive programming is good, the precondition checks can easily be re-created in your own library.

Copy and attribute

Providing the license allows it, it is also okay to simply copy sections of your dependent library directly into your own shared library and remove the dependency. Be sure to attribute where you got it from, but otherwise copying small to medium sized dependencies is okay.

Relocate and attribute

Finally, if all of the above are failing for you and, again if the licensing allows it, you can use a tool like shadow/shade to relocate a dependent library directly into you own library. These tools can will rebuild your resulting jar so that your dependencies are no longer transitive and all references to their old location have been changed to somewhere in your package. So, for example, you could relocate com.apache.commons to com.hrakaroo.apache.commons.

This will increase your jar size so it should really be used as the last resort, but it will guarantee that no one can later change the depenency version to something which is incompatable.

Whatever your approach, you should take the time when creating a shared library to minimize it’s transitive dependencies as much as possible. By doing so you will help minimize the risk to developers which use your library of creating their own diamon dependency nightmare.