Hrakaroo

Example Talk

Sat, 01 Jun 2030 13:00:00 +0000

Click on the Slides button above to view the built-in slides feature.

Slides can be added in a few ways:

Create slides using Hugo Blox Builder’s Slides feature and link using slides parameter in the front matter of the talk file
Upload an existing slide deck to static/ and link using url_slides parameter in the front matter of the talk file
Embed your slides (e.g. Google Slides) or presentation video on this page using shortcodes.

Further event details, including page elements such as image galleries, can be added to the body of this page.

Ray the Caster

Mon, 24 Jun 2024 00:00:00 +0000

Start with On-Call

Mon, 22 Apr 2024 00:00:00 +0000

This is my second post on running a standup, you can read my first post about walking the board here.

Start with On-Call: A Healthier Way to Begin the Day

On-call work can be isolating, exhausting, and easy to overlook if you’re not the one holding the pager. I’ve had nights where pages hit back-to-back, leaving me exhausted and irritable the next morning. When that happens, the idea of opening a standup with project updates feels disconnected from reality. What I really want to talk about is why I was paged—and how the team can help make sure it doesn’t happen again.

Rethinking Standup: Begin with On-Call

Standup is usually the first meeting of the day and exists to align the team. But before diving into project updates and task boards, start with one question:

Did the on-call engineer get paged overnight, and if so, why?

This isn’t a courtesy check-in—it’s a signal that the well-being of the team matters, and that operational issues are real work. If the overnight experience was rough, it might be appropriate to pause project tasks and focus on the underlying problem.

Classify Every Page

To make these conversations productive, each page should land in one of three categories:

1. Known Issue Already Being Worked On

If the root cause is already under investigation—underprovisioned services, a bug in progress—these pages can be deprioritized. Ideally, you’d mute the alerts, but that risks hiding new issues. Just be honest and precise: don’t group unrelated pages together.

2. Rare, One-Off Events

A “rare” page should truly be rare. Over time, the definition should stretch from once a month to once a quarter to once a year. If you’re getting paged weekly, it’s not a one-off.

3. New, Uninvestigated Issues

Anything that isn’t known or rare belongs here. These deserve investigation tickets. Failing to follow up leads to the same interruptions night after night—and often much bigger problems down the road.

Prevent Escalation: Keep Small Problems Small

Investigation work should be treated as a top priority for the on-call engineer and ideally the secondary as well. Small problems are easy to ignore when you’re busy, but they’re even easier to fix before they escalate. As the backpacking adage goes: keep small problems small.

Protect Sleep: Rotate When Needed

Being paged at 2 AM isn’t just inconvenient—it’s damaging if it happens repeatedly. If someone was paged between 9 PM and 9 AM, consider swapping them with the secondary for the next night. Guaranteeing a solid night’s rest can make all the difference—and reinforces shared responsibility. In some cases, you may want to also let the on-call person log off early to recover.

Why This Matters

In many teams, managers don’t participate in on-call rotations. When page volume is high, that disconnect can breed resentment, especially if nighttime interruptions feel invisible. Starting the day by focusing on on-call sends a different message: the team’s experience matters, and operational pain is a shared concern.

A small shift—like opening standup with one intentional question—can change how teams operate, collaborate, and care for each other. And for the person who held the pager last night, that shift can mean everything.

Glob Library for Go

Tue, 09 Jan 2024 00:00:00 +0000

42 Inch Round Table in White Oak

Mon, 11 Dec 2023 00:00:00 +0000

Background

For at least the last 20 years we’ve had a 48 inch round dining room table in our kitchen nook. Like most folks, we ended up using this as our primary dining table and while it was great, it was always a bit over sized for the space. Recently, while we were updating other parts of our house, we decided to finally replace the table with a 42 inch one that felt less bulky.

Jokes aside, six inches may not seem like a lot but if you multiply it out, a 48 inch table has a surface area of approximately 1809 squared inches, while a 42 inch table has a surface area of approximately 1385 squared inches. That’s a difference of 414 squared inches, so almost three full square feet smaller, which is considerable.

We knew we wanted a solid hardwood table and we liked the look of white oak. Most of the tables we found on-line were either super expensive ($3000+) and/or had straight or chunky legs which we didn’t love. What we wanted was something closer to our outdoor table which has legs that curve into a center ring and then out again.

After a fair bit of research I decided to try tackling the table build myself.

Design

Using SketchUp Maker (2017) I worked up some designs until I had something I was happy with. I knew I wanted to curve the legs in, but I wasn’t entirely sure how to do it. My initial idea (which you can sort of see in the sketch up) was to build the legs by laminating three thinner boards together. By overlapping them at the curve I thought I could build a strong joint. I still think this is a good design idea that would create a very strong curved leg, but in the end I went with a more traditional joint and just used dowels.

Wood

After making a cut list I went down to Crosscut Hardwoods and picked up ~30 feet of 6/4 for the table top and ~15 feet of 8/4 for the legs. The total cost for the wood was around $400.

Once I had the wood home I let it acclimate in the house for about a week before starting to mill it down. While I do have a planer I don’t have a jointer so I had to use a sled for the planer to get one side flat and a simple jig for my table saw to get a straight edge.

Table Top

After finding an arrangement I liked for the top I glued the boards together and let them dry overnight.

Mistake #1

I opted to not use any dowels or biscuits as they don’t really do much for adding strength. Not using anything for alignment proved to be a mistake as I didn’t get the boards perfectly flush and after it dried I had to take a hand plane and resurface both the top and bottom. Next time I’m probably going to put at least a couple of dowels in to assist with alignment

Once the top was dry I used a circle cutting jig and a spiral bit on my plunge router to cut the circle. I only cut it 3/4 of the way down before flipping the top over and using a flush trim bit on my trim router to finish the circle.

Mistake #2

I wasn’t entirely sure how far to plunge cut before flipping the table over. Several videos I watched on this seemed to suggest you only needed to make one or two passes with the router before flipping the table over. Thankfully I took it down further as my trim router was barely able to cut through what I had left. In the future it’s probably safer to take this down to 1/4 inch before flipping.

Legs

For the legs I first used cardboard to try out several different variations until I found something I liked. Although I had my sketchup design it helps to actually see it in person. Plus, once I had my cardboard template I was able to work backwards to figure out how big my leg blanks needed to be.

To do this I took my cardboard template and cut it right where I was picturing my joint to be. Then I was able to flip over the top piece and tape it back together which gave me an approximate template for how big I needed to cut each leg blank. Since I kept top and bottom leg angles the same, the resulting template was now perfectly straight and the taped cut line gave me the exact angle I needed.

Lining up the dowels for the joints was a process but everything came together pretty well in the end.

I did have some concerns that the dowel joints might not be strong enough as the pressure from the table to would basically be trying to pry them apart. To counter act this I made sure that when I jointed in my cross pieces that I used four dowels which really locked in this joint.

(And yes, I used more glue than what is shown. I had forgotten to take a picture, so I stopped in the middle of my glue up to snap this one.)

Mistake #3

Clamping the legs proved more difficult than I anticipated. I first tried the trick of gluing cauls to some blue tape, but those popped off pretty quickly when I started to tighten the clamps. Thankfully I had created some angle extensions that I was able to clamp to the individual pieces and then add an additional clamp. If I had to do it over again I would plan this part out better.

Once set I built a simple jig so that I could make parallel cuts in the top and bottom of the leg, as well as create the flat inner cut. Using this I was able to get all four legs nearly identical.

Next I made a router template from a piece of MDF I had lying around and used it to cut the inner edge of each leg. I did the same process of only cutting part of the way through from the top, then removing the template and flipping the piece over to cut the rest with the flush trim bit.

Finally I cut the center joining pieces and created a lap joint for them to fit together. Then I glued up each set of legs.

Sanding

Then came the sanding. All in all I probably spent 2 full days sanding everything. I started with 80 grit sandpaper and took it all the way up to 220 and used a wand light to make sure there were no imperfections.

Mistake #4

The finish I was planning on using was Rubio Monocoat which actually does not recommend going past 150 grit, so I then made one more pass to bring it back down to 150.

Related, although I like Rubio Monocoat, it is wildly expensive. On my next project I’m likely going to try to use Natura Onecoat as it reportedly works similar to Rubio and is about half the cost.

Mistake #5

After testing out several different samples I ended up going with Pure which, in retrospect, was probably a bit of a mistake. I still really like how the table looks, but its a little too honey colored for my tastes. I think something with a small percentage of white would have offset the honey color, but its too late now and overall I’m pretty happy with the way it turned out.

Final thoughts

This project took about two months to complete. I was mostly working on it during the weekends and some evenings. I did allow myself to buy a planer for this project and I needed to buy a couple of router bits as well. So all included (wood, planer, router bits, Rubio Monocoat, …) I probably spent about $1200 in total on this project. And while this is a lot, it’s still a far cry from the $3000 we were seeing on-line and I’m pretty proud to have built this myself.

Summiting Mt. Hood 2022

Sun, 08 May 2022 00:00:00 +0000

Background

In March 2022 I was accepted into the Mazamas BCEP program which taught basic rock and snow climbing skills. This was a fantastic experience and I highly recommend the program if you are in the Northwest and are at all interested in hiking and climbing. Once the program completed I signed up to join the Mt. Hood Climb scheduled for May 8, 2022 (mothers day), which was eventually pushed back to the following Wednesday (May 11th) due to weather conditions. There were a total of nine people (including the climb leaders) in the climbing group.

The day before

The day before the climb I drove up to Timberline Lodge and parked in the lower parking lot that allows climbers to camp/park overnight. In the weeks before the climb I had built a flat “bed” in the back of my Honda Pilot so I could put a sleeping pad down and stretch out.

It’s a really good idea to get to the parking lot the night before and start to acclimate to the altitude. Timberline lodge is at around 6,000 ft of elevation and the top of Mt. Hood is 11,249 so the more time you have to acclimate the better.

Six of the nine people from our group opted to show up the night before with three people driving up the morning of the climb. Initially we were planning to start our climb at 1am the next morning, but due to temperatures being lower than expected we were able to push our start time back to 2pm.

As the temperature heats up it can cause the ice and snow to soften creating dangerous climbing conditions as rocks and ice are more likely to break away and fall on you. Additionally, warmer temperatures weaken the structure of snow bridges over crevasses which increases risk. Ideally you want to climb when everything is frozen solid and stable.

So after rechecking my pack several times I finally went to sleep around 8pm.

The day of

Getting Ready

For a start time of 2am I should ideally have woken up at 1am to begin getting ready, but due to nerves I awoke up closer to 12am and was unable to fall back asleep. So at 12:30am I gave up trying and started getting ready for the climb. The day before I had prepared oatmeal in a thermos with hot water which made for nice warm breakfast. After breakfast I started to get my gear on and mentally prepare for the climb.

In addition to everything in my pack, I was also carrying one of the climbing ropes we were bringing in case we needed it going over the bergschrund. This added around 15 lbs to my pack and was a bit awkward to carry as you have to sling it over your pack. (I was also carring the fuel for the teams stove in my pack as we were required to bring it, but this only added a nominal amount of weight.)

Starting the climb

Just before 2am we started off. The night was amazingly clear but exceedingly cold. As we were hiking up we could see the headlights from other climbers further up the mountain who had left earlier.

The first part of the climb takes you to the top of the Palmer Lift and the route you take is along the “road” the snowcats create when coming back after grooming the ski slope. So the snow is pretty compacted and you can climb it easily in just your mountaineering boots and without crampons. (Crampons are great, but they are heavy, increase the likelihood of tripping, and generally slow you down.)

Silcox Hut

At approximately 3am we reached Silcox Hut and two things had already happened. First, two people decided to drop out of the climb as they were not feeling well. (It should be noted that both of these people were from the group that decided to drive up the morning of the climb so they did not have the extra six plus hours to acclimate to the altitude.)

Secondly, my water was starting to freeze. I had made the mistake of using a bladder for my water with an attached hose for drinking. This was the same setup I had used on multiple hikes and ski trips so I thought that as long as, after taking a drink, I blew air into my hose I could keep it mostly thawed. The problem was that the nozzle had collected water and was mostly frozen solid. After a lot of work I managed to get it somewhat unclogged and had, what would turn out to be, my last sip of water until after I was off the summit. In retrospect this was a huge and possibly dangerous mistake on my part and I should have carried backup water in my pack.

Somewhere between Silcox Hut and the top of the Palmer lift we started to get whiffs of the fumaroles on Mt. Hood which smells strong of sulfur.

Top of Palmer

At 4:40am we reached the top of the Palmer lift. The sun was starting to come up a bit but it was still really cold.

For the climb up to the Hogsback (and summit beyond) we would be in deep snow and no longer on a groomed trail so we needed our crampons, helmets and ice axes.

I was also able to switch off carrying the climbing rope to another climber so for the rest of the time my pack was 15 lbs lighter which made a considerable difference.

We had brought snowshoes for the climb, but after looking at the steps already dug for us by the previous climbers, we decided to stash our snowshoes near Palmer and go on without them.

You can see the shadow of the mountain on the landscape below.

Hogsback

Once we reached the Hogsback we were able to see the different routes to the summit. The Pearly Gates looked safe enough and the bergschrund was sufficiently covered that we felt comfortable climbing to the summit without ropes. This allowed us to stash our ropes and harnesses which removed a considerable amount of weight from our packs.

Fumaroles

At the Hogsback you can also see the fumaroles to either side of the path to the Pearly Gates. It’s important to watch your step near them because while they are warm, it’s really the lack of oxygen that will kill you before the heat will. That said, they are rather stunning in the snow.

Pearly Gates

We were forced to wait a bit at Hogsback as there were a lot of people trying to get through the Pearly Gates and the section right before the Gates themselves is the bergschrund, which you want to pass over as quickly as possible.

Going through the Pearly Gates was one of the more surreal experiences of the entire climb. First, the angle is rather steep so a lot of the climb is on your hands and feet. The climbers before us had already cut steps into the snow so it isn’t particularly hard, but it is slow going and some areas are only wide enough for two people (one going up and one going down) so you are forced to go as slow as the person ahead of you is going. Secondly, the natural ice sculptures that have been created from the snow and wind are stunningly beautiful.

Of course, the entire time I was climbing up through the Pearly Gates I had it in the back of my mind that I would shortly be having to climb down this as well.

The Summit

Once you pass through the Pearly Gates the top of the mountain actually levels off a bit and the summit is mostly flat with only about a 10-15% incline.

The view from the top is nothing short of breathtaking. The sun was out and we could see for miles. It was, however, very cold and we still had to down climb through the Pearly Gates.

The climb down

While climbing up through the Pearly Gates you can plan out where you are going to step, but the way back down requires a lot more trust. This is where having spikes on the front of your crampons and mountaineering boots really pays off. For most steps I would kick my foot into the snow a couple of times until I had created a slight indentation and then use the spikes to hold the majority of my weight. My mountaineering boots had a full shank so as long as my toes were secure I could stand up securely with the majority of my foot suspended by just my toes.

Initially this was hard for me to relax into, but after a while I really started to trust my equipment and the climb down became a lot easier.

Plunge stepping down to Palmer

Once we were back at the Hogsback my water tube finally thawed enough that I was able to get a drink. After that we collected the ropes and harnesses we had stashed and headed down to Palmer.

We started out on the trail we had taken getting to the Hogsback, but quickly drifted off and were mostly plunge stepping down to Palmer. Some of the folks in our group tried glissading down, but the snow was pretty deep which made for slow going.

Glissading

At Palmer we collected the snow shoes we had stashed and got back onto the groomed snowcat trail. We also quickly discovered that while glissading in the deep snow was difficult, glissading on the groomed snowcat trail was really easy. I had brought with me some medium gauge plastic that worked okay as a sled as I was able to glissade for probably 75% of the way from Palmer back to Timberline.

Back at the car

Finally back at the car we packed up our gear and drove into Welches to have lunch. After lunch I went into the parking lot, got into my sleeping bag and took a four hour nap before finally driving home.

Gear

What follows is a full listing of all the gear (to the best of my memory) that I had with me on the hike. With the exception of my water situation and some slight changes for my feet and hands, the rest of my gear really held up well and I don’t think I would change much, if anything, for the next time.

Feet

Hiking socks
Scapra Charmoz HD mountaineering boots
Hikenture leg gaiters
Grivel crampons
Snowshoes

I also have some Sealskinz socks which I’ll likely be wearing next time and I’m experimenting around with a couple different brands of heated socks. The Sealskinz, in addition to being waterproof which is a nice just-in-case feature, really trap in the heat, so these with the heated socks should be pretty nice.

The snowshoes I borrowed from a friend, but since this trip I have purchased my own pair from REI using their 20% off coupon. Not long after I purchased mine I noticed that Costco was also selling snowshoes as a seasonal item at the start of winter so that might have been another option, but I don’t know how heavy they are.

Pants

Zero Degree base layer leggings
Kuhl Snow/Hiking Pants

The Kuhl are fantastic. They are rugged enough for hiking in, insulated and priced reasonably.

Torso

Zero Degree long sleeve base layer
Quick dry t-shirt
Mountain Hardware Gore-Tex Infinium thin hooded coat
32 Degree Heat down jacket
Columbia hooded rain coat

The down jacket was a cheap one I bought off of Amazon for like $40 for camping and the Mountain Hardware and Columbia jackets I purchased at the Columbia employee store for a discount.

Head

Balaclava
Thin beanie hat
Mountaineering Helmet
Headlamp
Glacier glasses

Before leaving I loosened my helmet so that I was sure I could wear it easily over the balaclava and beanie. The balaclava was actually something I received for free from a local go kart race track. It was cold enough that the condensation that collected on my balaclava from breathing through it froze as well.

The beanie hat is an Omni-Heat one from Columbia so it has gold dots on the inside which really do a great job of reflecting the heat back.

Hands

Glove liners
Three finger gloves
Garmin Fenix 7

Mittens would have been ideal, but in order to try to keep some dexterity I opted for three finger gloves which have a dedicated finger for your thumb and index finger, and then a mitten like pouch for your remaining three fingers. This caused my index finger to get really cold so next time I’m either going to invest in heated gloves, or get some mittens. If I do get mittens I’ll probably use some of the chemical heating packets which I can stuff down in the outer glove.

The Garmin is useful as it has an altimeter so you can figure out how close you are to different milestones. It also has the ability to record your entire climb and has a full map which you can use in a pinch if you get lost.

Backpack

Ten essential items
Rock climbing harness
2 Nalgene 1L water (next time)
Thick hat (for when I wasn’t wearing my helmet)
Blue bag (just in case)
Extra glove liners
Extra glove shell (five fingers)
Small waffle sitting pad
Cell phone

My backpack is a Mountain Hardware 25L Scrambler I purchased from the Columbia employee store. In addition to being a perfect size it also has loops on the outside for two ice axes and I was able to easily attach my hiking poles and snow shoes.

I’ve listed two 1L Nalgene bottles which, of course, I didn’t have on my trip but really really wished I did. It’s important that you store the bottle upside down in your pack so when it starts to freeze it doesn’t seal the bottle shut. Instead I had my water bladder (which I didn’t list) but which was such a bad idea I’m leaving it off my list.

For the beginning part of the climb I had my crampons in my backpack which worked out well, but wrapping them so they don’t rip your backpack is a bit of a trick. I ended up using a mailing envelope but I might look to see if I can get/make a leather pouch for next time or see if I can somehow attach them to the outside of my backpack.

Misc

Ice axe
Hiking poles
Snacks and energy cubes
Glissading sled

Although I was only carrying one ice axe, I really think two would have been a better idea and wouldn’t have added much weight.

For the glissading sled I had made mine out of some shelf liner plastic from Home Depot but it really didn’t hold up well. While riding down one corner folded in on itself and I kept having to stop to fix things and by the time I was finally down it was mostly ruined. Since this trip I’ve gone to Tap Plastics and purchased a sheet of thicker, but still flexible, plastic and have constructed a better sled.

Closing thoughts

10/10 would do it again

I had a fantastic time overall, the weather was near perfect and the climbing was excellent. The ice sculptures near the summit are unworldly and something that you really need to see in person to believe. It was also amazing standing at the highest point in all of Oregon and being able to see for miles around you.

The cold was no joke

I think overall I underestimated how cold it was going to get. In addition to my water tube freezing and leaving me without water for the majority of the climb, my fingers and toes also suffered. I suspect that I have Raynaud’s disease which causes my body to abandon my fingers and toes pretty quickly when I get cold. Although I was wearing gloves and glove liners, my fingers still became very cold and numb. Plus, putting on my crampons, opening my backpack and even just getting my snack bag opened required me to take off at least my outer glove layer which didn’t help. I also started to lose feeling in my toes around Palmer and continued to lose more feeling in my feet for the rest of the climb.

Normally once I warm up the feeling comes back into my fingers and toes pretty quickly, but not this time. For at least the first week after the climb (and some of the next) I still had loss of feeling in my toes, thumbs and forefingers. This was unnerving and at the time I wasn’t quite sure if I would ever regain feeling. Eventually, slowly, the feeling did come back and now I’m back to normal, but it’s not something I really want to repeat.

For next time I have purchased some battery powered socks and mittens that I’m hoping will keep me warmer. Plus, I’m going to be taking some of those chemical hand warmers with me as well.

In Defense of Obscure Names

Sun, 07 Nov 2021 00:00:00 +0000

In defense of obscure names

Naming things is hard

As the joke goes

There are only two hard things in Computer Science: cache invalidation and naming things. – Phil Karlton

and while this is funny (perhaps only to those in Computer Science) underlying the quote is a real issue, naming things is really hard.

The challenge is not just in giving something an appropriate name for right now, but in trying to predict how it may evolve and be used in the future.

Maybe you start with a database of bicycles, but as powered bicycles are introduced you start adding those but from there it’s a slippery slope to electric motorcycles and then gas powered motorcycles. Pretty soon your database of bicycles includes all motorcycles and perhaps those three wheeled things I’ve seen driving on the freeway. Unless you had amazing forethought and were able to predict exactly how the industry was going to evolve, chances are whichever name you select is going to be incorrect at some point in the future.

Renaming isn’t always easy

So, if the scope does evolve and the original name is no longer correct we should just rename it, right? Sometimes this is the answer. Tools for software development have come a long way and doing a code refactoring isn’t nearly as hard as it once was. So if we have a variable name, function name, or even a class/object name that is no longer accurate we usually have the ability to safely rename it across all dependencies to a new and more appropriate name.

But the coding level is often where this capability ends. The industry has not yet developed the capability to reliability refactor service names across a distributed environment and trying to do it manually is time consuming and often fraught with errors that can negatively impact uptime.

In defense of the obscure

So if changing the name of something is difficult and if naming things correctly now is hard, (and predicting how they may evolve is nearly impossible,) then maybe the best thing we can do is to purposely give them an obscure name.

When I was in college we were able to select our own email address. Like most people I knew, I opted for the straightforward and used my last name for my email address: gerth@lclark.edu. However, one of my classmates opted for something entirely different, she selected the email address of squidlips@lclark.edu. At first I thought she was just trying to rebel, but her reasoning was actually pretty interesting. Her name was something rather common, (think Jane Doe) and her argument was that while doe@lclark.edu runs the risk of being mistaken for someone else, squidlips@lclark.edu is almost guaranteed not to.

This proved to be more than just an academic observation for as it happened, I was not the only Gerth at Lewis & Clark College. There was one other one, my father, a tenured professor, and I would occasionally receive email intended for him. People had seen gerth@lclark.edu and had incorrectly assumed I was my father, and were sending me information clearly not intended for the students. (Nothing salacious I’m sad to say).

In this case the name (my email address) was even 100% correct and it was still getting used incorrectly. And while while no one may have guessed who Squid Lips was, she never got any email not intended for her.

Obscure is better than wrong

We make snap decisions all the time and if I was looking to get a list of motorcycles and came across a service named bicycles I would immediately cross it off my list. But, if instead the service was named x34a5a or even horsetail I would at least have to consider it. I may first go looking for something more specific, but at least I wouldn’t cross it off my list.

(I also prefer the name horsetail over an obscure id designation as it’s much easier to remember horsetail and distinguish it from pigfoot than trying to remember a20b23 and distinguish it from c23s40.)

So if a thing is going to exist for a while, is likely to evolve in scope, and changing it’s name is non trivial, then giving it an obscure name may actually be better and ultimately easier than trying to come up with the perfect name. So in some cases, naming things may not be all that hard after all.

Avoiding Read/Write Locks

Sat, 06 Nov 2021 00:00:00 +0000

Intuitively, read/write locks seem to be a ideal solution. They prevent the read/write and write/write race conditions while making sure multiple read routines don’t block one another. So as long as the write operations are optimized to be as fast as possible, the overall throughput should be high.

In most cases this is true, but there is a subtlety to read/write locks that can end up causing them to be considerably more expensive than they initially appear.

Consider the following:

Thread 1 requests read access to a shared read/write lock.

At this point in time no one else has requested the lock for either read or write access so a read lock is granted to Thread 1.

Thread 2 requests read access to the same read/write lock.

Since the lock is currently only read locked by Thread 1, Thread 2 is also given read access.

Thread 3 requests write access to the same read/write lock.

Since the lock has been locked for read access by both Thread 1 and Thread 2, Thread 3 is put on a wait list for write access until Thread 1 and Thread 2 release their read locks.

Thread 4 requests read access to the same read/write lock.

What happens here?

A first guess might be to give Thread 4 the read lock, along with Threads 1 and 2, since Thread 3 has not yet obtained the write lock, but lets consider the fall out if we let this happen. Imagine a system where every read operation took a minute to execute and requests are comming in every 30 seconds. If every read request is granted a read lock so long as no write thread had been given a write lock, new read requests would always come in before previous ones finished ensuring that the lock was never free (ie not read locked.) In this situation the write operation would be blocked from every achieving write access. aka, lock starvation.

In order to prevent lock starvation it is necessary that as soon as a write lock is requested every subsequent read lock request is blocked from obtaining their lock until the thread requesting the write lock has been given the write lock, executes, and then releases their lock.

This means that as soon as a write lock is requested any read threads currently running will block all new read requests. So a read/write lock is, worse case, as “expensive” as the time it takes to execute the write operation plus the worse case time it takes to execute a read operation.

Therefore, even if you optimize your system so the write operations are as fast as possible, your overall throughput may still be dependent on your slowest running read operation.

What read/write locks are good for.

Read/Write are useful when you need a memory barrier and every thread on your service should only see the state of some memory either before or after a write operation, but never but never both. Meaning that it is not allowable for some threads to see the value of the memory before the write operations and other threads running at the same time to see the memory value after the write operation.

But, as we saw above, this comes at a cost and in many cases it may not be needed.

For example, suppose we have a data structure that holds the account balance for an individual. Suppose also that this account receives a constant stream of read requests that, for what ever reason, take several minutes to calculate. If we were to use a traditional read/write lock here then every time there is a deposit or withdrawal (a write operation) all read operations will be blocked until all running read operations finish and the write executes.

Although the write operation may be quick, the long execution time on the read operation is going to cause the system to appear blocked while it waits for the write lock. But if we can tolerate concurrent threads seeing two different values then we can remove the block all together and achieve higher overall throughput.

To achieve this we first need to make the data structure immutable. Updates would instead be handled by the write thread copying the immutable data structure locally, modifying it (or modify on copy) and then using an atomic operation to swap in the new data structure.

Read operations running against the old data structure would continue to run and would report the old value, while any new read operation would run over the new data structure and report the new value. Eventually all old read operations would finish and the only view would be of the new data structure, but for a time it would be possible for different threads to report different values.

Destroying the old data structure can be handled either automatically by letting the garbage collector pick it up, or we can add read counters to objects which can tell us when an object is no longer being accessed and we can delete it manually.

We could also re-introduce a simple lock for just the writer operations to ensure that multiple write operations always happen in sequence so they can never overlap and they would never cause inconsistent results.

So while read/write locks often appear, at first blush, like a panacea, they can actually carry a substantial hidden cost. And in some cases the functionality provided by a read/write lock may not even be what you really need.

Creating a workshop in my garage

Wed, 01 Apr 2020 00:00:00 +0000

Background

I’ve always wanted a small woodshop.

With both of my parents in theatre I spent a lot of my childhood in the shop they used for building sets. When we bought our first house I purchased a couple of woodworking tools, but my work area was in the garage which was shared with our cars. This made it hard to work on bigger projects and sawdust was always getting all over everything.

Our second house had a three car garage which meant I finally had a dedicated space, but sawdust control was still an issue. Plus, it rains a lot where I live so dampness control was also an issue, especially as we went in and out of the garage.

When the pandemic hit I suddenly found myself in desperate need of a project and decided to see if I could construct a wall between the main garage and the third bay to create a small, dedicated woodshop.

This is the before picture. Since the third bay was not isolated it often became just a dumping ground for things.

My first step was to take all the sheetrock off the post. Once I did that I was pretty much committed to the project. I knew I wanted to use the post to divide the wall in half, structurally at least, and I figured it would be better to work on the smaller half first, even though it was going to have a door in it which, at the time, I had no idea how to actually do.

But first, I had to put in the header and footer boards to attach the vertical 2x4s to.

In order to nail into the concrete I bought a hammer drill to drill anchor holes. This worked, but for the other half of the wall I borrowed a powder actuated nail gun from a friend of mine, which basically uses a 22 cartridge to shoot a nail into the concrete. Very loud, but much faster.

Once the header and footer were attached I started putting in the vertical 2x4s. I read up a lot on how to build a door frame with a king stud, jack stud and header plate.

Satisfied with the framing I went on to do the other side, using a laser level to make sure the wall was straight.

For this side I wasn’t as lucky with the ceiling beams and they were parallel with the wall instead of perpendicular, as they had been for the first section. This meant that I had to climb up in the attic space and attach cross beams first before I could then attach the header boards.

Insulation

Initially I was not going to add insulation as I hate working with it. If you don’t cover up enough you can get thousands of tiny glass slivers that seem to take weeks to work their way out. But in the end I decided to add it, mostly because I was going to be running some power tools and wanted the extra sound insulation.

I also added in some 20amp outlets and ran all the wiring, but ended up hiring an electrician to actually hook it into my electrical panel. I’ve added breakers to my panel before, but the whole thing scares the hell out of me and is not something I wanted to do again.

I also installed a 240v 50amp circuit on the garage side for a future electric car plug in, and a 240v 30amp circuit on the shop side for my table saw.

Sheetrock

I debated about installing the sheetrock myself or hiring out. Although I’ve done some small patch repairs, I haven’t actually done a full install before so this could be an opportunity to learn. That said, sheetrock is heavy and although I probably could have rigged up something to help me install it, it is much faster as a two person job. So in the end I just hired out for this part as well. We had temporary access to a truck so I was able to pick up the sheets myself and then two pros came in and got the whole thing installed and mudded in less than a week.

Painting

I did decide to paint it myself so I started with primer and was surprised at how much a new wall soaks in the primer. I went through way more than I was expecting.

I also opted to not add any texture to the wall as most of the walls in the garage were already pretty flat and I would have had to hire out for the texture as well as I’ve heard it’s a trick to get right the first time.

Door

I had never hung a door before, but it turned out easier than I was expecting. In the three years since I installed this I still haven’t had any issues with expansion or jamming.

Moulding

Next I attached most of the moulding. After going around the door I got busy with other projects and still, as of this writing, have not entirely finished the moulding. Seeing as how we are now modernizing the moulding in our house I may just wait and replace this with the new moulding as well.

Lights

Adding lights was the last step. There was already a plug in the ceiling where a garage door opener could plug in, so I bought some led strip lights and plugged them in with a remote on/off switch. (Initially I was thinking I might actually wire up a switch, but in the end this was easier and cheaper.)

The led lights are fantastic and really do a great job of lighting up the woodshop. I have since pinned the wires to the ceiling so they are no longer dangling down like you see here.

Final thoughts

This was a fantastic project which I both super enjoyed working on, and also left me with a very usable space for a woodshop. I still have a lot of work to do on organizing and building out the actual shop, but having a dedicated space has been a dream come true.

Summiting Mount St. Helens in 3+1 Days

Wed, 01 Apr 2020 00:00:00 +0000

Background

In July 2019 my friend Tobin and I decided to hike the full Loowit loop trail around Mount St. Helens. Both of us are experienced hikers and in reasonable shape and figured we could probably complete about 10-14 miles a day. The total distance around the mountain is about 35 miles, so we figured it should take us approximately 3 days to hike. We also managed to secure two passes to climb to the top of St. Helens for Aug 2nd which extended our trip by one day.

In order to climb to the top of Mount St. Helens you need a permit, which can be difficult to obtain depending on when you want to climb. The summer permits often sell out very quickly so initially we assumed we would not be able to climb to the top. Even so, we signed up on a reseller web site and about a week before we were set to climb we were contacted by someone who had two extra passes as some people in their party backed out at the last moment. Further, the person lived in Seattle and was only a short distance from Tobin, so two days before we were set to leave Tobin drove to their house and bought the two passes from the seller. The guy was really nice and didn’t mark up the cost and just charged us the $20 or so per pass that he had paid.

The trip

Where to start

The first challenge was figuring out where to enter the Loowit trail and which direction to hike (clockwise or counter clockwise). Since there is only one trail up to the summit and our passes were for Aug 2nd we had a hard requirement of being near Monitor Ridge on the night of Aug 1st. Initially we thought we would enter at Sheep Canyon trailhead and hike clockwise on the trail. This would put us around Windy Pass on the first night and close to June Lake on the second night. (In retrospect this was probably not a great plan as Windy Pass isn’t named ironically. It is windy as hell and camping there would have been a trick.) However, the night before we were planning on leaving we realized the trailhead we were going to use had been washed out and we had to scramble to come up with a new plan.

The new plan was to park at the Windy Ridge trailhead and hike counter clockwise around the mountain. Tobin is from Seattle and I’m from Portland so we met at Spiffy’s at I-5 and Highway 12, stashed his car in a gravel parking near the 76 gas station and drove my car to the trailhead.

Spiffy’s is a bit of a dive restaurant off I-5 but there were not a lot of other places around and we knew we wanted to get some take away sandwiches. I showed up first, walked in and started looking at the menu. While I was deciding, an older guy with a “get off my lawn” look to him walked in with a small lap dog in his arms. The hostess informed him that the dog would need to wait outside which caused him to get really upset and he demanded/yelled that it was a “therapy dog.” The hostess wasn’t having any of it and told him that he needed a permit or some sort of license if it was a therapy dog. At that point the guy yelled at the hostess “go fuck yourself you fucking whore” and stormed out. That was also the exact moment at which Tobin walked into the restaurant and only heard the insult as the guy stored out. We looked at each other and both cracked up laughing as it was such a bizarre way to start the trip. To her credit, the hostess handled it really well and the sandwiches were pretty good.

The trail head is at the end of NF-99, super easy to find and there is plenty of parking and a flush toilet bathroom. There are also some interesting informational signs that talk about the eruption and slow recovery of the area. We were a bit eager to get started so we didn’t linger too long. We ate the sandwiches we had picked up at Spiffy, sunscreened up and headed off.

Our packs weighed around 24lbs and since Mount St. Helens is well known for not having much water each of us was also carrying close to three liters.

Day 1 - 12.2 miles

Starting

For the first 1.7 miles the hike is along a winding gravel road. It’s blocked off from cars and well maintained, albeit not terribly interesting. At the end of the gravel road it’s still another 1.2 miles until you actually connect to the Loowit trail.

As we were hiking counter clockwise around the mountain we headed to the right and started on the Loowit trail. At this point the scenery looks sort of like rolling hills with the mountain in the distance. You can clearly see where the “blow out” was from the eruption but the walking is pretty easy.

Loowit Falls

Around a mile into the trail there is a turn out to go see Loowit Falls. It’s about 0.6 miles off the tail but well worth checking out. Although you can see the falls, you can’t actually get near it and even getting down to the water is a challenge.

We did meet up with a forest ranger just before we got to the turn off for the falls and chatted with him a bit. We asked the ranger if the waterfall was worth the extra hike (as we had a long ways that day). His answer was something like “yeah, it’s pretty good” which we didn’t find very convincing. I think we even said, in a joking way, “you are not exactly selling it here” to which he then responded very excitedly “it’s a 200 ft waterfall made of glacier melt coming out an active Volcano!” He was awesome and convinced us to go and waterfall was amazing.

Once we were back on the main trail it was another 0.8 miles until Willow Springs. Near Willow Springs there was actually water running across the path and if we had needed water this would have been a great place to fill up. It was also the last time we would find water until we stopped for camp that night.

Restricted Zone

After Willow Springs the scenery gets significantly more barren and rocky and the mountain looks blown out.

This is also the recovery area so no camping is allowed and really, it would have been hard to find a good spot to camp if we had wanted. This goes on for 5.4 miles, but the trail is pretty easy to see and mostly flat. In a couple of places we did drift slightly off the trail but thankfully others have erected rock piles that make getting back on the trail pretty easy.

Castle Ridge

As you get close to Castle Ridge the scenery starts to change significantly. The trail gets close to the ridge itself in sections and it’s a steep drop down. Tobin also recognized Huckleberries so we stopped at several big patches before continuing on.

From Castle Ridge it’s 1.4 miles down to the Toutle River. The start of this section is a switch back on a sandy incline that could be challenging to walk at time. At times it could take a fair bit of concentration but it wasn’t impossible. (I would not advise either this section or the ridge for smaller children.) After the sandy incline you enter the forest and continue your switchbacks down. It was here that Tobin discovered wild Salmonberries within reach of the trail and we were again stopping periodically.

Toutle River

The last section down to the river itself is basically a steep cliff for which the forest rangers have attached ropes for hikers to use. It requires going down backwards, with your feet walking down the cliff as you go hand over hand on the rope. Once you get over the initial shock it is actually rather easy to do.

When we were there the Toutle River was around 5 feet across and we were able to cross easily by jumping across rocks. We set up camp next to the river and the sandy floor made for a really plush ground. It was honestly one of the best night sleeps I’ve ever had camping.

There were lots of places to set up our tents and when we arrived we were the only people there, although another group did arrive just before dusk.

Day 2 - 10.2 miles

In the morning we packed up, filled our water bottles and used the ropes on the other side to climb out. The path to climb out was easier than the one to climb down and we probably could have managed it without the rope.

Our goal was to hike down near the Summit trail, find a spot to camp and hopefully water nearby. The other campers at the river were hiking the mountain in a clockwise direction and said they didn’t see any water other than June Lake. So our fall back plan was to find a camping spot around the Summit trail, dump our packs and hike down to June Lake for water.

Crescent Ridge

Right out of the river the hike is a brutal uphill climb that takes you from 3200 ft to 4700 ft in 1.7 miles. It’s almost entirely in the trees so you don’t get much of a view either. We did stop to take a couple of breaks on the hike up as we were still finding wild Salmonberries and Huckleberries. The Huckleberries especially were perfectly ripe and really good.

About 1.5 miles after reaching the top of that climb the trail crosses another ravine which again requires the use of attached ropes to climb in and out of. There was no water to cross in this one.

After that the next 4 miles is a really nice and relaxing hike through some beautiful scenery..

However, once we reached the trail for Summit Route we realized that “finding a camping spot near by” was virtually impossible. The trail is really cut into a steep mountain side and the flora is pretty thick on either side of the trail. We either had to hike back a mile or so to where there were some flat areas to camp, or hike forward and hope it flattened out. Hiking back would have taken us further away from water so we decided to go forward.

Chocolate Falls

Almost exactly two miles from Summit Route is Chocolate Falls and we decided to make camp there as there are plenty of flat areas nearby just before you cross over the falls. However, if you venture a little ways towards the cliff (and away from the falls) you will find a really nice circular camping area, complete with a fire pit (which we did not use). However, being so far off the path was really nice as we were not bothered at all by people passing on the trail and we felt secure about leaving our stuff there while we hiked to the summit the next day.

Chocolate Falls is named as such because the water contains a lot of sediment and takes on a chocolate milk color/consistency. But there was water and by straining it through a bandana and our Sawyer squeeze bag we were able to filter it enough to drink.

Using the bandana was not part of our inital plan but the water had a lot of silt in it and Tobin had a brand new bandana with him. Regretfully the bandana was red and had probably only been washed once so when we were straining the water it took on a very pink color. Even after using the Sawyer squeeze bag it still had a bit of a pinkish hue to it, but it didn’t seem to contribute any taste.

Day 3 - 8.2 miles - The Summit

Up until now our weather had been perfect but this morning it was raining. We had initially thought we would start on our summit hike at 8am, but the rain was coming down hard enough that we just stayed in our tents for a couple of hours to see if it would clear. By 11am the rain was starting to let up enough so we decided to get going before it was too late. Both of us took our backpacks but only brought with us some food, water, warm clothes and our permit.

We now had to backtrack the two miles back to the Summit Route. This was mostly uphill and because of the rain we were getting soaked every time we brushed up against the plants on the sides of the trail.

Summit Ridge

This was not an easy hike. The very start of this path is like a normal, albeit steep, trail but it quickly turns into climbing over volcanic boulders. There isn’t really a trail to follow anymore, but the route is marked by posts which are placed just close enough together that you can just see the next one.

We passed a couple of people coming down who had gone up very early that morning and mentioned that the entire top of the mountain was pretty well fogged in and there was not much to see. Still, we were determined and continued up. The entire hike is about 2.1 miles, but it is very slow going.

About 2/3rds of the way up we met a forest ranger who told us that “there is a crack at the rim and it appears that a section is about to fall in,” so we should stay back several meters from the edge. He also mentioned that because of the weather he was guessing that of the 100 passes that were issued for that day, probably about 20 of them were going to actually be used.

Pumice Path

As hard as the hike was so far, the pumice path that is at the end is even harder. The incline is very steep and with every step up you take you slip back down almost a half a step. It’s like this for almost half a mile and the pumice easily doubles the distance. There is also not a clear visual for how far you still have to go so it can seem a bit unending at times.

The top

The good thing about waiting those three hours before leaving camp is that by the time we did reach the top the clouds were clearing and we could see all around the mountain. Unfortunately, the clouds in the crater did not clear so we never really got a good full look inside, although from time to time we got a couple of good glimpses.

Otherwise the view from the top is really amazing and I’m glad we made the trip. We did walk part way around the edge and we did see the crack the ranger was talking about. Because of the scale of everything, looking over the edge didn’t seem like a huge drop, but apparently it’s something like a 2600 ft drop.

After spending about 30 mins at the top we headed down which gives you a totally new perspective on just how far you have climbed and how much on a ridge you really are.

By the time we reached our camp at Chocolate Falls it was getting dark. We decided to grab some more water but then ended up just leaving it in our bottles so we could filter it the next day.

Day 4 - 12.3 miles

The next morning we were awakened by the sounds of runners going down the trail. It turns out that there were several races going on that day, with one of them being a full run around the mountain.

It also turns out that Chocolate Falls does not normally run in the mornings as it needs the afternoon sun to melt the snow. So it was really fortunate we had collected water the night before as it was totally dry that morning.

It was a 1.2 mile hike to the cut off for June Lake and shortly past that past a creek that was flowing and allowed us to get more water. This water was clear and didn’t need to be double filtered. After that it was 4.8 miles to Pumice Butte. The hike was really beautiful and you can see several other mountains in the distance. We stopped often for Huckleberries and the very occasional Salmonberry and to move off the path at times for runners.

Pumice Butte

Several people claim you can get water near Pumice Butte but all the water we saw was very stagnant and didn’t look good at all. The 1.8 miles of The Plains of Abraham are flat, long and because of the time of day we were there, very hot.

At the end of the Flats we could have opted to take a short cut back to the car (and my feet were killing me by this point), or we could continue on the Loowit trail and complete the full loop. We opted to complete the full loop and just before we headed up Windy pass we crossed a running creek and were able to fill up on water one last time.

Windy pass

The climb over Windy pass was harder than I was expecting as you have a steep climb up, and short jog forward and then a steep switchback back down. Also, deceptively, they have put posts that run up to the left, giving the impression you have to climb much higher than you actually do. I’m not sure if this was someones idea of a joke or if there is another trail up that way.

In any case we made it down and eventually made it back to the spot where we first started on the Loowit trail. From there it was 2.5 miles back to the car and by now we were feeling every step of it. Also the winding gravel road has several false turns where you can easily fall into thinking that the car is “just ahead”.

The car

Getting back to the car we changed into clean(er) clothes and headed back to Tobins car, stopping off at Plaza Jalisco’s Mexican restaurant for dinner.

Retrospective

Equipment

Boots

Tobin was wearing hiking shoes while I was wearing hiking boots. Although my boots did better when it was wet, and when we were climbing over the boulders, I think in the long run wearing hiking shoes would have been a better call. They are lighter and since we were on a trail for a majority of the time I think the boots were, perhaps, overkill. And by the last day, especially that last 2.5 miles, my feet were killing me. Tobin commented that his feet hurt as well, but I think the boots were adding to my discomfort.

Socks

I should have brought more socks. In the past when I’ve hiked for a couple of days (2-4) I usually bring just two pairs of socks, but for those hikes I’m often spending the day at the destination or wearing my camp shoes. For this trip we were hiking every day and it would have been really nice to put on a clean pair of socks every morning. Tobin washed his out in the river which seems to have helped, but really I just think at least one pair of socks a day is the minimum and perhaps two.

Camera

Tobin and I both brought cameras with us, but while I just brought my cell phone (because, hey, it has a camera in it) Tobin brought an actual digital camera. However, since I had my phone off for the majority of the trip, taking a picture meant that I had to stop, take out my phone, wait for it to turn on, take a picture and then shut it down and put it away again. Tobin, on the other hand, had a handy pouch on his belt and was able to quickly take out his camera to take pictures. I had thought that non-SLR digital cameras were a thing of the past, but after this trip I now included a digital camera in my packing list.

Water

Similar to the camera, Tobin’s backpack also enabled him to carry at least one liter of water where he could easily reach and drink from it. Mine was attached to the back of my backpack making it a pain and often I had to ask Tobin to get my water for me. As a consequence I probably drank less than I should have. I need to either figure out a better way to hook up a bottle, or get one of those camel packs.

More space

I had brought with me some fleece pants and a fleece top and while these don’t weigh much, they do take up a fair bit of space. Weight wasn’t really an issue on this trip, but space was. As it was, my bag was packed really tight which required extra time packing up in the morning and trying to get to anything in my pack. Since this trip I have replaced some of the gear in my pack with things that just take up less space so I can treat my bag as more of a simple stuff sack.

Bivy or Tent

I really like sleeping in a bivy but at the last moment I decided to swap my bivy for a tent which I feel was a good call. I like using a bivy because they are fast to set up and overall just more fun. However, on this trip it would have been miserable sleeping in the bivy when it was raining and the 3 hours we hung out in the tent before climbing to the summit. I’m also not sure I would have been as keen on leaving my bivy out when we were hiking.

Sleeping pad

I had a blow up air pad while Tobin had the waffle mat. I think the air pad is more comfortable, but you do run the risk that it could rupture and then you can be in some real trouble. I did have a small patch kit with me, but I’ve never used it and I’m not sure how well it would work if I had to patch my air pad at night. Additionally, the air pad is noisy as hell when you move around on it. It makes a sound that is similar to rubbing a balloon. For next season I think I’m going to try using a waffle pad and see how that works. I also like that the set up for the waffle pad is super quick.

Hiking gloves

I didn’t have a good pair of hiking gloves with me and I really wish I did. I did have some fingerless gloves (aka hobo gloves) which convert to mittens, and while they were warm enough they were not the best for climbing with. Ultimately I need to get a good pair of gloves that are warm enough for the evenings, but also rugged enough to climb over boulders with. Or, if no such glove exists, then perhaps two pairs, one for warmth and one for climbing.

Folding chair

I have also added to my bag a folding chair. Yeah, I could just sit on stones and trees and the such, but after a long day, and in the morning, it is nice to be able to sit on a real chair. Sitting on the stones and logs meant that I was constantly trying to find a comfortable one or leaning to the side as I am sitting. The new folding chairs are down around a pound now so I’m hoping to try this out on my next hike.

Sawyer squeeze bottles

These things are awesome. When we got home and cleaned it out it was amazing how brown the water was and how much they filtered things. They are awesome, I just wanted to point that out.

Hike

Bugs

Normally I’m bothered by mosquitoes but I don’t think we encountered any on this trip. Perhaps this is due to the general lack of water but even the flies were not that bad.

Overall

All in all we had a fantastic time and hiked a total of 42.9 miles. The changes in the scenery were really interesting and the rope climbs and wild berries made for an interesting trek. Even though we had to scramble to find an alternative, I really think entering at Windy Pass and working counter clockwise worked out well for us. The only disappointment was that the weather wasn’t clearer when we were at the top, but it just means we will have to return sometime.

Forest Rangers

And finally, I want to give special thanks to the awesome forest rangers we met along the way. They are super fun to talk with and a treasure trove of information. They are also the ones that go through each year and set the ropes so you can climb down and up the cliffs. So thank you!

Samba4 Domain Controller on Ubuntu 18.04

Fri, 27 Mar 2020 00:00:00 +0000

Quick Background

I volunteer as a Systems Administrator for a small medical clinic. They have a very slim IT budget so while their desktop systems are primarily Windows 11, I’ve configured most of their server infrastructure with Linux. A Proxmox server is used to manage their virtualized environment and I’m using Samba4 for their Active Directory Domain Controller.

When I first set up Samba4 as their Active Directory I was able to use CentOS 6 with the Samba4 binary packages from SerNet. Although this has been working well enough, it has always been a little quirky and I had only ever set up a single Domain Controller. So recently I decided to upgrade their Domain Controllers to the newest version and add a second Domain Controller.

Initially I tried to find the SerNet packages but it appears they have rebranded themselves as Samba+ and now charge a subscription fee for the binary packages. While I would love to support them, paying a subscription fee is not an option for this clinic.

I’d really like to stick with CentOS as the server distribution as I have a lot of experience with it and have always found it to be a solid, minimial distribution. However, for reasons RedHat has decided that the Samba4 build that comes with Fedora should not include the ability to act as an Active Directory Domain Controller.

I could build the sources from scratch and still use CentOS, but that would require installing all the build tools on each server and I’m trying to keep them as light as possible. Plus, building from scratch often introduces its own host of challenges and I really just want to set this up and move on to my next project. So instead I’m going to give a go at using Ubuntu, which I really like as a desktop system but have never been terribly thrilled with as a server system.

For setting this up I am mostly following the instructions from tecmint and specifically this one. However, these instructions were written for Ubuntu 16.04 so I’m making this post to document the steps I took in setting up Samba4 on an Ubuntu 18.04 system.

Networking overview

For reference, the main gateway is a pfSense system set up at 10.0.1.1. This runs as the primary DNS and DHCP server for the network. The domain name I’ll use in these instructions is hrakaroo.lan.

The Active Directory domain is ad.hrakaroo.lan. I’ve specifically set this up so that Active Directory is on it’s own sub domain as I don’t want the rest of the server infrastructure to be dependent on Active Directory. I’d rather isolate Active Directory to its own area.

Creating a new system in Proxmox

In Proxmox I’ve created the following new host

Hostname: adc1
Memory: 2 G
Disk: 32 G
Processors: 1
Cores: 2

In retrospect (by looking at the usage charts in Proxmox) it looks like I could have cut the memory, disk and cores in half and it would have been fine, but I’m going to leave it as is for now.

Once the vm is created, start the server.

Most of the installation is pretty straightforward so I’m only going to highlight when I didn’t select the default.

Installer update available

During the install it suggests that an update to the installer is available and asks if you want to update. I’m not sure if it matters much, but I said yes.

Networking

Edit the IPv4 configuration and select Manual

Subnet: 10.0.1.0/24
Address: 10.0.1.25
Gateway: 10.0.1.1
Name servers: 10.0.1.26
Search domains: ad.hrakaroo.lan

10.0.1.26 is the name servers for the existing Active Directory Domain Controller. Once the system is fully configured I’ll change this, but it makes the initial setup much easier if this points at your existing Domain Controller.

Profile Setup

This is the first thing I don’t like about Ubuntu. I’d rather just have a password set for the root account. I get the reason why they are doing this, but for a super small setup this is more of an annoyance than a help. Sadly you also can’t use general accounts like ‘admin’ or ‘staff’ either. So I ended up creating a personal account for Joshua Gerth. (This turned out to be a bit of a mistake, more on this can be found in the additional things at the bottom.)

SSH Setup

[X] Enable install the OpenSSH server.

Basic Server Configuration

For almost all of these commands I find it easier to work by ssh’ing into the box rather than using the Proxmox console as copy/paste and editing all work better.

Update and install emacs

Okay, once the server is up and running I always run the update commands to make sure everything is up to date:

$ sudo apt update 
$ sudo apt upgrade
$ sudo apt dist-upgrade

and also, because vi is terrible I always install emacs:

$ sudo apt install -y emacs

Turn off IPv6

This may not be necessary, but I find that debugging things when only IPv4 is enabled to be a lot easier. The environment I’m installing this in is small enough that we are not at risk of running out of IP numbers any time soon and since everything gets NAT’ed anyhow I don’t really see a need to enable IPv6.

That said, this is the second thing that I don’t care for about Ubuntu. In CentOS turning off IPv6 is sort of trivial, but on Ubuntu I this proved to be rather difficult. Most of the existing suggestions only turned off part of IPv6 and it was only after a lot of trial and error did I happen on a post with the best answer.

Basically you need to edit /etc/default/grub and set

  GRUB_CMDLINE_LINUX="xxxxx ipv6.disable=1"

(For me there was nothing in the xxxxx area, but if you have any existing options then leave them there and add the ipv6.disable at the end.)

and then run

$ update-grub

Turn off dnsmasq

This is the third thing that I don’t like about Ubuntu, it runs a stub dnsmasq as a DNS caching server. Again, I get why they did this, but I’ve never found this to be useful on a server and nine out of ten times it just causes problems.

To disable it I’m mostly following these instructions.

Edit /etc/systemd/resolved.conf and add

DNSStubListener=no

Then remove the existing symlink at /etc/resolv.conf and rebuild it

$ rm /etc/resolv.conf
$ ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

Since this has been such a thorn in my side I prefer to reboot here to make damn sure the new settings have taken.

/etc/resolv.conf should now show the values that you entered on initial setup.

NTP

Domain Controllers are sensitive to clock drift and need to be configured to use a network time server. Normally I would configure ntpd on the host, but Ubuntu has decided to ship with timedatectl instead which claims to be a lightweight ntp client. (Humorously, 90% of the google searches for timedatectl suggest turning it off and replacing it with a real ntpd, however, I’m going to try sticking with timedatectl for now.)

First I like to set the timezone so I don’t need to convert everything

$ timedatectl set-timezone America/Los_Angeles

Now, edit the /etc/systemd/timesyncd.conf file and set the NTP entry to point to your NTP server. (I’m running an NTP server on another VM at 10.0.1.22).

[Time]
NTP=10.0.1.22

restart the timesyncd

$ systemctl restart systemd-timesyncd

and verify it with

$ cat /var/log/syslog | grep systemd-timesyncd
   XXX adc1 systemd-timesyncd[1416]: Synchronized to time server 10.0.1.22:123 (10.0.1.22).

Install Samba

Since we are actually about to install Samba I like to do one more reboot here.

Update the hosts file

The /etc/hosts file should have an entry for every domain controller, including itself. (I also removed the old IPv6 stuff)

127.0.0.1 localhost
10.0.1.25 adc1.ad.hrakaroo.lan adc1
10.0.1.26 oldadc.ad.hrakaroo.lan oldadc

Again, 10.0.1.25 is this box, the new Active Directory server and 10.0.1.26 is the old Active Directory server.

Install Samba4

Actually install samba

$ apt install -y samba krb5-user krb5-config winbind libpam-winbind libnss-winbind

If you already have things correctly setup then this should automatically find the kerberos SRV records for your domain which are being hosted on your existing Active Directory server, if not it may prompt you for them. If it does prompt you enter the domain in all upper case for the Default realm and in regular case for the Kerberos servers and Administrative server.

   Default Kerberos realm:  AD.HRAKAROO.LAN
   Kerberos servers realm:  ad.hrakaroo.lan
   Administrative server:   ad.hrakaroo.lan

Verify kerberos works by

$ kinit jgerth@AD.HRAKAROO.LAN
   Password for jgerth@AD.HRAKAROO.LAN:  <passwd>
$ klist
   Ticket cache: FILE:/tmp/krb5cc_0
   Default principal: jgerth@AD.HRAKAROO.LAN

   Valid starting     Expires            Service principal
   03/27/20 18:54:49  03/28/20 04:54:49  krbtgt/AD.HRAKAROO.LAN@AD.HRAKAROO.LAN
           renew until 03/28/20 18:54:46

Join the Domain

Remove the existing smb.conf as it will be recreated by samba-tool.

$ systemctl stop samba-ad-dc smbd nmbd winbind
$ mv /etc/samba/smb.conf /etc/samba/smb.conf.initial

Use samba-tool to join the domain

$ samba-tool domain join ad.hrakaroo.lan DC -U "AD\jgerth"

Now configure the server to start samba-ad-dc automatically

$ systemctl mask smbd nmbd winbind
   Created symlink /etc/systemd/system/smbd.service → /dev/null.
   Created symlink /etc/systemd/system/nmbd.service → /dev/null.
   Created symlink /etc/systemd/system/winbind.service → /dev/null.
$ systemctl unmask samba-ad-dc
   Removed /etc/systemd/system/samba-ad-dc.service.
$ systemctl enable samba-ad-dc

Use the host command to verify your configuration. This apparently hits every nameserver listed so you may get some errors when it tries to hit the non-active directory DNS servers.

$ host ad.hrakaroo.lan
   ad.hrakaroo.lan has address 10.0.1.26
   ad.hrakaroo.lan has address 10.0.1.25
   Host ad.hrakaroo.lan not found: 3(NXDOMAIN)
   Host ad.hrakaroo.lan not found: 3(NXDOMAIN)

$ host -t SRV _kerberos._udp.ad.hrakaroo.lan
   _kerberos._udp.ad.hrakaroo.lan has SRV record 0 100 88 oldadc.ad.hrakaroo.lan.
   _kerberos._udp.ad.hrakaroo.lan has SRV record 0 100 88 adc1.ad.hrakaroo.lan.

$ host -t SRV _ldap._tcp.ad.hrakaroo.lan
   _ldap._tcp.ad.hrakaroo.lan has SRV record 0 100 389 oldadc.ad.hrakaroo.lan.
   _ldap._tcp.ad.hrakaroo.lan has SRV record 0 100 389 adc1.ad.hrakaroo.lan.

Change DNS

At this point you could be done as the server is now up and running. However, it is technically using the other domain controller for DNS lookups. Even if I wasn’t planning on decommissioning the older server this would still be creating an out of band dependency between them and as Active Directory is, itself, a DNS server I prefer to update the server to use itself for DNS lookups. (Similar to what Ubuntu was initially trying to do with the dnsmasq stuff).

To do this, modify /etc/netplan and set 127.0.0.1 as first in your nameserver list and then the upstream DNS server second.

Also verify that search is set to use the ad domain first and then your global domain. Like this

            nameservers:
                addresses:
                - 127.0.0.1
                - 10.0.1.1
                search:
                - ad.hrakaroo.lan
                - hrakaroo.lan

As always, reboot and check the values in the /etc/resolv.conf and do a couple of tests with nslookup.

Configure Proxmox to boot the server

I also configure Proxmox to start the service at boot. This way if there is a power outage Proxmox will be sure to start your domain controllers.

Options -> Start at boot -> True

Additional things

Decommissioning the old Active Directory Controller

With the new Domain Controller up you can now decommission the old Domain Controller by moving the roles (7) from the old DC to your new one. However, the version of Samba which shipps with Ubuntu 18.04 is 4.7 which, apparently, has a bug with the python code that you use to move the roles. To fix this I had to edit /usr/lib/python2.7/dist-packages/samba/netcmd/fsmo.py

and insert line

import samba.drs_utils

just after

import samba

around line 20. After doing this the command to move the roles worked just fine and I was able to fully decommission the older active domain controller.

Removing the profile

It turns out that creating the profile as I did during the initial set up of Ubuntu was a bit of a mistake. It didn’t cause an issue for this server but it did on a file server I set up later which was also on Ubuntu. The issue was that the Ubuntu account jgerth conflicted with the Active Directory accouunt jgerth and made testing Windows File Sharing under my account difficult.

I could have addressed this by using a unique login on the Ubuntu server (like jogerth) but that feels like a pretty hacky solution. Plus, it raises the bigger issue that ultimately someday someone else is probably going to take over for me as the sysadmin and I don’t really want them to have to use my account. The right way to solve this is to create an LDAP server for the Linux boxes and host the logins centrally, but this feels a bit like overkill to me as we currently only have around 5 Linux systems and, at least for the time being, I would be the only person in the LDAP server.

I suppose I could bind the Linux boxes in to the Active Directory LDAP server, but that creates a bit of a chicken and egg dependency on Active Directory that I’m just not comfortable with.

What I really want is just a generic sysadmin account, but if you are going to do that then why not use the one that already comes with every Unix/Linux system, root. So in the end I configured an explicit password for root, enabled SSH login for root and deleted the account I created on setup. I would never recommend this for a larger environment, but for a small setup this worked out best.

Closing thoughts

In the end I’m glad I set this up from scratch rather than use the build and instructions from SerNet. I have a better understanding of what everything is doing and the weird quirkiness I was experiencing with the old server has gone away (although, to be fair, that could have been due to the older version I was running.) I also feel more comfortable with the samba tool and adding and removing domain controllers.

I’m still not a huge fan of Ubuntu as a server. I don’t care for the dnsmasq stub, the replacement for ntpd or how it forces you to create a non-root account, but since it is a popular system I was able to search for most of the errors I did encounter.

Supporting Windows 11 22H2

In the two years since I set this system up it has been running nearly flawlessly. The only real issue is that with Windows 11 update 22H2 Microsoft changed the encryption protocol (or something similar) and they are no longer able to bind to the domain controller. (This does not seem to impact existing systems that are already bound to the domain.)

The following Reddit link talks about the issue in depth https://www.reddit.com/r/sysadmin/comments/xoqend/samba_495_windows_11_22h2_kerberos/

The fix is to update Samba to version 4.16.0, which isn’t directly available for Ubuntu 18.04. The recommended path is probably to upgrade to Ubuntu 22.x, but since that’s bound to cause it’s own host of complications I instead opted to just change the source for Samba by following these instructions

https://ubunlog.com/en/ya-fue-liberada-la-nueva-version-de-samba-4-16-0-y-estos-son-sus-cambios/

sudo add-apt-repository ppa:linux-schools/samba-latest
sudo apt-get update
sudo apt install samba

This was enough to upgrade samba on 18.04 to 4.16.0. Once I restarted both domain controllers the Windows 11 with update 22H2 system was able to bind to the Domain.

Fractal Rust

Sun, 01 Mar 2020 00:00:00 +0000

Glob Library for Java

Thu, 26 Dec 2019 00:00:00 +0000

Ray the Tracer

Thu, 21 Nov 2019 00:00:00 +0000

Go for Java Developers - Part 2

Tue, 01 Oct 2019 00:00:00 +0000

This is my second post on things I’m finding interesting about the Go language. For additional background on this series you might want to read Part 1.

Dependency Management

This time I’m going to focus on dependency management and how go approaches the diamond dependency issue.

First off, Go does not support pre-compiled libraries the way Java does. (Personally, I think compiled Jar files are from a by-gone time and these days cause more harm than good, but that’s not why Go doesn’t have them.) Go doesn’t have them because Go code compiles down into native assembly so it would be impossible to distribute anything precompiled by Go and expect it to work on more than one platform.

So instead, Go publishes libraries as source code with Git tags and avoids artifactory all together. When you want to use a library you create a dependency on a git repo with a specific tag version.

This doesn’t exactly remove the diamond dependency issue as it is entirely possible to have a transitive dependency on two different versions of a library. In this case, as with Java, you still have to select the version you want to use. However, there are two substantial changes.

Can’t ignore the problem

First, you don’t have the option of ignoring or miss the issue. This means that if your service builds in Go, you can be confident your code does not contain any “hidden” mismatch errors. This effectively removes an entire class of errors that you have with Java. (TBH, these don’t pop up too often, but the more common a library is the greater the chance for this to happen. I’ve seen this error triggered with the Guava libraries.) Being able to completely remove them as a class of problems would be nice.

Conflicts identified at the source code level

Second, conflicts are identified at the source code level and not at the version level. This means that PATCH version mismatches won’t break your build so long as they don’t change function signature (and PATCHES really shouldn’t).

I’m not saying that the way Go deals with this is perfect but it does move the problem into a first class issue and forces a resolution. Java could take this same approach if you copied all of your dependencies into your code base as source code. But I did find it as an interesting consequence on how Go handles dependency management.

Go for Java Developers - Part 1

Thu, 01 Aug 2019 00:00:00 +0000

For the last 20+ years I’ve been mostly focused on Java development. But recently I had an opportunity to join a team which was focused on Go and having spent the early part of my career writing C++ I was intrigued by some of the decisions Go made. So this is the first (of possibly several) posts on things I found interesting about the language and since my I have been doing mostly Java development I’m going to be comparing and contrasting from that perspective.

Go interfaces

For this first post I’m going to discuss one specific implication of how Go handles interfaces.

If you are unfamiliar with Go, it uses structural typing for interfaces (which is more generally called “duck typing” meaning “if it walks like a duck and talks like a duck … it’s a duck.”) This means that if a struct in Go has functions on it which match the methods of an interface, instances of that struct are types of that interface. This sounds more complicated than it actually is so lets look at a quick example:

Suppose you wanted to create a Person class and a Named interface in Java, that might look like:

interface Named {
	String getName();
}

class Person implements Named {
	String name;

	String getName() {
    	return name;
	}
}

Pretty straight forward. Now, if we wanted to do a similar thing in Go it would probably look something like:

type Named interface {
     getName() string
}

type Person struct {
     name string
}

func (p Person) getName() string {
     return p.name
}

Ignoring the syntax differences, what is interesting is that in Go we don’t declare Person as implementing Named. This is implied by the function on Person matching the method in the Named interface. The compiler figures this out for us and instances of a Person are automatically of type Named.

So while this isn’t necessarily a new thing (Python mostly lets you do the same thing), this was something new for a statically typed language.

When I first saw this I thought “meh, so you can leave off the ‘implements’ keyword, whatever.” But I think this feature has a really interesting implication.

The setup

Back in Java land, let’s suppose you have a function that takes an SQL PreparedStatement as one of its arguments. Something like:

class Db {
   void setValue(PreparedStatement stmt, String value) {
 	stmt.setString(1, value);
   }
}

You might use this like:

PreparedStatement stmt = conn.prepareStatement("SELECT …");
Db db = new Db();
db.setValue(stmt, "someValue");

Writing a unit test for this would probably look something like:

class DbTest {

	@Test
	public void setValueTest() {
    	Db db = new Db();
    	PreparedStatement stmt = ??
    	db.setValue(stmt, "dog");

    	// todo - verify the statement received the value of "dog"
	}
}

The challenge is creating the test PreparedStatement. You could use one of the mocking frameworks in Java (Mockito, PowerMock, …) to mock the PreparedStatement and test that setValue is called with the value “dog”. While this does work, most/all of the mocking frameworks are basically a nice user interface over a rats nest of reflection calls. (Used incorrectly reflection is a tool to move compile time errors back to the runtime.) However, it is also possible to test this without using a mocking framework by building our own class which implements PreparedStatement:

class TestPreparedStatement implements PreparedStatement {

	String value;

	void setString(int parameterIndex, String x) {
    	this.value = x;
	}

	...
}

An instance of our TestPreparedStatement can now be passed into our setValue method and we can later verify that the internal ‘value’ is set to “dog”.

Ahhh … but there is a devil hidden in these details. PreparedStatement is a massive interface with well over 50 methods. In order to stub out the one method you want (setString) you are going to need to also stub out all of the other ones as well. We don’t ever use them so they can all throw a RuntimeException (and thankfully most modern IDEs can automatically generate this code for you) … but you are still dealing with a lot of boilerplate code.

The switch

Now let’s take a look at this same problem in Go land. (For the sake of argument let’s assume that PreparedStatement both exists in Go and works much in the same way as its Java counterpart).

type Db struct {
}

func (db Db) setValue(stmt PreparedStatement, value string) {
    stmt.setString(1, value)
}

And again it is used like:

stmt := conn.prepareStatement("SELECT …")
db := Db{}
db.setValue(stmt, "someValue")

Now let’s write the unit test:

func SetValueTest(t* testing.T) {
    db := Db{}
    stmt := ??
    db.setValue(stmt, "dog")

    // todo - verify the statement received the value of 'dog
}

We’ve once again hit the same issue. As with Java, we can use the Go’s mock framework (which also uses reflection) or, as before, we can try to roll our own.

Approaching it directly we could build out a TestPreparedStatement as:

type TestPreparedStatement struct {
    value string
}

func (tps TestPreparedStatement) setString(parameterIndex int, x string) {
     tps.value = x
}

But then we are going to have the same issue as we had with Java where we also need to stub out all of the other 50+ methods in the PreparedStatement interface which is just as wasteful as it was in Java. But we do have one other option.

Duck Typing

What if we didn’t stub out all of the other methods and instead created a new interface which only contained the one method we are using? So something like:

type StringSetter interface {
    setString(parameterIndex int, x string)
}

Now, since our TestPreparedStatement implements this one method it is automatically a type of StringSetter. But guess what, the object returned by conn.prepareStatement is also now a type of StringSetter since it too implements setString (by virtue of it being a PreparedStatement which also requires the same method.)

Now, we only need to change the signature of our setValue method to:

func (db Db) setValue(stmt StringSetter, value string) {
    stmt.setString(1, value)
}

This is where the power of the duck typing comes into play as doing this in Java would require modifying the definition of PreparedStatement so it explicitly extend StringSetter, which is virtually impossible. Yet in Go this is trivial as we can assign interfaces to objects by simply copying over the methods we want into the interface.

I found this to be a surprising feature about Go and is not something I had considered before. It feels really powerful and, to be honest, I’m not yet sure if this is a good feature about the language or not, but it is an interesting one.

Three Questions to Ask When Writing a New API

Thu, 06 Jun 2019 00:00:00 +0000

The following is blog post that was written from a speech I presented about questions to consider when writing an API.

Writing a functional API is relatively easy, but writing a good one that’s functional and empowers your users takes planning and patience. Designing a good API is about creating a sense of clarity and simplicity—it’s the bridge between your intention and your users.

Like most software development, building an API is a creative process; it’s impossible to completely define a hard-and-fast set of rules that will work in all cases. Nevertheless, three key questions—derived from what I consider the key characteristics of a good API—can serve you well as functional guideposts as you design and write your API:

Is your API’s usage discoverable?
Is your API composable?
Is your API safe to use?

Let’s take a closer look at each question.

Is your API’s usage discoverable?

In his famous book, The Design of Everyday Things, Don Norman coined the term discoverability. “When we interact with a product,” Norman wrote, “we need to figure out how to work it. This means discovering what it does, how it works, and what operations are possible.”

Take doors, for example: We interact with these standard physical objects every day. Often, based on the presence of affordances like knobs, handles, and push bars, it’s pretty clear how to open or close a door. But on occasion, a door’s design will suggest the opposite of how it actually works, and, as a result, we require instructions before we can properly use it. Just think of how many times you pulled a handle that actually needed to be pushed.

When we use a door the wrong way, we feel silly and stupid, but it’s not our fault. Actually it’s the design that’s bad.

Something similar can happen with a poorly designed API.

Consider the last API you used. How did you learn to use it? Did you read all the documentation first, or did you just jump right in? Maybe you weren’t sure about all of the parameters, so you sent in null for a few values and guessed at others. Did the API throw an error message when you did something wrong, or did it fail silently without any feedback? Did the error message clearly define which parameters were optional and which were not? Did you just keep plugging away until you got it right?

This is how most users will learn your API.

Your users are going to learn just enough to bootstrap themselves, and then they’ll figure the rest out as they go. With this fact in mind, you can help them along the away by increasing your API’s discoverability. You can do this through documentation; adhering to conceptual models; and using concise, symmetrical language.

Assume your users won’t read the documentation—until they need to

Just because your users won’t read your documentation doesn’t mean that you don’t need to provide it. You definitely do. But don’t design your API with the assumption that everyone will read the docs before they use it.

Some users would rather experiment than look up an answer in the docs. Every time I use Java’s substring() method, for example, I can never remember if the second value is an offset or a length, so I just write a little program to try it out both ways. This is usually quicker for me, and more fun, than looking up the answer.

In many cases, users who’ve learned to distrust documentation won’t read the docs anyway, at least not until they get desperate. Documentation is notorious for being out of date or just wrong. Now, this obviously isn’t true of all documentation, but think of how many times you’ve consulted documentation—or a help system or knowledge base—and found that either it provided answers that were totally useless, or it didn’t provide any related answers at all. Plenty of documentation does a poor job of anticipating the questions users might ask or how they might ask them. Additionally, even if users have a sense of what task they want to achieve, they may lack the exact vocabulary or use different terms for that task than the docs, which can make searching difficult.

You should also provide plenty of examples in your documentation—because users want them. Typically, examples are the first things users look for when learning a new API. Only after they gain a little context will they go look at the rest of the documentation. Examples are how users come to understand your API as a whole.

Create a conceptual model of how your API works

Don Norman explains that a conceptual model is “an explanation, usually highly simplified, of how something works.” Conceptual models are not schematics, and they should relate to other known conceptual models.

A good example of a conceptual model is the file system structure used on personal computers. File systems, like those on Mac and Windows operating systems, were intentionally based on the concept of files and folders that we were already familiar with in the physical world. This made it easy for non-technical users to understand and discover how to copy, store, and retrieve files on their PCs.

Even today, Unix uses this conceptual model of files and folders anytime a user attaches a device (e.g. a phone or external hard drive) to an operating system, which has completely eliminated the need for users to “discover” a new API every time they attach a device.

“Objects” in object-oriented programming are another example of a conceptual model. They’re specifically called objects so that we think of them as self-defining entities. Just as a ball object on the computer might support a bounce method, as well as other methods like throw, a ball in real life, through its design, also supports bounce and throw operations. In data-oriented programming, however, you don’t get this conceptual model, so you’re more likely to have a bounce function that will throw an error if you send it anything other than a ball.

Another example of working within conceptual models is the use of “object” in object-oriented programming. In this programming model, objects represent physical objects from the real world, such as servers, databases, and load balancers, and developers create relationships between those objects via APIs.

Use clear, consistent, and symmetrical language

In addition to documenting your API, you should also develop and publish a terminology dictionary for your API—and then use it consistently. For example, I commonly see APIs use terms like host and hostName, and account and accountId, almost interchangeably. Forcing your users to guess what the right call might be, or constantly changing the language, does not promote discoverability.

Like conceptual models, symmetrical language helps users work with your API with certain expectations in place. If your language is symmetrical, an open operation will be balanced with a close, and an add operation will be balanced with a delete.

In Python, for example, you use pop to remove an element, so the expectation would be that you’d use push to add an element, as that’s how it works in most other languages. Instead, Python uses append… and there’s plenty of Google search results from people confused by this poor discoverability.

Is your API composable?

When you build a composable API, you are letting your users select components of the API and use them in whatever pattern they want.

Small and composable methods are easier to describe and document than larger methods that contain a long chain of steps and caveats. They’re also easier to run regression and end-to-end tests against.

Most importantly, though, employing composable components gives your users the tools they need to build their own workflows with your API. You can’t predict all your users’ needs, so don’t force them into one execution pattern. Instead, create composable components and then use your examples to show how to combine them into larger execution patterns.

For example, consider the following methods:

setName(firstName, lastName)

vs.

setFirstName(firstName)
setLastName(lastName)

The second option is more composable than the first, as the second method allows you to easily update the value for lastName. With the first method you would first have to fetch the value of firstName so you could send it back in with the new value for lastName.

The second option is also more extensible, as you can easily add a method to set the middle name:

setMiddleName(middleName)

Finally, the second option is also 100% backwards compatible with existing code. If you were to update the first method to

setName(firstName, middleName, lastName)

you’d break the existing code.

Both you and your users will undoubtedly enjoy the free backwards compatibility, as building from smaller, composable components makes it much easier to extend your API as it grows; and to continue supporting support old operations alongside new ones.

Is your API safe to use?

Ensuring that your API is safe to use—that it won’t behave differently than users expect or break their workflows— is related to the discoverability of an API. But safety is so important that I want to call out the topic separately. When you publish your API, you create a relationship with your users that should be based on trust and transparency. Here’s how to make that happen:

Practice the principle of least astonishment

The principle of least astonishment tells us that a component of a system should behave in a way that most users will expect it to behave. The behavior should not astonish or surprise users.

The setDate method in GNU’s Coreutils, for example, surprises me every time I use it because I expect a set method to set a value and not alter it. If you set the year to any value less than 68, it automatically adds 2000 to the value; and if you set any value between 68 and 100, it automatically adds 1900. Every time I use this method, I’m astonished and have to re-read the documentation to make sure I’m using it correctly.

Follow the contract

Don’t try to interpret what you think your user is trying to do. For example, if your API expects a number, and the user provides a string, don’t try to parse a number out of the string. You aren’t doing anyone any favors: What happens when users enter an empty string: Is that 0 or null?

Design your API so that it’s deterministic and strict.

Trust nothing and fail fast

Similarly, your API should verify everything that users send, and immediately fail on errors. More specifically, garbage-in should not equal garbage-out. Garbage-in should fail. If your users are calling your methods with incorrect values, they may be in discovery mode, intentionally testing the boundaries and trying to figure out what is possible.

Help them understand what’s possible and what isn’t.

Plan to version from the start and aggressively deprecate old versions If you change the signature or external behavior of your API, version it.

And when you do roll an API’s version forward, dedicate time and resources to aggressively migrate users. If that’s not possible, try to rewrite older versions so they proxy to the new implementation. These steps will help avoid creating technical debt—which, like financial debt, definitely accrues interest over time. The longer an outdated version of your API sits around, the more ingrained it becomes in your user base, and the harder it will be to move users off of it. Set a migration date, and make it happen.

If you release a version that is likely to change quickly, make that fact explicit by tagging it as “incubating,” “unstable,” or “beta.” This helps provide breathing room if you need to turn off old versions of your API as you release new ones.

Separate your API from your implementation

Finally, publish your API version separately from its implementation. The implementation is likely to change faster than the API, so don’t tie the two together.

When versioning a library, for example, the API and its implementation are in the same package, so you can’t help but release them at the same time. But you can at least use semantic versioning to make it clear which parts are backwards compatible.

For a service, though, you can publish an API separately from its implementation. In fact, there are plenty of tools, including Apache Thrift, FlatBuffers, and Swagger, that allow you to write your API separately. With these tools, you write your spec and then build your implementation so that it implements the spec.

Nail that first impression

Your API is often a user’s first impression of your system. Spend time on discoverability, composability, and safety to make sure that first impression is a good one. Proper planning and design is critical to the effectiveness and success of your API. Taking the time to think things through will help to make your API a first-class feature—not a mere afterthought or means to an end.

Slides

Tue, 05 Feb 2019 00:00:00 +0000

Create slides in Markdown with Hugo Blox Builder

Hugo Blox Builder | Documentation

Features

Efficiently write slides in Markdown
3-in-1: Create, Present, and Publish your slides
Supports speaker notes
Mobile friendly slides

Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home
Finish: End
Overview: Esc
Speaker notes: S
Fullscreen: F
Zoom: Alt + Click
PDF Export

Code Highlighting

Inline code: variable

Code block:

porridge = "blueberry"
if porridge == "blueberry":
    print("Eating...")

Math

In-line math: $x + y = z$

Block math:

$$ f\left( x \right) = ;\frac{{2\left( {x + 4} \right)\left( {x - 4} \right)}}{{\left( {x + 4} \right)\left( {x + 1} \right)}} $$

Fragments

Make content appear incrementally

{{% fragment %}} One {{% /fragment %}}
{{% fragment %}} **Two** {{% /fragment %}}
{{% fragment %}} Three {{% /fragment %}}

Press Space to play!

One Two Three

A fragment can accept two optional parameters:

class: use a custom style (requires definition in custom CSS)
weight: sets the order in which a fragment appears

Speaker Notes

Add speaker notes to your presentation

{{% speaker_note %}}

- Only the speaker can read these notes
- Press `S` key to view
  {{% /speaker_note %}}

Press the S key to view the speaker notes!

Themes

black: Black background, white text, blue links (default)
white: White background, black text, blue links
league: Gray background, white text, blue links
beige: Beige background, dark text, brown links
sky: Blue background, thin dark text, blue links

night: Black background, thick white text, orange links
serif: Cappuccino background, gray text, brown links
simple: White background, black text, blue links
solarized: Cream-colored background, dark green text, blue links

Custom Slide

Customize the slide style and background

{{< slide background-image="/media/boards.jpg" >}}
{{< slide background-color="#0000FF" >}}
{{< slide class="my-style" >}}

Custom CSS Example

Let’s make headers navy colored.

Create assets/css/reveal_custom.css with:

.reveal section h1,
.reveal section h2,
.reveal section h3 {
  color: navy;
}

Questions?

Ask

Documentation

SQL is a terrible API

Thu, 01 Nov 2018 00:00:00 +0000

A long time ago…

In 1974 the first relational database was created and along with it a new user interface language was developed to make it easy to query the data. This language (or rather sub-language) was called Structured English Query Language or SEQUEL. SEQUEL would later evolved into SQL and become the preeminent language for querying relational databases. In the 45+ years since it’s introduction SQL has evolved slightly but it is still largely similar to the version introduced in the 70s. What has changed during that time is how we interact with the database. Today, SQL is used significantly more by applications querying the database than by end users. This usage pattern has created its own ecosystem of utilities and libraries which help applications build SQL queries. One of the more popular libraries for building SQL queries in Java is Hibernate’s Criteria Builder which uses an annotation pre-processor and a builder pattern to facilitate building SQL queries. (Java’s Persistence API was based heavily on Hibernate’s Criteria Builder). But, at the end of the day the builder still generates a basic SQL string which is sent to the target database, parsed and then executed. Hibernate is just one example, but there are hundreds of other libraries that exist across various languages which all seek to provide the same basic functionality. To provide an API which abstracts away the actual building of an SQL query.

So why do these all tools exist? Is an SQL query so complicated to construct that we need libraries and utilities to help us? In general, no, most SQL queries are relatively straightforward to construct. The problem is that SQL was designed for humans and not computers. The inclusion of the word ‘English’ in the original name was not by chance, SQL was intended to be similar enough to English that it would be self descriptive and would only require minimal transformation from the spoken question to the actual query. SQL was not written to make querying easy from other applications. In order to query a database from an application, the application needs to build the SQL query programmatically at run time, which means that all errors in the SQL query string are also going to be discovered at run time.

An Impedance Mismatch

Compiled languages offer an enormous advantage over interpreted languages in that you can be confident that, if the application compiles, then it does not contain any syntax errors. This assurance removes an entire class of errors that exist in pure scripting languages. Yet, by having an application generate SQL you are re-introducing the possibility of a run time syntax error. Furthermore, the errors which do get introduced are almost always related to the construction of the SQL query rather than actual column names or keywords in the query. Meaning the column names and keywords are not as likely to be the source of syntax errors as they don’t normally change based on the user request. For example, an application which allows the searching of available flights is less likely to return different a type of data depending on the destination city. You may get back more or less result, but it will usually be the same basic set of information. Syntax errors around the column names and keywords are often found through basic testing and fixed.

The more common (and harder to find) syntax errors are introduced in the construction of the query itself.

Did you remember a space after the ‘SELECT’ and ‘FROM’ keywords?
Did you join your select fields with a comma, but remember to not include a comma after the last one?
If this is the first filter then we need to add the ‘WHERE’ keyword, but if this is the second one we need to add the ‘AND’ keyword, and don’t forget about the parentheses.

These types of syntax errors are often introduced based on the users search criteria and can be significantly more difficult to find with basic testing due to the sheer number of permutations. This is the real advantage of using a helper library like Hibernate Criteria Builder. They provide an assurance that if you use their libraries, the generated SQL will be free of construction syntax errors. This assurance is most often achieved through massive test coverage and a responsive development team which quickly patches any errors that are found. It would be safe to say that in the 45+ years since the introduction of SQL, millions of lines of code have been written in various languages, all attempting to work around this same basic problem. It’s a huge waste of intellectual effort for what is essentially a self inflicted problem.

Where to go from here

SQL and similar English based DSLs (domain specific languages) are immensely powerful for building complex queries quickly. As an end user, nothing is more frustrating than trying to build a complex search criteria through a form based UI. First enter the subject, then select the predicate, then enter the object, then click the plus sign to add another filter, rinse and repeat. This becomes incredibly tedious very quickly and makes building complex groupings all but impossible.

This is where an English based DSL really shines. It allows a seasoned user to rapidly create complex queries as they are doing it interactively and have to do minimum mapping from what they are trying to search for to the actual query they are running (just as SQL was initially designed for). But that is where the DSL should remain, as a tool for the end user. Do not just create an API which takes the DSL directly as, at best, you will be recreating the same impedance mismatch and likely forcing your users to create the same set of “builder” libraries.

Instead, define a flexible structure which can be used to describe a query. JSON, XML or even one of the compressed protocols like Thrift or Flatbuffers could work. Users can then create their query programmatically and completely remove the risk of introducing a construction based error because they forgot a space or included a comma at the wrong location. Your API can either accept this structure directly for querying, or you can split it up and have different endpoints accept different categories of queries based on the structure of the return type. Either way, creating a well defined structure programmatically is straight forward and can easily be supported from a multitude of languages.

As for your UI, you are going to have to create a parser for your DSL anyhow which is going to build an intermediary representation of your query. If you merge your intermediary representation with your defined query structure it should be trivial to show the user what they would need to send to the API in order to reproduce their query in the API.

In this scenario your query language can continue to evolve separate from your query structure. You can even support other English based DSLs as long as they can all generate the same query structure. But what ever you do, keep the two domains separate. SQL is a terrible API.

Creating Simple and Effective Guidelines for Code Reviews

Tue, 09 Oct 2018 00:00:00 +0000

The following is a blog post that was written from a speech I gave about how to set the ground rules for code reviews.

Following New Relic’s Project Upscale—an innovative reorganization intended to make our development teams more autonomous—the engineering organization formed several new teams, one of which was the New Relic Database (NRDB) team. As the name implies, the NRDB team is responsible for the development of our events database, which powers the New Relic Insights tool as well as several other products.

When we formed the NRDB team, it included several senior-level software engineers. This was a highly skilled and very passionate group of developers reviewing one another’s pull requests.

When passion turns toxic

Being passionate about your work is one of New Relic’s core values. In this case, however, we may have experienced too much of a good thing: our code reviews soon became collision points, and we increasingly turned to passive-aggressive communications to settle our differences.

In the example on the left, the reviewer left the PR in an in-between state. They didn’t explicitly reject it, but they didn’t approve it either. In the example on the right, the reviewer made a highly subjective request, and the author just made the change, but from their tone you can kind of guess that they didn’t appreciate the feedback.

As a result, the NRDB team’s developers grew increasingly frustrated, team trust eroded, and several members (myself included) contemplated switching to other teams. We were in trouble.

Refining our process—and saving the team

We decided as a team to take a step back; we resolved to figure out what was going on, why it was happening, and what we could do to fix it. Since most of our frustration was tied to our code reviews, we started by asking a simple question: how could we give one another more effective and constructive feedback?

We answered the question by developing four basic guidelines for code reviews. We think you’ll find them useful, too, but before we spell them out, we want to share the full story behind what happened to divide our team and what was really as stake for us.

A flawed approach to the code review process

Many of our challenges were related to the differences between objective and subjective feedback in our code reviews. Being able to differentiate clearly between these two types of feedback can be critical to the success of a code review, and to the effectiveness of a development team. In too many cases, we weren’t handling subjective feedback in a constructive manner—in fact, just the opposite was true.

We probably aren’t the only ones who struggle with this issue. Many developers are trained from the start to downplay differences between the two types of feedback. In fact, students in academic software engineering programs rarely learn how to give or receive critical feedback of any sort.

When I went to school, this certainly was the case. The computer science curriculum focused on algorithm analysis, data modeling, and problem solving. Our instructors treated code review as a functional quality-assurance task; they rarely presented it as a creative process. Code review feedback tended to be straightforward: The code either worked, or it didn’t. Because of this kind of training—or rather, lack of training— many software engineers still treat all aspects of code reviews as completely objective activities.

It’s useful to contrast this approach with the one employed in an academic creative writing program. There, instructors conduct workshops that include training on how to give critical feedback. Creative writing instructors understand that giving and receiving critical feedback is an essential part of the creative process. They also understand, however, that critical feedback can be harmful and create resentment unless it is handled properly. The goal is to provide feedback in a positive and constructive way that helps to hone a writer’s ideas, enhance their creativity, and leave both parties enriched by the process.

The struggle over subjectivity

Many facets of a code review, however, are not straightforward. In particular, there are issues that demand subjective assessments for which there are no “correct” answers. This is where the rigid emphasis on code review as a totally objective activity, and the failure to consider the creative nature of software development, can become a problem.

Many elements of a modern code review process are now fully automated. Editors and IDEs will find syntax errors, evaluate Boolean logic, and warn about infinite loops. As a result, the bugs that survive are much harder to find, especially when you’re at the end of the process and are just looking at a code snippet with limited context.

Editors and IDEs, however, can’t detect—or prevent developers from focusing on—subjective issues such as confusing method names, questionable style preferences, and bad variable formatting. And when we dislike and disagree with what we find in such cases, we often forget that these “flaws” are subjective matters of opinion—not objective matters of fact.

This approach also makes it easy to forget that a debate over subjective issues during a code review can get emotional and heated very quickly.

Some teams try to regulate this problem out of existence by creating style guides that make objective rules out of subjective preferences. This approach rarely succeeds: software development is full of subjective choices, and there is no way to cover every subjective choice that developers may face in the course of project.

When a team lacks a clear communication channel for subjective feedback, the problem gets even worse. Reviewers may mix their subjective and objective comments without acknowledging the differences; here too, the process can end in resentment, frustration, and a breakdown in team communication.

Our four guidelines for code reviews

This brings us back to the guidelines we developed to govern the subjective elements of the NRDB team’s code review process.

First, as a preliminary to our four guidelines, we agreed to define who is ultimately responsible for the correct execution of any code changes. This was important to us because in a subjective debate, the opinions of the person who has the ultimate responsibility—in other words, verifying code execution— should carry the most weight. As a result, we decided that “The author of the code change is responsible for the correct execution of the change.”

This may seem obvious, but not all teams work that way. Some teams, for example, treat the review process as a QA process where the reviewer is ultimately responsible for verifying correct execution.

We found that subjective comments were most often presented as objective feedback at the pull request stage of the process. As a result, this is where we focused our code review guidelines.

In creating these rules, we laid a foundation for team members to clearly identify what a code reviewer should look for, and how to give both subjective and objective feedback. Here are the guidelines:

The reviewer should identify errors that will cause an issue in production. It’s a code review, after all, so the reviewer should identify missing semicolons, unending loops, or missing error handling. Reviewers aren’t responsible for finding all such errors (that’s still the responsibility of the author), but they should be on the lookout for obvious issues that will break the system if they’re are deployed into production. Such issues are a valid reason to block the pull request.
The reviewer should verify that the stated goal of the code change aligns with the changes being made. If the author submits a pull request that says they’re making changes to the networking code of a service, reviewers should expect that all of the changes are in and around the service’s networking code. This seems obvious, but it’s no secret that developers have a tendency to try to pack in multiple changes in such cases. This isn’t even necessarily a wrong practice, as long as the changes are mostly co-located. When you align a code change to its stated goal, however, you make it easier to determine if the pull request potentially submits any new bugs. Here, too, we agreed that failing to align the code change with its stated goal would justify blocking the pull request.
The reviewer should verify that any changes align with the team’s coding standards. I’ll cover this more in a bit, but as an example, if the team has decided that all variables must use camel case, and the reviewer finds a variable that does not use camel case, they should block the pull request.
The reviewer should look for anything they personally disagree with. This guideline addresses any comment which the first three rules do not cover. We want reviewers to give feedback, even if it’s not covered by the first three rules. We didn’t want our guidelines to suppress feedback, which is essential for how we learn from one another. Because these comments are clearly subjective, however, we agreed that they do not justify blocking the pull request.

To remove all confusion, we ask that reviewers specifically call out their comments as either blocking or non-blocking; and to add those comments as tags in their reviews. For example:

Objective comments

“Blocking: You are missing a semicolon.”
“Blocking: This loop never ends.”
“Blocking: You are missing some error handling here”

Subjective comments

“Non-blocking: Your method name is not clear enough.”
“Non-blocking: You should put the open curly brace on the line above.”
“Non-blocking: You should use camel case for your variable here and not snake case.”

Working within our code review guidelines

As we adopted these guidelines, the team had the most difficulty with the fourth one. Adopting this meant we had to accept two conditions:

The code our team produced did not need to be uniform. This meant overcoming a trend in our industry that says you should strive to remove all fingerprints from your code that identifies who wrote what part. We found the ROI on following this trend was pretty low, and trying to do so just led us back into the same subjective debate: If a developer writes code in a manner slightly different than their peer would, does that mean the code is incorrect? Clearly, we decided, that wasn’t the objective case.
If a reviewer adds non-blocking feedback, the author should take the time to consider it. Early on, some team members were concerned that authors would simply ignore all non-blocking comments, as their code was no longer blocked by subjective feedback. Our solution, then, was to reiterate that “we trust our teammates.” If, as reviewers, we took the time to enter a comment, we trusted that the author would take the time to read and consider it.

These both were contentious points, and the team spent a long time debating them. But ultimately, we found that the only way to work through these issues successfully is to live with the guidelines and give them a chance.

Sponsoring a coding standard

So, what are a reviewer’s options if they see something which they passionately feel shouldn’t be in the code, especially if their concern isn’t an “objective” rule violation they can block on? For such concerns, we agreed that a reviewer could choose to sponsor an addition to the team’s coding standards.

Every two weeks, we hold a retrospective meeting where team members are welcome to suggest changes or additions to our coding standards. There are two restrictions to this activity:

We cannot describe coding standards in subjective language. For example, a sponsor can’t say, “variables must not be ambiguous,” as ambiguity is subjective. But, the sponsor could add a standard that states, “variables must use Hungarian notation,” as this is objective and easily enforceable.
If you sponsor a coding standard, you must support it. The sponsor must provide documentation and training as needed. If there was a plugin or other tool the team needs to installed, the sponsor is responsible for supporting it. This restriction ensures the sponsor is passionate about anything they want to add to the team’s coding standards.

Finding respect and compromise in code reviews

After agreeing to these guidelines, we cleared all our existing coding standards and started over. For the first few weeks it was hard to break old habits, and we had to remind several team members to add the blocking and non-blocking tags during their pull request reviews. But once we got rolling with the new guidelines, we saw a number of successes.

First, by forcing reviewers to clearly identify those comments that were subjective, we noticed a change in how reviewers phrased their comments.Reviewers can no longer demand changes that meet their preferences; instead, they must request changes politely, and explain why they’re requesting the change. When we provide more explanation and context in this manner we create an environment that makes it easier for teammates to learn from one another. Plus, asking for changes, rather than demanding them, shows respect and acknowledges that the code’s author has valid feelings about their work, as well.

We also noticed that when a reviewer did write a non-blocking comment asking for a change, the author typically made the requested change or came up with a compromise—even though the author had the option of ignoring the comment. This demonstrates why asking for changes, rather than demanding them, builds stronger teams: the author feels less resentful, and the reviewer feels that the author genuinely appreciated their feedback.

We’ve identified a few other terrific benefits from this process. By limiting the scope of what qualifies as a blocking comment, for example, we reduced the time it took us to approve and merge changes, which resulted in greater overall project velocity. We have also reduced the time required to onboard new new team members and to get them up to speed with our code review process.

We have also updated our training materials to reflect our new code review process: We distribute one page that documents our guidelines, and another page that documents our coding standards. New team members now know exactly what they should be looking for and how best to communicate their suggestions.

We also expected the number of coding standards to increase greatly as reviewers sponsored new standards for items they could no longer block on. At the beginning, we did adopt several new coding standards, but after an initial burst, the number of new agreements fell off significantly. We concluded that since reviewers felt that authors were taking into consideration their subjective feedback, they did not feel as motivated to “convert” them to objective constraints based on their point of view.

These guidelines aid in team autonomy

The most important thing about these guidelines is that they support team autonomy; in no way do these guidelines dictate which coding standards teams should adopt. Teams are free to choose their own style guides, and they decide how strict they want to to be. These guidelines simply explain how to define coding standards and how reviewers should look for and give feedback.

We have come to appreciate the role that a strong and effective feedback process can have on building team morale, increasing team trust and communication, and improving development velocity. We implemented guidelines to strengthen the feedback process and to address issues that put the process at risk—and so far, I think we’re getting exactly what we hoped to get from these improvements.

Trying to solve Java's Gordian Knot

Fri, 01 Jun 2018 00:00:00 +0000

Java’s dependency management suffers from the dreaded diamond dependency issue. And while this issue is not unique to Java, it is, perhaps, more acute due to the precompiled nature of Java’s Jar files.

A Quick Refresher

If you are unfamiliar with this issue, here is a quick refresher:

Suppose there exists a library which contains a useful method that takes a string as its parameter:

class Common {
    void helpful(String a) {
        ...
    }
}

This library is published to an artifactory as

    com.hrakaroo : common : 1.0

Now, suppose there exists two other libraries which both use this helpful method

class Dog {
    void method1() {
        Common c = new Common();
        c.helpful("dog");
    }
}

and

class Cat {
    void method2() {
        Common c = new Common();
        c.helpful("cat");
    }
}

Each of these are also published to artifactory as:

    com.hrakaroo : dog : 1.0
    com.hrakaroo : cat : 1.0

And each of these have a transitive dependency on Common. Okay, now you decide to build your service which uses Dog and Cat so your dependency tree looks like

    com.hrakaroo : dog : 1.0
        com.hrakaroo : common : 1.0
    com.hrakaroo : cat : 1.0
        com.hrakaroo : common : 1.0

All is good. But now the folks who created Common come out with a new version which changes the signature of helpful and adds a boolean flag. So the new version looks like

class Common {
    void helpful(String a, boolean flag) {
        ...
    }
}

And knowing they are going to break some things publish this under a new version in artifactory

    com.hrakaroo : common : 2.0

The folks who made the Cat library realize this new flag will fix a bug they have had so they update to it as

class Cat {
    void method2() {
        Common c = new Common();
        c.helpful("cat", true);
    }
}

And publish it under

    com.hrakaroo : cat : 2.0

But the Dog library makers don’t need the new functionality so they don’t bother to update.

And finally, you decide to update your service to use the newest version of Cat which has the bug fix you need. This changes your dependency tree to:

    com.hrakaroo : dog : 1.0
        com.hrakaroo : common : 1.0
    com.hrakaroo : cat : 2.0
        com.hrakaroo : common : 2.0

In Java 8 you can not bring in the same dependency more than once with different versions. There are two common ways to deal with this, and they are both wrong.

First, you can not do anything. (This is probably the most popular solution.) In this situation gradle will pick one (usually the latest version) and use that as its version. In this case it will select com.hrakaroo : common : 2.0 which means that when your service calls Dog.method1 it will give you a runtime exception as the JVM will be unable to find the definition for helpful(String).

Or, if you are using gradle, you can use it’s force tag and force the version down to com.hrakaroo : common : 2.0 which means that when your service calls Cat.method2 it will give you a runtime exception as the JVM will be unable to find the definition for helpful(String, boolean)

The only “correct” solution here is to use gradle’s failOnVersionConfict() which will fail to compile your project unless both your dog and cat dependencies use the same version. This means you will be forced to fix the issue before your project can compile, but this may not be practical as a large project has lots of moving parts and compatible versions may not be available. Additionally, failOnVersionConflict() doesn’t understand semantic versioning so it will fail on PATCH level differences which often makes this a very painful and non-practical solution.

Most people just choose to go with the plug-and-pray approach where they just hope they never call a code path which encounters a definition which doesn’t exist.

Cutting the knot

As the two easy solutions are wrong and the one correct solution is impratical, the only real answer here is to avoid the problem altogether. When building a library, limit your dependencies.

Building a service requires a different approach from building a shared library. From the technologies you use to the way you version and test it are substantially different. Services are often just thin wiring together of different frameworks and libraries while libraries are more single tasked. And yet, they too often I don’t see people appreciate this difference. Instead they hack together libraries like they do services.

Stick to vanilla Java

I really like the Kotlin language, but I don’t think it has a place (yet) in shared libraries. Part of what makes Kotlin fun are all of the extension and infix libraries which are all packaged in the kotlin stdlib dependency. Any time you have more than two kotlin library dependencies you are just about assured to have a version conflict on the kotlin stdlib. Libraries should be written in Java to remove as many dependencies as possible.

Avoid huge common or utility libraries

Apache’s commons-lang3 is a fantastic library, but too often I’ve seen brought in so that the developer can use the StringUtils.join() method. Not only is this method trivial to write, but with Java 8 this can be done directly off the stream using .collect(Collectors.joining(...)).

The same can be said for Google’s Guava library, which is an enormous library that often makes non-compatible changes. In one library I reviewed the author had brought in a dependency on Guava so that they could use the Preconditions checks. While I think this type of defensive programming is good, the precondition checks can easily be re-created in your own library.

Copy and attribute

Providing the license allows it, it is also okay to simply copy sections of your dependent library directly into your own shared library and remove the dependency. Be sure to attribute where you got it from, but otherwise copying small to medium sized dependencies is okay.

Relocate and attribute

Finally, if all of the above are failing for you and, again if the licensing allows it, you can use a tool like shadow/shade to relocate a dependent library directly into you own library. These tools can will rebuild your resulting jar so that your dependencies are no longer transitive and all references to their old location have been changed to somewhere in your package. So, for example, you could relocate com.apache.commons to com.hrakaroo.apache.commons.

This will increase your jar size so it should really be used as the last resort, but it will guarantee that no one can later change the depenency version to something which is incompatable.

Whatever your approach, you should take the time when creating a shared library to minimize it’s transitive dependencies as much as possible. By doing so you will help minimize the risk to developers which use your library of creating their own diamon dependency nightmare.

The Dance Recital

Sun, 01 Oct 2017 00:00:00 +0000

The following is the speech I wrote for the Humorous Speaking Contest for Toastmasters. I placed first for the Division award but had to back out of competing at the Area level due to a travel conflict.

This speech is intended to be performed so it is not a formal writing.

My wife, who usually took our 10 year old daughter to her dance recitals, had been asked to speak at a conference that coincided with a recital, so I was tasked with taking our daughter instead. While this might stress out some fathers, it so happened that I was an experience dancer dad. My work schedule was very flexible so I had become primarily responsible for taking our daughter to her dance classes. I had spent hours at the dance studio and I’d seen it all. From temper tantrums, to crying and yelling and throwing things… and that was just the moms. I wasn’t worried in the least.

My first task was to take her to get her makeup for the recital. “Black eyeliner, red lipstick.” That seemed straightforward, but when we got to the makeup counter we found out there isn’t just one black eyeliner. There’s regular black, black out, onyx black, darkest black, ferocious black and blackest black, just to name a few. I must not be the most color aware person as all the different shades of black looked pretty similar to me. There were also water proof versions of everything which I ruled out instantly as we were certainly not going swimming. So we eventually narrowed it down to darkest black and blackest black, which seems like an absurd differentiation. I don’t know how you get to darkest without also being the blackest but far be it from me to question a multi-billion dollar industry. After much back and forth we finally settled on blackest black as we figured that way no one could complain it wasn’t black enough. The lipstick was a bit easier as the instructions had included a specific shade number.

However, it wasn’t until the day of the recital that I realized neither my daughter nor I had ever actually applied either eyeliner or lipstick. These days you could probably find an instruction video on-line, but this was pre-YouTube days. We decided to tackle the eyeliner first. For those that don’t know, an eyeliner pencil is essentially a sharpened stick which you need to get really close to the eye. The entire concept seems barbaric and on my first attempt I couldn’t bring myself to actually get right up next to her eye. This left her with a ring around her eye. Knowing this wasn’t quite correct I tried to fix it by coloring from the edge of the ring closer in towards her eye. Although in theory this worked, the end result was far more goth than I had intended.

The lipstick proved to be almost as challenging as the eyeliner, although less dangerous. To me, lipstick seems like a sticky crayon and your lips have a natural line on them that separates them from the rest of your face, so I figured if I just colored in between the lines I would be good to go. What I didn’t account for was that there is no natural stopping point inside your mouth and I spent a good while in front of the mirror making various lip poses trying to get a sense of how far in I should color.

“Dad!”

None of this was sitting well with my daughter who was becoming more and more panicked with each facial gesture. After deciding on a general plan I proceeded to put on the lipstick much like you would color in a coloring book, with rapid back-and-forth type motion. Again, although in theory this worked it left her with a very thick layer of lipstick on her lips.

Still, with the makeup done we rushed off to the dance recital. I got her checked in and took my seat in the audience.

The recital started normally but it didn’t take long before the hot overhead lights, heavy costumes and general physical activity started to take their toll. Like most of the dancers, my daughter started to sweat … heavily. This is apparently what waterproof eyeliner is for. Without it the sweat mixed with the thick ring I had applied and proceeded to run down her cheeks, giving her an Alice Cooper look. Meanwhile, the lipstick I had applied in her mouth had been rubbing against her teeth turning them a frightening shade of red and causing her smile to take on a menacing grimace. This added new meaning to the dance as it appeared my daughter was melting into a nightmarish black swan, threatening to eat the other dancers.

Stuck in the audience I could only watch with mounting horror as the thick layer of lipstick started to work itself outside her lips and smear around her face. My first thought was “my wife is going to kill me.”

Thankfully some of the other moms took pity on her and tried to clean her up between routines. When my wife returned and saw the recital photos she decided that she would never miss another dance recital again. My daughter recovered but has learned that dad is not who you go to for makeup advice. Still, whenever my kids ask me if they should wear the blue outfit or the black one, I like to respond with “now is that the darkest black one or the blackest black one?”

Thank you.

Walking The Board

Sun, 09 Jul 2017 00:00:00 +0000

In the traditional standup format everyone stands in a circle and each person, in order, gives an update. The updates follow the agile format:

What they did yesterday
What their plans are for today
Are there any blockers

I have a number of problems with this format.

Context Switching

People don’t usually stand in any particular order so the updates from person to person often jump between the project(s) the team is working on and support tasks. Additionally, if we’ve sized our tasks correctly, then a single person could potentially be working on a new task each day. This makes sense to the person giving the update, but for everyone else, trying to track actual progress becomes increasingly difficult.

Project Status

Related to context switching, it’s nearly impossible tell the overall project status from this type of update. Even if you could figure out how many tickets were closed, without looking at the kanban board it’s nearly impossible to get even a sense of what is left todo on the project. Plus, this format does nothing to encourage people to actually update the kanban board, compounding the issue.

Public Speaking

Public speaking does not come naturally to everyone, even when presenting just to the team. I’m much more relaxed with it now, but there was a time when this type of setting would cause me enough stress and I would ignore whatever was being said and just rehearse what I was going to say. Then, after I gave my update, I would spend rest of the standup obsessing over the things I had messed up or forgotten to say.

Competition for Busiest

Without focus, the standup can become a passive competition for who is the busiest person.

I’ve got a meeting with a customer today to talk over a feature they want added.
Well I’ve got two meetings today to talk over features.
I’ve got meetings pretty much all day long so I’m not likely going to get much else done.

It sounds silly, but it’s something I’ve seen happen at multiple companies, especially if people are feeling insecure (imposter syndrome). It can take the form of lamenting (but really bragging) about the number of meetings, or making the tasks you completed sound like herculean efforts. It’s pointless and unhealthy for the team.

Forced Update

Finally, it forces everyone to talk when they may not want to. Maybe they just got back from vacation and are still working at getting caught up. There have also been times in my career where my personal life has taken over and I’ve needed to take some time off, be it medical or family related. At the update I don’t want to go into details about what I’m dealing with. In fact, I may be at work specifically to take a break from the other stresses going on. The last thing I want to do is be forced to talk, or worse, explain why I don’t want to talk.

As an individual contributor I find this standup format stressful and as a project lead and manager I find it unhelpful. Instead I prefer using the standup time to walk the kanban board.

Walking the board

With most ticket tracking systems you can use swim lanes to split up your kanban board into project tasks and support tasks. Then, starting with the top swim lane, go down the Done column and each each owner can give an update for their ticket. (This also avoids that initial awkward moment of trying to figure out who should be the first to speak.)

Once the Done column is finished, move to column to the left (likely In Progress or Blocked) and repeat the process. Continue until you are done with all assigned tickets for that swim lane. Rinse and repeat for the next swim lane until all updates are done. The context switching is much less and the team can focus on what is being said instead of trying to memorize what they are going to say.

(Since Done is often an end state and you don’t necessarily want to re-walk this entire list every time, some teams will create a Closed column and use the standup as an opportunity to move tasks from Closed to Done.)

Closing comments

One critique of this format is that it is possible for a team member not working on a task to feel left out. There are a couple of ways to handle this: First although standup is suppose to be project focused, so I’ve always stressed that if you don’t have any updates for the project it is okay to not talk. (This assumes your team is a safe space where individuals won’t feel like they are being judged if they don’t talk.) But, if an individual really wants to explain to the team why they haven’t picked up any tasks, they should save that for the post-standup discussion. If, in the rare case, an individual is checking out and not picking up any tasks, that is something the manager should bring up during their 1:1 and not in front of the team.

As a project lead and manager I find walking the board to be significantly more useful as I can tell exactly the progress of the project (as I’m looking at the tickets) and I can also be sure that tickets for the project are up to date. As an individual contributor I don’t feel like I have to prepare as much for this standup and I can just attend and focus on what everyone else is saying.

One Seat Open

Wed, 27 Apr 2016 00:00:00 +0000

One Seat Open, an invitation system for limited seating events.

History

This project primarily came from a personal frustration in hosting my own game nights. Most invitation systems I researched were open ended (no wait list support) or were so chock full of ads that I could barely navigate them. At the same time I was also looking for a project with which to learn front end development as I have been mostly focused on backed development. So this seemed like a perfect opportunity to build my own service.

Development

After researching several frontend frameworks I eventually settled on Vue.js with the Vuetify plugin for several reasons. Vue.js was established enough to have good tutorials, training videos and a robust presence on Stack Overflow for questions. It also had an easy ramp up and it didn’t originate from Facebook. (Petty, I know, but I have a personal distain of Facebook).

Status

In November of 2019 I finished the initial version of and started to host a couple of game nights on the system. It was working pretty well but then the pandemic hit and I halted my home games, which halted my motiviation for working on the service. So it sat mostly idle until recently when I decided to start working on it again.

I initially wrote this service in Kotlin while I was doing a lot of Kotlin development at work. After changing companies I switched to primarily using Go for my work development and haven’t really done a lot of Kotlin development since then.

I still really like the Kotlin language, but it is much easier to develop in a language you are currently using. Plus, I often use my personal projects as a way to explore different solutions to technical challenges at work, so aligning the two can be useful.

All that said, I’ve decided to rewrite the project completely in Go with Vue3, but still using the Vuetify plugin. However, I’ve decided to keep this code base closed source for the time being. I’m not 100% sure yet what I want to do with it once/if I finish it and want to keep my options open for now.

Rebound

Sat, 10 Feb 2007 00:00:00 +0000

Soda Water Ray Tracer

Sun, 21 Nov 2004 00:00:00 +0000