Stop Over-Engineering

I have had it with over engineering!! If I have to see another code review with over-the-top abstractions and plenty of DB schema changes for a Forgot Password field I’m out of software. Fully giving up to become a bar keep in the west…

You’ve lost me, what’s over-engineering?

Sorry about that, tough day at the office…

So, over-engineering is when a software engineer builds too much to solve a problem, and essentially over complicates the solution. It’s like trying to remove milk that is long past it’s use by date in the fridge so you naturally build a massive missile carrier to get rid of the problem. NO, you say? You’re saying thats too much effort, and that I should pick up the milk and put it in the waste bin? You may have a point.

Why is it so bad though? It’s bad because when you are spending time over-engineering you could have been solving the real problem, or moving on to an issue that will change the business for the better. In terms of startups, you are not failing fast if you are over-engineering your solution.

Another reason over-engineering little things is an issue is down to “marginal gains”. David Brailsford took over as the Performance Director of the British Cycling; Team Sky. His approach was relatively simple if you take a 1% improvement in everything thing you do, these gains will add up to a significant improvement. He promised that Team Sky would win the Tour de France for the first time within 5 years. He was right, they won it in 2013, two years quicker.

Back to over-engineering, if you refactor an hour extra here for no real reason, or if you add a new design pattern just to try it out… these hours add up. Get the code live, learn from the metrics and don’t be tempted to over engineer.

Got it!! Real World examples plz

Wrappers – How many times have you written a wrapper around a 3rd party library and you haven’t swapped it out? Nowadays 3rd party integration is far better, these are tested code bases written by excellent engineers. It’s 2017!!

Make It All Generic – Need to write to the database? Create a generic query. Pass it parameters? Pass generic params. Need to create a factory? Create generic factory-factory. Don’t do it. Duplication is better than the wrong abstraction.

In House Tools – Writing your own tool is going to cause the business to sink time and money into them over the coming years. Tiny open source projects take a lot of effort to maintain and no guarantee they will be used elsewhere. Reuse others, Fork others and contribute to others.

There are more examples here

Make It Stop

There are several ways, to stop over engineering but the first of course is to be aware of it. If you’re a scrum master or a developer, ask questions to the team. “What scenario are you trying to solve?” and dive deep into the scenario.

In standup, or group discussion ask “How could we fail faster on this feature?” Remind the team what is valuable, seeing those metrics change. If the team are over-engineering reminding them that it’s first more valuable to get the feature out than it is to have the feature perfect.

It’s a team effort. Ensure you are all working on vital parts. Keep your pair honest, if you don’t need to refactor X – don’t. For businesses over-engineering leads to the loss revenue, for a selfish gain. And for startups… over-engineering could cost you success.

Advertisements

Commit.

I’ve worked in several different companies over the last ten years and I’ve worked with several different project management styles; Agile, Lean, with OKRs, 4DX and heck – I’ve even dabbled into Waterfall and PRINCE2. In the more recent months of working at FindMyPast and Roadie I have seen what aids delivery. Commitment.

Whats your issue “dude”?!

My issue is simple; I have worked on projects and at companies where we struggled to deliver.  In order to get to market, or get that new feature out swiftly, engineers and teams must deliver – effectively. I feel commitments, aid deliver – massively.

Boy! How did you ever come to this realisation?

The engineering team I lead at FMP, Team Rex, work using commitments as a part of our weekly process. Although, we stopped a couple of months back and we stopped focussing on the right tasks which meant the project was more stagnant. It was during this point I realised how powerful commitments had become in our team.

To explain this further, let me take you back OKRs – it’s a simple goal setting technique used by several companies including Google and even us at Roadie. The idea is to connect the company, the team and the individuals to a single objective and move towards that.

I first had experience working with this framework at FMP and to educate myself I read Radical Focus by Christina Wodtke. Not only that but I watched several videos of Christina talking about the subject matter and I loved it. I loved the idea of increasing focus on the goal and not getting distracted by X,Y and Z.

OKR’s are Objective’s and Key Results.

Objective – A “shoot for the moon” objective, an ambitious goal to aim for. here is our first Objective from Roadie:

“Objective: Create a leading platform that allows driving instructors to increase their business whilst lowering costs”

Key Results – A metric that is measurable that shows you have met your objective. Examples for our first stab at Roadie are:

KR: 95% of registered instructors compete each section of their profile
KR: 75% instructor registrations from initial invite

Radical Focus explained the entire company had to live and breathe OKRs, and that’s where commitments came in.

Christina said every week the development team should meet and talk about each KR, and set commitments to how they are going to change their current status. The team would commit to 2-3 things a week and they would focus on changing these metrics. This increased focus every single week as the team would keep themselves accountable. During standups the team then start to challenge each other, ensure they are working on the right tickets that will help them achieve all their commitments that week.

But in the real world…

So that’s the theory, and let me tell you – IT WORKS! Every stand up meeting, when an engineer wants to “refactor x” we state “yeah that needs to be done, but lets get our commitments done first”. It’s a powerful technique to keep the team on track.

At FMP we used this during our Internal Customer Service project – we delivered 6 weeks early.

During every Roadie session we start the session by defining what we want out of it, we then write them on the whiteboard and start the work. It keeps our sessions on point.

My advice is have the team come up with these commitments at the start of the week. Write them down, print them out and keep them visible at all times. Every time one is ticked off, make a big deal – celebrate the win. But overall, make sure you are committing to the right things – make sure it’s tied to the Metric That Matters

Start with a monolith

So building a new product eh? Reading about mico-services are we? Hmmmm.

Don’t do it. That’s what I say.

Don’t over complicate your product at the beginning, yes a micro-service architecture will allow you to move faster but you don’t know what your product is yet. It’s important in early days to build something that consumers want rather than something with a fancy architecture.

Take Roadie for example, tonight we built the beginnings of our Review Service… and I thought “should I build this as a separate app?” because I know we will build more than reviews for driving instructors, but at moment we are just building them for driving instructors… why start going a bit crazy with architecture when we haven’t even got paying customers yet?!

So we aren’t. We knowingly are going to build a monolith, with clear boundaries, easily maintainable code that we can break off into service when we need to. Focus on customer value right now and let’s get into some tech debt.

Baby step into Continuous Improvement

So you keep these buzz words around the water cooler huh? “Continuos Improvement”, “Continuous Delivery”,  “Kanban”, “Kaizen” and “Value Stream Mapping” but what does it all mean Basil?

Well I don’t want to dive deep into all that right now but I do want to sell you the idea of trying out Continuous Improvement, even for a week or two.

Give me one good reason you know it all schmuck!!

Improving your work flow will allow teams to deliver faster.

Simple right? It is yet so many engineering teams do not focus on this during week or sprints, it’s all features or bugs!

If the engineering team can deliver faster they will deliver features faster, hopefully meaning the faster revenue comes through the door.

But How?! I’m only a man!!

First, of you need to visual your current work flow.

  • Where does work come from?
  • When do product teams see the work?
  • When do engineers see the work?
  • How is it released?

Estimate how long it takes from one section to another and you will quickly see the pain points.

Excuse the crudity of this model

screen-shot-2017-02-23-at-21-09-45

You can see here the later steps in this diagram have times indicating how long it takes to progress from one stage to the next. You can identify issues with certain areas like tickets could stay in UAT for 3 days or each release takes 40 minutes.

If we focus on the latter stat of a release taking 40 minutes, how much time is the business wasting on releases every week? If you do 10 release a week thats almost 7 hours per week!

Continuous Improvement is tweaking this development cycle in order to get work through it faster. If we want to baby step into Continuous Improvement, instead of having a team taking on 20 points of feature work per week, take on 15 points and use 5 the remaining points on something that will help speed up delivery.

Some improvements could be:

  • tweaking the build pipeline that will allow teams to release faster.
  • write a tool that will auto generate files that are needed for new pages, controller or modules.
  • Engineers having trouble with a tech stack, allow time for learning and knowledge sharing

The idea is to come back to the above diagram every few weeks, and seeing how the team can tighten those times down. Try this for a couple of weeks and see how the team does, you could save hours or even days per feature meaning the business gets to market faster, which means you can fail faster… wait what? Oh, more on that later…

Metrics Driven Development

You are months into a project, knee deep in user stories, bugs and of course technical debt.

Where did it all go wrong?

It doesn’t matter right now, leave that for the project post-mortem. The real problem is how to take the project that is in flux to completion. That’s where Metrics Driven Development (MDD) comes in. This is a framework that will not so much hold your hand to the end of the project but drag you faster to those far off goals.

What you talking ’bout?

Metrics Driven Development. Let’s say you are moving data from one Database to another, breaking a monolith application into an array of services or you are creating a new project and trying to convert customers through the funnel. This is where MDD will help you succeed. It focuses the developers, the project managers and the entire business to recognise what success looks like. That’s really important.

Metrics Driven Development helps identify what success looks like.

So half way through a project, it’s useful to say “okay, where are we now?” and to look at the state of play. What factors demonstrate that the project/feature are complete? What metric(s) says to everyone “we are code complete?”

The engineering team then write this on their white board or the wall and never forget the goal.

  • Everyday at stand up, you pick up the ticket and say “how does that get us closer to our target?”
  • Every time a ticket is moved into “Done” you should see that graph move closer to the end result.

If it is not helping you progress, chances are you are working on the wrong things. This is a culture shift for sure. Although, if the task does not feed directly into moving that graph, it needs to either be a technical task that will help the team deliver faster, or fixing a defect. This should change the stand-up conversation entirely. “What are we doing to get closer to our goal?” This allows engineers to be more accountable for the success of the project as they are pushing for the wider picture than just individual coding tasks.

A more recent example of MDD in use would be the project my development team are finishing up; migration of traffic from a monolith application to a new micro-service. The goal is to have 100% traffic going through this new API rather than the old one. We only work on tasks that move that percentage closer to our goal. We have this percentage and the graph on big TVs, and every release we do we are looking at how our changes impact these metrics.

Where on Earth….

This all came from several years of failing to deliver quick enough. This helped finally putting the finger on it as to why myself and companies I have worked for have been failing for so long.

In the past I worked on a financial reporting application that was moving from an old system to a new one, 10 months in and we still had no visibility of where we were. That is until we built a dashboard comparing the old system to the new one, and we were getting realtime data everyday. That changed the stand up conversations to what everyone is working on to “How do we get closer to 100%?” And that fed into our tickets and planning too.

Sean Covey has coined “4 Disciplines of Execution” that focuses on talking about Lead Measures and Lag Measures which we used later in that same company. And Christina Wodtke has written a book “Radical Focus” based on Objectives and Key Results (OKRs) and this really is to allow you to focus on the Key Results that will allow you to achieve your goal. Take ideas from both these methodologies you come to what I call Metrics Driven Development:

  • Goal. Write down what Goal of your project or feature is.
  • Visualise. What does success look like on a graph?
  • Work. Only pick up tasks or plan tasks that are going to move you closer to that goal.
  • Commit. At the start of the week commit to what % or # you want to achieve for and plan for it.

If you stick to this plan you will deliver faster, which will increase the likely hood of success and prevent the team from ended up in no mans land.

Presenting: Roadie

In the last nine months I have been working with a friend on a new web application using Lean Development methods. That application is:

logo

Roadie is built for learner drivers to find the perfect instructor to help them pass their test. The learner can compare, filter driving instructors and book lessons online.

You can look here: https://www.goroadie.com/

During this project I have learnt many different technologies; Rails, Devise, Heroku, AWS, RDS, S3, Cloudfront, HTTPS and really that list continues to grow. In the last two weeks we had our first customers – 3 users booked driving lessons through us.

The next goal? Scale and increase our number of bookings.

New Project: Edinburgh Air

I love to travel. It’s something I do well and I do it a couple times in a year. In the last four years I have visited Australia, America, Canada, Thailand, Malaysia, Vietnam, and many other countries. It’s a definitely a passion of mine.

So, it wouldn’t be surprising that my next project would be travel focussed. I am hoping to build a site that will give me the latest offers from Edinburgh Airport. It could tweet these deals, it could give the latest information on the airport. Completely automated. I wonder if I could make it useful and interesting for other users.

I feel such an application may need too much human interaction to make it worthwhile. So, I am attempting to build this automated system that gives news and flight details that does not seem too spam-bot-like.

My primary hypothesis is that:

Given “Edinburgh Air” is able to display relevant airport information regarding Edinburgh Airport AND is able to deliver up to date flight deals users will use my site to book flights from Edinburgh.

Check out what I have so far: http://www.edinburghair.com