Technological Thoughts by Jerome Kehrli

In my current company, we embrace agility down the line, from the Product Management Processes and approaches down to the Software Development culture.
However, from the early days and due to the nature of our activities, we understood that we had two quite opposed objectives: on one side the need to be very flexible and change quickly priorities as we refine our understanding of our market and on the other side, the need to respect commitment taken with our customers regarding functional gaps delivery due dates.
In terms of road-mapping and forecasting, these two challenges are really entirely opposed:

• Strong delivery due dates on project gaps with hard commitment on planning. Our sales processes and customer delivery projects are all but Agile. We know when in the future we will start any given delivery project and we know precisely when the production rollout is scheduled, sometimes up to 12 months in advance. We have most of the tine a small set of Project Gaps required for these projects. Since we need to provide the delivery team with these functional gaps a few weeks prior to the production rollout, it turns out that we have actually strong delivery due dates for them, sometimes 12 months in advance.
• Priorities changing all the time as our sales processes and market understanding progress. We are an agile company and mid-term and even sometimes short-term focus changes very frequently as we sign deals and refine our understanding of our market, not to mention that the market itself evolves very fast

These two opposed challenges are pretty common in companies that are refining their understanding of their Product-Market Fit. Having to commit strongly on sometimes heavy developments up to more than a year in advance, while at the same time changing the mid-term and short-term priorities very often is fairly common.

In this article, I would like to propose a framework for managing such a common situation by leveraging on a roadmap as a communication, synchronization and management tool by inspiring from what we do in my current company (leveraging on some elements brought by Mr. Roy Belchamber - whom I take the opportunity to salute here).

There are three fundamental cultural principles and practices that are absolutely crucial in our context to handle such opposed objectives. These three elements are as follows:

1. Multiple interchangeable development teams: multiple teams that have to be interchangeable are required to be able to spread the development effort among flexible evolutions - that can be reprioritized at will - and hard commitments - that need to be considered frozen and with a fixed delivery due date.
2. Independent and autonomous development teams: these development team need to be able to work entirely independently and without and friction from any other team. This is essential to have reliable estimations and forecasts. A lot of the corollary principles and practices I will be presenting in this article are required towards this very objective.
3. An up to date and outcome-based Roadmap. Having a roadmap that crystallizes the path and the foreseen development activities in the next 2 years is absolutely key. Such a roadmap is all of an internal communication tool, an external communication support, a management and planning tool..

Agile Roadmap Example

In this article, I intend to present the fundamental principles behind the design and maintenance of such a roadmap that are required to make it a powerful and reliable tool - and not yet another good looking but useless drawing - along with everything that is required in terms of Agile principles practices.

The Search for Product-Market Fit is the sinews of war in a startup company. While the concept and whereabouts are well known by most founders, the importance of this event in the company and product building process, what it means to be Before-Product-Market-Fit and After-Product-Market-Fit and the fundamental differences in terms of objectives, processes, culture, activities, etc. between these two very distinct states is almost always underestimated or misunderstood.

Product-Market Fit is this sweet spot that startups reach when they feel eventually that the market is really pulling out their product. It's what happens when they found out that they can only deliver as fast as the customers buys their product or when they can only add new servers in the SaaS cloud as fast as is required to sustain the rise in workload.
Product-Market Fit is so important because it has to be a turn point in the life of a young company.

• Pre-Product-Market Fit, the startups needs to focus on the leanest possible ways to solve Problem-Solution Fit, define and verify its business model and eventually reach Product-Market-Fit.
• Post-Product-Market Fit, the company becomes a scale up, needs to ramp up its marketing roadmap and effort, build and scale it's sales team, build mission-centric departments, hire new roles and recruit new competencies, etc.

Dan Olsen designed the following pyramid to help figure what Product-Market Fit means (we'll be discussing this in length in this article):

Understanding Product-Market Fit and being able to measure and understand whether it's reached or not is crucial. Reaching PMF should be the core focus of a startup in its search phase and understanding whether it's reached is key before scaling up.
This article is an in-depth overview of what Product-Market-Fit means and the various perspective regarding how to get there. We will present the Lean-Startup fundamentals required to understand the process and the tools to reach product market fit, along with the Design thinking fundamentals, the metrics required to measure it, etc.

Test Driven Development is a development practice from eXtreme Programming which combines test-first development where you write a test before you write just enough production code to fulfill that test and refactoring.
TDD aims to improve the productivity and quality of software development. It consists in jointly building the software and its suite of non-regression tests.

The principle of TDD is as follows:

1. write a failing test,
2. write code for the test to work,
3. refactor the written code,

and start all over again.

Instead of writing functional code first and then the testing code afterwards (if one writes it at all), one instead writes the test code before the functional code.
In addition, one does so in tiny small steps - write one single test and a small bit of corresponding functional code at a time. A programmer taking a TDD approach shall refuse to write a new function until there is first a test that fails - or even doesn't compile - because that function isn't present. In fact, one shall refuse to add even a single line of code until a test exists for it. Once the test is in place one then does the work required to ensure that the test suite now passes (the new code may break several existing tests as well as the new one).
This sounds simple in principle, but when one is first learning to take a TDD approach, it does definitely require great discipline because it's easy to "slip" and write functional code without first writing or extending a new test.

In theory, the method requires the involvement of two different developers, one writing the tests, then other one writing the code. This avoids subjectivity issues. Kent Beck has more than a lot of examples of why and how TDD and pair programming fit eXtremely well together.
Now in practice, most of the time one single developer tends to write tests and the corresponding code all alone by himself which enforces the integrity of a new functionalities in a largely collaborative project.

There are multiple perspective in considering what is actually TDD.
For some it's about specification and not validation. In other words, it's one way to think through the requirements or design before one writes the functional code (implying that TDD is both an important agile requirements and an agile design technique). These considers that TDD is first and foremost a design technique.
Another view is that TDD is a programming technique streamlining the development process.
TDD is sometimes perceived as a way to improve quality of software deliverables, sometimes as a way to achieve better design and sometimes many other things.

I myself believe that TDD is all of this but most importantly a way to significantly reduce the "Total Cost of Ownership (TCO)" of software development projects, especially when long-term maintenance and evolution is to be considered.
The Total Cost of Ownership (TCO) of enterprise software is the sum of all direct and indirect costs incurred by that software, where the development, for in-house developped software, is obviously the biggest contributor. Understanding and forecasting the TCO and is a critical part of the Return on Investment (ROI) calculation.

This article is an in depth presentation of my views on TDD and an attempt to illustrate my perspective on why TDD is first and foremost a way to get control back on large Software Development Projects and significantly reduce their TCO.

Agility in Software Development is a lot of things, a collection of so many different methods. In a recent article I presented the Agile Landscape V3 from Christopher Webb which does a great job in listing these methods and underlying how much Agility is much more than some scrum practices on top of some XP principles.
I really like this infographic since I can recover most-if-not-all of the principles and practices from the methods I am following.

Recently I figured that I have written on this very blog quite a number of articles related to these very Agile Methods and after so much writing I thought I should assemble these articles in a book.
So here it is, The Agile Methods Collection book.

The Agile Methods Collection book is simply a somewhat reformatted version of all the following articles:

• Agile Landscape from Deloitte
• Agile Software Development, lessons learned
• Agile Planning : tools and processes
• DevOps explained
• The Lean Startup - A focus on Practices
• Periodic Table of Agile Principles and Practices

So if you already read all these articles, don't download this book.
If you didn't so far or want to have a kind of reference on all the methods from the collection illustrated above, you might find this book useful.
I hope you'll have as much pleasure reading it than I had writing all these articles.

After writing my previous article, I wondered how I could represent on a single schematic all the Agile Principles and Practices from the methods I am following, XP, Scrum, Lean Startup, DevOps and others.
I found that the approach I used in in a former schematic - a graph of relationship between practices - is not optimal. It already looks ugly with only a few practices and using the same approach for the whole set of them would make it nothing but a mess.

So I had to come up with something else, something better.
Recently I fell by chance on the Periodic Table of the Elements... Long time no see... Remembering my physics lessons in University, I always loved that table. I remembered spending hours understanding the layout and admiring the beauty of its natural simplicity.
So I had the idea of trying the same layout, not the same approach since both are not comparable, really only the same layout for Agile Principles and Practices.

The result is hereunder: The Periodic Table of Agile Principles and Practices:

The layout principle is and the description of the principles and practices is explained hereafter.

All the work on Agility in the Software Engineering Business in the past 20 years, initiated by Kent Beck, Ward Cunningham and Ron Jeffries, comes from the finding that traditional engineering methodologies apply only poorly to the Software Engineering business.

If you think about it, we are building bridges from the early stages of the Roman Empire, three thousand years ago. We are building heavy mechanical machinery for almost three hundred years. But we are really writing software for only fifty years.
In addition, designing a bridge or a mechanical machine is a lot more concrete than designing a Software. When an engineering team has to work on the very initial stage of the design of a bridge or mechanical machine, everyone in the team can picture the result in his mind in a few minutes and breaking it down to a set of single Components can be done almost visually in one's mind.

A software, on the other hand, is a lot more abstract. This has the consequence that a software is much harder to describe than any other engineering product which leads to many levels of misunderstanding.

The waterfall model of Project Management in Software Engineering really originates in the manufacturing and construction industries.
Unfortunately, for the reasons mentionned above, despite being so widely used in the industry, it applies only pretty poorly to the Software Engineering business. Most important problems it suffers from are as follows:

• Incomplete or moving specification: due to the abstract nature of software, it's impossible for business experts and business analysts to get it right the first time.
• The tunnel effect: we live in a very fast evolving world and businesses need to adapt all the time. The software delivered after 2 years of heavy development will fulfill (hardly, but let's admit it) the requirements that were true two years ago, not anymore today.
• Drop of Quality to meet deadlines: An engineering project is always late, always. Things are just a lot worst with software.
• Heightened tensions between teams: The misunderstanding between teams leads to tensions, and it most of the time turns pretty ugly pretty quick.

So again, some 20 years ago, Beck, Cunningham and Jeffries started to formalize some of the practices they were successfully using to address the uncertainties, the overwhelming abstraction and the misunderstandings inherent to software development. They formalized it as the eXtreme Programming methodology.

A few years later, the same guys, with some other pretty well known Software Engineers, such as Alistair Cockburn and Martin Fowler, gathered together in a resort in Utah and wrote the Manifesto for Agile Software Development in which they shared the essential principles and practices they were successfully using to address problems with more traditional and heavyweight software development methodologies.

Today, Agility is a lot of things and the set of principles of practices in the whole Agile family is very large. Unfortunately, most of them require a lot of experience to be understood and then applied successfully within an organization.

Unfortunately, the complexity of embracing a sound Agile Software Development Methodology and the required level of maturity a team has to have to benefit from its advantages is really completely underestimated.
I cannot remember the number of times I heard a team pretending it was an Agile team because it was doing a Stand up in the morning and deployed Jenkins to run the unit tests at every commit. But yeah, honestly I cannot blame them. It is actually difficult to understand Agile Principles and Practices when one never suffered from the very drawbacks and problems they are addressing.

I myself am not an agilist. Agility is not a passion, neither something that thrills me nor something that I love studying in my free time. Agility is to me simply a necessity. I discovered and applied Agile Principles and practices out of necessity and urgency, to address specific issues and problems I was facing with the way my teams were developing software.

The latest problem I focused on was Planning. Waterfall and RUP focus a lot on planning and are often mentioned to be superior to Agile methods when it comes to forecasting and planning.
I believe that this is true when Agility is embraced only incompletely. As a matter of fact, I believe that Agility leads to much better and much more reliable forecasts than traditional methods mostly because:

• With Agility, it becomes easy to update and adapt Planning and forecasts to always match the evolving reality and the changes in direction and priority.
• When embracing agility as a whole, the tools put in the hands of Managers and Executive are first much simpler and second more accurate than traditional planning tools.

In this article, I intend to present the fundamentals, the roles, the processes, the rituals and the values that I believe a team would need to embrace to achieve success down the line in Agile Software Development Management - Product Management, Team Management and Project Management - with the ultimate goal of making planning and forecasting as simple and efficient as it can be.
All of this is a reflection of the tools, principles and practices we have embraced or are introducing in my current company.

I've seen this infographic from Christopher Webb at Deloitte (at the time) recently.
This is the most brilliant infographic I've seen for years.

Christopher Webb presents here a pretty extended set of Agile Practices associated to their respective frameworks. The practices presented are a collection of all Agile practices down the line, related to engineering but also management, product identification, design, operation, etc.

[Click to enlarge]
(Source : Christopher Webb - LAST Conference 2016 Agile Landscape - https://www.slideshare.net/ChrisWebb6/last-conference-2016-agile-landscape-presentation-v1)

I find this infographic brilliant since its the first time I see a "one ring to rule them all" view of what I consider should be the practices towards scaling Agility at the level of the whole IT Organization.

Very often, when we think of Agility, we limit our consideration to solely the Software Build Process.
But Agility is more than that. And I believe an Agile corporation should embrace also Agile Design, Agile Operations and Agile Management.
This infographic does a great job in presenting how these frameworks enrich and complements each others towards scaling Agility at the level of the whole IT Organization.

To be honest there are even many more frameworks that those indicated on this infographic and Chris Webb is presenting some additional - reaching 43 in total - in his presentation.
But I believe he did a great job in presenting the most essential ones and presenting how these practices, principles and framework work together to achieve the ultimate goal of every corporation: skyrocketing employee productivity and happiness, maximizing customer satisfaction and blowing operational efficiency up.

Now I would want to present why I think considering Agility down the line in each and every aspect around the engineering team and how these frameworks completing each other are important.

A few years ago, I worked intensively on a pet project: AirXCell (long gone ...)
What was at first some framework and tool I had to write to work on my Master Thesis dedicated to Quantitative Research in finance, became after a few months somewhat my most essential focus in life.
Initially it was really intended to be only a tool providing me with a way to have a Graphical User Interface on top of all these smart calculations I was doing in R. After my master thesis, I surprised myself to continue to work on it, improving it a little here and a little there. I kept on doing that until the moment I figured I was dedicated several hours to it every day after my day job.
Pretty soon, I figured I was really holding an interesting software and I became convinced I could make something out of it and eventually, why not, start a company.

And of course I did it all wrong.

Instead of finding out first if there was a need and a market for it, and then what should I really build to answer this need, I spent hours every day and most of my week-ends developing it further towards what I was convinced was the minimum set of feature it should hold before I actually try to meet some potential customers to tell them about it.
So I did that for more than a year and a half until I came close to burn-out and send it all to hell.

Now the project hasn't evolve for three years. The thing is that I just don't want to hear about it anymore. I burnt myself and I am just disgusted about it. Honestly it is pretty likely that at the time of reading this article, the link above is not even reachable anymore.
When I think of the amount of time I invested wasted in it, and the fact that even now, three years after, I still just don't want to hear anything about this project anymore, I feel so ashamed. Ashamed that I didn't take a leap backwards, read a few books about startup creation, and maybe, who knows, discover The Lean Startup movement before.
Even now, I still never met any potential customer, any market representative. Even worst: I'm still pretty convinced that there is a need and a market for such a tool. But I'll never know for sure.

Such stories, and even worst, stories of startups burning millions of dollars for nothing in the end, happen every day, still today.

Some years ago, Eric Ries, Steve Blank and others initiated The Lean Startup movement. The Lean Startup is a movement, an inspiration, a set of principles and practices that any entrepreneur initiating a startup would be well advised to follow.
Projecting myself into it, I think that if I had read Ries' book before, or even better Blank's book, I would maybe own my own company today, around AirXCell or another product, instead of being disgusted and honestly not considering it for the near future.
In addition to giving a pretty important set of principles when it comes to creating and running a startup, The Lean Startup also implies an extended set of Engineering practices, especially software engineering practices.

This article focuses on presenting and detailing these Software Engineering Practices from the Lean Startup Movement since, in the end, I believe they can benefit from any kind company, from initiating startup to well established companies with Software Development Activities.
By Software Engineering practices, I mean software development practices of course but not only. Engineering is also about analyzing the features to be implemented, understanding the customer need and building a successful product, not just writing code.

So ... I've read a lot of things recently on DevOps, a lot of very interesting things ... and, unfortunately, some pretty stupid as well. It seems a lot of people are increasingly considering that DevOps is resumed to mastering chef, puppet or docker containers. This really bothers me. DevOps is so much more than any tool such as puppet or docker.

This could even make me angry. DevOps seems to me so important. I've spent 15 years working in the engineering business for very big institutions, mostly big financial institutions. DevOps is a very key methodology bringing principles and practices that address precisely the biggest problem, the saddest factor of failure of software development projects in such institutions : the wall of confusion between developers and operators.

Don't get me wrong, in most of these big institutions being still far from a large and sound adoption of an Agile Development Methodology beyond some XP practices, there are many other reasons explaining the failure or slippage of software development projects.
But the wall of confusion is by far, in my opinion, the most frustrating, time consuming, and, well, quite stupid, problem they are facing.

So yeah... Instead of getting angry I figured I'd rather present here in a concrete and as precise as possible article what DevOps is and what it brings. Long story short, DevOps is not a set of tools. DevOps is a methodology proposing a set of principles and practices, period. The tools, or rather the toolchain - since the collection of tools supporting these practices can be quite extended - are only intended to support the practices.
In the end, these tools don't matter. The DevOps toolchains are today very different than they were two years ago and will be very different in two years. Again, this doesn't matter. What matters is a sound understanding of the principles and practices.

Presenting a specific toolchain is not the scope of this article, I won't mention any. There are many articles out there focusing on DevOps toolchains. I want here to take a leap backwards and present the principles and practices, their fundamental purpose since, in the end, this is what seems most important to me.

DevOps is a methodology capturing the practices adopted from the very start by the web giants who had a unique opportunity as well as a strong requirement to invent new ways of working due to the very nature of their business: the need to evolve their systems at an unprecedented pace as well as extend them and their business sometimes on a daily basis.
While DevOps makes obviously a critical sense for startups, I believe that the big corporations with large and old-fashioned IT departments are actually the ones that can benefit the most from adopting these principles and practices. I will try to explain why and how in this article.

After almost two years as Head of R&D in my current company, I believe I succeeded in bringing Agility to Software Development here by mixing what I think makes most sense out of eXtreme Programing, Scrum, Kanban, DevOps practices, Lean Startup practices, etc.
I am strong advocate of Agility at every level and all the related practices as a whole, with a clear understanding of what can be their benefits. Leveraging on the initial practices already in place to transform the development team here into a state of the art Agile team has been - and still is - one of my most important initial objectives.
I gave myself two years initially to bring this transformation to the Software Development here. After 18 months, I believe we're almost at the end of the road and its a good time to take a step back and analyze the situation, trying to clarify what we do, how we do it, and more importantly why we do it.

As a matter of fact, we are working in a full Agile way in the Software Development Team here and we are having not only quite a great success with it but also a lot of pleasure.
I want to share here our development methodology, the philosophy and concepts behind it, the practices we have put in place as well as the tools we are using in a detailed and precise way.
I hope and believe our lessons learned can benefit others.
As a sidenote, and to be perfectly honest, while we may not be 100% already there in regards to some of the things I am presenting in this article, at least we have identified the gap and we're moving forward. At the end of the day, this is what matters the most to me.

This article presents all the concepts and practices regarding Agile Software Development that we have put (or are putting) in place in my current company and gives our secrete recipe which makes us successful, with both a great productivity / short lead time on one side and great pleasure and efficiency in our every day activities on the other side.