niceideas.ch
Technological Thoughts by Jerome Kehrli
https://www.niceideas.ch/roller2/badtrash/feed/entries/atom
2024-03-13T10:05:04-04:00
Apache Roller
https://www.niceideas.ch/roller2/badtrash/entry/artificial-intelligence-and-fraud-prevention
Artificial Intelligence and fraud prevention with Netguardians' CTO, Jérôme Kehrli
Jerome Kehrli
2022-01-18T06:11:45-05:00
2022-01-18T06:11:45-05:00
<p>
I spoke to to <a href="https://www.linkedin.com/in/rudolf-falat/">Rudolf Falat</a>, founder of the <a href="https://www.linkedin.com/in/rudolf-falat/">Voice of FinTech</a> podcast about leveraging AI in anti-fraud prevention and cybersecurity.
</p>
<p>
The Podcast is <a href="https://www.voiceoffintech.com/episodes/netguardians">available here</a> and can be listened to directly from hereunder:
</p>
<iframe width="100%" height="180" frameborder="no" scrolling="no" seamless="" src="https://share.transistor.fm/e/41d2933f/dark"></iframe>
<p>
Happy listening !
</p>
<p>
<b>The full transcript is available hereunder</b>
</p>
<p>
I spoke to to <a href="https://www.linkedin.com/in/rudolf-falat/">Rudolf Falat</a>, founder of the <a href="https://www.linkedin.com/in/rudolf-falat/">Voice of FinTech</a> podcast about leveraging AI in anti-fraud prevention and cybersecurity.
</p>
<p>
The Podcast is <a href="https://www.voiceoffintech.com/episodes/netguardians">available here</a> and can be listened to directly from hereunder:
</p>
<iframe width="100%" height="180" frameborder="no" scrolling="no" seamless="" src="https://share.transistor.fm/e/41d2933f/dark"></iframe>
<p>
Happy listening !
</p>
<p>
<b>The full transcript is available hereunder</b>
</p>
<p>
<b><i>Jerome, can you introduce yourself? How did you get to do what you do today?</i></b>
</p>
<p>
I'm swiss, 43 years old and a proud father of 3 boys.
<br>
I guess I'm first and foremost a passionate software engineer and computer scientist. I remember putting my hands on my first computer - a commodore 128 - when I was 12 years old and knowing very well at that very moment that that would be my carrier.
<br>
I'm passionate by technology, artificial intelligence, programming, etc. for nearly 30 years now. I made all my career in financial institutions and fintechs and I wouldn't see myself working in another business nowadays. Financial institutions and financial markets form a very interesting domain of application in computer science due to the complexity of these systems and the wide range of concerns to be addressed, from real-time computing to highly mathematical applications.
<br>
I guess that my role today as CTO of NetGuardians is kind of a natural evolution in my career.
</p>
<p>
NetGuardians is a Swiss based software editor developing a Big Data Analytics platform that we package and deploy in financial institution mostly today to detect fraudulent activities and prevent fraudulent transactions.
</p>
<p>
<b><i>Why have you decided to join a start-up (or a scale-up) like NetGuardians?</i></b>
</p>
<p>
Before the NetGuardians co-founders reached out to me I was a consultant for a few years, working mostly for the major european banking institutions.
<br>
I really liked the job at the time, mostly the possibility to jump quickly from one topic to another, one customer to another.
</p>
<p>
But I did miss the product culture very much. As a consultant I was guiding other teams or leaders in adopting technologies, designing information systems, driving innovation projects, etc. But I was missing the deep implication and engagement that you get when you create a product from A to Z and sell it. Developing a software is the closest you can get in day job to having an actual child ;-)
</p>
<p>
So when NetGuardians pivoted to banking fraud fighting 7 years ago, the co-founders were looking for someone to lead the product research and development department, someone with a strong technical background and an extensive experience in finance.
<br>
Switzerland is a small country so they have been directed by some common relationship we had in their advisory board to my profile. So we met and they told me their story and shared their vision with me and I decided that I wanted to be part of it.
<br>
And today, 7 years after this first encounter I guess this company and this product are just as much my children as they're theirs.
</p>
<p>
<b><i>When will we finally have truly intelligent AI working on fraud prevention in banking ?</i></b>
</p>
<p>
Now that's a good question.
<br>
It's hard to answer since intelligent AI would need to be defined or precised. So first I would want to distinguish strong AI vs. weak AI and then share my perspective on what would be a truly intelligent AI.
</p>
<p>
If we qualify as a strong artificial intelligence, a software program able to contextualize, to show sensitivity, to show creativity or to exceed it's programming scope, then we don't have today the slightest trail of a proof that we'd be able one day to create such an program. This is downright science fiction. There is nothing in the real world anywhere close to the beginning of it.
<br>
The thing is that Artificial Intelligence is generating a lot of fantasy in the public's mind and I guess that the fact that we have given some of these algorithms names such as neural network is not helping in this regards. If we had given to neural networks the technical names they should have, such as largely convoluted and iterative statistical matrix model, I'm sure they wouldn't generate the same level of fantasy in people's minds.
<br>
Anyways.
</p>
<p>
Then if we qualify as a weak AI a software program able to optimize a mathematical function, solve a classification problem, or take a decision based on input data, then the progresses today are tremendous and new applications and solutions pop up nearly every week.
<br>
This technology evolves at a very fast pace and today's AI programs are a collection of sometimes hundreds of different algorithms working together to solve an analytical problem, such as driving a car autonomously for instance which is amazing.
</p>
<p>
Now when it comes to true intelligence, I strongly believe that the only true, actual intelligence is in the mind of the people developing these systems, not the machine, never the machine.
<br>
And then again, the progresses today are tremendous and essentially around 2 dimensions: the complexity of the individual Machine Learning algorithms and the number of these algorithms deployed together and working in conjunction in Artificial Intelligence Systems
</p>
<p>
And what we do at NetGuardians is a good illustration of all this evolution.
<br>
When we started in 2016, we were using one or two different methods to infer good features on events we were monitoring, mostly EBanking activities and financial transactions, as well as a single Supervised learning algorithm. Today we use a combination of multiple dozens of different unsupervised and supervised techniques all working together and each one of them focusing on a specific perspective, such as the timing of events, their frequency, their location, the destination of the funds, etc. or a specific step in the risk scoring process.
</p>
<p>
So yeah, again the true intelligence is in the mind of the guys developing these systems, not in the software.
</p>
<p>
<b><i>How good is anti-fraud AI today? What kind of AI are we talking about?</i></b>
</p>
<p>
Anti-fraud systems today form a very peculiar and passionate domain of application for artificial Intelligence. The nature of the problem to be solved makes it very specific.
<br>
Think of it, while some payments channel such as credit cards for instance experience a plethora of frauds, some other channels such as digital banking payments have typically only a few frauds for a million transactions a day.
</p>
<p>
Most sophisticated classification machine learning algorithms we have today perform very poorly on such datasets. They work well when the data is very much balanced between the positive and other populations.
<br>
As an example, every engineer knows today how to train a neural network to recognize pictures of cats, for instance, by feeding it with thousands of pictures of cats and thousands of pictures of other animals and other objects. Now if you try to train a neural network to recognize cats with only 6 pictures of cats and millions of random pictures of other animals and objects, the next picture of a cat you will present to the neural network will be classified as anything, such an elephant, right ? But there's no way an algorithm trained this way understands how to recognize cats.
</p>
<p>
And we're in the same situation. The very unbalanced nature of the data we're playing with makes all simple approaches simply irrelevant. So we have to do fairly complex stuff.
</p>
<p>
Our state of the art approach today at NetGuardians is a combination of multiple fairly evolved techniques and approaches working together.
<br>
I don't want to go to much into technical details but I would mention three categories of techniques we're using.
</p>
<p>
First, unsupervised learning techniques for anomaly detection ... with a wide range of different algorithms, from simple statistical or Poisson scoring down to clustering and peer group analysis. At the end of the day, fraudulent activities and transactions are always part of the set of anomalies.
<br>
Then, supervised learning techniques ... with a lot of different models being required from classification algorithms to risk scoring techniques, to distinguish between legitimate anomalies and highly potentially fraudulent transactions.
<br>
Last but not least, active learning and other supervision techniques to monitor the feedback we get from banking business users reviewing the hits, the activities or transactions being blocked in real-time by the system, etc.
<br>
And that is absolutely key because at the end of the day, our algorithms learn a lot from the feedback of these business people and they can only be as good as this feedback is. So supervising this quality is an essential concern.
</p>
<p>
So yeah, again, our approach at NetGuardians today is a combination of dozens of such techniques and algorithms deployed together to detect and block suspicious activities and transactions in real-time. And It works pretty good !
</p>
<p>
Another thing to consider: every transaction we block is investigated by a business expert within the bank who takes the eventual decision.
<br>
In a sense, we're not replacing the human decision process, but we're enhancing it. We give bankers a chance to review potentially or likely fraudulent activities before the funds leave the bank. And this is called Augmented Intelligence.
</p>
<p>
<b><i>Who are your target customers?</i></b>
</p>
<p>
At NetGuardians, we're working only for financial institutions. Our typical customers are Tier 1 and Tier 2 banks - big banks to medium size banks - where we detect fraudulent activities in a holistic fashion, fraudulent transactions and activities on digital channels just as much as internal fraud or scams.
<br>
In terms of types of financial institutions, we work just as much with massive retail banking institutions in Asia than private banking institutions in Switzerland.
<br>
Our key markets are Europe, our home market, Africa and Asia Pacific.
<br>
We support on premise deployment for Tier 1 banks who have a strong will to keep everything in house and onboard smaller institutions on one of our SaaS - Software as a Service - platforms on the cloud.
</p>
<p>
<b><i>How do you make money?</i></b>
</p>
<p>
Our customers pay an annual recurring licensing fee calculated from two metrics, their Asset under Management and their volume of financial transactions.
<br>
We bill delivery and integration costs when we integrate the solution ourselves but we intend to get away from this activity as much as possible and rely increasingly on local partners for integration.
<br>
We are not very much interested to sell services and would want to focus in the future on selling licenses only but that would require us to reach a critical mass and we're not there yet.But it's an ultimate objective for us, moving a way from a being both a product and service company today and turning to an only product company.
</p>
<p>
<b><i>Where are you based?</i></b>
</p>
<p>
We're a company founded in Switzerland and we are still today headquartered in Yverdon-les-bains, a small town north to Lausanne. We have offices in Nairobi where we manage our operations in Africa and in Singapore where we handle our Asian activities. We also have a commercial office in London and a near-shore development center in Warsaw .
</p>
<p>
<b><i>Where are you on your journey in terms of product development, geographic reach, funding, hiring? Any numbers you can share?</i></b>
</p>
<p>
We have today a very solid technology and product for the banking fraud detection and prevention problem. And in the short term we intend to leverage on our technology to extend our product to other financial crime use cases.
</p>
<p>
There are many different concerns in Financial crime fighting in banking institutions, Fraud detection is an essential one of them of course but then there's also AML - Anti Money Laundering - Transaction Monitoring, KYC - Know your customer and of course customer and transaction screening.
<br>
KYC and screening require very different technologies than the ones we've built so they're not in our short term focus. But AML Transaction Monitoring is very close from what we do on Fraud, just the perspective of the analytics is somewhat different.
<br>
Finding fraud is a lot about understanding where the money goes while AML is a lot about understanding where the money comes from, but from a technical standpoint it's really similar.
</p>
<p>
So long story short, we intend by the end of next year to extend our solution to state of the art AML transaction Monitoring leveraging on our technology. Eventually, over the next years, we intend to build a complete financial crime package by integrating third party solutions for KYC and screening.
</p>
<p>
In terms of geographic reach, we are today strong in Europe - our local market - and Africa. But we're really only building Asia. This is where we are investing our effort today and in the coming year to build a strong sales team, identifying and leveraging on the right partners, scaling the delivery team and eventually, hopefully, become a major player in Asia as well.
<br>
Interestingly, we have no intent to actively address the US market today aside of a few opportunistic leads through some of our partners.
</p>
<p>
To give you a few figures, we're today a 100 FTEs company and we have a little less than 80 financial institutions as customers.
</p>
<p>
Now regarding funding, I can't tell you much actually. We have raised roughly 30 million USD so far and we are in the process of challenging and building the next investment round. Building the proper structure in APAC to emerge as a major player here is not something on our reach today. We need support from investors to build this and we're working on that today.
</p>
<p>
<b><i>What are the next steps for you next year and beyond? Customers, incumbents as partners, investors?</i></b>
</p>
<p>
We intend to develop significantly in our three key markets, Europe, Africa and Asia. We are in the process of finalizing recruitment of the key people - such as regional sales director, etc. - who will be instrumental in driving our growth in these regions.
<br>
And as I said before, we need support from investors to build the proper structure in APAC, based in Singapore.
</p>
<p>
In terms of partnership, we have today very good partners in the core banking systems and banking package providers field where our strategy is to bundle our fraud detection engine with their Core Banking Package offering.
<br>
We are now in the process of looking for integration partners in the different regions to support our scaling and incrementally disengage our own people from delivery.
</p>
<p>
In terms of investment, we would also expect the next round to support our extension to completeness of offering in AML and more generally financial crime fighting as well as complete our transition to the cloud as lead deployment channel. We have still quite a path ahead of us to provider tier 1 banking institutions with a state of the art hybrid cloud approach.
<br>
A lot of tier 1 banking institution would sign up for a cloud deployment of NetGuardians if and only if we can provide them with means to guarantee that the confidential data remains within the bank information system boundaries. And the technology for that is called hybrid cloud which would be quite an evolution from what we do today.
</p>
<p>
<b><i>Where can interested parties reach you?</i></b>
</p>
<p>
Well, I guess the best way to get in touch with us is through our web site, <a href="https://netguardians.ch/">www.netguardians.ch</a>.
<br>
And the best way to contact me would be on linkedin I guess <a href="https://www.linkedin.com/in/jeromekehrli/">jerome KEHRLI</a>.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/modern-information-system-architectures
Modern Information System Architectures
Jerome Kehrli
2021-12-13T06:04:45-05:00
2021-12-13T06:18:06-05:00
<p>
For forty years we have been building <i>Information Systems</i> in corporations in the same way, with the same architecture, with very little innovations and changes in paradigms:
</p>
<ul>
<li>On one side the <b>Operational Information System</b> which sustains day-to-day operations and business activities. On the Operational Information Systems, the <i>3-tiers</i> architecture and the relational database model (RDBMS - Relational Database Management System / SQL) have ruled for nearly 40 years.
</li>
<li>On the other side the <b>Decision Support Information System</b> - or <i>Business Intelligence</i> or <b>Analytical Information System</b> - where the <i>Data Warehouse</i> architecture pattern has ruled for 30 years.
</li>
</ul>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/1d146989-ed7b-4480-bd8c-651f4085375b">
<img class="centered" style="width: 800px;" alt="legacy Information System Architecture" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/1d146989-ed7b-4480-bd8c-651f4085375b" />
</a>
<div class="centered">
<i>legacy / Information Systems Architecture for 40 years</i>
</div>
</div>
<br>
<p>
Of course the technologies involved in building these systems have evolved in all these decades, in the 80s COBOL on IBM hosts used to rule the Information Systems world whereas Java emerged quickly as a standard in the 2000s, etc.
<br>
But while the technologies used in building these information systems evolved fast, their architecture in the other hand, the way we design and build them, didn't change at all. The relational model ruled for 40 years along the 3-tiers model in the Operational world and in the analytical world, the Data Warehouse pattern was the only way to go for decades.
</p>
<p>
The relational model is interesting and has been helpful for many decades. its fundamental objective is to optimize storage space by ensuring an entity is stored only once (3rd normal form / normalization). It comes from a time when storage was very expensive.
<br>
But then, by imposing normalization and ACID transactions, it prevents horizontal scalability by design. An Oracle database for instance is designed to run on a single machine, it simply can't implement relational references and ACID transactions on a cluster of nodes.
<br>
Today storage is everything but expensive but Information Systems still have to deal with RDBMS limitations mostly because ... that's the only way we used to know.
</p>
<p>
On the Decision Support Information System (BI / Analytical System), the situation is even worst. in Data warehouses, data is <i>pushed</i> along the way and transformed, one step at a time, first in a staging database, then in the Data Warehouse Database and finally in Data Marts, highly specialized towards specific use cases.
<br>
For a long time we didn't have much of a choice since implementing such analytics in a <i>pull</i> way (data lake pattern) was impossible, we simply didn't have the proper technology. The only way to support high volumes of data was to <i>push</i> daily increments through these complex transformation steps every night, when the workload on the system is lower.
<br>
The problem with this <i>push</i> approach is that it's utmost inflexible. One can't change his mind along the way and quickly come up with a new type of data. Working with daily increments would require waiting 6 months to have a 6 months history. Not to mention that the whole process is amazingly costly to develop, maintain and operate.
</p>
<p>
So for a long time, RDBMSes and Data Warehouses were all we had.
</p>
<p>
It took the Internet revolution and the web giants facing limits of these traditional architectures for finally something different to be considered. The <b>Big Data revolution</b> has been the cornerstone of all the evolutions in Information System architecture we have been witnessing over the last 15 years.
</p>
<p>
The latest evolution in this software architecture evolution (or revolution) would be micro-services, where finally all the benefits that were originally really fit to the analytical information system evolution finally end up overflowing to the operational information system.
<br>
Where Big Data was originally a lot about scaling the computing along with the data topology - bringing the code to where the data is (data tier revolution) - we're today scaling everything, from individual components requiring heavy processing to message queues, etc.
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/f74719c9-f6f5-4f48-9bf2-ae584c02525f">
<img class="centered" style="width: 800px;" alt="Microservices Architecture" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/f74719c9-f6f5-4f48-9bf2-ae584c02525f" />
</a>
<div class="centered">
<i>Example of modern IS architecture: Microservices</i>
</div>
</div>
<br>
<p>
In this article, I would want to present and discuss how Information System architectures evolved from the universal 3 tiers (operational) / Data Warehouse (analytical) approach to the Micro-services architecture, covering Hadoop, NoSQL, Data Lakes, Lambda architecture, etc. and introducing all the fundamental concepts along the way.
</p>
<p>
For forty years we have been building <i>Information Systems</i> in corporations in the same way, with the same architecture, with very little innovations and changes in paradigms:
</p>
<ul>
<li>On one side the <b>Operational Information System</b> which sustains day-to-day operations and business activities. On the Operational Information Systems, the <i>3-tiers</i> architecture and the relational database model (RDBMS - Relational Database Management System / SQL) have ruled for nearly 40 years.
</li>
<li>On the other side the <b>Decision Support Information System</b> - or <i>Business Intelligence</i> or <b>Analytical Information System</b> - where the <i>Data Warehouse</i> architecture pattern has ruled for 30 years.
</li>
</ul>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/1d146989-ed7b-4480-bd8c-651f4085375b">
<img class="centered" style="width: 800px;" alt="legacy Information System Architecture" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/1d146989-ed7b-4480-bd8c-651f4085375b" />
</a>
<div class="centered">
<i>legacy / Information Systems Architecture for 40 years</i>
</div>
</div>
<br>
<p>
Of course the technologies involved in building these systems have evolved in all these decades, in the 80s COBOL on IBM hosts used to rule the Information Systems world whereas Java emerged quickly as a standard in the 2000s, etc.
<br>
But while the technologies used in building these information systems evolved fast, their architecture in the other hand, the way we design and build them, didn't change at all. The relational model ruled for 40 years along the 3-tiers model in the Operational world and in the analytical world, the Data Warehouse pattern was the only way to go for decades.
</p>
<p>
The relational model is interesting and has been helpful for many decades. its fundamental objective is to optimize storage space by ensuring an entity is stored only once (3rd normal form / normalization). It comes from a time when storage was very expensive.
<br>
But then, by imposing normalization and ACID transactions, it prevents horizontal scalability by design. An Oracle database for instance is designed to run on a single machine, it simply can't implement relational references and ACID transactions on a cluster of nodes.
<br>
Today storage is everything but expensive but Information Systems still have to deal with RDBMS limitations mostly because ... that's the only way we used to know.
</p>
<p>
On the Decision Support Information System (BI / Analytical System), the situation is even worst. in Data warehouses, data is <i>pushed</i> along the way and transformed, one step at a time, first in a staging database, then in the Data Warehouse Database and finally in Data Marts, highly specialized towards specific use cases.
<br>
For a long time we didn't have much of a choice since implementing such analytics in a <i>pull</i> way (data lake pattern) was impossible, we simply didn't have the proper technology. The only way to support high volumes of data was to <i>push</i> daily increments through these complex transformation steps every night, when the workload on the system is lower.
<br>
The problem with this <i>push</i> approach is that it's utmost inflexible. One can't change his mind along the way and quickly come up with a new type of data. Working with daily increments would require waiting 6 months to have a 6 months history. Not to mention that the whole process is amazingly costly to develop, maintain and operate.
</p>
<p>
So for a long time, RDBMSes and Data Warehouses were all we had.
</p>
<p>
It took the Internet revolution and the web giants facing limits of these traditional architectures for finally something different to be considered. The <b>Big Data revolution</b> has been the cornerstone of all the evolutions in Information System architecture we have been witnessing over the last 15 years.
</p>
<p>
The latest evolution in this software architecture evolution (or revolution) would be micro-services, where finally all the benefits that were originally really fit to the analytical information system evolution finally end up overflowing to the operational information system.
<br>
Where Big Data was originally a lot about scaling the computing along with the data topology - bringing the code to where the data is (data tier revolution) - we're today scaling everything, from individual components requiring heavy processing to message queues, etc.
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/f74719c9-f6f5-4f48-9bf2-ae584c02525f">
<img class="centered" style="width: 800px;" alt="Microservices Architecture" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/f74719c9-f6f5-4f48-9bf2-ae584c02525f" />
</a>
<div class="centered">
<i>Example of modern IS architecture: Microservices</i>
</div>
</div>
<br>
<p>
In this article, I would want to present and discuss how Information System architectures evolved from the universal 3 tiers (operational) / Data Warehouse (analytical) approach to the Micro-services architecture, covering Hadoop, NoSQL, Data Lakes, Lambda architecture, etc. and introducing all the fundamental concepts along the way.
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. Introduction</a></li>
<li><a href="#sec2">2. The Web giants and Big Data</a>
<ul>
<li><a href="#sec21">2.1 The Era of Power</a></li>
<li><a href="#sec22">2.2 The Web Giants</a></li>
<li><a href="#sec23">2.3 Data Deluge</a></li>
<li><a href="#sec24">2.4 The Moore Law</a></li>
<li><a href="#sec25">2.5 The Death of the Moore Law</a></li>
<li><a href="#sec26">2.6 Fundamentals of Big Data - the Web giants new paradigms</a></li>
</ul>
</li>
<li><a href="#sec3">3. The CAP Theorem</a>
<ul>
<li><a href="#sec31">3.1 The origins of NoSQL</a>
<ul>
<li><a href="#sec311">3.1.1 Flat files as data store</a></li>
<li><a href="#sec312">3.1.2 RDBMS and the relational model</a></li>
<li><a href="#sec313">3.1.3 criticism of the relational model</a></li>
</ul>
</li>
<li><a href="#sec32">3.2 Horizontal scalability</a>
<ul>
<li><a href="#sec321">3.2.1 Scaling up</a></li>
<li><a href="#sec322">3.2.2 Scaling out</a></li>
</ul>
</li>
<li><a href="#sec33">3.3 Data Distribution</a></li>
<li><a href="#sec34">3.4 Properties of a distributed system</a>
<ul>
<li><a href="#sec341">3.4.1 Eventual consistency</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#sec4">4. NoSQL / NewSQL</a>
<ul>
<li><a href="#sec41">4.1 NoSQL</a></li>
<li><a href="#sec42">4.2 NewSQL</a></li>
</ul>
</li>
<li><a href="#sec5">5. Hadoop and Data Lakes</a>
<ul>
<li><a href="#sec51">5.1 What is Hadoop ?</a></li>
<li><a href="#sec52">5.2 Hadoop Overview</a></li>
<li><a href="#sec53">5.3 Hadoop Architecture</a></li>
<li><a href="#sec54">5.4 The DataLake Architecture pattern</a></li>
</ul>
</li>
<li><a href="#sec6">6. Streaming Architectures</a>
<ul>
<li><a href="#sec61">6.1 Complex Event Processing</a></li>
<li><a href="#sec62">6.2 Lambda Architecture</a></li>
<li><a href="#sec63">6.3 Kappa Architecture</a></li>
</ul>
</li>
<li><a href="#sec7">7. Big Data 2.0</a>
<ul>
<li><a href="#sec71">7.1 Alternatives to Hadoop </a></li>
<li><a href="#sec72">7.2 Kubernetes</a></li>
</ul>
</li>
<li><a href="#sec8">8. Micro-services</a>
<ul>
<li><a href="#sec81">8.1. Micro-services discussion</a></li>
</ul>
</li>
<li><a href="#sec9">9. Conclusion</a></li>
</ul>
<a name="sec1"></a>
<h2>1. Introduction </h2>
<p>
As stated in the summary above, the way we build information systems really didn't evolve in so many decades. The technologies used underneath have evolved of course, a long way from COBOL to Java and Angular, but the architectures in use - the <i>3-tiers model</i> on the operational information system and the <i>data warehouse pattern</i> on the decision support system (a.k.a analytics system) - haven't evolve in more than 30 years.
<br>
The <i>Software Architecture</i> is defined as the set of <b>principal design decision</b> about the system. Software architecture is kind of the blueprint for the system's construction and evolution. Design decisions encompass the following aspects of the system under development: Structure, Behaviour, Interactions, Non-functional properties. (Taylor 2010)
<br>
And then again, the technologies under the hood, from the operating systems to the User Interfaces through the programming languages, have evolved drastically. We all remember 3270 green-on-black terminal screens and can only consider the terrific evolution to the fancy HTML5/bootstrap screens we see today.
<br>
But the design of the Information system components, their interactions and the technical processes in between didn't evolve at all!
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/1d146989-ed7b-4480-bd8c-651f4085375b">
<img class="centered" style="width: 800px;" alt="legacy Information System Architecture" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/1d146989-ed7b-4480-bd8c-651f4085375b" />
</a>
<div class="centered">
<i>Information Systems Architecture for 40 years</i>
</div>
</div>
<br>
<p>
I find it amazing to consider that if you put COBOL, 3270 and a few terms like that on this schema instead of the web elements you literally get what would have been the high-level architecture schema 40 years ago.
<br>
As stated above, RDBMS - Relational Database Management Systems - have a lot of limits and some benefits, namely the standardized querying language - SQL - and the optimization of storage space. But in today's digital world, the benefits don't stand up to the drawbacks, mostly the impossibility to scale.
<br>
The Data Warehouse pattern in use for 30 years on the analytical Information System is also a poor fit for today's pace of development of digital services. It is much too inflexible not to mention the cost in developing and maintaining it.
</p>
<p>
It took the web giants to face the limits of these current architecture paradigms and invent new ways of building information systems to finally see some evolutions in the way we are building them in corporations. The first evolutions came to the analytics system side with Big Data technologies and overflowed later to the operationnal IS side with NoSQL, streaming architectures and eventually micro-services.
</p>
<p>
In this article I want to present these evolutions from an historical perspective. We'll start with the Web giants and the Big Data revolution, cover NoSQL and Hadoop, run through Lambda and Kappa architectures, and end up discussing Kubernetes and Micro-services.
</p>
<a name="sec2"></a>
<h2>2. The Web giants and Big Data</h2>
<p>
The web giants have been the first to face the limits of traditional architectures in an unacceptable way. Can you imagine google running their Search Engine on an IBM mainframe ? Can you imagine what that would be for a machine and how much money (licensing fees) they would need to leave to IBM every year for running such a host ?
<br>
Can you imagine Amazon running their online retail business on an Oracle database with hundreds of millions of users connected and querying the DB at any time ? Can you imagine the price of a computer that would be able to support such volumes of data and concurrent requests ?
</p>
<p>
The Web giants had to invent both <i>new data storage technologies</i> and <i>programming paradigms</i> to run their business and support their volume of activities.
<br>
But let's start with the beginning.
</p>
<a name="sec21"></a>
<h3>2.1 The Era of Power</h3>
<p>
As an prequel to introducing Big Data, let's have a look at these both situations:
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/3acf2fe9-36ae-4b3b-a26c-81ba64780676">
<img class="centered" style="width: 800px;" alt="The Era of Power" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/3acf2fe9-36ae-4b3b-a26c-81ba64780676" />
</a>
<div class="centered">
<i>The Era of Power</i>
<br>
Source: <a href="https://pages.experts-exchange.com/processing-power-compared ">https://pages.experts-exchange.com/processing-power-compared </a>
</div>
</div>
<br>
<p>
These two computers are separated by only 30 years of technological evolutions.
<br>
The computer on the left is a Cray II. When it came out in 1985, it was a revolution since it was the fastest machine in the world, the first multi-processor computer from Seymour Cray and included many unique technological evolutions.
<br>
The computer on the right is a Samsung S6 Smartphone. It's 30 years younger that the Cray 2.
</p>
<p>
It's 30 years younger only and around 15 times more powerful that the Cray 2. While the later was by far bigger than a human being, the Samsung S6 fits in the palm. The Cray 2 has 4 processors while the S6 packages 8 processors.
<br>
Considering how the hardware technology progressed over one generation is mind-blowing.
</p>
<p>
Another comparison is even more impressive, 50 years before the Samsung S6, a computer has been used to send people to the moon. The S6 is a million times more powerful in terms of raw computing power than that very computer.
<br>
We have today a device so small that it fits in our palm, incredibly powerful, which enables us to be <i>interconnected everywhere, all the time and for every possible need</i>. This is the definition of the <i>digitization</i>
<br>
The smartphones are really an amazing piece of technology, but much more impressive are the apps behind and the services they enable us to use. This leads us to the Web Giants.
</p>
<a name="sec22"></a>
<h3>2.2 The Web Giants</h3>
<p>
The Web giants have been the first to face the limits of traditional architectures and the usual way information systems were built.
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/ea67caee-b68f-48cc-9db3-8ca352e2f9cd">
<img class="centered" style="width: 700px;" alt="The Web Giants" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/ea67caee-b68f-48cc-9db3-8ca352e2f9cd" />
</a>
<div class="centered">
<i>The Web Giants</i>
</div>
</div>
<br>
<p>
And the revolution came from them. They had to find new technical solutions to business-critical challenges such as :
</p>
<ul>
<li><b>Google: </b> Index the whole web, and keep a response time to any request below one second - or how to keep the search free for the user ?</li>
<li><b>Facebook: </b> Interconnect billions of users, display their feeds in near-real-time and understand how they use their product to optimize ads ?</li>
<li><b>Amazon: </b> How to build a product recommendation engine for dozens of millions of customers, on millions of products ?</li>
<li><b>EBAY : </b> How to do a search in ebay auctions, even with misspelling ?</li>
</ul>
<p>
These are just oversimple examples of course and the challenges faced by the web giants go much beyond such simple cases.
<br>
These business challenges are backed by technical challenges such as:
</p>
<ul>
<li>How to invert a square matrix that doesn't fit in memory in a reasonable time ?</li>
<li>How to query a database containing trillions of documents in real-time ?</li>
<li>How to read billions of files of multiple megabytes in a reasonable time ?</li>
<li>etc.</li>
</ul>
<p>
At the end of the day, it all boiled down to finding ways to manage volumes of data bigger by several orders of magnitude than the volumes of data that IT systems were used to manipulate so far.
</p>
<a name="sec23"></a>
<h3>2.3 Data Deluge</h3>
<p>
So the most striking problem they had to solve is getting prepared and ready for the <b>Data Deluge!</b>
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/5666cd1f-0852-4db7-8d56-910d659d8820">
<img class="centered" style="width: 700px;" alt="The Data Deluge" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/5666cd1f-0852-4db7-8d56-910d659d8820" />
</a>
<div class="centered">
<i>Data Deluge!</i>
</div>
</div>
<br>
<p>
Not only do we generate more and more data, but we have today the means and the technology to analyze, exploit and mine it and extract meaningful business insights.
<br>
The data generated by the company’s own systems can be a very interesting source of information regarding customer behaviours, profiles, trends, desires, etc. But also external data, Facebook, twitter logs, etc.
</p>
<a name="sec24"></a>
<h3>2.4 The Moore Law</h3>
<p>
<b>The Moore Law:</b><i>The number of transistors and resistors on a chip doubles every 24 months"</i> (Gordon Moore, 1965)
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/e5e9bb30-1ca4-47f6-bd06-f3403dd38fd2">
<img class="centered" style="width: 850px;" alt="The Moore Law" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/e5e9bb30-1ca4-47f6-bd06-f3403dd38fd2" />
</a>
<div class="centered">
<i>The Moore Law <b>Click to enlarge</b>-</i>
</div>
</div>
<br>
<p>
For a long time, the increasing volume of data to be handled by any given corporation in its Information System was not an issue at all.
<br>
The volume of data increases, the number of user increases, etc. but the processing abilities increases as well, sometimes even more.
<br>
The Moore law was there to cover our ass. The corporation CTO just had to buy a new machine to host the Information System every few years.
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/c568d819-5337-424c-8d3f-f202abe75079">
<img class="centered" style="width: 650px;" alt="IT computing abilities exponential growth" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/c568d819-5337-424c-8d3f-f202abe75079" />
</a>
<div class="centered">
<i>For the 40 years, the IT component capabilties grew exponentially</i>
<br>
Source: <a href="http://radar.oreilly.com/2011/08/building-data-startups.html">http://radar.oreilly.com/2011/08/building-data-startups.html</a>
</div>
</div>
<br>
<p>
This model has hold for a very long time. The cost are going down, the computing capacities are rising, one simply needs to buy a new machine to absorb the load increase.
<br>
This is especially true in the mainframe world. There wasn’t even any need to make the architecture of the systems (COBOL, etc.) evolve for 30 years.
<br>
Even outside the mainframe world. The architecture patterns and styles we are using in the operational IS world haven’t really evolve for the last 30 years. Despite new technologies such as Web, Web 2.0, Java, etc. of course. I’m just speaking about architecture plans and styles.
</p>
<a name="sec25"></a>
<h3>2.5 The Death of the Moore Law</h3>
<p>
But everything has an end.
<br>
Let's consider a fifth dimension, too often left aside when considering the evolutions of computer technologies and hardware architectures: the throughput of the connection between the data on the disk (long term storage) and the memory (i.e. hard drive controllers mostly, but also buses, etc.)
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d37e83aa-67e1-4a8d-a0a0-d737c2964e4a">
<img class="centered" style="width: 850px;" alt="The death of the moore law" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d37e83aa-67e1-4a8d-a0a0-d737c2964e4a" />
</a>
<div class="centered">
<i>The death of the Moore Law</i>
</div>
</div>
<br>
<p>
Issue: the throughput evolution is always lower than the capacity evolution.
</p>
<div class="centering">
<div class="centered">
<b>How read/write more and more data through an always thicker pipe?</b>
</div>
</div>
<br>
<p>
The throughput has become the biggest concern in <b>scaling</b> computer / platform hardware <b>up</b>. It did not progress in terms of efficiency in a way comparable to the four other dimensions.
<br>
We are able to store <b>more and more data</b>, of course, but we are <b>less and less able to manipulate this data</b> efficiently.
<br>
In practice, fetching all the disk data on a computation machine to fit it in RAM to process it is becoming more and more difficult.
</p>
<a name="sec26"></a>
<h3>2.6 Fundamentals of Big Data - the Web giants new paradigms</h3>
<p>
In order to workaround the limits of traditional architectures, the web giants invented new architecture paradigms and new ways of building information systems by leveraging on three fundamental ideas:
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/3fbee390-21ba-4be0-816b-5d01d8bae46f">
<img class="centered" style="width: 750px;" alt="Fundamentals of Big Data" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/3fbee390-21ba-4be0-816b-5d01d8bae46f" />
</a>
<div class="centered">
<i>Fundamentals of Big Data - the Web giants new paradigms</i>
</div>
</div>
<br>
<p>
In details:
</p>
<ol>
<li>
<b>Key idea 1 : distribution</b> - Since its impossible to fit the data in the RAM of one single machine, split it and distribute it on as many different machines as are required.
<br>
Distribution means <i>partitioning</i> the dataset - also called <i>sharding</i> it sometimes - but also always <i>replicate</i> the partitions or shards. We'll see exactly why and how later.
</li>
<li>
<b>Key idea 2 : Horizontal scalability</b> - Just as we split the data, let's split the computation and distribute it on as many nodes as are required to support the workload, even if it means multiple datacenters.
</li>
<li>
<b>Key idea 3 : Data tier revolution</b> - So we distribute both the data on a cluster of computers - or nodes - and the processing as well. We end up using the data nodes as processing nodes. This is the data tier revolution, which is in complete opposition to what was usually done so far in traditional architectures: fetching the required data to the place where the computation occurs.
<br>
But it goes further than that.
<br>
Most of the time we end up distributing different types or categories of data. Every time a specific business process needs to compute something out of a specific piece of all this data, it's crucial to ensure the processing will happen on the very nodes where this specific piece of data is located. This is called <b>co-local processing</b> or <i>data locality optimization</i>.
</li>
</ol>
<p>
As a summary, the web giants have designed new architectures and programming paradigms where distributing the data and the processing (ideally in a co-local way) on a cluster of nodes was the most fundamental principle.
</p>
<a name="sec3"></a>
<h2>3. The CAP Theorem</h2>
<p>
But moving from a mainframe world - where everything is on the same computer and the data to compute always fits in the memory of that computer - to a distributed system world most definitely has benefits, but it also has some consequences. And that's the topic of this chapter.
</p>
<a name="sec31"></a>
<h3>3.1 The origins of NoSQL</h3>
<p>
Let's start with a bit of history.
</p>
<a name="sec311"></a>
<h4>3.1.1 Flat files as data store</h4>
<p>
In the early days of digital data, before 1960, the data within a Computer Information System was mostly stored in rather flat files (sometimes indexed) manipulated by top-level software systems.
<br>
The primitives provided by the operating system were really very low level, basically just the possibility to read or write file or file increments.
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a8eb454f-c28a-4cb9-a21b-c148ab9b1fac">
<img class="centered" style="width: 350px;" alt="An indexed flat file example" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a8eb454f-c28a-4cb9-a21b-c148ab9b1fac" />
</a>
<div class="centered">
<i>Indexed flat file</i>
</div>
</div>
<br>
<p>
Directly using flat files was cumbersome and painful. Different uncovered needs emerged at the time:
</p>
<ul>
<li>Data isolation</li>
<li>Access efficiency</li>
<li>Data integrity</li>
<li>Reducing the time required to develop brand new applications</li>
</ul>
<p>
Addressing such needs by relying on indexed flat files required solutions to be implemented artificially by the applications using such files.
<br>
It was highly difficult, inefficient, time consuming, etc. And the wheel had to be re-invented all the time all over again.
<br>
So something else was required.
</p>
<a name="sec312"></a>
<h4>3.1.2 RDBMS and the relational model</h4>
<p>
So in 1969, Edgar F. Cood, a British engineer, invented the relational model. In the relational model, business entities are modeled as <i>Tables</i> and <i>Associations</i> (relations).
<br>
The relational model is at the root of <b>RDBMS</b> - Relational DataBase Management Systems - that ruled the Data Storage world for 30 years.
</p>
<p>
The relational model is conceived to reduce redundancy in order to optimize disk space usage. At the time of its creation Disk storage was very expensive and limited. And the volume of data in Information Systems was rather small.
<br>
The relational model avoids redundancy to optimize disk space usage by guaranteeing:
</p>
<ul>
<li><b>Structure:</b> using normal design forms and modeling techniques</li>
<li><b>Coherence:</b> using transaction principles and mechanisms</li>
</ul>
<p>
An example relational model would be as follows, illustrating an Exam Grade Management application
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/288402d5-a103-4e0b-8c67-a211087c93bc">
<img class="centered" style="width: 350px;" alt="a relational model example" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/288402d5-a103-4e0b-8c67-a211087c93bc" />
</a>
<div class="centered">
<i>Relational Model Example</i>
</div>
</div>
<br>
<p>
In this example, if we want to know the subject assigned to a student on his profile screen, we would need to
</p>
<ol>
<li>Extract the personal data from the "student" table</li>
<li>Fetch its subject if from the "relation" table</li>
<li>Read the subject title from the "subject" table.</li>
</ol>
<p>
<span style="color: red;">
Why, oh why, separate all this information in different tables when in practice 99% of the time we want to fetch all of this together ?
</span>
</p>
<a name="sec313"></a>
<h4>3.1.3 criticism of the relational model</h4>
<p>
The relational model comes from a time where storage was expensive. The fundamental idea behind its design is rationalizing storage space by ensuring every piece of information is stored only once.
<br>
But nowadays long-term storage space is not expensive at all anymore. A Terabyte of SSD storage is not more that a few dozens of dollars. Optimizing the storage space at all cost makes little sense today.
<br>
In addition, the relational model is not the best way to represent some information. Let's see some examples
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/111e754b-ac70-44fd-a4ea-260d6c76c68c">
<img class="centered" style="width: 800px;" alt="other models" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/111e754b-ac70-44fd-a4ea-260d6c76c68c" />
</a>
<div class="centered">
<i>Other models</i>
</div>
</div>
<br>
<ul>
<li>Tabular information fits naturally well in the relational model but not only. Every time we can naturally divide a business problem into well-defined and predefined entities and relations among them, the relational model is usually a good fit.</li>
<li>But then think of other type of information, such as registration forms, product descriptions, etc. Such types of semi-structured data fit very poorly in the relational model.</li>
<li>Also molecular data or graph data would be way better stored in very different types of databases.</li>
</ul>
<p>
The web giants had to get away from the mainframe pattern, and if you challenge that, the very fundamental architecture pattern on which all information systems were built, why wouldn't you challenge all the rest, including the relational model ?
<br>
We'll get back to this.
</p>
<a name="sec32"></a>
<h3>3.2 Horizontal scalability</h3>
<p>
The mid and late 2000’s were times of major changes in the IT landscape. Hardware capabilities significantly increased and eCommerce and internet trade, in general, exploded.
<br>
Some internet companies- the "Web giants" (Yahoo!, Facebook, Google, Amazon, Ebay, Twitter, ...), pushed traditional databases to their limits. Those databases are by design hard to scale.
<br>
Traditional RDBMS and traditional architecture can only <b>scale up</b>. And scaling up is tricky.
</p>
<a name="sec321"></a>
<h4>3.2.1 Scaling up</h4>
<p>
With relational RDBMSes, the only way to improve performance is by scaling up, i.e. getting bigger servers (more CPU, more RAM, more disk, etc.). There's simply nothing else that can be done.
<br>
But one eventually hits a hard limit imposed by the current technology.
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/5a4c7dd0-9d31-48f5-9960-d1513616ea7d">
<img class="centered" style="width: 800px;" alt="scaling up" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/5a4c7dd0-9d31-48f5-9960-d1513616ea7d" />
</a>
</div>
<br>
<p>
With traditional architectures and RDMBS, all the workload happens on one single machine. And while running a few thousands operations or transactions on one single machine is perhaps possible, going much beyond it just doesn't work. The programming paradigm we use - mostly around thread synchronizations and context switches - make it impossible to run effectively a million threads on one single machine for instance.
</p>
<p>
But there's worst than that.
<br>
Imagine that a machine A with 4 CPUs, 64 GB RAM and 1 TB hard drive costs 10'000 USD.
<br>
Do you think that a machine B with twice the power so 8 CPUs, 128 GB RAM and a 2 TB hard drive would cost the double, hence 20'000 USD ?
<br>
<b>No!</b> It would be much more than that, perhaps four or five times the price, so more than 40k USD.
</p>
<p>
The price of individual machines doesn't scale linearly with the processing power of the machine, it's exponential !
</p>
<a name="sec322"></a>
<h4>3.2.2 Scaling out</h4>
<p>
By rethinking the architecture of databases, the web giants have been able to make them scale at will, by adding more servers to clusters instead of upgrading the servers.
<br>
When scaling out, instead of buying bigger machines, one buys more machines and add them in a processing cluster, working together on distributed data and processing.
<br>
The servers are not made of expensive, high-end hardware; they are qualified as <i>commodity hardware</i> (or commodity servers).
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/9f763a29-a1f7-40cf-b6f6-c47e039eb5a0">
<img class="centered" style="width: 800px;" alt="scaling out" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/9f763a29-a1f7-40cf-b6f6-c47e039eb5a0" />
</a>
</div>
<br>
<p>
When scaling out, the limits vanish, one can add as many nodes as one wants in a processing cluster.
<br>
And there's a cherry on the cake, recall the example of machine A above, buying 10 machine A is not even 10 times the price of a single machine A, since one can get discounts from the number being bought.
</p>
<p>
The only drawback is that the application leveraging on scaling out, or the information system as a whole, needs to be designed from the grounds up for distribution. And there are constraints for this, we'll see that further in this article.
</p>
<p>
Scaling out is also called Horizontal scalability while scaling up is called vertical scalability.
</pa>
<a name="sec33"></a>
<h3>3.3 Data Distribution</h3>
<p>
With most NoSQL databases, the data is not stored in one place (i.e. on one server). It is distributed among the nodes of the cluster. When created, an object is split in a number of shards, for instance 4 shards, A, B, C, D and these shards are assigned to a node in the cluster.
<br>
This is called <b>sharding</b> - or <b>partitioning</b> - the portion of data assigned to a node is called a shard - or a partition.
</p>
<p>
Having more cluster nodes implies a higher risk of having some nodes crash, or a network outage splitting the cluster in two. For this reason, and to avoid data loss, objects are also replicated across the clusters. The number of copies, called replicas, can be tuned. 3 replicas is a common figure.
<br>
Imaging that the specifications of a given computer indicates that there is a 10% chance for the computer to experience any kind of hardware failure in its first year of exploitation. Then imagine you have 10 nodes like that one in a cluster, what is the probability that at least one of these nodes experiences an hardware failure ? Yes, you can be nearly sure at least one of them will fail.
<p>
<p>
For this reason, when we start to distribute data on a cluster of multiple machines, we have to design for failures.
<br>
In data management, this means creating multiple copies of every shard in such a way that we maximize the chances of one of them always being available.
<br>
This is called <B>replication</b>.
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/1d8730dc-6e30-48ad-a7fd-73a8d718f92a">
<img class="centered" style="width: 250px;" alt="data distribution" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/1d8730dc-6e30-48ad-a7fd-73a8d718f92a" />
</a>
</div>
<br>
<p>
We can see here that the objects has been split in 4 shards A, B, C, D and that every shard has three replicas.
</p>
<p>
The objects may move, as nodes crash or new nodes join the cluster, ready to take charge of some of the objects. Such events are usually handled automatically by the cluster; the operation of shuffling objects around to keep a fair repartition of data is called <b>rebalancing</b>.
</p>
<a name="sec34"></a>
<h3>3.4 Properties of a distributed system</h3>
<p>
In RDBMSes, we expect DB transactions to respect some fundamental properties, identified by <b>ACID</b>: <i>Atomicity, Consistency, Isolation and Durability</i>.
<br>
In distributed systems, we consider things a little differently and consider the following properties:
</o>
<ul>
<li>
<b>Availability</b>
<br>
Availability (or lack thereof) is a property of the database cluster. The cluster is available if a request made by a client is always acknowledged by the system, i.e. it is guaranteed to be taken into account.
<br>
That doesn’t mean that the request is processed immediately. It may be put on hold. But an <i>available</i> system should at a minimum always acknowledge it immediately.
<br>
Practically speaking, availability is usually measured in percents. For instance, 99.99% availability means that the system is unavailable at most 0.01% of the time, that is, at most 53 min per year.
<br>
</li>
<li>
<b>Partition tolerance</b>
<br>
Partition Tolerance is verified if a system made of several interconnected nodes can stand a partition of the cluster; if it continues to operate when one or several nodes disappear. This happens when nodes crash or when a network equipment is shut down, taking a whole portion of the cluster away.
<br>
Partition tolerance is related to availability and consistency, but it is still different. It states that the system continues to function internally (e.g. ensuring data distribution and replication), whatever its interactions with a client.
</li>
<li>
<b>Consistency</b>
<br>
When talking about distributed databases, like NoSQL, consistency has a meaning that is somewhat different than in the relational context.
<br>
It refers to the fact that all replicas of an entity, identified by a key in the database, have the same value whatever the node being queried.
<br>
With many NoSQL databases, updates take a little time to propagate across the cluster. When an entity’s value has just been created or modified, there is a short time span during which the entity is not consistent. However the cluster guarantees that it will eventually be, when replication has occurred. This is called eventual consistency
</li>
</ul>
<p>
These 3 properties, Consistency, Availability and Partition tolerance, are not independent.
<br>
The CAP theorem - or Brewer’s theorem - states that <b>a distributed system cannot guarantee all 3 properties at the same time</b>.
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/7e1bf188-d074-409d-a70e-26fff0d5e80c">
<img class="centered" style="width: 850px;" alt="The CAP theorem" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/7e1bf188-d074-409d-a70e-26fff0d5e80c" />
</a>
</div>
<br>
<p>
This is a theorem. That means it is formally true, but in practice it is less severe than it seems.
<br>
The system or a client can often choose CA, AP or CP according to the context, and "walk" along the chosen edge by appropriate tuning.
<br>
Partition splits happen, but they are rare events (hopefully).
</p>
<p>
Traditional Relational DBMSes would be seen as CA - consistency is a must
<br>
Many NoSQL DBMSes are AP - availability is a must. Big clusters failures happen all the time so they better live with it. Consistency is only eventual.
</p>
<a name="sec341"></a>
<h4>3.4.1 Eventual consistency</h4>
<p>
Consistency refers to the fact that all replicas of an entity, identified by a key in the database, have the same value at any give time whatever the node being queried.
</p>
<p>
With many NoSQL databases, the preferred working mode is AP and <i>all-the-time consistency</i> is sacrificed.
<br>
Favoring performance, updates take a little time to propagate across the cluster. When an entity’s value has just been created or modified, there is a short time span during which the entity is not consistent. One node being queried at that moment could show the previous value while another node at the same time would already show the new value.
<br>
However the cluster guarantees that it will eventually be, when replication has occurred. This is called <b>eventual consistency</b> and this is an essential notion.
</p>
<p>
While all-the-time consistency is sacrificed, <b>eventual consistency is a must and is guaranteed by most-if-not-all NoSQL Database</b>.
</p>
<a name="sec4"></a>
<h2>4. NoSQL / NewSQL</h2>
<p>
NoSQL databases are a new type of databases emerging from the web giants technologies mostly, scaling out natively and renouncing to the usual behaviours and features of RDBMS - Relational Database Management Systems.
</p>
<a name="sec41"></a>
<h3>4.1 NoSQL</h3>
<p>
A NoSQL - originally referring to "<b>not-SQL</b>" for "non-relational" - database provides a mechanism for storage and retrieval of data that is modeled in <b>means other than the tabular relations</b> used in relational databases.
<br>
Such databases have existed since the late 1960s, but the name "NoSQL" was only coined in the early 21st century, triggered by the needs of Web 2.0 companies.
<br>
NoSQL databases are increasingly used in Big Data and Real-Time Web applications.
<br>
NoSQL systems are also sometimes called "Not only SQL" to emphasize that they may support SQL-like query languages or sit alongside SQL databases in polyglot-persistent architectures.
<br>
(Wikipedia - <a href="https://en.wikipedia.org/wiki/NoSQL">https://en.wikipedia.org/wiki/NoSQL</a>)
</p>
<p>
The fundamental idea behind NoSQL is as follows:
</p>
<ul>
<li>Because of the need to distribute data (Big Data), the Web giants have abandoned the whole idea of ACID transactions (only eventual consistency is possible).</li>
<li>So if we drop ACID Transactions - which we always deemed to be so fundamental - why wouldn't we challenge all the rest - the relational model and table structure?</li>
</ul>
<p>
There are 4 common types of NoSQL databases:
</p>
<ul>
<li><b>Document-oriented </b>, e.g. MongoDB, ElasticSearch
<li><b>Column-family</b> (aka BigTable), e.g. Cassandra
<li><b>Key/Value pairs</b>, e.g. Redis
<li><b>Graph</b>, e.g. Neo4J
</ul>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/c5f80a99-798d-4be2-969a-67f55d95700f">
<img class="centered" style="width: 850px;" alt="Types of NoSQL database" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/c5f80a99-798d-4be2-969a-67f55d95700f" />
</a>
</div>
<br>
<p>
Document-oriented databases are really the most wide spread with market leaders such as MongoDB, ElasticSearch, CouchDB, etc.
<br>
Column-oriented databases are also wide spread with multiple good open source solutions.
<br>
Key/Value pairs databases are really <b>distributed caching</b> products in in the end. Multiple good solutions are available on the market but most of them are proprietary software with sometimes a limited open-source version (unfortunately).
<br>
In terms of graph oriented databases, the lead player is Neo4J.
</p>
<p>
The following schema provides an illustration of the way data is structured and stored in these Databases:
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/81727cc6-f37c-4295-b54d-d3adccb4a673">
<img class="centered" style="width: 850px;" alt="NoSQL data storage examples" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/81727cc6-f37c-4295-b54d-d3adccb4a673" />
</a>
</div>
<br>
<p>
The NoSQL landscape is y very rich ecosystems with hundreds of different products and solution and growing continuously, with nearly every week a new product appearing.
</p>
<a name="sec42"></a>
<h3>4.2 NewSQL</h3>
<p>
What is NewSQL ?
</p>
<p>
NewSQL refers to relational databases that have adopted upon some of the NoSQL genes, thus exposing a relational data model and SQL interfaces to distributed, high volume databases.
</p>
<p>
NewSQL, contrary to NoSQL, enables an application to keep
</p>
<ul>
<li>The relational view on the data</li>
<li>The SQL query language</li>
<li>Response times suited to transactional processing</li>
</ul>
<p>
Some were built from scratch (e.g. VoltDB), others are built on top of a NoSQL data store (e.g. SQLFire, backed by GemFire, a key/value store)
</p>
<p>
The current trend is for some proven NoSQL databases, like Cassandra, to offer a thin SQL interface, achieving the same purpose
<br>
Generally speaking, the frontier between NoSQL and NewSQL is a bit blurry… SQL compliance is often sought for, as the key to integrating legacy SQL software (ETL, reporting) with modern No/NewSQL databases
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d1973df4-a625-4bdd-a98b-e17c7348172e">
<img class="centered" style="width: 600px;" alt="NewSQL examples" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d1973df4-a625-4bdd-a98b-e17c7348172e" />
</a>
</div>
<br>
<a name="sec5"></a>
<h2>5. Hadoop and Data Lakes</h2>
<p>
Around 2006, Google published two papers, "GFS - The Google FileSystem" where they explained how they designed an implemented a distributed filesystem and "Map Reduce" where they presented the distributed programming paradigm they used to process data stored on GFS.
<br>
A few years after, google published "Big Table", a paper presenting how they designed and implemented a <i>Column-oriented database</i> on top of HDFS and MapReduce.
</p>
<p>
Doug Cutting, the leader of the Apache Lucene Project at the time discovered these papers and decided to work on an Open-Source implementation of these concepts.
<br>
Hadoop was born.
</p>
<a name="sec51"></a>
<h3>5.1 What is Hadoop ?</h3>
<p>
Hadoop is an <b>Open Source Platform</b> providing:
<ul>
<li>A distributed, scalable and fault tolerant storage system as a grid</li>
<li>Initially, a single parallelism paradigm : MapReduce to reuse the storage nodes as processing nodes</li>
<li>Since Hadoop V2 and YARN, other parallelization paradigms can be implemented on Hadoop</li>
<li>Schemaless and optimized sequential write once and read many times</li>
<li>Querying and processing DSL (Hive, Pig)</li>
</ul>
<p>
Hadoop was initially primarily intended for Big Data Analytics. It is the middleware under the Data Lake Architecture pattern and intents to revolution the architecture of analytical information systems / decision support systems.
<br>
Nowadays Hadoop can be an infrastructure for much more, such as Micro-services architecture (Hadoop V3) or Real-time Architectures.
</p>
<p>
Hadoop is declined in different distributions: Fundation Apache, Cloudera, HortonWorks, MapR, IBM, etc.
</p>
<a name="sec52"></a>
<h3>5.2 Hadoop Overview</h3>
<p>
Hadoop is designed as a layered software where technologies in every layer can be interchanged at will:
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/690fe110-474d-411b-96a5-2757af8b3906">
<img class="centered" style="width: 850px;" alt="Hadoop Overview" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/690fe110-474d-411b-96a5-2757af8b3906" />
</a>
</div>
<br>
<ul>
<li>
<b>Distributed storage</b>: Hadoop packages HDFS as the default underlying distributed filesystem. But for instance the MapR Hadoop distribution uses the MAPR filesystem instead.
</li>
<li>
<b>Parallel Computing Framework / MapReduce Processing Engine</b>: In Hadoop V1, MapReduce was the only parallel computing paradigm available on top of Hadoop, taking care of the processing distribution as well as the resources negotiation on the Hadoop cluster.
<br>
With Hadoop 2.0, The MapReduce paradigm has been split from the Resource negotiation which became YARN - Yet Another Resource Negotiator. With this split, it has become possible to use Hadoop with different parallel processing constructs and paradigms, MapReduce becoming just one possibility among others.
</li>
<li>
<b>Machine Learning / Processing</b>: This is in the end the most essential layer on top of Hadoop core. Hadoop is designed first and foremost for Big Data Analytics. There are numerous solutions that were initially either implemented on top of MapReduce or ported to MapReduce.
<br>
Nowadays, with YARN, software doesn't need anymore to be ported to MapReduce to run on Hadoop, it just needs to integrate with YARN.
</li>
<li>
<b>Orchestration</b>: Numerous different solution can be used to operate Hadoop and orchestrate processes.
</li>
<li>
<b>Querying</b>: A lot of NoSQL / NewSQL databases have been implemented as an Hadoop querying framework, such as HBase or Hive. Some more advanced tools goes beyond querying with as Pig.
</li>
<li>
<b>Reporting</b>: User have multiple choices of software specialized in building reports on the data in Hadoop.
</li>
<li>
<b>IS Integration</b>: Integrating Hadoop in the Information System, specifically building data import / Export between Hadoop and the operational information system components is a key concern. Numerous different solutions have been developed for this and are packaged with most Hadoop distributions.
</li>
<li>
<b>Supervision and Management</b>: Most Hadoop distributions provide their own management tool. Some tools are available as Apache projects.
</li>
</ul>
<p>
Hadoop is a very large ecosystems of hundreds of different software in all these different layers.
<br>
The most common ones would be as follows:
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/29b960d3-03ef-4d5f-bce2-bb677989dc27">
<img class="centered" style="width: 850px;" alt="Hadoop Components" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/29b960d3-03ef-4d5f-bce2-bb677989dc27" />
</a>
</div>
<br>
<p>
But then again there are really many more components than that in a typical Hadoop distribution.
<br>
Most Hadoop distributions are quite behemoth software stacks that would be very difficult to integrate and configure manually, which is the very reason behind the success of these distributions.
<br>
Hadoop core on its own is fairly complex to install, configure and fine tune so whenever one needs Hadoop core only for his specific software (e.g to run spark), it's sometimes a more appropriate approach to search for a lighter cluster management system such as <i>Apache Mesos</i> for instance; more on that later in this article.
</o>
<a name="sec53"></a>
<h3>5.3 Hadoop Architecture</h3>
<p>
A simplified view of Hadoop core components deployment architecture would be as follows:
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/217b1b5e-1555-46e2-8033-7e86f8de2f2a">
<img class="centered" style="width: 850px;" alt="Hadoop Architecture" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/217b1b5e-1555-46e2-8033-7e86f8de2f2a" />
</a>
<div class="centered">
<i>Hadoop Architecture</i>
</div>
</div>
<br>
<p>
Since Hadoop 2, having two master nodes for high-availability and avoiding <i>single points of failure</i> on the master components is highly advised.
<br>
The components from Hadoop core are deployed as follows:
</p>
<ul>
<li>
The <b>HDFS Name node</b> (and secondary name node) is the center piece of the HDFS File System. It acts as the HDFS Master and keeps the directory tree and tracks where on the cluster the file data is kept. The <b>HDFS Data Nodes</b> acts as slave processes, run on individual cluster nodes and take care of data storage.
</li>
<li>
The <b>YARN Resource Manager</b> (and secondary resource manager) is the master that arbitrates all the available cluster resources and thus helps manage the distributed applications running on the YARN system. It works together with the per-node <b>NodeManagers</b> and the per-application <b>ApplicationMaster</b>.
</li>
<li>
The <b>MapReduce JobTracker</b> is the service within Hadoop that farms out MapReduce tasks to specific nodes in the cluster, ideally the nodes that have the data for <b>co-local processing optimization</b>, or at least are in the same rack. Client applications submit jobs to the Job tracker. <b>MapReduce TaskTrackers</b> run on individual cluster nodes, execute the tasks and report the status of tasks to the JobTracker.
</li>
</ul>
<a name="sec54"></a>
<h3>5.4 The DataLake Architecture pattern</h3>
<p>
From <a href="https://en.wikipedia.org/wiki/Data_lake ">Wikipedia</a>:
<br>
A data lake is a system or repository of data stored in its natural/raw format.
</p>
<ul>
<li>
It's is usually a single store of data including raw copies of source system data, sensor data, social data etc. and transformed data used for tasks such as reporting, visualization, advanced analytics and machine learning.
</li>
<li>
It can include structured data from relational databases, semi-structured data (CSV, logs, XML, JSON), unstructured data (emails, documents, PDFs) and binary data (images, audio, video).
</li>
</ul>
<p>
With the continued growth in scope and scale of analytics applications using Hadoop and other data sources, then the vision of an enterprise data lake can become a reality.
<br>
In a practical sense, a data lake is characterized by three key attributes:
</p>
<ul>
<li>
<b>Collect everything</b>. A data lake contains all data, both raw sources over extended periods of time as well as any processed data.
<br>
A data lake is characterized by a <b>Big Volume</b> of data.
</li>
<li>
<b>Dive in anywhere</b>. A data lake enables users across multiple business units to refine, explore and enrich data on their terms.
<br>
In a Data Lake, one doesn't know <i>a priori</i> the analytical structures.
</li>
<li>
<b>Flexible access</b>. A data lake enables multiple data access patterns across a shared infrastructure: batch, interactive, online, search, in-memory and other processing engines.
<br>
As a result, a data lake delivers maximum scale and insight with the lowest possible friction and cost.
</li>
</ul>
<p>
The fundamental approach of Data Lakes is to <b>pull</b> the required data from the raw data storage, transforming it and processing it dynamically, as required per the use case being executed. It's entirely dynamic, queries and processes are designed on the fly.
<br>
The storage principle is to store everything, all the raw data from the operational Information System as well as all the data produced by the IS, log files, usage metrics, etc. (Collect everything pattern).
<br>
Hadoop is kind of the <i>Operating System</i> underneath the Data Lake pattern and with Hadoop's power, there is nearly no analytics use case that can't be implemented in a dynamic fashion, requiring at worst a few hours of runtime processing before providing the expected results
</p>
<p>
This is in complete opposition with the Data Warehouse pattern where the data was <b>pushed</b> in statically predefined transformation pipelines. The most critical drawback of this approach is the impossibility to come up with a new use case in a quick time. Most of the time, when a corporation requires a new KPI to be computed by the analytical system, if the required data was not already collected for another use case, it was impossible to provide quickly, requiring for instance to wait 6 months before providing the KPI on a 6 months period.
<br>
Hadoop finally made it possible at a cheap cost to get away fro this <i>push</i> pattern.
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/eb3fe0db-418c-48df-a404-146963fb779e">
<img class="centered" style="width: 850px;" alt="DataLake Architecture" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/eb3fe0db-418c-48df-a404-146963fb779e" />
</a>
<div class="centered">
<i>DataLake Architecture</i>
</div>
</div>
<br>
<p>
The <i>Data Lake</i> architecture pattern and its hadoop engine form a tremendous opportunity to finally get away from the <i>Data Warehouse pattern</i>.
<br>
But there are pitfalls of course and many corporations experienced it the hard way.
<br>
It has been stated to much everywhere that data can be incorporated "as is" in data lakes that way too many corporations took it too literaly, forgetting about one essential aspect, even in Data Lakes.
<br>
A minimum of data cleaning, cleansing and preparation is always required. The most crucial aspect than can nevfer be neglected is the need to alway have proper <b>correlation ID</b>s in every single piece of data that is being ingested in a data lake.
<br>
Without correlation IDs, data is unusable. And your Data Lake turns into a <i>Data Swamp</i>.
</p>
<a name="sec6"></a>
<h2>6. Streaming Architectures</h2>
<p>
Streaming data refers to data that is continuously generated, usually in high volumes and at high velocity. A streaming data source would typically consist of a stream of logs that record events as they happen - such as a user clicking on a link in a web page, or a sensor reporting the current temperature.
</p>
<p>
A <b>streaming data architecture</b> is a framework of software components built to ingest and process large volumes of streaming data from multiple sources. While traditional data solutions focused on writing and reading data in batches, a streaming data architecture consumes data immediately as it is generated, persists it to storage, and may include various additional components per use case - such as tools for real-time processing, data manipulation and analytics.
<br>
A <b>real-time</b> system is an event-driven system that is available, scalable and stable, able to take decisions (actions) with a latency defined as <i>below the frequency of events</i>.
</p>
<p>
Streaming Architectures are not strictly related to the web giants and the Big Data revolution and CEP - Complex Events Processing - Engines exists since the early 2000s.
<br>
However, streaming architectures evolved significantly with products emerging from the needs of the web giants in Lambda Architecture first and then Kappa Architecture.
</p>
<!--
6.1 CEP RT architectures
6.2 Lambda Architecture
6.3 Kappa architecture
-->
<a name="sec61"></a>
<h3>6.1 Complex Event Processing</h3>
<p>
From <a href="https://en.wikipedia.org/wiki/Complex_event_processing">Wikipedia</a>
<br>
Complex event processing, or CEP, consists of a set of concepts and techniques developed in the early 1990s for processing real-time events and extracting information from event streams as they arrive. The goal of complex event processing is to identify meaningful events (such as opportunities or threats) in real-time situations and respond to them as quickly as possible.
</p>
<p>
In a <i>Complex Event Processing Architecture</i> :
</p>
<ul>
<li>Historical data is regularly and consistently updated with live data.</li>
</li>Live data is available to the end user </li>
<li>
Both types or data (historical and live) are not necessarily presented consistently to the end user.
<ul>
<li>Both sets of data can have their own screens or even application</li>
<li><i>A consistent view on both sets of data would be proposed by Lambda Architecture (next topic in this presentation)</i></li>
</ul>
</li>
</ul>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/3f822b48-b325-4d2b-a36e-7245dceb80f2">
<img class="centered" style="width: 850px;" alt="Complex Event Processing" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/3f822b48-b325-4d2b-a36e-7245dceb80f2" />
</a>
<div class="centered">
<i>Complex Event Processing</i>
</div>
</div>
<br>
<p>
A few notes on typical CEP deployments, in a raw fashion:
</p>
<ul>
<li><b>The rules GUI</b> is often a user friendly editor supporting <i>hot</i> updates of rules and made available to business users.</li>
<li><b>The capture middleware</b> should support very high throughput of thousands of events per second, just as the whole processing line and negligible latency.</li>
<li><b>The CEP engine</b> needs to support very high throughput as well and usually a maximum latency of a few dozen milliseconds to hundreds milliseconds. Fault tolerance and state coherence are common concerns.</li>
</ul>
<p>
Complex Event Processing engines and architecture are heavily used in the industry in the world of real-time computing systems, such as trading systems, payment monitoring systems, etc.
<br>
Such engines form however a quite legacy technology and have limits in terms of analytics. Most if not all CEP engines on the market even nowadays are really some sort of evolved <i>rules-engines</i>.
<br>
And that would be the most common limit of CEP engines, the fact that its really only about rules. Machine learning and AI use cases are limited on CEP engines by the difficulty of these systems to derive features requiring correlation with large historical datasets.
</p>
<p>
The rise of Big Data analytics technologies have opened the door for much more advanced analytics use cases in real-time. Lambda Architecture and Kappa Architectures are much more recent approaches to real-time analytics.
</p>
<a name="sec62"></a>
<h3>6.2 Lambda Architecture</h3>
<p>
The Lambda Architecture, first proposed by Nathan Marz, attempts to provide a combination of technologies that together provide the characteristics of a web-scale system that satisfies requirements for availability, maintainability, fault-tolerance and low-latency.
</p>
<p>
Quoting <a href="https://en.wikipedia.org/wiki/Lambda_architecture">Wikipedia</a>: "<i>Lambda architecture is a data-processing architecture designed to handle massive quantities of data by taking advantage of both batch- and stream-processing methods.
<br>
This approach to architecture attempts to balance latency, throughput, and fault-tolerance by using batch processing to provide comprehensive and accurate views of batch data, while simultaneously using real-time stream processing to provide views of online data. The two view outputs may be joined before presentation.
<br>
The rise of lambda architecture is correlated with the growth of big data, real-time analytics, and the drive to mitigate the latencies of map-reduce.</i>"
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a63d7668-8d45-47ec-a2f5-57eb64eb6b00">
<img class="centered" style="width: 50px; " alt="Lambda Symbol" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a63d7668-8d45-47ec-a2f5-57eb64eb6b00" />
</a>
</div>
<br>
<p>
At a high level, the Lambda Architecture is designed to handle both real-time and historically aggregated batched data in an integrated fashion. It separates the duties of real-time and batch processing so purpose-built engines, processes, and storage can be used for each, while serving and query layers present a unified view of all of the data.
<br>
The efficiency of this architecture becomes evident in the form of increased throughput, reduced latency and negligible errors. While we mention data processing we basically use this term to represent high throughput, low latency and aiming for near-real-time applications.
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/0b993a55-6c71-4eab-811d-426e084385b6">
<img class="centered" style="width: 850px;" alt="Lambda Architecture" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/0b993a55-6c71-4eab-811d-426e084385b6" />
</a>
</div>
<br>
<p>
As new data is introduced to the system, it is processed simultaneously by both the batch layer, and the speed layer. The batch layer is an append-only repository containing unprocessed raw data. The batch layer periodically or continuously runs jobs that create views of the batch data-aggregations or representations of the most up-to-date versions. These batch views are sent to the serving layer, where they are available for analytic queries.
<br>
At the same time that data is being appended to the batch layer, it is simultaneously streaming into the speed layer. The speed layer is designed to allow queries to reflect the most up-to-date information-necessary because the serving layer's views can only be created by relatively long-running batch jobs. The speed layer computes only the data needed to bring the serving layer's views to real time-for instance, calculating totals for the past few minutes that are missing in the serving layer's view.
<br>
By merging data from the speed and serving layers, low latency queries can include data that is based on computationally expensive batch processing, and yet include real-time data. In the Lambda Architecture, the raw source data is always available, so redefinition and re-computation of the batch and speed views can be performed on demand. The batch layer provides a big data repository for machine learning and advanced analytics, while the speed and serving layers provide a platform for real-time analytics.
<br>
The Lambda Architecture provides a useful pattern for combining multiple big data technologies to achieve multiple enterprise objectives.
</p>
<p>
There are numerous solutions nowadays to build every layer of a Lambda Architecture:
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/56ffadc9-952a-4e2c-9867-6e8799ffcc26">
<img class="centered" style="width: 850px;" alt="Lambda Architecture components" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/56ffadc9-952a-4e2c-9867-6e8799ffcc26" />
</a>
</div>
<br>
<p>
The takeaway here is that we have gone a long way since <i>Complex Event Processing</i> architectures and we have now numerous solutions to build new generations of streaming architectures able to extend real-time streaming to much more advanced analytics use cases, embracing <i>Real-time Artificial Intelligence</i> use cases.
</p>
<p>
<b>Pros and Cons of Lambda Architecture.</b>
</p>
<p>
<b>Pros</b>
</p>
<ul>
<li>Batch layer of Lambda architecture manages historical data with the fault tolerant distributed storage which ensures low possibility of errors even if the system crashes.</li>
<li>It is a good balance of speed and reliability.</li>
<li>Fault tolerant and scalable architecture for data processing.</li>
</ul>
<p>
<b>Cons</b>
</p>
<ul>
<li><b>It can result in coding overhead due to the need to implement the same analytics logic twice:</b> one time in the speed layer and one time all over again in the batch layer.</li>
<li>Re-processes every batch cycle which is not beneficial in certain scenarios.</li>
<li>A data modeled with Lambda architecture is difficult to migrate or reorganize.</li>
</ul>
<a name="sec63"></a>
<h3>6.3 Kappa Architecture</h3>
<p>
In 2014 Jay Kreps started a discussion where he pointed out some discrepancies of Lambda architecture that further led the big data world to another alternate architecture that used less code resource and was capable of performing well in certain enterprise scenarios where using multi layered Lambda architecture seemed like extravagance.
<br>
Kappa Architecture cannot be taken as a substitute of Lambda architecture on the contrary it should be seen as an alternative to be used in those circumstances where active performance of batch layer is not necessary for meeting the standard quality of service.
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/fd9e852b-718f-48b3-8c89-c0ab9c596488">
<img class="centered" style="width: 250px; " alt="Lambda vs Kappa Symbol" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/fd9e852b-718f-48b3-8c89-c0ab9c596488" />
</a>
</div>
<br>
<p>
Kappa architecture is a streaming-first architecture deployment pattern. With most recent Stream Processing technologies (Kafka Streams, Flink, etc.) the interest and relevance of the batch layer tend to diminish. The streaming layer matches computation abilities of the batch layer (ML, statistics, etc.) and stored data as it processes it.
<br>
A batch layer would only be needed to kick-start the system on historical data but then Apache Flink can very well do that
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/1ad9d53b-e708-4f20-8fab-2f8b3988f0af">
<img class="centered" style="width: 850px;" alt="Lambda Architecture" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/1ad9d53b-e708-4f20-8fab-2f8b3988f0af" />
</a>
</div>
<br>
<p>
Kappa architecture can be deployed for those data processing enterprise models where:
</p>
<ul>
<li>Multiple data events or queries are logged in a queue to be catered against a distributed file system storage or history.</li>
<li>The order of the events and queries is not predetermined. Stream processing platforms can interact with database at any time.</li>
<li>It is resilient and highly available as handling Terabytes of storage is required for each node of the system to support replication.</li>
</ul>
<p>
<b>Pros and Cons of Kappa architecture</b>
</p>
<p>
<b>Pros</b>
</p>
<ul>
<li>Kappa architecture can be used to develop data systems that are online learners and therefore don’t need the batch layer.</li>
<li>Re-processing is required only when the code changes.</li>
<li>It can be deployed with fixed memory.</li>
<li>It can be used for horizontally scalable systems.</li>
<li>Fewer resources are required as the machine learning is being done on the real time basis.</li>
</ul>
<p>
<b>Cons</b>
</p>
<ul>
<li>Absence of batch layer might result in errors during data processing or while updating the database that requires having an exception manager to reprocess the data or reconciliation.</li>
</ul>
<a name="sec7"></a>
<h2>7. Big Data 2.0</h2>
<p>
When google published their papers in the early 2000s, it was quite a tsunami in the Computer Engineering World. Doug Cutting and the guys behind Hadoop started working on Hadoop but a lot of other initiatives kicked off as well.
<br>
With their approach - scaling information systems on commodity hardware, it turned out that massive computational systems suddenly became affordable and it gave a whole new level of interest in distributed systems and distributed computing.
<br>
<p>
There are now an entire range of engines that transcend the Hadoop framework and are dedicated to specific verticals (e.g. structured data, graph data, streaming data, etc.)
<br>
Nowadays, The NoSQL ecosystem provides incredibly efficient alternatives to HDFS storage in the storage layer. In the processing layer, there is a plethora of solutions available from Kafka Streams to Apache Flink through Spark, etc.
<br>
On the resource negotiation side as well, multiple initiatives provide lightweight and interesting alternatives to Hadoop's YARN.
</p>
<a name="sec71"></a>
<h3>7.1 Alternatives to Hadoop </h3>
<p>
A specific project kicked off by the University of California retained quite a bit of attention at the time, the <i>Nexus</i> project renamed later <i>Mesos</i> and given to the Apache Software Foundation.
</p>
<p>
Apache Mesos intended to be kind of the <i>Operating System</i> of a computer cluster, somehow in the same way the <i>Linux Kernel</i> for instance is operating a single machine. Mesos intended to provide the same kind of primitives for resources management at the scale of a whole cluster.
<br>
Pretty early in the Mesos development story, support of docker containers has been added to enable users to deploy and scale applications in the form of docker containers.
</p>
<p>
A few years Later, some folks inspired from <i>Google Borg</i> and created in their turn a <i>cloud container orchestration system</i> for automating computer application deployment, scaling, and management. They named it Kubernetes.
<br>
With Mesos and Kubernetes gaining a lot of traction since scaling applications in the form of docker containers is extremely convenient, the folks at Hadoop added support to deploying applications in the form of docker containers as well in YARN in Hadoop 3.0.
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/3cb0a91f-9959-4f4f-b55b-79b4c2d1c57a">
<img class="centered" style="width: 650px;" alt="Big Data 2.0" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/3cb0a91f-9959-4f4f-b55b-79b4c2d1c57a" />
</a>
</div>
<br>
<p>
Nowadays in 2021, with Hadoop 3, these 3 technologies converge tend to converge to the same possibilities. Hadoop 3 supports deploying jobs as docker containers just as Mesos and Kubernetes.
<br>
Mesos and Kubernetes can use alternatives to HDFS such as Ceph, GlusterFS, Minio, (of course Amazon, Azure, ...) etc.
</p>
<p>
So while Kubernetes was really oriented to scale application in the <b>Operational Information System</b> space initially, it tends now to overflow to analytics use case as well.
<br>
And the other way around, while Hadoop is still first and foremost oriented to deploy applications in the <b>Analytical Information System</b> space, Hadoop 3 tends to be deployed increasingly in the operational space as well.
<br>
Apache Mesos can well be used on both sides and was forming an interesting alternatives to Hadoop YARN in both worlds for quite some time. Today, Apache Mesos, even though from my perspective an amazing software, is not heavily maintained anymore and support for Mesos tends to vanish in latest versions of software stacks.
</p>
<p>
Kubernetes (and/or technologies based on Kubernetes) is today a market standard for the Operational IS just as Hadoop remains a market standard for the Analytical IS.
</p>
<a name="sec72"></a>
<h3>7.2 Kubernetes</h3>
<p>
Kubernetes is an Open Source Platform providing:
</p>
<ul>
<li>Automated software applications deployment, scaling, failover and management across cluster of nodes.</li>
<li>Management of application runtime components as Docker containers and application units as Pods.</li>
<li>Multiple common services required for service location, distributed volume management, etc. (pretty much everything one requires to deploy application on a Big Data cluster).</li>
</ul>
<p>
Kubernetes is originally largely inspired and even based on Google Borg, (one of) Google’s initial cluster management system(s). It has been released as Open-Source component in Google in 2014 and the
first official release was in 2015.
</p>
<p>
Kubernetes is emerging as a standard as a <b>Cloud Operating System</b>.
<br>
In comes in the flavour of many distributions. The mains ones are:
</p>
<ul>
<li>PKS (Pivotal Container Service)/li>
<li>Red-Hat OpenShift/li>
<li>Canonical Kubernetes/li>
<li>Google / AWS / Azure</li>
</ul>
<p>
Kubernetes deployment architecture would be as follows:
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/5593b148-4030-42a8-a170-314f8f5492ca">
<img class="centered" style="width: 850px;" alt="Kubernetes Architecture" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/5593b148-4030-42a8-a170-314f8f5492ca" />
</a>
<div class="centered">
<i>Kubernetes Architecture</i>
</div>
</div>
<br>
<p>
With the ever-growing popularity of containerized cloud-native applications, Kubernetes has become the leading orchestration platform to manage any containerized application.
<br>
Again, nowadays <b>Kubernetes</b> is emerging as a market standard to scale the <b>Operational Information System</b>, while <b>Hadoop</b> largely remains a market standard to scale the <b>Analytical Information System</b>.
</p>
<a name="sec8"></a>
<h2>8. Micro-services</h2>
<p>
From <a href="https://en.wikipedia.org/wiki/Microservices">Wikipedia</a>:
<br>
<b>Microservice architecture</b> - a variant of the <b>Service-Oriented Architecture (SOA)</b> structural style - arranges an application as a collection of loosely-coupled services. In a microservices architecture, services are fine-grained and the protocols are lightweight. Its characteristics are as follows:
</p>
<ul>
<li>Services in a microservices architecture (MSA) are small in size, messaging-enabled, bounded by contexts, autonomously developed, independently deployable, decentralized and built and released with automated processes.</li>
<li>Services are often processes that communicate over a network to fulfill a goal using technology-agnostic protocols such as HTTP.</li>
<li>Services are organized around business capabilities.</li>
<li>Services can be implemented using different programming languages, databases, hardware and software environment, depending on what fits best (Note JKE : this is not a strict requirement, e.g. Spring boot)</li>
</ul>
<p>
From <i>Martin Fowler</i>:
<br>
A Microservices-based architecture has the following properties:
</p>
<ul>
<li>Independent services lifecycles leads to a continuous delivery software development process. A change to a small part of the application only requires rebuilding and redeploying only one or a small number of services.</li>
<li>Adheres to principles such as fine-grained interfaces to independently deployable services, business-driven development (e.g. domain-driven design).</li>
</ul>
<p>
As early as 2005, Peter Rodgers introduced the term "Micro-Web-Services" during a presentation at the Web Services Edge conference. The architectural style name was really adopted in 2012.
<br>
Kubernetes democratized the architectural approach. The two big players in this field are Spring Cloud and Kubernetes
</p>
<p>
A typical micro-services infrastructure architecture would be as follows:
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/9ef528e8-40a3-42a8-8d73-db7ea47ff4d6">
<img class="centered" style="width: 850px;" alt="Micro-services infrastructure Architecture" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/9ef528e8-40a3-42a8-8d73-db7ea47ff4d6" />
</a>
<div class="centered">
<i>Micro-services Architecture</i>
</div>
</div>
<br>
<a name="sec81"></a>
<h3>8.1. Micro-services discussion</h3>
<p>
Ask yourself : do you need microservices ?
</p>
<ul>
<li><b>Microservices are NOT Big Data !</b>. In Big Data Analytics, one needs to scale the processing linearly with the storage. Hadoop and for instance Spark with Mesos on ElasticSearch are designed for that very key aspect to be respected: co-local processing optimization. Micro-services are not designed for this. The scaling approach in micro-services is at the component / service level. Heavy resources consuming services are scaled widely while light services run typically on a few nodes mostly for high-availability concerns. </li>
<li><b>You don’t need microservices or Kubernetes to benefit from Docker</b>. Docker is a tremendous way to package and deploy applications as a whole or individual application components. Unless you need horizontal scalability and high-availability, you might not need Kubernetes or a micro-services infrastructure.</li>
<li><b>You’re not scaling anything with synchronous calls.</b> This is essential. A fundamental element in the design of a micro-services architecture resides in the usage of asynchronous calls as the communication paradigm. Think of it. If services call each others using synchronous calls, then scaling them is useless since they will all synchronize with the slowest of them.</li>
</ul>
<p>
As a consequence, don’t do microservices unless:
</p>
<ul>
<li>You <b>need</b> independent service-level scalability (vs. storage / processing scalability - Big Data).</li>
<li>You <b>need</b> a strong SOA - Service-Oriented Architecture.</li>
<li>You <b>need</b> independent services lifecycle management.</li>
</ul>
<p>
There are various challenges to be accounted when implementing micro-services:
</p>
<ul>
<li><b>Distributed caching vs reloading the world all over again.</b> If every service is a fully independent application, then all the reference and master data need to be reloaded all over again by all services. This needs to be accounted and distributed caching needs to be considered.</li>
<li><b>Not all applications are fit for asynchronous communications.</b> Some applications require fundamentally synchronous calls.</li>
<li>
<b>Identifying the proper granularity for services.</b>
<ul>
<li>Enterprise architecture view is too big</li>
<li>Application architecture view is too fine</li>
</ul>
</li>
<li><b>Data consistency without distributed transactions</b>. Applications need to be designed with this in mind.</li>
<li>
<b>Weighting the overall memory and performance waste.</b>
<ul>
<li>A Spring boot stack + JVM + Linux Docker base for every single service ?</li>
<li>HTTP calls in between layers ?</li>
</ul>
</li>
</ul>
<a name="sec9"></a>
<h2>9. Conclusion</h2>
<p>
We went a long way in this article, from the web giants and their needs to scale their information systems horizontally, the reasons behind it and the challenges this implies, down to Micro-services and the scaling of individual Information Systems components.
<br>
The Web giants needs were really related initially to their massive amount of data to be manipulated and the need to scale the processing linearly with the storage distribution. Nowadays, cloud computing and SaaS - Software As A Service on the cloud form somehow a different needs.
<br>
Initial Big Data technologies were really oriented towards Data Analytics use cases and the Analytical Information System space. Later technologies, namely NoSQL / NewSQL and now Kubernetes and micro-services are much more oriented towards scaling or deploying on the cloud Operational Information System components
</p>
<p>
The Strong frontier between Operational IS and Analytical IS will tend to vanish in the future.
</p>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/e177c115-cc09-4659-ae72-dfa906030a62">
<img class="centered" style="width: 500px;" alt="Operational / Analytical Frontier" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/e177c115-cc09-4659-ae72-dfa906030a62" />
</a>
</div>
<br>
<p>
Increasingly, in Hadoop 3 with YARN able to manage and deploy docker containers, Hadoop is not so strictly limited to the Analytical IS.
<br>
On the other side, Kubernetes make it increasingly feasible to scale heavy data analytics applications as well.
<br>
Even today NoSQL, Streaming, Lambda and Kappa architectures are increasingly overflowing to the Operational IS and as such provide a common ground for operational processes and analytical processes.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/powerful-big-data-analytics-platform
Powerful Big Data analytics platform fights financial crime in real time
Jerome Kehrli
2021-09-03T05:17:04-04:00
2021-09-03T05:17:04-04:00
<p>
<i>(Article initially published on <a href="https://blog.netguardians.ch/powerful-big-data-analytics-platform-fights-financial-crime-in-real-time">NetGuardians' blog</a>)</i>
</p>
<p>
NetGuardians overcomes the problems of analyzing billions of pieces of data in real time with a unique combination of technologies to offer unbeatable fraud detection and efficient transaction monitoring without undermining the customer experience or the operational efficiency and security in an enterprise-ready solution.
</p>
<p>
When it comes to data analytics, the more data the better, right? Not so fast. That’s only true if you can crunch that data in a timely and cost-effective way.
</p>
<p>
This is the problem facing banks looking to Big Data technology to help them spot and stop fraudulent and/or non-compliant transactions. With a window of no more than a hundredth of a millisecond to assess a transaction and assign a risk score, banks need accurate and robust real-time analytics delivered at an affordable price. Furthermore, they need a scalable system that can score not one but many thousands of transactions within a few seconds and grow with the bank as the industry moves to real-time processing.
</p>
<p>
AML transaction monitoring might be simple on paper but making it effective and ensuring it doesn’t become a drag on operations has been a big ask. Using artificial intelligence to post-process and analyze alerts as they are thrown up is a game-changing paradigm, delivering a significant reduction in the operational cost of analyzing those alerts. But accurate fraud risk scoring is a much harder game. Some fraud mitigation solutions based on rules engines focus on what the fraudsters do, which entails an endless game of cat and mouse, staying up to date with their latest scams. By definition, this leaves the bank at least one step behind.
</p>
<p>
At NetGuardians, rather than try to keep up with the fraudsters, we focus on what we know and what changes very little – customers’ behavior and that of bank staff. By learning “normal” behavior, such as typical time of transaction, size, beneficiary, location, device, trades, etc., for each customer and internal user, and comparing each new transaction or activity against those of the past, we can give every transaction a risk score.
</p>
<p>
<i>(Article initially published on <a href="https://blog.netguardians.ch/powerful-big-data-analytics-platform-fights-financial-crime-in-real-time">NetGuardians' blog</a>)</i>
</p>
<p>
NetGuardians overcomes the problems of analyzing billions of pieces of data in real time with a unique combination of technologies to offer unbeatable fraud detection and efficient transaction monitoring without undermining the customer experience or the operational efficiency and security in an enterprise-ready solution.
</p>
<p>
When it comes to data analytics, the more data the better, right? Not so fast. That’s only true if you can crunch that data in a timely and cost-effective way.
</p>
<p>
This is the problem facing banks looking to Big Data technology to help them spot and stop fraudulent and/or non-compliant transactions. With a window of no more than a hundredth of a millisecond to assess a transaction and assign a risk score, banks need accurate and robust real-time analytics delivered at an affordable price. Furthermore, they need a scalable system that can score not one but many thousands of transactions within a few seconds and grow with the bank as the industry moves to real-time processing.
</p>
<p>
AML transaction monitoring might be simple on paper but making it effective and ensuring it doesn’t become a drag on operations has been a big ask. Using artificial intelligence to post-process and analyze alerts as they are thrown up is a game-changing paradigm, delivering a significant reduction in the operational cost of analyzing those alerts. But accurate fraud risk scoring is a much harder game. Some fraud mitigation solutions based on rules engines focus on what the fraudsters do, which entails an endless game of cat and mouse, staying up to date with their latest scams. By definition, this leaves the bank at least one step behind.
</p>
<p>
At NetGuardians, rather than try to keep up with the fraudsters, we focus on what we know and what changes very little – customers’ behavior and that of bank staff. By learning “normal” behavior, such as typical time of transaction, size, beneficiary, location, device, trades, etc., for each customer and internal user, and comparing each new transaction or activity against those of the past, we can give every transaction a risk score.
</p>
<h2>Billions of pieces of data</h2>
<p>
To do this effectively means taking into account thousands of pieces of information every time a customer makes a transaction. Multiply that by the number of customers a bank has on its books, and it quickly gets to billions.
</p>
<p>
Such high volumes would overwhelm most platforms, slowing the analytics to an unacceptable speed for the demands of real-time banking. At NetGuardians, we have solved this by using a combination of technologies that allows us to regularly batch process all the data for super-accurate models and supplement these batch models in real time by checking and adding smaller data sets as they arrive. This allows our software to accurately assess huge volumes of transactions in real-time.
</p>
<p>
The technologies we use are:
</p>
<ul>
<li>Apache Kafka</li>
<li>Elasticsearch</li>
<li>Apache Mesos</li>
<li>Apache Spark</li>
</ul>
<p>
All are open-source and run on our proprietary <a href="https://www.niceideas.ch/roller2/badtrash/entry/lambda-architecture-with-kafka-elasticsearch">Lambda architecture</a>-driven platform. Together, they make up a powerful and affordable solution for analyzing every transaction accurately in real time. In fact, our platform catches up to 99% of fraud, with 85 percent fewer false alerts, cutting investigation time by 95 percent compared with alternative rule-based solutions.
</p>
<p>
While this is key, it’s not the best bit.
</p>
<p>
At NetGuardians, we help our customers reap the benefits of cutting-edge and state-of-the-art open-source technologies without them suffering any of the drawbacks. We integrate these technologies, fine-tune and secure them and, critically, we implement enterprise-grade requirements on top. This means banks can use our solution out of the box.
</p>
<h2>Enterprise-Ready Big Data Platform</h2>
<p>
NetGuardians combines all the appropriate technologies in a way to make them work together 100 percent of the time, perfectly fine-tuned and secure, providing a bank everything it requires for an enterprise environment. This includes high availability, data and communication encryption, disaster recovery processes, state-of-the-art authorization, identification and authentication frameworks, single sign on, backup and restore procedures and much more. In this way, banks using our software enjoy the benefits of open source – easy integration and further development/fine-tuning – with the security and resilience of proprietary software.
</p>
<p>
With NetGuardians, banking institutions get the best of both worlds. But the cherry on the cake is that our banks don’t have to do anything. The NetGuardians platform takes care of everything and operates itself automatically, benefitting from strong NoSQL and DevOps genes. And that is unique to us.
</p>
<p>
Should it want to, though, a bank can create its own analytics on top of the open-source components on which the NetGuardians’ platform is built for its own use cases. A bank may want to use our version of Kafka for its own data-streaming use cases, for example, or it can open our 360 vision of the customer and user activities in ElasticSearch and expose that data through a secured API to in-house, third party software. This allows it to use the data for whatever it wants or needs to do - perhaps AML use cases or enriching the CRM application with NetGuardians’ data about customers.
</p>
<h2>The future of finance is real-time payments</h2>
<p>
Typically, many banks access the anonymized data we collect and store on our platform as a financial crime data lake to enrich their own customer 360 views in front office applications with risk indicators and a consolidated view of customers’ activities on their account. This is important because real time payments are growing fast. In 2020, 54 percent of consumers had used real-time payment app PayPal https://www.paymentsjournal.com/real-time-payments-everything-you-need-to-know/. Similar apps such as Venmo and Zelle are also growing fast – with the latter claiming 13 percent of consumers using its app in 2020, up from 1 percent in 2017.
</p>
<p>
While retail payments are important, it’s in business that the big volumes lie and in one survey 80 percent of businesses said they wanted real time banking. Already this is translating into action - in the US, 2020 saw a fivefold increase year on year in financial institutions implementing real-time payments.
</p>
<p>
Such huge growth means banks, big and small, will need affordable fraud detection in real time that can cope with these volumes. For the big banks, the solution will need to scale fast; for the smaller ones, they need a platform that can deliver accurate real-time risk scoring with smaller data sets. NetGuardians, with its unique combination of proprietary and open-source technologies, satisfies both. That is why banks worldwide – from Tier 1 to credit unions and co-ops – are turning to NetGuardians fraud-mitigation software to keep their customers’ cash safe.
</p>
<p>
<i>(Article initially published on <a href="https://blog.netguardians.ch/powerful-big-data-analytics-platform-fights-financial-crime-in-real-time">NetGuardians' blog</a>)</i>
</p>
https://www.niceideas.ch/roller2/badtrash/entry/a-proposed-framework-for-agile
A proposed framework for Agile Roadmap Design and Maintenance
Jerome Kehrli
2021-06-11T04:25:42-04:00
2021-06-11T04:25:42-04:00
<!--<h1>A proposed framework for Agile Roadmap Design and Maintenance</h1>-->
<p>
In my current company, we embrace agility down the line, from the <i>Product Management</i> Processes and approaches down to the Software Development culture.
<br>
However, from the early days and due to the nature of our activities, we understood that we had two quite opposed objectives: on one side the need to be very flexible and change quickly priorities as we refine our understanding of our market and on the other side, the need to respect commitment taken with our customers regarding functional gaps delivery due dates.
<br>
In terms of road-mapping and forecasting, these two challenges are really entirely opposed:
</p>
<ul>
<li>
<b>Strong delivery due dates on project gaps with hard commitment on planning</b>. Our sales processes and customer delivery projects are all but Agile. We know when in the future we will start any given delivery project and we know precisely when the production rollout is scheduled, sometimes up to 12 months in advance. We have most of the tine a small set of <i>Project Gaps</i> required for these projects. Since we need to provide the delivery team with these functional gaps a few weeks prior to the production rollout, it turns out that we have actually <b>strong delivery due dates</b> for them, sometimes 12 months in advance.
</li>
<li>
<b>Priorities changing all the time as our sales processes and market understanding progress</b>. We are an agile company and mid-term and even sometimes short-term focus changes very frequently as we sign deals and refine our understanding of our market, not to mention that the market itself evolves very fast
</li>
</ul>
<p>
These two opposed challenges are pretty common in companies that are refining their understanding of their <a href="https://www.niceideas.ch/roller2/badtrash/entry/the-search-of-product-market">Product-Market Fit</a>. Having to commit strongly on sometimes heavy developments up to more than a year in advance, while at the same time changing the mid-term and short-term priorities very often is fairly common.
</p>
<p>
In this article, I would like to propose a framework for managing such a common situation by leveraging on <b>a roadmap as a communication, synchronization and management tool</b> by inspiring from what we do in my current company (leveraging on some elements brought by Mr. Roy Belchamber - whom I take the opportunity to salute here).
</p>
<p>
There are three fundamental cultural principles and practices that are absolutely crucial in our context to handle such opposed objectives. These three elements are as follows:
</p>
<ol>
<li>
<b>Multiple interchangeable development teams</b>: multiple teams that have to be interchangeable are required to be able to <b>spread the development effort</b> among flexible evolutions - that can be reprioritized at will - and hard commitments - that need to be considered frozen and with a fixed delivery due date.
</li>
<li>
<b>Independent and autonomous development teams</b>: these development team need to be able to work entirely independently and without and friction from any other team. This is essential to have <b>reliable estimations and forecasts</b>. A lot of the corollary principles and practices I will be presenting in this article are required towards this very objective.
</li>
<li>
<b>An up to date and outcome-based Roadmap</b>. Having a roadmap that crystallizes the path and the foreseen development activities in the next 2 years is absolutely key. Such a roadmap is all of an internal communication tool, an external communication support, a management and planning tool.</b>.
</li>
</ol>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/cc46c114-ab4d-4433-b61c-9ad3d3833ea1">
<img class="centered" style="width: 850px;" alt="An Agile roadmap sample" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/cc46c114-ab4d-4433-b61c-9ad3d3833ea1" />
</a>
<div class="centered">
Agile Roadmap Example
</div>
</div>
<br>
<p>
In this article, I intend to present the fundamental principles behind the design and maintenance of such a roadmap that are required to make it a powerful and reliable tool - and not yet another good looking but useless drawing - along with everything that is required in terms of Agile principles practices.
</p>
<!--<h1>A proposed framework for Agile Roadmap Design and Maintenance</h1>-->
<p>
In my current company, we embrace agility down the line, from the <i>Product Management</i> Processes and approaches down to the Software Development culture.
<br>
However, from the early days and due to the nature of our activities, we understood that we had two quite opposed objectives: on one side the need to be very flexible and change quickly priorities as we refine our understanding of our market and on the other side, the need to respect commitment taken with our customers regarding functional gaps delivery due dates.
<br>
In terms of road-mapping and forecasting, these two challenges are really entirely opposed:
</p>
<ul>
<li>
<b>Strong delivery due dates on project gaps with hard commitment on planning</b>. Our sales processes and customer delivery projects are all but Agile. We know when in the future we will start any given delivery project and we know precisely when the production rollout is scheduled, sometimes up to 12 months in advance. We have most of the tine a small set of <i>Project Gaps</i> required for these projects. Since we need to provide the delivery team with these functional gaps a few weeks prior to the production rollout, it turns out that we have actually <b>strong delivery due dates</b> for them, sometimes 12 months in advance.
</li>
<li>
<b>Priorities changing all the time as our sales processes and market understanding progress</b>. We are an agile company and mid-term and even sometimes short-term focus changes very frequently as we sign deals and refine our understanding of our market, not to mention that the market itself evolves very fast
</li>
</ul>
<p>
These two opposed challenges are pretty common in companies that are refining their understanding of their <a href="https://www.niceideas.ch/roller2/badtrash/entry/the-search-of-product-market">Product-Market Fit</a>. Having to commit strongly on sometimes heavy developments up to more than a year in advance, while at the same time changing the mid-term and short-term priorities very often is fairly common.
</p>
<p>
In this article, I would like to propose a framework for managing such a common situation by leveraging on <b>a roadmap as a communication, synchronization and management tool</b> by inspiring from what we do in my current company (leveraging on some elements brought by Mr. Roy Belchamber - whom I take the opportunity to salute here).
</p>
<p>
There are three fundamental cultural principles and practices that are absolutely crucial in our context to handle such opposed objectives. These three elements are as follows:
</p>
<ol>
<li>
<b>Multiple interchangeable development teams</b>: multiple teams that have to be interchangeable are required to be able to <b>spread the development effort</b> among flexible evolutions - that can be reprioritized at will - and hard commitments - that need to be considered frozen and with a fixed delivery due date.
</li>
<li>
<b>Independent and autonomous development teams</b>: these development team need to be able to work entirely independently and without and friction from any other team. This is essential to have <b>reliable estimations and forecasts</b>. A lot of the corollary principles and practices I will be presenting in this article are required towards this very objective.
</li>
<li>
<b>An up to date and outcome-based Roadmap</b>. Having a roadmap that crystallizes the path and the foreseen development activities in the next 2 years is absolutely key. Such a roadmap is all of an internal communication tool, an external communication support, a management and planning tool.</b>.
</li>
</ol>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/cc46c114-ab4d-4433-b61c-9ad3d3833ea1">
<img class="centered" style="width: 850px;" alt="An Agile roadmap sample" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/cc46c114-ab4d-4433-b61c-9ad3d3833ea1" />
</a>
<div class="centered">
Agile Roadmap Example
</div>
</div>
<br>
<p>
In this article, I intend to present the fundamental principles behind the design and maintenance of such a roadmap that are required to make it a powerful and reliable tool - and not yet another good looking but useless drawing - along with everything that is required in terms of Agile principles practices.
</p>
<p>
This article is available as a <a href="https://www.slideshare.net/JrmeKehrli/a-proposed-framework-for-agile-roadmap-design-and-maintenance">Slideshare presentation</a>
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. A Software Development Roadmap </a>
<ul>
<li><a href="#sec12">1.2 Roadmap elements</a></li>
<li><a href="#sec13">1.3 Hierarchy</a></li>
<li><a href="#sec14">1.4 A realistic Roadmap</a></li>
<li><a href="#sec15">1.5 In the next chapters</a></li>
</ul>
</li>
<li><a href="#sec2">2. Prerequisites </a>
<ul>
<li><a href="#sec21">2.1 A. Agile Software Development Teams </a>
<ul>
<li><a href="#sec211">2.1.1 eXtreme Programming </a>
<ul>
<li><a href="#sec2111">2.1.1.1 Small-releases</a></li>
<li><a href="#sec2112">2.1.1.2 Testing, testing and more testing</a></li>
<li><a href="#sec2113">2.1.1.3 On-site customer</a></li>
<li><a href="#sec2115">2.1.1.5 The Planning Game</a></li>
<li><a href="#sec2116">2.1.1.6 XP Takeaways</a></li>
</ul>
</li>
<li><a href="#sec212">2.1.2 Lean Startup</a>
<ul>
<li><a href="#sec2121">2.1.2.1 Pizza Teams</a></li>
<li><a href="#sec2122">2.1.2.2 Feature Teams</a></li>
<li><a href="#sec2123">2.1.2.3 Lean Startup Takeaways</a></li>
</ul>
</li>
<li><a href="#sec213">2.1.3 DevOps</a>
<ul>
<li><a href="#sec2131">2.1.3.1 Infrastructure as Code </a></li>
<li><a href="#sec2132">2.1.3.2 Continuous Delivery</a></li>
<li><a href="#sec2133">2.1.3.3 DevOps Takeaways</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#sec22">2.2 B. The 3 Horizons Framework</a>
<ul>
<li><a href="#sec221">2.2.1 Business economics </a></li>
<li><a href="#sec222">2.2.2 The three Horizon framework from McKinsey</a></li>
<li><a href="#sec223">2.2.3 Three Horizons framework - takeaways</a></li>
</ul>
</li>
<li><a href="#sec23">2.3 C. The Estimation process</a>
<ul>
<li><a href="#sec231">2.3.1 The roles and rituals involves in the estimation process</a></li>
<li><a href="#sec232">2.3.2 Rituals are scheduled</a></li>
<li><a href="#sec233">2.3.3 Now the Estimation Process</a></li>
<li><a href="#sec234">2.3.4 Team Sprint velocity</a></li>
<li><a href="#sec235">2.3.5 Forecasting</a></li>
<li><a href="#sec236">2.3.6 The Estimation process - Takeaways</a></li>
</ul>
</li>
<li><a href="#sec24">2.4 D. Roadmap timeline</a>
<ul>
<li><a href="#sec241">2.4.1 Monthly roadmap update</a></li>
<li><a href="#sec242">2.4.2 Back on Continuous Delivery</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#sec3">3. Conclusions and final notes</a></li>
</ul>
<a name="sec1"></a>
<h2>1. A Software Development Roadmap </h2>
<p>
A <i>good</i> roadmap implies some important characteristics:
</p>
<ul>
<li>Aligns with company strategy, rallies all around this and steers us towards delivering on this strategy</li>
<li>Focuses on delivering customer value & articulating benefits</li>
<li>Excites our customers about our company & product direction</li>
<li>Reflects what we have learned, over time, as an organisation</li>
<li>Does not pretend to have all the answers, neither needs to</li>
</ul>
<p>
If it is done well, <b>a roadmap is a strategic communication tool, a statement of intent and direction</b>.
<br>
A good roadmap has to set a clear direction and simultaneously embrace uncertainty inherent in product development
</p>
<p>
Some pitfalls have to be avoided when designing a good roadmap:
</p>
<ul>
<li>Excessive granularity – too much focus on detail and dates means it will inevitably soon be inaccurate or even obsolete!</li>
<li>Mistakenly thinking every item in the roadmap demands upfront design and estimation - this is impossible and wasteful. Only the short-to-mid-term elements have to be estimated accurately.</li>
<li>Believing each stakeholders must personally value every item</li>
<li>Ensuring our roadmap is not conflated with our product release plan!</li>
</ul>
<a name="sec12"></a>
<h3>1.2 Roadmap elements</h3>
<p>
The essential elements of a roadmap are as follows:
</p>
<ul>
<li>A good roadmap starts with a <b>vision</b> of where we are going, guides us there and explains the stops along the way. The vision is the guiding principle.</li>
<li><b>Broad timeframes</b> avoid overcommitment - it's the sequence that matters (now, next, future). As we move along the sequence, accurate estimations become less important. Down to the point where we don't really care about the time that might take something on which we are not going to be working before several years.</li>
<li>Focus on <b>outcomes</b> not outputs. Themes are not granular features and functions.</li>
<li>What <b>goals</b> will our product accomplish? What outcomes?</li>
<li>Protects against claims of broken promises by explaining that changes can happen.</li>
</ul>
<p>
These elements would be illustrated as follows for instance for a <i> RIAO - Rich Internet Application Organizer</i> (as introduced in my <a href="https://www.slideshare.net/JrmeKehrli/from-product-vision-to-story-map-lean-agile-product-shaping">"Lean / Agile Product shaping" slideshare</a> and detailed in my <a href="https://www.slideshare.net/JrmeKehrli/introduction-to-modern-software-architecture-249211724">"Introduction to Modern Software Architecture" slideshare.</a>):
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/475e949d-312d-4296-81f0-13e241879d3d">
<img class="centered" style="width: 850px;" alt="Roadmap elements" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/475e949d-312d-4296-81f0-13e241879d3d" />
</a>
<div class="centered">
Roadmap Elements<br>
(inspired from Roy Belchamber @ NetGuardians)
</div>
</div>
<br>
<a name="sec13"></a>
<h3>1.3 Hierarchy</h3>
<p>
In terms of hierarchy, we have to wonder why we are doing this, why it supports our strategy, what customer problems it will solve, and finally how to solve it. <br>
The <b>Roadmap</b> is about <i>what</i> customer problem we're about to solve. And the <b>Product Backlog</b> is about how will we solve them and what solutions do we need to implement to solve these. <br>
The roadmap is a product management concern, while the backlog is an R&D concern.
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d63adcea-c7c7-44e1-b7e7-2e85a84d2928">
<img class="centered" style="width: 850px;" alt="The why / what / how to to roadmapping" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d63adcea-c7c7-44e1-b7e7-2e85a84d2928" />
</a>
<div class="centered">
Roadmap hierarchy<br>
(© Roy Belchamber @ NetGuardians)
</div>
</div>
<br>
<a name="sec14"></a>
<h3>1.4 A realistic Roadmap</h3>
<p>
Let's look at something more realistic.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/cc46c114-ab4d-4433-b61c-9ad3d3833ea1">
<img class="centered" style="width: 850px;" alt="An Agile roadmap sample" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/cc46c114-ab4d-4433-b61c-9ad3d3833ea1" />
</a>
<div class="centered">
Realistic Agile Roadmap
</div>
</div>
<br>
<p>
This example is a slightly reworked (anonymized and generified) version of the roadmap we use today (April 2021) in my current company. It is really an instantiation of all the fundamental principles expressed above and we will be using it throughout the remainder of this article to illustrate all key practices required to make it a living and yet fairly useful tool.
</p>
<p>
A first important thing is to be noted.
<br>
As we move along the timeline of the roadmap, the confidence and certainty diminishes down to the point where, in 2 years from now, the roadmap is more a collection of long term development ideas and in no way any kind of actual commitment on working on any of these.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/78f69efe-5730-490e-a5dd-5927f64ce804">
<img class="centered" style="width: 850px;" alt="Uncertainty on the roadmap" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/78f69efe-5730-490e-a5dd-5927f64ce804" />
</a>
</div>
<br>
<a name="sec15"></a>
<h3>1.5 In the next chapters</h3>
<p>
In the next chapters of this article, we will be covering all the fundamental concepts, principles and practices that need to be understood and adopted to make this roadmap a useful communication and planning tool.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/c06fba7b-d0ed-412b-b785-ae96832f6e9a">
<img class="centered" style="width: 850px;" alt="Next chapters" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/c06fba7b-d0ed-412b-b785-ae96832f6e9a" />
</a>
</div>
<br>
<a name="sec2"></a>
<h2>2. Prerequisites </h2>
<p>
In order to be able, to design and maintain such a roadmap, some organizational principles as well as some product management principles are required.
<br>
Then, if one intends such a <i>Product Roadmap</i> to be more than a fancy marketing tool, really a strong communication, management and planning tool, it's essential that the forecasts are as reliable and realistic as possible. And this is fairly complicated since it relies on the ability of the individual development teams working on a specific topic to be able to <b>work autonomously, independently and without any friction</b> with other teams. And this in turn requires an in depth adoption of <i>Agile Principles and practices</i> not only in the development team but at product Management level as well.
</p>
<p>
In this chapter, we will be looking at each and every of these prerequisites.
</p>
<a name="sec21"></a>
<h3>2.1 A. Agile Software Development Teams </h3>
<p>
First, if we want to be in a situation where we can respect the roadmap timeline and have reliable forecasts, the multiple development teams working in parallel on the multiple streams need state-of-the-art agile culture, principles and practices.<br>
The whole problem is <i>How to build a team with a culture, an organization and a set of principles and practices that make estimations possible and forecasting reliable and accurate ?</i>
<br>
The answer is <i>by adopting a state of the art Agile Engineering methodologies</i>.
</p>
<p>
The schema below is a slide that I've designed when I was a consultant carrying on digital transformation projects in big corporations. Very often I was meeting IT teams that were not agile at all. I needed a slide to explain to them what are the prerequisites if one want to embrace digital transformation.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/80f83d69-2302-40f6-a088-f862c09aba59">
<img class="centered" style="width: 800px;" alt="Agile prerequisites" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/80f83d69-2302-40f6-a088-f862c09aba59" />
</a>
</div>
<br>
<p>My message here was as follows: <br>
<i>"Look, if you want to go digital, if you want to come up with digital products and be able to develop and adapt them at the pace that is required in the digital world, it's going to be fairly difficult if you're not an agile Corporation, if you haven't <b>scaled agile principles and practices</b> at the level of the whole organization.
<br>
And then scaling agile, if you haven't embraced the <b>Lean Startup</b> methodology, and if you don't have company wide monitoring and improvement approaches such as <b>Kanban and Kaizen</b>, will be difficult.
<br>
Then doing Lean Startup, Kanban, and Kaizen if you don't have a <b>state of the art Agile software development methodology</b>, and if you haven't embraced <b>DevOps</b> principles, will be difficult.
<br>
And finally doing Agile and DevOps, if you're not state of the art regarding <b>eXtreme Programming</b> principles and practices will be difficult."
</i>
</p>
<p>
Now, if you want to be in a situation where you have independent and autonomous teams which can benefit from reliable estimations, and eventually come up with reliable forecasting abilities, then you have to have raised agility adoption to this level in your company.
</p>
<p>
Another way to illustrate, these agile methodologies such as DevOps, Lean Startup and others would be <a href="https://www.niceideas.ch/roller2/badtrash/entry/periodic-table-of-agile-principles">The Periodic table of Agile principles and practices</a>.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/eaa33988-0916-4c98-93d8-efce52c72bea">
<img class="centered" style="width: 850px;" alt="Periodic Table of Agile Principles and practices" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/eaa33988-0916-4c98-93d8-efce52c72bea" />
</a>
<div class="centered">
Periodic Table of Agile Principles and Practices
</div>
</div>
<br>
<p>
All the principles and practices with a red border are effectively needed for accurate planning and road mapping. All these practices are required to have independent and autonomous teams giving you the ability to parallelize development epics and be in a position where you can accurately estimate the velocity of these teams, in such a way that eventually you're able to do reliable estimations and forecasting on the items for which you need to have an accurate due date.
<br>
Now in reality the situation is more complex since at the end of the day all the practices from the table heavily depend on one another so in the end you need to have embraced most of them.
</p>
<p>
We will now review the most essential practices with direct impact on planning and forecasting abilities from these various methodologies.
<br>
We can't review all of the required practices, of course, but at least those that I believe are the most essential ones from eXtreme Programming, Lean Startup and DevOps that one needs to adopt, if one want to be able to reach the ultimate goal, which is having autonomous and independent team, giving one the ability to paralyze development items, while still being able, within teams, to have accurate planning capacity and estimations, and eventually reliable forecasting.
</p>
<a name="sec211"></a>
<h4>2.1.1 eXtreme Programming </h4>
<p>
The eXtreme Programming practices are as follows. This representation is interesting since it shows that even only within XP, all these practices depend from each others.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/be3fc61c-68dc-4d23-b356-3560430dda38">
<img class="centered" style="width: 600px;" alt="eXtreme Programming Practices" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/be3fc61c-68dc-4d23-b356-3560430dda38" />
</a>
<div class="centered">
XP Practices
</div>
</div>
<br>
<p>
Let's discuss more in detail 4 essential principles when it comes to planning: Small-releases, Testing, On-Site Customer and the Planning Game.
</p>
<a name="sec2111"></a>
<h5>2.1.1.1 Small-releases</h5>
<p>
And the first part I want to mention is XP's <i>Small Releases</i> principle, which DevOps streamlines as <i>Continuous Delivery</i>. The fundamental idea behind it is that if you want to master your release process, you have to release as often as possible, for multiple reasons. First, because if you release as often as possible, the releases are small. And as a result, your chances of mastering the process are much higher than if you have rare and then very large releases.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d591f1d9-6449-4e57-9826-dbf4d50ede6f">
<img class="centered" style="width: 800px;" alt="XP Small Releases" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d591f1d9-6449-4e57-9826-dbf4d50ede6f" />
</a>
<div class="centered">
XP Practices
</div>
</div>
<br>
<p>
Having smaller gaps and smaller changes in your releases significantly reduces the risk inherent to the release.
<br>
But there's another reason behind it. If you ask an engineer to do something very often, he will automate it. This is not necessarily the case if an engineer does only a few releases a year, why would one bother automating it?
<br>
But if you ask your development team, your engineering team, to release at the end of every single sprint a <i>production-ready</i> and <i>shippable version</i> of the product, then your development team will automate the release process. That's what engineers do.
</p>
<a name="sec2112"></a>
<h5>2.1.1.2 Testing, testing and more testing</h5>
<p>
The next topic is about testing. XP's practitioners are always saying that one has to invest the 20% of time required to reach 80% coverage of the cyclomatic complexity of the code or branch / line coverage, and not more (since the remaining 20% would require 80% of the investment) .
<br>
But if one wants to go to continuous delivery, if one wants to be in a situation where one is able to entirely automate the release process, then the 80% target is not sufficient anymore, one has to target 100% coverage.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/499760ff-0cea-4723-aef0-ab346dbd6422">
<img class="centered" style="width: 800px;" alt="TDD" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/499760ff-0cea-4723-aef0-ab346dbd6422" />
</a>
</div>
<br>
<p>
And this is doable with different types of test. Covering more than 80% of the code complexity with unit test only is impossible. But integration tests enable to go beyond, for instance typically 90%. Then if one is able to have the product entirely built and deployed automatically on a production like environment, then one is able to implement <i>end-to-end</i> / functional tests using for instance protractor, Selenium or whatever. And with end-to-end / functional tests, you are able to cover the remaining 10% to reach nearly 100% coverage with your whole automated tests suite.
<br>
And this is essential if one wants to be able to have reliable forecasting abilities. Because stop the world releases are killing it. It's fundamentally unpredictable how much time it takes to complete acceptance tests, fix the remaining issues and eventually release the software if one waits for periodic releases.
<br>
One needs to entirely automate non-regression tests <b>and acceptance tests</b> using functional, integration and unit tests. And this will be absolutely key to reach continuous delivery.
</p>
<p>
But there's something else that needs to be accounted. In this next chart, we're looking at three typical situations of a feature development. The first situation is without any automated test, the second is with some tests implemented after the development is done, and the last situation is when embracing TDD.
<br>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/tdd-test-driven-development-is#sec45">This chart is explained in detail here in a previous article.</a>
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/c625635e-a891-4769-9a58-73b67523dd38">
<img class="centered" style="width: 800px;" alt="3 scenarii" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/c625635e-a891-4769-9a58-73b67523dd38" />
</a>
</div>
<br>
<p>This is interesting. We have the illusion that skipping the development of tests makes us gain time. And if you look at the blue box, it does indeed make us gain time in terms of pure development time. But the time lost after the development phase exceeds greatly the time gained by skipping tests.
<br>
Writing some tests after implementing the code really helps, we can see that the development time takes longer but is really compensated by the time gained on debugging and manual testing. Not to mention the formidable documentation that unit and integration tests form.
<br>
But the striking approach is when we're putting in place TDD - enabling us to reach a close to 100% coverage of the code with our automated tests - where we spend most of the time doing pure development, which takes longer of course, but the return on investment is absolutely brilliant.
</p>
<p>
And the interesting thing when it comes to planning and forecasting is that the blue box can be estimated. This is typically what we estimate, when we do the planning game with Story Points. It's possible to estimate the time it takes to develop a feature, including the tests. On the contrary the time that it would take to debug it, manually testing it or re-understanding it is fundamentally unpredictable.
<br>
<b>TDD brings most of the development activities back to something that can be estimated</b>, which is the pure development time. And this is not optional. If one wants to be able to have reliable forecasts and reliable estimations, one needs to bring most of the development activities back to something that can be estimated. This is absolutely precious with TDD.
</p>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/tdd-test-driven-development-is">For more information on TDD, refer to my previous article.</a>
</p>
<a name="sec2113"></a>
<h5>2.1.1.3 On-site customer</h5>
<p>
XP insists on the need to have an <i>on-site customer</i> to streamline the development pace by being able to answer questions, provide refine specifications and, importantly, do acceptance testing continuously as development tasks are being completed.<br>
The notion of on-site customer - having someone with a true business expertise - is replaced in scrum with the notion of <b>product Owner</b>.
<br>
If one wants the development teams to be independent and autonomous then one needs to make sure that this team can work on a feature peacefully and in isolation down to production readiness without any interruption and without any dependency on an external actor.
<br>
The product owner enables to do <i>continuous acceptance testing</i> thus avoiding stop the world events and streamlines acceptance testing as part of the development activities.
<br>
For this reason mostly, the on-site customer - or the scrum product owner - is crucial.
</p>
<a name="sec2115"></a>
<h5>2.1.1.5 The Planning Game</h5>
<p>
And finally, the planning game. You need to think of it this way: imagine you're looking at a big stone somewhere in a field. Estimating <i>out of the blue</i> the weight of the stone is very difficult. Estimating absolute figures is a very difficult game.
<br>
But answering to another question which would be "is the stone heavier or lighter than this other stone ?" is a completely different story. It's a comparison game. And that's surprisingly easier. Finding out whether a stone is somewhat bigger than another one, but also somewhat smaller than a third one, is a much easier game.
<br>
Story points enabled to transform a very difficult estimation game into a much more a much easier comparison game.
<br>
And that's what the planning poker with Story Points is all about.
</br>
It's about finding ways to transform a difficult problem in an easier problem. And after a while, the team becomes quite good at it
</p>
<p>
And if you're able to accurately estimate a task in story points using the comparison game, then you're able to compute how many story points a team is able to do in a sprint. And if you know how many story points a feature cost and what is the <i>sprint velocity</i> of a team, then you can compute how many sprints does the team need to implement this very feature.
<br>
And with this, you can do forecasting and planning ... if and only if the team is able to work independently, autonomously, without any synchronization point, without stop the world events and without any friction coming from the need to collaborate with another team.
<br>
And we will see what these teams are to enable such independence.
</p>
<p>
More information on the Planning game in <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-planning-tools-and-processes">a previous article</a> and further in this very article.
</p>
<a name="sec2116"></a>
<h5>2.1.1.6 XP Takeaways</h5>
<p>
Summing things up:
</p>
<ul>
<li>
First, the product owner (i.e. <i>on-site customer</i>) - enables the team to be autonomous and work on its own without interruption by answering questions as they pop up and, more importantly, running acceptance test continuously, so that a developer can consider his task being finished as soon as possible before he moves to the next one.
</li>
<li>
Small releases - which we will see as <i>Continuous Delivery</i> in the DevOps methodology - enables to avoid periodic releases as <i>stop the world</i> events, which would break both the autonomy and the pace of the team by forcing synchronization points everywhere on the roadmap timeline.
</li>
<li>
TDD enables to have reliable forecasting by bringing most of the development team activities to something that can actually be estimated and forecasted, which is code implementation time. Debugging sessions, testing and re-understanding the code are activities that are fundamentally impossible to estimate.
</li>
<li>
And finally, the planning poker estimation game enables to have reliable estimation and forecasting abilities, surprisingly and counter-intuitively much more than traditional waterfall.
</li>
</ul>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/e1b9f1dd-afc9-4166-9abe-c557a90dc41e">
<img class="centered" style="width: 850px;" alt="XP Takeaways" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/e1b9f1dd-afc9-4166-9abe-c557a90dc41e" />
</a>
<div class="centered">
XP takeaways
</div>
</div>
<br>
<a name="sec212"></a>
<h4>2.1.2 Lean Startup</h4>
<p>
Eric Ries presents Lean Startup as the <i>Build - Measure - Learn</i> loop.
<br>
Steve Blank presents it as the <i>Four Steps to the Epiphany</i> process as follows. We won't be discussing Lean Startup in depth today but I would want to discuss <i>two Lean Startup Practices</i> that are crucial to shape autonomous and independent, and yet interchangeable and equivalent teams: <b>Pizza Teams</b> and <b>Feature Teams</b>.
<br>
One might refer to <a href="https://www.niceideas.ch/roller2/badtrash/entry/lean-startup-a-focus-on">my previous article on lean startup</a> to get an overview of all Lean Startup principles and practices.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/81a7edce-4619-452d-8364-5cbb5a0c83a4">
<img class="centered" style="width: 850px;" alt="Lean Startup" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/81a7edce-4619-452d-8364-5cbb5a0c83a4" />
</a>
<div class="centered">
Lean startup - The four steps to the Epiphany.
</div>
</div>
<br>
<p>
Let's discuss shortly these two important principles related to Agile Teams.
</p>
<a name="sec2121"></a>
<h5>2.1.2.1 Pizza Teams</h5>
<p>
I discussed in length Pizza Teams in <a href="https://www.niceideas.ch/roller2/badtrash/entry/lean-startup-a-focus-on#sec341">my previous article about Lean Startup</a> so I will only recap a few things here.
</p>
<p>
The reason for small teams is that the more people you have in your team, the more you explode the number of <i>one-to-one</i> communication channels. So one needs to keep a team being sufficiently small to make it so that everyone is able to understand what everyone else is working on.
<br>
One needs to keep the team sufficiently small to have an efficient organization within the team, and so the ability to have people collaborating together, for instance, the UI UX expert with the backend developer, the DevOps engineer, etc.
<br>
But the teams needs to be sufficiently large to enable efficient brainstorming, the ability to generate new ideas, interchangeability of essential resources, etc.
<br>
In my current company, we believe that the ideal size for our <i>Feature Teams</i> is between 5 and 8 engineers.
</p>
<a name="sec2122"></a>
<h5>2.1.2.2 Feature Teams</h5>
<p>
Just as for Pizza Teams, I discussed in length <i>Feature Teams</i> in <a href="https://www.niceideas.ch/roller2/badtrash/entry/lean-startup-a-focus-on#sec342">my previous article about Lean Startup</a> so I will only recap a few things here.
</p>
<p>
The key point with Feature Teams is to have teams that are independent and autonomous as possible without any synchronization needs or friction with any other team. This is essential to guarantee that the team cam deliver a feature from A to Z on its own, where Z is the production rollout or release. This in turn is key to have reliable estimations and eventually forecasting abilities.
<br>
Organizing an R&D department with Feature Teams is striking in this regards.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/4ba88a93-48b5-4953-99cf-c7f3baa3d3dd">
<img class="centered" style="width: 850px;" alt="Feature Teams" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/4ba88a93-48b5-4953-99cf-c7f3baa3d3dd" />
</a>
<div class="centered">
Feature Teams<br>
(Source : Large Scale Scrum : <a href="http://less.works/">http://less.works/</a>)
</div>
</div>
<br>
<p>
To understand this, consider the following.
<br>
Most of the time, the way software engineering development projects or IT departments are organized in companies is by having component teams. In the component teams model, one team is taking care of the UI, another team takes care of developing the back-end, there is perhaps a data science team, another team takes are of data management and database infrastructure, and so on.
<br>
These multiple teams have to collaborate to develop a feature or to implement an evolution on the set of products that they develop or maintain. And that's where the problem is, every single team cannot go faster than the slowest of all the teams on which they all have a dependency.
<br>
Imagine that each and every feature requires some changes in the database model. And these can only be done by the database team. Then no team can go faster than the database team because they all have to wait on the database team to finally implement the changes to the database model that they require.
<br>
Components team are killing the performance of the IT development department or project as a whole.
</p>
<p>
A feature team on the other hand is able to implement a feature from A to Z, from refined specifications down to production deployment, entirely independently and autonomously.
<br>
This can only work if you have within the feature team all the competencies that are required for that: developers, UI/UX experts, data scientists, QA engineers, DevOps engineers - not necessarily to operate the software in production, but to <b>automate</b> the deployment and operation of the software in production - and so forth.
<br>
And since these feature teams are able to work on a feature entirely independently and autonomously, then <b>if you know the velocity of your feature team, you know how long it will take it to implement a feature</b> that you have been able to estimate.
<br>
Feature teams are multiplying the performance of the whole development or software development organization by several orders of magnitude. And the side benefit is they are autonomous and independent by design.
</p>
<p>
Just a little note about component teams, people are sometimes confusing component teams and product teams. While it is not acceptable to have component teams (one team working on the UI, another team working on the back end, another team working on the research, yet another working on the database and so on=, it is on the contrary crucial to have product teams.
<br>
Component teams are killing performance by introducing strong dependencies between teams.
<br>
But having a Feature Team linked not to a component, but to a product makes a lot of sense. We want to leverage and develop <b>business expertise of the feature team</b>. And this can't happen if a feature team is moving all the time from one product to another.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a9f2679f-e8a0-47ce-91bf-0eef1db8e081">
<img class="centered" style="width: 850px;" alt="Feature Teams" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a9f2679f-e8a0-47ce-91bf-0eef1db8e081" />
</a>
<div class="centered">
Product Teams
</div>
</div>
<br>
<p>
A feature team is fundamentally linked to a product or perhaps a consistent set of products (or product line), because that's the product they will start to be familiar with in terms of business understanding.
<br>
And that is essential to avoid wasting time. With experience on a product, the team can become even more autonomous and slowly reduce its dependency on the product owner to be there and working with the team all the time to understand what it has to do in terms of business requirements.
</p>
<a name="sec2123"></a>
<h5>2.1.2.3 Lean Startup Takeaways</h5>
<p>
Summing things up:
</p>
<ul>
<li>
First, a feature team is an independent and autonomous team that's able to implement a feature from A to Z. And the fundamental idea behind that is that as soon as a team has any dependency on another team, its estimation and forecasts are simply not reliable anymore. Multi-team consolidated planning and forecasting is a much more difficult problem.
<br>
<i>From A to Z</i> means that the team needs to have everything it takes to carry on its mission, 3rd level support, documentation, maintenance, automated test development, code reviews, IT testing, acceptance testing with the product owner, releasing, continuous delivery, continuous deployment, deployment and delivery automation, everything... And the <i>DoD - Definition Of Done</i> on a task or epic has to account all these elements.
<br>
And the takeaway here is that because the team is autonomous and independent, its planning and forecasts are accurate and reliable.
</li>
<li>
The reason you want to have many of these independent teams is that you want to be able to work on different things in parallel. You want to have a fair share of spread between the project gaps and the short term development, while retaining the ability to work on long term topics.
<br>
And how you want to do this if you don't have different teams able to work on different things at the same time?
<br>
So you want to have multiple feature teams in your project development organization to be able to have different core focuses at the same time.
</li>
<li>
Finally, because a feature is being given to one autonomous team, and that team is able to work without any friction without an interruption, and more importantly, without any stop the world event, the team's velocity calculation enables to do reliable forecasting.
<br>
As a side note, something that's important to understand as well is that the development team should never ever be exposed to customers. If all the time the team has to answer customer requests then its performance will critically suffer. First and second levels of Support need to be external to R&D.
</li>
</ul>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/bd19e59e-8e9c-4194-8663-901be7cd97ac">
<img class="centered" style="width: 850px;" alt="Lean Startup Takeaways" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/bd19e59e-8e9c-4194-8663-901be7cd97ac" />
</a>
<div class="centered">
Lean Startup Takeaways
</div>
</div>
<br>
<a name="sec213"></a>
<h4>2.1.3 DevOps</h4>
<p>
At the very root of DevOps is the wall of confusion between developers and operators. And the wall of confusion is a crystallization of the fact that developers and operators have fundamentally different objectives, and fundamentally different cultures.
<br>
A developer is challenged to deliver new functionalities to production as fast as possible. An operator, on the other hand, has the fundamental mission of maintaining the production stability, which is precisely what a developer pushing changes all the time is compromising. These two different roles in an organization have entirely different and completely opposed objectives.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/7fc925f4-e338-4fc4-87b8-a53c6fd1723d">
<img class="centered" style="width: 850px;" alt="DevOps" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/7fc925f4-e338-4fc4-87b8-a53c6fd1723d" />
</a>
<div class="centered">
DevOps
</div>
</div>
<br>
<p>
Interestingly, the web giants have built organizations making this wall of confusion nearly vanish. DevOps is a lot about understanding how traditional industries can inspire from the web giants to streamline the interactions, and smoothen the relationship between developers and operators.
<br>
And DevOps relies on three pillars:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/b8141259-13e3-48ab-87a7-0df32674b057">
<img class="centered" style="width: 500px;" alt="DevOps pillars" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b8141259-13e3-48ab-87a7-0df32674b057" />
</a>
<div class="centered">
DevOps pillars
</div>
</div>
<br>
<p>
I described these principles and practices in length in <a href="https://www.niceideas.ch/roller2/badtrash/entry/devops-explained">my previous article on DevOps</a> so I wont be repeating much more here.
<br>
I will just emphasize a little two aspects that are utmost essential for planning and estimations that are <i>Infrastructure as code</i> and <i>Continuous Delivery</i>
</p>
<a name="sec2131"></a>
<h5>2.1.3.1 Infrastructure as Code </h5>
<p>
Again, this is very much detailed in my previous article so have a look at the section on <a href="https://www.niceideas.ch/roller2/badtrash/entry/devops-explained#sec2">Infrastructure as Code.</a>
</p>
<p>
The reason why <i>Infrastructure As Code</i> and <i>Continuous Delivery</i> are essential to planning and forecasting is because both enable to automate entirely the deployment and release process - the deployment process in case of a cloud or SaaS application, and the release process in case of an application deployed on premise.
<br>
Automating release and/or deployment is crucial since we don't want <i>stop the world</i> events. We want a feature team to be able to implement changes on the software, release them and push them to a customer, without any friction and without any contention coming from other teams that have different speed or timelines. And this can only work if the whole release or deployment process is entirely automated. And at the end of the day, this is what DevOps is about.
<br>
We have all the possible tools today to automate machine provisioning, system configuration, application deployment, etc.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a58fd379-8035-450b-9ea3-60132c5cef02">
<img class="centered" style="width: 850px;" alt="IaaS" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a58fd379-8035-450b-9ea3-60132c5cef02" />
</a>
<div class="centered">
Infrastructure as Code
</div>
</div>
<br>
<p>
Automating all of this is difficult, I'm not saying it's easy. it's everything but easy when you implement features that involve significant technology changes to keep up to date the automated deployment process, the automated release process, etc. At the end of the day, there is a reason why <i>courage</i> is the most essential value in eXtreme Programming.
<br>
But again, you don't have much of a choice. <b>That's the only way you can make teams independent from each other and avoid stop the world events</b>. And 20 years ago, that would have been crazy. But with the tools we have today, with virtual machines, Docker containers, Ansible, Chef, Puppet, with everything you can build around these tools to make it so that you click a button and the release is entirely automated with all the tests being executed, etc. this can definitely be done. And it has to be done.
<br>
The <i>Return On Investment</i> in building and maintaining such automation infrastructure is absolutely striking. It enables a team to deploy to production (or create a release) independently from every other team. And this is absolutely key when planning and forecasting matters.
</p>
<a name="sec2132"></a>
<h5>2.1.3.2 Continuous Delivery</h5>
<p>
The fundamental idea behind continuous delivery is that doing deployments is difficult. And because it's difficult, one has to do it as often as one can.
<br>
There are two reasons for this:
</p>
<ol>
<li>
The first reason is that the more often you release, the smallest the changeset will be. And the smallest the changeset is, the more you master it, the more you can control it and the smallest the inherent risks are.
</li>
<li>
But there's another reason: if you ask an engineering team to do something all the time, they will automate it, that's the way engineers work. They will invest the time to make it so that the next time they need to do something it's as easy as pushing a button. And that's absolutely key, because if you automate your release and production rollout processes, then you can do it as often as you can.
</li>
</ol>
<b>A little note about <i>on premise</i> deployment.</b>
<p>
If you want to release an application, for instance at every end of sprint or at the end of every feature and push it to a customer, then internally you have to organize the team in such a way that it makes it possible to release at every end of sprint a shippable, production ready version of the product.
<br>
Releasing the product and deploying it to any given customer has to be a product management decision. As far as the development team or the R&D is concerned, every single sprint has to finish with a production ready, shippable version. Period.
<br>
For this to be possible, you have to be in a situation where if one day you push, for instance, version 7.3.4 to a customer, then then a few months or years later you release a 9.46.5, you need to be able to push that upgrade to that very customer automatically. For this you have to build a framework that enables you to apply these upgrades entirely automatically. It should be possible all the time to push upgrades automatically. If you don't do that, if applying an upgrade to a customer involves manual steps, manual data migration, manual configuration migration, etc. then you're screwed because it would force you to potentially maintain all the individual versions you have in production at your customers. And that would kill the performance of your team with its ability to focus on the last development version only.
</p>
<p>
In short, Continous delivery is key for planning!
</p>
<ul>
<li>You don’t want to loose time on Deployment</li>
<li>You want deployment to be automated</li>
<li>You want deployment and production release to happen without you worrying about it or even only noticing</li>
<li>You want to be able to develop on one single branch, the latest development branch, and push any version to any customer thanks to the data migration framework maintained along the software.
<li>it's key to avoid multiple team synchronization</li>
</ul>
<p>
I won't detail continuous delivery any further here and would refer to <a href="https://www.niceideas.ch/roller2/badtrash/entry/devops-explained">my article on DevOps</a>, specifically the sections about:
</p>
<ul>
<li><a href="https://www.niceideas.ch/roller2/badtrash/entry/devops-explained#sec34">Continuous Delivery deployments</a></li>
<li><a href="https://www.niceideas.ch/roller2/badtrash/entry/devops-explained#sec35">Zero Downtime Deployments</a></li>
</ul>
<a name="sec2133"></a>
<h5>2.1.3.3 DevOps Takeaways</h5>
<p>
Summing things up:
</p>
<ul>
<li>
You have to be in a situation where releasing a new version of the product or pushing it to the SaaS of cloud environment is a <i>push-a-button</i>, anecdotal event. You don't want to lose time on releasing or deploying in production because that would mean you have <i>stop the world</i> events in your entire development department, killing the ability of your teams to work independently on their own feature.
<br>
If every three months or every two months or even every month a team has to stop everything because a release process is going on, it's impossible to respect the estimations and forecasts.
</li>
<li>
Automated tests are essential. You want to be in a situation where you push a button to deploy the software to production. This only works if you have a complete suite of automated tests and end-to-end tests that ensure the software is working 100% before it's automatically pushed to a production environment or automatically released.
</li>
</ul>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/6bac9d55-d610-4110-a16d-cccc69592c38">
<img class="centered" style="width: 850px;" alt="DevOps Takeaways" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/6bac9d55-d610-4110-a16d-cccc69592c38" />
</a>
<div class="centered">
DevOps Takeaways
</div>
</div>
<br>
<a name="sec22"></a>
<h3>2.2 B. The 3 Horizons Framework</h3>
<p>
Your might have noticed these H1, H2 and H3 notions on the roadmap. It's now time to explain this. This comes from McKinsey's <b>3 Horizons framework</b>
<br>
In order to introduce the three horizons framework by McKinsey, I need to start by explaining a few concepts of <i>Business Economics</i>.
</p>
<a name="sec221"></a>
<h4>2.2.1 Business economics </h4>
<p>
So let's start by looking at the following charts:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/f2ed9fee-290e-4e53-b547-d39a9cbbac0f">
<img class="centered" style="width: 850px;" alt="Business Economics" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/f2ed9fee-290e-4e53-b547-d39a9cbbac0f" />
</a>
</div>
<br>
<p>
On the left here, we have what we call the <i>Technology S Curve</i>. And it's quite typical in product development stories with a strong innovation dimension.
<br>
The <i>technology S curve</i> illustrates that when you invent a new product, when you invent a new technology or come up with a new offering for specific market, the progresses you make at first from a technology standpoint are huge. But after a while, continuing to invest more and more in your product or your technology doesn't make a whole lot of sense. Because after a while, the amount of investment you have to consent to develop your technology further is exploding. After a while, you have reached the maximum you can do with your technology or your innovative product. And it doesn't make a whole lot of sense at that point to keep investing in it, you better move your idea to the next level, or look at adjacent segments, or even entirely new products.
</p>
<p>
And this is shown pretty well by the <i>technology profit curve</i> on the right as well, which is saying basically the same thing, but from a different perspective.
<br>
When you have a new idea, when you put a new product on the market, in a real <a href="https://www.niceideas.ch/roller2/badtrash/entry/the-search-of-product-market">Product-Market fit</a> situation, the money you make out of your product gets very high quite early. You make very fast a lot of profits. But after a while, this technology is being disrupted in its turn - you have competitors offering the same idea, you have alternative offering coming up, and your profit will go down. And when this happens, you have to find the next level investment, the next level innovation in your idea or develop something completely different. And again this is quite typical in companies developing product with a strong innovation dimension.
</p>
<a name="sec222"></a>
<h4>2.2.2 The three Horizon framework from McKinsey</h4>
<p>
This is at the root of the three horizons framework from McKinsey.
<br>
McKinsey is basically saying that, in order to keep developing your company, and perhaps your solution or your technology, you have to be aware, monitor, foresee and manage these situations. If you're working on your current technology, you have to be aware that it has a deadline. After a while it doesn't make a whole lot of sense to keep investing in what you're doing today. You have to identify your next idea, you have to identify the next level evolution to enhance your idea or technology that will make you reach the next level.
<br>
And also, you have to keep in mind that eventually you might need to come up with something completely new to keep developing your company further.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/20384e68-12e2-41ed-963a-9766d291cb59">
<img class="centered" style="width: 850px;" alt="Three Horizons Framework" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/20384e68-12e2-41ed-963a-9766d291cb59" />
</a>
<div class="centered">
Three Horizons Framework
</div>
</div>
<br>
<p>
This reads as follows:
</p>
<ul>
<li><b>Horizon 1</b> is about maintaining and strengthening your core business.</li>
<li><b>Horizon 2</b> is about expanding your offering further for new opportunities or emerging markets. </li>
<li><b>Horizon 3</b> is about genuinely new businesses, competencies and possibilities, perhaps on top of your current technology, but likely on something completely new, yet related to your <i>product vision</i>, the next level idea. </li>
</ul>
<p>
The best example of that is Uber. They came up with their initial application very fast. And after a while, developing the Uber application itself didn't make much sense anymore. It was working perfectly. And it started to be challenged by competitors offering. So Uber came with Uber Eats leveraging on their technology to provide a completely new product on a new market.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/623ba280-d810-4215-9a99-aa73322ec9d6">
<img class="centered" style="width: 850px;" alt="Three Horizons description" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/623ba280-d810-4215-9a99-aa73322ec9d6" />
</a>
</div>
<br>
<p>
Again:
</p>
<ul>
<li>
The first horizon involves implementing innovations that are improving the current operations or the UX of the product or covering functional aspects that were not covered so far.
</li>
<li>
The second horizon is about innovations that extend the human competencies or technology abilities into new related markets. It's about looking at new verticals or adjacent segments.
</li>
<li>
The third is about disruption or high-end innovations that will change the nature of your industry and generate entirely new possibilities and competencies.
</li>
</ul>
<p>
McKinsey speaks of the horizon 1 being one to two years, horizon 2 being two to four years and Horizon 3 being three to five years.
<br>
In my current company, this doesn't speak to us, since, well, we have absolutely no idea of what we're going to be doing in five years, this is like forever to us. And it doesn't make a whole lot of sense to think of it so much today aside of some high-level orientations. For us, our interpretation in terms of timeline is as follows. Horizon 1 is now to 12 months, Horizon 2 is six to 24 months, Horizon 3 is three years from now.
</p>
<p>
An important aspect of the 3 horizons model is that at every single moment, you should have a fair share of investment on the thre horizons, meaning elements from the 3 horizons in your current backlog.
<br>
If you have not reached the flattening of the profit curve, the you have mostly H1 innovations or elements, but you should also have a fair share of Horizon 2 and 3 elements.
<br>
If you have reached the flattening of your profit curve, then the ratio of Horizon 2 and 3 elements in your backlog should be higher.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/cbfa9c34-8bf6-4644-888f-da0a19c46e69">
<img class="centered" style="width: 850px;" alt="Three Horizons - fair share of focuses" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/cbfa9c34-8bf6-4644-888f-da0a19c46e69" />
</a>
</div>
<br>
<a name="sec223"></a>
<h4>2.2.3 Three Horizons framework - takeaways</h4>
<p>
Summing things up:
</p>
<ul>
<li>
You should have a fair share between the different Horizons in your backlog and your sprint development tasks. If most of the time, the majority of your investment should be on Horizon 1, there should still be a fair share of Horizon 2 and 3 elements. Whenever your reach the flattening of your curves, the ratio should change in favor of Horizon 2. But you can't avoid investing on Horizon 2 and 3 already today. If you don't, you will die.
</li>
<li>
A little note about the stories related to individual customer integration projects or individual delivery project: for some of them, you might not know before you actually start the project or the specific implementation, all of that very customer needs, the precise scope, and hence you can't estimate accurately. In this case, you have to take some reserves. And it's fine to take reserves, you can absolutely be in a situation where you don't know exactly what the team will be working on. There are always backlog fillers tasks, tiny tasks like a typo fixing or another small change somewhere that are literally filling up your backlog. These tasks are fillers. And this is what your teams work on when they need to fill in the blanks.
</li>
<li>
In a normal situation though, these fillers - all these tiny tasks - shouldn't be prioritized and scheduled independently, it's impossible. So as far as roadmapping is concerned, all these small tasks have to be grouped together in batches, consistent batches - grouping them together by theme, or scope or value. Such evolution batches are prioritized and scheduled as one big block. And this kind of makes sense, you're almost forced to implement all these tasks in a continuous way all the time. But you might decide that in the next six months, we will implement this batch of consistent evolution together on the platform.
</li>
<li>
Regarding what are these stories or epics on the roadmap, most of them comes from the PMC identified topics or Project Gaps. There's also some long and large functional evolution coming from business analyst or from the product owners just as technology evolutions and maintenance come from the CTO. The project gaps are coming from delivery of course. The takeaway here is that the cardinality has to be larger, big, important topics. And if you want to schedule smaller things, group them together in consistent evolution batches.
</li>
<li>
If you have a team that's working on H1 topics, such as project gaps or evolution batches for six months, make it so that six months after they work on H2 or H3 topics. This is crucial to avoid frustration. You can tell a team that in the coming four months they will be working on tiny and not so passionate improvements here and there, but only as long as they know that the next big technology evolution is coming up and is for them. If you have a fair amount of tasks in the teams among H1, H2, H3, everybody keeps being motivated.
<br>
And then, if you don't do that you're somehow back to having component teams, which doesn't make a whole lot of sense.
</li>
<li>
Finally, you want to have independent and autonomous team for estimations and reliable forecasting. And you want to have many of them to keep the possibility to have a fair share of investment between H1, H2 and H3 at any given time in your product development timeline.
</li>
</ul>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/3288679a-cdfe-4689-8031-104f1da1e452">
<img class="centered" style="width: 850px;" alt="3 Horizons Takeaways" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/3288679a-cdfe-4689-8031-104f1da1e452" />
</a>
<div class="centered">
3 Horizons Takeaways
</div>
</div>
<br>
<a name="sec23"></a>
<h3>2.3 C. The Estimation process</h3>
<p>
I won't spend much time on the estimation process since I detailed a few years ago all of this in <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-planning-tools-and-processes">a dedicated article on this very blog</a>.
<br>
So I will just be repeating some essential information regarding how epics or stories on the roadmap are estimated and how <i>team velocity</i> is estimated.
</p>
<a name="sec231"></a>
<h4>2.3.1 The roles and rituals involves in the estimation process</h4>
<p>
Let's start by defining all the roles involved in the estimation process. In this regards, the most important roles are the Product Manager, the CTO, the Tech Leads and Architects, the Team Leaders and of course the Product Owners.
<br>
Their core responsibilities are as follows, at least in my current company.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/efbcda67-4675-4076-8af1-ff34191e0788">
<img class="centered" style="width: 850px;" alt="Roles involved in estimation" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/efbcda67-4675-4076-8af1-ff34191e0788" />
</a>
<div class="centered">
Roles involves in the estimation process
</div>
</div>
<br>
<p>
For the estimation process in our context, the important rituals are the <i>Product Management Committee</i>, and the <i>Architecture committee</i>. The first is central to identifying evolutions and prioritizing them while the second is central to designing and estimating them.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/40354c47-0d2d-4081-b58f-2f7d30ae695c">
<img class="centered" style="width: 850px;" alt="Rituals involved in estimation" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/40354c47-0d2d-4081-b58f-2f7d30ae695c" />
</a>
<div class="centered">
Rituals involved in estimation
</div>
</div>
<br>
<p>
The Product Management Committee identifies the opportunities and evolutions and prioritize them.
<br>
The Architecture committee is responsible to proceed with everything that is required for accurate estimations: the detailed specification of these epics and stories, their breakdown in tasks and their estimations.
<br>
It's key to estimate the short to mid-term elements accurately, but then we don't care so much in estimating something on which we will work in two years. Why would you care how much time it takes to develop something we are not even sure we will be doing? We want at all cost to have accurate estimations of the things on which we know we will be working, to prioritize and plan them efficiently. But for long term ideas, a <i>T-Shirt sizing</i> approach is sufficient.
</p>
<a name="sec232"></a>
<h4>2.3.2 Rituals are scheduled</h4>
<p>
All precautions should be taken to avoid interupting the development sprint all the time. For this reason, rituals are clearly scheduled and take place at a predefined time and pace. There should be no unforeseen interruption of the sprint course.
<br>
In my company, we stick to the following month organization, with 2 sprints per month:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/7728d1d1-77c8-4dd7-a9b0-07bc9690e353">
<img class="centered" style="width: 850px;" alt="Rituals are scheduled" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/7728d1d1-77c8-4dd7-a9b0-07bc9690e353" />
</a>
</div>
<br>
<p>
This reads as follows:
</p>
<ul>
<li>
A sprint ends with a <b>Sprint Retrospective</b> (Kaizen) and starts with a <b>Sprint Planning</b> where the sprint backlog is filled up to the <i>team sprint velocity</i> by the Product Owner and the Team Leader. Tasks are discussed with all the team at the same time and some estimations may be reviewed.
<br>
In my current company, we do both on Friday since we want everything to be ready for Monday morning regardless of the timezone of the teams.
</li>
<li>
Of course every single day there is a <b>Daily Scrum</b> in the morning where everyone presents where they are and where problems are discussed and escalated to the Team Leader and Tech Lead who might schedule dedicated meetings to discuss them further if required.
<br>
Also, every day finishes with an automated deployment of the whole product in a production-like setup on the <b>Integration Environment</b>.
</li>
<li>
At the end of the sprint, a production-ready, shippable version of the product is automatically deployed on the <i>Test Environment</i>. This internal release is considered as a <i>customer release</i> as far as the development teams are concerned.
</li>
<li>
The PMC - <b>Product Management Committee</b> occurs once a month. It's sometimes a very short meeting, just reviewing the updated roadmap and fine tuning priorities and sometimes a very long meeting finishing late at night, when multiple opportunities have to be discussed and prioritized.
</li>
<li>
The ARCHCOM - <b>Architecture Committee</b> - is a fairly central ritual in my current company. This is where we do all of:
<ul>
<li>Dispatch stories among Architects, Tech Leads, Product Owners and even the CTO for design, refined specifications and tasks breakdown.</li>
<li>Discuss open points, comment and amend designs and breakdowns.</li>
<li>Proceed with task estimations</li>
<li>Identify and Challenge technical evolutions and maintenance </li>
</ul>
As far as the estimation process is concerned, this ritual is really central. It can last a few dozens of minutes in some cases, and long hours until late in the evening sometimes. We run it every week at the same time.
</li>
</ul>
<a name="sec233"></a>
<h4>2.3.3 Now the Estimation Process</h4>
<p>
Shortly put, the estimation process consists in identifying topics and evolutions in the PMC, specifying them with details, having the ARCHCOM design them and then proceeding with the breakdown in tasks for estimations, having the tasks estimated and eventually computing the overall epic or story estimations. With everything in the short-to-mid term properly estimated, the Product Manager can proceed with updating and maintaining the roadmap.
<br>
The process looks as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/483fb6a6-9c28-4cc8-b6ef-32bbc1993e81">
<img class="centered" style="width: 850px;" alt="Estimation Process" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/483fb6a6-9c28-4cc8-b6ef-32bbc1993e81" />
</a>
</div>
<br>
<p>
Which reads as follows:
</p>
<ul>
<li>
The PMC decides to prioritize a new feature, a new element or an evolution, as a <b>new story</b> and we shall proceed with its estimation so that it can be positioned accurately on the roadmap.
</li>
<li>
So the product manager or a product owner will be specifying the story in a detailed way with the help of the CTO perhaps, focusing on identifying as precisely as possible its functional elements. The result in our context is called a <b>detailed story</b>
<br>
This closes the specification phase.
</li>
<li>
Now comes the design phase.
<br>
Then the CTO, a PO, an architect or sometimes a teach lead will be transforming the <i>detailed story</i> - which is a marketing / product formalism - into a <b>development epic</b> - which is a technical formalism where we cover functional design, solution design, application design, perhaps data identification, data research needs and so on.
<br>
It can happen that the CTO creates himself some development epics for mandatory technology evolutions, refactorings, etc.
</li>
<li>
The ARCHCOM will challenge the design and other technical elements and discuss / refine open points before an ARCHCOM member will be assigned the task to proceed with the <b>breakdown in tasks</b> of the development epic. <br>
The breakdown in task is challenged and discussed at ARCHCOM before being validate.
<br>
It can happen that tasks are created directly from architects or tech leads for refactorings for instance or transformed from delivery wishes. In such case, they are bound to an existing epic or a container epic is created for them (for instance for a specific Project Gaps).
</li>
<li>
Then all individual tasks can be <b>estimated</b>.
</li>
<li>
With the individual task estimations finalized, the CTO or a PO will compute the total amount of <b>Story Points at Epic level</b>.
</li>
<li>
The CTO or a PO will then liaise with the PM to communicate the <b>estimations on the story</b>.
</li>
<li>
Finally the PM will <b>update the roadmap</b> accordingly and send the updated roadmap to the PMC. The next PMC can challenge the view, re-prioritize or end up taking a different decision regarding the evolution or new feature now that it knows how much it costs.
</li>
</ul>
<a name="sec234"></a>
<h4>2.3.4 Team Sprint velocity</h4>
<p>
The approach is fairly usual, one needs to monitor the number of story points a team has been able to implement in the past five sprints. Then the extreme values are eliminated because they make no sense (people tend to go on holidays, people are sick, some tasks have been postponed to another sprint, etc.).
<br>
We take the second and the last but one and with these we're able to compute an upper and a lower bound. This range addresses the uncertainty inherent to software development.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/6bb888cc-6c23-4042-bab1-e5917dc74a68">
<img class="centered" style="width: 700px;" alt="Team Sprint capacity" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/6bb888cc-6c23-4042-bab1-e5917dc74a68" />
</a>
<div class="centered">
Computing Team Sprint capacity
</div>
</div>
<br>
<p>
As far as estimations and planning are concerned, we will take the lower bound to compute the <b>team sprint velocity</b> since we have to be defensive.
</p>
<a name="sec235"></a>
<h4>2.3.5 Forecasting</h4>
<p>
Now that we know how many Story Points a team can do in a sprint, we know how many it can do in a month of 2 sprints (the additional days are ignored, they form a reserve).
<br>
Add if we know how many Story Points a team can do in a month, then we know how many months a team needs to implement any given story that has been properly estimated.
<br>
Boom. Done.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/0fda5e39-7a75-4b88-ad8e-6cb7c738a027">
<img class="centered" style="width: 650px;" alt="Forecasting delivery dates" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/0fda5e39-7a75-4b88-ad8e-6cb7c738a027" />
</a>
<div class="centered">
Forecasting delivery dates
</div>
</div>
<br>
<p>
As far as delivery date estimations are concerned, we will take the pessimistic estimation coming from the lower bound of the Team Sprint capacity.
</p>
<a name="sec236"></a>
<h4>2.3.6 The Estimation process - Takeaways</h4>
<p>
Summing things up:
</p>
<ul>
<li>
First a note about something specific to our context in my project company. In our model, the project gaps are the only items on which we decide to have hard commitments.
<br>
In our model, we know that we have to start any given customer delivery project at a precise moment in time and that it has to be in production at another precise moment in time. So we need to make sure we will deliver the missing functionalities sufficiently in advance for our integration team or delivery team to be able to integrate them at the customer sufficiently in advance.
<br>
So we don't have much of a choice, we need to have at least some elements of our roadmap that are strong commitments. And in our model, these are the project gaps.
<br>
To handle this, we make sure the project gaps are well balanced between the different teams. They become the only items from the roadmap that can't be moved, they are frozen items; we want to ensure that we will respect their delivery dates.
<br>
The way we do that is that we consider them as frozen once they are scheduled. Once they are planned, they can't move anymore.
</li>
<li>
The estimation game tells us the total story point for each and every story of epic that we want to follow on the roadmap. So assuming two things, the amounts with the team sprint velocity, we know how many sprints and how many months will be required to implement a story or an epic. So we can put it on the roadmap. The key word here is <i>reserve!</i>. Take reserves, more reserves and even more reserves.
<br>
In our model, sufficient reserve is taken by the fact that we schedule these elements for the end of a period, the end of the month in the three to six months timeline, and then the end of a quarter in the next periods, or even the end of the year in the following period. And by doing so, most of the time, we find out who have sufficient reserve taken when laying these elements down the roadmap.
</li>
</ul>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/538bd676-bf44-4c16-aeb7-3f07d573febf">
<img class="centered" style="width: 850px;" alt="Estimation Process Takeaways" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/538bd676-bf44-4c16-aeb7-3f07d573febf" />
</a>
<div class="centered">
Estimation Process Takeaways
</div>
</div>
<br>
<a name="sec24"></a>
<h3>2.4 D. Roadmap timeline</h3>
<p>
The roadmap timeline is also an important aspect. The whole principle is to avoid having a timeline with too many different deadlines since they would be impossible to follow and only bring noise, but we have to have a sufficient number since they form our scheduling unit.
<br>
A good number of terms of scheduling deadlines is eight. Eight deadlines are easy to follow and with a reducing granularity approach, they provide enough scheduling and synchronization points to successfully draw a 24 to 36 months roadmap.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a2f03749-a2e2-4647-b171-8b9a9d2429f2">
<img class="centered" style="width: 800px;" alt="Roadmap timeline" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a2f03749-a2e2-4647-b171-8b9a9d2429f2" />
</a>
<div class="centered">
Roadmap timeline
</div>
</div>
<br>
<p>
Which reads as follows:
</p>
<ul>
<li>
The <b>Next 3 months</b> are followed on three elements. This makes it more than 1/3 of the roadmap dedicated to this shortest term and is a reflection of the fact that we follow and monitor things carefully on this short term roadmap. In addition, we're having strong commitment there and any change on the activity plan of the next three months should really be considered and weighted very carefully since we commit on most of these elements.
</li>
<li>
The next 2 elements are related to the <b>following 2 quarters</b>. This makes 1/4 of the roadmap timeline dedicated to a 6 months period and is an indication that we want a fine control over the next priorities past the shortest term 3 months period. This is where changes can happen frequently as we discover our market or sign new deals. Priorities can change thre but it has to be backed by strong business objectives.
</li>
<li>
The following 2 elements are related to the <b>following 2 semesters</b>. This is the mid term roadmap basically related to <i>what we will do in a year or so</i>. This is really rather indicative and <b>estimations there do not necessarily need to be precise</b>, a <i>T-shirt sizing</i> approach can be sufficient.
<br>
The elements on the mid-term roadmap yet need to be realistic and correspond to the vision we have today, even though that vision can entirely change.
</li>
<li>
The last element being the <b>following year</b> is really related to the long-term roadmap, a basket of ideas.
</li>
</ul>
<a name="sec241"></a>
<h4>2.4.1 Monthly roadmap update</h4>
<p>
The roadmap is updated every month and a trick needs to be found to ensure that whatever happens and wherever we are within the year, we keep tracking 8 baskets, not more and importantly not less.
<br>
This is done by tricking the vision and adapting some periods to fit the specific situation in which we are within the year.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/2edd8777-8816-4953-b667-e35a6b80f263">
<img class="centered" style="width: 850px;" alt="Timeline Update" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/2edd8777-8816-4953-b667-e35a6b80f263" />
</a>
<div class="centered">
Roadmap timeline
</div>
</div>
<br>
<p>
By tricking the period this way every month, we make sure that we only follow eight different buckets in the in the roadmap since anything beyond eight wouldn't make a whole lot of sense, at least in our model.
</p>
<p>
Again, one aspect that is absolutely critical to be in a position where we can have reliable forecasts is to avoid at all cost <i>stop-the-world</i> events (refactorings, releases, etc.)
<br>
<i>Stop-the-world</i> releases or refactoring, or big technology evolutions that require synchronizing the teams, are killing the ability of a team to work independently and autonomously. So eventually, it's killing the ability of a team to respect the forecast on schedule and planning. These have to be avoided at all cost.
<br>
And interestingly, even though implementing a big refactoring or technology evolution without <i>stopping the world</i> is much more difficult, from a purely technical standpoint, most of the time, it's <b>absolutely doable</b>. It will be more expensive. It will indeed take longer. But it's absolutely doable.
<br>
And it's absolutely worth it. Because avoiding <i>stop the world</i> events enable the team to respect and fulfill expectations in terms of forecasts and planning.
</p>
<a name="sec242"></a>
<h4>2.4.2 Back on Continuous Delivery</h4>
<p>
Continous delivery is very hard to reach in the mind of software development teams working on SaaS platforms.
<br>
But think of the web giants. Think of what they are doing ...
<br>
Let's see some examples. Amazon is deploying code in production every 11 seconds, on average. Netflix is pushing code to production 1000 times a day. They have even developed <i>Chaos Monkeys</i>, which is a software literally killing VMs, services and containers all the time in production, as a way to force developers to <i>design for failure</i>. Facebook, before 2016, were cherry picking the features in a few release trains a day. They don't do that anymore, they push the master branch directly in a dozen of release trains a day.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/16cff642-0b83-4b41-a082-cbca59af25ce">
<img class="centered" style="width: 850px;" alt="Web Giants continuous delivery" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/16cff642-0b83-4b41-a082-cbca59af25ce" />
</a>
</div>
<br>
<p>
Now that's what the Web Giants manage to do. If you embrace the same techniques - they are using state of the art XP, agile, Scrum, DevOps and lean startup principles and practices, then you can do this as well.
</p>
<p>
And even if you do <i>on premise</i> deployment, it still makes sense to do continuous delivery since it forces you to automate your release process.
<br>
As far as the development teams are concerned, every sprint ends up with a shippable and production-ready version of the product. And because Database migration scripts are maintained along the software, you end up in a situation where it's perfectly feasible from a technology standpoint to push this version at a customer. It's perfectly doable because you have designed the product and developed it this way.
<br>
If the developments teams stick to this, then releasing the product as an actual end-customer version becomes purely a Product management decision, not any more a technical concern. At the end of every feature development, Product Management has the possibility to decide to transform the internal release into a customer release and give it a proper version number.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/4a1c4998-6939-4add-b745-94e129fba4c6">
<img class="centered" style="width: 850px;" alt="Releases" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/4a1c4998-6939-4add-b745-94e129fba4c6" />
</a>
<div class="centered">
Releases
</div>
</div>
<br>
<a name="sec3"></a>
<h2>3. Conclusions and final notes</h2>
<p>
In terms of conclusion I would say that embracing the core Agile practices from XP, Scrum, DevOps and Lean Startup enables a company to have reliable forecasting and planning abilities which eventually lead to design a relevant and useful roadmap.
<br>
This principles and practices enables the company to have autonomous and independent teams, which are then able to work on a development topic or technology evolution without interruptions and without and friction with other teams, which is key to have their real-life development pace aligned with estimations and forecasts.
<br>
In addition, multiple autonomous teams have the possibility to work in parallel on multiple development topics which enables the company to have a fair share of development in the multiple Horizons.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/82a6044e-22b4-4d9b-8cf6-61780600a0ac">
<img class="centered" style="width: 850px;" alt="Conclusion - Agile roadmap requirements" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/82a6044e-22b4-4d9b-8cf6-61780600a0ac" />
</a>
</div>
<br>
<p>
Some tools are required to give life to all of this and I will leave the reader to discover them hereunder:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/4dd1a1b4-d127-4265-bf4d-ec57c4019914">
<img class="centered" style="width: 850px;" alt="Tools" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/4dd1a1b4-d127-4265-bf4d-ec57c4019914" />
</a>
</div>
<br>
<p>
My final words would be that such an Agile roadmap has the potential to be much more that a communication tool, it's really an internal alignment tool and provides the high-level management view to align everyone in the company towards the common product development objectives.
</p>
<p>
This article is available as a <a href="https://www.slideshare.net/JrmeKehrli/a-proposed-framework-for-agile-roadmap-design-and-maintenance">Slideshare presentation</a>
</p>
https://www.niceideas.ch/roller2/badtrash/entry/netguardians-3d-ai-technology
NetGuardians' 3D AI Technology
Jerome Kehrli
2021-06-08T03:18:54-04:00
2021-09-03T05:12:10-04:00
<p>
<i>(Article initially published on <a href="https://blog.netguardians.ch/netguardians-3d-ai-delivers-global-analytics-supremacy-in-fraud-detection">NetGuardians' blog</a>)</i>
</p>
<p>
Whenever our software is run head-to-head in a pitch situation against that of our rivals, we always come out top. We always find more fraud with a lower number of alerts. For some, this is a surprise – after all, we are one of the youngest companies in our field and one of the smallest. To us, it is no surprise. It is testament to our superior analytics.
</p>
<h2>A focus on customer behavior</h2>
<p>
We began working in fraud prevention in 2013 and quickly realized the futility of rules engines in this endless game of cat-and-mouse with the fraudsters. The criminals will always tweak and reinvent their scams; those trying to stop the fraud with rules engines will always be left desperately working as fast as possible to identify and incorporate the latest scams into their surveillance. Far better to focus on what we know changes very little – customer behavior.
</p>
<p>
If a bank knows how a customer spends money, it can spot when something is awry by looking for anomalies in transaction data. However meticulous the fraudster is at trying to hide, every fraudulent transaction will have anomalous characteristics. People’s lives are constantly changing – they buy from new suppliers, they move house, go on holiday and their children grow up – all of which will affect their spending and transaction data. Every change will throw up false alerts that will undermine the customer experience unless you train your models correctly.
</p>
<h2>The three pillars of 3D AI</h2>
<p>
We train our models using what we call our 3D AI approach. This enables them to assess the risk associated with any transaction with extraordinary accuracy, even if it involves new behavior by the customer. This also keeps false alerts to the minimum.
</p>
<div class="centering">
<a href=" https://www.niceideas.ch/roller2/badtrash/mediaresource/f0469115-795b-45d9-be64-e5059c14745f">
<img class="centered" style="width: 400px;" alt="3D AI" src=" https://www.niceideas.ch/roller2/badtrash/mediaresource/f0469115-795b-45d9-be64-e5059c14745f" />
</a>
</div>
<br>
<p>
Developed by us at NetGuardians, this approach has three pillars, each of which uses artificial intelligence (AI) to constantly update and hone the models.
</p>
<p>
The pillars are: <b>anomaly detection</b>, <b>fraud-recognition</b> training analytics and <b>adaptive feedback</b>. Together, they give our software a very real advantage by not only spotting fraud and helping banks stop fraudulent payments before any money has left the account, but also by minimizing friction and giving the best possible customer experience. This is what differentiates our software in head-to-head pitches.
</p>
<p>
<i>(Article initially published on <a href="https://blog.netguardians.ch/netguardians-3d-ai-delivers-global-analytics-supremacy-in-fraud-detection">NetGuardians' blog</a>)</i>
</p>
<p>
Whenever our software is run head-to-head in a pitch situation against that of our rivals, we always come out top. We always find more fraud with a lower number of alerts. For some, this is a surprise – after all, we are one of the youngest companies in our field and one of the smallest. To us, it is no surprise. It is testament to our superior analytics.
</p>
<h2>A focus on customer behavior</h2>
<p>
We began working in fraud prevention in 2013 and quickly realized the futility of rules engines in this endless game of cat-and-mouse with the fraudsters. The criminals will always tweak and reinvent their scams; those trying to stop the fraud with rules engines will always be left desperately working as fast as possible to identify and incorporate the latest scams into their surveillance. Far better to focus on what we know changes very little – customer behavior.
</p>
<p>
If a bank knows how a customer spends money, it can spot when something is awry by looking for anomalies in transaction data. However meticulous the fraudster is at trying to hide, every fraudulent transaction will have anomalous characteristics. People’s lives are constantly changing – they buy from new suppliers, they move house, go on holiday and their children grow up – all of which will affect their spending and transaction data. Every change will throw up false alerts that will undermine the customer experience unless you train your models correctly.
</p>
<h2>The three pillars of 3D AI</h2>
<p>
We train our models using what we call our 3D AI approach. This enables them to assess the risk associated with any transaction with extraordinary accuracy, even if it involves new behavior by the customer. This also keeps false alerts to the minimum.
</p>
<div class="centering">
<a href=" https://www.niceideas.ch/roller2/badtrash/mediaresource/f0469115-795b-45d9-be64-e5059c14745f">
<img class="centered" style="width: 400px;" alt="3D AI" src=" https://www.niceideas.ch/roller2/badtrash/mediaresource/f0469115-795b-45d9-be64-e5059c14745f" />
</a>
</div>
<br>
<p>
Developed by us at NetGuardians, this approach has three pillars, each of which uses artificial intelligence (AI) to constantly update and hone the models.
</p>
<p>
The pillars are: <b>anomaly detection</b>, <b>fraud-recognition</b> training analytics and <b>adaptive feedback</b>. Together, they give our software a very real advantage by not only spotting fraud and helping banks stop fraudulent payments before any money has left the account, but also by minimizing friction and giving the best possible customer experience. This is what differentiates our software in head-to-head pitches.
</p>
<!-- read more -->
<p>
The first pillar is <b>anomaly detection</b>, which is mostly <b>unsupervised learning</b>. At this stage, we are looking for anomalies and working out the level of risk associated with them. This involves examining a set of parameters such as time of transaction, counterparty, location, amount and currency. As seen above, on their own, these parameters aren’t enough to prevent an unacceptable level of false alerts. By including peer-group behavior, we begin to reduce the number of false alerts. For example, people don’t buy a car often and when they do, they don’t want their bank to block and query the transaction because of its rarity. But if you place the customer among peers, the size of the transaction can trigger associations that bring its risk level down. This, along with other techniques including Poisson Law help us understand the timing and regularity of the transaction, resulting in a highly nuanced picture of risk.
</p>
<p>
The second pillar is <b>fraud-recognition</b> training analytics, mostly relying on <b>supervised learning</b> techniques. Typically, a T2 or T3 bank might see 10 frauds a year out of 100 million transactions. This level of fraud is insufficiently high to train a complex algorithm. So we have developed models using anonymized bank data that can be overlaid on a bank’s own data. The models cover different situations, regions, size of bank and type of customer, which allow us to create analytics that look at the context of the data. Our software is even capable of deciding which model is best for the analysis.
</p>
<p>
For a T1 bank, it takes just a few hours to train the algorithms, with perhaps some manual intervention to confirm the final choice of models. Smaller banks take one to two hours and the process is fully automated.
</p>
<p>
The final element is <b>adaptive feedback</b> using <b>active learning</b>. This is absolutely crucial to reduce false alerts to the lowest possible level while minimizing the risk of missing a fraud. Our adaptive feedback technology monitors, controls, challenges and supervises feedback from the alert investigators – the bank’s back- and middle-office employees who review alerts and decide when to call the customer – to ensure that it is of sufficient quality before re-injecting it into the machine learning models. As this is unique to the NetGuardians approach, it’s worth going into a little more detail.
</p>
<h2>Fine-tuning and machine learning</h2>
<p>
All alerts raised by the NetGuardians software come up on a dashboard and have to be reviewed manually. If the alert turns out not to be a fraud, we ask the bank to classify the risk of transaction.
</p>
<ul>
<li><b>High</b> = not a fraud but confirmed as high-risk, so continue to block similar transactions</li>
<li><b>Medium</b> = not a fraud but can see why NetGuardians software thought it would be. Proceed with care</li>
<li><b>Low</b> = not a fraud and unclear why the software thought it was. Never block similar again</li>
</ul>
<p>
This feedback retrains unsupervised and supervised learning and is key to the precision of our solution. Further tuning comes from the ability of our software to query feedback to ensure it is high quality.
</p>
<p>
For example, someone has been working in fraud investigation for many years. They have “learnt” that usually just two parameters reveal suspicious behavior – let’s say the amount and the beneficiary – and use only those two to decide the level of risk. When applied over and over again, a model would learn to look at just those two features, undermining its ability to see and learn from the whole picture – a form of “overfitting”. This reduces its accuracy. But our software does something very clever. If it believes something is anomalous, it will ask the investigator a direct question about another parameter, forcing them to re-examine the transaction. It then uses this additional feedback to learn more about the customer, the bank and the transactions.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/56510626-5ea6-4a69-bc0b-a0e8c35dfe06">
<img class="centered" style="width: 850px;" alt="3D AI Cycle" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/56510626-5ea6-4a69-bc0b-a0e8c35dfe06" />
</a>
</div>
<br>
<p>
Taken together, these three pillars of AI are responsible for our market-beating performance. Only NetGuardians uses these three together. And only NetGuardians is able to find all the fraud a bank knows about in historic data as well as up to nearly one-fifth more. Our software doesn’t need to be taught new fraud types because it isn’t looking for them. And it doesn’t add customer friction – in fact it reduces false alerts by as much as 83 percent. This is because it is always learning and refining its models across the broadest possible perspective, resulting in our analytics supremacy.
</p>
<p>
<i>(Article initially published on <a href="https://blog.netguardians.ch/netguardians-3d-ai-delivers-global-analytics-supremacy-in-fraud-detection">NetGuardians' blog</a>)</i>
</p>
https://www.niceideas.ch/roller2/badtrash/entry/covid-19-just-get-the
COVID-19 - Just get the vaccine as soon as possible!
Jerome Kehrli
2021-05-11T10:57:10-04:00
2021-05-11T10:57:10-04:00
<p>
Well. For once this will be an article far away from the kind of stuff I use to post on this blog. This is a post about COVID-19 and its vaccines. But it's perhaps the most important thing I will have ever written.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/74aa55fc-6df1-4a81-abc7-410f541dade5">
<img class="centered" style="width: 400px;" alt="sars-COV-2" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/74aa55fc-6df1-4a81-abc7-410f541dade5" />
</a>
</div>
<br>
<p>
Since the start of the week, various events have cruelly reminded us of how dangerous COVID is.
<br>
Particularly the situation in India, where infections are increasing at an absolutely terrifying rate, is very worrying. India this week set a world record for new daily COVID cases with more than 400,000 daily infections in recent days (and this is probably underestimated) and more than 2,000 daily deaths.
</p>
<p>
It is time for everyone to realize that our only way out of this long-term humanitarian disaster is clearly through herd immunity.
<br>
We have finally had an incredible chance for a few months: in Europe and the US we have wide access to many vaccines. The speed of vaccination has increased.
And that is great.
</p>
<p>
The reason that prompts me to write today is that the #AntiVax are now emerging as the main pitfall on the way out of this disaster and that we - scientists or simply educated people - have the responsibility to react. All over Western countries, the same signs are being seen: some vaccination centers have already gone from insufficient supply a few weeks ago to insufficient demand today.
And that is a disaster.
</p>
<ul>
<li>
For example, in the US, according to the CDC, the average daily immunization has dropped by 20% since the beginning of April. A lot of centers find themselves with excess doses and possible appointments remaining, for example in South Carolina (see US media). Another example, without the county of Palm Beach, where a large number of vaccination centers have opened in recent weeks, the health department announces that it has 10,000 possible appointments on the various sites that remain vacant.
</li>
<li>
In Europe, the situation is unfortunately no better. In France, for instance, several vaccination centers claim that people are not registering for the vaccine as quickly as they had expected and that others are not showing up. The bottom line is that multiple doses are not given at the end of the day and some must be thrown away. This is madness.
</li>
</ul>
<p>
A worrying and growing number of people are reluctant to get the free COVID vaccine.
<br>
However, these vaccines could save not only your life, but also the life of the people around you.
</p>
<p>
The bottom line is that for the Coronavirus, the herd immunity threshold would be between 70 and 90% of the population. It is this threshold that we have to reach as quickly as possible. This must be our most sacred goal.
</p>
<p>
In the US, for example, a survey found that if 60% of American adults have either received a first dose of the vaccine or wish to be vaccinated, 18% respond "maybe" and 22% categorically refuse.
<br>
Let us imagine that in the long term the undecided ones convince themselves to get vaccinated, that only places us at 78%. And unfortunately, this poll does not take into account children who are not currently eligibale to bgetting vaccinated and who represent about 22% of the population. From this perspective, achieving herd immunity today seems illusory.
</p>
<p>
We must therefore ensure that the vaccine is given to as many adults as possible as quickly as possible. And that means we really all need to get vaccinated.
</p>
<p>
I can understand that young, athletic and healthy people do not see the benefits of vaccination. We might not get seriously ill from COVID or even get sick at all, right?
<br>
But it could always be inadvertently passed on to someone who could then die.
<br>
And only vaccinating people at risk is not satisfactory since the circulation of the virus would not be stopped. And this is the real problem: the more the virus circulates, the more likely it is that we will see mutations appear that make it more dangerous, and perhaps even strains capable of fully resisting current vaccines, thus bringing us back to The starting point.
</p>
<p>
The point is, the current hesitation over vaccination is terribly problematic.
</p>
<p>
So in the rest of this article, I would like to review and discuss most of the "arguments" of the #Antivax groups.
</p>
<p>
Well. For once this will be an article far away from the kind of stuff I use to post on this blog. This is a post about COVID-19 and its vaccines. But it's perhaps the most important thing I will have ever written.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/74aa55fc-6df1-4a81-abc7-410f541dade5">
<img class="centered" style="width: 400px;" alt="sars-COV-2" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/74aa55fc-6df1-4a81-abc7-410f541dade5" />
</a>
</div>
<br>
<p>
Since the start of the week, various events have cruelly reminded us of how dangerous COVID is.
<br>
Particularly the situation in India, where infections are increasing at an absolutely terrifying rate, is very worrying. India this week set a world record for new daily COVID cases with more than 400,000 daily infections in recent days (and this is probably underestimated) and more than 2,000 daily deaths.
</p>
<p>
It is time for everyone to realize that our only way out of this long-term humanitarian disaster is clearly through herd immunity.
<br>
We have finally had an incredible chance for a few months: in Europe and the US we have wide access to many vaccines. The speed of vaccination has increased.
And that is great.
</p>
<p>
The reason that prompts me to write today is that the #AntiVax are now emerging as the main pitfall on the way out of this disaster and that we - scientists or simply educated people - have the responsibility to react. All over Western countries, the same signs are being seen: some vaccination centers have already gone from insufficient supply a few weeks ago to insufficient demand today.
And that is a disaster.
</p>
<ul>
<li>
For example, in the US, according to the CDC, the average daily immunization has dropped by 20% since the beginning of April. A lot of centers find themselves with excess doses and possible appointments remaining, for example in South Carolina (see US media). Another example, without the county of Palm Beach, where a large number of vaccination centers have opened in recent weeks, the health department announces that it has 10,000 possible appointments on the various sites that remain vacant.
</li>
<li>
In Europe, the situation is unfortunately no better. In France, for instance, several vaccination centers claim that people are not registering for the vaccine as quickly as they had expected and that others are not showing up. The bottom line is that multiple doses are not given at the end of the day and some must be thrown away. This is madness.
</li>
</ul>
<p>
A worrying and growing number of people are reluctant to get the free COVID vaccine.
<br>
However, these vaccines could save not only your life, but also the life of the people around you.
</p>
<p>
The bottom line is that for the Coronavirus, the herd immunity threshold would be between 70 and 90% of the population. It is this threshold that we have to reach as quickly as possible. This must be our most sacred goal.
</p>
<p>
In the US, for example, a survey found that if 60% of American adults have either received a first dose of the vaccine or wish to be vaccinated, 18% respond "maybe" and 22% categorically refuse.
<br>
Let us imagine that in the long term the undecided ones convince themselves to get vaccinated, that only places us at 78%. And unfortunately, this poll does not take into account children who are not currently eligibale to bgetting vaccinated and who represent about 22% of the population. From this perspective, achieving herd immunity today seems illusory.
</p>
<p>
We must therefore ensure that the vaccine is given to as many adults as possible as quickly as possible. And that means we really all need to get vaccinated.
</p>
<p>
I can understand that young, athletic and healthy people do not see the benefits of vaccination. We might not get seriously ill from COVID or even get sick at all, right?
<br>
But it could always be inadvertently passed on to someone who could then die.
<br>
And only vaccinating people at risk is not satisfactory since the circulation of the virus would not be stopped. And this is the real problem: the more the virus circulates, the more likely it is that we will see mutations appear that make it more dangerous, and perhaps even strains capable of fully resisting current vaccines, thus bringing us back to The starting point.
</p>
<p>
The point is, the current hesitation over vaccination is terribly problematic.
</p>
<p>
So in the rest of this article, I would like to review and discuss most of the "arguments" of the #Antivax groups.
</p>
<p>
<b>VACCINES DO NOT WORK. THEY DO NOT PREVENT YOU FROM GETTING SICK OR INFECTING OTHERS</b>
</p>
<p>
We hear that often and I kind of understand it. Especially when you have to receive two injections, you really want the thing to be foolproof (nothing is foolproof).
</p>
<p>
The argument most often made by conspirators is the fact that public health departments in Western countries recommend that vaccinated people (even force them) to continue to wear masks when in contact with vulnerable people.
<br>
However, this is in no way an element able to determine the uselessness of vaccines.
</p>
<p>
Clinical trials have shown that vaccines are amazingly successful in preventing people from contracting a severe form of the disease.
<br>
As to whether they protect you from the spread of the virus, the tests were not designed to assess that aspect. But the evidence so far indicates that they have significantly reduced transmission. The best proof of this is Israel, which with 80% of its population vaccinated has virtually eliminated the circulation of the virus.
</p>
<p>
The reason masks and distancing are still recommended is because public health organizations are as cautious as possible and apply the precautionary principle. In my opinion, that makes sense during a global pandemic, right?
</p>
<p>
What is crisp in all of this. is the fact that the #AntiVax groups were impatiently awaiting an event like this. They have a unique window of opportunity to propagate vaccine doubt and they have profited greatly.
<br>
And what's really scary is that they don't even really need to convince people that they're right. They just need to convince people that no one else is about the situation and the vaccines to get people to buy into myths, with lots of conspiracy videos spreading untruths and studies and claims taken ont of context.
<br>
So in the US, for example, only 4% of the population believes that the COVID vaccine is more dangerous than the disease. But 25% say they don't know. And this is a tragedy because if the #AntiVax groups manage to spread enough disinformation to convince people to "not be vaccinated for lack of information", then they ruin any chance of getting out of this crisis for everyone.
</p>
<p>
To be clear, I'm sure most of the hesitant aren't fanatics or conspirators, most are just people trying to make the best decisions for themselves and their families. We all know a bunch of people like that. But by swallowing the myths of the conspirators, they are complicit in the current situation.
</p>
<p>
<b>BILL GATES USES THE VACCINE TO PUT A MICROCHIP IN EACH OF US</b>
</p>
<p>
This myth fascinates me specifically because it touches a topic with which I am more than very familiar.
<br>
And the answer is really very simple. The technology for this simply does not exist, neither today nor in the near future.
</p>
<p>
Make no mistake, we have the technology today to miniaturize some very limited feature devices at this scale.
<br>
But miniaturizing a worthwhile chip (with remote connection capabilities, a reliable power supply, and useful analytical capabilities or the ability to influence the human body) is simply not possible today. It's science fiction. Believe me, it's definitely my job.
</p>
<p>
And also - and I don't stress how essential this is - monitoring you, stalking you in any way possible, and observing you all day long is precisely what your phone does with all your consent possible. Why would anyone bother to design a chip for this when everyone is buying these smartphones with such excellent analytics and tracking capabilities so greedily?
</p>
<p>
So to put it once and for all: there is no microchip in the COVID vaccine.
<br>
This rumor is based on the fact that the Gates Foundation funded research years ago that has often been taken out of context. In this study, researchers looked at creating an invisible ink that could potentially be injected with a vaccine so populations like refugee children could keep immunization records without paperwork. Over the time, the original context has been skewed by conspirators and made to become a microchip story that Bill Gates wants to inject into vaccines. Which, if you think about it for a second, doesn't make any sense.
</p>
<p>
Again, Bill Gates, just like facebook, google etc. already has all the means possible to track you with your smartphone.
</p>
<p>
<b>VACCINES ARE TESTED ON US, THE POPULATION</b>
</p>
<p>
Many people think that pharmaceutical companies use the general population as guinea pigs. And that hardly encourages, eh? No one wants to be used as a guinea pig.
</p>
<p>
It's worth understanding exactly how the vaccine got to the market so quickly. The very first thing to understand is that researchers have been working on vaccines against other coronaviruses for years. So when COVID-19 emerged, they got a big head start. "Operation Warp Speed", as it was called, was not about rushing scientific research. It was simply a matter of drastically reducing the bureaucracy that might otherwise have slowed it down.
</p>
<p>
The vaccine researchers have explained in detail (and in full BTW if you get the information in the right places, not on youtube) that they were able to develop vaccines in 12 months because they were able to compress the vaccine. calendar. While the various steps necessary for putting a vaccine into circulation are normally done in a linear fashion - from A, then B, then C and so on up to Z - they were able, this time around, to start steps in parallel - by doing the F and the E for example at the same time as the A and the B.
<br>
They took steps that usually happen sequentially and saved time by performing them simultaneously.
</p>
<p>
It is this aspect above all, but also the massive investments of the private sector and Western states in research which have made it possible to be so rapid, to the benefit of the population, in no way to their detriment.
</p>
<p>
<b>mRNA-BASED VACCINES MAY DAMAGE DNA AND CAUSE MUTATIONS</b>
</p>
<p>
Many have heard or read online that the Pfizer and Moderna vaccines were the first to be allowed to use messenger RNA, which is true, but it has given rise to speculation about what messenger RNA is capable of.
<br>
The most common fear is that anyone who takes these vaccines will invariably have neurological problems within a year, and most of them will die within 10 years.
<br>
Huh ... okay. I think it is important to clarify certain things.
</p>
<p>
First, other than a few conspiratorial videos that claim to have evidence that people have never seen, there is absolutely no credible evidence or study to support any of this.
<br>
As for the claim that mRNA vaccines can modify DNA, it is very important to know that messenger RNA from vaccines does not enter the genome. It's impossible. It does its job far away from the nucleus of the cell (where DNA is located).
<br>
Fear of what the vaccine contains or what it might do, however, seems to be common. For example, some evangelicals in the US fear that it contains cells from aborted fetuses, which is not the case either.
<br>
For most of the people who left #AntiVax insinuate doubt in their mind, they actually fear that the vaccine (still based on DNA modifications) could alter how their body works.
<br>
We have heard some individuals on facebook claim that these vaccines cause infertility; these rumors were fueled by a blog post, which falsely claimed that Pfizer's vaccine can cause the female body to attack a protein that plays a crucial role in the development of the placenta. It is a myth. This is absolutely wrong. Whatever these famous experts say, there is absolutely no evidence to support this claim.
<br>
There is, however, already quite satisfactory evidence that Pfizer's vaccine does not cause infertility, since during trials carried out in 2020, several women became pregnant ... and the only one to have suffered a miscarriage received the placebo. So this is simply completely wrong.
</p>
<p>
If you know a biologist or doctor among your friends, ask them to explain to you what messenger RNA is and how polymerase enzymes translate DNA into RNA and how the latter is used by ribosomes to synthesize the proteins that drive our body. It is really quite fascinating.
<br>
The bottom line is: Messenger RNA is not designed to enter a cell nucleus, let alone alter its DNA. It just doesn't work that way.
</p>
<p>
<b>THE RISKS OF VACCINES ARE GREATER THAN THE RISKS OF COVID</b>
</p>
<p>
This is a perception fueled by the constant circulation of misleading headlines about people falling ill or dying after receiving their dose of vaccine.
</p>
<p>
For example, you may have seen this story which was widely shared about 23 people in Norway who died within a week of taking the first dose of the vaccine, which sounds scary indeed. But that headline obscures an absolutely essential piece of context: At that time, in Norway, the vaccine was being given to the oldest or sickest people, and a certain percentage of them would statistically die that week. anyway, vaccine or not vaccine. 400 people die every week in nursing homes in Norway.
<br>
You see what I mean ?
<br>
If someone who has just received their first dose dies of a car crash on their way out of the vaccination center, we should be able to exclude the vaccine as the cause of the death, nope ?
</p>
<p>
When the WHO looked at these incidents, it found no unexpected or untoward increases in the number of deaths, which makes sense. Correlation is not causation. The vaccine protects against COVID, not the concept of mortality, fuck it.
<br>
It's strange to have to clarify this: but you have to know that you are all going to die one day, all of you, vaccine or not vaccine.
</p>
<p>
This also applies to the stories that are popping up in the US that are linked to the frightening exaggeration of data from VAERS (Vaccine Adverse Event Reporting System) - the system for collecting vaccine-related adverse events. VAERS a database that collects all medical events following vaccination for any US citizen. The problem with VAERS is that reports can be entered by anyone and are not verified. The goal is to collect everything and since the users of the system are doctors or medical researchers, the system relies on their skills to separate the wheat from the chaff themselves.
<br>
So any layman must treat the resulting data with extreme caution, which journalists obviously do not do.
<br>
A self-styled doctor once claimed that the flu shot turned him into the Hulk. And this report was obviously accepted and entered into the database. Again the goal is to collect everything.
<br>
See where this takes us, eh?
</p>
<p>
The reason the US CDC collects this data is that if a pattern emerges, action can be taken. This is exactly what happened with Johnson and Johnson's vaccine. The CDC has found a potential pattern of rare blood clots and, with due caution, halted the large-scale deployment of this vaccine pending further testing and protocols.
<br>
Skeptics pointed to this fact as proof they were right that vaccines were dangerous. But in reality, this obviously proves the opposite: the risk to public safety of vaccines is analyzed and rigorously monitored, and in any case not buried in secret and requiring conspirators with an IQ 2 to be revealed to the public.
</p>
<p>
Obviously I am not claiming that there are no side effects to vaccines, of course not. But serious side effects like anaphylaxis are incredibly rare: 4.7 cases per million for Pfizer, 2.5 cases per million for Moderna. Also, it is essential to understand that these side effects mainly occur in people with a history of severe allergies.
<br>
The point is, the vast majority of people can expect the most common cold or flu symptoms within the first few days after their injection, or maybe just arm pain, or even nothing at all most of the time. time.
</p>
<p>
But most importantly, the bottom line, the key thing to remember is that no vaccine side effect is worse than the alternative COVID, a disease that has killed more than 3 million people worldwide, while one again to this day, there is no conclusive evidence that the vaccine killed anyone (and not that no one who received the vaccine died of course).
</p>
<p>
<b>IN CONCLUSION</b>
</p>
<p>
It is of course more than natural to have questions. But there are reassuring answers. You just have to look for them in the right places, so not on youtube or facebook.
</p>
<p>
The goal for everyone must be to achieve herd immunity, as quickly as possible. This is the only way out of this human, social and economic crisis in the long term.
<br>
And to get there, we must at all costs convince all those who can be to take these vaccines.
</p>
<p>
Talk to your loved ones and bring these arguments to those who hesitate, in the most benevolent way possible.
<br>
We need to get out of this crisis, and vaccines are the only solution.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/the-search-of-product-market
The Search for Product-Market Fit
Jerome Kehrli
2020-08-17T04:31:43-04:00
2023-03-30T10:06:32-04:00
<p>
The Search for Product-Market Fit is the sinews of war in a startup company. While the concept and whereabouts are well known by most founders, the importance of this event in the company and product building process, what it means to be Before-Product-Market-Fit and After-Product-Market-Fit and the fundamental differences in terms of objectives, processes, culture, activities, etc. between these two very distinct states is almost always underestimated or misunderstood.
</p>
<p>
Product-Market Fit is this sweet spot that startups reach when they feel eventually that the market is really pulling out their product. It's what happens when they found out that they can only deliver as fast as the customers buys their product or when they can only add new servers in the SaaS cloud as fast as is required to sustain the rise in workload.
<br>
Product-Market Fit is so important because it has to be a turn point in the life of a young company.
</p>
<ul>
<li>Pre-Product-Market Fit, the startups needs to focus on the leanest possible ways to solve Problem-Solution Fit, define and verify its business model and eventually reach Product-Market-Fit.
</li>
<li>
Post-Product-Market Fit, the company becomes a scale up, needs to ramp up its marketing roadmap and effort, build and scale it's sales team, build mission-centric departments, hire new roles and recruit new competencies, etc.
</li>
</ul>
<p>
Dan Olsen designed the following pyramid to help figure what Product-Market Fit means (we'll be discussing this in length in this article):
</P>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/1098b6c5-6e4e-4846-84d6-0de114355ae5">
<img class="centered" style="width: 500px; " alt="Product Market Fit Pyramid" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/1098b6c5-6e4e-4846-84d6-0de114355ae5" />
</a>
<div class="centered">
</div>
</div>
<br>
<p>
Understanding Product-Market Fit and being able to measure and understand whether it's reached or not is crucial. Reaching PMF should be the core focus of a startup in its search phase and understanding whether it's reached is key before scaling up.
<br>
This article is an in-depth overview of what Product-Market-Fit means and the various perspective regarding how to get there. We will present the Lean-Startup fundamentals required to understand the process and the tools to reach product market fit, along with the Design thinking fundamentals, the metrics required to measure it, etc.
</p>
<!-- The Search for Product-Market Fit
TODO: replace design thinking circle with inner arrows
-->
<p>
The Search for Product-Market Fit is the sinews of war in a startup company. While the concept and whereabouts are well known by most founders, the importance of this event in the company and product building process, what it means to be Before-Product-Market-Fit and After-Product-Market-Fit and the fundamental differences in terms of objectives, processes, culture, activities, etc. between these two very distinct states is almost always underestimated or misunderstood.
</p>
<p>
Product-Market Fit is this sweet spot that startups reach when they feel eventually that the market is really pulling out their product. It's what happens when they found out that they can only deliver as fast as the customers buys their product or when they can only add new servers in the SaaS cloud as fast as is required to sustain the rise in workload.
<br>
Product-Market Fit is so important because it has to be a turn point in the life of a young company.
</p>
<ul>
<li>Pre-Product-Market Fit, the startups needs to focus on the leanest possible ways to solve Problem-Solution Fit, define and verify its business model and eventually reach Product-Market-Fit.
</li>
<li>
Post-Product-Market Fit, the company becomes a scale up, needs to ramp up its marketing roadmap and effort, build and scale it's sales team, build mission-centric departments, hire new roles and recruit new competencies, etc.
</li>
</ul>
<p>
Dan Olsen designed the following pyramid to help figure what Product-Market Fit means (we'll be discussing this in length in this article):
</P>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/1098b6c5-6e4e-4846-84d6-0de114355ae5">
<img class="centered" style="width: 500px; " alt="Dan Olsen's Product Market Fit Pyramid" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/2546b5e4-b772-4a08-a2d5-9b5d58b6f07c" />
</a>
<div class="centered">
</div>
</div>
<br>
<p>
Understanding Product-Market Fit and being able to measure and understand whether it's reached or not is crucial. Reaching PMF should be the core focus of a startup in its search phase and understanding whether it's reached is key before scaling up.
<br>
This article is an in-depth overview of what Product-Market-Fit means and the various perspective regarding how to get there. We will present the Lean-Startup fundamentals required to understand the process and the tools to reach product market fit, along with the Design thinking fundamentals, the metrics required to measure it, etc.
</p>
<p>
This article is available as a <a href="https://www.slideshare.net/JrmeKehrli/the-search-for-productmarket-fit-237078155">Slideshare presentation</a> and a <a href="https://www.youtube.com/watch?v=k41qAv3NwqM">Youtube video</a>.
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. Introduction - Product Market Fit</a>
<ul>
<li><a href="#sec11">1.1 Startups ... </a></li>
<li><a href="#sec12">1.2 From Product-Market Fit to "Lean Startup"</a></li>
<li><a href="#sec13">1.3 Defining "Product-Market Fit"</a></li>
<li><a href="#sec14">1.4 Four myths about Product-Market Fit</a></li>
<li><a href="#sec15">1.5 Feeling Product-Market Fit</a></li>
<li><a href="#sec16">1.6 Dan Olsen's PMF Pyramid</a></li>
<li><a href="#sec17">1.7 A first high-level process</a></li>
</ul></li>
<li><a href="#sec2">2. Lean Startup Fundamentals</a>
<ul>
<li><a href="#sec21">2.1 Lean Startup </a></li>
<li><a href="#sec22">2.2 Key principles </a></li>
<li><a href="#sec23">2.3 The Feedback loop </a></li>
<li><a href="#sec24">2.4 The Four steps to the Epiphany </a></li>
<li><a href="#sec25">2.5 Customer development - the practices </a></li>
<li><a href="#sec26">2.6 Get out of the building</a></li>
<li><a href="#sec27">2.7 Problem interview</a></li>
<li><a href="#sec28">2.8 Solution interview</a></li>
<li><a href="#sec29">2.9 MVP</a></li>
<li><a href="#sec210">2.10 Fail Fast</a></li>
<li><a href="#sec211">2.11 Metrics Obsession</a></li>
<li><a href="#sec212">2.12 Pivot</a></li>
<li><a href="#sec213">2.13 The Lean Canvas</a></li>
<li><a href="#sec214">2.14 The Value Proposition Canvas</a></li>
</ul></li>
<li><a href="#sec3">3. Design Thinking Fundamentals</a>
<ul>
<li><a href="#sec31">3.1 Design Thinking </a></li>
<li><a href="#sec32">3.2 The Design Thinking Process </a></li>
<li><a href="#sec33">3.3 The Design Thinking Framework </a></li>
<li><a href="#sec34">3.4 Thinking Outside of the Box </a></li>
<li><a href="#sec35">3.5 Sum-up</a></li>
</ul></li>
<li><a href="#sec4">4. Reaching Product Market fit - Different perspectives</a>
<ul>
<li><a href="#sec41">4.1 The Lean Startup Perspective </a></li>
<li><a href="#sec42">4.2 The "MVP-Centric" Perspective </a></li>
<li><a href="#sec43">4.3 The "Lean Canvas-Centric" Perspective </a></li>
<li><a href="#sec44">4.4 The "Design Thinking-Centric" Perspective </a></li>
</ul></li>
<li><a href="#sec5">5. Measure Obsession</a></li>
<ul>
<li><a href="#sec51">5.1 Net Promoter Score</a></li>
<li><a href="#sec52">5.2 CLV to CAC Ratio</a></li>
<li><a href="#sec53">5.3 Retention Ratio / Curve </a></li>
<li><a href="#sec54">5.4 Growth Rate </a></li>
<li><a href="#sec55">5.5 Further readings: pirate metrics </a></li>
</ul></li>
<li><a href="#sec6">6. Conclusion </a></li>
</ul>
<a name="sec1"></a>
<h2>1. Introduction - Product Market Fit</h2>
<p>
<i>"The term product/market fit describes "the moment when a startup finally finds a widespread set of customers that resonate with its product'."</i>
<br>by Eric Ries
</p>
<p>
In this chapter, we will detail what is Product-Market Fit, what it means and how it's defined.
</p>
<a name="sec11"></a>
<h3>1.1 Startups ... </h3>
<p>
<b>A startup</b>
</p>
<p>
We can find multiple definitions of a <i>Startup</i> online:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/dc7f3495-3c31-4f89-a3a8-941756a9aa89">
<img class="centered" style="width: 600px;" alt="various definitions of a startup" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/dc7f3495-3c31-4f89-a3a8-941756a9aa89" />
</a>
</div>
<br>
<p>The <i>Most of the time ...</i> definition is from my perspective downright wrong. A startup is not a scaled down version of a company. There are important inherent differences between a running company and a startup (we'll come back to this later)
</p>
<p>
The <i>wikipedia</i> definition is better. There is the important notion of search - a startup is indeed still searching the Product-Market Fit, the very important notion of validating - need to make assumptions, test them and confirm or contradict them and the notion of scaling.
</p>
<p>
But Eric Ries' definition is the best from my perspective since it emphasizes three important aspects of a startup:
</p>
<ol>
<li>The Notion of Human institution is better than company or project - a startup can be many things, a guy working in his garage on his idea is in some ways a startup</li>
<li>The notion of <b>new</b> product or service</li>
<li>Most importantly, the notion of <b>Extreme uncertainty</b> - that's the root of the problem</li>
</ol>
<p>
<b>Startup often fail!</b>
</p>
<p>
Most frequently cited reasons for startup failures:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/2d0ca791-67b5-4f5b-8c08-e887878a1a3a">
<img class="centered" style="width: 800px; " alt="Top reasons why startup fail" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/2d0ca791-67b5-4f5b-8c08-e887878a1a3a" />
</a><br>
<div class="centered">
(Source: <a href="https://www.statista.com/chart/11690/the-top-reasons-startups-fail/">CB Insights - https://www.statista.com/chart/11690/the-top-reasons-startups-fail/
</a>)
</div>
</div>
<br>
<p>
One of the main reasons why products fail is because they don't meet customer needs in a way that is better than other alternatives. That is the essence of <b>product-market fit</b>.
</p>
<p>
Most startup fail because the founder have an idea, work in a tunnel for multiples months or even year to build their idea, and only eventually rise their head looking for customers and face the ugly truth: the idea may well be brilliant indeed, but there's no market. I've seen this so many times, so many times.
<br>
Whenever one has an idea of a product, a new technology, before writing the single Line of Code, before investing a single dollar on it, one needs to answer the single and only question that matters: <i>Is there a market for it and what is that market?</i>
</p>
<p>
In the reasons why startups fail mentioned above, there are a few others interesting things to note:
</p>
<ul>
<li>Ran out of cash - why spend so much cash before Product / Market Fit. If your each Product-Market fit before heavily investing in your product / company, you can't run out of cash because investors will be killing themselves to put money in your product / company. When you reached Product - Market fit, you WILL have the data points to raise investments, BIG investments.
</li>
<li>
Not the right team - when you reach Product Market fit , you can raise investments and then hire the right team - Founders do not necessarily need to remain CEO and COO of their company.
</li>
</ul>
<p>
Interestingly, The Lean startup Methodology with its processes, principles and practices gives a solution to most of the top 10 problems listed above. Today, I intend to focus a lot on Lean Startup in this article.
</p>
<a name="sec12"></a>
<h3>1.2 From Product-Market Fit to "Lean Startup"</h3>
<p>
The "<b>Product-market fit</b>" term and concept is widely misattributed to Marc Andreessen by bloggers and writers, but Andy Rachleff coined the term. In a 2007 article, "The only thing that matters," Andreessen credits Rachleff for the term and synthesizes much of Rachleff's thinking
</b>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/159d63a7-0203-4507-a1a8-a7d0c91bfb0c">
<img class="centered" style="width: 800px;" alt="History of Lean startup" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/159d63a7-0203-4507-a1a8-a7d0c91bfb0c" />
</a>
</div>
<br>
<p>
Alex Osterwalder is a swiss guy living in Lausanne. He wrote "<i>Business Model Generation</i>" where he presents the business model canvas and the lean approach to it along with a lot of hints on how to efficiently (and cheaply) relieve uncertainties in startup with concepts such as prototyping, getting feedback from the market, challenging the status quo, etc.
<br>
Eric Ries is a Silicon Valley Serial Entrepreneur with failures and successes. His failures make him think of them and come up with the <b>Lean Startup</b> way, putting the customer at the center of the process.
<br>
Ash Maurya is an entrepreneur from Austin that understood on his end as well most of the Lean Startup principles. In his Running Lean book he details a lot of the Lean Startup principles and practices and came up with the Lean Canvas, a version of Osterwalder's Business Model Canvas adapted for Startups.
<br>
Steve Blank is the grand-father of the Lean Startup. He is a professor in Stanford University and a Silicon Valley Serial entrepreneur (search for him on Linkedin) - Steve Blank designed the <b>customer development</b> methodology that he presents in his "<i>Four Steps to the Epiphany</i>" book. <i>Four steps to the Epiphany</i> is a process for finding Product Market Fit and eventually scaling the company.
</p>
<p>
With Lean Startup, Product Market Fit is the step separating a startup from a scale up and Customer development methodology is the path to Product Market Fit.
</p>
<a name="sec13"></a>
<h3>1.3 Defining "Product-Market Fit"</h3>
<p>
The product/market fit (PMF) concept was developed and named by Andy Rachleff.
</p>
<p>
It answers the question some might wonder about: what correlates the most to success, team, product, or market?
<br>
Or, more bluntly, what causes success? And, for those of us who are students of startup failure, what's most dangerous: a bad team, a weak product, or a poor market?
<br>
If you ask entrepreneurs or VCs which of team, product, or market is most important, many will say team.
<br>
On the other hand, if you ask engineers, many will say product.
<br>
But actually market is the most important factor in a startup's success or failure.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/de1a7c40-4a0a-48c0-a580-06e9b51bd455">
<img class="centered" style="width: 700px; " alt="Rachleff's laws of Startup" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/de1a7c40-4a0a-48c0-a580-06e9b51bd455" />
</a><br>
<div class="centered">
(Source: June 25, 2007 / Marc Andreessen - The only thing that matters (blog post))
</div>
</div>
<br>
<p>
In a great market - a market with lots of real potential customers - the <b>market pulls product out</b> of the startup.
<br>
The market needs to be fulfilled and the market will be fulfilled, by the first viable product that comes along.
<br>
The product doesn't need to be great; it just has to basically work. And, the market doesn't care how good the team is, as long as the team can produce that viable product.
<br>
Conversely, in a terrible market, you can have the best product in the world and an absolutely killer team, and it doesn't matter -- you're going to fail.
</p>
<p>
You can obviously screw up a great market - and that has been done, and not infrequently - but assuming the team is baseline competent and the product is fundamentally acceptable, a great market will tend to equal success and a poor market will tend to equal failure. Market matters most.
<br>
To quote Tim Shephard : <i>"A great team is a team that will always beat a mediocre team, given the same market and product." </i>
</p>
<p>
Second question: Can't great products sometimes create huge new markets?
<br>
And the answer is yes, this is possible. But it's really exceptional.
<br>
VMWare is a good example of this since it was so profoundly transformative out of the box that it catalyzed a whole new movement towards operating system virtualization, which turned out to be a monster market. But then again this is an exception.
</p>
<p>
Rachleff's corollary of startup success gives us the first and most crucial definition of Product Market Fit :
</p>
<div class="centering">
<div class="centered">
<b>
"Product market fit means being in a good market with a product that can satisfy that market"
</b>
</div>
</div>
<br>
<p>
<i>A good market is essential</i>, there needs to be a market and the product needs to <i>satisfy that market</i>, give a solution to the market's problem.
</p>
<p>
<b>So what is Product-Market Fit?</b>
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/315715d1-8236-46cb-a418-55eccf47307b">
<img class="centered" style="width: 550px; " alt="Human centric View of Product Market Fit" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/315715d1-8236-46cb-a418-55eccf47307b" />
</a><br>
<div class="centered">
(Source: <a href="https://medium.com/@briantod/about-product-market-fit-what-ive-learned-about-the-goal-the-process-and-the-nuance-e7b317740f43
">https://medium.com/@briantod/about-product-market-fit-what-ive-learned-about-the-goal-the-process-and-the-nuance-e7b317740f43
</a>)
</div>
</div>
<br>
<p>
Looking at things from a Human Centered Design perspective - Product Market fit is the intersection between:
</p>
<ul>
<li>A problem that a sizeable group of people really need solved = i.e.: Desirability</li>
<li>A product that can actually be built well to fully solve that problem = i.e.: Feasibility</li>
<li> business model that can be executed to be profitable at some point in time = i.e.: Viability</li>
</ul>
<p>
If all three of these elements are not well identified, assessed, controlled and balanced, you can't reach PMF.
</p>
<ul>
<li>If desirability is weak, not many people want what you can make and believe you can sold => waste</li>
<li>If feasibility is weak, your are not able to build the product => failure</li>
<li>If viability is weak, you're not making money => failure</li>
<li>Etc.
</ul>
<p>
<b>Another view</b>
</p>
<p>
Product-market fit occurs when your product or service solves a problem that directly affects your target customers/audience. But it can't be just a few people. It's got to fill a gap and fix a problem for a large market. So when approximately 40% of your customers say that they can't imagine living or working without, then you know you have product-market fit.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/993c07ab-aca2-49f6-b506-665737cff7ac">
<img class="centered" style="width: 450px; " alt="Human centric View of Product Market Fit" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/993c07ab-aca2-49f6-b506-665737cff7ac" />
</a><br>
<div class="centered">
(Source: <a href="https://startupdevkit.com/guide-to-product-market-fit-with-everything-you-need-to-know/">https://startupdevkit.com/guide-to-product-market-fit-with-everything-you-need-to-know/
</a>)
</div>
</div>
<br>
<p>
However, there are times when there's a gap in the market, but most of the target audience doesn't know about this gap. That is, they don't know about the problem until you show them how your solution improves their lives. By doing this, you create the need for the market and you directly solve an existing problem in their niche or market.
</p>
<a name="sec14"></a>
<h3>1.4 Four myths about Product-Market Fit</h3>
<p>
(Source : <a href="https://blog.pmarca.com/2010/03/20/the-revenge-of-the-fat-guy/">https://blog.pmarca.com/2010/03/20/the-revenge-of-the-fat-guy/</a>)
</p>
<p>
<b>MYTH 1 : Product market fit is always a discrete, big bang event</b>
</p>
<p>
Some companies achieve primary product market fit in one big bang. But Most don't, instead getting there through partial fits, a few false alarms, and a big pile of perseverance.
Most of the time it's a lot of trial and error, running around it until finally all indicators are green
</p>
<p>
<b>MYTH 2: It's patently obvious when you have product market fit</b>
<p>
I am sure that Twitter knew when it achieved product market fit, but it's far blurrier for most startups. Twitter is a good example because most of these myths were actually true for Twitter. But Twitter is an exception.
<br>
Determining if you reached product market fit will require you to have very good understanding of your market and your product and give a lot of thoughts into coming up with the proper metrics and indicators and the recipe to interpret them.
<br>
Pretty much every product and market will require a different set of indicators and a specific way to interpret them.
</p>
<p>
<b>MYTH 3: Once you achieve product market fit, you can't lose it.</b>
<br>
And
<br>
<b>MYTH 4: Once you have product-market fit, you don't have to sweat the competition.</b>
</p>
<p>
These two myths really go together and are obviously wrong.
<br>
It's fine to stay lean if you are not quite sure that you have product market fit and there are no competitors in your face every day. But usually there are. In fact, the best markets are usually the ones in which competition is fierce because the opportunity is big. The number and quality of competitors is actually a fairly good indicator of the market.
<br>
The big principle here is that post PMF, monitoring and watching competitors closely should be an every day concern, just as staying very close the customer and keeping to run lean.
</p>
<a name="sec15"></a>
<h3>1.5 Feeling Product-Market Fit</h3>
<p>
These two quotes from Marc Andreessen are spot on and give a good perspective on how Product Market Fit can be felt.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a6571420-ca42-4ac2-ac23-1cb9a2c47874">
<img class="centered" style="width: 650px; " alt="Feeling Product Market Fit" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a6571420-ca42-4ac2-ac23-1cb9a2c47874" />
</a><br>
<div class="centered">
(Source: <a href="https://hackernoon.com/product-market-fit-heres-why-youre-probably-confused-about-it-1dr73zgd
">https://hackernoon.com/product-market-fit-heres-why-youre-probably-confused-about-it-1dr73zgd
</a>)
</div>
</div>
<br>
<a name="sec16"></a>
<h3>1.6 Dan Olsen's PMF Pyramid</h3>
<p>
Dan Olsen represents Product-Market Fit using the following pyramid:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/1098b6c5-6e4e-4846-84d6-0de114355ae5">
<img class="centered" style="width: 500px; " alt="Product Market Fit Pyramid" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/2546b5e4-b772-4a08-a2d5-9b5d58b6f07c" />
</a><br>
<div class="centered">
(Source: <a href="https://www.mindtheproduct.com/the-playbook-for-achieving-product-market-fit/
">https://www.mindtheproduct.com/the-playbook-for-achieving-product-market-fit/
</a>)
</div>
</div>
<br>
<p>
<b>The Problem Space</b>
</p>
<p>
A market is a set of related customer needs, which rests squarely in problem space or you can say "problems" define market, not "solutions". A market is not tied to any specific solutions that meet market needs. It is a broader space. There is no product or design that exists in problem space. Instead, problem space is where all the customer needs that you'd like your product to deliver live. You shouldn't interpret the word "needs" too narrowly: Whether it's a customer pain point, a desire, a job to be done, or a user story, it lives in problem space.
</p>
<p>
<b>The Solution Space</b>
</p>
<p>
If I speak of solution space, any product or the product design - such as mock-ups, wire-frame, prototype, depends on and is built upon problem space, but is in solution space. So we can say problem space is at the base of solution space. Solution space includes any product or representation of a product that is used by or intended for use by a customer. When you build a product, you have chosen a specific implementation. Whether you've done so explicitly or not, you've determined how the product looks, what it does, and how it works.
</p>
<p>
<b>The What and How Approach</b>
</p>
<p>
"What" the product needed to accomplish for customers is Problem space. The "what" describes the benefits product should give to the target customer.
Whereas, "how" the product would accomplish it, is solution space. The "how" is the way in which the product delivers the "what" to target customer. The "how" is the design of the product and the specific technology used to implement the product.
</p>
<p>
The best problem space learning often comes from feedback you receive from customers on the solution space.
</p>
<a name="sec17"></a>
<h3>1.7 A first high-level process</h3>
<p>
Based on Dan Olsen's pyramid, we can introduce here already a first idea of a process to reach Product-Market Fit:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/697409ef-e06a-467d-ac37-13b1c84b08d3">
<img class="centered" style="width: 700px;" alt="A first idea of a process" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/697409ef-e06a-467d-ac37-13b1c84b08d3" />
</a>
</div>
<br>
<p>
This is a first idea only, we will see different processes and different perspective in chapter <a href="#sec4">Different Perspective</a>
</p>
<a name="sec2"></a>
<h2>2. Lean Startup Fundamentals</h2>
<p>
In this chapter we will be covering the most fundamentals aspects of Lean Startup required to understand the different perspective in the Search for Product-Market Fit presented in chapter <a href="#sec1">1. Introduction</a>.
</p>
<p>
<b>The Lean Startup</b> is a movement, initiated and supported by some key people that I presented in the previous section.
<br>
But it's also a framework, an inspiration, an approach, a methodology with a set of fundamental principles and practices for helping entrepreneurs increase their odds of building a successful startup.
<br>
Lean Startup cannot be thought as a set of tactics or steps. Don't expect any checklist (well, at least not only checklists) or any recipe to be applied blindly, but it gives you a set or processes, principles and practices to reach Product-Market Fit and eventually scale the company.
</p>
<a name="sec21"></a>
<h3>2.1 Lean Startup </h3>
<p>
<b>Lean Movement (1990)</b>
</p>
<p>
<b>Lean thinking</b> is a <b>business methodology</b> that aims to provide a new way to think about how to organize human activities to deliver more benefits to society and value to individuals while <b>eliminating waste</b>.
<br>
The aim of lean thinking is to create a <b>lean enterprise</b>, one that <b>sustains growth</b> by aligning customer satisfaction with employee satisfaction, and that <b>offers innovative products</b> or services profitably while <b>minimizing unnecessary over-costs</b> to customers, suppliers and the environment.
<br>
The Lean Movement finds its roots in Toyotism and values <b>performance, visual management</b> (Kanban) and <b>continuous improvement</b> (Kaizen)
</p>
<p>
<b>Lean Startup (2010)</b>
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/223c22b4-f903-4441-92ce-51d1e227ceb3">
<img class="centered" style="width: 800px;" alt="Lean Startup origins" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/223c22b4-f903-4441-92ce-51d1e227ceb3" />
</a>
</div>
<br>
<p>
Blank, Ries, Osterwalder and Maurya are the founders or initiators of the <i>Lean Startup Movement</i>. Eric Ries is considered as the leader of the movement, while Steve Blank considers himself as its godfather.
<br>
Osterwalder and Maurya's work on business models is considered to fill a gap in Ries and Blank's work on processes, principles and practices. In Steve Blank's "The four Steps the the Epiphany", the business model section is a vague single page.
<br>
Moreover, Maurya's "<i>Running Lean</i>" magnificently completes Blank's work on <i>Customer Development</i>. We'll get to that.
</p>
<a name="sec22"></a>
<h3>2.2 Key principles </h3>
<p>
Before digging any further into Lean Startup, below are the essential principles that characterize <i>The Lean Startup</i> approach, as reported by Eric Ries' book.
</p>
<p>
<b>Entrepreneurs are everywhere</b>
</p>
<p>
You don't have to work in a garage to be in a startup. The concept of entrepreneurship includes anyone who works within Eric Ries' definition of a startup, which I repeat here:
</p>
<div class="centering">
<div class="centered">
<b>A startup is a human institution designed to create new products and services under conditions of extreme uncertainty.</b>
</div>
</div>
<br>
<p>
That means entrepreneurs are everywhere and the Lean Startup approach can work in any size company, even a very large enterprise, in any sector or industry.
</p>
<p>
<b>Entrepreneurship is management</b>
</p>
<p>
A startup is an institution, not just a product, and so it requires a new kind of management specifically geared to its context of extreme uncertainty.
<br>
In fact, Ries believes "<i>entrepreneur</i>" should be considered a job title in all modern companies that depend on innovation for their future growth
</p>
<p>
<b>Validated learnings</b>
</p>
<p>
Startups exist not just to make stuff, make money, or even serve customers. They exist to learn how to build a sustainable business. This learning can be validated scientifically by running frequent experiments that allow entrepreneurs to test each element of their vision.
</p>
<p>
<b>Innovation accounting</b>
</p>
<p>
To improve entrepreneurial outcomes and hold innovators accountable, we need to focus on the boring stuff: how to measure progress, how to set up milestones, and how to prioritize work. This requires a new kind of accounting designed for startups-and the people who hold them accountable.
</p>
<p>
<b>Build-Measure-Learn</b>
</p>
<p>
The fundamental activity of a startup is to turn ideas into products, measure how customers respond, and then learn whether to <b>pivot or persevere</b>. All successful startup processes should be geared to accelerate that <b>feedback loop</b>.
</p>
<a name="sec23"></a>
<h3>2.3 The Feedback loop </h3>
<p>
The feedback loop is represented as below.
<br>
The five-part version of the <i>Build-Measure-Learn</i> schema helps us see that the real intent of building is to test "<i>ideas</i>" - not just to build blindly without an objective.
<br>
The need for "<i>data</i>" indicates that after we measure our experiments we'll use the data to further refine our learning. And the new learning will influence our next ideas. So we can see that the goal of Build-Measure-Learn isn't just to build things, the goal is to build things to validate or invalidate the initial idea.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/6da0f813-d05e-420f-92e8-178f31b2122b">
<img class="centered" style="width: 800px;" alt="The feedback loop" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/6da0f813-d05e-420f-92e8-178f31b2122b" />
</a>
</div>
<br>
<p>
Again, the goal of <i>Build-Measure-Learn</i> is not to build a final product to ship or even to build a prototype of a product, but to <b>maximize learning</b> through incremental and iterative engineering.
<br>
In this case, learning can be about product features, customer needs, distribution channels, the right pricing strategy, etc.
<br>
The "<i>build</i>" step may refer to building an <b>MVP</b> - Minimal Viable Product - or simply a prototype, mock-up or even simply a hand sketch, whatever works for collecting market / customer feedback.
<br>
<p>
In the end, the <i>Build-Measure-Learn</i> framework let startups be fast, agile and efficient by validating every single assumption of the Problem, The Solution fit and the Business Model before consenting to any heavy investment.
</p>
<a name="sec24"></a>
<h3>2.4 The Four steps to the Epiphany </h3>
<p>
Most startups lack a process for discovering their markets, locating their first customers, validating their assumptions, and growing their business.
<br>
The <b>Customer Development Model</b> creates the process for these goals.
</p>
<p>
The life of any startup can be divided into two parts: before product/market fit (call this "BPMF") and after product/market fit ("APMF").
<br>
When you are BPMF, focus obsessively on getting to product/market fit.
<br>
Do whatever is required to get to product/market fit. Including changing out people, rewriting your product, moving into a different market, telling customers no when you don't want to, telling customers yes when you don't want to, raising that fourth round of highly dilutive venture capital, whatever is required! When you get right down to it, you can ignore almost everything else.
</p>
<p>
Whenever you see a successful startup, you see one that has reached product/market fit, and usually along the way screwed up all kinds of other things, from channel model to pipeline development strategy to marketing plan. And the startup is still successful.
</p>
<p>
PMF means it's safe to scale !!
<br>
If you decide to scale-up a SaaS company without proven Product/Market Fit, you're taking a huge risk. There's no guarantee that a market for your product exists. Even if it does, it might not be able to sustain your business.<br>
Without PMF, major investments into marketing, sales and customer success are premature.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a4aff1ad-cca8-416b-b499-550b9173a41e">
<img class="centered" style="width: 800px;" alt="The four steps to the Epiphany" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a4aff1ad-cca8-416b-b499-550b9173a41e" />
</a>
</div>
<br>
<ol>
<li><b>Customer discovery</b>: is to determine who your customers are, and whether the problem you're solving is important to them. In this phase, you may spend a lot of time conducting primary research, with surveys and interviews, or looking through secondary research. For example, in the case of Uber, Travis Kalanick, decided to build the business model as a private black cab service for himself. Gradually as the service was shared with friends, they began to realize demand from others.
</li>
<li><b>Customer validation</b>: is when you build a sales process that can be repeated by a sales and marketing team. This process is validated by selling the product to early customers for money. In the case of Uber, customers were paying for the ride from the get go, hence the business model was validated. And for Facebook, in its early days, Mark Zuckerberg was selling banners to local college businesses as a proof that the freemium monetization model will work.
</li>
<li><b>Customer creation / Get new Customers</b>: seeks to increase demand for a product and scale the business. In the case of Uber, the referral bonus program with ride subsidies was the key to its rapid growth, or customer creation.
</li>
<li><b>Customer building / Company Creation</b>: is when a company transitions into a more formalized structure where different specialized departments are created to specialize functions such as sales, marketing, and business development.
</li>
</ol>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/f8101263-5b88-43c1-9013-6639f43fdd81">
<img class="centered" style="width: 800px;" alt="The four steps to the Epiphany - Details" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/f8101263-5b88-43c1-9013-6639f43fdd81" />
</a>
</div>
<br>
<p>
Shortly put, Steve Blank proposes that companies need a <b>Customer Development process</b> that complements, or even in large portions replaces, their <i>Product Development Process</i>. The <i>Customer Development process</i> goes directly to the theory of <a href="https://en.wikipedia.org/wiki/Product/market_fit">Product/Market Fit</a>.
<br>
In "<i>The four steps to the Epiphany</i>", Steve Blank provides a roadmap for how to get to Product/Market Fit.
</p>
<a name="sec25"></a>
<h3>2.5 Customer development - the practices </h3>
<p>
So I want to present the most essentials principles and practices introduced and discussed by <i>the Lean Startup</i> approach.
<br>
These principles and practices are presented on the following schema attached to the stages of the <i>Customer Development</i> process where I think they make more sense:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/29ff320d-3e48-4261-b2ad-a165165fd1e4">
<img class="centered" style="width: 800px; " alt="Lean Startup Practices" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/29ff320d-3e48-4261-b2ad-a165165fd1e4" />
</a>
</div>
<br>
<p>
We will be focusing now on the relevant practices to reach product-market fit.
</p>
<a name="sec26"></a>
<h3>2.6 Get out of the building</h3>
<p>
<b>If you're not Getting out of the Building, you're not doing Customer Development and Lean Startup.</b>
<br>
There are no facts inside the building, only opinions.
</p>
<p>
If you aren't actually talking to your customers, you aren't doing Customer Development. And talking here is really speaking, with your mouth. Preferably in-person, but if not, a video call would work as well, messaging or emailing doesn't.
</p>
<p>
As Steve Blank said "<i>One good customer development interview is better for learning about your customers / product / problem / solution / market than five surveys with 10'000 statistically significant responses.</i>"
</p>
<p>
The problem here is that tech people, especially software engineers, try to avoid going out of the building as much as possible. But this is so important. Engineers need to fight against their nature and get out of the building and talk to customers as much as possible; find out who they are, how they work, what they need and what your startup needs to do, to build and then sell its solution.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/2d7ea9a3-3537-47f3-8a28-abdf0a4220e6">
<img class="centered" style="width: 800px; " alt="Get Out of the Building" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/2d7ea9a3-3537-47f3-8a28-abdf0a4220e6" />
</a>
</div>
<br>
<p>
Again, getting out of the building is not getting in the parking lot, it's really about getting in front of the customer.
<br>
At the end of the day, it's about <i>Customer Discovery</i>. And <i>Customer Discovery</i> is not sales, it's a lot of listening, a lot of understanding, not a lot of talking.
</p>
<p>
A difficulty that people always imagine is that young entrepreneurs with an idea believe that they don't know anybody, so how to figure out who to talk to ?
<br>
But at the time of Linkedin, Facebook, Twitter, it's hard to believe one cannot find a hundred of people to have a conversation with.
</p>
<p>
And when having a conversation with one of them, whatever else one's asking (<a href="#sec27">Problem interview</a>, <a href="#sec28">Solution interview</a>), one should ask two very important final questions:
</p>
<ol>
<li>
"<i>Who else should I be talking to ?</i>"
<br>
And because you're a pushy entrepreneur, when they give you those names, you should ask "<i>Do you mind if I sit here while you email them introducing me ?</i>"
</li>
<li>
"<i>What should I have really asked you ?</i>"
<br>
And sometimes that gets into another half hour related to what the customer is <i>really</i> worried about, what's really the customer's problem.
</ol>
<p>
Customer Discovery becomes really easy once you realize you don't need to get the world's best first interview.
<br>
In fact its the sum of these data points over time, it's not one's just going to be doing one and one wants to call on the highest level of the organization.
<br>
In fact you actually never want to call on the highest level of the organization because you're not selling yet, you don't know enough.
<br>
What one actually wants is to understand enough about the customers, their problems and how they're solving it today, and whether one's solution is something they would want to consider.
</p>
<a name="sec27"></a>
<h3>2.7 Problem interview</h3>
<p>
Problem Interview is Ash Maurya's term for the interview you conduct to validate whether or not you have a real problem that your target audience has.
</p>
<p>
In the Problem Interview, you want to find out 3 things:
</p>
<ol>
<li><b>Problem</b> - What are you solving? - How do customers rank the top 3 problems?</li>
<li><b>Existing Alternatives</b> - Who is your competition? - How do customers solve these problems today?</li>
<li><b>Customer Segments</b> - Who has the pain? - Is this a viable customer segment?</li>
</ol>
<p>
Talking to people is hard, and talking to people in person is even harder. The best way to do this is building a script and sticking to it. Also don't tweak your script until you've done enough interviews so that your responses are consistent.
<br>
The main point is to collect the information that you will need to validate your problem, and to do it face-to-face, either in-person or by video call. It's actually important to see people and be able to study their body language as well.
</p>
<p>
The interview script - at least the initial you should follow until you have enough experience to build yours - is as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/b54ec66a-fda7-4938-94b4-10456335746d">
<img class="centered" style="width: 500px; " alt="Problem Interview" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b54ec66a-fda7-4938-94b4-10456335746d" />
</a>
</div>
<br>
<p>
If you have to remember just three rules for problem interviews here they are:
</p>
<ol>
<li>Do not talk about your business idea or product. You are here to understand a problem, not imagine or sell a solution yet.</li>
<li>Ask about past events and behaviours</li>
<li>No leading question, learn from the customer</li>
</ol>
<p>
After every interview, take a leap backwards, analyze the answers, make sure you understand everything correctly and synthesize the results.
<br>
After a few dozen of interviews, you should be a able to make yourself a clear understanding of the problem and initiate a few ideas regarding the solution to it.
<br>
Finding and validating your solution brings us to the next topic: the <i>Solution Interview</i>.
</p>
<p>
And what if a customer tells you that the issues you thought are important really aren't? Learn that you have gained important data.
</p>
<a name="sec28"></a>
<h3>2.8 Solution interview</h3>
<p>
In the Solution Interview, you want to find out three things:
</p>
<ol>
<li><b>Early Adopters</b> - Who has this problem? - How do we identify an early adopter?</li>
<li><b>Solution</b> - How will you solve the problems? - What features do you need to build?</li>
<li><b>Pricing/Revenue</b> - What is the pricing model? - Will customers pay for it?</li>
</ol>
<p>
The key point here is to understand how to come up with a solution fitting the problem, step by step getting to the right track with your prototype and also understanding what could be a pricing model.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/9ef49888-50eb-4778-bc52-3f4584eeabf7">
<img class="centered" style="width: 500px; " alt="Solution Interview" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/9ef49888-50eb-4778-bc52-3f4584eeabf7" />
</a>
</div>
<br>
<p>
A <i>demo</i> is actually important. Many products are too hard to understand without some kind of demo. If a picture is worth a thousand words, a demonstration is probably worth a million.
</p>
<p>
Identifying early adopters is also key.
<br>
Think of something: if one of the guys you meet tells you that you definitely hold something, ask him if he would want to buy it. If he says he would definitely buy it when it's ready and available, ask him if he would commit to this. If he says he commits to this, ask him if he would be ready to pay half of it now and have it when its ready, thus becoming a partner or an investor.
<br>
If you find ten persons committing on already paying for the solution you draw, you may not even need to search for investors, you already have them. And that is the very best proof you can find that your solution is actually something.
<br>
And customers or partners are actually the best possible type of investors.
</p>
<a name="sec29"></a>
<h3>2.9 MVP</h3>
<p>
The <b>Minimum Viable Product</b> is an engineering product with just the set of features required to gather <i>validated learnings</i> about it - or some of its features - and its continuous development.
<br>
This notion of <i>Minimum Feature Set</i> is key in the MVP approach.
</p>
<p>
The key idea is that it makes really no sense developing a full and finalized product without actually knowing what will be the market reception and if all of it is actually worth the development costs.
<br>
Gathering insights and directions from an MVP avoids investing too much in a product based on wrong assumptions. Even further, The <i>Lean Startup</i> methodology seeks to avoid assumptions at all costs, see <a href="#sec23">The Feedback Loop </a> and <a href="#sec5">Metrics Obsession</a>.
</p>
<p>
The <i>Minimum Viable Product</i> should have just that set of initial features strictly required to have a valid product, usable for its very initial intent, and nothing more. In addition these features should be as minimalist as possible but without compromising the overall <i>User Experience</i>. A car should move, a balloon should be round and bounce, etc.
<br>
when adopting an MVP approach, the MVP is typically put at disposal at first only to <i>early adopters</i>, these customers that may be somewhat forgiving for the "naked" aspect of the product and more importantly that would be willing to give feedback and help steer the product development further.
</p>
<p>
Eric Ries defines the MVP as:
</p>
<div class="centering">
<div class="centered">
<b>
"The minimum viable product is that version of a new product a team uses to collect the maximum amount of validated learning about customers with the least effort."
</b>
</div>
</div>
<br>
<p>
The definition's use of the words <i>maximum</i> and <i>minimum</i> means it is really not formulaic. In practice, it requires a lot of judgement and experience to figure out, for any given context, what MVP makes sense.
</p>
<p>
The following chart is pretty helpful in understanding why both terms <i>minimum</i> and <i>viable</i> are equally important and why designing an MVP is actually difficult:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/abd771cd-b5a4-4a74-91c4-03951407517f">
<img class="centered" style="width: 300px; " alt="Minimum and Viable" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/abd771cd-b5a4-4a74-91c4-03951407517f" />
</a>
</div>
<br>
<p>
When applied to a new feature of any existing product instead of a brand new product, the MVP approach is in my opinion somewhat different. It consists of implementing the feature itself not completely; rather, a mock-up or even some animation simulating the new feature should be provided.
<br>
The mock-up or links should be properly instrumented so that all user reactions are recorded and measured in order to get insights on the actual demand of the feature and the best form it should take (<a href="#sec331">Measure Obsession</a>),
<br>
This is called a <b>deploy first, code later</b> method.
</p>
<p>
<a href="http://www.expressiveproductdesign.com/minimal-viable-product-mvp/">Fred Voorhorst' work</a> does a pretty good job in explaining what an MVP is:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/756e983e-5fd7-4dc1-a395-f5f1a69747f8">
<img class="centered" style="width: 700px; " alt="MVP - How-To" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/756e983e-5fd7-4dc1-a395-f5f1a69747f8" />
</a><br>
<div class="centered">
(Fred Voorhorst - Expressive Product Design - <a href="http://www.expressiveproductdesign.com/minimal-viable-product-mvp/">http://www.expressiveproductdesign.com/minimal-viable-product-mvp/</a>)
</div>
</div>
<br>
<p>
Developing an MVP is most definitely not the same as developing a sequence of elements which maybe, eventually combine into a product. A single wheel is not of much interest to a user wanting a personal transporter like a car, as illustrated by the first line.
<br>
Instead, developing an MVP is about developing the vision. This is not the same as developing a sequence of intermediate visions, especially not, if these are valuable products by themselves. As an example, a skateboard will likely neither interest someone in search for a car, as illustrated by the second line.
</p>
<p>
Developing an MVP means developing a sequence of prototypes through which you explore what is key for your product idea and what can be omitted.
</p>
<p>
<b>Sidenote on Product Design Artifacts</b>
</pa>
<p>
<b>This is important</b>: The MVP is not the only solution to capture user feedback, there are multiple different tools.
<br>
For instance during the first customer solution interviews, something as stupid as a multiples hand sketch move around with your hands may well be sufficient to capture feedback.
<br>
You should always settle to the simplest form of demonstration that is required to capture the feedback you need from your customer to verify your assumption, your hypothesis or your idea.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/019896bb-51a3-415f-900c-1b184e6b70a3">
<img class="centered" style="width: 700px; " alt="Product Design Artifcats" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/019896bb-51a3-415f-900c-1b184e6b70a3" />
</a><br>
<div class="centered">
(Source: <a href="https://www.slideshare.net/LeanStartupConf/a-playbook-for-achieving-productmarket-fit">https://www.slideshare.net/LeanStartupConf/a-playbook-for-achieving-productmarket-fit
</a>)
</div>
</div>
<br>
<a name="sec210"></a>
<h3>2.10 Fail Fast</h3>
<p>
The key point of the "<b>fail fast</b>" principle is to quickly abandon ideas that aren't working. And the big difficulty of course is not giving up too soon on an idea that could potentially be working. should one find the right channel, the right approach.
<br>
Fail fast means getting out of planning mode and into testing mode, eventually for every component, every single feature, every idea around your product or model of change. <i>Customer development</i> is the process that embodies this principle and helps you determine which hypotheses to start with and which are the most critical for your new idea.
</p>
<p>
It really is OK to fail if one knows the reason of the failure, and that is where most people go wrong. Once a site or a product fails then one needs to analyse why it bombed. It's only then that one can learn from it.
<br>
The key aspect here is really learning. And learning comes from experimenting, <b>trying things, <a href="#sec5">measuring</a> their success and <a href="#sec212">adapting</a></b>.
<br>
An entrepreneur should really be a pathologist investigating a death and finding the cause of the failure. Understanding the cause of a failure can only work if the appropriate measures and metrics around the experiment are in place.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/559f0efe-95eb-4e4e-bdf1-3f6bb998932a">
<img class="centered" style="width: 350px; " alt="Success - what it really looks like" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/559f0efe-95eb-4e4e-bdf1-3f6bb998932a" />
</a>
</div>
<br>
<p>
Now failing is OK as long as we learn from it and as long as we <b>fail as fast as possible</b>. Again, the whole <i>lean</i> idea is to avoid waste as much as possible and there's no greater waste than keeping investing on something that can ultimately not work. Failing as fast as possible, adapting the product, <a href="#sec212">pivoting</a> the startup towards its next approach as soon as possible is key.
<br>
But then again, the big difficulty is not to give up too soon on something that could possible work.
</p>
<div class="centering">
<div class="centered">
<b>
Fail fast, <br>
Learn faster, <br>
Succeed sooner !
</b>
</div>
</div>
<br>
<p>
So how do you know when to turn, when to drop an approach and adapt your solution ? How can you know it's not too soon?
</p>
<p>
<a href="#sec5">Measure, measure, measure</a> of course!
</p>
<p>
The testing of new concepts, failing, and building on failures are necessary when creating a great product.
<br>
The adage, "<i>If you can't measure it, you can't manage it</i>" is often used in management and is very important in <i>The Lean Startup</i> approach. <br>
Lean Startup is about verifying all your assumptions and hypothesis, and the only way to verify them is to take measures, compute metrics, infer insights and adapt.
</p>
<a name="sec211"></a>
<h3>2.11 Metrics Obsession</h3>
<p>
In the <i>build-measure-learn</i> loop, there is measure ... <i>The Lean Startup</i> makes from measuring everything an actual obsession. And I believe that this is a damn' good thing.
<br>
Think of it: what if you have an idea regarding a new feature or an evolution of your product and you don't already have the metrics that can help you take a sound and enlightened decision? You'll need to introduce the new measure and wait until you get the data. Waiting is not good for startups.
</p>
<p>
This is why I like thinking of it as a <b>Metrics Obsession</b>. Measure everything, everything you can think of!
<br>
And repeat a hundred times:
</p>
<div class="centering">
<div class="centered">
<b>
I will never ever again think that ... <br>
Instead I will <i>measure</i> that ...
</b>
</div>
</div>
<br>
<p>
Or as Edward Deming said :
</p>
<div class="centering">
<div class="centered">
<b>
"In god we trust, all others must bring data"
</b>
</div>
</div>
<br>
<a href="#sec5">We'll come back to this</a>
<a name="sec212"></a>
<h3>2.12 Pivot</h3>
<p>
In the process of learning by iterations, a startup can discover through field returns with real customers that its product is either not adapted to the identified need, that it does not meet that need.
<br>
However, during this learning process, the startup may have identified another need (often related to the first product) or another way to answer the original need.
<br>
When the startup changes its product to meet either this new need or the former need in a different way, it is said to have performed a <b>Pivot</b>.
<br>
A startup can <i>pivot</i> several times during its existence.
</p>
<p>
A <i>pivot</i> is ultimately a <b>change in strategy</b> without <i>a change in vision</i>.
<br>
It is defined as a structured course correction designed <b>to test a new fundamental hypothesis</b> about the product, business model and engine of growth.
</p>
<p>
The vision is important. A startup is created because the founder has a vision and the startup is really built and organized around this vision. If the feedback from the field compromises the vision, the startup doesn't need to pivot, it needs to resign, cease its activities and another startup, another organization aligned to the new vision should perhaps be created.
</p>
<p>
There are various kind of pivots:
</p>
<ul>
<li><b>Zoom-In :</b> a single feature becomes the whole product </li>
<li><b>Zoom-Out :</b> the whole initial product becomes a feature of a new product </li>
<li><b>Customer segment :</b> Good product, bad customer segment </li>
<li><b>Customer need :</b> Repositioning, designing a completely new product (still sticking to the vision)</li>
<li><b>Platform : </b> Change from an application to a platform, or vice versa</li>
<li>Many others ...</li>
</ul>
<p>
<b>Pivot or Persevere</b>
</p>
<p>
Since entrepreneurs are typically emotionally attached to their product ideas, there is a tendency to hang in there too long. This wastes time and money. The pivot or persevere process forces a non-emotional review of the hypothesis.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/0ea0e3ba-a1ce-42cb-a805-c14c5a757537">
<img class="centered" style="width: 360px;" alt="Pivot or Persevere" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/0ea0e3ba-a1ce-42cb-a805-c14c5a757537" />
</a>
</div>
<br>
<p>
Unsuprisingly, knowing when to pivot is an art, not a science. It requires to be well thought through and can be pretty complicated to manage.
<br>
At the end of the day, knowing when to pivot or persevere requires experience and, more importantly, metrics: proper performance indicators giving the entrepreneur clear insights about the market reception of the product and the fitting of customer needs.
</p>
<p>
One thing seems pretty clear though, if it becomes clear to everyone in the company that another approach would better suit the customer needs, the startup needs to pivot, and fast.
</p>
<a name="sec213"></a>
<h3>2.13 The Lean Canvas</h3>
<p>
Evolution on Business Models and the relative processes were surprisingly missing or poorly addressed from Ries' and Blank's initial work.
<br>
Fortunately, Osterwalder and Maurya caught up and filled the gap.
</p>
<p>
<b>Business Model Canvas</b>
</p>
<p>
The <a href="https://en.wikipedia.org/wiki/Business_Model_Canvas">Business Model Canvas</a> is a strategic management template invented by Alexander Osterwalder and Yves Pigneur for developing new business models or documenting existing ones.
<br>
It is a visual chart with elements describing a company's value proposition, infrastructure, customers, and finances. It assists companies in aligning their activities by illustrating potential trade-offs.
</p>
<p>
<b>Lean Canvas</b>
</p>
<p>
The Lean Canvas is a version of the Business Model Canvas adapted by Ash Maurya specifically for startups. The Lean Canvas focuses on addressing broad customer problems and solutions and delivering them to customer segments through a unique value proposition.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/ef01cc87-68ed-4ec9-8fd4-ad9694fda417">
<img class="centered" style="width: 800px; " alt="Lean Canvas" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/ef01cc87-68ed-4ec9-8fd4-ad9694fda417" />
</a>
</div>
<br>
<p>
So how should one use the Lean Canvas?
</p>
<ol>
<li><b>Customer Segment and Problem</b><br>
Both Customer Segment and Problem sections should be filled in together.
<br>
Fill in the list of potential <i>customers</i> and <i>users</i> of your product, distinguish customers (willing to pay) clearly from users, then refine each and every identified customer segment. Be careful not no try to focus on a too broad segment at first, think of Facebook whose first segment was only Harvard students.
<br>
Fill in carefully the problem encountered by your identified customers.
<br>
Identify carefully your early adopters since they will help you test and refine your business model
</li>
<li><b>UVP - Unique Value Proposition</b><br>
For new products, the initial battle is about how to get noticed ? How will you get the customer's attention ?
<br>
The UVP is the unique characteristic of your product or your service making it different from what is already available on the market an that makes it worth the consideration of your customers. Focus on the main problem you are solving and what makes your solution different.
</li>
<li><b>Solution</b><br>
Filling this is initially is tricky, since knowing about the solution for real requires trial and error, build-measure-learn loop, etc. In an initial stage one shouldn't try to be to precise here and keep things pretty open.
</li>
<li><b>Channels</b><br>
This consists in answering: how should you get in touch with your users and customers ? How do you get them to know about your product ? Indicate clearly your communication channels.
<br>
it's one of the riskiest item on your canvas! Start testing from day 1! (Social networks, Newsletter, Ads, Friends, Events, SEO, Etc.)
</li>
<li><b>Revenue Stream and Cost Structure</b><br>
Both these sections should also be filled in together.
<br>
At first, at the time of the initial stage of the startup, this should really be focused on the costs and revenues related to launching the MVP (how to interview 50 customers ? Whats the initial burn rate ? etc.)
<br>
Later this should evolve towards an initial startup structure and focus on identifying the <i>break-even</i> point by answering the question : how many customers are required to cover my costs ?
</li>
<li><b>Key Metrics</b><br>
Ash Maurya refers to Dave McClure Pirate Metrics to identify the relevant KPIs to be followed : <br>
Aquisition - How do user find you ?<br>
Activation - Do user have a great first experience ?<br>
Retention - Do users come back ?<br>
Revenue - How do you make money ?<br>
Referral - Do users tell others ?
</li>
<li><b>Unfair Advantage</b><br>
This consists in indicating the adoption barriers as well as the competitive advantages of your solution. An <i>unfair advantage</i> is defined as something that cannot be copied easily neither bought.
<br>
Examples: Insider Information, Personal authority, A dream team, Existing customers, "Right" celebrity endorsement, Large network effect, Community, SEO ranking, Patents, Core values, etc.
</li>
</ol>
<p>
<b>Lean Startup : test your plan !</b>
</p>
<p>
Using the new "Build - Measure - Learn" diagram, the question then becomes, "What hypotheses should I test?". This is precisely the purpose of the initial Lean Canvas,
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/9400cf47-0a75-4f96-87ed-662a381ae070">
<img class="centered" style="width: 750px; " alt="Canvas Principle" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/9400cf47-0a75-4f96-87ed-662a381ae070" />
</a>
</div>
<br>
<p>
<b>Product Market Fit on the Lean Canvas</b>
</p>
<p>
The Lean-Canvas is a formidable tool to capture the assumptions and hypothesis leading to Product Market Fit.
<br>
Product Market Fit happens here on the Lean Canvas:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/6324b197-c8b1-4f92-8ebc-90819519dd39">
<img class="centered" style="width: 800px; " alt="Product Market Fit on the Lean Canvas" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/6324b197-c8b1-4f92-8ebc-90819519dd39" />
</a>
</div>
<br>
<p>
Filling up these two parts can be challenging, so another tool comes in the game to help identify Assumptions leading to Product-Market Fit.
</p>
<a name="sec214"></a>
<h3>2.14 The Value Propostion Canvas</h3>
<p>
The Value Proposition Canvas is a tool which can help ensure that a product or service is positioned around what the customer values and needs.
<br>
The Value Proposition Canvas was initially developed by Dr Alexander Osterwalder as a framework to ensure that there is a fit between the product and market. It is a detailed look at the relationship between two parts of the Osterwalder's broader Business Model Canvas; customer segments and value propositions.
<br>
The Value Proposition Canvas can be used when there is need to refine an existing product or service offering or where a new offering is being developed from scratch.
</p>
<p>
<b>Customer Profile</b>
</p>
<ol>
<li><b>Customer jobs</b> - the functional, social and emotional tasks customers are trying to perform, problems they are trying to solve and needs they wish to satisfy.</li>
A customer profile should be created for each customer segment, as each segment has distinct gains, pains and jobs.</li>
<li><b>Gains</b> - the benefits which the customer expects and needs, what would delight customers and the things which may increase likelihood of adopting a value proposition.</li>
<li><b>Pains</b> - the negative experiences, emotions and risks that the customer experiences in the process of getting the job done.</li>
</ol>
<p>
<b>Value Map</b>
</p>
<ol>
<li><b>Gain creators</b> - how the product or service creates customer gains and how it offers added value to the customer.</li>
<li><b>Pain relievers</b> - a description of exactly how the product or service alleviates customer pains.</li>
<li><b>Products and services</b> - the products and services which create gain and relieve pain, and which underpin the creation of value for the customer.</li>
</ol>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/2bc8b37e-f920-4b92-991e-85579c1d2c0b">
<img class="centered" style="width: 800px; " alt="Value Proposition Canvas" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/2bc8b37e-f920-4b92-991e-85579c1d2c0b" />
</a>
</div>
<br>
<p>
<b>Achieving fit between the value proposition and customer profile</b>
</p>
After listing gain creators, pain relievers and products and services, each point identified can be ranked from nice to have to essential in terms of value to the customer. A fit is achieved when the products and services offered as part of the value proposition address the most significant pains and gains from the customer profile.
Identifying the value proposition on paper is only the first stage. It is then necessary to validate what is important to customers and get their feedback on the value proposition. These insights can then be used to go back and continually refine the proposition.
<a name="sec3"></a>
<h2>3. Design Thinking Fundamentals</h2>
<p>
Just as the previous chapter intended to cover the Lean Startup fundamentals required to present the different perspective in the Search for Product-Market fit presented in the next chapter, this one covers the most essential Design Thinking Fundamentals
</p>
<a name="sec31"></a>
<h3>3.1 Design Thinking </h3>
<p>
Design Thinking is <b>an iterative process</b> in which we seek to <b>understand the user, challenge assumptions, and redefine problems</b> in an attempt to <b>identify alternative and innovative strategies and solutions</b> that might not be instantly apparent with our initial level of understanding (Creative thinking, Outside-the-box thinking, ...).
<br>
Design thinking is a way of thinking and working as well as a collection of hands-on methods.
</p>
<p>
The <i>Design Thinking Process</i> involves five phases: Empathize, Define, Ideate, Prototype and Test-it is most useful to tackle problems that are ill-defined or unknown.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/ce4f907e-cc10-4174-b533-214242dcafe6">
<img class="centered" style="width: 400px; " alt="Design Thinking Big Picture" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/ce4f907e-cc10-4174-b533-214242dcafe6" />
</a>
</div>
<br>
<p>
Design Thinking revolves around a deep interest in developing an understanding of the people for whom we're designing the products or services. Experience says customers are not likely to communicate their needs clearly. It's not how the human brain works. We have a natural tendency to think in terms of solutions.
<br>
Design Thinking is based on the assumption that designers' work processes can help us systematically extract, teach, learn, and apply these human-centered techniques to solve problems in a creative and innovative way. It is kind of a capture of the best practices in use by designers for ages, formalized and collected in a process and set of tools. It's an attempt to leverage designer's ways of working and thinking to other business fields where brainstorming is required to converge to a solution to a given problem.
</p>
<a name="sec32"></a>
<h3>3.2 The Design Thinking Process </h3>
<p>
Design thinking starts with Empathy and uses collaborative and participatory methods, repeating all 5 steps as many times as needed to achieve a complete solution.
<br>
The process helps not skipping to solution thinking before crystal clear problem understanding and formulation !
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/1418f19e-700b-4868-97b6-fcf12a168c3e">
<img class="centered" style="width: 700px; " alt="Design Thinking Process" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/1418f19e-700b-4868-97b6-fcf12a168c3e" />
</a>
</div>
<br>
<p>
Design Thinking is an iterative and non-linear process. This simply means that the design team continuously use their results to review, question and improve their initial assumptions, understandings and results. Results from the final stage of the initial work process inform our understanding of the problem, help us determine the parameters of the problem, enable us to redefine the problem, and, perhaps most importantly, provide us with new insights so we can see any alternative solutions that might not have been available with our previous level of understanding.
</p>
<ul>
<li>Get back to the customer for further refinement of the problem expression</li>
<li>Working on the prototype give new ideas : challenge them and reprioritize!</li>
<li>Tests give new ideas : challenge them and reprioritize!</li>
<li>Tests reveal insights that redefine the problem </li>
</ul>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/927f8e98-a7f9-4d1b-b216-f64a579e92e6">
<img class="centered" style="width: 650px; " alt="Design thinking - Different perspectrives" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/927f8e98-a7f9-4d1b-b216-f64a579e92e6" />
</a><br>
<div class="centered">
(Source : <a href="https://www.slideshare.net/ChrisJackson43/i-design-think-therefore-i-am-a-uxer">https://www.slideshare.net/ChrisJackson43/i-design-think-therefore-i-am-a-uxer</a>)
</div>
</div>
<br>
<p>
Designers don't become designers from day 1, there are design schools, it requires experience etc.
<br>
Design thinking is just the same. It requires a lot of practice and familiarity with the process and the tools to become good at it.
</p>
<a name="sec33"></a>
<h3>3.3 The Design Thinking Framework </h3>
<p>
This is intended as a map of the different tools and practices in use in the different stages of the design thinking process.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/f8d76eab-062b-4d82-966c-be39c28a2398">
<img class="centered" style="width: 800px; " alt="Design Thinking Framework" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/f8d76eab-062b-4d82-966c-be39c28a2398" />
</a>
</div>
<br>
<p>
I wont go any deeper today in detailing these practices and tools and let the reader google them.
<br>
I will likely dedicate a full article to design thinking on this very blog in the short term
</p>
<p>
At the end of the day, Design thinking is a lot about bringing Agility and Lean practices to the design and problem solving process.
<br>
In this perspective, it is different to the <i>traditional</i> thinking process in many ways, just as Agile Development is different than Waterfall Development.
</p>
<div class="centering">
<table class="centered" style="border: solid 1px #999999; border-collapse: collapse; padding: 4px;"><tr>
<td class="story" style="text-align: center; border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;"><b></b></td>
<td class="story" style="text-align: center; border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;"><b>Traditional thinking</b></td>
<td class="story" style="text-align: center; border-bottom: solid 1px #CCCCCC; padding: 4px;"><b>Design thinking</b></td>
</tr><tr>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;"><b>Style</b></td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;">Directed</td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px;">Emergent</td>
</tr><tr>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;"><b>Process</b></td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;">Planning of flawless intellect</td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px;">Enlightened trial and error</td>
</tr><tr>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;"><b>Path to success</b></td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;">Avoid failure, secure</td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px;">Fail fast</td>
</tr><tr>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;" rowspan=4><b>Factor of success</b></td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;">Expert Advantage</td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px;">Ignorance advantage</td>
</tr><tr>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;">Right answers</td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px;">Right questions</td>
</tr><tr>
<td class="story" style="padding: 3px; border-right: solid 1px #CCCCCC;">Rigorous Analysis</td>
<td class="story" style="padding: 3px;">Rigorous Testing</td>
</tr><tr>
<td class="story" style="padding: 3px; border-right: solid 1px #CCCCCC;">Subject experts</td>
<td class="story" style="padding: 3px;">Process experts</td>
</tr><tr>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;"><b>Rituals</b></td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;">Presentations and meetings</td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px;">Experiments and experiences</td>
</tr><tr>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;"><b>Communication</b></td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;">Telling</td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px;">Showing</td>
</tr><tr>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;"><b>Base</b></td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;">Headquarters</td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px;">In the field</td>
</tr><tr>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;" rowspan=2><b>Customer involvement</b></td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;">Arm's length customer research</td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px;">Deep customer immersion</td>
</tr><tr>
<td class="story" style="padding: 3px; border-right: solid 1px #CCCCCC;">Periodic</td>
<td class="story" style="padding: 3px;">Continuous</td>
</tr><tr>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;"><b>Activities</b></td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px; border-right: solid 1px #CCCCCC;">Thinking and planning</td>
<td class="story" style="border-bottom: solid 1px #CCCCCC; padding: 3px;">Doing</td>
</tr></table>
</div>
<br>
<a name="sec34"></a>
<h3>3.4 Thinking Outside of the Box </h3>
<p>
The best way to illustrate this key aspect of <i>Design Thinking</i> is with the following quote from Henri Ford:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/8acb4773-60a5-4d8f-9b5f-fa87afcdbe46">
<img class="centered" style="width: 700px; " alt="Henri Ford - faster horsed" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/8acb4773-60a5-4d8f-9b5f-fa87afcdbe46" />
</a>
</div>
<br>
<p>
I'd like to illustrate this quote with the following process as an example, to show what a over-simplied design thinking
process to the problem above could look like:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/2d02c29c-9884-4e23-a382-b27fa0791916">
<img class="centered" style="width: 800px;" alt="Design thinking Example process" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/2d02c29c-9884-4e23-a382-b27fa0791916" />
</a>
</div>
<br>
<a name="sec35"></a>
<h3>3.5 Sum-up</h3>
<p>
Design Thinking is essentially
</p>
<ul>
<li>a problem-solving approach specific to design, </li>
<li>which involves assessing known aspects of a problem and </li>
<li>identifying the more ambiguous or peripheral factors that contribute to the conditions of a problem. </li>
</ul>
<p>
It contrasts with a more scientific approach where the concrete and known aspects are tested in order to arrive at a solution.
</p>
<p>
Design Thinking is
</p>
<ul>
<li>an iterative process to identify alternative strategies and solutions that might not be instantly apparent with our initial level of understanding. </li>
<li>often referred to as "outside the box thinking', as designers are attempting to develop new ways of thinking that do not abide by the dominant or more common problem-solving methods - just like artists do. </li>
</ul>
<p>
At the heart of Design Thinking is the intention to improve products by analyzing how users interact with them and investigating the conditions in which they operate.
<br>
Design Thinking offers a means of digging that bit deeper to uncover ways of improving User eXperiences.
</p>
<a name="sec4"></a>
<h2>4. Reaching Product Market fit - Different perspectives</h2>
<p>
Now with all we have covered above - lean startup and design thinking fundamentals - we can come back to this very article topic, the <b>search for Product-Market Fit</b>.
<br>
When searching online articles or posts about Product Market Fit, you will mostly likely fall in one of the following four perspectives
</p>
<ol>
<li>
<b>The <i>Lean-Startup</i> perspective</b>: with actually two sub-cases that converge to the same thing:
<ul>
<li>
<b>The <i>Feedback-loop</i> perspective</b>: Searching Product-market fit is applying the <i>Build-Measure-Learn</i> feedback loop comprehensively throughout the product identification and design lifecycle and the business plan definition to shape a product fulfilling perfectly the customer needs.
</li>
<li>
<b>The <i>Four-Steps-to-the-Epiphany</i> perspective</b>: Product-market fit is the result of the search phase, when the solution to the customer problem is clearly identified along with its feature set, market potential, business plan, foreseen evolutions, etc.
</li>
</ul>
Again, these two perspectives actually converge to what I will describe hereunder as the <i>Lean Startup</i> Perspective.
</li>
<li>
<b>The <i>MVP-Centric</i> perspective</b>: For many, searching product-market fit is iterating around a <i>Minimum Viable Product</i>. It is the result of a process centered around the MVP design iterations, when the MVP and what we learned from it enabled to identify the product fulfilling the market needs.
</li>
<li>
<b>The <i>Lean-Canvas-Centric</i> perspective</b>: For others, Product-market fit happens when you succeeded in designing great value propositions that match your customer needs and jobs-to-be-done and helps solve their problems.
</li>
<li>
<b>The <i>Design Thinking</i> perspective</b>: Product-market-fit is what happens when applying successfully Lean-Startup principles to the last design-thinking process stages to reach maturity and the growth stage.
</li>
</ol>
<p>
These different perspectives are all, well, perspectives ... Different visions of the same thing: putting the customer needs at the center of the Solution Search and Design process and being <i>lean-by-the-book</i> as long as the Problem, the Solution, the Market and the Product along with its minimum features are not well identified.
<br>
We should now detail these different perspectives.
</p>
<a name="sec41"></a>
<h3>4.1 The Lean Startup Perspective </h3>
<p>
Again, actually the <i>Feedback loop</i> and the <i>Four Steps to the Epiphany</i> - both described and referenced often in the literature - converge to the very same thing: The Lean Startup way, which can be represented this way:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d3a297a2-0dba-4c91-903e-81589bd5ab57">
<img class="centered" style="width: 800px;" alt="The Lean Startup Perspective to Product Market Fit" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d3a297a2-0dba-4c91-903e-81589bd5ab57" />
</a>
</div>
<br>
<p>
The Lean Startup perspective to product market fit consists, well, in being Lean Startup by the book:
</p>
<ul>
<li>"<i>4-steps to the Epiphany</i>" as a high level process</li>
<li>Lower-level process is represented by the Lean-Startup Feedback Loop
<ul>
<li>First Problem-Solution Fit</li>
<li>Then Business-Model validation</li>
<li>Eventually Product-Market Fit</li>
</ul>
</li>
<li>MVP happens late in the process, most assumptions are verified during interviews with mock-ups and prototype (Lean-by-the-book)</li>
</ul>
<p>
At the end of the day Lean Startup <b>is</b> about reaching Product-Market Fit.
</p>
<p>
<b>Key Aspects</b>
</p>
<p>
The <i>Lean Startup</i> way to product market fit is fundamentally Customer-centric: <i>Get out of the building</i>, work with your customers:
</p>
<ul>
<li>Understand your customer's problem</li>
<li>Understand if your solution works for your customer!</li>
<li>Understand your market, capture its constraints, abilities, means.</li>
</ul>
<p>
It's lean-by-the-book, only very little investment should be made upfront, focus on Problem-Solution Fit first, then design your business mode, all of this can be done almost for free.
<br>
Only then one should develop an MVP - which requires some investment - at the latest stage, in any case after Problem-Solution Fit.
<br>
The whole process is fundamentally Data-driven: make an hypothesis, test it, measure, learn, adapt or persevere, move to next assumption, etc.
<br>
Product-Market Fit is reached when the metrics measured from the MVP confirms it.
</p>
<div class="centering">
<table class="centered" style="border: 0 none; border-collapse: collapse; margin: 2px; padding: 4px;"><tr>
<td class="story" style="text-align: center; border: 0 none; border-right: solid 1px #CCCCCC; padding: 3px;"><b>Advantages</b></td>
<td class="story" style="text-align: center; border: 0 none; padding: 4px;"><b>Drawbacks</b></td>
</tr><tr>
<td class="story" style="text-align: left; border: 0 none; border-right: solid 1px #CCCCCC; padding: 3px;">
<ul>
<li>
Little investment on MVP, everything remains theoretical before Problem-Solution Fit and Business Model validation - Lean by the book!
</li>
<li>
Only when most assumptions are verified one moves to developing MVP (which requires money!)
</li>
</ul>
</td>
<td class="story" style="text-align: left; border: 0 none; padding: 4px;">
<ul>
<li>
While this approach - working a lot upfront before starting to work on an MVP - is seducing (less investment required), it is also challenging
<ul>
<li>
The first feedback we get from the MVP often challenges a lot the initial assumptions, even though they were validated with customers.
</li>
<li>
This is because users and customers have a strong tendency not to be able to clearly state what they need and want before something concrete us put in front of them
</li>
</ul>
</li>
</ul>
</td>
</tr></table>
</div>
<br>
<a name="sec42"></a>
<h3>4.2 The "MVP-Centric" Perspective </h3>
<p>
The <i>MVP-centric</i> perspective is very similar to the <i>Lean-Startup</i> perspective, with only one fundamental difference.
<br>
Instead of remaining lean too long and focusing a lot on Problem / Solution fit and the Business Model Design from a theoretical perspective (through interview, design sessions with customers, etc.), for some it makes more sense to rush it to the MVP and capture better feedback based on something concrete, the MVP, instead of remaining nearly theoretical too long.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/70f2eef4-cddc-42a1-bee1-46ba1ac2e5b3">
<img class="centered" style="width: 600px;" alt="The MVP-Centric Perspective to Product Market Fit" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/70f2eef4-cddc-42a1-bee1-46ba1ac2e5b3" />
</a>
</div>
<br>
<p>
The MVP-Centric approach puts the iterations on MVP at the center of the process.
<br>
it is very relevant approach for online and very wide audience services, such as SaaS platform, online services, etc.
</p>
<p>
<b>Fundamental Idea </b>
</p>
<ul>
<li>
Jump to the MVP development stage as fast as possible - without neglecting Problem / Solution fit though - and iterate on MVP as long as required to reach PMF
</li>
<li>
Getting IRL - In Real Life - and live as fast as possible to optimize feedback
</li>
</ul>
<p>
<b>Note about MVP</b>
</p>
<p>
Refer to section <a href="#sec29">MVP</a>, the first version of the MVP in this context can very well be a prototype or a simple mock-up of even sketch-up.
<br>
What the MVP-centric approach is saying is that at every stage (Problem/Solution fit, etc.) you should have something concrete to present to the customer to have him react on something real, not just open questions or theoretical solutions.
<br>
In this perspective, every kind of feedback that is not measured on something real (in the sense of existing, as real as a simple sketch-up can be) is simply useless. This is opposed to the previous <i>Lean-by-the-book</i> perspective where the first stages can me made through simple interviews.
<br>
<p>
<b>Key aspects</b>
</p>
<p>
Some believe that reasoning on a concrete MVP is the best way (again, the definition of MVP in this context is wide)
<br>
The principles behind the underlying process are:
</p>
<ul>
<li>Build MVP, put it live in real-life and start getting feedback</li>
<li>Feedback and metrics collection automation is key</li>
<li>A/B Testing / UX Metrics / etc.</li>
</ul>
<p>
In contrary to the Lean-Startup perspective, in this approach there some more significant investment up-front required, one needs to develop the MVP.
<br>
The Problem-Solution fit search phase is not neglected, but shorten as much as possible to reach the more concrete MVP stage faster.
<br>
Product-Market Fit is reached when the metrics measured from the MVP confirms it (as usual).
</p>
<div class="centering">
<table class="centered" style="border: 0 none; border-collapse: collapse; margin: 2px; padding: 4px;"><tr>
<td class="story" style="text-align: center; border: 0 none; border-right: solid 1px #CCCCCC; padding: 3px;"><b>Advantages</b></td>
<td class="story" style="text-align: center; border: 0 none; padding: 4px;"><b>Drawbacks</b></td>
</tr><tr>
<td class="story" style="text-align: left; border: 0 none; border-right: solid 1px #CCCCCC; padding: 3px;">
<ul>
<li>
People have difficulties reasoning on abstractions and customers have trouble expressing clearly what they need as long as they don't see anything concrete (the infamous "that's not what I wanted"
</li>
<li>
Moving AFAP to the MVP enables to address this and get feedback on the real-thing as fast as possible
</li>
<li>
Very well suited for online and wide audience services (e.g. Netflix, Google, Facebook, etc.)
</li>
</ul>
</td>
<td class="story" style="text-align: left; border: 0 none; padding: 4px;">
<ul>
<li>
More up-front investment, need to develop the MVP and put it live
</li>
<li>
Does not apply to all businesses / products - how to develop an MVP in Pharma, Bio-tech or heavy industry?
</li>
</ul>
</td>
</tr></table>
</div>
<br>
<a name="sec43"></a>
<h3>4.3 The "Lean Canvas-Centric" Perspective </h3>
<p>
The <i>Lean Canvas-Centric</i> perspective is kind of the the symmetric of the <i>MVP-Centric</i> perspective if the <i>Lean Startup</i> perspective is the pivot point.
<br>
It's fundamental idea is the exact opposite of the MVP Centric approach: postponing the MVP and the investment required in it as much as possible and stay lean and theoretical as long as possible, even as long as is required to reach "<i>theoretical Product-Market fit</i>"
<br>
The Lean-Canvas centric approach puts a strong emphasis on the theoretical work and customer representatives / market experts interviews instead of the MVP
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/96fd46a9-5ba8-4123-9bf5-f449c649cd39">
<img class="centered" style="width: 800px;" alt="The Lean Canvas-Centric Perspective to Product Market Fit" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/96fd46a9-5ba8-4123-9bf5-f449c649cd39" />
</a>
</div>
<br>
<p>
<i>Theoretical Product-Market fit</i> would be defined as "<i>Designing a Product and a Business Model that has the potential to have Product-Market Fit</i>" as measured by whatever possible means to confirm most assumptions without having the actual product or even only an MVP.
<br>
The Lean Canvas and the Value proposition Canvas form a formidable tool to drive researches with customers towards Product-Market fit and discussion with Investors. They sum up the findings and validated assumptions based on theoretical work (such as prototype, academic research, scientific findings, etc.) leads to theoretical product market-fit before actually building anything on the product
<br>
Developing, evolving and maintaining these canvas is
</p>
<ul>
<li>Useful during the search phase for every startup</li>
<li>Essential when an MVP is not possible or very expensive. These canvas provide a guiding line for the theoretical search phase.</li>
</ul>
<p>
<b>Key aspects</b>
</p>
<p>
Designing a lean-Canvas, maintaining and evolving it , along with a Value proposition canvas, always makes sense and should always be done to drive the initial assumptions on the Problem solution fit, the Business Model and the Product Market-Fit and their evolutions.
<br>
But most of the time, coming back over and over again to the Lean Canvas is dropped in favor of iterations around the MVP and it's predominant usage to capture customer feedback and converge to Product-Market Fit.
<br>
When working with and around the MVP is not possible - heavy industry, Bio-Tech, Pharma, etc. - the lean canvas and its maintenance remains the principal guideline when searching for Product Market Fit
Every customer interview, expert consulting, scientific research should lead to evolving the Lean Canvas and the Value Proposition Canvas. The Lean canvas is the Big Picture of the Business Plan leading discussions with investors
<br>
The Lean canvas is the map to the data points that need to be collected before talking to investors.
</p>
<div class="centering">
<table class="centered" style="border: 0 none; border-collapse: collapse; margin: 2px; padding: 4px;"><tr>
<td class="story" style="text-align: center; border: 0 none; border-right: solid 1px #CCCCCC; padding: 3px;"><b>Advantages</b></td>
<td class="story" style="text-align: center; border: 0 none; padding: 4px;"><b>Drawbacks</b></td>
</tr><tr>
<td class="story" style="text-align: left; border: 0 none; border-right: solid 1px #CCCCCC; padding: 3px;">
<ul>
<li>
The Lean-Canvas helps create a quick visualization of an idea, share it and get feedback.
</li>
<li>
The Value Proposition canvas helps capture how Product Market fit will be reached.
</li>
<li>
Sometimes - when working with an MVP is not possible - every single word on these canvas capture an essential assumption that has been verified and that is key to build the eventual product and the company.
</li>
</ul>
</td>
<td class="story" style="text-align: left; border: 0 none; padding: 4px;">
<ul>
<li>
Again, if working with these canvas is important in the initial stage and perhaps at later stage when discussing with potential customers and investors, iterating and evolving these canvas takes a lesser importance in favor of working with an MVP as e move forward in the search phase.
</li>
</ul>
</td>
</tr></table>
</div>
<br>
<a name="sec44"></a>
<h3>4.4 The "Design Thinking-Centric" Perspective </h3>
<p>
The <i>Design Thinking-Centric</i> perspective is actually not a variation of the previous ones, but rather a complementary approach.
<br>
Lean startup doesn't tell a lot about how to conduct brainstorming and the thinking process towards solutions. This is where <i>Design Thinking</i> kicks in.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/26986622-a945-4c40-9f39-ccc923552b7d">
<img class="centered" style="width: 550px;" alt="The Design Thinking-Centric Perspective to Product Market Fit" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/26986622-a945-4c40-9f39-ccc923552b7d" />
</a>
</div>
<br>
<p>
The Design-Thinking Perspective is actually a way to structure and formalize the Problem-solving approach in the search phase.
<br>
There's some overlap between Lean Startup and Design Thinking
</p>
<ul>
<li>Design thinking emphasizes on Problem / Solution Fit, MVP design and UX as ultimate results from a Design perspective, </li>
<li>while Lean Startup focuses on reaching Product Market Fit before scaling.</li>
</ul>
<p>
Both are very complementary!
</p>
<ul>
<li>The design thinking process is a very good fit for the Lean-Startup Search Phase</li>
<li>Lean Startup doesn't give much processes and tools on how to get to Problem / Solution Fit (aside from some general principles, Get out of the building, Problem / Solution interview, etc.), how to design the MVP, etc.</li>
<li>This is where Design thinking kicks in.</li>
</ul>
<p>
Applying the Design-thinking process to the required brainstorming in the Lean-Startup search phase is a striking fit.
</p>
<p>
<b>Key aspects</b>
</p>
<p>
Lean-Startup insists on the need to reach Problem Solution Fit, a working Business Model and eventually Product Market fit and gives principles and practices to it (Get out of the Building, Converge to MVP, Lean Canvas, etc.).
<br>
But Lean startup doesn't give much recipe for how to conduct brainstorming, how to search for solution, how to design the MVP, etc.
</p>
<p>
Here comes Design thinking!
</p>
<ul>
<li>
The Design thinking process can be applied every time a solution to a problem, a design job or simply a brainstorming exercise has to be performed in the search phase … or after.
</li>
<li>
Design thinking and Lean Startup share some genes (getting feedback, iterate, etc.), but they are rather very much complementary with each other
</li>
</ul>
<div class="centering">
<table class="centered" style="border: 0 none; border-collapse: collapse; margin: 2px; padding: 4px;"><tr>
<td class="story" style="text-align: center; border: 0 none; border-right: solid 1px #CCCCCC; padding: 3px;"><b>Advantages</b></td>
<td class="story" style="text-align: center; border: 0 none; padding: 4px;"><b>Drawbacks</b></td>
</tr><tr>
<td class="story" style="text-align: left; border: 0 none; border-right: solid 1px #CCCCCC; padding: 3px;">
<ul>
<li>
Helps structuring the search for solutions to various problems and aspects in the search phase
<ul>
<li>
Problem-Solution fit (striking application for the design thinking process)
</li>
<li>
MVP Design
</li>
<li>
Commercial and marketing issues
</li>
</ul>
</li>
<li>
Very much relevant only when a lot of thoughts need to be put in the design of the product or the search for a solution to the customer problem.
</li>
</ul>
</td>
<td class="story" style="text-align: left; border: 0 none; padding: 4px;">
<ul>
<li>
When the solution is clear after the sets of interviews with key customers, or on the contrary when searching for a solution requires a lot of scientific research, Design thinking is out of scope.
<ul>
<li>
If the solution is crystal clear, a structured brainstorming process such as design thinking is not required.
</li>
<li>
If the solution requires a lot of scientific research, design thinking is not of a great help
</li>
</ul>
</li>
</ul>
</td>
</tr></table>
</div>
<br>
<a name="sec5"></a>
<h2>5. Measure Obsession</h2>
<p>
There is one fairly important topic that I haven't covered in this paper and that would require a dedicate blog post on its own.
<br>
And that is "<i>How do you know when you have reached Product-Market Fit</i>" ?
<br>
I said in the introduction that a lot of it is about feeling, when you really feel the market <i>pulling out</i> your product.
<br>
But fortunately, knowing whether your startup reached product-market fit or whether simply you are going in the right direction is a lot more than feelings. It's all about <b>metrics</b> !
</b>
<p>
Or, as W. Edwards Deming said:
</p>
<div class="centering">
<div class="centered">
<b>
"In God we trust, All others must bring data."
</b>
</div>
</div>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/8786f243-6a78-44f9-8090-bdb9707cdfb1">
<img class="centered" style="width: 650px; " alt="Measure Obsession" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/8786f243-6a78-44f9-8090-bdb9707cdfb1" />
</a><br>
<div class="centered">
(Source : <a href="http://fr.slideshare.net/sperin/les-pratiques-des-geants-du-web">Les pratiques des géants du web / Stephen Perrin - OCTO Technology</a>)
</div>
</div>
<br>
<p>
Now the question of course is which metrics makes sense in measuring whether one is going in the right direction (Product-Market Fit)?
</p>
<p>
Honestly there is no magic silver bullet and it can in fact be pretty difficult to pick up the right metric that would be most helpful to validate a certain hypothesis. However, metrics should at all cost respect the three A's.
<br>
Good metrics
</p>
<ul>
<li>are actionable,</li>
<li>can be audited and</li>
<li>are accessible</li>
</ul>
<p>
An actionable metric is one that ties specific and repeatable actions to observed results. The actionable property of picked up metrics is important since it prevents the entrepreneur from distorting the reality to his own vision.
<br>
We speak of <i>Actionable</i> vs. <i>Vanity</i> Metrics.
<br>
Meaningless metrics such as "<i>How many visitors ?</i>", "<i>How many followers ?</i>" are vanity metrics and are useless.
</p>
<p>
Ultimately, your metrics should be useful to measure progress against your own questions.
</p>
<p>
Now giving you a list of metrics and the proper way to interpret them is a topic on his own and I might write another article on this blog in a near future to define and present such metrics.
<br>
Since this article is already long enough this way, I'll just mention four metrics that I believe should be among the minimum set of metrics that any startup retain:
</p>
<ol>
<li><b>NPS</b> - Net Promoter Score</li>
<li><b>CLV (or LTV) to CAC Ratio</b> Customer LifeTime Value to Customer Acquisition Cost Ratio</li>
<li><b>Retention Ratio</b></li>
<li><b>Growth Rate</b></li>
</ol>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/b5634003-834c-40c6-aae6-ddb94161f1d2">
<img class="centered" style="width: 500px;" alt="PMF Metrics" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b5634003-834c-40c6-aae6-ddb94161f1d2" />
</a>
</div>
<br>
<a name="sec51"></a>
<h3>5.1 Net Promoter Score</h3>
<p>
The <b>Net promoter Score</b> - or NPS - is perhaps both the simplest metrics to gather and computer as well as one of the most meaningful.
<br>
It consists in understanding how great your product is by capturing how much your users are so enthusiastic about it that they would recommend it to others. On other words, it's really about how much your product is susceptible to generate a <i>Wow</i> effect.
</p>
<p>
The <b>Net Promoter Score</b> is a metric that has become a standard for measuring customer loyalty and satisfaction by many companies.
<br>
It is built on the power of one simple question: "<i>how likely is it that you would recommend your product to a friend or colleague?</i>"
<br>
it's now used by companies of all sizes in virtually every industry all over the world.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d51343f3-f4e6-4fce-9874-9c8f4d8373aa">
<img class="centered" style="width: 500px;" alt="NPS Intro" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d51343f3-f4e6-4fce-9874-9c8f4d8373aa" />
</a>
</div>
<br>
<p>
While NPS is a good leading indicator of business growth, it can also be a vanity metric if used alone without looking at the context of why your customers would recommend (or not recommend) your product.
<br>
If your company is committed to measuring NPS, here's a tip that you can use to understand the "why" behind your NPS score and potentially increase it. Follow up the standard NPS question with one additional question, "<i>What would it take for you to recommend my product to someone you know?</i>", and target the people who are not your promoters, which means they rated their likelihood to recommend your product an 8 or lower. You can then analyze those open-ended responses to identify key trends in the data.
</p>
<p>
<b>Computing the Net Promoter Score</b>
</p>
<div class="centering">
<div class="centered">
<b>Net Promoter Score</b>
=
<span style="color: green;">% Promoters</span>
-
<span style="color: red;">% Detractors</span>
</div>
</div>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/123905db-2c96-4a3c-b18c-1fc63721d4ff">
<img class="centered" style="width: 600px; " alt="NPS calculation elements" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/123905db-2c96-4a3c-b18c-1fc63721d4ff" />
</a><br>
<div class="centered">
(Source : <a href="https://www.netigate.net/articles/customer-satisfaction/nps-ultimate-guide-to-net-promoter-score">https://www.netigate.net/articles/customer-satisfaction/nps-ultimate-guide-to-net-promoter-score</a>)
</div>
</div>
<br>
<p>
<b>Understanding the Net Promoter Score</b>
</p>
<p>
There's a simple rule of thumb:
</p>
<ul>
<li>A positive value is OK/li>
<li>A value above 20% is what you want to reach/li>
<li>A value above 50% is extremely good</li>
</ul>
<a name="sec52"></a>
<h3>5.2 CLV to CAC Ratio</h3>
<p>
The <b>CLV to CAC Ratio</b> is an expression of how much money you can make built on two key figures:
</p>
<ul>
<li>
<b>CAC - Customer Acquisition Costs</b> - is the figure representing what it costs to your company in average to acquire a new customer.
<br>
It's the total cost of converting a prospect or convincing a potential customer to become an actual customer. It's the total cost devoted to your sales and marketing effort - cross department, domains and worldwide divided by the number of your new customers over the period.
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/eff44437-8801-4061-8ba7-d6e914e0b568">
<img class="centered" style="width: 800px;" alt="CAC" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/eff44437-8801-4061-8ba7-d6e914e0b568" />
</a>
</div>
<br>
</li>
<li>
<b>CLV - Customer Lifetime Value</b> (or often <b>LTV</b> in the literature) - is the figure representing how much money you make in average with one of your customer. CLV is more difficult to compute and no formula working out of the box can be expressed easily since one need to account upselling, sales model such as subscriptions vs. one time license, etc.
<br>
One can however simplify it by considering incomes only incomes from new customers
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/f5a37c09-190d-4ede-a65b-8ccf7e62320d">
<img class="centered" style="width: 700px;" alt="CLV" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/f5a37c09-190d-4ede-a65b-8ccf7e62320d" />
</a>
</div>
<br>
</li>
</ul>
<p>
The <b>CLV to CAC ratio</b> gives you an indication of how much your business is profitable.
<br>
The metric is computed by dividing LTV by CAC. It is a signal of customer profitability, and sales and marketing efficiency.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/ab4b3a13-1c74-42a6-a8dd-a6f4ce81e60d">
<img class="centered" style="width: 800px; " alt="CAC to CLV Ratio" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/ab4b3a13-1c74-42a6-a8dd-a6f4ce81e60d" />
</a><br>
<div class="centered">
(Sources :
<br>
<a href="https://www.klipfolio.com/resources/kpi-examples/saas/customer-lifetime-value-to-customer-acquisition-cost">https://www.klipfolio.com/resources/kpi-examples/saas/customer-lifetime-value-to-customer-acquisition-cost</a>
<br>
<a href="https://www.forentrepreneurs.com/startup-killer/">https://www.forentrepreneurs.com/startup-killer/ </a>)
</div>
</div>
<br>
<a name="sec53"></a>
<h3>5.3 Retention Ratio / Curve </h3>
<p>
Acquisition isn't the whole answer. Retention is even more important!
</p>
<p>
<b>Definitions:</b>
</p>
<ul>
<li><b>N-Day Retention</b>: The proportion of users who come back on the 'Nth' day after first use.</li>
<li><b>Retention Curve</b>: A line graph depicting the average percentage of active users for each day within a specified timeframe.</li>
</ul>
<p>
At a high level, retention is a measure of how many users return to your product over time.
<br>
It is the mathematical inverse of the customer churn (which can be another metric)
</p>
<p>
The point is, every improvement that you make to retention also improves all of these other things-virality, LTV, payback period. It is literally the foundation to all of growth, and that's really why retention is the king
</p>
<p>
A good way of visualizing retention rate is by plotting a retention curve:
<br>
Retention can actually indicate if you have a product-market fit problem; if you plot out your retention numbers as a percentage of active users over time and you have a flat line that reaches zero instead of a curve that stabilizes-you need to solve a product-market fit problem, not a retention problem.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/29851f5f-2609-4134-bdb1-6ad23c391868">
<img class="centered" style="width: 600px; " alt="Retention Curve" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/29851f5f-2609-4134-bdb1-6ad23c391868" />
</a><br>
<div class="centered">
(Source : <a href="https://amplitude.com/mastering-retention/why-care-about-user-retention">https://amplitude.com/mastering-retention/why-care-about-user-retention </a>)
</div>
</div>
<br>
<a name="sec54"></a>
<h3>5.4 Growth Rate </h3>
<p>
The <b>Growth Rate</b> is an expression of the speed at which your business is growing.
<br>
The Growth Rate is unfortunately a metric that is simple to understand, yet fairly difficult to compute since a lot of different elements need to be accounted.
</p>
<p>
Imagine a situation where the growth would be 10% monthly, composed by 40% new customers every months and 30% customers leaving or stopping to use the product.
<br>
In such a situation, even though the monthly growth seems interesting, the company is actually losing all its customers every 3 months!
<br>
Under such conditions, the survival of the company is almost impossible after the hype effect passes.
<br>
For this reason, the growth rate metrics needs to account the ability of the company to retain its customers, the churn rate, etc.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/2432fd5a-c280-4361-8263-2a77d5171d43">
<img class="centered" style="width: 800px; " alt="Growth Rate example" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/2432fd5a-c280-4361-8263-2a77d5171d43" />
</a><br>
<div class="centered">
(Sources : <br>
<a href="https://tribecap.co/a-quantitative-approach-to-product-market-fit/">https://tribecap.co/a-quantitative-approach-to-product-market-fit/</a>
<br>
<a href="https://www.lightercapital.com/blog/how-to-establish-product-market-fit/">https://www.lightercapital.com/blog/how-to-establish-product-market-fit/</a>)
</div>
</div>
<br>
<p>
<b>Growth accounting framework</b>
</p>
<p>
The <i>Growth Accounting Framework</i> proposed on tribecap.co at <a href="https://tribecap.co/a-quantitative-approach-to-product-market-fit/">https://tribecap.co/a-quantitative-approach-to-product-market-fit/</a> presents a fairly relevant approach to computing the Growth Rate.
</p>
<p>
Shortly put, it consists of working with the <b>Compound Monthly Growth Rate</b> over past X months as illustrated here:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/428e095f-9f33-42ba-8b35-181dd96ab687">
<img class="centered" style="width: 700px; " alt="Compund Monthly Growth Rate" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/428e095f-9f33-42ba-8b35-181dd96ab687" />
</a><br>
<div class="centered">
(Source : <a href="https://tribecap.co/a-quantitative-approach-to-product-market-fit/">https://tribecap.co/a-quantitative-approach-to-product-market-fit/</a>)
</div>
</div>
<br>
<p>
Product Market Fit is confirmed when these indicators - the cmgr3, cmgr6 and cmgr12 go up consistently.
<br>
If you intend to work to use the <i>Growth Rate</i> as key metric, you definitely should read very carefully the article above.
</p>
<a name="sec55"></a>
<h3>5.5 Further readings: pirate metrics </h3>
<p>
In this article, I have presented four example metrics, the ones that are used most of the time. But there are many more.
</p>
<p>
The reader should get familiar with the <b>Pirate Metrics</B> framework proposed by Dave McClure.
<br>
Pirate metrics is a helpful customer-lifecycle framework invented by Dave McClure from 500 startups that you can use to determine where you should focus on optimizing your marketing funnel, to make the most of your scarcest resource - your time.
<br>
Pirate metrics is essentially a way of categorizing different metrics and KPIs, and is made up of the metric “categories” Awareness, Acquisition, Activation, Revenue, Retention, Referral - or AAARRR for short (like a pirate. Pirate metrics, get it?).
</p>
<p>
The reader might want to head on over to Slideshare to read the <a href="https://www.slideshare.net/dmc500hats/startup-metrics-for-pirates-long-version">original slide deck outlining Pirate Metrics</a>.
</p>
<a name="sec6"></a>
<h2>6. Conclusion </h2>
<p>
<b>So there is a recipe for success. </b>
</p>
<p>
Entrepreneurship has a magic power, it triggers positive energies and it leaves people with an irresistible willingness to start doing things.
<br>
However, all these positive energies can very easily become negative when they are not channeled in the correct direction. And by negative we mean: having quit a day job, having spent most of our savings, having re-mortgaged the house and ultimately having trouble explaining to our life partner, family and friends why we have done all of that and we still haven't been able to succeed. That's awful.
<br>
<b>Luckily there is a process that we can follow, developed after having worked with hundreds of entrepreneurs.</b>
</p>
<p>
<b>Instead of starting to develop a product and hiring people immediately, these are the questions that we need to answer</b> in order to build and launch a product that customers need and for which there is market demand:
</p>
<ul>
<li>Which problem are we going to solve?</li>
<li>Who has the problem in our market?</li>
<li>Who are the early adopters?</li>
<li>What is the value proposition able to satisfy their needs?</li>
<li>How much are they willing to pay for it?</li>
<li>What is the minimum set of features required for launch?</li>
</ul>
<p>
The way to answer most of these questions is to <b>engage with customers from the very early stage of a new business idea</b>, get to know them profoundly, create a value proposition based on the insights captured that relies on the company's key strengths to create a competitive advantage customers care about, and test and iterate that proposition on the market until we reach <b>Product-Market Fit</b>.
</p>
<p>
Most of this can be done before investing any substantial resource into the business, and that's really the best thing about Lean Startup methodology. The most difficult thing is to resist from the instinct to jump into “build mode”. Instead, we invest some time to de-risk an idea before investing heavily in it. Everyone is in love with their idea, and the last thing we want to know is that it's not a good one. But the sooner we realize an idea is flawed - i.e. there is no market need for it - the better it is.
<br>
In terms of practical steps, this is a possible process to validate a new business idea and achieve product-market fit:
</p>
<ol>
<li>Compile a lean canvas to have clarity of a business idea (with Value Proposition Canvas)</li>
<li>Identify the riskiest assumptions for the business idea.
<br>
Hint: usually these are the ones around target segment, problems and market size.</li>
<li>(Problem Interview) Conduct a round of qualitative interviews with target customers, to understand if they have the problem, how big it is, and what they currently do to solve it.</li>
<li>Run a collaborative workshop with the entire team to refine the value proposition based on customer insights collected so far. This is how disruptive ideas are generated!</li>
<li>Prepare a cheap and quick form of a prototype.</li>
<li>(Solution Interview) Conduct another round of qualitative interviews with target customers, to understand if the solution prepared solves the problem and if they are willing to pay for it. At this stage it is mandatory to attempt to get a commitment from them.</li>
<li>Iterate the solution, and conduct new interviews if they didn't commit already.</li>
<li>(MVP) When we get enough commitment, we define a MVP (Minimum Viable Product) and start proper development (when an MVP is not possible - heavy industry, pharma, etc. - we conduct additional confirmation research and engage with more potential customers)</li>
<li>Put MVP on the market, collect feedback on MVP and evolve / adapt / pivot as required until reaching Product-Market-Fit</li>
</ol>
<p>
<b>An on-going health check</b>
</p>
<p>
<n>The Innovator's Dilemma?</b>
<br>
A new solution achieves PMF, and manages to capture the lion's share of the market.
<br>
They become the dominant player, and stay that way, until new technology appears and supersedes their solution.
<br>
By failing to stay ahead of changing technology, the incumbent company loses market share to a smaller, disruptive business, who in turn, go on to dominate the market.
<br>
Rinse and repeat.
</p>
<p>
In this narrative, the market for a product evolved over time, and the definition of Product/Market Fit changed with it. As a result of developing technology, a solution that fits the market in the here-and-now might not fit the same market in the future.
</p>
<p>
PMF is like an ongoing health check for your business, allowing you to periodically test the key assumptions that underpin your business:
</p>
<ul>
<li>Does the problem we solve still exist?</li>
<li>Is the problem important enough?</li>
<li>Is the market for our product still a 'good' market?</li>
</ul>
https://www.niceideas.ch/roller2/badtrash/entry/tdd-test-driven-development-is
TDD - Test Driven Development - is first and foremost a way to reduce the TCO of Software Development
Jerome Kehrli
2020-01-18T17:23:56-05:00
2020-01-19T05:44:53-05:00
<p>
<b>Test Driven Development</b> is a development practice from <i>e<b>X</b>treme <b>P</b>rogramming</i> which combines <i>test-first development</i> where you write a test before you write just enough production code to fulfill that test and <i>refactoring</i>.
<br>
TDD aims to improve the <b>productivity</b> and <b>quality</b> of software development. It consists in jointly building the software and its suite of non-regression tests.
</p>
<p>
The principle of TDD is as follows:
</p>
<ol>
<li>write a failing test,</li>
<li>write code for the test to work, </li>
<li>refactor the written code, </li>
</ol>
<p>
and start all over again.
</p>
<p>
Instead of writing functional code first and then the testing code afterwards (if one writes it at all), one instead <b>writes the test code before the functional code</b>.
<br>
In addition, one does so in tiny small steps - write one single test and a small bit of corresponding functional code at a time. A programmer taking a TDD approach shall refuse to write a new function until there is first a test that fails - or even doesn't compile - because that function isn't present. In fact, one shall refuse to add even a single line of code until a test exists for it. Once the test is in place one then does the work required to ensure that the test suite now passes (the new code may break several existing tests as well as the new one).
<br>
This sounds simple in principle, but when one is first learning to take a TDD approach, it does definitely require great discipline because it's easy to "<i>slip</i>" and write functional code without first writing or extending a new test.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d99df247-37ed-471e-83b4-70df826121b7">
<img class="centered" style="width: 500px; " alt="TDD Principle" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d99df247-37ed-471e-83b4-70df826121b7" />
</a>
</div>
<br>
<p>
In theory, the method requires the involvement of two different developers, one writing the tests, then other one writing the code. This avoids subjectivity issues. Kent Beck has more than a lot of examples of why and how TDD and <i>pair programming</i> fit eXtremely well together.
<br>
Now in practice, most of the time one single developer tends to write tests and the corresponding code all alone by himself which enforces the integrity of a new functionalities in a largely collaborative project.
</p>
<p>
There are multiple perspective in considering what is actually TDD.
<br>
For some it's about specification and not validation. In other words, it's one way to think through the requirements or design before one writes the functional code (implying that TDD is both an important agile requirements and an agile design technique). These considers that TDD is first and foremost a design technique.
<br>
Another view is that TDD is a programming technique streamlining the development process.
<br>
TDD is sometimes perceived as a way to improve quality of software deliverables, sometimes as a way to achieve better design and sometimes many other things.
<p>
I myself believe that TDD is <b>all of this</b> but most importantly a way to significantly <b>reduce the "Total Cost of Ownership (TCO)" of software development projects</b>, especially when long-term maintenance and evolution is to be considered.
<br>
The <i>Total Cost of Ownership (TCO)</i> of enterprise software is the sum of all direct and indirect costs incurred by that software, where the development, for in-house developped software, is obviously the biggest contributor. Understanding and forecasting the TCO and is a critical part of the <i>Return on Investment (ROI)</i> calculation.
</p>
<p>
This article is an in depth presentation of my views on TDD and an attempt to illustrate my perspective on why TDD is first and foremost a way to get control back on large Software Development Projects and <b>significantly reduce their TCO</b>.
</p>
<p>
<b>Test Driven Development</b> is a development practice from <i>e<b>X</b>treme <b>P</b>rogramming</i> which combines <i>test-first development</i> where you write a test before you write just enough production code to fulfill that test and <i>refactoring</i>.
<br>
TDD aims to improve the <b>productivity</b> and <b>quality</b> of software development. It consists in jointly building the software and its suite of non-regression tests.
</p>
<p>
The principle of TDD is as follows:
</p>
<ol>
<li>write a failing test,</li>
<li>write code for the test to work, </li>
<li>refactor the written code, </li>
</ol>
<p>
and start all over again.
</p>
<p>
Instead of writing functional code first and then the testing code afterwards (if one writes it at all), one instead <b>writes the test code before the functional code</b>.
<br>
In addition, one does so in tiny small steps - write one single test and a small bit of corresponding functional code at a time. A programmer taking a TDD approach shall refuse to write a new function until there is first a test that fails - or even doesn't compile - because that function isn't present. In fact, one shall refuse to add even a single line of code until a test exists for it. Once the test is in place one then does the work required to ensure that the test suite now passes (the new code may break several existing tests as well as the new one).
<br>
This sounds simple in principle, but when one is first learning to take a TDD approach, it does definitely require great discipline because it's easy to "<i>slip</i>" and write functional code without first writing or extending a new test.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d99df247-37ed-471e-83b4-70df826121b7">
<img class="centered" style="width: 500px; " alt="TDD Principle" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d99df247-37ed-471e-83b4-70df826121b7" />
</a>
</div>
<br>
<p>
In theory, the method requires the involvement of two different developers, one writing the tests, then other one writing the code. This avoids subjectivity issues. Kent Beck has more than a lot of examples of why and how TDD and <i>pair programming</i> fit eXtremely well together.
<br>
Now in practice, most of the time one single developer tends to write tests and the corresponding code all alone by himself which enforces the integrity of a new functionalities in a largely collaborative project.
</p>
<p>
There are multiple perspective in considering what is actually TDD.
<br>
For some it's about specification and not validation. In other words, it's one way to think through the requirements or design before one writes the functional code (implying that TDD is both an important agile requirements and an agile design technique). These considers that TDD is first and foremost a design technique.
<br>
Another view is that TDD is a programming technique streamlining the development process.
<br>
TDD is sometimes perceived as a way to improve quality of software deliverables, sometimes as a way to achieve better design and sometimes many other things.
<p>
I myself believe that TDD is <b>all of this</b> but most importantly a way to significantly <b>reduce the "Total Cost of Ownership (TCO)" of software development projects</b>, especially when long-term maintenance and evolution is to be considered.
<br>
The <i>Total Cost of Ownership (TCO)</i> of enterprise software is the sum of all direct and indirect costs incurred by that software, where the development, for in-house developped software, is obviously the biggest contributor. Understanding and forecasting the TCO and is a critical part of the <i>Return on Investment (ROI)</i> calculation.
</p>
<p>
This article is an in depth presentation of my views on TDD and an attempt to illustrate my perspective on why TDD is first and foremost a way to get control back on large Software Development Projects and <b>significantly reduce their TCO</b>.
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. So what is TDD exactly ?</a>
<ul>
<li><a href="#sec11">1.1 Principle of TDD</a></li>
<li><a href="#sec12">1.2 Advantages of TDD over tests after or even no tests</a></li>
<li><a href="#sec13">1.3 Different types of tests</a></li>
<li><a href="#sec14">1.4 Styles of TDD</a>
<ul>
<li><a href="#sec141">1.4.1 Inside-Out TDD (Bottom Up / Detroit School / Classic Approach)</a></li>
<li><a href="#sec142">1.4.2 Outside-In TDD (Top Down / London School / Mockist Approach)</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#sec2">2. Improving Design</a>
<ul>
<li><a href="#sec21">2.1 Design by testing and initial design</a></li>
<li><a href="#sec22">2.2 Emergent Design</a></li>
<li><a href="#sec23">2.3 Design principles to identify refactoring opportunities</a></li>
</ul>
</li>
<li><a href="#sec3">3. Reducing TCO </a>
<ul>
<li><a href="#sec31">3.1 Implementing Automated tests</a></li>
<li><a href="#sec32">3.2 Embracing TDD</a></li>
</ul>
</li>
<li><a href="#sec4">4. An example to illustrate the TCO reduction</a>
<ul>
<li><a href="#sec41">4.1 Illustration Example</a></li>
<li><a href="#sec42">4.2 No automated tests whatsoever (A)</a></li>
<li><a href="#sec44">4.4 Strict TDD following Bottom-Up approach (C)</a></li>
<li><a href="#sec45">4.5 How do these methods compare with each others in regards to TCO?</a></li>
</ul>
</li>
<li><a href="#sec5">5. Conclusion / Take Aways</a></li>
</ul>
<a name="sec1"></a>
<h2>1. So what is TDD exactly ?</h2>
<a name="sec11"></a>
<h3>1.1 Principle of TDD</h3>
<p>
The principle of TDD is simple: when one wants to develop a new feature, one starts by writing the test that assesses how it shall work. In the next step, the functional code is developed so that the test is validated. And nothing more!
<br>
Focusing on functionalities avoids writing code without meeting a requirement satisfied by a validated test.
</p>
<p>
The principle then consists in working in small iterative cycles consisting of:
</p>
<ul>
<li>writing the minimum possible code to pass the test;</li>
<li>enriching the test base with a new test;</li>
<li>rewriting the minimum code to pass the test;</li>
<li>and so on...</li>
</ul>
<p>
This practice mostly comes from Kent Beck, one of the signatories of the Agile Manifesto. It encourages a simple, clean and sound design of software products and makes the developer more confident in the ability of his code to do what he wants correctly, without hiding a few bugs.
</p>
<p>
Let's take a closer look at the different stages of the TDD cycle.
</p>
<ol>
<li>
<b>Write a test.</b> The first thing to do when one wants to implement a new feature is to write a test. It involves understanding the functionality that one has to develop beforehand, which is a very good thing.
</li>
<li>
<b>Execute the test(s).</b> Then one has to run the test that he just wrote. In practice, the new test is executed, along with all those already existing. This implies that they must be very quick to execute, otherwise too much time is wasted waiting for feedback. Some IDEs even push it to the extent of running the tests continuously during the development, in order to have an even faster response.
<br>
The test must fail, since no code has been written to make it pass. In general, it doesn't event compile, because the method / class doesn't even exist.
</li>
<li>
<b>Write the code.</b> Then, one writes the strictly minimum functional code required to make the test pass and nothing more. If the written code is not perfect yet, or makes the test inelegantly, it doesn't really matter for now.
</li>
<li>
<b>Execute the test(s) again.</b> The developer then re-runs all the tests ane makes sure that they run successfuly and that everything is working fine.
</li>
<li>
<b>Refactoring.</b> In this phase, one shall improve the code he has written. This helps to see if it may be simplified, written better, made generic, factorized, etc. One shall get rid of duplications, rename variables if some are not utmost meaningful, as well as methods, classes, etc., so that the code is clean, simple and clearly expresses its intentions. One shall separate responsibilities, maybe extract some design patterns, etc.
</li>
</ol>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/16bf09a8-bc9a-4b8a-8071-471f4df6875d">
<img class="centered" style="width: 600px; " alt="TDD Principle details" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/16bf09a8-bc9a-4b8a-8071-471f4df6875d" />
</a>
</div>
<br>
Following this virtuous development approaches enforces <i>single responsibilities</i>, <i>separation of concerns</i>, important <i>code coverage</i>, etc. and comes with multiple benefits described in <a href="#sec12">the next section</a>.
<p>
<b>What to do when a bug manages to make it through to production?</b>
<p>
<p>
When TDD is properly applied, it makes it simply impossible for the vaste majority of bugs to make it through to production.
<br>
However, some very tricky corner case situations may be difficult to assess with automated tests and as a consequence, even when TDD is applied by the book, it may happen that a bug passes through the cracks and is discovered late in production, sometimes after months, when the specific situation triggering the bug occurs.
</p>
<p>
This is an interesting situation and is worth discussing: what shall happen with TDD whenever a bug still manages to make it through to production?
<br>
Long story short: whenever a bug is spot in production, the resolution of the bug shall follow TDD as well:
</p>
<ul>
<li>
First, implement a new unit test or integration test that reproduces the bug. The test shall fail at first, since the bug exists.
</li>
<li>
Then do whatever it takes to have the test passing.
</li>
</ul>
<p>
This method shall simply <b>always</b> be respected whenever a bug that passes through the cracks is encountered. Eventually, these bug resolution tests will form the most important assets in the non-regression tests suite.
</p>
<p>
<b>A note about 100% functional code coverage</b>
</p>
<p>
When following the TDD methodology, one shall target 100% coverage of the <b>functional code</b> with automated tests, both in terms of <i>Lines of Code</i> and <i>Conditions Branches</i>.
<br>
This does not mean that sonar with its default configuration - or other code coverage measuring tools - shall necessarily report 100% coverage.
<br>
In practice, some boilertplate code doesn't neet to be tested. It's note considered as <i>functional code</i>.
<br>
For instance in Java, some exception catch blocks - that may be mandatory for the code to compile but that simply can't happen in practice because of the impossibility of the functional code to enter the specific branch triggering the exception - shall not be tested. Testing those would be a waste of time.
<br>
On the other hand, if the exception catch block corresponds to a specific exceptional business situation that can happen in practice, it has to have a proper unit test assessing it and it shall be covered.
</p>
<p>
Most of the time, the code coverage computation tool or the quality assessment tool - such as sonar - shall be properly configured to exclude from the coverage computation the blocks of code that would be a waste of time to test.
<br>
In practice that is rarely the case and the reported coverage never reaches 100%. This is not a problem as long as the <i>functional code</i> coverage reaches nearly 100%.
</p>
<p>
For this reason, there is an important distinction between the <i>functional code</i> and the <i>whole code</i>.
<br>
The essential point is that the functional code - the business meaningful conditions - are covered 100%. The technical boilerplate code doesn't need to be covered 100%.
</p>
<a name="sec12"></a>
<h3>1.2 Advantages of TDD over tests after or even no tests</h3>
<p>
Implementing automated tests - mostly unit tests and some integration tests - is a formidable development tool.
</p>
<p>
<b>Even without TDD, automated tests provide significant benefits:</b>
</p>
<ul>
<li>
In software development, the written functional code has to be tested continuously as it is written. Without automated tests, one needs to test the code on a live running application to assess its behaviour. In addition, when some misbehaviour is happening, one is left with the debugger to figure what is going wrong. There is no less efficient mean to figure what some code is doing than using a debugger (even though sometimes it's the only way). Implementing unit tests assessing some conditions capturing what the code under testing is doing is a much simpler way. Automated tests enable to understand the code behaviour in a simple and unitary way. Again, going through the debugger once is a while to understand why an assessment fails is still required, but to a much lesser extent.
</li>
<li>
Automated tests form a formidable non-regression tests suite. Instead of testing the live running application over and over again to search for regressions, on simply re-runs the whole automated test suite and, if it succeeds, one can be confident that no regression have been introduced by some maintenance or evolution.
<br>
This non-regression benefit also helps during the development process. Running the existing tests indicates whether the last change breaks something in the existing code base.
</li>
<li>
Bugs passing through the cracks and making it to production are significantly reduced. Unit testing and integration testing provides a much larger coverage of code both in terms of lines of codes and condition branches over manual testing. On large software products, there is simply no way manual testing can compete with automated tests, regardless of the size, the complexity and the maturity of the test plan.
</li>
<li>
Developers are getting more productive since they get confident in changing and evolving the code. When programming, the bigger the codebase gets, the harder it gets to move further or to change the code because it's easier to mess up. When one has automated tests, they become the safety net, allowing one to see what the mistake are, where they are, and how they affects the system. It helps identify errors in a really short period of time. These tests give developers very fast feedback when something breaks.
</li>
<li>
Automated tests, mostly unit tests, form a very good form of detailed specification and documentation. This not only streamlines the development process but helps re-understanding the code quickly when it has to be maintained, sometimes several months or even years after its initial development.
</li>
</ul>
<p>
<b>TDD, or <i>driving the software development with the tests</i>, brings additional benefits over <i>writing the tests after</i>:</b>
</p>
<ul>
<li>
TDD is about getting feedback. Some define TDD as being a mental model (discipline) which relies on a very short feedback loop at the code level. Getting short and frequent feedback about what the code is doing streamlines the whole development process. Quick and small feedback is much easier to understand than late and large feedback. TDD gives the ability to think more about simplicity, focusing on writing only the code strictly necessary to pass an assumption. In that sense, with TDD one challenges the wrong assumptions as early as possible and one identifies errors and problems very quickly.
</li>
<li>
The number of bugs passing through the cracks are reduced even further with TDD.
<br>
TDD enables to reach an almost exhaustive coverage of the code both in terms of <i>Lines of Code</i> and <i>Condition branches</i>.
</li>
<li>
The increase in code coverage by tests also makes it much more straightforward to proceed with large refactorings. And without refactoring ability, one's screwed to ensure best possible design since, unless one is a genius, achieving the best possible design relies mostly on his ability to refactor (improve the design)
</li>
<li>
TDD is ultimately about design. One is forced to write small class focused about one concern and small methods dedicated to one responsibility. TDD enforces SOLID design rules (see below). TDD enforces clean, simple and sound design since it simply makes it difficult not to say impossible to write convoluted code with TDD. One of the principal reasons behind this is that writing the tests first requires one to really consider what do he wants from the code a the very beginning.
</li>
<li>
All of this leads to significant reduction of the TCO (Total Cost of Ownership) of Software Development
</li>
</ul>
<a name="sec13"></a>
<h3>1.3 Different types of tests</h3>
<p>
There is a distinction between automated tests and unit tests. Not all automated tests are necessarily unit tests and while TDD is mostly about unit testing, other types of tests also make a lot of sense and bring value when embracing TDD.
</p>
<p>
There are basically three types of automated tests:
</p>
<ol>
<li>
<b>Unit tests</b>: are meant to test individual modules of an application in isolation (without any interaction with dependencies) to confirm that an individual piece of code - typically a method - is doing things right.
</li>
<li>
<b>Integration tests</b>: are meant to check if different modules are working fine when combined together as a group and that their interactions is producing the expected results.
</li>
<li>
<b>Functional tests</b>: are meant to test a slice of functionality in the system (may interact with dependencies) to confirm that the code is doing the right things.
</li>
</ol>
<p>
Functional tests are comparable to integration tests in some way, with the difference that they are intended to ensure the sound behaviour of the entire application's functionality with all the code running together and deployed in a realistic way, nearly somehow a <i>super integration test</i>.
<br>
Unit tests considers checking a single component of the system whereas functional tests are intended to assess the conformity of a whole feature with its specifications (such as the user story and its acceptance criterias).
<br>
Functional tests are intended as a safety net and form a good way to automate acceptance tests. Working with the Product Owner, the Product Managers or Business Experts, developers can formalize acceptance criterias and automate acceptance tests in the form of functional tests. This is actually the best possible way to implicate business experts in assessing the product quality.
</p>
<p>
Unit tests alone are not sufficient since it's nearly impossible to achieve near to 100% coverage of the functional code with unit tests. For instance, assessing behaviour related to interactions between modules is by design impossible with unit tests.
<br>
This is where integration tests kick in. Integrations tests are meant to cover and assess behaviour that unit tests are not targeting by assessing the well behaviour of the different units when working with each others.
</p>
<p>
These different kind of tests don't have the same level of complexity and the same target in terms of coverage. Yet they have a large overlap of scope which can be represented as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/9cab545d-60f0-48f9-a4ae-e4aa3a904c78">
<img class="centered" style="width: 700px; " alt="Different types of tests" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/9cab545d-60f0-48f9-a4ae-e4aa3a904c78" />
</a>
</div>
<br>
<p>
All these tests shall be operated in a consistent way, using maven for instance if maven is the chosen tool to build and package the software.
<br>
In terms of technology though, while unit tests and integration tests usually share a common base, this is not necessarily the case for automated tests which often rely on a completely different technical stack.
</p>
<p>
The technologies quite widely used for these different kinds of tests can be reprsented as follows :
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/817ef8b6-ba7c-400f-bb10-1d4e056abcc0">
<img class="centered" style="width: 700px; " alt="Different technologies for tests" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/817ef8b6-ba7c-400f-bb10-1d4e056abcc0" />
</a>
</div>
<br>
<p>
Again, while TDD is mostly about unit testing, and depending on the approach as well as the stage of the development, some other types of tests are fully part of the TDD scope.
<br>
Eventually, <b>all these tests together form the <i>non-regression test suite</i>.</b>
</p>
<p>
For instance while unit tests are intended to tests a method (or other forms of units) specifically (perhaps under multiple conditions), integration tests are intended to while functional tests are expected to tests end to end features and mimim user behaviour on the software.
</p>
<a name="sec14"></a>
<h3>1.4 Styles of TDD</h3>
<p>
There are actually two quite opposed approaches when applying TDD on a large software development project, the Inside-Out approach and the Outside-In approach:
</p>
<a name="sec141"></a>
<h4>1.4.1 Inside-Out TDD (Bottom Up / Detroit School / Classic Approach)</h4>
<p>
The first approach is <b>Inside-Out</b> TDD, which is sometimes called the "<i>Detroit School</i>" of TDD, or <i>bottom-up</i>, or even <i>classical</i> approach. With Inside-Out TDD, one starts by writing tests and implementation for small aspects of the system. The aim is to grow the design through a process of refactoring and generalizing the codebase as tests become increasingly specific.
</p>
<p>
Although all developers should be mindful of the bigger picture, Inside-Out TDD enables developers to focus on one thing at a time. Every component (i.e. an individual module or single class) is created one after the other and pile up until the whole application is built up.
<br>
One one hand, individual components written this way could be deemed somewhat worthless until they are connected together by higher level components and working together. Also wiring the system together at a late stage may constitute a higher risk in terms of overall design consistency. On the other hand, focusing on one component at a time helps parallelize development work efficiently within a team and refactoring is here to ensure the overall design when the components starts to pile up.
</p>
<p>
The main characteristics of the Inside-Out approach are as follows:
</p>
<ul>
<li>Emergent Design happens during the refactoring phase.</li>
<li>Very often tests are state-based tests.</li>
<li>During the refactoring phase, the unit under test may grow to multiple classes.</li>
<li>Mocks are rarely used, unless when isolating external systems.</li>
<li>No or little up-front design considerations are made except for breaking it in small features. Overall design emerges from code and is improved with refactoring..</li>
<li>Inside-Out TDD is often used in conjunction with the 4 Rules of Simple Design.</li>
</ul>
<p>
Its advantages are often considered as being the following:
</p>
<ul>
<li>It's a great way to avoid over-engineering.</li>
<li>Easier to understand and adopt due to state-based tests and little design up-front.</li>
<li>Good for exploration, when one knows what the input and desired output are but one doesn't really know how the implementation looks like at the early stage.</li>
<li>Great for cases where one can't rely on a domain expert or domain language (data transformation, algorithms, etc.)</li>
</ul>
<p>
Of course, it suffers from some commonly accepted drawbacks:
</p>
<ul>
<li>Exposing state for tests purpose only.</li>
<li>Refactoring phases are normally bigger and more complexe when compared to Outside-In approach (more on that below).</li>
<li>Unit under test becomes bigger than a class when classes emerge during the refactoring phase. This is fine when we look at that test in isolation but as classes emerge, they evolved and are extended significantly as they are being reused by other parts of the application. As these other classes evolve, they often break completely unrelated tests, since the tests use their real implementation instead of <i>mocks</i>.</li>
<li>The refactoring step (improvement of the design) is often skipped or not done properly by inexperienced practitioners, leading to a cycle that looks more like RED-GREEN-RED-GREEN-...-RED-GREEN-MASSIVE AND YET NOT EFFICIENT REFACTORING.</li>
<li>Due to its exploratory nature, some classes under test are created according to the "<i>I think I'll need this class with this interface (public methods)</i>", making them not fit well when connected to the rest of the system and requiring further more complex refactorings.</li>
<li>Can be slow and wasteful since quite often one already knows that one cannot have so many responsibilities in the class under test. The classicist advice is to wait for the refactoring phase to fix the design, only relying on concrete evidence to extract other classes. Although this is good for novices, this is often somewhat a waste of time for more experienced developers.</li>
</ul>
<p>
An illustration of the inside-out method could be as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/9fce8a45-9fc7-477a-b728-cd21dd0cc474">
<img class="centered" style="width: 850px; " alt="TDD Inside-Out process" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/9fce8a45-9fc7-477a-b728-cd21dd0cc474" />
</a>
</div>
<br>
<a name="sec142"></a>
<h4>1.4.2 Outside-In TDD (Top Down / London School / Mockist Approach)</h4>
<p>
Second approach is <b>Outside-In</b> TDD, which is sometimes called "<i>London School</i>" of TDD, or <i>Top-Down</i> or even <i>Mockist Approach</i>. Using this approach, development begins at the very top of the system's architecture and is grown downwards. The aim is to progressively implement increasing functionality of lower levels, one layer of the system at a time.
<br>
As a result, a reliance on mocks is required to simulate the functionality of lower level components.
</p>
<p>
Outside-In TDD lends itself well to having a definable route through the system from the very start, even if some (most if not all at first) parts are initially mocked.
<br>
The tests are based upon user-requested scenarios (or user stories with proper acceptance criteria well defined), and components are wired together from the beginning. This allows a fluent API to emerge and integration is performed from the very start of development.
</p>
<p>
By focusing on a complete flow through the system from the start, knowledge of how different parts of the system interact with each other is required from the very beginning. A little time to come up with a proper architecture of the system and even a rough design of every layer and functional blocks is required.
<br>
As required components emerge, they are <i>mocked</i> or <i>stubbed</i> out, which allows their detail to be deferred until later, when their time comes. This approach means that the developer needs to know how to test interactions up front, either through a mocking framework or by writing their own test doubles. The developer will then loop back, providing the real implementation of the mocked or stubbed components through new unit tests as the development moves forward.
</p>
<p>
The main characteristics of the Outside-In approach are as follows:
</p>
<ul>
<li>Different from the classicist, Outside-In TDD prescribes a direction in which we start test-driving our code: from outside (first class to receive an external request) to the inside (classes that will contain single pieces of behaviour that satisfy the feature being implemented).</li>
<li>One normally starts with an acceptance test which verifies if the feature as a whole works. The acceptance test also serves as a guide for the implementation as it progresses to lower layers.</li>
<li>With a failing acceptance test informing why the feature is not yet complete (no data returned, no message sent to a queue, no data stored in a database, etc.), one starts writing unit tests. The first class to be tested is the class handling an external request (a controller, queue listener, event handler, the entry point for a component, etc.)</li>
<li>As one already knows that the entire application won't be built in a single class, one makes some assumptions of which type of collaborators the class under test will need. One then writes tests that verify the collaboration between the class under test and its collaborators.</li>
<li>Collaborators are identified according to all the things the class under test needs to do when its public method is invoked. Collaborators names and methods should come from the domain language (nouns and verbs).</li>
<li>Once a class is tested, one picks the first collaborator (which was created with no implementation) and test-drive its behaviour, following the same approach one used for the previous class. This is why Outside-In is called this way: one starts from classes that are closer to the input of the system (outside) and move towards the inside of the application as more collaborators are identified.</li>
<li>Design starts in the red phase, while writing the tests.</li>
<li>Tests are rather about collaboration and behaviour, and only little about state.</li>
<li>Design is refined during the refactoring phase.</li>
<li>Each collaborator and its public methods are always created to serve an existing client class, making the code read very well.</li>
</ul>
<p>
It's advantages are often considered as being the following:
</p>
<ul>
<li>Since most classes are designed to serve the client calling code, the design is <i>client-centric</i>. This is not only better conceptually but also helps enforcing the domain language when naming methods.</li>
<li>Refactoring phases are much smaller, when compared to the classicist approach.</li>
<li>Promotes a better encapsulation since usually less state is exposed for test purposes only.</li>
<li>More aligned to the original ideas of <i>Object Oriented Programming</i>: tests are about objects sending messages to other objects instead of checking their state.</li>
<li>More suitable for business applications, where names and verbs can be extracted from user stories and acceptance criteria (domain model, domain language).</li>
</ul>
<p>
Of course, it also suffers from some commonly accepted drawbacks:
</p>
<ul>
<li>The architecture needs to be defined up-front and a significant design work needs to be done as well before starting to work on the first feature.</li>
<li>Much harder for novices to adopt since a higher level of design skill is necessary.</li>
<li>Developers don't get feedback from code in order to create collaborators. They need to visualize collaborators while writing the test.</li>
<li>May lead to over-engineering due to premature type (collaborators) creation.</li>
<li>Less suitable for exploratory work or behaviour that is not specified in a user story (data transformation, algorithms, etc).</li>
<li>Bad design skills may lead to an explosion of mocks.</li>
<li>Behavioural tests are harder to write than state tests.</li>
<li>Knowledge of <i>Domain Driven Design</i> and other design techniques, including 4 Rules of Simple Design (SOLID, see below), are required while writing tests.</li>
<li>Doesn't enforce simple and clean design as much as the classical approach (the emergent design is weaken)</li>
</ul>
<p>
An illustration of the Outside-In method could be as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/fd80cb2a-d018-4688-8db8-d3d6b2b75e0c">
<img class="centered" style="width: 850px; " alt="TDD Outside-In process" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/fd80cb2a-d018-4688-8db8-d3d6b2b75e0c" />
</a>
</div>
<br>
<p>
At the end of the day, every development team shall ask itself what it is more comfortable with, what makes more sense considering the product management and development organization around it, the maturity of the software architects and their ability to come up with a design first or a good breaking down in functionality.
<br>
One method is not better than the other one even though, again, Inside-Out fits better exploratory work and technical software while Outside-In works better for Business Applications and large software development projects.
<br>
But at the end of the day it's more related to the culture of the software develpment team and different cultures might prefer one or the other.
</p>
<a name="sec2"></a>
<h2>2. Improving Design</h2>
<a name="sec21"></a>
<h3>2.1 Design by testing and initial design</h3>
<p>
TDD is eventually a tool to help us design faster, first because of the necessity to write testable code, and second by easing refactoring.
<br>
The fact that one needs to write code that fullfills a unit test, magically forces to write code with simple, clear and sound design.
<br>
There really is some kind of magic in this which is interesting to explain a little.
</p>
<p>
Whenever one starts by writing some code and perhaps only then write a few unit tests (most of the time only for the methods that are easy to test this way), the testability of the code is not a key concern and it's likely that significant portions of it won't be testable using unit tests.
<br>
For the sake of solely testing and test coverage, integration tests can help a little of course, but as far as design is concerned, they don't.
</p>
<p>
Even when one tries to write code with testability in mind, ending up with code that is really only a well-thought collection of single-responsibility classes and methods each doing one and only one clearly identified functional action is really hard, not to say impossible.
</p>
<p>
With TDD, one writes the unit test first.
<br>
Writing a unit test that tests and assesses a single and unique clearly identified behaviour (or responsibility) is natural to everyone, even junior developers. When code is implemented by strictly following a logic of "<i>making this unit test pass</i>", it naturally and logically ends up in being a collection of <i>single concern</i> methods and class.
<br>
This really happens magically because it simply becomes natural to write the code this way. Implementing a unit test that would require a very convoluted code to make it pass is nearly impossible.
</p>
<p>
This is the magic part in TDD.
<br>
Interestingly, following TDD, even very junior developers end up with an initial design that is way simpler, cleaner and sound than what experienced developers could do without TDD.
</p>
<p>
Specifically, TDD is especially good at ensuring <i>Low Coupling</i> between different modules, different classes, etc. by forcing to think and design interactions and dependencies very carefully.
</p>
<p>
But that is not all in TDD that related to design, the next section is even more important.
</p>
<a name="sec22"></a>
<h3>2.2 Emergent Design</h3>
<p>
The next level of clarity and simplicity is then achieved with <i>refactoring</i>, which TDD makes easy and natural, thanks to the way it promotes nearly 100% functional code coverage both in terms of lines of code and condition branches.
<br>
TDD is a design help tool. The quality of the design one gets out of TDD depends largely on the capacity of the developer to use refactoring to <i>Design Patterns</i>, or refactoring to <i>SOLID principles</i>.
<br>
We say that the developer makes the design <i>emerge</i> using <i>continuous refactoring</i>. Applying TDD without doing constant refactoring is missing half of the job and will often lead to systems not being designed as good as they could / should be.
</p>
<p>
TDD is always associated with this important notion of "<b>emergent design</b>". In agile, one often builds the software incrementaly, feature by feature. So one can't know right from the start what <i>fine design</i> will be required, it will evolve / emerge as the development moves forward. So any time one adds a new piece of functionality, one does some refactoring to improve the design of the application. It's continuous / incremental design. That's why TDD is key in agile development processes.
</p>
<p>
Doing a lot of design upfront (BDUF = Big Design Up Front) is not incompatible with TDD though, on the contrary. There is nothing wrong with starting a piece of sofware while having the design already in mind. TDD will then enable one to put that design in place quickly. And in the case the design one thought about was wrong, TDD will allow one to refactor it nicely and safely. Again, it's just a tool, it's there to help one develop his ideas faster and design stuff safely and faster.
<br>
Now RDUF - Rough Design Up Front - probably makes more sense when embracing TDD.
<br>
When using Outside-In TDD, the RDUF is a strong requirement along with a proper pre-identification of the architecture of the software product.
</p>
<p>
In every case, one should never try to do emergent design without being willing to do some constant refactoring, they both go together and it does really require a lot of discipline
</p>
<a name="sec23"></a>
<h3>2.3 Design principles to identify refactoring opportunities</h3>
<p>
In the world of Agile projects and Agile design, several principles shall be respected to help the code keep a clean, simple and sound design.
</p>
<p>
First, the SOLID principles:
</p>
<ul>
<li>
<b>Single responsibility principle (SRP)</b> : A class should have one and only one reason to change, meaning that a class should only have one job, one single respnsibility. Note that the same should apply to a method, a package, one could consider even a whole application, with different levels of abstraction of course.
</li>
<li>
<b>Open-closed Principle (OCP)</b> : Objects or components should be open for extension, but closed for modification.
Open for extension means that we should be able to add new features or components to the application without breaking existing code.
Closed for modification means that we should not be able to introduce breaking changes to existing functionality, because that would force one to refactor a lot of existing code.
</li>
<li>
<b>Liskov Substitution Principle (LSP)</b> : Every subclass / derived class should be substitutable for its base / parent class. In other words, a subclass should override the parent class methods in a way that does not break functionality from a client's point of view. <br>
The LSP stats that whenever one is tempted to introduce some inheritance between classes but would break this principle in doing so, then one should consider composition instead of inheritance.
<br>
At the end of the day, it's about answering the question "<i>Is that X really an Y ?</i>" and if the answer is positive, then inheritance between X and Y can be used. For instance, the answer to "<i>Is a cat really an animal ?</i> is clearly yes. But the answer to the question "<i>Is than appartment really a room ?</i>" is clearly negative even though they share some common properties - such as size, volume, number of light switches, etc. - some common methods - such as Enter, Leave, etc. This is a good indication that <i>Apartment</i> should not inherit from <i>Room</i>, an appartment should own a collection of rooms. Now The Composite pattern would help factorize the common stuff.
</li>
<li>
<b>Dependency Inversion Principle (DIP)</b> : Entities must depend on abstractions not on concretions. It states that the high level module must not depend on the low level module, but they should depend on abstractions.
</li>
</ul>
<p>
Then, some common sense principles:
</p>
<ul>
<li>
<b>YAGNI - You ain't gonna need it</b> : Don't implement today something that is not strictly required today. When one thinks of some cool feature and one's tempted to implement it because it may one day be required, one shall simply never implement it today, rather implement it that one day, when the need is confirmed, not today. This way, if eventually it's not required, one doesn't loose the amount of time needed to develop it.
</li>
<li>
<b>DRY - Don't repeat yourself</b> : Use sonar or else to identify every piece of duplicated code or feature and factorize the it to eliminate the duplication. Take the opportunity to identify the duplicated code responsibility and introduce a new proper abstraction.
</li>
<li>
<b>KISS - Keep It Simple and Stupid</b> : Keep design as simple as possible. This sound easy ... but it's not. Actually coming up with the simplest possible design is much harder and requires a lot more thoughts than settling for the first idea that comes in mind.
</li>
<li>
<b>Design Patterns</b> : Introduce Design Patterns when identifying an opportunity for it.
</li>
</ul>
<a name="sec3"></a>
<h2>3. Reducing TCO </h2>
<a name="sec31"></a>
<h3>3.1 Implementing Automated tests</h3>
<p>
Automated tests reduce maintenance costs significantly since they:
</p>
<ul>
<li>form a formidable form of documentation aimed at understanding the code much faster when it needs to be maintained and/or evoluted, sometimes months or years after the initial development, </li>
<li>prevent from long sessions of manual tests to assess the behaviour of the application on edge cases, </li>
<li>prevent from deploying the code over and over again in a live running application to test it as the development is ongoing,</li>
<li>prevent from relying exclusively on the debugger to understand misbehaviour. Using the debugger is highly inefficient. Unit and Integration tests don't entirely anihilate the needs to use the debugger once in a while, but significantly reduce this need,</li>
<li>finally, unit tests (as well as integration tests) prevents a lot of bugs from passing through the cracks and making it to production, being discovered weeks or months later when a specific conditions occurs and making the business users as well as the developers loose a lot of time to figure and fix.</li>
</ul>
<p>
All these benefits that reduce the TCO comes out of the box when one reaches a good coverage of the lines of code and the condition branches with automated tests.
<br>
A good coverage of the code with unit tests means reaching 80% of lines of code and condition branches covered.
<br>
The 80/20 rule states the following : <i>"If 100 days would be required to cover 100% of the functional code in lines and condition branches coverage, then it's likely that only 20 days are required to cover 80% of them and the remaining 20% to be covered require the additional 80 days. This overwhelming investment is not worth it and one is better of limiting the invested development effort to the 20 days to cover 80%. of the code"</i>.
</p>
<a name="sec32"></a>
<h3>3.2 Embracing TDD</h3>
<p>
Implementing unit tests after the code suffers from two important drawbacks:
</p>
<ul>
<li>It doesn't enforce, simple, clean and sound design. The emergent design approach - relying on testability of the code and refactoring - is not enforced whenever one writes tests after the code.</li>
<li>Because of the previous problem, covering (nearly) 100% of the functional code with tests is overkill and one limits to the 80/20 rule. </li>
<li>Since the design is not as simple, clean and sound as it shall be, long term maintenance and evolution of the code will be more expensive that when the design is as good as it can be with</li>
</ul>
<p>
These drawbacks have direct consequences on the TCO.
</p>
<p>
<b>Hence the reason for Tests Driven Development.</b>
</p>
<p>
With TDD:
</p>
<ul>
<li>Because a unit test is written first and the code written after limited to each and every line of code required for the unit test to run successfully, the functional code coverage both in terms of lines coverage and condition branches reaches (almost) 100%.</li>
<li>The code is forced to be simple and clean because it has to fullfill the specification of a single unit test. When following TDD, it's impossible to write convoluted code since it would have been impossible to implement a unit test for this convoluted code first.</li>
<li>Thanks to the nearly 100% functional code coverage by automated tests and to the simple design, refactoring is not only always possible but also simple and straightforward.</li>
</ul>
<p>
These advantages have direct benefits on the maintenance cost since:
</p>
<ul>
<li>the need to spend long hours re-understanding the code over and over again every time it needs to be maintained is significantly reduced futher, benefiting from the simple, clean and sound design enforced by TDD,</li>
<li>the need to use a debugger to figure and understand misbehaviour the code, thanks to both the documentation that the test form and the simple and clean design, is almost eliminated,</li>
<li>the need to deploy the code in a live running application to test is reduced significantly further.</li>
</ul>
<p>
TDD is really about embracing unit and other formats of automated testing to the next level which benefits first and foremost to the TCO.
<br>
The next section will illustrate this statement with an example.
</p>
<a name="sec4"></a>
<h2>4. An example to illustrate the TCO reduction</h2>
<a name="sec41"></a>
<h3>4.1 Illustration Example</h3>
<p>
Let's take an example as an illustration of the gain in TCO when coding with TDD.
<br>
This example assume that some code must be developed, representing some 10 days of work.
</p>
<p>
This code will be developed following 3 methods:
</p>
<ul>
<li><b>A. No automated tests whatsoever</b></li>
<li><b>B. Automated tests implemented after the code</b></li>
<li><b>C. Strict TDD following Bottom-Up approach</b></li>
</ul>
<p>
For the sake of illustrating the potential TCO gain with TDD, the code will experience some maintenance after a few weeks (next maintenance) and then a major evolution (further evolution) after a few months.
</p>
<p>
In details, the development and maintenance tasks in the illustration scenario are as follows:
</p>
<ul>
<li>
<b>Initial development</b> : initial development of the feature down to production rollout
<ul>
<li><b>Development Time</b> : this is the initial development time of the first version of the feature</li>
<li><b>Debugging Time</b> : this is the debugging at development time, on the live running application to polish the behaviour</li>
<li><b>Manual Testing time</b> : this is the manual testing of the application at development time on the live running application</li>
<li><b>Pre and Post-Production Debugging Time</b> : this is the additional debugging just before and after the features enters production, most of the time required when the test coverage is not good.</li>
</ul>
</li>
<li>
<b>After a few weeks maintenance</b> : few weeks after, a small set of changes are required
<ul>
<li><b>Next maintenance re-understanding time</b> : time lost on re-understanding the code and doing the small changes</li>
<li><b>Next maintenance Debugging time</b> : time lost again in debugging the code to figure and assess its behaviour</li>
</ul>
</li>
<li>
<b>After a few months evolution</b> : few months after, an important evolution is required.
<ul>
<li><b>Further evolution re-understanding time</b> : time lost on re-understanding the code</li>
<li><b>Further evolution implementation time</b> : time required to implement the evolution</li>
<li><b>Further evolution manual testing</b> : time required to tests the evolution manually</li>
<li><b>Further evolution non-regression testing</b> : time required to re-test the feature as a whole and assessing the evolution didn't break anything.</li>
</ul>
</li>
</ul>
<p>
This is a simplification over what could be a real development and maintenance scenario of course since it leaves out all aspects that are not relevant to compare the different methods such as Documentation, Acceptance testing by business users or the product owner, etc.
</p>
<p>
Each and every approach listed above is discussed hereunder in terms of advantages and drawbacks related to costs.
</p>
<a name="sec42"></a>
<h3>4.2 No automated tests whatsoever (A)</h3>
<p>
The costs for the different steps above of the software component development and maintenance lifecycle in the case of the "<i>no tests</i>" method are as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/e57cc240-512c-4a99-9875-e138f9bd2080">
<img class="centered" style="width: 850px; " alt="No tests TCO" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/e57cc240-512c-4a99-9875-e138f9bd2080" />
</a>
</div>
<br>
<p>
We can see that the actual coding cost ("<i>Development Time</i>" and "<i>Further evolution implementation time</i>") is only a tiny part of the whole development and maintenance lifecycle.
<br>
Debugging, manual testing and struggling to understand the software again after a few months represents a significant portion of the whole TCO.
</p>
<a name="sec43"></a>
<h3>4.3 Automated tests implemented after the code (B)</h3>
<p>
The costs for the different steps above of the software component development and maintenance lifecycle in the case of the "<i>test after</i>" method are as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/6969812a-3bcc-4fdd-a7b1-d5352b2303c9">
<img class="centered" style="width: 850px; " alt="Tests after TCO" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/6969812a-3bcc-4fdd-a7b1-d5352b2303c9" />
</a>
</div>
<br>
<p>
The actual coding time becomes a lot more significative compared to the other activities (debugging, manual testing, etc.) which are significantly reduces thanks to the introduction of automated tests. Their cumulated cost remains significant though.
</p>
<a name="sec44"></a>
<h3>4.4 Strict TDD following Bottom-Up approach (C)</h3>
<p>
The costs for the different steps above of the software component development and maintenance lifecycle in the case of the "<i>test before</i>" (TDD) method are as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/710ac473-0f74-4666-8e95-612fd61dc513">
<img class="centered" style="width: 850px; " alt="TDD TCO" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/710ac473-0f74-4666-8e95-612fd61dc513" />
</a>
</div>
<br>
<p>
We can see that most of the TCO is related to coding activity, either writing tests or the functional code.
<br>
Other activities are reduced to marginal levels.
</p>
<a name="sec45"></a>
<h3>4.5 How do these methods compare with each others in regards to TCO?</h3>
<p>
We can now compare how these methods compare together in termd of TCO.
</p>
<p>
Let's first explain how these different method diverge from each others on each and every task of our scenario :
</p>
<table class="allborder">
<tr style="background: #CCCCCC; font-weight: bold;">
<td>
</td>
<td>
No automated tests
</td>
<td>
Tests implemented after code
</td>
<td>
TDD
</td>
</tr>
<tr>
<td style="font-weight: bold;"">
Development Time
</td>
<td>
Not implementing any automated tests makes the "purely" development part of the process quicker indeed. There is much less code to be written. However this illusional gain will need to be paid later on. In addition, the need to deploy the application to run live tests after every block of code implemented reduces the gain. One is better off assessing the code with unit tests instead of with a running application.
</td>
<td>
Implementing tests after writing the business logic code is better than nothing, it prevents from the need to test the code within the live running application. It comes with a little additional cost, the need to write these unit tests. The problem here is mostly that impementing tests after the code doesn't benefit from the first advantage of TDD which is ensuring a clear, simple and sound design. In addition, when writing the tests after the code, one struggles to have a good code coverage. In most-if-not-all cases the code coverage is way below what is achieved with TDD. This will be paid later on.
</td>
<td>
With TDD, tests are implemented first. This forces the code to have a clear, simple and sound design and to be utmost testable. This comes with an additional cost over tests implemented after. However, the testing, the maintainance and the future evolution of the code will tremendously benefit from the almost exhaustive coverage of the code by automated tests and the simple design.
</td>
</tr>
<tr>
<td style="font-weight: bold;"">
Debugging Time
</td>
<td>
Without unit tests, one is left with a debugger to spot the misbehaviour of the code. Debugging a running application to figure what the code is doing and where the problems are form the worst possible way to develop software.
</td>
<td>
The unit tests are preventing from loosing so much time with a debugger here. However, the poor coverage makes it so that one still needs to rely quite a lot on the debugger to figure the interactions between the different parts of the code and understand misbehaviours.
</td>
<td>
The almost exhaustive coverage by unit tests makes it almost entirely useless to debug the running application to figure mishbehaviours and side effects. Everything, including edge cases, is properly covered by unit tests and the debugger is only required very rarely to understand tricky code parts.
</td>
</tr>
<tr>
<td style="font-weight: bold;"">
Manual Testing time
</td>
<td>
Debugging is one thing, but the worst aspect without unit tests is that one needs to tests the whole behaviour of the code manually. And this is where it can get quite tricky, sometimes several minutes of manipulations are required on the UI of the running application to put in place the conditions required to test a specific edge case of the business logic.
</td>
<td>
Tests prevents manual testing to some extend only when they are not exhaustive. Most of the time when tests are implemented after the code, edge cases are not covered and as such manual testing is still required quite extensively to assess the behaviour of the code on edge cases.
</td>
<td>
This is one of the most striking advantages of TDD. The almost exhaustive coverage of condition branches and code by automated tests reduces significantly the need for manual testing. Just the tricky integration aspects remain to be tested manually.
</td>
</tr>
<tr>
<td style="font-weight: bold;"">
Pre and Post-Production Debugging Time
</td>
<td>
Without integration tests, most of the time when the application is prepared for production and/or integrated in a realistic production environment for the first time, a whole new range of corner cases appear and require a new set of very lengthy debugging session, not to mention the need to reproduce the production environment on the developer's computer first. In addition, after the production roll-out, specific conditions triggering new bugs will most certainly occur and make busines users as well as developers loose a lot of time to figure and fix.
</td>
<td>
Unfortunately, the poor coverage of condition branches as well as the lack of good integration tests reproducing different production situations most of the time prevent from benefitting from the advantages of the tests. When tests are implemented after the code, most of the time an important level of debugging under the specific conditions of the production environment is still required. Nevertheless, automated tests prevents the majority of bugs to pass through the cracks and make through post-production rollout.
</td>
<td>
Unit and integration tests can easily reproduce the whole range of specificities of the possibles conditions around the code being tested and assessed. Especially with intgeration tests, developers have the possibility to reproduce different production conditions and assess the well behaviour of the code under these specific conditions. This prevents most-if-not-all of the production debugging nightmare.
</td>
</tr>
<tr>
<td style="font-weight: bold;"">
Next maintenance re-understanding time
</td>
<td>
Unit and integration tests form a formidable form of documentation for the code. Without any of these, whenever a developer needs to apply a maintenance on some piece of code after a few months, he needs first to dedicate the required amount of time to understand the code all over again.
</td>
<td>
With unit and integration tests, the developer benefits from a surprisingly good form of documentation to understand the code very quickly and be fast in a position where he can apply the maintenance changes. However, writing the tests after the code doesn't enforce the clean, simple and sound design that TDD brings. As such, without TDD, some time is still lost due to the need to understand sometimes quite convoluted code.
</td>
<td>
With TDD, the developer doesn't only benefit from the exhaustive automated tests forming a good documentation, he also benefits from the fact that TDD enforces clean, simple and sound design and can underdstand the code produced this way much faster.
</td>
</tr>
<tr>
<td style="font-weight: bold;"">
Next maintenance Debugging Time
</td>
<td>
Without tests, the need to debug the code over and over again at every maintenance kicks in. Deploying the code in a live application is the only way to figure and understand it's misbehaviours.
</td>
<td>
The unit tests are preventing from loosing so much time with a debugger here. However, the poor coverage doesn't entirely prevent from its usage to understand some tricky part of the code or some complex interactions and side effects.
</td>
<td>
The almost exhaustive coverage by automated tests makes it almost entirely useless to debug the running application to figure mishbehaviours and side effects. This is especially important when maintaining the code or evoluting it months or years after it's been initially written. Finally, with TDD the proper reaction whenever a bug is encountered is implement a unit or integration test that reproduces the bug and assess the wrong behaviour and then fixing the failing tests. This is a much more efficient way of fixing a bug than debugging.
</td>
</tr>
<tr>
<td style="font-weight: bold;"">
Further evolution re-understanding time
</td>
<td>
Same as above. Without unit tests documenting the behaviour, one is left with reading the code itself to figure what it does.
</td>
<td>
Same as above, the developer benefits from unit tests to understand and assess the expected behaviour of the code which comes with a significant gain of time when needed to maintain or evolve the code sometimes several months after the initial development.
</td>
<td>
TDD comes with better and more tests, making the whole process even more efficient. In addition, the enforcement of a simple, clean and sound design makes the code itself much more readable which comes with a great increase of TCO gains.
</td>
</tr>
<tr>
<td style="font-weight: bold;"">
Further evolution implementation time
</td>
<td>
Once all the required time to understand the code all over again is invested, the developer can proceed with implementing the evolution. Not writing any tests is again quicker of course. But that gain is an illusion and without tests a lot of time will be lost further on the process.
</td>
<td>
Writing the tests after the development does take some additional development time, of course. But for all the reasons already presented, this time lost will be regained with huge benefits further on the process. Now again writing the tests after the code doesn't lead neither to an optimal code coverage nor to the best possible design which will have consequences later.
</td>
<td>
Here as well again the development of the tests will require some additional coding time but the other acitivites will be significantly reduced thanks to these almost exhaustive automated tests suite, not to mention the simple design which makes the whole evolution process easier.
</td>
</tr>
<tr>
<td style="font-weight: bold;"">
Further evolution manual testing
</td>
<td>
Same as above. Without unit tests, one is left with manual testing of every aspect of the feature and all corner cases on the live running application. This takes a lot of time and more importantly this has to be done over and over again everytime the feature evolves.
</td>
<td>
Again, writing unit tests after the code is better than nothing of course. But doing so, one struggles to come up with a sufficient coverage of the code with automated tests. And in this case at least some level of manual testing of the feature and egde cases is required.
</td>
<td>
Again, with an almost exhaustive coverage of code both in terms of lines of code and in terms of condition branches, the need for manual testing is significantly reduced. Only specific integration concerns and very rare border cases need to be assessed on the live running application. And most of the time when a glitch is discovered, it comes from a lack of prevision of some corner cases, almost never from a bug passing through.
</td>
</tr>
<tr>
<td style="font-weight: bold;"">
Further evolution non-regression testing
</td>
<td>
This is perhaps the biggest problem from which a software development project not leveraging on automated tests will suffer. Without a proper suite of automated tests to assess the non-regression of the software, one is left with manually testing almost the whole application each and every time some code is changed. This comes with the most amazing hidden cost and is the price to pay when not investing on automated tests at the development time.
</td>
<td>
Automated tests, even when written after the code, form a formidable protection against regressions. Most of the manual testing needed against regressions is prevented by the suite of automated tests. In case the coverage of the functional code is not 100%, which is the case most of the time when tests are written after the code, a little level of manual testing is still required.
</td>
<td>
This is another one of the most striking benefits from TDD: the fact that the test suite form a formidable non-regression testing approach. With an almost exhaustive coverage of the code with automated tests, non-regression testing boils down to simply running these tests.
</td>
</tr>
</table>
<p>
As a consequence, the TCO in terms of required man/days diverge quite a lot between the three different approaches:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/c625635e-a891-4769-9a58-73b67523dd38">
<img class="centered" style="width: 850px; " alt="Approaches TCO Comparison" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/c625635e-a891-4769-9a58-73b67523dd38" />
</a>
</div>
<br>
<p>
The scenario above gives us the following figures in terms of man/days required for every approach:
</p>
<ul>
<li>37 M/D for the approach without any test</li>
<li>30 M/D for the approach with tests written after the code</li>
<li>26 M/D for the TDD approach</li>
</ul>
<p>
Which represents the following difference
</p>
<ul>
<li>20% TCO gain when working with a consistent suite of automated tests over the no tests approach</li>
<li>10% addition gains with TDD over the <i>tests written after</i> approach.</li>
</ul>
<p>
<b>This represents a 30% reduction of TCO when embracing TDD over an approach without a comprehensive suite of automated tests.</b>
<br>
On software development projects requiring millions of dollars of investment, this represents a more than significant gain.
</p>
<a name="sec5"></a>
<h2>5. Conclusion / Take Aways</h2>
<p>
I believe that the most important take away when reading an article about TDD is that TDD is eventually the only way to recover some level of mastery on software development processes.
<br>
Please allow me to develop this statement.
</p>
<p>
Software Engineering forms a very specific and peculiar domain in the engineering business. Let's compare its situation with <i>Civil Engineering</i> for instance. We are building bridges for litterally several thousands of years. Today, even a 10 years old child can have a basic understanding of how a bridge shall be built, some pilars should be anchored in the ground and a deck shall be layed on top of these, etc.
<br>
Everyone is able to figure what would be the trivial steps when building a bridge.
</p>
<p>
A software product is something completely different. Due to its very abstract nature, building large software products is very hazardous. In contrary to other engineering domains, it's nearly impossible to estimate the required effort to develop a large software component and the reality shifts simply always from the plan.
<br>
And this is not even accounting debugging, maintenance and evolutions.
</p>
<p>
Without TDD, whenever a development team believe it's quite close to completing the project is most of the time also the very moment it just starts figuring the tons of bugs that will need to be solved and the tremedous amount of work that actually still remains to be done.
</p>
<p>
TDD is a way to get the control back.
</p>
<p>
TDD enables to reduce significantly maintenance and evolution costs and at the same time master the software development process. With TDD, the implemented code is most of the time almost production ready from a functional perspective and pre-production debugging sessions are largely reduced.
<br>
But more importantly TDD enables to smoothing the future evolutions of the software product by significantly improving its design and providing an exhaustive set of non-regression tests out of the box.
</p>
<p>
At the end of the day, this significant reduction of the TCO is the most important aspect of TDD. The pressure to deliver should never dictate whether one uses TDD or not. The time gained at development time when skipping automated tests is an illusion. Eventually much more time will be lost without tests. And then again TDD is not only about tests...
</p>
https://www.niceideas.ch/roller2/badtrash/entry/ai-opportunities-and-challenges-for
AI - opportunities and challenges for Swiss banks
Jerome Kehrli
2019-12-06T11:00:58-05:00
2019-12-06T11:30:29-05:00
<p>
Yesterday we were amazed by the first smartphones. Today they have almost become an extension of ourselves.<br>
People are now used to be connected all the time, with highly efficient devices on highly responsive services, everywhere and for every possible need.
</p>
<p>
This is a new industrial revolution - the digitization . and it forces corporations to transform their business models to meet customers on these new channels.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/cbe06ed8-664a-49e5-a91e-f829ab3e011d">
<img class="centered" style="width: 650px; " alt="AI and banking" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/cbe06ed8-664a-49e5-a91e-f829ab3e011d" />
</a>
</div>
<br>
<p>
Banks worldwide are on the first line in this regards and for many years now they have well understood the urgency in proclaiming digitization as a key objective.<br>
From a user perspective, the digitization confers enormous benefits in the form of ease, speed and multiple means of access and a paradigm shift in engagement. Since banking as a whole benefits from going digital, it is only a matter of time before operations turn completely digital.
</p>
<p>
The journey to digital transformation requires both strategy investments as well as tactical adjustments in orienting operations for the digital road ahead.<br>
Fortunately, if technology can be perceived as a challenge, it is also a formidable opportunity.<br>
And in this regards, Artificial Intelligence is a category on its own.
</p>
<p>
Yesterday we were amazed by the first smartphones. Today they have almost become an extension of ourselves.<br>
People are now used to be connected all the time, with highly efficient devices on highly responsive services, everywhere and for every possible need.
</p>
<p>
This is a new industrial revolution - the digitization . and it forces corporations to transform their business models to meet customers on these new channels.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/cbe06ed8-664a-49e5-a91e-f829ab3e011d">
<img class="centered" style="width: 650px; " alt="AI and banking" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/cbe06ed8-664a-49e5-a91e-f829ab3e011d" />
</a>
</div>
<br>
<p>
Banks worldwide are on the first line in this regards and for many years now they have well understood the urgency in proclaiming digitization as a key objective.<br>
From a user perspective, the digitization confers enormous benefits in the form of ease, speed and multiple means of access and a paradigm shift in engagement. Since banking as a whole benefits from going digital, it is only a matter of time before operations turn completely digital.
</p>
<p>
The journey to digital transformation requires both strategy investments as well as tactical adjustments in orienting operations for the digital road ahead.<br>
Fortunately, if technology can be perceived as a challenge, it is also a formidable opportunity.<br>
And in this regards, Artificial Intelligence is a category on its own.
</p>
<h3>Artificial Intelligence and it's potential in the banking business.</h3>
<p>
AI provides a unprecedented opportunity to make banks smarter. Deploying AI solutions in banking leads to better customer intelligence and better customer experience. <br>
Both are key to increase benefits and reduce operational costs.
</p>
<p>
There are multiple applications for AI solutions in the banking business around three major axis:
</p>
<ol>
<li> Customer Experience revolution when putting technology in direct contact with the customer</li>
<li> AI analytics improving operational efficiency in various domains (e.g. investment research, credit scoring, etc.) or providing personalized advisory to customers</li>
<li> Risk mitigation with better fraud detection, more efficient AML, more efficient compliance controls, etc.</li>
</ol>
<p>
One of the most impressive opportunity on the customer experience revolution axis is formed by chat-bots and voice assisted banking. The need for physical presence is definitely fading and technology empowers customers to use banking services using voice commands and touch screens.
</p>
<p>
In regards to improving operational efficiency within the bank, the most promising evolution comes form the conjunction of Real-time Big Data processing with Machine Learning. The technology can provide personalized, value-added products to customers as it learns about spending habits or investment profiles, but it can also automate most analytics duties within the bank.<br>
Data-driven AI applications are intended in the future to cover the whole range of financial decisions: advisory, calculations, scoring and forecasts, for the bank as well as for the customers. For instance if approving a commercial real estate loan was traditionally a several days process within a bank, using AI will reduce it to a few dozen of minutes.
</p>
<p>
Last but not least, embracing AI has been at the root of significant improvements in Fraud detection and AML. Companies like MasterCard and Visa have been using AI to detect fraudulent transaction patterns for several years now. At NetGuardians we deploy AI solutions for digital banking fraud prevention and internal fraud detection for several years as well.
</p>
<p>
AI solutions are key to react proactively and inform the customer before the funds leave the bank. AI enables to implement Transaction analytics but also behaviour analysis aimed at catching more complex fraud patterns.
</p>
<h3>What about Swiss banks ?</h3>
<p>
Interestingly, while most would describe Switzerland as less innovative than other countries such as UK for instance, especially in the retail banking space, the reality is a quite differentiated picture.
</p>
<p>
The digital solutions of the major Swiss banks are among the best in the world and Machine Learning algorithms are used down the line on the three axis described above. Due to their conservative nature, the major Swiss banks have a tendency to rather follow the market best practices and state of the art in terms of customer experience evolution but on the backend side - the technology running under the hood - they are rather very well in advance.
</p>
<p>
The situation of smaller Swiss retail banking institutions is somewhat similar. Their strong footprint in their regions, their attractive conditions as well as their good digital banking solutions in general relieves the pressure. The biggest difference with major banks is that smaller institutions don't necessarily have the ability to research AI or Machine Learning technology on their own so they rely on third party providers such as NetGuardians for fraud prevention or other fintechs for other use cases.<br>
In this sense, keeping a close proximity with engineering schools and universities is a tremendous opportunity to stay on the top of the game and get in touch with the numerous fintechs flourishing in Switzerland. One could only advise them to be less timorous when it comes to supporting these startups since investing in them is eventually their only way to support the development of the technology that will be available to them in the future.
</p>
<p>
Private banking institutions on the other hand are more vulnerable today, at least for the smaller ones. Their margin is reducing and their wealth management business is increasingly cannibalized by other actors such as External Asset Managers, Fintechs or bigger institutions - even retail institutions where AI has been instrumental into making them reach a level of proximity in advisory that was so far the exclusive privilege of private banks.<br>
Private banking institutions need to understand the urgency in revolutionizing the private banking customer experience and recover the lead in this regards from the other actors. Here as well the opportunities for Artificial Intelligence applications are striking: operational efficiency, advisory, etc.
https://www.niceideas.ch/roller2/badtrash/entry/dissecting-swift-message-types-involved
Dissecting SWIFT Message Types involved in payments
Jerome Kehrli
2019-04-05T05:40:51-04:00
2019-04-06T15:24:06-04:00
<p>
In my current company, we implement a state-of-the art banking Fraud Detection system using an Artificial Intelligence running on a Big Data Analytics platform. When working on preventing banking fraud, looking at SWIFT messages is extremely interesting. 98% of all cross-border (international) funds transfers are indeed transferred using the SWIFT Network.
<br>
The SWIFT network enables financial institutions worldwide to send and receive information about financial transactions in a secure, standardized and reliable environment. Many different kind of information can be transferred between banking institution using the SWIFT network.
</p>
<p>
In this article, I intend to dissect the key SWIFT Messages Types involved in funds transfers, present examples of such messages along with use cases and detail the most essential attributes of these payments.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/b5614a1d-2b9e-4be6-b582-470d92744b3b">
<img class="centered" style="width: 850px; " alt="SWIFT messaging scheme" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b5614a1d-2b9e-4be6-b582-470d92744b3b" />
</a>
</div>
<br>
<p>
These key messages are as follows:
</p>
<ul>
<li>MT 101 - Request for Transfer</li>
<li>MT 103 - Single Customer Credit Transfer</li>
<li>MT 202 - General Financial Institution Transfer</li>
<li>MT 202 COV - General Financial Institution Transfer for Cover payments</li>
</ul>
<p>
This article presents each and every of these messages, discuss their typical use cases and details key SWIFT fields involved.
</p>
<!-- Dissecting SWIFT Message Types involved in payments -->
<p>
In my current company, we implement a state-of-the art banking Fraud Detection system using an Artificial Intelligence running on a Big Data Analytics platform. When working on preventing banking fraud, looking at SWIFT messages is extremely interesting. 98% of all cross-border (international) funds transfers are indeed transferred using the SWIFT Network.
<br>
The SWIFT network enables financial institutions worldwide to send and receive information about financial transactions in a secure, standardized and reliable environment. Many different kind of information can be transferred between banking institution using the SWIFT network.
</p>
<p>
In this article, I intend to dissect the key SWIFT Messages Types involved in funds transfers, present examples of such messages along with use cases and detail the most essential attributes of these payments.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/b5614a1d-2b9e-4be6-b582-470d92744b3b">
<img class="centered" style="width: 850px; " alt="SWIFT messaging scheme" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b5614a1d-2b9e-4be6-b582-470d92744b3b" />
</a>
</div>
<br>
<p>
These key messages are as follows:
</p>
<ul>
<li>MT 101 - Request for Transfer</li>
<li>MT 103 - Single Customer Credit Transfer</li>
<li>MT 202 - General Financial Institution Transfer</li>
<li>MT 202 COV - General Financial Institution Transfer for Cover payments</li>
</ul>
<p>
This article presents each and every of these messages, discuss their typical use cases and details key SWIFT fields involved.
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. Introduction to SWIFT</a>
<ul>
<li><a href="#sec11">1.1 Key SWIFT aspects</a></li>
<li><a href="#sec12">1.2 Correspondent banking</a>
<ul>
<li><a href="#sec121">1.2.1 Correspondent Bank</a></li>
<li><a href="#sec122">1.2.2 Transferring Money Using a Correspondent Bank</a></li>
<li><a href="#sec123">1.2.3 VOSTRO and NOSTRO accounts</a></li>
</ul>
</li>
<li><a href="#sec13">1.3 Key SWIFT Message Types</a>
<ul>
<li><a href="#sec131">1.3.1 Detailed presentation of key SWIFT messages</a></li>
<li><a href="#sec132">1.3.2 Typical situation and use cases in banking institutions</a></li>
</ul>
</li>
<li><a href="#sec14">1.4 Serial and cover payments</a>
<ul>
<li><a href="#sec141">1.4.1 Cover method details</a></li>
<li><a href="#sec142">1.4.2 Serial method details</a></li>
</ul>
</li>
<li><a href="#sec15">1.5 SWIFT Message Structure</a></li>
<li><a href="#sec16">1.6 SWIFT BIC Code</a>
<ul>
<li><a href="#sec161">1.6.1 Structure of the SWIFT BIC Code</a></li>
</ul>
</li>
<li><a href="#sec17">1.7 Other specific details</a></li>
</ul>
</li>
<li><a href="#sec2">2. Dissecting key SWIFT Mesages involved in payments (Funds Transfers)</a>
<ul>
<li><a href="#sec21">2.1 SWIFT MT101 Detailed Analysis</a>
<ul>
<li><a href="#sec211">2.1.1 MT101 Introductory examples</a>
<ul>
<li><a href="#sec2111">2.1.1.1 MT101 Example 1: Simplest case</a></li>
<li><a href="#sec2112">2.1.1.2 MT101 Example 2: beneficiary with another institution</a></li>
<li><a href="#sec2113">2.1.1.3 MT101 Example 3: Multiple payments in MT101</a></li>
<li><a href="#sec2114">2.1.1.4 MT101 Example 4: Payment from a subsidiary account</a></li>
<li><a href="#sec2115">2.1.1.5 MT101 Example 5: Fund repatriation</a></li>
</ul>
</li>
<li><a href="#sec212">2.1.2 MT101 Parsing and Data Mapping </a></li>
<li><a href="#sec213">2.1.3 Additional notes on MT101</a></li>
</ul>
</li>
<li><a href="#sec22">2.2 SWIFT MT103 Detailed Analysis</a>
<ul>
<li><a href="#sec221">2.2.1 MT103 Introductory examples</a>
<ul>
<li><a href="#sec2211">2.2.1.1 MT103 Example 1: Simplest case</a></li>
<li><a href="#sec2212">2.2.1.2 MT103 Example 2: more realistic example</a></li>
<li><a href="#sec2213">2.2.1.3 MT103 Example3: forwarded serial message</a></li>
<li><a href="#sec2214">2.2.1.4 MT103 Example 4: Announce message (cover method)</a></li>
</ul>
</li>
<li><a href="#sec222">2.2.2 MT103 Parsing and Data Mapping </a></li>
<li><a href="#sec223">2.2.3 Additional notes on MT103</a></li>
</ul>
</li>
<li><a href="#sec23">2.3 SWIFT MT202 Detailed Analysis</a>
<ul>
<li><a href="#sec231">2.3.1 SWIFT MT202 Introductory Exampkes</a>
<ul>
<li><a href="#sec2311">2.3.1.1 MT202 Example 1: simplest case</a></li>
<li><a href="#sec2312">2.3.1.2 MT202 Example 2: other bank case</a></li>
<li><a href="#sec2313">2.3.1.3 MT202 Example 3: routed message</a></li>
</ul>
</li>
<li><a href="#sec232">2.3.2 MT202 Parsing and Data Mapping </a></li>
<li><a href="#sec233">2.3.3 Additional notes on MT202</a></li>
</ul>
</li>
<li><a href="#sec24">2.4 SWIFT MT202 COV Detailed Analysis</a>
<ul>
<li><a href="#sec241">2.4.1 MT202 COV Introductory examples</a>
<ul>
<li><a href="#sec2411">2.4.1.1 MT202 COV Example: Cover payment</a></li>
</ul>
</li>
<li><a href="#sec242">2.4.2 MT202 COV Parsing and Data Mapping </a></li>
<li><a href="#sec243">2.4.3 Additional notes on MT202 COV</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#sec3">3. Conclusion</a></li>
</ul>
<a name="sec1"></a>
<h2>1. Introduction to SWIFT</h2>
<p>
SWIFT - Society for Worldwide Inter-bank Financial Telecommunication - is a Belgian company, located in Belgium, and is a trusted and closed network used for communication between banks around the world. It is overseen by a committee composed of the US Federal Reserve, the Bank of England, the European Central Bank, the Bank of Japan and other major banks.
<br>
SWIFT is used by around 11 thousands institutions in more than 200 countries and supports around 25 million communications a day, most of them being money transfer transactions, the rest are various other types of messages.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a887fd53-655c-48a9-9397-6899371af97d">
<img class="centered" style="width: 150px; " alt="SWIFT logo" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a887fd53-655c-48a9-9397-6899371af97d" />
</a>
</div>
<br>
<p>
The majority of international inter-bank messages use the SWIFT network.
<br>
SWIFT does not facilitate funds transfer: rather, it sends payment orders, which must be settled by correspondent accounts that the institutions have with each other. For two financial institutions to exchange banking transactions, they must have a banking relationship beforehand.
</p>
<a name="sec11"></a>
<h3>1.1 Key SWIFT aspects</h3>
<p>
Internationally standardized messaging means that every transaction between every financial institution is recorded in exactly the same way, providing all the details in a clear and transparent manner.
<br>
Every financial institution has its own unique code that provides information on the name and location of the bank. Each transaction contains a unique reference number, bank operation code and details of charges incurred during the transaction.
</p>
<p>
Because SWIFT uses internationally standardized messages, it is a transparent way for institutions to communicate between each other and securely relay the details of any transaction. There are a number of known benefits to using SWIFT:
</p>
<ul>
<li>
Transparency. SWIFT payments clearly detail the amounts involved in the transaction, the route it takes between banks, the details of all charges and the nature of the payment (along with many other details). This information allows all parties involved to track the transaction and to understand the costs and time period involved.
</li>
<li>
Traceability. Because SWIFT details the route of the transaction between banks and the amount of money involved, it provides clear and recognized proof of payment.
</li>
<li>
Consistency. Due to the consistency of how messages are structured, payment information is easy to decipher regardless of country or language barriers.
</li>
</ul>
<a name="sec12"></a>
<h3>1.2 Correspondent banking</h3>
<p>
Correspondent banking is an important aspect of international banking and a key concept underneath the SWIFT network.
</p>
<a name="sec121"></a>
<h4>1.2.1 Correspondent Bank</h4>
<p>
A correspondent bank is a financial institution that provides services on behalf of another financial institution. It can facilitate wire transfers, conduct business transactions, accept deposits and gather documents on behalf of another financial institution. Correspondent banks are most likely to be used by domestic banks to service transactions that either originate or are completed in foreign countries, acting as a domestic bank's agent abroad.
</p>
<p>
Generally speaking, the reasons domestic banks employ correspondent banks include:
</p>
<ul>
<li>limited access to foreign financial markets and the inability to service client accounts without opening branches abroad,</li>
<li>act as intermediaries between banks in different countries or as an agent to process local transactions for customers abroad,</li>
<li>accept deposits, process documentation and serve as transfer agents for funds.</li>
</ul>
<p>
The ability to execute these services relieves domestic banks of the need to establish a physical presence in foreign countries.
</p>
<a name="sec122"></a>
<h4>1.2.2 Transferring Money Using a Correspondent Bank</h4>
<p>
International wire transfers often occur between banks that do not have an established financial relationship. When agreements are not in place between the bank sending a wire and the one receiving it, a correspondent bank must act as an intermediary. For example, a bank in Geneva that has received instructions to wire funds to a bank in Japan cannot wire funds directly without a working relationship with the receiving bank.
<br>
Most if not all international wire transfers are executed through SWIFT. Knowing there is not a working relationship with the destination bank, the originating bank can search the SWIFT network for a correspondent bank that has arrangements with both banks.
</li>
<p>
Interestingly, when a bank wants to send some funds to another bank on the other side of the world, it happens often that the sending bank has no banking relationships with any bank having itself a relationship with the target bank. In this case, the order needs to be transferred through several, sometimes many, distinct banking institutions through the SWIFT network.
<br>
These intermediate banks are called routing banks.
</p>
<a name="sec123"></a>
<h4>1.2.3 VOSTRO and NOSTRO accounts</h4>
<p>
General usage of NOSTRO/VOSTRO (they refer to the same account but different bank perspective):
<br>
A NOSTRO account is a reference used by Bank A to refer to "our" account held by Bank B. NOSTRO, is a shorthand way of talking about "our money that is on deposit at your bank."
<br>
VOSTRO is the term used by Bank B, where bank A's money is on deposit. VOSTRO is a reference to "yours" and refers to "your money that is on deposit at our bank."
</p>
<a name="sec13"></a>
<h3>1.3 Key SWIFT Message Types</h3>
<p>
When it comes to fund transfers, only a subset of the SWIFT messages are relevant:
</p>
<table class="allborder">
<tr style="background: #CCCCCC; font-weight: bold;">
<td>Message Identification and Name</td>
<td>Space</td>
<td>Reference document</td>
</tr>
<tr>
<td>MT 101 Request for Transfer</td>
<td>Customer-to-Bank and Interbank</td>
<td rowspan="5">
<a href="https://www2.swift.com/uhbonline/books/public/en_uk/us2m_20180720/index.htm">Standards MT November 2018</a>
</td>
</tr>
<tr>
<td>
MT 103 Single Customer Credit Transfer
</td>
<td>
Interbank
</td>
</tr>
<tr>
<td>
MT 202 General Financial Institution Transfer
</td>
<td>
Interbank
</td>
</tr>
<tr>
<td>
MT 202 COV General Financial Institution Transfer
</td>
<td>
Interbank
</td>
</tr>
<tr>
<td>
MT 9XX - Confirmations and statements
</td>
<td>
Customer-to-Bank and Interbank
</td>
</tr>
</table>
<a name="sec131"></a>
<h4>1.3.1 Detailed presentation of key SWIFT messages</h4>
<p>
These SWIFT messages are described as follows:
</p>
<ul>
<li>
The <b>SWIFT MT101</b> message is a request for transfer, enabling the electronic transfer of funds from one account to another. Funds are transferred from ordering customers account to a receiving financial institution or account servicing financial institution. For us right now, the important thing to note is that the message format that enables this transfer is the SWIFT MT101 format.
</li>
<li>
The <b>MT103</b> is a SWIFT message format used for making payments. MT103 SWIFT payments are known as international wire transfers, telegraphic transfers, standard EU payments (SEPA payments), LVTS in Canada, etc.
</li>
<li>
Swift <b>MT202</b> Requests the movement of funds between financial institutions, except if the transfer is related to an underlying customer credit transfer that was sent with the cover method, in which case the MT 202 COV must be used.
</li>
<li>
<b>MT202 COV</b> is a SWIFT message format for financial institution (FI) funds transfer between financial institutions. MT202's are used primarily for two purposes, bank-to-bank payments (i.e. interest payments and settlement of FX trades) and Cover Payments.
<br>
MT202 COV was implemented in 2009 to create traceability of the origination of funds (institution and account) through to the destination of funds (institution and account.) This was in response to anti-money laundering and associated banking requirements.
<br>
Prior to MT202 COV, The message format, MT202, did not include origination/destination financial institution information. Particularly for Cover Payments, where a combination of MT103 and MT202 are used to direct funds transfers to a beneficiary account, the intermediate banks in the MT202 had no ability to understand and perform risk analysis/AML/compliance checks on the funds transfer based on the original and destination of the funds. Thus, intermediate banks could be unwittingly involved in illegal transactions under new regulations.
</li>
</ul>
<a name="sec132"></a>
<h4>1.3.2 Typical situation and use cases in banking institutions</h4>
<p>
In regards to this set of messages, the various situations that a banking institution is confronted to can be represented as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/b5614a1d-2b9e-4be6-b582-470d92744b3b">
<img class="centered" style="width: 850px; " alt="SWIFT messaging scheme" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b5614a1d-2b9e-4be6-b582-470d92744b3b" />
</a>
</div>
<br>
<p>
With the following details:
</p>
<ul>
<li>
A customer can request a fund transfer on his behalf using either a bank native channel (email, EBanking, branch, etc.) or by sending an MT101 to the bank. Big corporate customers can indeed be connected themselves to the bank network and send requests to the bank using the MT101 Message Type which is intended for this purpose.
</li>
<li>
Most of the time, the reception of an MT101 by the bank will make it issue an MT103 to proceed further with the Fund Transfers. But if the account to be debited is not by the bank, it can as well transfer the MT101 further to the other institution owning the account to be debited.
</li>
<li>
MT103 are sent following a customer request to transfer the funds further (serial method) or announce to the receiving institution that the funds will be coming following an MT202.COV (cover method).
</li>
<li>
An MT202 is sent on behalf of bank itself and not following a customer request, in contrary to MT103.
</li>
<li>
If the bank is a routing bank in the middle of a routing chain, MT103 and MT202 are sent further. Actually new messages are sent, the sent MT103 or MT202 can be different than the received one.
</li>
</ul>
<a name="sec14"></a>
<h3>1.4 Serial and cover payments</h3>
<p>
SWIFT Serial and Cover payments originate from the two methods that are used to settle transactions in the SWIFT network and specifically in the field of correspondent banking: Serial method and Cover method.
<br>
When sender and receiver are located in different currency zones, they send or receive funds through their correspondents. And in this case, either of both methods can be used.
</p>
<p>
With the cover method, two different messages are initiated by the sender to settle the funds, an MT103 and an MT202.COV. The MT101 message is used to inform the creditor bank that funds are coming, it is an announcement. The MT202.COV, called cover message, moves the funds between correspondent accounts.
<br>
With the serial method, one single message is initiated by the sender to settle the funds, an MT103. That MT103 is in this case not an announce, but the fund transfer itself that moves for one party to the next in the payment chain until it reaches the beneficiary bank.
</p>
<a name="sec141"></a>
<h4>1.4.1 Cover method details</h4>
<p>
The MT103 announcement is sent to the beneficiary bank to announce that funds are coming for a specific beneficiary. It does not carry the funds but rather just informs the bank of the beneficiary that funds are coming for which beneficiary customer and which correspondent (of the beneficiary bank) will receive the funds.
</p>
<p>
The cover payment (MT202 COV) is sent by the sender to its correspondent. This is the message that really moves the funds. The MT202 COV enables the sender to ask its correspondent to debit its account (of the sender, with the correspondent) and credit the beneficiary bank's account with its correspondent.
</p>
<p>
Most of the time, the announcement is created and sent before the cover but that is not a requirement so receiving the cover payment before the announce is a situation that needs to be taken into account by the receiver.
</p>
<a name="sec142"></a>
<h4>1.4.2 Serial method details</h4>
<p>
Using an MT103, the funds moves from one party to another until it reaches the final beneficiary. Following a customer request, the sender sends an MT103 serial to its correspondent which transfers the funds to the intermediary institution. The intermediary institution is the correspondent of the beneficiary most of the time. The intermediary institution on its turn credits the account of the creditor bank and eventually the beneficiary account.
</p>
<p>
In the SWIFT MT103 Serial Message, the fields 56a and 57a are used while the fields 53a and 54a are used in the MT103 Announcement Message (cover method).
<br>
Intermediary institution and receiver's correspondent are usually two names to designate the same thing. The account with institution is the bank that holds the beneficiary account, so just another name for Creditor bank.
</p>
<a name="sec15"></a>
<h3>1.5 SWIFT Message Structure</h3>
<p>
This section presents how an ISO 15022 message looks like and decomposes it to its particular parts. The description of the structure is intended as a guidance to build a SWIFT message parser.
</p>
<p>
A message consists of blocks enclosed in curly braces. The first colon separates the block name and content. The block content can consist of sub-blocks.
</p>
<table class="allborder">
<tr style="background: #CCCCCC; font-weight: bold;">
<td>
Block Ident.
</td>
<td>
Block Name
</td>
<td>
Mandatory or Optional
</td>
<td>
Description
</td>
<td>
Comments
</td>
</tr>
<tr>
<td>
1
</td>
<td>
Basic Header
</td>
<td class="center">
M
</td>
<td>
The only mandatory block is the basic header. The basic header contains the general information that identifies the message, and some additional control information. The FIN interface automatically builds the basic header.
</td>
<td>
Common to all SWIFT messages. It contains five fields that are all mandatory.
</td>
</tr>
<tr>
<td>
2
</td>
<td>
Application Header
</td>
<td class="center">
O
</td>
<td>
The application header contains information that is specific to the application. The application header is required for messages that users, or the system and users, exchange. Exceptions are session establishment and session closure.
</td>
<td>
Common to all SWIFT messages. There are two variations: One block for input messages which may contain up to six fields and one for output messages which may have up to seven fields.
</td>
</tr>
<tr>
<td>
3
</td>
<td>
User Header
</td>
<td class="center">
O
</td>
<td>
The user header is an optional header.
</td>
<td>
Common to all SWIFT messages. All fields of the user header (except the tag 103 for FINCopy Service) are optional. Fields are populated in specific situations.
</td>
</tr>
<tr>
<td>
4
</td>
<td>
Text
</td>
<td class="center">
O
</td>
<td>
The text is the actual data to transfer.
</td>
<td>
This is the block found in the Message Reference Guide.
</td>
</tr>
<tr>
<td>
5
</td>
<td>
Trailers
</td>
<td class="center">
O
</td>
<td>
The trailer either indicates special circumstances that relate to message handling or contains security information.
</td>
<td>
Common to all SWIFT messages. Like the block 3, this block consists of only non-mandatory fields except the checksum.
</td>
</tr>
</table>
<p>
The various header blocks contain different kinds of information but not all of them are interesting for our use cases in my current company. The next chapters will present which field we extract and for what usage.
</p>
<a name="sec16"></a>
<h3>1.6 SWIFT BIC Code</h3>
<p>
The SWIFT BIC code is much more than an entity identifier. It is used to route the financial messages from the issuing institution to the receiving institutions. The SWIFT BIC Code therefore plays a crucial role in payment messaging. Without it, a message cannot be transported to the receiving entity over SWIFT Net.
</p>
<p>
The BIC code contains the identity and the location of the participants that are used to find out and reach the message destination.
</p>
<p>
The SWIFT BIC code is composed of exactly 8 or 11 alphanumeric characters structured as followed:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/e06c3953-d548-4614-9f78-01f4d54048ba">
<img class="centered" style="width: 350px; " alt="BIC Code Structure" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/e06c3953-d548-4614-9f78-01f4d54048ba" />
</a>
</div>
<br>
<a name="sec161"></a>
<h4>1.6.1 Structure of the SWIFT BIC Code</h4>
<p>
The structure of the BIC code is as follows:
</p>
<ul>
<li>
alphabetical characters that indicate the identification of the institution (bank or corporate)
</li>
<li>
2 alphabetical characters for the ISO code of the country in which the institution is located
</li>
<li>
2 alphabetic or numeric characters used to locate the institution head office in the country or the head office in a particular region in the country.
<ul>
<li>
When the second character takes the value "0", it is typically a test BIC used in test systems as opposed to a BIC used on the live network (also production).
</li>
<li>
When the second character takes the value "1", it denotes a passive participant in the SWIFT network. Passive participants cannot be contacted directly over the SWIFT Network. These BICs are sometimes referred to as ‘BIC1', ‘non-SWIFT BIC' and ‘non-connected BIC'. A non-connected BIC is not allowed in the header of a SWIFT message, otherwise the message is rejected by the SWIFT system.
</li>
<li>
When second character takes the value "2", it indicates a reverse billing BIC, where the recipient pays for the message as opposed to the more usual mode where the sender pays for the message.
</li>
</ul>
</li>
<li>
3 alphabetical or numeric characters to indicate a branch or agency of the institution. Unlike the first 8 characters, these last 3 are not mandatory. They are mainly used by banks and less by corporations.
</li>
</ul>
<p>
Few examples to illustrate the above explanations:
</p>
<ul>
<li>
DEUTDEFF is the BIC of Deutsche Bank (DEUT) / in Germany (DE) / Main office of Frankfurt (FF)
</li>
<li>
DEUTDESS is the BIC of Deutsche Bank (DEUT) / in Germany (DE) / Main office of Stuttgart (SS)
</li>
<li>
DEUTDESS648 is the BIC of Deutsche Bank (DEUT) / in Germany (DE) / Main office of Stuttgart (SS). 648 is the branch located in Vaihingen-Enz in the same region.
</li>
<li>
DEUTDES0 and DEUTDES0648 are test BIC for DEUTDESS and DEUTDESS648
</li>
<li>
LAFAFRPP is the BIC of Lafarge (LAFA) / in France (FF) / Main office of Paris (PP)
</li>
</ul>
<a name="sec17"></a>
<h3>1.7 Other specific details</h3>
<p>
Below are some specific noteworthy details about SWIFT messages in a raw fashion:
</p>
<ul>
<li>
<b>Caution: about the little "a" in field names</b>
<br>
Very often in documents and papers about SWIFT, fields are indicated with a little "a" as suffix, such as 54a, 56a, etc.
<br>
This lower case "a" needs not to be confused with the upper case "A" which indicates the variant of the field.
<br>
For instance, Field 50a exists in 3 variants: 50A, 50F, 50K. The little suffix "a" is just a convention to mention the field 50, it is not the indication of the variant "A"
</li>
<li>
<b>About SWIFT Input and Output messages</b>
<br>
In SWIFT, the notion of output and input related to the SWIFT network, not to the bank.
<br>
A SWIFT input message is an outbound message: a message emitted by the bank and send to another bank on the SWIFT network.
<br>
A SWIFT output message is an inbound message: a message received to a bank from the SWIFT network.
<br>
Qualifying messages as Input or Output is relative to the SWIFT network, i.e. inverse to the bank perspective (Input are messages sent by the bank, Output are messages received by the bank)
</li>
</ul>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/68fa3f37-3bc8-4069-8860-2adbee333ff1">
<img class="centered" style="width: 650px; " alt="SWIFT Input / Output Messages" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/68fa3f37-3bc8-4069-8860-2adbee333ff1" />
</a>
</div>
<br>
<a name="sec2"></a>
<h2>2. Dissecting key SWIFT Mesages involved in payments (Funds Transfers)</h2>
<p>
This chapter presents key SWIFT messages as well as some examples aimed at understanding the meaning and usage of most important SWIFT fields.
</p>
<a name="sec21"></a>
<h3>2.1 SWIFT MT101 Detailed Analysis</h3>
<p>
The SWIFT MT101 Request for Transfer is a payment initiation message used to send domestic and/or international payments instructions to their banks.
</p>
<p>
The situation of SWIFT MT101 in a banking institution is as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/20a4957e-1153-4690-b17f-0519e717b347">
<img class="centered" style="width: 750px; " alt="SWIFT MT101 situation" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/20a4957e-1153-4690-b17f-0519e717b347" />
</a>
</div>
<br>
<p>
Most of the time when the bank receives an MT101 it will either perform the operation if both debited and credited accounts are with itself, or it will issue an MT103 to proceed with the crediting of the destination account with another banking institution.
</p>
<p>
But there are situations as well where the account to be debited is not by the bank itself, in which case the MT101 can be routed further.
</p>
<a name="sec211"></a>
<h4>2.1.1 MT101 Introductory examples</h4>
<p>
This section presents various examples of SWIFT MT101 corresponding to different situations
</p>
<a name="sec2111"></a>
<h5>2.1.1.1 MT101 Example 1: Simplest case</h5>
<p>
A corporate customer - Robert Corporation - of bank XYZ located in Switzerland wants to send 50k CHF to Vinino SARL in Geneva, another customer of bank XYZ. Robert Corporation being a big corporation, it is connected to the SWIFT network and trades on his account by the bank using SWIFT MT 101 messages.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/72868344-56dd-4b66-8b41-809192084027">
<img class="centered" style="width: 620px; " alt="MT101 Simplest example" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/72868344-56dd-4b66-8b41-809192084027" />
</a>
</div>
<br>
<p>
In this situation, there is no SWIFT messages emitted further whatsoever since both customers are customer of bank XYZ.
</p>
<p>
Details of the MT101:
</p>
<ul>
<li>
There is one single transaction in this MT101.
</li>
<li>
Both accounts are with bank XYZ so no <i>Account with Institution</i> or <i>Account Servicing Institution</i> need to be indicated.
</li>
</ul>
<a name="sec2112"></a>
<h5>2.1.1.2 MT101 Example 2: beneficiary with another institution</h5>
<p>
A corporate customer - Robert Corporation - of bank XYZ located in Switzerland wants to send 500k EUR to Dupont SARL in Paris.
<br>
The corporation Dupont SARL is not a customer of bank XYZ, it is a customer of bank BNP Parisbas in Paris.
<br>
Bank XYZ and BNP Paribas have a direct banking relationship.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/b5c5cf6a-fbf1-4141-ad82-33cc16b08d64">
<img class="centered" style="width: 780px; " alt="MT101 Example 2" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b5c5cf6a-fbf1-4141-ad82-33cc16b08d64" />
</a>
</div>
<br>
<p>
Since Bank XYZ and BNP Paribas have a direct banking relationship, Bank XYZ can send an MT103 to BNP Paribas to make it credit the customer account from its VOSTRO account by BNP Paribas.
</p>
<p>
Details of the MT101:
</p>
<ul>
<li>
There is one single transaction in this MT101.
</li>
<li>
The beneficiary account to be credit is not by bank XYZ, so the eventual banking institution the funds need to be sent to is indicated in <i>Account With Institution</i> - 57A.
</li>
</ul>
<a name="sec2113"></a>
<h5>2.1.1.3 MT101 Example 3: Multiple payments in MT101</h5>
<p>
In this third example, Corporation Robert Corp of bank XYZ located in Switzerland wants to send a first payment of 100k EUR to Dupont Sarl in Paris and a second payment of 200 K to Jacob SA in France.
<br>
Dupont Sarl is a customer of Bank BNP Paribas in Paris and Jacob SA is a customer of bank Credit Agricole in France as well.
</p>
<p>
Bank XYZ in Switzerland and BNP Paribas Paris have a direct banking relationship, and BNP Paribas and Credit Agricole have a direct banking relationship as well.
</p>
<p>
He wants to use different accounts for both transactions and he is submitting both transactions in the very same MT101.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/bb13b9d0-9d25-4ba2-822f-4c397de74392">
<img class="centered" style="width: 850px; " alt="MT101 Example 3" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/bb13b9d0-9d25-4ba2-822f-4c397de74392" />
</a>
</div>
<br>
<p>
The MT101 makes it possible to input as many transaction in a single message, having at least one occurrence of the repetitive sequence is mandatory.
</p>
<p>
Since Credit Agricole and Bank XYZ have no direct banking relationship, the second transaction, crediting Jacob SA has to go through BNP Paribas as well.
</p>
<p>
Details of the MT101:
</p>
<ul>
<li>
This MT101 contains two distinct transactions
</li>
<li>
In both case <i>Account with Institution</i> - field 57A - identifies the eventual destination of the funds for both transactions
</li>
<li>
The <i>ordering customer</i> - field 50H - is the same from a customer perspective but the accounts used for both fund transfers are different. For this reason the ordering customer is indicated in the repetitive sequence, not in sequence A.
</li>
</ul>
<a name="sec2114"></a>
<h5>2.1.1.4 MT101 Example 4: Payment from a subsidiary account</h5>
<p>
A parent company can also use the SWIFT MT101 to pay from own account on behalf of multiple subsidiaries.
<p />
<p>
In this example, the parent company Robert Corp located in Switzerland has received an invoice from Jacob SA for various services that Jacob SA provided to the local company Robert Corp France SARL, a subsidiary of Robert Corp located in France.
</p>
<p>
The parent company Robert Corp decides to use the account of the subsidiary company in France to pay for the invoice (there can be various reasons for that).
<br>
The subsidiary has granted permission to the parent company for trading with its account with BNP Paris.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/bd536ad7-a1a4-4b87-a8c8-e0e6fcd77943">
<img class="centered" style="width: 850px; " alt="MT101 Example 4" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/bd536ad7-a1a4-4b87-a8c8-e0e6fcd77943" />
</a>
</div>
<br>
<p>
The trick here is in the field 50a Instructing Party. In the SWIFT standard we read the following under the usage rules of that field: "This field must only be used when the instructing customer is not also the account owner."
<br>
The field 50a Instructing Party specifies the subsidiary company - FR12983459182931 / Robert Corp France Sarl - on behalf of which the parent company, RBCPCHAA02A, sends the payment instruction.
</p>
<p>
Details of the MT101:
</p>
<ul>
<li>
<i>Account Servicing Institution</i> - field 52A - is identifying the fact that the MT101 needs to be transferred further to the institution owning the account of the subsidiary where the funds need to be debited from: BNP Paris / BNPSFRZA93B.
</li>
<li>
The <i>Account with institution</i> - field 57A - is identifying the eventual recipient of the funds, the institution owning the account to be credited: Bank Credit Agricole / CACIFRXA12C.
</li>
<li>
<i>Instructing Party</i> - field 50L - identifies that the parent corporation is actually the one at the origin of this MT101, acting on behalf of the <i>ordering customer</i> - 50 H - the subsidiary in France.
</li>
</ul>
<a name="sec2115"></a>
<h5>2.1.1.5 MT101 Example 5: Fund repatriation</h5>
<p>
There is another possibility: funds repatriation.
<br>
Funds repatriation simply means to move the funds available on one account to another account held either by the same financial institution or by another financial institution. Funds repatriation is performed for cash pooling inside a company or a group of companies. Corporations resort to cash pooling to optimize the liquidity usage.
</p>
<p>
In this example, Robert Corp (a corporate customer) has an account by bank XYZ in Switzerland and an account by bank BNP Paribas Paris which he uses for his operations in EUR.
<br>
Robert Corp. wants to repatriate all the funds he has on his BNP Paris account back to his Bank XYZ Account.
<br>
Bank XYZ account is called centralized account, master account or leader account. The BNP Paris account is called secondary account or slave account.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/cad13f64-be4d-4d5f-9484-11afb7c27ace">
<img class="centered" style="width: 630px; " alt="MT101 Example 5" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/cad13f64-be4d-4d5f-9484-11afb7c27ace" />
</a>
</div>
<br>
<p>
In this case the MT101 is transmitted further to the bank owning the debited account and the funds are repatriated with the MT103.
</p>
<p>
Details of the MT101:
</p>
<uL>
<li>
<i>Account Servicing Institution</i> - field 52A - is identifying the fact that the MT101 needs to be transferred further to the institution owning the account: BNP Paris / BNPSFRZA93B.
</li>
<li>
The <i>Account with institution</i> - field 57A - is identifying the eventual recipient of the funds, the institution owning the account to be credited: Bank XYZ / BXYZCHZZ80A.
</li>
<li>
Using the <i>instruction code</i> - field 23 - CMZB - Code to Zero Balance, it is not necessary to specify any amount, the code will be balanced down to zero and all the available funds repatriated.
</li>
</ul>
<a name="sec212"></a>
<h4>2.1.2 MT101 Parsing and Data Mapping </h4>
<p>
The MT101 parsing details are presented in the table below. Only most essential fields are discussed.
</p>
<table class="allborder">
<tr style="background: #CCCCCC; font-weight: bold;">
<td rowspan="2">
Meaning
</td>
<td colspan="2">
SWIFT
</td>
<td rowspan="2">
Example
</td>
<td rowspan="2" colspan="2">
Comment
</td>
</tr>
<tr style="background: #CCCCCC; font-weight: bold;">
<td>Field</td>
<td>Variant</td>
</tr>
<tr>
<td>
AppID
</td>
<td colspan="2">
Block1/ApplId
</td>
<td>
F
</td>
<td colspan="2">
The Application Identifier identifies the application within which the message is being sent or received. The available options are: F = FIN , A = GPA, etc.
<br>
These values are automatically assigned by the SWIFT system and the user's CBT
</td>
</tr>
<tr>
<td>
ServiceID
</td>
<td colspan="2">
Block1/Servid
</td>
<td>
01
</td>
<td colspan="2">
The Service Identifier consists of two numeric characters. It identifies the type of data that is being sent or received and, in doing so, the type of the following message
</td>
</tr>
<tr>
<td>
Sender
(Sending bank / BIC)
</td>
<td colspan="2">
Block1/
LTaddrBlk1 (I)
<br>
Or
<br>
Block2/
LTaddrBlk2 (O)
</td>
<td>
SGOBFRPP
</td>
<td colspan="2">
Sender BIC appears in header block (Block 1) in the MT101 Input and in the application block (Block 2) in the MT101 Output
<br>
(Input and output related to the SWIFT network, not the bank). Need to use field Block2 / Inoutind to find out
</td>
</tr>
<tr>
<td>
Message Type
</td>
<td colspan="2">
Block2/
Msgtype
</td>
<td>
101
</td>
<td colspan="2">
SWIFT Message Type = MT 101
</td>
</tr>
<tr>
<td>
Receiver
(Receiving Bank / BIC)
</td>
<td colspan="2">
Block2/
LTaddrBlk21 (I)
<br>
Or
<br>
Block1/
LTaddrBlk1 (O)
</td>
<td>
RBOSGB2L
</td>
<td colspan="2">
The Receiver BIC appears in header block (Block1) in the MT101 Output and in the application block (Block 2) in the MT101 Input.
<br>
(Input and output related to the SWIFT network, not the bank). Need to use field Block2 / Inoutind to find out.
<br>
The receiver of the message is the eventual beneficiary only if no field 57 says otherwise.
</td>
</tr>
<tr>
<td>
Sender's Reference
</td>
<td colspan="2" class="center">
20
</td>
<td>
ORDERREF1234
</td>
<td colspan="2">
This field is mandatory and of format 16x. It is a reference assigned by the Sender to unambiguously identify the message.
</td>
</tr>
<tr>
<td>
Customer Specified Reference
</td>
<td class="center">
21
</td>
<td class="center">
R
</td>
<td>
GBSUPPLIERS5668
</td>
<td colspan="2">
The field is optional and of format 16x. It is a reference to the entire message assigned by either the instructing party, when present or by the ordering customer, when the instructing party is not present.
</td>
</tr>
<tr>
<td>
Message Index/Total
</td>
<td class="center">
28
</td>
<td class="center">
D
</td>
<td>
1/1
</td>
<td colspan="2">
It is mandatory and of format : 5n/5n for (Message Index)/(Total) If you send 5 messages for the same order, this field will take the value 1/5 in the first message, 2/5 in the second message, and so on.
<br>
1/1 means there is only one message sent for this order.
</td>
</tr>
<tr>
<td>
Sender Msg. Sending Timestamp
</td>
<td colspan="2">
(O) Block2 / ‘
Intime + Inmate
(I) System.time()
</td>
<td>
1538070522
</td>
<td colspan="2">
(O) = Output only : SWIFT timestamp for an Output message (HHMMYYMMDD) or local date/time for an Input Message.
</td>
</tr>
<tr>
<td>
In / Out-put flag
</td>
<td colspan="2">
Block2 / Inoutind
</td>
<td>
I
</td>
<td colspan="2">
Single letter ‘I' or ‘O'
</td>
</tr>
<tr>
<td rowspan="3">
Ordering Customer
</td>
<td rowspan="3" class="center">
50
</td>
<td class="center">
F
</td>
<td>
/DE20700800000...<br>
1/Essilor International<br>
2/147 Rue de Paris<br>
3/FR/Charenton 94220
</td>
<td>
Line 1 (subfield Party Identifier)<br>
/34x (Account)<br>
Lines 2-5 : (Number/Name and Address)<br>
1!n/33x (Number)(Details)
</td>
<td rowspan="3" class="lx">
The field ordering customer is mandatory. In can be given either in the message details (here) or per transaction in the repeating sequence
</td>
</tr>
<tr>
<td class="center">
G
</td>
<td>
/31926819<br>
UBSCH123FNX
</td>
<td>
Line 1 (subfield Party Identifier)<br>
/34x (account)<br>
Line 2 (subfield bank)<br>
BIC (8 or 11 characters)
</td>
</tr>
<tr>
<td class="center">
H
</td>
<td>
/31926819<br>
Compagnie de Saint Gobain<br>
118 Rue Lauriston<br>
75016 Paris
</td>
<td>
Line 1 (subfield Party Identifier)<br>
/34x (Account)<br>
Lines 2-5 : (Number/Name and Address)<br>
4*35x (Name and Address)
</td>
</tr>
<tr>
<td>
Authorization
</td>
<td colspan="2" class="center">
25
</td>
<td>
12DF64BG345A
</td>
<td colspan="2">
Optional, 35x (authorization code)
</td>
</tr>
<tr>
<td>
Requested Execution Date
</td>
<td colspan="2" class="center">
30
</td>
<td>
181017
</td>
<td colspan="2">
The value date, mandatory and of format 6!n (YYMMDD). It is the date on which all subsequent transactions should be initiated by the executing bank.
</td>
</tr>
<tr>
<td colspan="6" style="background-color: #AACCFF; font-weight: bold;">
Repeating Sequence
</td>
</tr>
<tr>
<td>
Transaction Reference
</td>
<td colspan="2" class="center">
21
</td>
<td>
35863REFOFTRX1
</td>
<td colspan="2">
This field is mandatory and of format 16x. It is a reference assigned by the Sender to unambiguously identify a unique transaction.
</td>
</tr>
<tr>
<td>
F/X Deal Reference
</td>
<td class="center">
21
</td>
<td class="center">
F
</td>
<td>
FXDEALID78685
</td>
<td colspan="2">
Optional, if there is an underlying foreign exchange deal to this transaction, then this field specifies the FX deal reference
</td>
</tr>
<tr>
<td>
Instruction Code
</td>
<td class="center">
23
</td>
<td class="center">
E
</td>
<td>
CMZB (= Code to Zero balance the account. This transaction contains a cash Management instruction, requesting to zero balance the account of the ordering customer)
<br>
INTC =( Code for Intra-Company Payment.)
</td>
<td colspan="2">
It is optional and of format :4!c[/30x] (Instruction Code)(Additional Info.)
<br>
Optional instruction codes that identifies the operation types.
<br>
<b>Caution : there can be several instruction codes (all with same code) in the SWIFT message. </b>
</td>
</tr>
<tr>
<td>
Charges Account
</td>
<td class="center">
25
</td>
<td class="center">
A
</td>
<td>
/FR763000402837...
</td>
<td colspan="2">
This is the ordering customer's account number to which applicable transaction charges should be separately applied by the debtor.
</td>
</tr>
<tr>
<td>
Currency,<br>
Transaction Amount
</td>
<td class="center">
32
</td>
<td class="center">
B
</td>
<td>
GBP50000
</td>
<td colspan="2">
Mandatory and of Format 3!a15d (Currency)(Amount)
</td>
</tr>
<tr>
<td>
Currency,
Original Ordered Amount
</td>
<td class="center">
33
</td>
<td class="center">
B
</td>
<td>
EUR200000
</td>
<td colspan="2">
This optional field is provided in format 3!a15d. It specifies the original currency and amount as specified by the ordering customer.
</td>
</tr>
<tr>
<td>
Exchange Rate
</td>
<td colspan="2" class="center">
36
</td>
<td>
1,1382
</td>
<td colspan="2">
Mandatory since field 33B is present and 'amount' in field 32B is not equal to zero. Provided in format 12d. The integer part of Rate must contain at least one digit. A decimal comma is mandatory and is included in the maximum length.
</td>
</tr>
<tr>
<td>
Instructing Party
</td>
<td class="center">
50
</td>
<td class="center">
L
</td>
<td>
Compagnie de Saint Gobain<br>
118 Rue Lauriston<br>
75016 Paris
</td>
<td colspan="2">
This field is optional and to be provided if the sending customer (instructing party) is different than the owner of the account (but has an authorization to make payment on account given by owner)
<br>
Compagnie de Saint Gobain instructs the payment but does not own the account to be debited. It is authorized by it's the owner (e.g. a subsidiary) to pay form the ordering customer account provided below.
</td>
</tr>
<tr>
<td rowspan="3">
Ordering Customer
</td>
<td rowspan="3" class="center">
50
</td>
<td class="center">
F
</td>
<td>
/DE207008000...<br>
1/Essilor International<br>
2/147 Rue de Paris<br>
3/FR/Charenton 94220
</td>
<td>
Line 1 (subfield Party Identifier)<br>
/34x (Account)<br>
Lines 2-5 : (Number/Name and Address)<br>
1!n/33x (Number)(Details)
</td>
<td rowspan="3" class="lx">
The field ordering customer is mandatory. In can be given either in the message details or per transaction in the repeating sequence (here)
</td>
</tr>
<tr>
<td class="center">
G
</td>
<td>
/31926819<br>
UBSCH123FNX
</td>
<td>
Line 1 (subfield Party Identifier)<br>
/34x (account)<br>
Line 2 (subfield bank)<br>
BIC (8 or 11 characters)
</td>
</tr>
<tr>
<td class="center">
H
</td>
<td>
/31926819<br>
Compagnie de Saint Gobain<br>
118 Rue Lauriston<br>
75016 Paris
</td>
<td>
Line 1 (subfield Party Identifier)<br>
/34x (Account)<br>
Lines 2-5 : (Number/Name and Address)<br>
4*35x (Name and Address)
</td>
</tr>
<tr>
<td>
Account Servicing Institution
</td>
<td class="center">
52
</td>
<td class="center">
A
</td>
<td>
//083098 (BSB)
<br>
NATAAU33 (bic)
</td>
<td colspan="2">
This is to notify the receiving bank that the institution owning the account to debit is another institution So the MT101 needs to be transferred to that other institution identified here.
<br>
The BSB (Bank State Branch) code and the BIC are provided.
<br>
Even if it is not mandatory in the SWIFT standards to use the BSB code preceded by a double slash. Many banks impose it.
<br>
We need to parse the BIC out of it
<br>
Format :<br>
&bnsp;&bnsp;&bnsp;&bnsp;[/1!a][/34x] (Party Identifier)<br>
&bnsp;&bnsp;&bnsp;&bnsp;4!a2!a2!c[3!c] (Identifier Code)
</td>
</tr>
<tr>
<td>
Intermediary
Institution
</td>
<td class="center">
56
</td>
<td class="center">
A
</td>
<td>PNBPUS3N (bic)
</td>
<td colspan="2">
This is the Correspondent of the Creditor Bank. It holds the account in currency of the creditor bank.
<br>
It is optional and can be provided in option A, C or D.
<br>
<i>Formats C or D are rarely used and most of the time not supported by banks.</i>
<br>
Option A is formatted [/1!a][/34x] (Party Identifier)
<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)
</td>
</tr>
<tr>
<td rowspan="2">
Account With Institution
</td>
<td rowspan="2" class="center">
57
</td>
<td class="center">
A
</td>
<td>
BARCGB22XXX (bic)<br>
or e.g.<br>
//939400 (BSB)<br>
AMPBAU2SXXX (bic)
</td>
<td>
We need to parse the BIC out of it<br>
Format :<br>
[/1!a][/34x] (Party Identifier )<br>
4!a2!a2!c[3!c] (Identifier Code)
</td>
<td rowspan="2" class="lx">
Optional. The Account with Institution is provided because the beneficiary does not have an account with the debtor bank, but with another bank .(i.e. an MT103 will be sent further)
<br>
It can be provided in option A, B, C or D.
<br>
Formats B, C are rarely used and most of the time not supported by banks.
<br>
For instance, in case the Creditor bank is not the same as the debtor bank (most of the time), it is identified in field 57a. The BSB (Bank State Branch) code and the BIC of AMP Bank are provided.
<br>
We need to parse the BIC out of it or hash the address
<br>
(priority over header)
</td>
</tr>
<tr>
<td class="center">
D
</td>
<td>
Hong Kong Banking Assoc.<br>
Avenue du Léman<br>
1204 Genève - CH<br>
Switzerland
</td>
<td>
If no BIC is available to identify the target institution, option D is used.
<br>
In principle minimum 3 lines with name and address should be provided
<br>
Format:<br>
[/1!a][/34x] (Party Identifier)<br>
4*35x (Name and Address)<br>
In this case, take country from receiver BIC
</td>
</tr>
<tr>
<td rowspan="3">
Beneficiary
</td>
<td rowspan="3" class="center">
59
</td>
<td class="center">
(no letter)
</td>
<td>
/26351-38947<br>
Company One<br>
CITY STREET 50<br>
LONDON, UK
</td>
<td>
Line 1 : <br>
[/34x] (Account) IBAN format or else<br>
Line 2-5:<br>
4*35x (Name and Address)
</td>
<td rowspan="3" class="lx">
Beneficiary customer information is Mandatory
</td>
</tr>
<tr>
<td class="center">
A
</td>
<td>
NBPUS3N<br>
or e.g.<br>
/12345678901<br>
PNBPUS3N
</td>
<td>
Option A is formatted <br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)
</td>
</tr>
<tr>
<td class="center">
F
</td>
<td>
/10078074<br>
1/Company One<br>
2/CITY STREET 50<br>
3/GB/LONDON
</td>
<td>
Line 1 (subfield Party Identifier)<br>
[/34x] (Account)<br>
Lines 2-5 : (Number/Name and Address)<br>
4*(1!n/33x) (name and address)
</td>
</tr>
<tr>
<td>
Remittance Information
</td>
<td colspan="2" class="center">
70
</td>
<td>
Payment from<br>
Compagnie de Saint Gobain<br>
/INV/7828728292
</td>
<td colspan="2">
Remittance information is optional and provided in format 4*35x if available. Up to 4 lines of up to 35 X characters each. The name of the parent company is provided here, so that the beneficiary can see it is the parent company that is paying.
</td>
</tr>
<tr>
<td>
Details of Charges
</td>
<td class="center">
71
</td>
<td class="center">
A
</td>
<td>
OUR
</td>
<td colspan="2">
It is mandatory and of format 3!a. It can take 3 values: BEN, OUR and SHA.
<br>
<ul>
<li>
OUR means charges are to be borne by the ordering customer.
</li>
<li>
SHA means charges are shared between Ordering and beneficiary customers.
</li>
<li>
BEN means charges are to be borne by the beneficiary
</li>
</ul>
</td>
</tr>
</table>
<a name="sec213"></a>
<h4>2.1.3 Additional notes on MT101</h4>
<p>
Some complementary notes:
</p>
<ul>
<li>
MT 101 have a repeating sequence (several transactions in same message). In my currently company, we consider every repeating sequence B (repeating) as a different transaction. The common sequence (A) is duplicated for each.
</li>
<li>
We still need a way identify uniquely a message. Unfortunately SWIFT doesn't have such thing as a unique transaction identified. We usually use as "business_reference" = [Transaction Reference / 21 / transaction_business_reference] concatenated to a UUID as suffix.
</li>
<li>
What to do in case of 57D if we cannot find a country code? One way is to use the SWIFT message Receiver BIC country (sometimes wrong but better than nothing, only sometimes since when 57D is used we are likely in the same country
</li>
</ul>
<a name="sec22"></a>
<h3>2.2 SWIFT MT103 Detailed Analysis</h3>
<p>
The MT103 is a SWIFT message format used for making payments. MT103 SWIFT payments are known as international wire transfers, telegraphic transfers, standard EU payments (SEPA payments), LVTS in Canada, etc.
</p>
<p>
The situation of SWIFT MT103 in a banking institution is as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/b1e34a62-0eae-4551-9d8a-ebe8108d684f">
<img class="centered" style="width: 750px; " alt="SWIFT MT103 situation" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b1e34a62-0eae-4551-9d8a-ebe8108d684f" />
</a>
</div>
<br>
<p>
There can be many different situations:
</p>
<ul>
<li>
An MT103 can be sent after a request from a customer of the bank to send funds cross-border to another institution. The customer request can come from any channel including the reception of an MT101.
</li>
<li>
The MT103 can be a routed message in case the bank is simply a routing bank in the chain or an explicit intermediary institution.
</li>
<li>
Then MT103 can be simple announces - in the cover method - or actual funds transfers - in the serial method.
</li>
</ul>
<a name="sec221"></a>
<h4>2.2.1 MT103 Introductory examples</h4>
<p>
This section presents various examples of SWIFT MT101 corresponding to different situations
</p>
<a name="sec2211"></a>
<h5>2.2.1.1 MT103 Example 1: Simplest case</h5>
<p>
In this example, the customer "John Robert" of bank XYZ in Switzerland wants to send 500k EUR to Dupont SARL, a corporation located in Paris which is a BNP Paribas customer.
</p>
<p>
Bank XYZ and BNP Paribas have a direct banking relationship.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/05f42c09-f59f-4093-87d4-d8bba7e1ed83">
<img class="centered" style="width: 740px; " alt="SWIFT MT101 Example 1" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/05f42c09-f59f-4093-87d4-d8bba7e1ed83" />
</a>
</div>
<br>
<p>
A single MT103 is sufficient to implement the fund transfer.
</p>
<a name="sec2212"></a>
<h5>2.2.1.2 MT103 Example 2: more realistic example</h5>
<p>
This example is a more realistic example. The previous one is from a pure SWIFT specification perspective entirely valid. But in practice, a few more fields are almost always present in a SWIFT message.
</p>
<p>
This example is more or less the same as the previous one: the customer "John Robert" of bank XYZ in Switzerland wants to send 30k EUR to Dupont SARL, a corporation located in Paris which is a BNP Paribas customer.
<br>
Bank XYZ and BNP Paribas have a direct banking relationship.
</p>
<p>
The first thing is that Bank XYZ obviously has several VOSTRO account by BNP Paribas, and it wants to choose which of these accounts it wants to use for the reimbursement. It wants to use account 12345678901.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/405bf70f-f869-448f-a667-892b602a2367">
<img class="centered" style="width: 740px; " alt="SWIFT MT101 Example 12" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/405bf70f-f869-448f-a667-892b602a2367" />
</a>
</div>
<br>
<p>
The field 53 normally indicates the Sender's correspondent (i.e. a banking institution), but it's common practice to use the variant 53B to indicate the account by the receiving institution to be used for the reimbursement.
</p>
<p>
In addition, in this more realistic example the charges details are indicated as well as remittance information.
</p>
<a name="sec2213"></a>
<h5>2.2.1.3 MT103 Example3: forwarded serial message</h5>
<p>
In the case the bank we are operating within is neither the <i>ordering institution</i> (initial sender) neither the eventual beneficiary institution, it is just a routing bank. In this case, specific fields are used to identify the initial sending institution as well as the eventual beneficiary institution. These fields are illustrated in this example.
</p>
<p>
In this example, customer Max Prank of bank BCVs in Switzerland wants to send 30k EUR to Alfred SARL in Samoens (France), a customer of "Banque Populaire" in France.
<br>
BCVs in Switzerland and Banque Populaire in France have no direct relationship together so the fund transfer has to go through several routing bank, one of them being bank XYZ.
</p>
<p>
We are here interested in the MT 103 sent by Banque XYZ, a routing bank to the next bank in the routing chain: BNP Paribas.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/98b2922b-79aa-4587-b775-7ee972bcfc1f">
<img class="centered" style="width: 850px; " alt="SWIFT MT101 Example 3" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/98b2922b-79aa-4587-b775-7ee972bcfc1f" />
</a>
</div>
<br>
<p>
Since bank XYZ is not the initial sending institution, and the receiver of the SWIFT message, bank BNP Paribas is not the eventual beneficiary institution, these two other institutions (initial sending and eventual beneficiary) needs to be indicated in the SWIFT messages.
</p>
<p>
Details of the SWIFT MT103:
</p>
<ul>
<li>
The field <i>Ordering institution</i> - field 52A - is clearly identifying the initial sending institution, bank BCVs
</li>
<li>
The field <i>Account with institution</i> - field 57A - is clearly identifying the eventual beneficiary institution of the funds
</li>
</ul>
<p>
It is relevant to look at the two other SWIFT messages from the chain to be able to compare them:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/faf8fcf2-f238-44d8-b578-0099c0099910">
<img class="centered" style="width: 850px; " alt="SWIFT MT101 Example 3 - B" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/faf8fcf2-f238-44d8-b578-0099c0099910" />
</a>
</div>
<br>
<p>
Another noteworthy field is the field <i>Intermediary account</i> - field 56A - which identified that the message has to go through BNP Paribas after Bank XYZ.
</p>
<a name="sec2214"></a>
<h5>2.2.1.4 MT103 Example 4: Announce message (cover method)</h5>
<p>
We'll now go through a case where the MT103 is just an announce as part of the cover payment method. This is the case when the initial sending institution and the eventual beneficiary institution have no relationship together and decide to go through their correspondent.
</p>
<p>
In case the MT103 is just an announce, it can be sent directly from the initial sending institution to the eventual beneficiary institution regardless of the fact that they have no banking relationship with each other.
</p>
<p>
In this example, the customer John Trump of bank XYZ in Switzerland wants to send 1'000'000 USD to Cowboy Corp. in Kensas City, a customer of "Kensas Credit"
<br>
Due to the nature of the transaction and their missing banking relationship together, they decide to go through correspondent banks :
</p>
<ul>
<li>
An MT103 is sent directly to the beneficiary bank, regardless of the fact they have no banking relationship together
</li>
<li>
An MT202.COV will be routed through correspondent(s) and routing banks
</li>
</ul>
<p>
We are here interested in the MT 103 sent by Bank XYZ, the announcement, which can be sent directly to beneficiary bank
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/c99c7a41-eb49-4637-ba93-6e83cb15b354">
<img class="centered" style="width: 850px; " alt="SWIFT MT101 Example 4" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/c99c7a41-eb49-4637-ba93-6e83cb15b354" />
</a>
</div>
<br>
<p>
The MT103 announce clearly indicates that the fund transfer will go through correspondents with the usage of the fields <i>Sender's correspondent</i> - field 53A - and <i>Receiver's Correspondent</i> - field 54A.
<br>
Aside from this, there is no specific difference between this announce and a serial MT103 fund transfer.
</p>
<a name="sec222"></a>
<h4>2.2.2 MT103 Parsing and Data Mapping </h4>
<p>
The MT103 parsing details are presented in the table below. Only most essential fields are discussed.
</p>
<table class="allborder">
<tr style="background: #CCCCCC; font-weight: bold;">
<td rowspan="2">
Meaning
</td>
<td colspan="2">
SWIFT
</td>
<td rowspan="2">
Example
</td>
<td rowspan="2" colspan="2">
Comment
</td>
</tr>
<tr style="background: #CCCCCC; font-weight: bold;">
<td>Field</td>
<td>Variant</td>
</tr>
<tr>
<td>
AppID
</td>
<td colspan="2">
Block1/ApplId
</td>
<td>
F
</td>
<td colspan="2">
The Application Identifier identifies the application within which the message is being sent or received. The available options are: F = FIN , A = GPA, etc.
<br>
These values are automatically assigned by the SWIFT system and the user's CBT.
</td>
</tr>
<tr>
<td>
ServiceID
</td>
<td colspan="2">
Block1/Servid
</td>
<td>
01
</td>
<td colspan="2">
The Service Identifier consists of two numeric characters. It identifies the type of data that is being sent or received and, in doing so, the type of the following message
</td>
</tr>
<tr>
<td>
Sender
(Sending bank / BIC)
</td>
<td colspan="2">
Block1/
LTaddrBlk1 (I)
<br>
Or
<br>
Block2/
LTaddrBlk2 (O)
</td>
<td>
SGOBFRPP
</td>
<td colspan="2">
Sender BIC appears in header block (Block 1) in the MT103 Input and in the application block (Block 2) in the MT103 Output
<br>
(Input and output related to the SWIFT network, not the bank). Need to use field Block2 / Inoutind to find out
</td>
</tr>
<tr>
<td>
Message Type
</td>
<td colspan="2">
Block2/
Msgtype
</td>
<td>
103
</td>
<td colspan="2">
SWIFT Message Type = MT 103
</td>
</tr>
<tr>
<td>
Cover or Serial Transfer Type
</td>
<td colspan="2">
</td>
<td>
If field 56a or 57a are present : then transfer type = "serial" <br>
If field 53a (ex 53B) or 54a are present : then transfer type = "announce (cover)"
<br>
(Default : serial)
</td>
<td colspan="2">
Knowing whether the MT103 is a serial transfer or the announce preceding a cover transfer is important. For a given use case, we actually expect a banking institution always to use the very same method.
</td>
</tr>
<tr>
<td>
Receiver
(Receiving Bank / BIC)
</td>
<td colspan="2">
Block2/
LTaddrBlk2(I)
<br>
Or
<br>
Block1/
LTaddrBlk1 (O)
</td>
<td>
RBOSGB2L
</td>
<td colspan="2">
The Receiver BIC appears in header block (Block1) in the MT103 Output and in the application block (Block 2) in the MT103 Input.<br>
(Input and output related to the SWIFT network, not the bank). Need to use field Block2 / Inoutind to find out.<br>
The receiver of the message is the eventual beneficiary only if no field 57 says otherwise.
</td>
</tr>
<tr>
<td>
Unique End-to-end Transaction Reference
</td>
<td colspan="2">
Block3/ Tag 121
</td>
<td>
b03c6901-bbed-4aa9-afdh-A5bc26d19257
</td>
<td colspan="2">
This reference is provided in the user block (Block 3) and transported end-to-end. It is mandatory in MT103 but can still be missing as well as have duplicates.
</td>
</tr>
<tr>
<td>
Sender's Reference
</td>
<td colspan="2" class="center">
20
</td>
<td>
ORDERREF1234
</td>
<td colspan="2">
This field is mandatory and of format 16x. It is a reference assigned by the Sender to unambiguously identify the message.
</td>
</tr>
<tr>
<td>
Sender Msg. Sending Timestamp
</td>
<td colspan="2">
(O) Block2 / ‘
Intime + Indate<br>
(I) Sys.time()
</td>
<td>
1538070522
</td>
<td colspan="2">
(O) = Output only : SWIFT timestamp for an Output message (HHMMYYMMDD)
<br>
or local date/time for an Input Message.
</td>
</tr>
<tr>
<td>
In / Out-put flag
</td>
<td colspan="2">
Block2 / Inoutind
</td>
<td>
I
</td>
<td colspan="2">
Single letter ‘I' or ‘O'
</td>
</tr>
<tr>
<td>
Bank operation code
</td>
<td class="center">
23
</td>
<td class="center">
B
</td>
<td>
CRED
</td>
<td colspan="2">
It is mandatory and of format 4!c.
</td>
</tr>
<tr>
<td>
Instruction code
</td>
<td class="center">
23
</td>
<td class="center">
E
</td>
<td>
PHOB/+34.91.397.6789
</td>
<td colspan="2">
It is optional and of format :4!c[/30x] (Instruction Code)(Additional Info.)
<br>
Code PHOB means Phone Benef.. The sender requests the benef. bank to contact benef. by phone when funds are received.
</td>
</tr>
<tr>
<td>
Value Date / currency / interbank settled amount
</td>
<td class="center">
32
</td>
<td class="center">
A
</td>
<td>
180816USD2325,
</td>
<td colspan="2">
It is mandatory and of format 6!n3!a15d (Date)(Currency)(Amount).<br>
Note the trailing coma (i.e. decimal part is not mandatory if 0)
</td>
</tr>
<tr>
<td>
Currency / Instructed Amount
</td>
<td class="center">
33
</td>
<td class="center">
B
</td>
<td>
USD2350,
</td>
<td colspan="2">
Normally optional in the standard. It may be provided for instance because the sender has taken fees. See field 71F below.<br>
Format 3!a15d (Currency)(Amount)
</td>
</tr>
<tr>
<td rowspan="3">
Ordering Customer
</td>
<td rowspan="3" class="center">
50
</td>
<td class="center">
A
</td>
<td>
/DE3750070010...<br>
DEUTDEFF
</td>
<td>
Line 1 (subfield Party Identifier)<br>
/34x (account)<br>
Line 2 (subfield bank)<br>
4!a2!a2!c[3!c] (Identifier Code)
</td>
<td rowspan="3" class="lx">
The field ordering customer is mandatory. In can be given either in the message details (here) or per transaction in the repeating sequence.<br>
The ordering customer is customer of the sender only if there is no field 52. The ordering customer remains constant in the message chain.
</td>
</tr>
<tr>
<td class="center">
F
</td>
<td>
/DE207008000...<br>
1/Essilor International<br>
2/147 Rue de Paris<br>
3/FR/Charenton-le-Pont, 94220
</td>
<td>
Line 1 (subfield Party Identifier)<br>
/34x (Account)<br>
Lines 2-5 : (Number/Name and Address)<br>
1!n/33x (Number)(Details)
</td>
</tr>
<tr>
<td class="center">
K
</td>
<td>
/CH570483509...<br>
GALLMAN COMPANY GMBH<br>
RAEMISTRASSE, 71<br>
8006 ZURICH<br>
SWITZERLAND
</td>
<td>
Line 1 : (subfield party identified)<br>
/34x (Account)<br>
Line 2-5 (subfield Address)<br>
4*35x (Name and Address)
</td>
</tr>
<tr>
<td rowspan="2">
Ordering institution
</td>
<td rowspan="2" class="center">
52
</td>
<td class="center">
A
</td>
<td>
BNPAFRPP<br>
or e.g.<br>
/FR123509321...<br>
BNPAFRPP
</td>
<td>
Format<br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code)
</td>
<td rowspan="2" class="lx">
Ordering institution is optional and can be provided in two options A (usual) and D (less common).<br>
The sender populates this field to indicate that the initial instruction comes from another institution (ordering institution).<br>
The ordering institution remains constant in the message chain.<br>
(priority over header)
</td>
</tr>
<tr>
<td class="center">
D
</td>
<td>
BANQUE DELUBAC ET CIE<br>
16 PL SALEON<br>
TERRAS<br>
07160 LE CHEYLARD
</td>
<td>
Format<br>
[/1!a][/34x] (Party Identifier)<br>
4*35x (Name and Address)
</td>
</tr>
<tr>
<td rowspan="2">
Sender's correspondent
</td>
<td rowspan="2" class="center">
53
</td>
<td class="center">
A
</td>
<td>
PNBPUS3N<br>
or e.g.<br>
/12345678901<br>
PNBPUS3N
</td>
<td colspan="2">
<b>Cover payments only</b><br>
Correspondent of sender. Sender has an account in Currency with this banking institution.<br>
Option A is formatted <br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)
</td>
</tr>
<tr>
<td class="center">
B
</td>
<td>
/12345678901
</td>
<td colspan="2">
<b>Caution : Serial and cover payments.</b><br>
Field 53B indicates the account number of the Sender, serviced by the Receiver, which is to be used for reimbursement (debit) in the transfer. This the account of the sender by the receiver (VOSTRO)<br>
Option B Format is:<br>
[/1!a][/34x] (Party Identifier)<br>
[35x] (Location)<br>
The field is optional but In practice, the account number is almost always provided.
</td>
</tr>
<tr>
<td>
Receiver's correspondent
</td>
<td class="center">
54
</td>
<td class="center">
A
</td>
<td>
IRVTUS3N<br>
or e.g.<br>
/9876412-1234/123<br>
IRVTUS3N
</td>
<td colspan="2">
<b>Cover payments only.</b><br>
Correspondent of receiver. Receiver has an account in currency with this banking institution. <br>
Option A is formatted <br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)
</td>
</tr>
<tr>
<td>
Intermediary Institution
</td>
<td class="center">
56
</td>
<td class="center">
A
</td>
<td>
IRVTUS3N (bic)<br>
or e.g.<br>
/939400 (BSB or account)<br>
AMPBAU2SXXX (bic)
</td>
<td colspan="2">
<b>Serial payments only.</b><br>
This is the Correspondent of Creditor Bank. It holds the account in currency of the creditor bank. It is used instead of over 54a (Receiver's Correspondent) in case of a serial payment transfer.<br>
It is optional and can be provided in option A, C or D. <br>
<i>Formats C or D are rarely used and most of the time not supported by banks.</i><br>
Option A is formatted <br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)
</td>
</tr>
<tr>
<td rowspan="2">
Account With Institution
</td>
<td rowspan="2" class="center">
57
</td>
<td class="center">
A
</td>
<td>
BARCGB22XXX (bic)<br>
or e.g.<br>
//939400 (BSB)<br>
AMPBAU2SXXX (bic)
</td>
<td>
We need to parse the BIC out of it <br>
Format :<br>
[/1!a][/34x] (Party Identifier )<br>
4!a2!a2!c[3!c] (Identifier Code)
</td>
<td rowspan="2" class="lx">
<b>Serial payments only.</b><br>
Account with institution is optional and can be provided in option A, B, C or D. <br>
Formats B, C are rarely used and most of the time not supported by banks.<br>
Field 57 is used when the receiver of the SWIFT MT103 doesn't own the beneficiary account and needs to send the Message Further. In the final MT103 on the chain, the holder of the account will be the receiver and no field 57 will be required anymore. <br>
We need to parse the BIC out of it or hash the address <br>
(priority over header)
</td>
</tr>
<tr>
<td class="center">
D
</td>
<td>
Hong Kong Banking Assoc.<br>
Avenue du Léman<br>
1204 Genève - CH<br>
Switzerland
</td>
<td>
If no BIC is available to identify the target institution, option D is used.<br>
In principle minimum 3 lines with name and address should be provided<br>
Format :<br>
[/1!a][/34x] (Party Identifier )<br>
4*35x (Name and Address)<br>
In this case, take country from receiver BIC.
</td>
</tr>
<tr>
<td rowspan="3">
Beneficiary
</td>
<td rowspan="3" class="center">
59
</td>
<td class="center">
(no letter)
</td>
<td>
/26351-38947<br>
Company One<br>
CITY STREET 50<br>
LONDON, UK
</td>
<td>
Line 1 : <br>
[/34x] (Account) (IBAN format or else)<br>
Line 2-5:<br>
4*35x (Name and Address)
</td>
<td rowspan="3" class="lx">
Beneficiary customer information is Mandatory. <br>
The beneficiary remains constant in the message chain.
</td>
</tr>
<tr>
<td class="center">
A
</td>
<td>
PNBPUS3N<br>
or e.g.<br>
/12345678901<br>
PNBPUS3N
</td>
<td>
Option A is formatted <br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)
</td>
</tr>
<tr>
<td class="center">
F
</td>
<td>
/10078074<br>
1/Company One<br>
2/CITY STREET 50<br>
3/GB/LONDON
</td>
<td>
Line 1 (subfield Party Identifier)<br>
[/34x] (Account)<br>
Lines 2-5 : (Number/Name and Address)<br>
4*(1!n/33x) (name and address)
</td>
</tr>
<tr>
<td>
Remittance Information
</td>
<td colspan="2" class="center">
70
</td>
<td>
/INV/18042-090715
</td>
<td colspan="2">
Remittance information is optional and provided in format 4*35x if available. Up to 4 lines of up to 35 X characters each.<br>
Usually the remittance information is generated by the beneficiary and sent to the ordering customer (or debtor). The beneficiary requests the debtor to provide it the payment message, so that the beneficiary can easily reconcile the payment with an invoice for instance.
</td>
</tr>
<tr>
<td>
Details of Charges
</td>
<td class="center">
71
</td>
<td class="center">
A
</td>
<td>
OUR
</td>
<td colspan="2">
It is mandatory and of format 3!a. It can take 3 values: BEN, OUR and SHA.
<br>
<ul>
<li>
OUR means charges are to be borne by the ordering customer.
</li>
<li>
SHA means charges are shared between Ordering and beneficiary customers.
</li>
<li>
BEN means charges are to be borne by the beneficiary
</li>
</ul>
</td>
</tr>
<tr>
<td>
Sender's charges
</td>
<td class="center">
71
</td>
<td class="center">
F
</td>
<td>
EUR2,50
</td>
<td colspan="2">
Optional. When 71A is BEN (or SHA), 71G contains amount of the charges due, which have been deducted from the interbank settlement amount.<br>
Interbank settled amount = Instructed amount - Sender's charges.<br>
Format 3!a15d <br>
<b>Caution: there can be several different 71F in a same MT103.</b>
</td>
</tr>
<tr>
<td>
Receiver's charges
</td>
<td class="center">
71
</td>
<td class="center">
G
</td>
<td>
EUR2,50
</td>
<td colspan="2">
Optional. When 71A is OUR (or SHA), 71G contains amount of the charges due, which have been prepaid and included in the interbank settlement amount.<br>
Format 3!a15d<br>
<b>Caution : there can be several different 71G in a same MT103. </b>
</td>
</tr>
<tr>
<td>
Sender to Receiver Information
</td>
<td colspan="2" class="center">
72
</td>
<td>
/INS/BNPAFRPP
</td>
<td colspan="2">
This is an optional field. It takes the Format 6*35x.<br>
There can be many codes indicating additional information. <br>
INS is a code indicating that BNPAFRPP is the instructing institution. Without field 72, the receiver may not know it since that information is not provided somewhere else in the message when sender is the next bank on the routing chain and ordering institution is another bank before the instructing one.
</td>
</tr>
</table>
<a name="sec223"></a>
<h4>2.2.3 Additional notes on MT103</h4>
<p>
Some complementary notes:
</p>
<ul>
<li>
The option field 53B (only variant B) is really only used to indicate which account at the correspondent (receiver) should be debited.
</li>
<li>
Fields 51A seems to be not supported by most banks (at least all I found such as UBS, etc.)
</li>
<li>
When no correspondent is used neither on sender side (Tag 53A) nor on receiver side (Tag 54A) and No reimbursement party (Tags 56a and 57a) is indicated in the SWIFT MT103 message. It means
<ul>
<li>there is a direct account relationship, in the currency of the transfer, between the Sender and the Receiver. Money will be taken from account and credited to the beneficiary.</li>
<li>Beneficiary customer account (:59) is hold by the receiver.</li>
</ul>
</li>
<li>
The fields 56a and 57a are used for serial transfers while the fields 53a (ex 53B) and 54a are used in the MT103 Announcement Message (cover method).
</li>
<li>
What to do in case of 57D if we cannot find a country code? We Use SWIFT message Receiver BIC country (sometimes wrong but better than nothing, only sometimes since when 57D is used we are likely in the same country)
</li>
<li>
Routing
<ul>
<li>When there is no ordering institution (Tag 52) in the SWIFT MT103 message. That means implicitly that the ordering customer is customer of the Sender.</li>
<li>When the ordering institution (Tag 52D) is provided in the MT103 SWIFT Message. This means the ordering customer is not customer of the Sender. </li>
<ul>
<li>Either the sending institution sends the MT103 on behalf of the ordering institution in 52D. This happens when the ordering institution is a small bank that has an agreement with a major bank (sending bank) for the processing and settlement of currency transactions. The small bank can use the correspondent network of sending institution.</li>
<li>Or the Sender is a routing bank on the chain</li>
</ul>
</li>
</ul>
</li>
<li>
Field 57 is used when the receiver of the SWIFT MT103 doesn't own the beneficiary account and needs to send the Message Further.
<br>
In the final MT103 on the chain, the holder of the account will be the receiver and no field 57 will be required anymore.
<ul>
<li>This indicates that Sender and Beneficiary customer's Bank do not have direct account relationship in the currency of the transaction (USD). Otherwise the sender would send the message directly to the beneficiary customer's Bank.</li>
</ul>
</li>
<li>
Sender's reference is new for every message in the routing chain, but end-to-end reference remains constant.
</li>
</ul>
<a name="sec23"></a>
<h3>2.3 SWIFT MT202 Detailed Analysis</h3>
<p>
SWIFT MT202 Messages are used for interbank funds transfers. There is no customer involved when issuing an MT202. Funds are not sent on behalf of a customer but on behalf of the initial sending banking institution itself.
</p>
<p>
The situation of SWIFT MT202 in a banking institution is as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d5018e99-e484-4633-b558-a738539a9dff">
<img class="centered" style="width: 750px; " alt="SWIFT MT202 situation" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d5018e99-e484-4633-b558-a738539a9dff" />
</a>
</div>
<br>
<p>
There can be two different situations:
</p>
<ul>
<li>
Either the bank is the initial sending institution, in which case it will be identified both as the sender of the SWIFT message and perhaps as the Ordering Institution (52a)
</li>
<li>
Or the bank is just a routing bank in the routing chain in which case it is the sender but shall be different that the Ordering Institution (52a)
</li>
</ul>
<a name="sec231"></a>
<h4>2.3.1 SWIFT MT202 Introductory Exampkes</h4>
<p>
This section presents various examples of SWIFT MT202 corresponding to different situations
</p>
<a name="sec2311"></a>
<h4>2.3.1.1 MT202 Example 1: simplest case</h4>
<p>
In this first example, bank XYZ wants to send 1 million euros from its general VOSTRO account 1234-5678 by BNP Paribas to another of its own account FR982381827331 also by BNP Paribas.
<br>
The field "Receiver's Correspondent" - 53B - is diverted for the usage of identifying the source account to use for debit.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/adf80790-2744-4778-82d3-44f7dd25f774">
<img class="centered" style="width: 550px; " alt="SWIFT MT202 Example 1" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/adf80790-2744-4778-82d3-44f7dd25f774" />
</a>
</div>
<br>
<p>
Details of the SWIFT MT202:
</p>
<ul>
<li>
The <i>beneficiary institution</i> is a field that doesn't exist in MT103 and introduced by MT202. It does not identify the eventual institution owning the account to be credited but the Banking institution owning that account, actually the bank being the customer of the institution owning its correspondent account.
</li>
</ul>
<a name="sec2312"></a>
<h4>2.3.1.2 MT202 Example 2: other bank case</h4>
<p>
In this second example, bank XYZ wants to send 1 million euros from its general VOSTRO account 1234-5678 by BNP Paribas to an account belonging to another financial institution: Banque Populaire in France. This other banking institution also owns an account by BNP Paribas.
<br>
The account of Banque Populaire by BNP Paribas is FR98238182733.
</p>
<p>
The field "<i>Receiver's Correspondent</i>" - 53B - is diverted for the usage of identifying the source account to use for debit.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/02aec4a2-49d6-498d-8301-3d5997e37727">
<img class="centered" style="width: 580px; " alt="SWIFT MT202 Example 2" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/02aec4a2-49d6-498d-8301-3d5997e37727" />
</a>
</div>
<br>
<p>
The details of this SWIFT MT 202 are fundamentally similar to the one from the previous example:
</p>
<ul>
<li>The field <i>Beneficiary Institution</i> - 58A - identifies clearly the eventual beneficiary institution owning the account by the Account with Institution - field 57A, in this case Banque Populaire.
</li>
</ul>
<a name="sec2313"></a>
<h4>2.3.1.3 MT202 Example 3: routed message</h4>
<p>
In this example we’ll see how a routed MT202 looks like. In this example, bank BCVs wants to send money to Banque Populaire. They are not using correspondent, the initial account debited is by BCVs and the eventual beneficiary account is by Banque Populaire.
</p>
<p>
Since BCVs and Banque Populaire have no relationship together, the MT202 needs to be routed through banks having a banking relationship. In this example we look at the MT202 forwarded by bank XYZ, one of the bank on the routing chain.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/9a25620b-6fc9-4493-93f2-dda68ac42ade">
<img class="centered" style="width: 680px; " alt="SWIFT MT202 Example 3" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/9a25620b-6fc9-4493-93f2-dda68ac42ade" />
</a>
</div>
<br>
<p>
Details of this SWIFT MT202 message:
</p>
<ul>
<li>The field <i>account with Institution</i> - 57A has the same value than the field beneficiary institution - 58A - since the funds make it to Banque Populaire.</li>
</ul>
<a name="sec232"></a>
<h3>2.3.2 MT202 Parsing and Data Mapping </h3>
<p>
The MT202 parsing details are presented in the table below. Only most essential fields are discussed.
</p>
<table class="allborder">
<tr style="background: #CCCCCC; font-weight: bold;">
<td rowspan="2">
Meaning
</td>
<td colspan="2">
SWIFT
</td>
<td rowspan="2">
Example
</td>
<td rowspan="2" colspan="2">
Comment
</td>
</tr>
<tr style="background: #CCCCCC; font-weight: bold;">
<td>Field</td>
<td>Variant</td>
</tr>
<tr>
<td>
AppID
</td>
<td colspan="2">
Block1/ApplId
</td>
<td>
F
</td>
<td colspan="2">
The Application Identifier identifies the application within which the message is being sent or received. The available options are: F = FIN , A = GPA, etc.
<br>
These values are automatically assigned by the SWIFT system and the user's CBT.
</td>
</tr>
<tr>
<td>
ServiceID
</td>
<td colspan="2">
Block1/Servid
</td>
<td>
01
</td>
<td colspan="2">
The Service Identifier consists of two numeric characters. It identifies the type of data that is being sent or received and, in doing so, the type of the following message
</td>
</tr>
<tr>
<td>
Sender
(Sending bank / BIC)
</td>
<td colspan="2">
Block1/
LTaddrBlk1 (I)
<br>
Or
<br>
Block2/
LTaddrBlk2 (O)
</td>
<td>
SGOBFRPP
</td>
<td colspan="2">
Sender BIC appears in header block (Block 1) in the MT202 Input and in the application block (Block 2) in the MT202 Output
<br>
(Input and output related to the SWIFT network, not the bank). Need to use field Block2 / Inoutind to find out
</td>
</tr>
<tr>
<td>
Message Type
</td>
<td colspan="2">
Block2/
Msgtype
</td>
<td>
202
</td>
<td colspan="2">
SWIFT Message Type = MT 202
</td>
</tr>
<tr>
<td>
Validation Flag
</td>
<td colspan="2">
Block3/Tag119
</td>
<td>
COV (if MT202 is a cover payment)
</td>
<td colspan="2">
This validation flag is provided the user block (Block 3) and transported end-to-end. It indicates that the message is a MT202 Cover.
</td>
</tr>
<tr>
<td>
Receiver
(Receiving Bank / BIC)
</td>
<td colspan="2">
Block2/
LTaddrBlk2(I)
<br>
Or
<br>
Block1/
LTaddrBlk1 (O)
</td>
<td>
RBOSGB2L
</td>
<td colspan="2">
The Receiver BIC appears in header block (Block1) in the MT292 Output and in the application block (Block 2) in the MT202 Input.
<br>
(Input and output related to the SWIFT network, not the bank). <b>Need to use field Block2 / Inoutind to find out.</b>
<br>
The receiver of the message is the eventual beneficiary only if no field 57 says otherwise.
</td>
</tr>
<tr>
<td>
Unique End-to-end Transaction Reference
</td>
<td colspan="2">
Block3/ Tag 121
</td>
<td>
b03c6901-bbed-4aa9-afdh-A5bc26d19257
</td>
<td colspan="2">
This reference is provided in the user block (Block 3) and transported end-to-end. It is mandatory in MT103 but can still be missing as well as have duplicates.
</td>
</tr>
<tr>
<td colspan="6" style="background-color: #AACCFF; font-weight: bold;">
Sequence A - General Information (Matching MT 202 format)
</td>
</tr>
<tr>
<td>
Sender's Reference
</td>
<td colspan="2" class="center">
20
</td>
<td>
ORDERREF1234
</td>
<td colspan="2">
This field is mandatory and of format 16x. It is a reference assigned by the Sender to unambiguously identify the message.
</td>
</tr>
<tr>
<td>
Related reference
</td>
<td colspan="2" class="center">
21
</td>
<td>
123456789ABCDEF
</td>
<td colspan="2">
This field is mandatory and of format 16x.
</td>
</tr>
<tr>
<td>
Sender Msg. Sending Timestamp
</td>
<td colspan="2">
(O) Block2 / ‘
Intime + Indate
<br>
(I) Sys.time()
</td>
<td>
1538070522
</td>
<td colspan="2">
(O) = Output only : SWIFT timestamp for an Output message (HHMMYYMMDD)
<br> or local date/time for an Input Message.
</td>
</tr>
<tr>
<td>
In / Out-put flag
</td>
<td colspan="2">
Block2 / Inoutind
</td>
<td>
I
</td>
<td colspan="2">
Single letter ‘I’ or ‘O’
</td>
</tr>
<tr>
<td>
Value Date / currency / interbank settled amount
</td>
<td class="center">
32
</td>
<td class="center">
A
</td>
<td>
180816USD2325,
</td>
<td colspan="2">
It is mandatory and of format 6!n3!a15d (Date)(Currency)(Amount). <br>
Note the trailing coma (i.e. decimal part is not mandatory if 0)
</td>
</tr>
<tr>
<td rowspan="2">
Ordering institution
</td>
<td rowspan="2" class="center">
52
</td>
<td class="center">
A
</td>
<td>
BNPAFRPP<br>
or e.g.<br>
/FR123509321...<br>
BNPAFRPP
</td>
<td>
Format<br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code)<br>
</td>
<td rowspan="2" class="lx">
Ordering institution is optional and can be provided in two options A (usual) and D (less common). <br>
The sender populate this field to indicate that the initial instruction comes from another institution (ordering institution). <br>
The ordering institution remains constant in the message chain. <br>
(priority over header)
</td>
</tr>
<tr>
<td class="center">
D
</td>
<td>
BANQUE DELUBAC ET CIE<br>
16 PL SALEON<br>
TERRAS <br>
07160 LE CHEYLARD
</td>
<td>
Format<br>
[/1!a][/34x] (Party Identifier)<br>
4*35x (Name and Address)
</td>
</tr>
<tr>
<td rowspan="2">
Sender's correspondent
</td>
<td rowspan="2" class="center">
53
</td>
<td class="center">
A
</td>
<td>
PNBPUS3N<br>
or e.g.<br>
/12345678901<br>
PNBPUS3N
</td>
<td colspan="2">
<b>Cover payments only</b><br>
Correspondent of sender. Sender has an account in Currency with this banking institution. <br>
Option A is formatted <br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)
</td>
</tr>
<tr>
<td class="center">
B
</td>
<td>
/12345678901
</td>
<td colspan="2">
<b>Caution : Serial and cover payments.</b><br>
Field 53B indicates the account number of the Sender, serviced by the Receiver, which is to be used for reimbursement (debit) in the transfer. This the account of the sender by the receiver (VOSTRO)<br>
Option B Format is:<br>
[/1!a][/34x] (Party Identifier) <br>
[35x] (Location)<br>
<i>The field is optional but In practice, the account number is almost always provided.</i><br>
</td>
</tr>
<tr>
<td>
Receiver’s correspondent
</td>
<td class="center">
54
</td>
<td class="center">
A
</td>
<td>
IRVTUS3N<br>
or e.g.<br>
/9876412-1234/123<br>
IRVTUS3N
</td>
<td colspan="2">
<b>Cover payments only.</b><br>
Correspondent of receiver. Receiver has an account in currency with this banking institution. <br>
Option A is formatted <br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)
</td>
</tr>
<tr>
<td>
Intermediary Institution
</td>
<td class="center">
56
</td>
<td class="center">
A
</td>
<td>
IRVTUS3N (bic)<br>
or e.g.<br>
/939400 (BSB or account)<br>
AMPBAU2SXXX (bic)
</td>
<td colspan="2">
<b>Serial payments only.</b><br>
This is the Correspondent of Creditor Bank. It holds the account in currency of the creditor bank. It is used instead of over 54a (Receiver's Correspondent) in case of a serial payment transfer.<br>
It is optional and can be provided in option A, C or D. <br>
<i>Formats C or D are rarely used and most of the time not supported by banks.</i><br>
Option A is formatted <br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)
</td>
</tr>
<tr>
<td rowspan="2">
Account With Institution
</td>
<td rowspan="2" class="center">
57
</td>
<td class="center">
A
</td>
<td>
BARCGB22XXX (bic)<br>
or e.g.<br>
//939400 (BSB)<br>
AMPBAU2SXXX (bic)
</td>
<td>
We need to parse the BIC out of it<br>
Format :<br>
[/1!a][/34x] (Party Identifier )<br>
4!a2!a2!c[3!c] (Identifier Code)
</td>
<td rowspan="2" class="lx">
<b>Serial payments only.</b><br>
Account with institution is optional and can be provided in option A, B, C or D. <br>
<i>Formats B, C are rarely used and most of the time not supported by banks.</i><br>
Option A is formatted <br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)<br>
Field 57 is used when the receiver of the SWIFT MT103 doesn’t own the beneficiary account and needs to send the Message Further. In the final MT103 on the chain, the holder of the account will be the receiver and no field 57 will be required anymore.<br>
We need to parse the BIC out of it or hash the address<br>
(priority over header)
</td>
</tr>
<tr>
<td class="center">
D
</td>
<td>
Hong Kong Banking Assoc.<br>
Avenue du Léman<br>
1204 Genève - CH<br>
Switzerland
</td>
<td>
If no BIC is available to identify the target institution, option D is used.<br>
In principle minimum 3 lines with name and address should be provided<br>
Format :<br>
[/1!a][/34x] (Party Identifier )<br>
4*35x (Name and Address)
</td>
</tr>
<tr>
<td rowspan="2">
Beneficiary Institution
</td>
<td rowspan="2" class="center">
58
</td>
<td class="center">
A
</td>
<td>
BNPAFRPP<br>
or e.g.<br>
/FR123509321...<br>
BNPAFRPP
</td>
<td>
Format<br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code)
</td>
<td rowspan="2" class="lx">
Beneficiary Institution is a mandatory field.<br>
In the Case of MT 202 and MT 202.COV, beneficiary institution is the most reliable and straightforward way to identify the eventual beneficiary institution of the founfs.
</td>
</tr>
<tr>
<td class="center">
D
</td>
<td>
BANQUE DELUBAC ET CIE<br>
16 PL SALEON<br>
TERRAS<br>
07160 LE CHEYLARD
</td>
<td>
Format<br>
[/1!a][/34x] (Party Identifier)<br>
4*35x (Name and Address)
</td>
</tr>
<tr>
<td>
Sender to Receiver Information
</td>
<td colspan="2" class="center">
72
</td>
<td>
/INS/BNPAFRPP
</td>
<td colspan="2">
This is an optional field. It takes the Format 6*35x.<br>
There can be many codes indicating additional information.<br>
INS is a code indicating that BNPAFRPP is the instructing institution. Without field 72, the receiver may not know it since that information is not provided somewhere else in the message when sender is the next bank on the routing chain and ordering institution is another bank before the instructing one.
</td>
</tr>
</table>
<a name="sec233"></a>
<h4>2.3.3 Additional notes on MT202</h4>
<p>
Some complementary notes:
</p>
<ul>
<li>
When no correspondent is used neither on sender side (Tag 53A) nor on receiver side (Tag 54A) and No reimbursement party (Tags 56a and 57a) is indicated in the SWIFT MT103 message. It means
<ul>
<li>There’s a direct account relationship, in the currency of the transfer, between the Sender and the Receiver. Money will be taken from account and credited to the beneficiary.</li>
<li>Beneficiary customer account (:59:/00012367493) is hold by the receiver.</li>
</ul>
</li>
<li>
What to do in case of 57D if we cannot find a country code? We Use SWIFT message Receiver BIC country (sometimes wrong but better than nothing, only sometimes since when 57D is used we are likely in the same country
</li>
<li>
Routing
<ul>
<li>When there is no ordering institution (Tag 52) in the SWIFT MT202 message. That means implicitly that the ordering customer is customer of the Sender.</li>
<li>When the ordering institution (Tag 52D) is provided in the MT202 SWIFT Message. This means the ordering customer is not customer of the Sender.
<ul>
<li>Either the sending institution sends the MT202 on behalf of the ordering institution in 52D. This happens when the ordering institution is a small bank that has an agreement with a major bank (sending bank) for the processing and settlement of currency transactions. The small bank can use the correspondent network of sending institution.</li>
<li>Or the Sender is a routing bank on the chain</li>
</ul>
</li>
</ul>
<li>
Field 57 is used when the receiver of the SWIFT MT103 doesn’t own the beneficiary account and needs to send the Message Further.
</li>
<li>
In the final MT103 on the chain, the holder of the account will be the receiver and no field 57 will be required anymore.
<ul>
<li>This indicates that Sender and Beneficiary customer's Bank do not have direct account relationship in the currency of the transaction (USD). Otherwise the sender would send the message directly to the beneficiary customer's Bank.</li>
</ul>
</li>
<li>
The field 58a identifies in a unique way the beneficiary institution. To be perfectly honest, it does not have the same meaning as the 57a in 103 for instance, it rather has the same beaning as 59 to identify the beneficiary, when the beneficiary is an institution and not a customer. We shall use the 58 for 202 and 202.cov to identify the eventual beneficiary institution.
</li>
</ul>
<a name="sec24"></a>
<h3>2.4 SWIFT MT202 COV Detailed Analysis</h3>
<p>
MT202 COV is a SWIFT message format for financial institution (FI) funds transfer between financial institutions. MT202's are used primarily for two purposes, bank-to-bank payments (i.e. interest payments and settlement of FX trades) and Cover Payments.
<br>
MT202 COV was implemented in 2009 to create traceability of the origination of funds (institution and account) through to the destination of funds (institution and account.) This was in response to anti-money laundering and associated banking requirements.
</p>
<p>
Prior to MT202 COV, The message format, MT202, did not include origination/destination financial institution information. Particularly for Cover Payments, where a combination of MT103 and MT202 are used to direct funds transfers to a beneficiary account, the intermediate banks in the MT202 had no ability to understand and perform risk analysis/AML/compliance checks on the funds transfer based on the original and destination of the funds. Thus, intermediate banks could be unwittingly involved in illegal transactions under new regulations.
</p>
<p>
The situation of SWIFT MT202 COV in a banking institution is as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/2bd7f8a6-2534-472b-ad7c-b4534354aceb">
<img class="centered" style="width: 750px; " alt="SWIFT MT202 COV situation" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/2bd7f8a6-2534-472b-ad7c-b4534354aceb" />
</a>
</div>
<br>
<p>
There can be two different situations:
</p>
<ul>
<li>
Either the bank is the initial sending institution, in which case it will be identified both as the sender of the SWIFT message and perhaps as the <i>Ordering Institution</i> (52a)
</li>
<li>
Or the bank is just a routing bank in the routing chain in which case it is the sender but shall be different that the <i>Ordering Institution</i> (52a)
</li>
</ul>
<a name="sec241"></a>
<h4>2.4.1 MT202 COV Introductory examples</h4>
<p>
This section presents various examples of SWIFT MT202 COV corresponding to different situations
</p>
<a name="sec2411"></a>
<h5>2.4.1.1 MT202 COV Example: Cover payment</h5>
<p>
We’ll now go through a case where the MT202 COV followed as the MT103 sent as an announce as part of the cover payment method as introduced in section [6.2.2.4 Example 4: Announce message (cover method)]. This is the case when the initial sending institution and the eventual beneficiary institution have no relationship together and decide to go through their correspondent.
<br>
The MT 202 COV in this case is the message actually carrying the funds.
</p>
<p>
In this example, the customer John Trump of bank XYZ in Switzerland wants to send 1’000’000 USD to Cowboy Corp. in Kensas City, a customer of “Kensas Credit”
<br>
Due to the nature of the transaction and their missing banking relationship together, they decide to go through correspondent banks :
</p>
<ul>
<li>
An MT103 is sent directly to the beneficiary bank, regardless of the fact they have no banking relationship together
</li>
<li>
An MT202.COV will be routed through correspondent(s) and routing banks
</li>
</ul>
<p>
We are here first having a look at the initial MT202 COV sent by bank XYZ, the sending institution:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d5be5c72-30f1-425e-810f-dafac5c4d10b">
<img class="centered" style="width: 850px; " alt="SWIFT MT202 COV Example 1" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d5be5c72-30f1-425e-810f-dafac5c4d10b" />
</a>
</div>
<br>
<p>
Details of the SWIFT MT 202 COV message:
</p>
<ul>
<li>
The field account with institution- 57a - ends up by Wells Fargo. The MT 202 chain doesn’t carry the funds any further.
</li>
<li>
But the account by Wells Fargo is the correspondent account of Kensas Credit by Well Fargo, the field Beneficiary Institution - 58a - identified the institution owning that account by Wells Fargo, hence Kensas Credit.
</li>
<li>
The field 59 identifies the beneficiary customer for which the funds are transferred, Cowbow corp, a customer of Kensas Credit.
</li>
</ul>
<p>
We shall have a look at all the MT202 COV of the chain to see how they differ from each other’s:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/1c731b22-d64d-410e-9fc9-a685e880fe5e">
<img class="centered" style="width: 850px; " alt="SWIFT MT202 COV Example 2" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/1c731b22-d64d-410e-9fc9-a685e880fe5e" />
</a>
</div>
<br>
<a name="sec242"></a>
<h3>2.4.2 MT202 COV Parsing and Data Mapping </h3>
<p>
The MT202 COV parsing details are presented in the table below. Only most essential fields are discussed.
</p>
<table class="allborder">
<tr style="background: #CCCCCC; font-weight: bold;">
<td rowspan="2">
Meaning
</td>
<td colspan="2">
SWIFT
</td>
<td rowspan="2">
Example
</td>
<td rowspan="2" colspan="2">
Comment
</td>
</tr>
<tr style="background: #CCCCCC; font-weight: bold;">
<td>Field</td>
<td>Variant</td>
</tr>
<tr>
<td>
AppID
</td>
<td colspan="2">
Block1/ApplId
</td>
<td>
F
</td>
<td colspan="2">
The Application Identifier identifies the application within which the message is being sent or received. The available options are: F = FIN , A = GPA, etc.
<br>
These values are automatically assigned by the SWIFT system and the user's CBT.
</td>
</tr>
<tr>
<td>
ServiceID
</td>
<td colspan="2">
Block1/Servid
</td>
<td>
01
</td>
<td colspan="2">
The Service Identifier consists of two numeric characters. It identifies the type of data that is being sent or received and, in doing so, the type of the following message
</td>
</tr>
<tr>
<td>
Sender
(Sending bank / BIC)
</td>
<td colspan="2">
Block1/
LTaddrBlk1 (I)
<br>
Or
<br>
Block2/
LTaddrBlk2 (O)
</td>
<td>
SGOBFRPP
</td>
<td colspan="2">
Sender BIC appears in header block (Block 1) in the MT202 Input and in the application block (Block 2) in the MT202 Output
<br>
(Input and output related to the SWIFT network, not the bank). Need to use field Block2 / Inoutind to find out
</td>
</tr>
<tr>
<td>
Message Type
</td>
<td colspan="2">
Block2/
Msgtype
</td>
<td>
202
</td>
<td colspan="2">
SWIFT Message Type = MT 202
</td>
</tr>
<tr>
<td>
Validation Flag
</td>
<td colspan="2">
Block3/Tag119
</td>
<td>
COV (if MT202 is a cover payment)
</td>
<td colspan="2">
This validation flag is provided the user block (Block 3) and transported end-to-end. It indicates that the message is a MT202 Cover.
</td>
</tr>
<tr>
<td>
Receiver
(Receiving Bank / BIC)
</td>
<td colspan="2">
Block2/
LTaddrBlk2(I)
<br>
Or
<br>
Block1/
LTaddrBlk1 (O)
</td>
<td>
RBOSGB2L
</td>
<td colspan="2">
The Receiver BIC appears in header block (Block1) in the MT292 Output and in the application block (Block 2) in the MT202 Input.
<br>
(Input and output related to the SWIFT network, not the bank). <b>Need to use field Block2 / Inoutind to find out.</b>
<br>
The receiver of the message is the eventual beneficiary only if no field 57 says otherwise.
</td>
</tr>
<tr>
<td>
Unique End-to-end Transaction Reference
</td>
<td colspan="2">
Block3/ Tag 121
</td>
<td>
b03c6901-bbed-4aa9-afdh-A5bc26d19257
</td>
<td colspan="2">
This reference is provided in the user block (Block 3) and transported end-to-end. It is mandatory in MT103 but can still be missing as well as have duplicates.
</td>
</tr>
<tr>
<td colspan="6" style="background-color: #AACCFF; font-weight: bold;">
Sequence A - General Information (Matching MT 202 format)
</td>
</tr>
<tr>
<td>
Sender's Reference
</td>
<td colspan="2" class="center">
20
</td>
<td>
ORDERREF1234
</td>
<td colspan="2">
This field is mandatory and of format 16x. It is a reference assigned by the Sender to unambiguously identify the message.
</td>
</tr>
<tr>
<td>
Related reference
</td>
<td colspan="2" class="center">
21
</td>
<td>
123456789ABCDEF
</td>
<td colspan="2">
This field is mandatory and of format 16x.
</td>
</tr>
<tr>
<td>
Sender Msg. Sending Timestamp
</td>
<td colspan="2">
(O) Block2 / ‘
Intime + Indate
<br>
(I) Sys.time()
</td>
<td>
1538070522
</td>
<td colspan="2">
(O) = Output only : SWIFT timestamp for an Output message (HHMMYYMMDD)
<br> or local date/time for an Input Message.
</td>
</tr>
<tr>
<td>
In / Out-put flag
</td>
<td colspan="2">
Block2 / Inoutind
</td>
<td>
I
</td>
<td colspan="2">
Single letter ‘I’ or ‘O’
</td>
</tr>
<tr>
<td>
Value Date / currency / interbank settled amount
</td>
<td class="center">
32
</td>
<td class="center">
A
</td>
<td>
180816USD2325,
</td>
<td colspan="2">
It is mandatory and of format 6!n3!a15d (Date)(Currency)(Amount). <br>
Note the trailing coma (i.e. decimal part is not mandatory if 0)
</td>
</tr>
<tr>
<td rowspan="2">
Ordering institution
</td>
<td rowspan="2" class="center">
52
</td>
<td class="center">
A
</td>
<td>
BNPAFRPP<br>
or e.g.<br>
/FR1235093212...<br>
BNPAFRPP
</td>
<td>
Format<br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code)<br>
</td>
<td rowspan="2" class="lx">
Ordering institution is optional and can be provided in two options A (usual) and D (less common). <br>
The sender populate this field to indicate that the initial instruction comes from another institution (ordering institution). <br>
The ordering institution remains constant in the message chain. <br>
(priority over header)
</td>
</tr>
<tr>
<td class="center">
D
</td>
<td>
BANQUE DELUBAC ET CIE<br>
16 PL SALEON<br>
TERRAS <br>
07160 LE CHEYLARD
</td>
<td>
Format<br>
[/1!a][/34x] (Party Identifier)<br>
4*35x (Name and Address)
</td>
</tr>
<tr>
<td rowspan="2">
Sender's correspondent
</td>
<td rowspan="2" class="center">
53
</td>
<td class="center">
A
</td>
<td>
PNBPUS3N<br>
or e.g.<br>
/12345678901<br>
PNBPUS3N
</td>
<td colspan="2">
<b>Cover payments only</b><br>
Correspondent of sender. Sender has an account in Currency with this banking institution. <br>
Option A is formatted <br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)
</td>
</tr>
<tr>
<td class="center">
B
</td>
<td>
/12345678901
</td>
<td colspan="2">
<b>Caution : Serial and cover payments.</b><br>
Field 53B indicates the account number of the Sender, serviced by the Receiver, which is to be used for reimbursement (debit) in the transfer. This the account of the sender by the receiver (VOSTRO)<br>
Option B Format is:<br>
[/1!a][/34x] (Party Identifier) <br>
[35x] (Location)<br>
<i>The field is optional but In practice, the account number is almost always provided.</i><br>
</td>
</tr>
<tr>
<td>
Receiver’s correspondent
</td>
<td class="center">
54
</td>
<td class="center">
A
</td>
<td>
IRVTUS3N<br>
or e.g.<br>
/9876412-1234/123<br>
IRVTUS3N
</td>
<td colspan="2">
<b>Cover payments only.</b><br>
Correspondent of receiver. Receiver has an account in currency with this banking institution. <br>
Option A is formatted <br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)
</td>
</tr>
<tr>
<td>
Intermediary Institution
</td>
<td class="center">
56
</td>
<td class="center">
A
</td>
<td>
IRVTUS3N (bic)<br>
or e.g.<br>
/939400 (BSB or account)<br>
AMPBAU2SXXX (bic)
</td>
<td colspan="2">
<b>Serial payments only.</b><br>
This is the Correspondent of Creditor Bank. It holds the account in currency of the creditor bank. It is used instead of over 54a (Receiver's Correspondent) in case of a serial payment transfer.<br>
It is optional and can be provided in option A, C or D. <br>
<i>Formats C or D are rarely used and most of the time not supported by banks.</i><br>
Option A is formatted <br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)
</td>
</tr>
<tr>
<td rowspan="2">
Account With Institution
</td>
<td rowspan="2" class="center">
57
</td>
<td class="center">
A
</td>
<td>
BARCGB22XXX (bic)<br>
or e.g.<br>
//939400 (BSB)<br>
AMPBAU2SXXX (bic)
</td>
<td>
We need to parse the BIC out of it<br>
Format :<br>
[/1!a][/34x] (Party Identifier )<br>
4!a2!a2!c[3!c] (Identifier Code)
</td>
<td rowspan="2" class="lx">
<b>Serial payments only.</b><br>
Account with institution is optional and can be provided in option A, B, C or D. <br>
<i>Formats B, C are rarely used and most of the time not supported by banks.</i><br>
Option A is formatted <br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)<br>
Field 57 is used when the receiver of the SWIFT MT103 doesn’t own the beneficiary account and needs to send the Message Further. In the final MT103 on the chain, the holder of the account will be the receiver and no field 57 will be required anymore.<br>
We need to parse the BIC out of it or hash the address<br>
(priority over header)
</td>
</tr>
<tr>
<td class="center">
D
</td>
<td>
Hong Kong Banking Assoc.<br>
Avenue du Léman<br>
1204 Genève - CH<br>
Switzerland
</td>
<td>
If no BIC is available to identify the target institution, option D is used.<br>
In principle minimum 3 lines with name and address should be provided<br>
Format :<br>
[/1!a][/34x] (Party Identifier )<br>
4*35x (Name and Address)
</td>
</tr>
<tr>
<td rowspan="2">
Beneficiary Institution
</td>
<td rowspan="2" class="center">
58
</td>
<td class="center">
A
</td>
<td>
BNPAFRPP<br>
or e.g.<br>
/FR123509321...<br>
BNPAFRPP
</td>
<td>
Format<br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code)
</td>
<td rowspan="2" class="lx">
Beneficiary Institution is a mandatory field.<br>
In the Case of MT 202 and MT 202.COV, beneficiary institution is the most reliable and straightforward way to identify the eventual beneficiary institution of the founfs.
</td>
</tr>
<tr>
<td class="center">
D
</td>
<td>
BANQUE DELUBAC ET CIE<br>
16 PL SALEON<br>
TERRAS<br>
07160 LE CHEYLARD
</td>
<td>
Format<br>
[/1!a][/34x] (Party Identifier)<br>
4*35x (Name and Address)
</td>
</tr>
<tr>
<td>
Sender to Receiver Information
</td>
<td colspan="2" class="center">
72
</td>
<td>
/INS/BNPAFRPP
</td>
<td colspan="2">
This is an optional field. It takes the Format 6*35x.<br>
There can be many codes indicating additional information.<br>
INS is a code indicating that BNPAFRPP is the instructing institution. Without field 72, the receiver may not know it since that information is not provided somewhere else in the message when sender is the next bank on the routing chain and ordering institution is another bank before the instructing one.
</td>
</tr>
<tr>
<td colspan="6" style="background-color: #AACCFF; font-weight: bold;">
Sequence B - Underlying Customer Credit Transfer Detail
</td>
</tr>
<tr>
<td>
Currency / Instructed Amount
</td>
<td class="center">
33
</td>
<td class="center">
B
</td>
<td>
USD2350,
</td>
<td colspan="2">
Normally optional in the standard. It may be provided for instance because the sender has taken fees. See field 71F below.<br>
Format 3!a15d (Currency)(Amount)
</td>
</tr>
<tr>
<td rowspan="3">
Ordering Customer
</td>
<td rowspan="3" class="center">
50
</td>
<td class="center">
A
</td>
<td>
/DE3750070010...<br>
DEUTDEFF
</td>
<td>
Line 1 (subfield Party Identifier)<br>
/34x (account)<br>
Line 2 (subfield bank)<br>
4!a2!a2!c[3!c] (Identifier Code)
</td>
<td rowspan="3" class="lx">
The field ordering customer is mandatory. In can be given either in the message details (here) or per transaction in the repeating sequence.<br>
The ordering customer is customer of the sender only if there is no field 52. The ordering customer remains constant in the message chain.
</td>
</tr>
<tr>
<td class="center">
F
</td>
<td>
/DE207008000...<br>
1/Essilor International<br>
2/147 Rue de Paris<br>
3/FR/Charenton-le-Pont, 94220
</td>
<td>
Line 1 (subfield Party Identifier)<br>
/34x (Account)<br>
Lines 2-5 : (Number/Name and Address)<br>
1!n/33x (Number)(Details)
</td>
</tr>
<tr>
<td class="center">
K
</td>
<td>
/CH5704835098...<br>
GALLMAN COMPANY GMBH<br>
RAEMISTRASSE, 71<br>
8006 ZURICH<br>
SWITZERLAND
</td>
<td>
Line 1 : (subfield party identified)<br>
/34x (Account)<br>
Line 2-5 (subfield Address)<br>
4*35x (Name and Address)
</td>
</tr>
<tr>
<td rowspan="3">
Beneficiary
</td>
<td rowspan="3" class="center">
59
</td>
<td class="center">
(no letter)
</td>
<td>
/26351-38947<br>
Company One<br>
CITY STREET 50<br>
LONDON, UK
</td>
<td>
Line 1 : <br>
[/34x] (Account) (IBAN format or else)<br>
Line 2-5:<br>
4*35x (Name and Address)
</td>
<td rowspan="3" class="lx">
Beneficiary customer information is Mandatory. <br>
The beneficiary remains constant in the message chain.
</td>
</tr>
<tr>
<td class="center">
A
</td>
<td>
PNBPUS3N<br>
or e.g.<br>
/12345678901<br>
PNBPUS3N
</td>
<td>
Option A is formatted <br>
[/1!a][/34x] (Party Identifier)<br>
4!a2!a2!c[3!c] (Identifier Code / BIC)
</td>
</tr>
<tr>
<td class="center">
F
</td>
<td>
/10078074<br>
1/Company One<br>
2/CITY STREET 50<br>
3/GB/LONDON
</td>
<td>
Line 1 (subfield Party Identifier)<br>
[/34x] (Account)<br>
Lines 2-5 : (Number/Name and Address)<br>
4*(1!n/33x) (name and address)
</td>
</tr>
<tr>
<td>
Remittance Information
</td>
<td colspan="2" class="center">
70
</td>
<td>
/INV/18042-090715
</td>
<td colspan="2">
Remittance information is optional and provided in format 4*35x if available. Up to 4 lines of up to 35 X characters each.<br>
Usually the remittance information is generated by the beneficiary and sent to the ordering customer (or debtor). The beneficiary requests the debtor to provide it the payment message, so that the beneficiary can easily reconcile the payment with an invoice for instance.
</td>
</tr>
<tr>
<td>
Details of Charges
</td>
<td class="center">
71
</td>
<td class="center">
A
</td>
<td>
OUR
</td>
<td colspan="2">
It is mandatory and of format 3!a. It can take 3 values: BEN, OUR and SHA.
<br>
<ul>
<li>
OUR means charges are to be borne by the ordering customer.
</li>
<li>
SHA means charges are shared between Ordering and beneficiary customers.
</li>
<li>
BEN means charges are to be borne by the beneficiary
</li>
</ul>
</td>
</tr>
<tr>
<td>
Sender's charges
</td>
<td class="center">
71
</td>
<td class="center">
F
</td>
<td>
EUR2,50
</td>
<td colspan="2">
Optional. When 71A is BEN (or SHA), 71G contains amount of the charges due, which have been deducted from the interbank settlement amount.<br>
Interbank settled amount = Instructed amount - Sender's charges.<br>
Format 3!a15d <br>
<b>Caution: there can be several different 71F in a same MT202 COV.</b>
</td>
</tr>
<tr>
<td>
Receiver's charges
</td>
<td class="center">
71
</td>
<td class="center">
G
</td>
<td>
EUR2,50
</td>
<td colspan="2">
Optional. When 71A is OUR (or SHA), 71G contains amount of the charges due, which have been prepaid and included in the interbank settlement amount.<br>
Format 3!a15d<br>
<b>Caution : there can be several different 71G in a same MT202 COV. </b>
</td>
</tr>
<tr>
<td>
Sender to Receiver Information
</td>
<td colspan="2" class="center">
72
</td>
<td>
/INS/BNPAFRPP
</td>
<td colspan="2">
This is an optional field. It takes the Format 6*35x.<br>
There can be many codes indicating additional information. <br>
INS is a code indicating that BNPAFRPP is the instructing institution. Without field 72, the receiver may not know it since that information is not provided somewhere else in the message when sender is the next bank on the routing chain and ordering institution is another bank before the instructing one.
</td>
</tr>
</table>
<a name="sec243"></a>
<h4>2.4.3 Additional notes on MT202 COV</h4>
<p>
Some complementary notes:
</p>
<ul>
<li>
The option field 53B (only variant B) is really only used to indicate which account at the correspondent (receiver) should be debited.
</li>
<li>
Fields 51A seems to be not supported by most banks (at least all I found such as UBS, etc.)
</li>
<li>
When no correspondent is used neither on sender side (Tag 53A) nor on receiver side (Tag 54A) and No reimbursement party (Tags 56a and 57a) is indicated in the SWIFT MT202 message. It means
<ul>
<li>there is a direct account relationship, in the currency of the transfer, between the Sender and the Receiver. Money will be taken from account and credited to the beneficiary.</li>
<li>Beneficiary customer account is hold by the receiver.</li>
</ul>
</li>
<li>
What to do in case of 57D if we cannot find a country code? We Use SWIFT message Receiver BIC country (sometimes wrong but better than nothing, only sometimes since when 57D is used we are likely in the same country
</li>
<li>
Routing
<ul>
<li>When there is no ordering institution (Tag 52) in the SWIFT MT202 message. That means implicitly that the ordering customer is customer of the Sender.</li>
<li>
When the ordering institution (Tag 52D) is provided in the MT202 SWIFT Message. This means the ordering customer is not customer of the Sender.
<ul>
<li>Either the sending institution sends the MT103 on behalf of the ordering institution in 52D. This happens when the ordering institution is a small bank that has an agreement with a major bank (sending bank) for the processing and settlement of currency transactions. The small bank can use the correspondent network of sending institution.</li>
<li>Or the Sender is a routing bank on the chain</li>
</li>
</ul>
</li>
</ul>
</li>
<li>
Field 57 is used when the receiver of the SWIFT MT202 doesn’t own the beneficiary account and needs to send the Message Further.
<br>
In the final MT103 on the chain, the holder of the account will be the receiver and no field 57 will be required anymore.
<ul>
<li>This indicates that Sender and Beneficiary customer's Bank do not have direct account relationship in the currency of the transaction (USD). Otherwise the sender would send the message directly to the beneficiary customer's Bank.
</li>
</ul>
</li>
<li>
Sender’s reference is new for every message in the routing chain, but end-to-end reference remains constant.
</li>
<li>
The field 58a identifies in a unique way the beneficiary institution. To be perfectly honest, it does not have the same meaning as the 57a in 103 for instance, it rather has the same meaning as 59 to identify the beneficiary, when the beneficiary is an institution and not a customer.
<br>
We shall use the 58 for 202 and 202.cov to identify the eventual beneficiary institution.
</li>
</ul>
<a name="sec3"></a>
<h2>3. Conclusion</h2>
<p>
The above is a collection of the most essential information we gathered at NetGuardians when implementing SWIFT parsing to monitor payments, specifically cross border payments. There are more Message Types relevant when it comes to monitoring payments than those mentioned in this article (for instance MT 205, MT 102, MT 203, etc.) but there usage is rarer.
<br>
In addition, the listing of fields provided in this article is far from complete and other fields of the SWIFT messages, especially from the various headers, may be interesting for your own business. I let the reader refer to the SWIFT MT specifications.
<br>
Last but not least, with the coming mandatory transition to XML format (SWIFT MX), I may well need to write a new version of this article pretty soon :-)
</p>
<p>
For the sake of exhaustivity, I wanted to complete this article by giving a very simplified view of how SWIFT kicks-in in a Banking Information System:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/f035bd9d-e1de-49f9-8d79-0cd3825b869c">
<img class="centered" style="width: 750px; " alt="SWIFT Architecture in Bank" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/f035bd9d-e1de-49f9-8d79-0cd3825b869c" />
</a>
</div>
<br>
<p>
Most importantly, the SWIFT interface is connected to the payment HUB of the banking institution. But interestingly, it is also used as an input channel for customers, since the later may send <i>Transfer Requests</i> such as MT101 to their banking institution using SWIFT.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/ai-what-do-we-do
AI - what do we do differently at NetGuardians ?
Jerome Kehrli
2019-02-18T02:42:15-05:00
2019-02-18T02:44:03-05:00
<p>
The world of fraud prevention in banking institutions has always been largely based on rules.
<br>
Bankers and their engineers were integrating rules engines on the banking information system to prevent or detect most common fraud patterns.
<br>
And for quite a long time, this was sufficient.
</p>
<p>
But today we are experiencing a change of society, a new industrial revolution.
<br>
Today, following the first iPhone and the later mobile internet explosion, people are interconnected all the time, everywhere and for all kind of use.
<br>
This is the digital era and the digitization of means and behaviours forces corporations to transform their business model.
</p>
<p>
As a consequence, banking institutions are going massively online and digital first. Both the bank users and customers have evolved their behaviours with the new means offered by the digital era.
<br>
And the problem is:
<br>
How do you want to protect your customer's assets with rules at a time when, for instance, people connect to their swiss ebanking platform from New York to pay for a holiday house rental in Morocco? How would you want to define rules to detect frauds when there are almost as many different behaviours as there are customers?
</p>
<div id="player_2" style="max-width: 100%;"></div>
<script>
player = new YT.Player('player_2', {
height: '360',
width: '640',
videoId: 'B9dY63KAG3I'
});
</script>
<p> </p>
<p>
The world of fraud prevention in banking institutions has always been largely based on rules.
<br>
Bankers and their engineers were integrating rules engines on the banking information system to prevent or detect most common fraud patterns.
<br>
And for quite a long time, this was sufficient.
</p>
<p>
But today we are experiencing a change of society, a new industrial revolution.
<br>
Today, following the first iPhone and the later mobile internet explosion, people are interconnected all the time, everywhere and for all kind of use.
<br>
This is the digital era and the digitization of means and behaviours forces corporations to transform their business model.
</p>
<p>
As a consequence, banking institutions are going massively online and digital first. Both the bank users and customers have evolved their behaviours with the new means offered by the digital era.
<br>
And the problem is:
<br>
How do you want to protect your customer's assets with rules at a time when, for instance, people connect to their swiss ebanking platform from New York to pay for a holiday house rental in Morocco? How would you want to define rules to detect frauds when there are almost as many different behaviours as there are customers?
</p>
<div id="player_148374" style="max-width: 100%;"></div>
<script>
player = new YT.Player('player_148374', {
height: '360',
width: '640',
videoId: 'B9dY63KAG3I'
});
</script>
<p> </p>
<p>
At NetGuardians, we prevent fraud using a completely different approach. We use Artificial Intelligence to monitor financial transactions and user behaviour in real time and detect suspicious transactions or activities.
<br>
With our Big Data Analytics Platform - NG|Screener - the machine analyzes the past transactions of the customers to understand their transactional behaviour, as well as past activities of users on the banking information system.
<br>
The machine is able to analyze a very important depth of history in real time, capturing what customers and users usually do in so called Dynamic Profiles.
<br>
Then, whenever a transaction is input on the system, the Artificial Intelligence is able to compare that specific transaction against the customer profile and compute a risk score for it. If the risk score is sufficiently high, the machine will decide to block the transaction and qualify it for further investigation by the bank.
</p>
<p>
Using advanced machine learning techniques, we are able to have a very broad spectrum of detection, while minimizing wrong alerts, these famous false positives, to an unprecedented low ratio.
Instead of focusing on identifying fraudsters, at NetGuardians we focus on understanding user and customer behaviours and habits. In order to find frauds, we don't look only for Fraud. Our approach is to detect and block every transaction that is simply too unusual and risky for the banking institution to afford letting it go out without a further investigation. And it turns out the frauds are simply always part of this set of risky transactions.
<br>
Our unique combination of unsupervised and supervised approaches makes it possible to minimize false positives, while still being able to detect fraud patterns never encountered before.
In the world of Fraud Prevention Solutions, this is the Holy Grail, being able to use models that can detect what has never been encountered before while at the same time keeping the amount of alerts very low. This is hardly seen in other AI solutions on the market today.
</p>
<p>
Over the time, we have been able to develop our Artificial Intelligence further to make it smarter and smarter. For instance, pretty soon we started to compare transactions against different background sets, the past customer or user activities of course, but also the banking institution past activities as a whole or even the customer or user peer group.
<br>
The peer groups are also built using Machine Learning to analyze customers and users past activities, this time to group them together, and achieve better and more accurate scoring of risky transactions or other activities.
</p>
<p>
Today, our unique Artificial Intelligence platform is able to use a dozen of different Analytics and Machine Learning approaches going much beyond solely transaction scoring. For instance, we are able to qualify the legitimacy of a specific interaction on the ebanking platform, or to monitor PSD2 providers activity, or even to do frequency and timing anomalies detection on card transactions.
</p>
<p>
At NetGuardians, we use AI to detect anomalies and prevent fraud by providing the experts and investigators within the bank with the tools aimed at making them more efficient than has ever been possible before.
<br>
We enhance the human investigation and analytics process, but we don't supplant it.
<br>
This is called Augmented Intelligence, the core of what we do at NetGuardians.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/interview-about-netguardians-and-fighting
Interview about NetGuardians and fighting fraud in the digital era
Jerome Kehrli
2019-02-04T06:10:18-05:00
2019-02-05T04:38:50-05:00
<!-- Interview about NetGuardians and fighting fraud in the digital era. -->
<p>
<b><i>The below is an extract from an interview I ran in February 2019 during the <a href="https://forward-sme.epfl.ch/">EPFL Forward event</a>.</i></b>
</p>
<p>
NetGuardians is a Swiss Software Publisher based in Yverdon-les-bains that edits a Big Data Analytics Solution deployed Financial Institution for one key use case: fighting financial crime and preventing banking Fraud.
<br>
Banking fraud is meant in the broad sense here: both internally and externally.
<br>
Internal fraud is when employees misappropriate funds under management and external fraud is when cyber-criminals compromise ebanking applications, mobile devices used for payment or credit cards.
</p>
<p>
In the digital age, the means of fraudsters and cyber-criminals have drastically increased.
</p>
<p>
Cyber-criminals have become industrialized, professionalized and organized. The same technology they use against banks is also what gives us the means to protect banks
</p>
<p>
At NetGuardians we deploy an Artificial Intelligence that monitors on a large scale, in depth and in real time all activities of users, employees of the bank, but also those of its customers, to detect anomalies.
<br>
We prevent bank fraud and fight financial crime by detecting and blocking all suspicious activity in real time.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/e870fa77-83dd-40fb-a0f8-ee508d2b653a">
<img class="centered" style="width: 600px; " alt="Digital Banking" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/e870fa77-83dd-40fb-a0f8-ee508d2b653a" />
</a>
</div>
<br>
<p>
<b><i>Jérôme Kehrli, how did you manage to convince a sector that is, in essence, very traditional, to trust you with your digital tools to fight against fraud?
<br>
Two different worlds, two languages, two visions?
</i></b>
</p>
<p>
The situation of the banks is a bit peculiar, the digitization and with it the evolution of the means and the behaviours of the customers in the digital age, was at the same time both a traumatic and a formidable solution.
</p>
<p>
The digital revolution was a traumatic because the banks, which by their very nature are very conservative, especially in Switzerland with our very strong private banking culture, were not prepared for the need to profoundly transform the customer experience of the banking world: to meet the customer where he is, on his channels, with mobile banking, this culture of all and everything immediately, with instant payments, the opening of the information system, with the explosion of the External Asset Managers model and external service providers with the PSD2 European standard, etc.
</p>
<p>
The digital revolution has imposed these changes, sometimes brutally, in banks and it is the source of a tremendous increase of the attack surface of banks.
</p>
<p>
But this same technology that spawned the digital revolution has proved to be the solution too.
<br>
Technology has made it possible to build digital banking applications that provide all of the bank's services on a mobile device.
<br>
Technology has made it possible to implement innovative solutions that secure the information system and protect client funds.
</p>
<p>
And in this perspective, Artificial Intelligence is really a sort of panacea: robot advisory, chatbots, personalization of financial advice and especially, especially the fight against financial crime: banking fraud and money laundering
</p>
<p>
In the end, if five years ago our solutions seemed somewhat avant-garde, not to say futuristic and sometimes aroused a bit of skepticism, today the banks are aware of the digital urgency and it is the bankers themselves who eagerly seek our solutions.
</p>
<p>
<b><i>
You support the digital shift of the banking sector.
<br>
Do banks sometimes have to change their way of operating, their habits, to be able to use your technologies?
<br>
(Do you have to prepare them to work with you?)
</i></b>
</p>
<p>
So of course the digital revolution profoundly transforms not only the business model but also the corporate culture, its tools, and so on.
</p>
<p>
At NetGuardians we have a very concrete example.
</p>
<p>
Before the use of Artificial Intelligence, banks protected themselves with rules engines. Hundreds of rules were deployed on the information system to enforce security policies or detect the most obvious violations.
<br>
The advantage with rules was that a violation was very easy to understand. A violation of a compliance rule reported in a clear and accurate audit report was easy to understand and so was the response.
<br>
The disadvantage, however, was that the rules were a poor protection against financial crime and that's why fraud has exploded over the decade.
</p>
<p>
Today with artificial intelligence, the level of protection is excellent and without comparison with the era of the rules.
<br>
But the disadvantage of artificial intelligence is that accurately understanding a decision of the machine is much more difficult.
</p>
<p>
At NetGuardians, we develop with our algorithms a Forensic analysis application that allows bankers to understand the operation of the machine by presenting the context of the decision.
<br>
This forensic analysis application, which presents the results of our algorithms, is essential and almost as important as our algorithms themselves.
</p>
<p>
This is a powerful application but requires a grip.
</p>
<p>
Tom Cruise in Minority Report who handles a data discovery application playing an orchestra conductor, it's easy in Hollywood, but it's not in reality.
<br>
In reality, we provide initial training to our users and then regular updates.
</p>
<p>
In the end, a data analysis and forensic application is not Microsoft Word. Our success is to make such an application accessible to everyone, but not without a little help.
<br>
In conclusion i would say that the culture transformation end the evolution of the tools do require some training and special care.
</p>
<p>
<b><i>
In general, what should a company prepare for, before making a digital shift?
</i></b>
</p>
<p>
In the digital age, many companies must transform their business model or disappear. Some services become obsolete, some new necessities appear.
<br>
We can mention Uber of course but also NetFlix, Booking, eBookers, etc.
</p>
<p>
For the majority of the industrial base, the digitalization of products and services is an absolute necessity, a question of survival.
</p>
<p>
Successful process and business model transformation often requires a transformation of the very culture of the company, down toits identity:
<br>
Among other things one could mention the following requirements:
</p>
<ul>
<li>scaling agility from product development to the whole company level</li>
<li>involving digital natives to identify and design digital services</li>
<li>realizing the urgency or if necessary create a sense of urgency</li>
<li>understanding the scale of the challenge and the necessary transformation. Some say <i>"if it does not hurt, it is not digital transformation"</i></li>
</ul>
<p>
In summary I would say that a company is "mature" for digitalization if it is inspired by the digitalization of our daily life to adapt its products and services AND if it has the ability to execute its ideas.
<br>
Ideas without the ability to execute leads to mess, the ability to execute without the ideas leads to the status quo.
</p>
<p>
From there I would say that a company must prepare itself on these two dimensions, bring itself the conditions and resources required to identify and to design its digital products and those required to realize them.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/artificial-intelligence-for-banking-fraud
Artificial intelligence for banking fraud prevention in the digital era
Jerome Kehrli
2018-07-04T15:34:04-04:00
2018-07-04T15:34:04-04:00
<p>
The digitalization with its changes of means and behaviours and the induced society and industrial evolution is putting increasingly more pressure on banks.
<br>
Just as if regulatory pressure and financial crisis weren't enough, banking institutions have realized that they need to transform the way they run their business to attract new customers and retain their existing ones.
<br>
I detailed already this very topic in a former article on this blog: <a href="https://www.niceideas.ch/roller2/badtrash/entry/the-digitalization-challenge-and-opportunities">The Digitalization - Challenge and opportunities for financial institutions</a>.
</p>
<p>
In this regards, <b>Artificial Intelligence</b> provides tremendous opportunities and very interesting initiatives start to emerge in the big banking institutions.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/10fc4c4f-ca5d-44a0-9711-8c90c91a1ba4">
<img class="centered" style="width: 600px; " alt="Artificial Intelligence in the Financial Industry" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/10fc4c4f-ca5d-44a0-9711-8c90c91a1ba4" />
</a>
</div>
<br>
<p>
In this article I intend to present these three ways along with a few examples and detail what we do at NetGuardians in this regards.
</p>
<!-- Artificial intelligence for banking fraud prevention in the digital era -->
<p>
The digitalization with its changes of means and behaviours and the induced society and industrial evolution is putting increasingly more pressure on banks.
<br>
Just as if regulatory pressure and financial crisis weren't enough, banking institutions have realized that they need to transform the way they run their business to attract new customers and retain their existing ones.
<br>
I detailed already this very topic in a former article on this blog: <a href="https://www.niceideas.ch/roller2/badtrash/entry/the-digitalization-challenge-and-opportunities">The Digitalization - Challenge and opportunities for financial institutions</a>.
</p>
<p>
In this regards, <b>Artificial Intelligence</b> provides tremendous opportunities and very interesting initiatives start to emerge in the big banking institutions.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/10fc4c4f-ca5d-44a0-9711-8c90c91a1ba4">
<img class="centered" style="width: 600px; " alt="Artificial Intelligence in the Financial Industry" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/10fc4c4f-ca5d-44a0-9711-8c90c91a1ba4" />
</a>
</div>
<br>
<p>
In this article I intend to present these three ways along with a few examples and detail what we do at NetGuardians in this regards.
</p>
<h2>Digital urgency</h2>
<p>
Since 10 years, since the first Iphone, the banking business is under intensive transformation, like the whole society. The iphone, then all the next generations of smartphones, has allowed the always and everywhere interconnection of everyone, 4 billion people today, more tomorrow. But even more than this interconnection, the real revolution was the user experience offered by these new devices, providing access to vital services at one finger touch away.
<br>
The current generations, millennials, and the rising Z generation, these young people born with a smartphone, do not share the values than their ancestors. These new generations are characterized by their need of absolute, of immediacy, everything and immediately, of individualization, <i>all about me, myself and I</i>, and of universal service, <i>where I want, when I want and especially how I want</i>.
</p>
<p>
As a result, banks, if they want to retain their customers, or seduce these young active people who are coming on the market, must adapt, transform, and digitalize their businesses. This topic is far from new, and the amazing slap given to the banks by the emergence of the fintechs and their cannibalization of the banking business was way enough to create the sense of urgency required to trigger the transformation of banks.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/6171b9a1-5ed9-4617-b960-ad0340ce0bb1">
<img class="centered" style="width: 900px; " alt="Digital Transformation" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/6171b9a1-5ed9-4617-b960-ad0340ce0bb1" />
</a>
</div>
<br>
<p>
Today all banks are adapting, developing more personalized and massively online services, meeting the new active force on its privileged channels, mobile first, but also social networks, youtube, etc
<br>
One might want to read my previous article on this very topic: <a href="https://www.niceideas.ch/roller2/badtrash/entry/the-digitalization-challenge-and-opportunities">The Digitalization - Challenge and opportunities for financial institutions</a>
</p>
<p>
In this race for digital, made vital by the crumbling margins, new technologies are both the source of the challenges and their solution. In this perspective, Artificial Intelligence and advanced algorithms are the next step, the way to scale and have the potential to make banks more innovative and smarter.
To meet the challenges of digital and the demands of this population with new behaviors and uses, but also to take advantage of opportunities related to digitalization, AI proves to be a panacea.
</p>
<p>
At every level, from financial research to fraud prevention, AI can do better, faster, farther, stronger.
<br>
Today an lot of highly innovative initiatives, taking advantage of the latest advances in artificial intelligence, emerge in the big banks, the most able to consent to the investments needed for the transformation.
</p>
<p>
These initiatives are mainly around 3 axes:
</p>
<h2>1. The customer experience</h2>
<p>
The physical presence vanishes, the contact with a human also and is replaced advantageously, at least for these digital generations, by a computer system: chatbot, personal banking assistant, voice assistant, etc.
</p>
<p>
Bank of America, for instance, has deployed a virtual assistant, Erica, that interacts with her customers by voice chat or chat and is responsible for answering customers' questions, but who is also able to provide financial advices, recommendations of investment, or to carry out the simplest banking operations such as payments, fund transfers, etc.
These new banking channels are available constantly, everywhere and in various forms, "<i>When I want, where I want and especially as I want.</i>"
</p>
<h2>2. Advanced analytics, operational efficiency and service customization</h2>
<p>
New technologies and machine learning techniques are able to substitute to humans on many analytical tasks in an advantageous way, be it financial research, investment optimization or customer profiling for the personalization of advisory.
</p>
<p>
UBS, for instance, has developed virtual research agents able to performing investment research tasks, from analyzing market data to company valuation at a level comparable to human analysts but much faster.
<br>
RBS has implemented a robot for the loan evaluation process able to approve a credit in 45 minutes, instead of several days as previously required.
</p>
<h2>3. Prevention of bank ingfraud and anti-money-laundering</h2>
<p>
The new channels to which banks must subscribe, the new usages of customers and the acceleration of business cause an increase of the attack surface of banks. These uses especially, and the behaviors of these digital generations, <i>share everything with everyone</i>, <i>all about myself</i>, my life online, social networks where everything is shared, facilitate all forms of attacks, from social engineering to theft of eBanking session.
</p>
<p>
Finally, the new technologies, on one side, since they offer new means to cyber-criminals, and the economic context on the other side, which makes the criminal enterprise attractive to a number of engineers and other qualified computer scientists, cause an explosion of fraud cases, increasingly external.
</p>
<h2>AI against banking fraud, a bit of history</h2>
<p>
In the early 2000s, the detection of banking fraud relies mostly on internal control and auditing. The effectiveness of these approaches is pretty low because of their inherent limitations. Working by sampling, internal control and audit leave a lot of frauds pass through the cracks. Some additional securities are implemented within the bank's operational information system, but here too their efficiency is quite relative.
<br>
At that time, the subprime crisis and the southern european countries sovereign debt crisis have not yet occurred, the margins are wide, people trust the banks and overall, they feel safe. The fight against fraud is not perceived as a priority.
</p>
<p>
In the second half of the 2000s, the maturity of cyber-criminals, their organization and the complexity of their attacks explode, multiplying the losses associated with fraud.
<br>
Banks react by massively deploying specific analytical systems aimed at detecting fraud. At that time, these systems are rule engines seeking behavioral patterns or pre-established and well-determined conditions or patterns in audit trails of the information system.
</p>
<p>
Today, the complexity of attacks and the means of cyber-criminals are such that these rule engines are defeated. We can mention the computer crash of the central bank of Bangladesh, where cyber-criminals, safe and untraceable, managed to steal $81 million or the Retefe worm, which despite the means deployed still manages to divert about fifty ebanking sessions every day, today, in Switzerland.
<br>
The rule engines are outdated for various reasons, including changes in usage, their multiplication and the complexity of bank customer behavior. How can the same set of rules effectively protect customers with different uses and behaviors, for example a simple saver on one side and an institutional account used to pay the suppliers of a company on another side?
</p>
<p>
Today, banks no longer have any choice and deploy large-scale AI techniques to protect their Assets Under Management and their customers.
</p>
<h2>Return of Experience, our approach at NetGuardians</h2>
<p>
In the first half of the current decade, we started at NetGuardians to develop our first AI approaches, leveraging the analytics capabilities of our Big Data platform. These approaches consist in analyzing in real time all the bank transactions, with a depth of analysis of several years, to let the machine learn the transactional behavior of both the bank's customers (external fraud) and its employees (internal fraud). With this in-depth understanding of the habits, behaviors and practices of these two populations, the machine can qualify each and every transaction as legitimate or potentially fraudulent, and, if necessary, block it before the funds have left the bank.
</p>
<p>
We then introduced other machine learning algorithms to dynamically build customers peer-groups with similar behavior, allowing us to compare an individual transaction not only with the profile of a specific customer but also with its peer group, and reducing thus the irrelevant alerts, these false-positives which must nonetheless be analyzed.
Later, we focused on broadening the vision of AI by trying to make it understand all patterns of interaction between humans, employees or customers, and the information system of the bank, by analyzing not only transactions but also all other types of interaction.
</p>
<p>
Today we are able to effectively block each suspicious transaction or activity, while drastically reducing the number of cases to be analyzed, these infamous false positives, and also the time required for the investigation of a case by the teams of the bank.
</p>
All of this is explained in details in yet another article on this blog: <a href="https://www.niceideas.ch/roller2/badtrash/entry/artificial-intelligence-for-fraud-prevention">Artificial Intelligence for Banking Fraud Prevention</a>
<h2>Customer experience, the machine in contact with customers</h2>
<p>
The next step consists in putting the IA directly in touch with the bank's customers. When a suspicious transaction is detected, instead of mandating a bank anti-fraud employee to analyze the situation, which investigation usually ends up with a call to the customer for re-confirmation, the future is to let the machine contact the customer itself, by means of an application installed on the mobile of the customer, or by means of a voice chatbot able to contact him and speak to him to obtain directly the confirmation required for the validation of the transaction.
</p>
<p>
The benefits are numerous. For the customers of the banks it is a question of bringing this "callback" as close as possible to the input of the transaction, from a few hours today to a few seconds in the future. For the banks it is a question of reducing the costs of intervention while eliminating the frauds by systematically delegating the re-confirmation of the suspicious transactions to the customer.
</p>
<p>
In the end, the bank protects its reputation, its Asset Under Management, the data of its customers, but also meets them - meet the challenges of the digital - while reducing its operational costs - benefit from its opportunities.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/interview-on-artificial-intelligence
Interview on Artificial Intelligence
Jerome Kehrli
2018-06-29T10:29:52-04:00
2018-06-29T10:29:52-04:00
<p>
This is a collection of three videos I recorded for the <a href="https://www.empowerment.foundation/">"empowerment fundation"</a> as part of their file on <a href="https://bee2.ch/intelligence-artificielle-2/">Artificial Intelligence</a>.
</p>
<p>
In parallel and in addition to <a href="https://www.empowerment.foundation/becurious">BeCurious</a>, the <i>Empowerment Foundation</i> launches in 2018 a project of <i>curation files</i> thematic through the bee² program.
</p>
<p>
Taking up the practice of curating video content, bee² means: exploring the issues that build our world, expand the perspectives of analysis, stimulate awareness to enable everyone to act in a more enlightened and responsible way facing tomorrow's challenges.
<br>
It's about bringing out specific issues and allowing everyone to easily discover videos the most relevant, validated by experts, on the given topic without having to browse many sources of information.
</p>
<p>
The three videos I contributed to are (in french, sorry):
</p>
<ul>
<li>
<a href="https://www.youtube.com/watch?v=6g87mOszUWQ">AI and Cybersecurity: Preventing Bank Fraud</a>
</li>
<li>
<a href="https://www.youtube.com/watch?v=jOH0akpSmos">How does the Google self driving car work ?</a>
</li>
<li>
<a href="https://www.youtube.com/watch?v=lQY5_aHt42A">What are the limits of AI?</a>
</li>
</ul>
<p>
The three videos can be viewed directly on this very page below.
</p>
<p>
This is a collection of three videos I recorded for the <a href="https://www.empowerment.foundation/">"Empowerment Fundation"</a> as part of their file on <a href="https://bee2.ch/intelligence-artificielle-2/">Artificial Intelligence</a>.
</p>
<p>
In parallel and in addition to <a href="https://www.empowerment.foundation/becurious">BeCurious</a>, the <i>Empowerment Foundation</i> launched in 2018 a project of <i>curation files</i> thematic through the bee² program.
</p>
<p>
Taking up the practice of curating video content, bee² means: exploring the issues that build our world, expand the perspectives of analysis, stimulate awareness to enable everyone to act in a more enlightened and responsible way facing tomorrow's challenges.
<br>
It's about bringing out specific issues and allowing everyone to easily discover videos the most relevant, validated by experts, on the given topic without having to browse many sources of information.
</p>
<p>
The three videos I contributed to are::
</p>
<ul>
<li>
<a href="https://www.youtube.com/watch?v=6g87mOszUWQ">AI and Cybersecurity: Preventing Bank Fraud</a>
</li>
<li>
<a href="https://www.youtube.com/watch?v=jOH0akpSmos">How does the Google self driving car work ?</a>
</li>
<li>
<a href="https://www.youtube.com/watch?v=lQY5_aHt42A">What are the limits of AI?</a>
</li>
</ul>
<p>
<b>The interview was in french</b> (sorry ...)
</p>
<p>
The three videos can be viewed directly on this very page below.
</p>
<div id="player_2" style="max-width: 100%;"></div>
<script>
player = new YT.Player('player_2', {
height: '360',
width: '640',
videoId: '6g87mOszUWQ'
});
</script>
<p> </p>
<div id="player_3" style="max-width: 100%;"></div>
<script>
player = new YT.Player('player_3', {
height: '360',
width: '640',
videoId: 'jOH0akpSmos'
});
</script>
<p> </p>
<div id="player_4" style="max-width: 100%;"></div>
<script>
player = new YT.Player('player_4', {
height: '360',
width: '640',
videoId: 'lQY5_aHt42A'
});
</script>
<p> </p>
<p>
Enjoy :-)
</p>
https://www.niceideas.ch/roller2/badtrash/entry/lambda-architecture-with-kafka-elasticsearch
Lambda Architecture with Kafka, ElasticSearch and Spark (Streaming)
Jerome Kehrli
2018-05-04T06:32:20-04:00
2018-05-05T08:29:19-04:00
<!-- Lambda Architecture with Kafka, ElasticSearch and Spark (Streaming)-->
<p>
The Lambda Architecture, first proposed by Nathan Marz, attempts to provide a combination of technologies that together provide the characteristics of a web-scale system that satisfies requirements for availability, maintainability, fault-tolerance and low-latency.
</p>
<p>
Quoting <a href="https://en.wikipedia.org/wiki/Lambda_architecture">Wikipedia</a>: "<i>Lambda architecture is a data-processing architecture designed to handle massive quantities of data by taking advantage of both batch- and stream-processing methods.
<br>
This approach to architecture attempts to balance latency, throughput, and fault-tolerance by using batch processing to provide comprehensive and accurate views of batch data, while simultaneously using real-time stream processing to provide views of online data. The two view outputs may be joined before presentation.
<br>
The rise of lambda architecture is correlated with the growth of big data, real-time analytics, and the drive to mitigate the latencies of map-reduce.</i>"
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a63d7668-8d45-47ec-a2f5-57eb64eb6b00">
<img class="centered" style="width: 60px; " alt="Lambda Symbol" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a63d7668-8d45-47ec-a2f5-57eb64eb6b00" />
</a>
</div>
<br>
In my current company - NetGuardians - we detect banking fraud using several techniques, among which real-time scoring of transactions to compute a risk score.
<br />
The deployment of Lambda Architecture has been a key evolution to help us evolve towards real-time scoring on the large scale.
</p>
<p>
In this article, I intend to present how we do Lambda Architecture in my company using Apache Kafka, ElasticSearch and Apache Spark with its extension Spark-Streaming, and what it brings to us.
</p>
<!-- Lambda Architecture with Kafka, ElasticSearch and Spark (Streaming)-->
<p>
The Lambda Architecture, first proposed by Nathan Marz, attempts to provide a combination of technologies that together provide the characteristics of a web-scale system that satisfies requirements for availability, maintainability, fault-tolerance and low-latency.
</p>
<p>
Quoting <a href="https://en.wikipedia.org/wiki/Lambda_architecture">Wikipedia</a>: "<i>Lambda architecture is a data-processing architecture designed to handle massive quantities of data by taking advantage of both batch- and stream-processing methods.
<br>
This approach to architecture attempts to balance latency, throughput, and fault-tolerance by using batch processing to provide comprehensive and accurate views of batch data, while simultaneously using real-time stream processing to provide views of online data. The two view outputs may be joined before presentation.
<br>
The rise of lambda architecture is correlated with the growth of big data, real-time analytics, and the drive to mitigate the latencies of map-reduce.</i>"
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a63d7668-8d45-47ec-a2f5-57eb64eb6b00">
<img class="centered" style="width: 60px; " alt="Lambda Symbol" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a63d7668-8d45-47ec-a2f5-57eb64eb6b00" />
</a>
</div>
<br>
In my current company - NetGuardians - we detect banking fraud using several techniques, among which real-time scoring of transactions to compute a risk score.
<br />
The deployment of Lambda Architecture has been a key evolution to help us evolve towards real-time scoring on the large scale.
</p>
<p>
In this article, I intend to present how we do Lambda Architecture in my company using Apache Kafka, ElasticSearch and Apache Spark with its extension Spark-Streaming, and what it brings to us.
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. Introduction</a>
<ul>
<li><a href="#sec11">1.1 NetGuardians' key big data software components</a></li>
<li><a href="#sec12">1.2 One ring to rule them all</a></li>
<li><a href="#sec13">1.3 Real-time readiness</a></li>
</ul>
</li>
<li><a href="#sec2">2. Lambda Architecture</a>
<ul>
<li><a href="#sec21">2.1 Lambda Architecture principles</a></li>
<li><a href="#sec22">2.2 Lambda Architecture with Kafka, ElasticSearch and Spark (Streaming)</a></li>
<li><a href="#sec23">2.3 Drawbacks and difficulties of Lambda Architecture</a></li>
</ul>
</li>
<li><a href="#sec3">3. Real-time computation with Lambda Architecture</a>
</li>
<li><a href="#sec4">4. Conclusion</a>
</li>
</ul>
<a name="sec1"></a>
<h2>1. Introduction</h2>
<a name="sec11"></a>
<h3>1.1 NetGuardians' key big data software components</h3>
<p>
NG|Screener, NetGuardians' flasgship product, is a Big Data Analytics Platform aimed at preventing fraud on the large scale within Financial Institutions.
<br>
Our platform manages and operates Big Data Analytics Use Cases detecting fraud attempts by analyzing user behaviours and financial transactions. Working in real-time, it can block suspicious business events, e.g financial transactions to prevent fraud effectively.
</p>
<p>
Our platform is built internally on four key Big Data Open Source Software components:
</p>
<table style="border: 0px none;">
<tr>
<td style="width: 100px; border: 0px none;">
<img style="width: 100px; min-width: 100px; border: 0px none;" alt="Kafka Logo" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/203ed798-13b0-4d43-9f3b-4d6047ecd959" />
</td>
<td style="width: 100%; border: 0px none;">
<p>
<b>Apache Kafka</b>: Kafka is an open-source stream processing software aimed at providing a unified, high-throughput, low-latency platform for handling real-time data feeds.
</p>
</td>
</tr>
</table>
<table style="border: 0px none;">
<tr>
<td style="width: 100px; border: 0px none;">
<img style="width: 100px; min-width: 100px; border: 0px none;" alt="ElasticSearch Logo" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/26059d4e-069e-49d1-857d-50c428591557" />
</td>
<td style="width: 100%; border: 0px none;">
<p>
<b>ElasticSearch</b>: ElasticSearch is a distributed, real-time, RESTful search and analytics document-oriented storage engine. It lets one perform and combine many types of searches - structured, unstructured, geo, metric - in real time.
</p>
</td>
</tr>
</table>
<table style="border: 0px none;">
<tr>
<td style="width: 100px; border: 0px none;">
<img style="width: 100px; min-width: 100px; border: 0px none;" alt="Mesos Logo" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b38d48c0-7b52-40cd-ad86-203ae0735d6d" />
</td>
<td style="width: 100%; border: 0px none;">
<p>
<b>Apache Mesos</b>: Mesos is a distributed systems kernel that runs on every machine and provides applications with API's for resource management and scheduling across entire datacenter and cloud environments.
</p>
</td>
</tr>
</table>
<table style="border: 0px none;">
<tr>
<td style="width: 100px; border: 0px none;">
<img style="width: 100px; min-width: 100px; border: 0px none;" alt="Spark Logo" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/4c1d16c3-c660-42a3-9440-067587918a63" />
</td>
<td style="width: 100%; border: 0px none;">
<p>
<b>Apache Spark</b>: Spark is a fast and general engine for large-scale data processing. It provides programmers with an API functioning as a working set for distributed programs that offers a versatile form of distributed shared memory.
</p>
</td>
</tr>
</table>
<a name="sec12"></a>
<h3>1.2 One ring to rule them all</h3>
<p>
The choice of these specific components under the hood is not anecdotal. Running Apache Spark on Apache Mesos is really still cutting edge nowadays and the choice of Apache Kafka and ElasticSearch, in addition to the good fit with our use case, answers a very important need we have.
</p>
<p>
We deploy our platform as much in tier 1 banks and big financial services providers than small private banks in Switzerland or even small Credit Institutions in Africa. Some of our customers have a few thousands of transactions daily while some others have dozens of millions of transactions per day.
<br>
Considering that some of our Analytics use cases require depth of analysis of several years, when we have billions of events to consider, we deploy our analytics platform on multiple-nodes clusters, sometimes up to a few dozen computation and storage nodes within the cluster. On the other hand, when we we work for small institutions with very low data volumes, we deploy it on a single small machine.
<br>
This need is at the very root of our technology choice, we needed technologies able to run efficiently on single small machines while still being able to scale our on hundreds of nodes should we require that.
</p>
<p>
ElasticSearch, Apache Spark, Apache Mesos and Apache Kaflka have been designed from the grounds up with this horizontal scalability in mind. But they have been implemented in such a way that they run also very well on a single little machine.
<br>
This is pretty uncommon in the Big Data Technology / NoSQL family of products. For instance, Apache Hadoop performs most of the time very poorly on single machines.
</p>
<p>
These products under the hood are key to sustain our "<i>one ring to rule them all</i>" approach. We develop one single platform that we can deploy everywhere, regardless of the volume of data of our customers.
</p>
<a name="sec13"></a>
<h3>1.3 Real-time readiness</h3>
<p>
In addition to their unique genes regarding vertical scalability described above, ElasticSearch, Apache Kafka and Apache Spark are providing our platform with another key feature.
</p>
<p>
With ElasticSearch, real-time updating (fast indexing) is achievable through various functionalities and search / read response time can be astonishingly deterministic.
</p>
<p>
Apache Kafka comes with the Kafka Stream extension. The Streams API, available as a Java library that is part of the official Kafka project, is the easiest way to write mission-critical real-time applications and microservices with all the benefits of Kafka's server-side cluster technology.
<br>
Despite being a humble library, Kafka Streams directly addresses both hardest problems in stream processing:
</p>
<ul>
<li>event-at-a-time processing with millisecond latency and </li>
<li>stateful processing including distributed joins and aggregations.</li>
</ul>
<p>
Kafka enables to implement fast processing on business events, e.g most often financial transactions in real-time and in event-at-a-time mode while dispatching micro-batches further to Spark Streaming.
The more complicated processing required by our Analytics use cases occurs then within Spark through the Spark Streaming extension.
</p>
<p>
Spark Streaming is able to process hundreds of thousands of records per node per second. When using Kafka as a source, it is able to consume nearly half million records per node per second which is striking. It also offers near linear scaling ability, another great perk.
<br>
In contrary to Kafka, Spark Streaming works using a micro-batches approach. It works as follows; received input streams and decided into small batches, which are processed by Spark engine and a processed stream of batches is return.
<br>
The micro-batches can be as small as a few milliseconds batches, thus enabling sub-second latency while still ensuring a very high throughput and access to the whole Spark power and versatility to implement high level analytics use cases.
</p>
<p>
This real-time readiness aspect of these components of our technology stack is key to deploy Lambda Architecture within the our platform.
</p>
<a name="sec2"></a>
<h2>2. Lambda Architecture</h2>
<p>
When it comes to processing transactions in real-time, our platform provides a state-of-the-art implementation of a Lambda Architecture.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a63d7668-8d45-47ec-a2f5-57eb64eb6b00">
<img class="centered" style="width: 60px; " alt="Lambda Symbol" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a63d7668-8d45-47ec-a2f5-57eb64eb6b00" />
</a>
</div>
<br>
<p>
Lambda architecture is a Big Data Architecture that enables us to reunite our real-time and batch analytics layers.
</p>
<a name="sec21"></a>
<h3>2.1 Lambda Architecture principles</h3>
<p>
Lambda architecture is a data-processing architecture designed to handle massive quantities of data by taking advantage of both batch- and stream-processing methods. This approach to architecture attempts to balance latency, throughput, and fault-tolerance by using batch processing to provide comprehensive and accurate views of batch data, while simultaneously using real-time stream processing to provide views of online data.
</p>
<p>
At a high level, the Lambda Architecture is designed to handle both real-time and historically aggregated batched data in an integrated fashion. It separates the duties of real-time and batch processing so purpose-built engines, processes, and storage can be used for each, while serving and query layers present a unified view of all of the data.
<br>
The rise of lambda architecture is correlated with the growth of big data and real-time analytics.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a63d7668-8d45-47ec-a2f5-57eb64eb6b00">
<img class="centered" style="width: 800px; " alt="Lambda Architecture" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/f36e9948-0053-4f07-a0ff-a1ebca0b5762" />
</a>
</div>
<br>
<p>
As new data is introduced to the system, it is processed simultaneously by both the batch layer, and the speed layer. The batch layer is an append-only repository containing unprocessed raw data. The batch layer periodically or continuously runs jobs that create views of the batch data-aggregations or representations of the most up-to-date versions. These batch views are sent to the serving layer, where they are available for analytic queries.
<br>
At the same time that data is being appended to the batch layer, it is simultaneously streaming into the speed layer. The speed layer is designed to allow queries to reflect the most up-to-date information-necessary because the serving layer's views can only be created by relatively long-running batch jobs. The speed layer computes only the data needed to bring the serving layer's views to real time-for instance, calculating totals for the past few minutes that are missing in the serving layer's view.
<br>
By merging data from the speed and serving layers, low latency queries can include data that is based on computationally expensive batch processing, and yet include real-time data.
In the Lambda Architecture, the raw source data is always available, so redefinition and re-computation of the batch and speed views can be performed on demand. The batch layer provides a big data repository for machine learning and advanced analytics, while the speed and serving layers provide a platform for real-time analytics.
<br>
The Lambda Architecture provides a useful pattern for combining multiple big data technologies to achieve multiple enterprise objectives.
</p>
<a name="sec22"></a>
<h3>2.2 Lambda Architecture with Kafka, ElasticSearch and Spark (Streaming)</h3>
<p>
Lambda defines a big data architecture that allows pre-defined and arbitrary queries and computations on both fast-moving data and historical data.
<br>
Using Kafka, ElasticSearch, Spark and SparkStreaming, it is achieved using the following layout:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a63d7668-8d45-47ec-a2f5-57eb64eb6b00">
<img class="centered" style="width: 800px; " alt="Lambda Architecture - Software Components" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/45ee2683-eb9f-4698-aea4-de5c18620c40" />
</a>
</div>
<br>
<p>
Lambda Architecture enables us to score transactions or other business events in real-time and still consider the most recent events as well as the whole transaction history in its scoring model.
</p>
<p>
By using Kafka at the beginning of the pipeline to accept inputs, it can be guaranteed that messages will be delivered as long as they enter the system, regardless of hardware or network failure.
</p>
<p>
The batch layer is largely build on the Apache Spark / Mesos coupled with ElasticSearch as large scale storing component underneath. The reasons why we are running on Spark, Mesos and ElasticSearch have been covered before in this document but interestingly, these components appear to behave extremely well together when it comes to addressing batch processing concerns, thanks to spark's ability to work largely in memory and proper optimization of data co-locality on ElasticSearch and Spark nodes.
</p>
<p>
In the streaming layer, Kafka messages are consumed in real time using Spark Streaming. In terms of core component to support the speed layer, the usual choice is between Apache Storm or Apache Spark Streaming.
The main selection criteria between the two depends on whether one is interested in ultra low latency (Apache Storm) or high throughput (Apache Spark Streaming). There are other factors, but these are some of the main drivers.
<br>
In my company, for our use cases, we can afford a little higher latency as long as we work under a second to score a business event (e.g. financial transaction). On the other hand, we face situations where burst of thousands of transactions to be scored per second are common. As such, high throughput is not optional for us, it's a key requirement and as such, the rationality behind the usage of Apache Spark Streaming.
<br>
Here, in the speed layer, ElasticSearch is key to reduce latency of integration concerns of the speed layers since it is a real-time querying database in addition to a very powerful database engine.
</p>
<p>
The Serving Layer, consolidating the batch layer and speed layer partial results, is largely home made in our case and relies on ElasticSearch's ability to fetch both partial sets in real-time.
</p>
<a name="sec23"></a>
<h3>2.3 Drawbacks and difficulties of Lambda Architecture</h3>
<p>
There is a natural tendency to duplicate logic between batch layer and speed layer which needs to be addressed through strict design and re-usable logic. Using Spark in batch mode on the batch layer and Spark Streaming on the speed layer in our case really helps us reuse business logic as much as possible between both worlds.
</p>
<p>
In addition, there is an operational complexity of the systems that are involved in implementing the lambda architecture. Thus the implementation of Lambda architecture is inherently difficult.
</p>
<a name="sec3"></a>
<h2>3. Real-time computation with Lambda Architecture</h2>
<p>
The demand for real-time analytics has led to demand for workflows that can effectively balance latency, throughput, scaling and fault tolerance.
<br>
In order to accommodate the demand for real-time analytics, we need to design a system that can provide balance between the concept of "single version of truth" and "real-time analytics". Lambda Architecture is one such method.
</p>
<p>
In my company, some of our analytics use cases require to consider very extended contextual information about trade and transaction activities, for instance, to build user and customer profiles or analyze their past behaviours.
<br>
Building such contextual information typical require analyzing over again and again billions of business events and peta-bytes of data.
<br>
Rebuilding these profiles or re-creating the aggregated statistical metrics would require several dozens of minutes even on large cluster in a typical batch processing approach.
<br>
Happily, all this information supports an incremental building way and as such we can benefit from Lambda architecture to rebuild the historical part while the latest data is taken into consideration by the speed layer to provide an up-to-date (as far as real-time) view of the reality. The serving layer consolidates both results to provide always up-to-date and accurate views of these profiles or other aggregated statistical metrics.
<br>
These real-time metrics are thus made available to our real-time scoring and classification systems.
</p>
<p>
The same technologies and approaches deployed in the speed layer to provide up-to-date views of the reality are used to score and classify business events, e.g. financial transactions in real-time.
<br>
Here as well, we have no requirements for strong real-time with millisecond-order latency. As long as we can provide a risk score or a classification for an event under a second, this is sufficient for our use cases.
<br>
On the other hand, it happens often that we have to compute burst of events of several hundreds of entries per second. As such, a system benefiting from an acceptable latency but a very high throughput such as Apache Spark Streaming is a key component of our processing platform.
</p>
<p>
In addition, within NG|Screener UI we provide our customers with a full-blend data discovery application (forensic application). Lambda Architecture is key in enabling us to provide our users with real-time updates and a second close up-to-date view of the reality.
</p>
<a name="sec4"></a>
<h2>4. Conclusion</h2>
<p>
Deploying Lambda architecture on our use cases has proven to be the simplest way to reach our objectives:
</p>
<ul>
<li>Up to date and second-close view of the reality in contextual information, user / customer profiles and other key periodic statistical metrics</li>
<li>Classification and scoring of business events with an under-a-second latency and a very high throughput</li>
<li>Resilience and fault tolerance of our business processes on large clusters, both on technical failures and human failures</li>
<li>Simplicity and maintenance, especially in our approach since we can share significant portions of codes between the batch layer and the speed layer since both are built on Apache Spark</li>
<li>Resolution of operational complexity of big computation on historical data by dividing the work to do in an incremental fashion.</li>
</ul>
<p>
Now of course, Lambda Architecture being the simplest way for us to reach our mission-critical objectives doesn't make it simple per se, on the contrary. Lambda Architecture is inherently difficult to deploy and maintain and requires sound design and implementation.
</p>
<p>
At NetGuardians, we could benefit from our mastery of cutting-edge technologies as well as our in-depth experience of batch computing systems and real-time computing systems to make it an advantage of our approach.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/artificial-intelligence-for-fraud-prevention
Artificial Intelligence for Banking Fraud Prevention
Jerome Kehrli
2018-04-30T08:57:44-04:00
2018-07-04T15:30:11-04:00
<p>
In this article, I intend to present my company's - NetGuardians - approach when it comes to deploying Artificial Intelligence techniques towards better fraud detection and prevention.
<br>
This article is inspired from various presentations I gave on the topic in various occasions that synthesize our experience in regards to how these technologies were initially triggering a lot of skepticism and condescension and how it turns our that they are now really mandatory to efficiently prevent fraud in financial institutions, due to the rise of fraud costs, the maturity of cybercriminals and the complexity of attacks.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/123503c0-bec6-43d3-95e5-5ba6eea322f6">
<img class="centered" style="width: 250px;" alt="Artificial Intelligence for Fraud Prevention" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/123503c0-bec6-43d3-95e5-5ba6eea322f6" />
</a>
</div>
<br>
<p>
Here financial fraud is considered at the broad scale, both internal fraud, when employees divert funds from their employer and external fraud in all its forms, from sophisticated network penetration schemes to credit card theft.
<br>
I don't have the pretension to present an absolute or global overview. Instead, I would want to present things from the perspective of NetGuardians, from our own experience in regards to the problems encountered by our customers and the <b>how Artificial Intelligence helped us solve these problems</b>.
</p>
<!-- AI for Fraud Prevention -->
<p>
In this article, I intend to present my company's - NetGuardians - approach when it comes to deploying Artificial Intelligence techniques towards better fraud detection and prevention.
<br>
This article is inspired from various presentations I gave on the topic in various occasions that synthesize our experience in regards to how these technologies were initially triggering a lot of skepticism and condescension and how it turns our that they are now really mandatory to efficiently prevent fraud in financial institutions, due to the rise of fraud costs, the maturity of cybercriminals and the complexity of attacks.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/123503c0-bec6-43d3-95e5-5ba6eea322f6">
<img class="centered" style="width: 250px;" alt="Artificial Intelligence for Fraud Prevention" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/123503c0-bec6-43d3-95e5-5ba6eea322f6" />
</a>
</div>
<br>
<p>
Here financial fraud is considered at the broad scale, both internal fraud, when employees divert funds from their employer and external fraud in all its forms, from sophisticated network penetration schemes to credit card theft.
<br>
I don't have the pretension to present an absolute or global overview. Instead, I would want to present things from the perspective of NetGuardians, from our own experience in regards to the problems encountered by our customers and the <b>how Artificial Intelligence helped us solve these problems</b>.
</p>
<p>
This article is available as a slideshare presentation here
<a href="https://www.slideshare.net/JrmeKehrli/artificial-intelligence-for-banking-fraud-prevention-95475760">https://www.slideshare.net/JrmeKehrli/artificial-intelligence-for-banking-fraud-prevention-95475760</a>
</p>
<p>
A video of the speech is available on youtube:
</p>
<!-- 1. The <iframe> (and video player) will replace this <div> tag. -->
<div id="player_aiii" style="max-width: 100%;"></div>
<script>
player = new YT.Player('player_aiii', {
height: '360',
width: '640',
videoId: 'QZbgBvZgFkA'
});
</script>
<h2>1. Early times, the 2000s</h2>
<p>
Before 2000, banking institutions are only poorly equipped when it comes to fight financial fraud.
</p>
<p>
For most of it, detecting fraud cases relies on manual verification and tests performed by
</p>
<ul>
<li>Internal Control</li>
<li>Internal Audit or</li>
<li>External Audits</li>
</ul>
<p>
And unfortunately, this implies a lot of issues:
</p>
<ul>
<li>
By working with samples only, Internal control and Audit let a lot of fraud cases pass through the cracks and are found only very late or even never.
</li>
<li>
Analysis are cumbersome and most often finding fraud cases is not the first and foremost objective of the auditors.
</li>
</ul>
<p>
Now of course, the most essential security rules and checks are implemented within the Operational Information System or in the form of procedures to be respected and audited.
<br>
Also, some banking institutions already have an Analytics System - or Business Intelligence - at the time and some ad'hoc reports are implemented on top of it that target fraud detection.
</p>
<p>
In these early times, neither the subprime crisis nor the south European countries debt crisis happened. Margins are important, people trust banks and all in all bankers are happy people.
<br>
Fraud cases, mostly internal, exist of course but financial institutions feel rather safe,
</p>
<h2>2. The late 2000s - fraud costs rise</h2>
<p>
In the second half of the 2000's, however, the costs linked to fraud, increasingly external, the complexity of attacks and the maturity of attackers rise.
<br>
Banking institutions react by deploying quite massively and for the first time specific analytics systems aimed at detecting banking fraud, both external and internal.
</p>
<p>
At this time, these systems are rules-engines that work by checking or searching pre-defined and well defined conditions within the data extracted from the information system.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/3094a30f-a2f6-4171-acc9-3b658993edf6">
<img class="centered" style="width: 250px;" alt="Rules Engines" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/3094a30f-a2f6-4171-acc9-3b658993edf6" />
</a>
</div>
<br>
<p>
In a way these systems can be considered as simple extensions of the security checks and rules implemented directly within the operational information system. These solutions come most of the time from the AML - Anti Money Laundering - World, their editors having understood that banking fraud was a an interesting opportunity to extend their sales.
</p>
<p>
A very simple rule example would be as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/f8d2142b-5976-4a17-a905-c1d5b89089b7">
<img class="centered" style="width: 450px;" alt="Rule Example" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/f8d2142b-5976-4a17-a905-c1d5b89089b7" />
</a>
</div>
<br>
<p>
At this time, a first set of papers have already been published on the success, still somewhat relative in this early days, of some Machine Learning approaches implemented towards banking fraud detection.
<br>
But Machine Learning and Artificial Intelligence are considered with a lot of condescension and skepticism.
<br>
Bankers and their engineers are not willing to consider an approach whose interpretation of results is deemed fuzzy.
</p>
<p>
NetGuardians has been founded at these times and the NetGuardians platform could be seen then as a gigantic rule engine.
</p>
<h2>3. The reality of fraud changes dramatically</h2>
<p>
Unfortunately, the reality of fraud and financial cybercrime evolved fast and dramatically.
</p>
Let me give you two examples
</p>
<h3>3.1 The Bangladesh Bank heist</h3>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/efa6f623-52fb-4999-bea1-263235c588ac">
<img class="centered" style="width: 500px;" alt="The Bangladesh Bank heist - initial" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/efa6f623-52fb-4999-bea1-263235c588ac" />
</a><br>
<div class="centered">
(Source : <a href="https://www.bankinfosecurity.com/bangladeshi-bank-hackers-steal-100m-a-8958">https://www.bankinfosecurity.com/bangladeshi-bank-hackers-steal-100m-a-8958</a>)
</div>
</div>
<br>
<p>
In February 2016, a group that we deem around 20 persons, composed by financial experts, software engineers and hackers have attacked the information system of the Bangladesh Central Bank.
<br>
They manage to compromise the bank internal gateway to the SWIFT Network. The SWIFT network is the international banking messaging network used by banks to communicate and transfer money through electronic wire. The pirates used the SWIFT network to withdraw money from the Bangladesh Central bank VOSTRO account by the US Federal Reserve.
<br>
They manage to transfer 81 millions USD to the Philippines and used the Philippino casinos to launder the stolen funds.
</p>
<p>
As a sidenote, the fact that they have stolen "only" 81 million USD is an amazing luck for the bank, or rather an amazing bad luck for the cybercriminals.
<br>
An Anti-Money laundering system - <i>rule-based</i> - deployed in the US federal Reserve for Anti-Money Laundering blocked the 6th transaction because the beneficiary name contained the word "Jupiter". Jupiter was on a sanction screening list in the US because a cargo ship navigating under Iranian flag was named "Jupiter" something. The 6th transaction being blocked, all the further ones, a little less than thirty, have been blocked as well.
<br>
But 5 transactions pass through before the 6th has been blocked by the Fed and went further through the correspondent banking network.
<br>
Another transaction has been blocked by the Deutsche Bank, a routing bank, because of a typo: "Shilka Fandation" instead of "Shilka Fundation".
<br>
So only 4 transactions our of 35 successfully arrived to the Philippines and as such the total loss have been reduced from 951 million USD initially intended to "only" 81 millions USD.
</p>
<!--
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/98aff47d-08ee-4694-8d02-a8bdd21b0387">
<img class="centered" style="width: 350px;" alt="The Bangladesh Bank heist - optimistic responsibles" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/98aff47d-08ee-4694-8d02-a8bdd21b0387" />
</a><br>
<div class="centered">
(Source : <a href="http://www.dhakatribune.com/business/banks/2017/03/28/muhith-stolen-heist-money-must-recovered/">http://www.dhakatribune.com/business/banks/2017/03/28/muhith-stolen-heist-money-must-recovered/</a>)
</div>
</div>
<br>
-->
<p>
As a fun note, a few week after the heist, all the responsibles of the financial institutions involved, the US Fed Reserve, the Bangladesh Central Bank, even the finance minister of the Philippines were all convinced that the money - or at least a significant part of it - would be recovered and that the cybercriminals would be caught.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/fd3ece3c-1f56-4f0a-8e77-39984a17b3ff">
<img class="centered" style="width: 500px;" alt="The Bangladesh Bank heist - 2 years after" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/fd3ece3c-1f56-4f0a-8e77-39984a17b3ff" />
</a>
</div>
<br>
<p>
Two years after, today, we know that we will never recover these funds.
<br>
The attacker are safe, untraceable and will never be found. We believe that this is a group of about 20 persons who worked on the heist preparation for about 18 months. 81 million USD is a pretty number.
</p>
<p>
Now you think ... But this is Bengladesh ... right ?
<br>
Here we are in Europe ... Even better, here we are in Switzerland ... right ?
And in Switzerland we don't really feel concerned by the numerous security holes in the Bangladesh Central Bank Information System.
So let me give you another example...
<h3>3.2 The retefe Worm</h3>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/4d566e47-ae5f-4cab-b540-801a0d685742">
<img class="centered" style="width: 600px;" alt="The Retefe Saga" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/4d566e47-ae5f-4cab-b540-801a0d685742" />
</a>
</div>
<br>
<p>
Quoting <a href="https://www.govcert.admin.ch/blog/33/the-retefe-saga">https://www.govcert.admin.ch/blog/33/the-retefe-saga</a>:
</p>
<div class="centering">
<div class="centered">
<p>
"This threat actor has already been around for more than four years...
<br>
Their goal remains the same: committing e-banking fraud in Switzerland and Austria.
</p>
<p>
In August 2017, Retefe still redirects between 10 and 90 e-banking sessions every day."
</p>
</div>
</div>
<p>
The Retefe worm is a worm developed by a team of cybercriminals targeting specifically the ebanking platforms of small and mid size Austrian And Swiss Banking Institutions.
<br>
The worm is used by the thieves to take control of the victim's ebanking sessions and to submit fraudulent transactions to the system.
</p>
<p>
This worm is 4 years old.
<br>
For 4 years, fraudsters keep on updating it, modifying it and extending it to counter the anti-viruses software and the specific protections put in place by the banks.
<br>
This worm is 4 years old and nevertheless, as pointed out by the Computer Security Section of the Federal Finance Separtment, it is still making today between 10 and 90 victims in Switzerland and Austria.
</p>
<p>
Today, in the swiss banks ...
</p>
<p>
My conclusion from these examples is as follows:
<br>
Today, fraudsters and cybercriminals are professionals. The time when fraud was mostly coming from little hackers working in their garage or back-office employees disappointed by their bonus, is over. Today, attackers are professionals who have industrialized their methods.
</p>
<h2>4. Facts and Projections</h2>
<p>
Some facts and projections to understand what reality banking institutions are facing nowadays...
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/0c3a4d41-4d11-4c18-b031-0f3f9d5580d8">
<img class="centered" style="width: 700px;" alt="Facts and projections" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/0c3a4d41-4d11-4c18-b031-0f3f9d5580d8" />
</a>
</div>
<br>
<p>
In frebruary 2016, a group of cybercriminals managed to steal 81 million USD from the VOSTRO account of the Bangladesh Central Bank by the US federal Reserve
<br>
This is one of the biggest bank heist in history and the most impressive cybercrime ever
</p>
<p>
In a report called "<i>Report to the nations</i>", the international association of Fraud Examiners estimated that in 2017, the total cost of fraud has been 3000 billions USD.
<br>
In banking fraud, a big part of this amount is related to internal fraud, when bank employees divert funds from their employer.
<br>
In Switzerland, of course, thanks to the maturity of the banking business as well as the security checks and practices put in place in banking institutions, internal fraud is marginal, compared to external fraud. But external fraud is a cruel reality, think of the Retefe Worm.
</p>
<p>
Finally, Cyber Security ventures estimates that by 2021 the total cost of cybercrime will reach 6000 billion USD.
</p>
<p>
The reality to which banks are confronted nowadays is this one.
</p>
<h2>5. Historical systems are beaten</h2>
<p>
The principal implication of this reality, the problem which banking institutions are confronted to nowadays is that historical systems deployed to counter fraud - rules engines - are beaten.
</p>
<p>
Let's assume that a banking institution wants to define a set of rules aimed at detecting when an attacker diverts a customer account to issue fraudulent transactions.
</p>
<ul>
<li>
Imagine the situation of a first customer, someone such as myself, using his ebanking account to pay his loan at the end of the month, his mortgage, his taxes, telephone bills, etc.
<br>
In my case, a big transaction withdrawing 20 k CHF from my account for a beneficiary located in Nigeria should raise an alert. It's clearly an anomaly, being completely outside of my usual habits and behaviour.
</li>
<li>
Imagine now the situation of a another customer, a responsible of acquisitions for a big corporation, a frequent traveler, spending most of his time abroad and using the corporate account to pay big amounts to providers all over the world.
<br>
In the case of this second customer, it is on the contrary a small payment benefiting to a counterparty in Switzerland that would be the anomaly and should raise an alert.
</li>
</ul>
<p>
If one wants to detect anomalies for these two different situations, one would end up implementing a completely different set of rules for the two distinct customers.
<br>
And this is impossible.
</p>
<p>
Every bank customer, and even user up to a certain level, is different.
<br>
Representing everyone's own and private situations with rules would require to implement and manage hundreds of thousands of rules on the system, which, obviously, is impossible.
</br>
Only the most common set of rules can be implemented, which means that:
</p>
<ol>
<li>A lot of frauds pass through the cracks.</li>
<li>In addition, in order to catch the biggest frauds, the limits enforced by the rules have to be very low, which has the consequence of flagging a lot of cases to be analyzed - the so called false-positives - requiring an army of analysts to be reviewed and discarded.
</li>
</ol>
<p>
The direct consequences for our customers are as follows:
<p>
<ul>
<li>
Financial impacts: frauds must be reimbursed. And in addition these analysts spending their days discarding <i>false positives</i> must be paid.
</li>
<li>
Reputation impacts: a fraud case being communicated in the newspapers is a nightmare for banking institutions. Even without a large scale communication, customers impacted by fraud will loose faith in their bank.
<br>
Then I do not need to explain the consequences that the thousands of papers published on the Bangladesh Bank heist had on the Bangladesh central bank.
</li>
</ul>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/eef79897-85b6-453d-825f-e6a3b40983aa">
<img class="centered" style="width: 600px;" alt="Consequence : rules engines are beaten" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/eef79897-85b6-453d-825f-e6a3b40983aa" />
</a>
</div>
<br>
<p>
Rule-based systems are beaten today.
<br>
Something else is required to protect efficiently Banking institutions from banking fraud.
</p>
<h2>6. Artificial Intelligence comes in help</h2>
<p>
Artificial Intelligence provides the solution to this problem.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/39bccb86-968e-4c78-af5e-d1016ede0ef1">
<img class="centered" style="width: 300px;" alt="AI helps" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/39bccb86-968e-4c78-af5e-d1016ede0ef1" />
</a>
</div>
<br>
<p>
In 2016, we started at NetGuardians to integrate the first advanced algorithms, so called Machine Learning algorithms, in our systems.
</p>
<p>
We let an Artificial Intelligence analyze continuously the history of billions of transactions in the system and learn about individuals habits and behaviours.
<br>
With big data technologies, AI can analyze a very extended depth of history and build dynamic profiles for each and every individual capturing his transactional behaviour.
<br>
Individuals can be both Customer and Users (Internal Employees):
</p>
<ul>
<li>Profiling customers is required for both Internal and External Fraud.</li>
<li>Profiling users is required for Internal Fraud. </li>
</ul>
<p>
Big Data technologies are key to maintain these profiles up-to-date in real time by tracking each and every interaction between the user and the bank systems.
<br>
In addition to a financial transaction direct characteristics such as the beneficiary, the target bank country, the amount of the transaction, its currency, etc., the machine can correlate a lot of indirect characteristics, such as where in the world was located the ATM where the user withdrawn money from, where was he connected to his ebanking session, etc.
</p>
<p>
For each and every individual a dynamic and up to date profile captures his behaviour and his habits.
<br>
Then, each and every financial transaction, regardless of its type, it being a security trade order, an ATM withdrawal or an ebanking payment, is compared against the user profile and a risk score is computed.
<br>
Based on this risk score, the machine eventually decides whether the transaction is genuine or not and whether it requires further investigation by a human analyst within the bank.
</p>
<p>
The gains for our customers of this new approach, based on customer profiling done by AI, is striking.
<br>
It has been a game changing shift of paradigm.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/11b20bad-1ee4-4606-b564-66dc59e27852">
<img class="centered" style="width: 600px;" alt="Consequences and benefits of AI" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/11b20bad-1ee4-4606-b564-66dc59e27852" />
</a>
</div>
<br>
<p>
In the banking institutions where we can deploy this new generation approach, we almost eliminate the amount of fraud cases passing through the cracks.
<br>
And that, by still reducing to 1/3 of what it was before the number of cases flagged by the system to be reviewed by an analyst or fraud investigator (most of them being the so-called false positives).
<br>
Not only the amount of cases, but the amount of time required to investigate a case could be reduced by 80% by having the machine presenting the profile of the customer and how the individual transaction deviates from it with relevant and meaningful visualization techniques.
<br>
Finally, the number of re-confirmation asked to customers could be reduce to 1/4.
</p>
<p>
Reducing the time required to investigate a case in addition to the amount of cases to be investigated has a direct financial impact: analysts spend less time investigating such cases and can focus on task with more added value. Drastically reducing fraud cases passing through also has obvious financial impacts.
<br>
Now all of this, especially reducing the number of times a re-confirmation is asked to customers has positive impacts on reputation
</p>
<p>
Now working on a per-customer basis is sometimes still sub-optimal. Sometimes a genuine transaction is always very unusual on a per-customer basis and it is required to broaden the view of the Artificial Intelligence.
<br>
Let me give you an example.
<br>
Let's imagine that tomorrow I buy a new Audi. That would be a transaction of 60 kCHF leaving my account for a beneficiary - Amag Audi Switzerland - that I never used before. Such a transaction, new beneficiary and huge amount is completely outside of my profile.
<br>
Based on this, the AI will decide to block the transaction, requiring a further validation from my end which will annoy me.
<br>
So how can we avoid that ?
<br>
If we look more carefully and globally at the transactions of this kind, big amounts benefiting to Amag Audi Swritzerland, among the customers with same profiles as myself, are quite usual.
</p>
<p>
The machine needs a broader view to understand that this transaction is not unusual.
</p>
<h2>7. The Machine can do better</h2>
<p>
The machine can look at the big picture and analyze transactions at a broader scale.
<br>
Recall the Audi example. When such a transaction is very unusual for a specific customer, looking at other customers with similar conditions, habits and behaviour is required.
</p>
<p>
And here again AI comes in help.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/123503c0-bec6-43d3-95e5-5ba6eea322f6">
<img class="centered" style="width: 250px;" alt="Artificial Intelligence for Fraud Prevention" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/123503c0-bec6-43d3-95e5-5ba6eea322f6" />
</a>
</div>
<br>
<p>
AI can analyze behaviours and habits of customers and group together the people with same patterns. People that are the same age, same wealth level, same origins, live in the same region, etc. will have a strong tendency to behave the same: for instance drive the same kind of car, such as an Audi, live in a appartments of the same size, pay the same amount of telephone bills at the end of the month, etc.
<br>
The machine can analyze customer activities and transactions on a very large scale and <i>cluster</i> together customers with same behaviour.
<br>
Then, these groups can be profiled just as individuals.
<br>
And finally, a transaction can be scored against the <b>customer group profile</b> in addition to the <b>customer profile</b>.
</p>
<p>
Recalling the Audi example. When scoring this specific payment against the individual profile, the transaction will be flagged as suspicious.
<br>
Scoring it against the group profile will clearly indicate that it's a genuine transaction. People buy new cars every day, especially in Switzerland
</p>
<p>
With this new approach, looking at the broader scale and comparing customers with each others instead of only scoring transactions in the individual context of a customer, we could improve our fraud detection system further.
</p>
<p>
The number of cases to be analyzed (false positives) could be reduced further.
<br>
In addition, the groups and their profiles happen to be an invaluable source of information for other use cases and concerns within the bank such as marketing, trend analysis, etc.
<br>
Of course reducing the number of cases to be handled by the investigation team has a direct impact on operational efficiency and induces further financial gains
</p>
<p>
Now all of this, transaction scoring and customer clustering works amazingly, but it works after the facts. The transaction has been input in the system and if we are not fast enough, depending on how we integrate within the bank information system, we can be too late, doing only fraud detection and not fraud prevention.
<br>
Our idea from here was:
</p>
<ul>
<li>
What if we could analyze the User or customer activities even before the transaction is input one the system and detect fraud before it happens ?
</li>
<li>
What if we could interpret weak signals coming from the analysis of how the Customer interacts with the banking information system to qualify him as legitimate or potentially fraudulent ?
</li>
</ul>
<p>
All of this require completely different analysis techniques.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/10b2587b-3006-451a-9237-fb614aafdc96">
<img class="centered" style="width: 600px;" alt="Consequences and benefits of AI - further improvements" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/10b2587b-3006-451a-9237-fb614aafdc96" />
</a>
</div>
<br>
<h2>8. Even further</h2>
<p>
Let me give you a simple example of what I mean by analyzing a customer's interaction with the banking Information system.
<br>
The interactions of a customer with the ebanking application is the simplest example I can come up with.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/dec86cd1-4705-4b1c-bc41-0b86f1658adf">
<img class="centered" style="width: 600px;" alt="Different ebanking session" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/dec86cd1-4705-4b1c-bc41-0b86f1658adf" />
</a>
</div>
<br>
<p>
Imagine the situation of a genuine user of the ebanking platform whose behaviour when inputting is payments is always the same:
</p>
<ul>
<li>He logs in the ebanking platform.</li>
<li>He looks at his account balance.</li>
<li>He performed all his payment, from input to validation, many of them.</li>
<li>He checks his pending orders, making sure he missed none of them.</li>
<li>He logs out the platform.</li>
</ul>
<p>
Now if a worm hijacks the ebanking session, the worm will do none of that:
</p>
<ul>
<li>The worm will likely go directly from login to payment input, validation, then logout.</li>
</ul>
<p>
Here I am only showing transitions but one can also consider User think time, keyboard stroke speed, etc.
</p>
<p>
AI can analyze all this behaviour and activity trails a user or customer leaves on the banking information system and build a probabilistic model capturing this behaviour as a succession of interactions.
<br>
Then, when an individual action is performed, the machine can compute the likelihood of that action to be performed by a legitimate user or an attacker based on the <i>path-to-action</i>.
<br>
And here as well, AI can build profiles of these activities and their likelihood both at individual level and group level through clustering techniques.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/fa9fc0a1-f96e-4b7a-b105-62417ccb10b6">
<img class="centered" style="width: 250px;" alt="Artificial Intelligence for interactions monitoring" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/fa9fc0a1-f96e-4b7a-b105-62417ccb10b6" />
</a>
</div>
<br>
<p>
With this kind of analysis, by looking at all the interactions of the users or customers with the banking information systems, AI can look at all individual events and qualify these interactions as legitimate or suspicious regardless of the financial transactions being input or not on the system.
</p>
<p>
AI can detect a fraud, or the intention to commit a fraud, even before a transaction is input on the system, by analyzing the user or customer activity, in the form of its interactions with the Banking Information System, before inputing the transaction.
<br>
In addition, by analyzing the behaviour of the customer as a whole, AI can qualify his interaction session (ebanking, mobile banking, PSD2, etc.) as legitimate or suspicious and kill the session in case of a doubt, thus protecting the information he sees and protecting his privacy in addition to his assets.
<br>
Finally, all this understanding of the user or customer habits and behaviour can be used to design even more advanced transaction scoring models.
</p>
<p>
This ability to detect fraud cases before they happen lead to further improvement of the operational efficiency and operational security of the banking institution.
<br>
Protecting the customers privacy in addition to their assets is important to protect the reputation of a financial institution. This is especially important for private banking institutions.
</p>
<p>
With «<i>AI vs AI</i>», I wanted to illustrate the current research topics we are working on today at NetGuardians to improve our algorithms further.
<br>
In a few words, we see today that cybercriminals are increasingly using advanced algorithms on their end to study the banks attack surface and discover means to attack the banks and their customers.
<br>
We are in a <i>"cat and mouse"</i> game where attackers attempt to counter the security systems put in place by banking institutions, which in their turn deploy new forms of algorithms and intelligence to protect them further.
<br>
I can only be looking forward to telling you more on this matter in a near future...
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/34b2d625-d85d-4dc7-82f5-d9174ca254ee">
<img class="centered" style="width: 600px;" alt="Consequences and benefits of AI - final improvements" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/34b2d625-d85d-4dc7-82f5-d9174ca254ee" />
</a>
</div>
<br>
<h2>9. Conclusion</h2>
<p>
Our own experience and conclusion with AI technologies and it's concrete application on our use cases is striking.
</p>
<p>
Introducing advanced algorithms, machine learning and advanced analytics techniques in our use cases has been key to help us improve the way we secure financial institutions and their customers.
<br>
We could:
</p>
<ul>
<li>
Reduce the fraud cases passing through and almost eliminate them.
</li>
<li>
Reduce the number of cases to be analyzed and make the detection system a lot more relevant.
</li>
<li>
Drastically reduce the amount of time required to investigate a case.
</li>
</ul>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/40057d11-9308-464e-b008-b3e3b9f332c5">
<img class="centered" style="width: 800px;" alt="Conclusion - Reminder on benefits" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/40057d11-9308-464e-b008-b3e3b9f332c5" />
</a>
</div>
<br>
<p>
Today, at our customers, Artificial Intelligence monitors every single interaction between individuals, both customers or employees, and the information system, to qualify their actions as legitimate or fraudulent, in addition to analyzing with highly sophisticated models financial transactions input in the system.
<br>
Today our reality is as follows: <b>Artificial intelligence monitors human behavior on a large scale to secure banks and their customers.</b>
</p>
<p>
But Science Fiction advances much faster than reality. Regarding artificial intelligence, the collective imagination, fed by Musk and Hollywood, is way ahead of reality.
<br>
In the public collective imagination, artificial intelligence today generates quite a lot of fantasies.
</p>
<p>
So let's agree on something if you do not mind.
<br>
If one calls <b>weak artificial intelligence</b>, a computer solution able to solve a problem in a strict context, to optimize a solution or a mathematical function, or to look for an answer to a question in a strict context, one calls a <b>strong artificial intelligence</b> an intelligence able to argue, contextualize or to show sensitivity or initiative.
<br>
If progress in weak artificial intelligence is today very fast and very impressive, we do not have the slightest little trace of a proof that would allow us to believe one day in the emergence of a strong artificial intelligence.
Strong artificial intelligence is science fiction.
</p>
<p>
The problem is that approach names like <i>Neural Network</i> are generating a lot of fantasy in the public imagination who takes this name literally.
<br>
With neural networks, the public imagines a digital brain, whereas the reality is that of <i>"matrices of convolutions"</i>, intensive iterative calculations carried out on gigantic numerical matrices. On the other hand, powerful technologies with less evocative names, <i>genetic algorithms</i>, <i>random forests</i> or <i>boosted gradient</i> raise less fantasies.
</p>
<h2>10. Artificial Intelligence vs. Augmented Intelligence</h2>
<p>
Today, these Artificial Intelligence techniques give the most impressive results when they support the human and not when they supplant it.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/554bf15d-f4d0-4014-a9e2-71af725d6a92">
<img class="centered" style="width: 600px;" alt="Artificial Intelligence vs. Augmented Intelligence" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/554bf15d-f4d0-4014-a9e2-71af725d6a92" />
</a>
</div>
<br>
<p>
Chess is one of the first areas in which computers started to beat humans.
<br>
The examples of algorithms that manage to defeat the great masters of chess in a regular if not systematic are legion.
<br>
But these are the so-called <i>"centaurs"</i>, most of the time amateur players, but helped by Artificial Intelligence, half-human, half-machines, who now win all the "freestyle" games.
</p>
<p>
I would like to mention a second example with an experience that has been performed last year.
<br>
Melanoma specialists have been asked to identify cancerous lesions based on photos of skin lesions.
<br>
These experts had a precision, a success rate of the order of 95%.
<br>
An AI based on a <i>Neural Network</i> deployed towards the same objective reached a pretty impressive 93% accuracy, yet failing to beat the experts.
<br>
But a set of interns, really rather students that actual doctors, accompanied and helped by an artificial intelligence have reached 97% accuracy, beating both Artificial Intelligence alone and experts
</p>
<p>
Today, the most impressive results of these technologies come from what is called <b>Augmented Intelligence</b>, when Artificial Intelligence intervenes in support of the human decision process and not to replace it.
<br>
And <i>Augmented intelligence</i> is exactly what we do at NetGuardians by providing bankers with the means to prevent fraud cases much more effectively.
</p>
<h2>11. AI Pillars at NetGuardians</h2>
<p>
The <b>key pillars</b> which enable us to deploy Artificial Intelligence technologies are as folows:
<p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/be5c5e61-8817-412a-8a97-faa964751874">
<img class="centered" style="width: 600px;" alt="Ai Pillars at NetGuardians" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/be5c5e61-8817-412a-8a97-faa964751874" />
</a>
</div>
<br>
<p>
All of this is pretty straightforward to understand.
<br>
I would just insist on two key notions:
</p>
<ul>
<li>
The ability to run these analyzes in real time. Be able to analyze the activity of bank customers and users in real time is at the root of the difference between preventing fraud and detecting fraud. It must be possible to work with very low processing times to characterize a transaction before it is placed on the market.
</li>
<li>
The user experience. The deployed algorithms can be as intelligent as one can imagine, if one is not able to provide investigators and analysts with clear, concise and precise information, allowing them to understand the context of the transaction and the reasons for the system to block it, all this does not work. Users reject the solution. Providing analysts with extremely intuitive and visual means to understand machine decisions is essential.
</li>
</ul>
<p>
(This article is available as a slideshare presentation here
<a href="https://www.slideshare.net/JrmeKehrli/artificial-intelligence-for-banking-fraud-prevention-95475760">https://www.slideshare.net/JrmeKehrli/artificial-intelligence-for-banking-fraud-prevention-95475760</a>)
</p>
https://www.niceideas.ch/roller2/badtrash/entry/video-of-jerome-kehrli-presenting
Presenting NetGuardians' Big Data technology (video)
Jerome Kehrli
2018-01-05T13:00:00-05:00
2018-06-29T10:57:33-04:00
<p>
I am presenting in this video NetGuardians' Big Data approach, technologies and its advantages for the banking institutions willing to deploy big data technologies for Fraud Prevention.
</p>
<!-- 1. The <iframe> (and video player) will replace this <div> tag. -->
<div id="player" style="max-width: 100%;"></div>
<script>
// 2. This code loads the IFrame Player API code asynchronously.
var tag = document.createElement('script');
tag.src = "https://www.youtube.com/iframe_api";
var firstScriptTag = document.getElementsByTagName('script')[0];
firstScriptTag.parentNode.insertBefore(tag, firstScriptTag);
// 3. This function creates an <iframe> (and YouTube player)
// after the API code downloads.
var player;
function onYouTubeIframeAPIReady() {
player = new YT.Player('player', {
height: '360',
width: '640',
videoId: 'wxESC2PCwp4',
events: {
'onReady': onPlayerReady,
'onStateChange': onPlayerStateChange
}
});
}
// 4. The API will call this function when the video player is ready.
function onPlayerReady(event) {
//event.target.playVideo();
}
// 5. The API calls this function when the player's state changes.
//var done = false;
function onPlayerStateChange(event) {
/*
if (event.data == YT.PlayerState.PLAYING && !done) {
setTimeout(stopVideo, 6000);
done = true;
}
*/
}
/*
function stopVideo() {
player.stopVideo();
}
*/
</script>
<p>
The speech is reported in textual form hereafter.
</p>
<p>
I am presenting in this video NetGuardians' Big Data approach, technologies and its advantages for the banking institutions willing to deploy big data technologies for Fraud Prevention.
</p>
<!-- 1. The <iframe> (and video player) will replace this <div> tag. -->
<div id="player_1231346" style="max-width: 100%;"></div>
<script>
player = new YT.Player('player_1231346', {
height: '360',
width: '640',
videoId: 'wxESC2PCwp4'
});
</script>
<p>
The speech is reported in textual form hereafter.
</p>
<p>
It keeps puzzling me to see how deploying Big Data Technologies in banking institutions for fraud prevention and other use cases seems to be so difficult.
<br>
A large number of such projects have simply failed over the past.
<br>
By failure I mean projects that, led to poor results, or exceeded the budget significantly, or even that have been simply cancelled.
</p>
<p>
When looking at why these projects failed, it always boils down to the two same major issues.
</p>
<p>
The first major issue is that extracting the required data to build the analytics use cases is a challenge on its own.
Let's say the bank managed to extract the required data, which is only a technical problem, but cleaning, enriching, normalizing and re-modeling it for banking fraud use cases is a whole new project.
</p>
<p>
The second major issue is that technological mastery alone is not sufficient for Big Data projects to succeed.
<br>
Implementing data analytics use cases requires a strong involvement from business experts.
<br>
It always amazes me to see so many projects had the illusion that putting a dozen of gifted Data Scientists in a room for a few years would be sufficient. Without a clear business understanding, Data Scientists are blind and can go nowhere.
</p>
<p>
And then even with a clear understanding of these both challenges, deploying big data technologies for fraud prevention is a 10 months to 2 years project.
At NetGuardians we typically deploy our technology at our new customers within a few weeks.
</p>
<p>
So how do we do that ?
</p>
<p>
First, we are using technology on the bleeding edge of the state of the art, not today's state of the art but tomorrow's, benefiting from the right data extraction approach and the right use cases.
</p>
<p>
In terms of technology, our NG|Screener platform is using key big data components underneath: ElasticSearch, Mesos and Spark.
</p>
<p>
Regarding the Data Ingestion System, we have developed at NetGuardians our Data Collection Framework that is simple, efficient and configurable.
<br>
Typical data extraction tools are either simple, or efficient or configurable. Our framework is all of that together, without any compromises.
</p>
<p>
Then, working with numerous financial institutions worldwide over the years made us understand the indispensable role of not only technology but also business expertise when it comes to developing Big Data analytics use cases.
</p>
<p>
Business experts in banking institutions are only hardly available, right ?
<br>
Not a problem for us, we have hired our own.
<br>
Today, we have our own business and risk experts with an impressive trackrecord in risk or other banking business departments.
</p>
<p>
At NetGuardians, we have this multi-competencies team that so many project struggle to build and together, we have designed and implemented the right use cases to make Big Data deployment projects happen smoothly at our customers, and bring them actual added value.
</p>
<p>
As a result, our customers are able to make sense of their available Big Data, save enormous amount of time, and implement the Big Data technology to proactively prevent growing fraud challenges.
</p>
<p>
From a personal perspective, I am utmost proud of what we have built, both in terms of technology and approach, as well as the privilege I have to work in a team with such brilliant minds and wonderful persons.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/the-agile-collection
The Agile Collection Book
Jerome Kehrli
2017-12-12T17:57:20-05:00
2017-12-13T06:50:10-05:00
<!-- The Agile Collection -->
<!-- agile-collection agile-methods agile agility -->
<p>
Agility in Software Development is a lot of things, a collection of so many different methods. <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-landscape">In a recent article</a> I presented the <i>Agile Landscape V3</i> from Christopher Webb which does a great job in listing these methods and underlying how much Agility is much more than some scrum practices on top of some XP principles.
<br>
I really like this infographic since I can recover <i>most-if-not-all</i> of the principles and practices from the methods I am following.
</p>
<p>
Recently I figured that I have written on this very blog quite a number of articles related to these very Agile Methods and after so much writing I thought I should assemble these articles in a book.
<br>
So here it is, <a href="https://www.niceideas.ch/the_agile_methods_collection.pdf">The Agile Methods Collection</a> book.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/5ad812bb-d038-4f3a-924d-b13af76941ca">
<img class="centered" style="width: 550px;" alt="The Agile Methods Tree" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/5ad812bb-d038-4f3a-924d-b13af76941ca" />
</a>
</div>
<br>
<p>
<a href="https://www.niceideas.ch/the_agile_methods_collection.pdf">The Agile Methods Collection</a> book is simply a somewhat reformatted version of all the following articles:
</p>
<ul>
<li><a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-landscape">Agile Landscape from Deloitte</a></li>
<li><a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-software-development-lessons-learned">Agile Software Development, lessons learned</a></li>
<li><a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-planning-tools-and-processes">Agile Planning : tools and processes</a></li>
<li><a href="https://www.niceideas.ch/roller2/badtrash/entry/devops-explained">DevOps explained</a></li>
<li><a href="https://www.niceideas.ch/roller2/badtrash/entry/lean-startup-a-focus-on">The Lean Startup - A focus on Practices</a></li>
<li><a href="https://www.niceideas.ch/roller2/badtrash/entry/periodic-table-of-agile-principles">Periodic Table of Agile Principles and Practices</a></li>
</ul>
<p>
So if you already read all these articles, don't download this book.
<br>
If you didn't so far or want to have a kind of reference on all the methods from the collection illustrated above, you might find this book useful.
<br>
I hope you'll have as much pleasure reading it than I had writing all these articles.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/deciphering-the-bengladesh-bank-heist
Deciphering the Bangladesh bank heist
Jerome Kehrli
2017-11-15T17:03:49-05:00
2020-09-03T10:15:31-04:00
<!-- Deciphering the Bengladesh bank heist -->
<p>
<b>The Bangladesh bank heist - or SWIFT attack - is one of the biggest bank robberies ever, and the most impressive cyber-crime in history</b>.
</p>
<p>
This is the story of a group of less than 20 cyber-criminals, composed by high profile hackers, engineers, financial experts and banking experts who gathered together to hack the worldwide financial system, by attacking an account of the central bank of Bangladesh, a lower middle income nation and one of the world's most densely populated countries, and steal around 81 million US dollars, successfully, after attempting to steal almost a billion US dollars.
</p>
<p>
In early February 2016, authorities of Bangladesh Bank were informed that about 81 million USD was illegally taken out of its account with the Federal Reserve Bank of New York using an inter-bank messaging system known as SWIFT.
The money was moved via SWIFT transfer requests, ending up in bank accounts in the Philippines and laundered in the Philippines' casinos during the chinese New-Year holidays.
</p>
<p>
Fortunately, the major part of the billion US dollars they intended to steal could be saved, but 81 million US dollars were successfully stolen and are gone for good.
</p>
<p>
The thieves have stolen this money without any gun, without breaking physically in the bank, without any form of physical violence. (There are victims though, there are always victims in such case, but they haven't suffered any form of physical violence)
<br>
These 81 million US dollars disappeared and haven't been recovered yet. The thieves are unknown, untroubled and safe.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/ac851db2-485d-496e-ab31-d040612d9a2f">
<img class="centered" style="width: 600px;" alt="Bengladesh bank hacker" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/ac851db2-485d-496e-ab31-d040612d9a2f" />
</a>
</div>
<br>
<p>
The Bangladesh bank heist consisted in hacking the Bangladesh central bank information system to issue fraudulent SWIFT orders to withdraw money from the banking institution. SWIFT is a trusted and closed network that bank use to communicate between themselves around the world. SWIFT is owned by the major banking institutions.
</p>
<p>
In terms of technological and technical mastery, business understanding, financial systems knowledge and timing, this heist was a perfect crime. The execution was brilliant, way beyond any Hollywood scenario. And the bank was actually pretty lucky that that the hackers didn't successfully loot the billion US dollars as they planned, but instead only 81 million.
<br>
As such, from a purely engineering perspective, studying this case is very exiting. First, I cannot help but admire the skills of the team of thieves team as well as the shape of the attack, and second, it's my job in my current company to design controls and systems preventing such attack from happening against our customers in the future.
</p>
<p>
In this article, I intend to present, explain and decipher as many of the aspects of the Bangladesh bank heist and I know.
</p>
<!-- Deciphering the Bengladesh bank heist -->
<p>
<b>The Bangladesh bank heist - or SWIFT attack - is one of the biggest bank robberies ever, and the most impressive cyber-crime in history</b>.
</p>
<p>
This is the story of a group of less than 20 cyber-criminals, composed by high profile hackers, engineers, financial experts and banking experts who gathered together to hack the worldwide financial system, by attacking an account of the central bank of Bangladesh, a lower middle income nation and one of the world's most densely populated countries, and steal around 81 million US dollars, successfully, after attempting to steal almost a billion US dollars.
</p>
<p>
In early February 2016, authorities of Bangladesh Bank were informed that about 81 million USD was illegally taken out of its account with the Federal Reserve Bank of New York using an inter-bank messaging system known as SWIFT.
The money was moved via SWIFT transfer requests, ending up in bank accounts in the Philippines and laundered in the Philippines' casinos during the chinese New-Year holidays.
</p>
<p>
Fortunately, the major part of the billion US dollars they intended to steal could be saved, but 81 million US dollars were successfully stolen and are gone for good.
</p>
<p>
The thieves have stolen this money without any gun, without breaking physically in the bank, without any form of physical violence. (There are victims though, there are always victims in such case, but they haven't suffered any form of physical violence)
<br>
These 81 million US dollars disappeared and haven't been recovered yet. The thieves are unknown, untroubled and safe.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/ac851db2-485d-496e-ab31-d040612d9a2f">
<img class="centered" style="width: 600px;" alt="Bengladesh bank hacker" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/ac851db2-485d-496e-ab31-d040612d9a2f" />
</a>
</div>
<br>
<p>
The Bangladesh bank heist consisted in hacking the Bangladesh central bank information system to issue fraudulent SWIFT orders to withdraw money from the banking institution. SWIFT is a trusted and closed network that bank use to communicate between themselves around the world. SWIFT is owned by the major banking institutions.
</p>
<p>
In terms of technological and technical mastery, business understanding, financial systems knowledge and timing, this heist was a perfect crime. The execution was brilliant, way beyond any Hollywood scenario. And the bank was actually pretty lucky that that the hackers didn't successfully loot the billion US dollars as they planned, but instead only 81 million.
<br>
As such, from a purely engineering perspective, studying this case is very exiting. First, I cannot help but admire the skills of the team of thieves team as well as the shape of the attack, and second, it's my job in my current company to design controls and systems preventing such attack from happening against our customers in the future.
</p>
<p>
In this article, I intend to present, explain and decipher as many of the aspects of the Bangladesh bank heist and I know.
</p>
<p>
(This article is available as a slideshare presentation here
<a href="https://www.slideshare.net/JrmeKehrli/deciphering-the-bengladesh-bank-heist">https://www.slideshare.net/JrmeKehrli/deciphering-the-bengladesh-bank-heist</a>)
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. Introduction</a></li>
<li><a href="#sec2">2. The SWIFT network - Key Concepts</a>
<ul>
<li><a href="#sec21">2.1 Correspondant Banking</a></li>
<li><a href="#sec21">2.2 Transfering Money</a></li>
<li><a href="#sec23">2.3 Application Architecture (Bengladesh case)</a></li>
<li><a href="#sec24">2.4 Introducing key SWIFT messages</a></li>
</ul>
</li>
<li><a href="#sec3">3. The attack</a>
<ul>
<li><a href="#sec31">3.1 Behaviour of the malware</a></li>
<li><a href="#sec32">3.2 Complete overview of the attack</a></li>
<li><a href="#sec33">3.3 Timeline of the attack</a></li>
<li><a href="#sec34">3.4 Laundering of the money</a></li>
</ul>
</li>
<li><a href="#sec4">4. Aftermath</a></li>
<li><a href="#sec5">5. Conclusion</a></li>
</ul>
<br>
<a name="sec1"></a>
<h2>1. Introduction</h2>
<p>
The Bangladesh Bank hack is one of the biggest bank heists in global financial history. There have been larger scams and scandals, but cyber heists from a single bank, this takes the cake.
</p>
<p>
The heist of over 80 million US dollars sent shock-waves through the global financial system and security experts scrambled to find out how it had happened. Political and administrative authorities played the blame game, as was expected of them. Resignations were offered and statements were issued. It was a complete chaos.
</p>
The hackers managed to break into the bank's security system and transferred more than 80 million USD from the New York Federal Reserve account to multiple bank accounts located in Sri Lanka and Philippines.
<br>
A significant number of transfer requests, 30 out of 35, were blocked by the Federal Reserve, saving the bank a loss of an additional 850 million US dollars
<br>
But the five requests that managed to pass through, amounting to more than a 80 million US dollars, were devastating enough in their consequences.
</p>
<p>
Perhaps the most troubling aspect of the whole episode was that the hackers managed to hack into the SWIFT software. SWIFT, lies at the heart of the global financial system and is a network which connects majority of the world's financial institutions and enables them to send and receive financial information about financial transactions.
</p>
<p>
However, It was the bank's own systems and controls that were compromised, not the SWIFT network connection software. The SWIFT software behaved as it was intended to, but was not operated by the intended person or process. This was really a bank problem, not a SWIFT problem.
</p>
<p>
In the next chapter, I will present the key concepts required to understand the attack before presenting the shape and timeline of the attack.
</p>
<a name="sec2"></a>
<h2>2. The SWIFT network - Key Concepts</h2>
<p>
Some key concepts about correspondent banking and the principles of the SWIFT network are required to grasp a basic understanding of the Bangladesh SWIFT attack. These are presented in this chapter.
</p>
<p>
SWIFT - Society for Worldwide Inter-bank Financial Telecommunication - is a belgian company, located in Belgium, and is a trusted and closed network used for communication between banks around the world. It is overseen by a committee composed of the US Federal Reserve, the Bank of England, the European Central Bank, the Bank of Japan and other major banks.
<br>
SWIFT is used by around 11 thousands institutions in more than 200 countries and supports around 25 million communications a day, most of them being money transfer transactions, the rest are various other types of messages.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a887fd53-655c-48a9-9397-6899371af97d">
<img class="centered" style="width: 150px;" alt="SWIFT logo" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a887fd53-655c-48a9-9397-6899371af97d" />
</a>
</div>
<br>
<p>
The majority of international inter-bank messages use the SWIFT network.
<br>
SWIFT does not facilitate funds transfer: rather, it sends payment orders, which must be settled by correspondent accounts that the institutions have with each other. For two financial institutions to exchange banking transactions, they must have a banking relationship beforehand.
</p>
<p>
A cool introduction to SWIFT is available on the Fin website : <a href="https://fin.plaid.com/articles/what-is-swift">https://fin.plaid.com/articles/what-is-swift</a>.
</p>
<a name="sec21"></a>
<h3>2.1 Correspondent Banking</h3>
<p>
<b>Correspondent Bank</b>
</p>
<p>
A correspondent bank is a financial institution that provides services on behalf of another financial institution.
<br>
It can facilitate wire transfers, conduct business transactions, accept deposits and gather documents on behalf of another financial institution.
<br>
Correspondent banks are most likely to be used by domestic banks to service transactions that either originate or are completed in foreign countries, acting as a domestic bank's agent abroad.
</p>
<p>
Generally speaking, the reasons domestic banks employ correspondent banks include:
</p>
<ul>
<li>limited access to foreign financial markets and the inability to service client accounts without opening branches abroad,</li>
<li>act as intermediaries between banks in different countries or as an agent to process local transactions for customers abroad,</li>
<li>accept deposits, process documentation and <b>serve as transfer agents for funds</b>. </li>
</ul>
<p>
The ability to execute these services relieves domestic banks of the need to establish a physical presence in foreign countries.
</p>
<p>
<b>NOSTRO / VOSTRO Account</b>
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/45ed631c-105f-4292-b5bc-bb9b839affd4">
<img class="centered" style="width: 750px;" alt="NOSTRO VOSTRO accounts" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/45ed631c-105f-4292-b5bc-bb9b839affd4" />
</a>
</div>
<br>
<p>
The accounts held between correspondent banks and the banks to which they are providing services are referred to as <i>NOSTRO</i> and <i>VOSTRO</i> accounts (latin words for <i>ours</i> and <i>yours</i>).
<br>
An account held by one bank for another is referred to by the holding bank as a <b>VOSTRON</b> account.
<br>
The same account is referred as a <b>NOSTRO</b> account by the owning bank (the customer). Generally speaking, both banks in a correspondent relationship hold accounts for one another for the purpose of tracking debits and credits between the parties.
</p>
<P>
NOSTRO and VOSTRO accounts are really to the same thing but from different perspectives. For example, Bank X has an account with Bank Y in Bank Y's home currency. To Bank X, that is a NOSTRO, meaning "our account on your books," while to Bank Y, it is a VOSTRO, meaning "your account on our books." These accounts are used to facilitate international transactions
</p>
<p>
<b>Transferring Money Using a Correspondent Bank</b>
</p>
<p>
International wire transfers often occur between banks that do not have an established financial relationship. When agreements are not in place between the bank sending a wire and the one receiving it, a correspondent bank must act as an intermediary. For example, a bank in Geneva that has received instructions to wire funds to a bank in Japan cannot wire funds directly without a working relationship with the receiving bank.
</p>
<p>
Most if not all international wire transfers are executed through SWIFT. Knowing there is not a working relationship with the destination bank, the originating bank can search the SWIFT network for a correspondent bank that has arrangements with both banks.
</p>
<p>
Interestingly, when a bank wants to send some funds to another bank on the other side of the world, it happens often that the sending bank has no banking relationships with any bank having itself a relationship with the target bank. In this case, the order needs to be transferred through several, sometimes many, distinct banking institutions through the SWIFT network.
<br>
These intermediate banks are called <b>routing banks</b>.
</p>
<a name="sec22"></a>
<h3>2.2 Transferring Money</h3>
<p>
In the scope of funds transfers, correspondent banking relationships often happen between commercial banks and central banks. This is especially useful when a bank has to process massive funds transfers in different currencies.
</p>
<p>
Imagine that a non-US commercial banking institution has to transfer a massive amount of US Dollars for one of its big customers to some other account in another financial institution abroad. It would be very inconvenient in this case to have to build up enough reserves of US dollars for this kind of transfers.
<br>
Instead of building such reserves, big commercial banks worldwide have the tendency to open a correspondent banking relationships with the US federal Reserve in New York - called the Fed - and use their VOSTRO accounts at the Fed to process such big transfers.
<br>
Such VOSTRO accounts have typically no limits and do not necessarily need to be credited beforehand. The settlement can happen afterwards, on a regular basis.
</p>
<p>
Let's illustrate this situation with 2 imaginary customers and the Bangladesh central bank:
<p />
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/22d1bc9c-05a4-4482-acf8-574726869c54">
<img class="centered" style="width: 610px;" alt="Correspondant bank money transfer" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/22d1bc9c-05a4-4482-acf8-574726869c54" />
</a>
</div>
<br>
<p>
In case the money transfer requested by a customer in a foreign currency (here USD for the Bangladesh bank) exceeds some limits or even the reserves of the Bangladesh bank in USDs, the bank decides to go through the VOSTRO account by the correspondent central bank related to the foreign currency (Here the US Fed for USD) and instruct it to proceed with the transfer.
<br>
Again, that VOSTRO account doesn't even necessarily need to be credited beforehand, the settlement can happen way later or even never.
</p>
<a name="sec23"></a>
<h3>2.3 Application Architecture (Bangladesh case)</h3>
<p>
SWIFT provides a centralized store-and-forward mechanism, with some transaction management. For bank A to send a message to bank B with a copy or authorization with institution C, it formats the message according to standard and securely sends it to SWIFT. SWIFT guarantees its secure and reliable delivery to B after the appropriate action by C. SWIFT guarantees are based primarily on high redundancy of hardware, software, and people.
</p>
<p>
<b>Principles</b>
</p>
<p>
SWIFT moved to its current IP network infrastructure, known as SWIFTNet, from 2001 to 2005, providing a total replacement of the previous X.25 infrastructure. During 2007 and 2008, the entire SWIFT Network migrated its infrastructure to a new protocol called SWIFTNet Phase 2.
<br>
<b>Today the SWIFT network can be seen as a highly secured private network over Internet</b>.
</p>
<p>
In order to have access to this network, a financial institution needs to obtain a SWIFT gateway running the SWIFT NetLink software. This is most of the time proprietary hardware running with Linux and requiring a physical security dongle storing cryptographic keys to access the network.
</p>
<p>
SWIFT also provides a whole lot of other software such as SWIFT Alliance Access that can be used by a financial institution to access the SWIFT network (always through the gateway) in a more convenient way and with higher level or simpler APIs.
</p>
<p>
<b>SWIFT Network Security</b>
</p>
<p>
SWIFT's security stems from two major sources. First, it's a private network, and most banks set up their accounts such that only certain transactions between particular parties are permitted. The network privacy means that it should be hard for someone outside a bank to attack the network, but if a hacker breaks into a bank-as was the case here-then that protection evaporates.
<br>
The Bangladesh central bank has all the necessary SWIFT software and authorized access to the SWIFT network. Any hacker running code within the Bangladesh bank also has access to the software and network.
</p>
<p>
<b>The Bangladesh bank architecture</b>
</p>
<p>
The Bangladesh central bank, at the time of the heist, was handling SWIFT connectivity from the <b>Banking Information System</b> to the <b>SWIFT network</b> using the specific <b>SWIFT Alliance Access</b> software running on a bridge server.
<br>
Alliance Access, integrated the way it was at the Bangladesh banks, was setup to <b>read/write SWIFT messages from/to files on the filesystem</b> and record transaction information in an Oracle database. In addition, confirmation and reconciliation messages were handled through a manual process after being sent to <b>a printer</b>.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/28997370-2fe2-4d74-be10-37be9e8bf8a4">
<img class="centered" style="width: 610px;" alt="SWIFT Application Architecture" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/28997370-2fe2-4d74-be10-37be9e8bf8a4" />
</a>
</div>
<br>
<p>
Again, the order reconciliation process in the Bangladesh central bank was a largely manual process.
<br>
In addition, passing through the filesystem to integrate the Banking Information System and the SWIFT network is a huge security weakness. We'll discuss this later.
</p>
<a name="sec24"></a>
<h3>2.4 Introducing key SWIFT messages</h3>
<p>
As we will see in the next chapter, the hackers created a malware that generated and manipulated key SWIFT messages to withdraw the money from the Bangladesh Central Bank's VOSTRO account at the Fed.
</p>
<p>
SWIFT messages consist of five blocks of the data including three headers, message content, and a trailer. Message types are crucial to identifying content.
<br>
All SWIFT messages include the literal "MT" (Message Type). This is followed by a three-digit number that denotes the message category, group and type
</p>
<p>
The key SWIFT messages in question here were of the following types:
</p>
<ul>
<li><b>MT103</b> is used for cash transfer specifically for cross border/international wire transfer.</li>
<li><b>MT202</b> is the general Financial Institution transfer order, used to order the movement of funds to the beneficiary institution.</li>
<li><b>MT950</b> is the Final Statement report on all settlement operations with specified account within a current business day. Can be seen as the confirmation for an MT 202.</li>
</ul>
<p>
The workflow between these messages can be seen as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d1c08dbe-c7e9-453e-8fcb-4c49bd74f832">
<img class="centered" style="width: 850px;" alt="SWIFT Messages" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d1c08dbe-c7e9-453e-8fcb-4c49bd74f832" />
</a>
</div>
<br>
<p>
A lot of stuff goes through SWIFT, here we focus on a really small subset of the supported message tapes, those related to transferring money.
<br>
First, the MT103 is an information message, basically announcing that a target counterparty account will receive money from an internal (or correspondent) account to be debited.
<br>
Then the MT202 is the inter-bank transfer order, it applies globally to transfer money from a banking institution to another banking institution, covering all individual account level transfer announces relating to the same target institution. There is a relationship between MT103 and MT202, they cross-reference each others (field 21 in MT103 reference MT202 and field 20 in MT202 references all related MT103)
<br>
Finally the MT950 is the extract that confirms all executed order. It is the bible and references all positions confirmed and executed by the correspondent bank. It is often an end of day extract used by banking institutions for reconciliations.
</p>
<p>
Again, when it comes to SWIFT messages processing from and to the Banking information System in the Bangladesh case, the important aspects here are:
</p>
<ul>
<li>SWIFT messages originating from the Banking Information System simply needed to be put on the filesystem somewhere to be <i>"slurped"</i> by the SWIFT Alliance Access software integrated the way is was integrated at Bangladesh Central Bank. In terms of security of the process, this is more that questionable.</li>
<li>Confirmation messages back from the SWIFT network were stored and printed. Reconciliation is a manual process from these printed messages. This is quite unusual (we'll get back to that)</li>
</ul>
<a name="sec3"></a>
<h2>3. The Attack</h2>
<p>
<b>Summary of the attack</b>
</p>
<p>
Capitalizing on weaknesses in the security of the Bangladesh Central Bank, hackers attempted to steal around a billion US dollars from the Bangladesh central bank's VOSTRO account with the US Federal Reserve Bank between February 4 and 5 when Bangladesh Bank's offices were closed.
</p>
<p>
The perpetrators managed to compromise Bangladesh Bank's computer network, observed how transfers were done, and gained access to the bank's credentials for payment transfers.
<br>
They used these credentials to authorize about three dozen requests to the Federal Reserve Bank of New York to transfer funds from the Bangladesh Bank VOSTRO to accounts in Sri Lanka and the Philippines.
</p>
<p>
Thirty transactions worth 851 million USD were flagged by the banking system for staff review, but five requests were granted; 20 million USD to Sri Lanka (later recovered), and 81 million USD lost to the Philippines, entering the Southeast Asian country's banking system on February 5, 2016.
<br>
This money was laundered through casinos and a little later transferred to Hong Kong.
</p>
<p>
The attack is impressive and stands out on various levels, in terms of technical means, maturity and complexity for several reasons:
</p>
<ol>
<li><b>Technical Mastery</b>: the usage of a custom worm (filename evtdiag.exe) to hack the SWIFT Bridge by the bank and likely other malwares to capture the administration credentials</li>
<li><b>In-depth understanding of the worldwide financial market</b>: both the attack shape and the money laundering scheme prove in-depth understanding of financial markets</li>
<li><b>SWIFT knowledge</b>: in-depth Knowledge of the SWIFT messaging details is not that much spread among software engineers</li>
</ol>
<p>
Let's see how this is proven by analyzing the details of the attack.
</p>
<a name="sec31"></a>
<h3>3.1 Behaviour of the malware</h3>
<p>
The hackers used a custom version of malware to hack software called SWIFT Alliance Access to both make the transactions and hide the evidence. The hackers used a version of the malware that removed integrity checks within the Alliance software and then monitored the transaction files sent through the system, searching the payment orders and confirmations for specific terms. These terms and the responses to them were specified by a <i>Command and Control</i> server in Egypt.
</p>
<p>
When a message with one of the search terms was found, the malware would do different things depending on the kind of message. Payment orders were modified to increase the amounts being moved, updating the Alliance database with new values. Confirmation messages from the SWIFT network were also modified. Confirmations are printed and stored in the database. Before being printed, the malware would alter the confirmations to show the original, correct transaction value; it also deleted confirmations from the Alliance database entirely.
</p>
<p>
It's still not clear how the initial transactions were entered into the system to trigger the malware in the first place.
</p>
<p>
Again, the SWIFT network key components haven't been compromised, the malware was targeting the Bangladesh Central Bank's own bride to the SWIFT infrastructure running the SWIFT Alliance Access Software:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/4a70014d-d9be-418d-a500-be0b0a7246b9">
<img class="centered" style="width: 610px;" alt="Worm infection on the Bank bridge to SWIFT" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/4a70014d-d9be-418d-a500-be0b0a7246b9" />
</a>
</div>
<br>
<p>
If an organization can't keep its endpoint secure, it leaves itself very vulnerable to being electronically robbed. That was the case here.
<br>
The bank lacked any firewalls and was using second-hand $10 switches on its network. These switches did not allow for the regular LAN to be segmented or otherwise isolated from the SWIFT systems. The lack of network security infrastructure has hindered the investigation.
<br>
It's still not known how the hackers penetrated the network, but it looks like the bank didn't make it difficult for them to do so.
</p>
<p>
How the attackers obtained administration credentials is still unclear. They might have obtained these credentials by using another malware or by exploiting a remotely available vulnerability (not impossible considering the weak security practices in place in the Bangladesh Central Bank) or it might also have been an insider job.
<br>
So far there are only speculations in this regards.
</p>
<p>
<b>Forging fraudulent SWIFT messages</b>
</p>
<p>
Simplifying a bit the reality, we can picture the malware as forging fraudulent SWIFT messages as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/63d62d1c-e899-4b82-a2ce-3aaa9e39afbc">
<img class="centered" style="width: 850px;" alt="Worm forging SWIFT messages" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/63d62d1c-e899-4b82-a2ce-3aaa9e39afbc" />
</a>
</div>
<br>
<p>
The view above is a simplification of the reality. Actually, the worm was brilliantly implemented since forging from scratch consistent SWIFT announces (MT103) and Money Transfer Orders (MT202) messages would have been more difficult.
<br>
Instead, the worm was tampering with genuine messages issued by the Banking Information System and changing the amounts and recipient. This is a lot easier than blank forging.
<br>
It is still unclear for now if the initial untampered messages were simply authentic and relevant messages, perhaps duplicated by the worm, or forged through other malwares on different systems. I couldn't find clear information in this regards in all that has been published (if a reader has additional information in this regards, I would be happy to learn about it).
</p>
<p>
Just as a sidenote, whenever an institution such as the Bengladesh central bank sends a SWIFT funds transfer order, it's always in behalf of one of its customer. The SWIFT message(s) indicates the customer for which the bank requests a funds transfer.
<br>
Now of course the target correspondent bank cannot know if such customer exist, it doesn't have access to the list of customers of the sending bank. The SWIFT messages tampered by the worm could have been related to any random customer of the Bengladesh bank, this doesn't matter.
<br>
The only important aspect was that the beneficiary account and banking institution were the ones intended by the attackers.
</p>
<p>
<b>Intercepting SWIFT confirmations</b>
</p>
<p>
Here as well, by simplifying a little the reality, we can picture the malware as intercepting SWIFT confirmations as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/27dfdbe0-56d6-4f47-a1ba-74085f6c23f6">
<img class="centered" style="width: 850px;" alt="Worm forging SWIFT messages" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/27dfdbe0-56d6-4f47-a1ba-74085f6c23f6" />
</a>
</div>
<br>
<p>
The malware was also developed in such a way that it was intercepting confirmation messages (MT950) back from the Fed (from the SWIFT network in fact). Confirmation of genuine orders were supposed to be allowed to pass through untampered while confirmation of fraudulent messages were supposed to be intercepted and hidden.
<br>
But the worm was buggy and while tampering with confirmations sent to the printer, it corrupted them somehow which caused the printer to crash. We'll get back to that later.
</p>
<p>
Interestingly, going as far as trying to tamper with confirmations was pure genius (even though it didn't work as expected). Had it worked, the bank might well have noticed the attack weeks after the facts since on both sides of the world (the Fed view and the Bengladesh Central bank view), positions would have been very different but yet consistent, the Fed knowing about the orders, taking them as genuine and the Bangladesh bank would have known nothing about them.
<br>
Also, one should note that transfer orders (MT202) are executed immediately. So trying to tamper with confirmations was not intended to give more chance to the transfer to succeed, it was really intended as a way to hide the theft until hopefully after the money is laundered.
</p>
<p>
<b>The Malware</b>
</p>
<p>
The malware, codenamed Dridex (Addendum 2020-Sep-03 - see note at the end of the article), filenamed <code>evtdiag.exe</code>, was designed to hide the hacker's tracks by changing information on a SWIFT database within Alliance Access and contained the IP address of a server in Egypt the attackers used to monitor the use of the SWIFT system by Bangladesh Bank staff.
<br>
It was likely part of a broader attack toolkit that was installed after the attackers obtained administrator credentials.
</p>
<p>
The malware was compiled close to the date of the heist, contained detailed information about the bank's operations and was uploaded from Bangladesh.
<br>
While that malware was specifically written to attack the Bangladesh Bank, the general tools, techniques and procedures used in the attack may allow the gang to strike again and as a matter of fact there have been attempts discussed by Reuters.
</p>
<p>
The malware was designed to make a slight change to the code of the Access Alliance software installed at the Bangladesh Central Bank, giving attackers the ability to modify a database that logged the bank's activity over the SWIFT network.
<br>
Once it had established a foothold, the malware could delete records of outgoing transfer requests altogether from the database and also intercept incoming messages confirming transfers ordered by the hackers.
<br>
It was also able to manipulate account balances on logs to prevent the heist from being discovered until after the funds had been laundered. Additionnaly, it manipulated the stream of confirmations sent to a printer that produced hard copies of transfer requests so that the bank would not identify the attack through those printouts.
<br>
This part went wrong and led the printer to crash.
</p>
<p>
More information on the malware is available <a href="https://d3pakblog.wordpress.com/2017/01/31/bangladesh-bank-heist-2016/">in this article</a> and <a href="https://asamborski.github.io/cs558_s17_blog/2017/03/23/bangladesh.html">this one</a>.
</p>
<a name="sec32"></a>
<h3>3.2 Complete overview of the attack</h3>
<p>
On February 4, likely after months of preparation, organizationally and technically, gaining access to the systems, developing the custom worm, obtaining credentials, infecting the systems, etc. unknown hackers sent more than three dozens of fraudulent money transfer requests to the Federal Reserve Bank of New York asking the bank to transfer millions of the Bangladesh Bank's VOSTRO account to bank accounts in the Philippines, Sri-Lanka and other parts of Asia.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/56d7365b-e516-4b9e-953a-4a42e96088bd">
<img class="centered" style="width: 850px;" alt="Overview of the attack" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/56d7365b-e516-4b9e-953a-4a42e96088bd" />
</a>
</div>
<br>
<p>
The hackers managed to get 81 million USD sent to Rizal Commercial Banking Corporation (RCBC) in the Philippines via four different transfer requests and an additional 20 million USD sent to Pan Asia Banking in a single request.
</p>
<p>
Fortunately 850 million USD in other transactions managed to be saved initially (thx to the Fed).
</p>
<p>
The 81 million USD was deposited into three accounts at a Rizal branch in Manila on Feb. 4. These accounts had all been opened a year earlier in May 2015, but had been inactive with just 500 USD sitting in them until the stolen funds arrived in February this year.
</p>
<p>
Another 20 million USD intended to be sent to Pan Asia Banking have been blocked later in the correspondent bank funds transfer routing process (thx to Deutsche Bank).
</p>
<p>
But the 81 million USD that went to Rizal Bank in the Philippines was gone. It had already been credited to multiple accounts, reportedly belonging to casinos in the Philippines, and all but 68 thousands USD of it was withdrawn on February 5 and 9 before further withdrawals were halted.
<br>
The stolen funds from Bangladesh Bank were transferred to money transfer company Philrem Services Corporation. Philrem converted into pesos some of the 81 million USD and delivered the money in cash tranches to a registered casino junket operator named Weikang Xu, Eastern Hawaii Leisure Company, and Bloomberry Hotels Incorporated (Solaire Resort & Casino).
</p>
<p>
The hackers might have stolen much more but most transfers have fortunately been stopped by the Fed and one transfer has been stopped by Deutsche Bank.
</p>
<p>
<b>Deutsche Bank saved 20 million USD</b>
</p>
<p>
Four requests to transfer a total of about 81 million USD to the Philippines went through, but the fifth one to transfer 20 million USD to a Sri Lankan non-profit organization was held up because the hackers misspelled the name of a non-existent NGO, Shalika Foundation, by writing "fandation" instead of "foundation".
</p>
<p>
This prompted the Deutsche Bank, a routing bank, to seek clarification from the Bangladesh central bank, thereby stopping the transaction.
</p>
<p>
<b>The Fed saved 850 million USD</b>
</p>
<p>
The Federal Reserve Bank did not execute 30 of the 35 transfers, worth around 851 million USD, officially due to "lack of details." These thirty transactions were flagged by the banking system for staff review.
</p>
<p>
The Fed was still tricked into paying out 101 million USD. But the losses could have been much higher had the name <b>Jupiter</b> not formed part of the address of a Philippines bank where the hackers sought to send hundreds of millions of dollars more.
<br>
By chance, <i>Jupiter</i> was also the name of an oil tanker and a shipping company under United States' sanctions against Iran. That sanctions listing triggered concerns at the New York Fed and spurred it to scrutinize the fake payment orders more closely.
</p>
<p>
It was a "total fluke" that the New York Fed did not pay out the 951 million USD requested by the hackers. There is no suggestion the oil tanker or shipping company was involved in the heist.
<br>
The Reuters examination has also found that the payment orders sent by the hackers were exceptional in several ways. They were incorrectly formatted at first; they were mainly to individuals; and they were very different from the usual run of payment requests from Bangladesh Bank.
<br>
Yet it was the word Jupiter that set the loudest alarm bells ringing at the New York Fed.
</p>
<p>
<b>The printer error</b>
</p>
<p>
A printer "error" helped Bangladesh Bank discover the heist. The bank's SWIFT bridge (running Alliance Access) was configured to automatically print out confirmations back from correspondent banks.
<br>
The printer works 24 hours so that when workers arrive each morning, they check the tray for transfers that got confirmed overnight.
<br>
But on the morning of Friday February 5, the director of the bank found the printer tray empty. When bank workers tried to print the reports manually, they couldn't. The software on the terminal that connects to the SWIFT network indicated that a critical system file was missing or had been altered.
<br>
The problem is deemed to be an unwanted bug with the worm, a failure in the attack if one likes, since the worm was programmed to remove confirmation of fraudulent payments from the confirmation stream being sent to the printer.
<br>
Fortunately, in this case, the Fed clarification requests and the Deutsche Bank request would have anyway alerted the bank, so even if the worm had functioned correctly, the bank would have been made aware of the attack.
</p>
<p>
When they finally got the software working the next day and were able to restart the printer, dozens of suspicious transactions spit out. The Fed bank in New York had apparently sent queries to Bangladesh Bank questioning dozens of the transfer orders, but no one in Bangladesh had responded.
<br>
Panic ensued as workers in Bangladesh scrambled to determine if any of the money transfers had gone through - their own records system showed that nothing had been debited to their account yet - and halt any orders that were still pending.
<br>
They contacted SWIFT and New York Fed, but the attackers had timed their heist well; because it was the weekend in New York, no one there responded. It wasn't until Monday that bank workers in Bangladesh finally learned that four of the transactions had gone through amounting to 101 million USD.
</p>
<p>
This article on the Fin website is magnificent and gives a lot of additional and cool information on the attack: <a href="https://fin.plaid.com/articles/anatomy-of-a-bank-heist">https://fin.plaid.com/articles/anatomy-of-a-bank-heist</a>.
</p>
<a name="sec33"></a>
<h3>3.3 Timeline of the attack</h3>
<p>
In every movie about bank robberies, timing is presented as critical. Here as well timing has been an essential concern and brilliantly mastered by the attackers.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/f241e7dc-6d4c-4f7e-932a-e8c90bb3d6a4">
<img class="centered" style="width: 850px;" alt="Timeline of the attack" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/f241e7dc-6d4c-4f7e-932a-e8c90bb3d6a4" />
</a>
</div>
<br>
<p>
In details:
</p>
<ul>
<li>
<b>May 15, 2015:</b> three dollar bank accounts in the Jupiter, Makati branch of the Rizal Commercial Banking Corporation (RCBC) were opened under the names of Enrico Teodoro Vasquez, Alfred Santos Vergara, Michael Francisco Cruz and Jessie Christopher Lagrosas, with an initial deposit of 500 USD each. These accounts, which were later found to be fake, remained idle until February 4, 2016.
</li>
<li>
<b>January 2016:</b> The hackers installed the malware on the bank's system some time in January, not long before they initiated the bogus money transfers on February 4. This was brilliant as well since installing it too soon might have made it detected before the heist and installing it too late might not have enabled them to assess its behaviour.
</li>
<li>
<b>February 4, 2016: </b> Under control of the hackers, the malware broke into Bangladesh Bank's VOSTRO account with the Federal Reserve Bank of New York, ordering 35 transfers worth 951 million USD, bulk of which to be transferred to RCBC Jupiter branch.
<br>
The Fed managed to detect and block 30 fraudulent transactions but 5 transfers worth 101 million USD haven't been blocked.
</li>
<li>
<b>February 5, 2016: </b> The fed tried to contact the Bangladesh Bank to get an explanation about these transfers, including the 5 non blocked.
<br>
<b>But Feb 5 was a banking holiday in the Bengladesh</b> and nobody could answer.
</li>
<li>
<b>February 5 to February 8, 2016: </b> The 5 transfer are executed by the correspondent banks and the routing banks.
<br>
One transaction of 20 million USD has been salvaged. This was after an instruction to a fake Sri Lankan foundation was put on hold by Deutsche Bank, one of the routing bank, because of a typographical mistake.
<br>
But the remaining 81 million USD stolen funds found their way to 4 fake bank accounts in RCBC.
</li>
<li>
<b>February 8, 2016:</b> Bangladesh Bank sent a "stop payment" order to RCBC. The request means the central bank was asking to refund the stolen funds or freeze the funds if these were not transferred yet.
<br>
<b>February 8 is a Chinese New Year non-working holiday for the Philippines.</b>
</li>
<li>
<b>February 9:</b> RCBC received a SWIFT code from Bangladesh Bank requesting for a refund or putting it on hold if the funds had been transferred or freeze them for proper investigation.
<br>
Despite the "stop payment" order, RCBC Jupiter branch still allowed withdrawals from the accounts.
<br>
Money was then consolidated and deposited in a dollar account of William So Go of DBA Centurytex Trading, which was opened on the same day.
<br>
In the following days the money was laundered in the Casinos.
</li>
</ul>
<p>
The timing was perfect. A unique Week-End preceded by a business holiday in Bengladesh and followed by a the Chinese New Year holiday in the Philippines, an ideal situation.
<nr>
The Fed couldn't get the required clarification from Bangladesh Bank on the next day and as such didn't attempt to recover the 5 orders that passed immediately.
<br>
On Monday, the stop orders sent by the Bangladesh bank couldn't be processed by the RCBC to freeze the funds since it was a banking holiday in Philippines.
<br>
In addition, the chinese new-year and the volume of exchanges in casinos at such occasion made the laundering of the money straightforward, not to mention the Philippines weak AML laws and practices.
</p>
<a name="sec34"></a>
<h3>3.4 Laundering of the money</h3>
<p>
Getting the money out is also difficult. Here the laundering scheme was both easy and magnificent. Money has been laundered through the Philippines.
</p>
<p>
The 81 million USD that was successfully stolen was sent to the Philippines to accounts at the Rizal Commercial Banking Corp (RCBC) held by two Chinese nationals who organize gambling junkets in Macau and the Philippines. The money was moved to several Philippine casinos and then subsequently to international bank accounts.
<br>
Laundering money in a Casino is fairly straightforward. One just need to touch the money as chips at any counter, loose one at any random game and then pretend one's lost enough and put it back in another account. Boom, laundered. The banking institution behind the account one withdrawn money from assumes it's been lost gambling while the other banking institution behind the account the money is put back one assumes it's been won in the Casino.
</p>
<p>
Philippine casinos are exempted of anti-money laundering law that requires them to report suspicious transactions, making them an attractive target for this kind of crime.
<br>
Plus can you imagine the amount of money transferred, spent, won and lost in Philippine's Casinos during Chinese New-Year ?
<br>
The volumes and the kind of operations makes everything absolutely untraceable.
</p>
<p>
Bamm ! Done. Money Laundered.
</p>
<a name="sec4"></a>
<h2>4. Aftermath</h2>
<p>
<b>What Does the Heist Mean?</b>
</p>
<p>
Even if the hackers didn't compromise the SWIFT network itself, such that all of SWIFT banks were vulnerable, it's still bad news for the global banking process. By targeting the methods that financial institutions use to conduct transactions over the SWIFT network, the hackers undermine a system that until now had been viewed as stalwart.
</p>
<p>
<b>Who's to Blame?</b>
</p>
<p>
Honestly only the attackers are really to blame.
<br>
But still, without such amazing security weaknesses in the Bengladesh Central Bank and with better control and stricter procedures in place at the Fed, the attack would not have been possible.
<br>
Of course, the Bangladesh Bank blames the Fed for allowing the money transfers to go through instead of waiting for confirmation from Bangladesh. The Fed counters that it contacted the bank to question and verify dozens of suspicious transfers and never got a response. Authorities at the Fed said that workers followed the correct procedures in approving the five money transfers that went through and blocking 30 others. Bangladesh Bank says the Fed bank should have blocked all money transfers until it got a response on the ones it deemed suspicious. And so on ...
</p>
<p>
<b>The Bengladesh Bank</b>
</p>
<p>
Aside from the loss of money, the Central Bank's governor, Atiur Rahman has resigned due to the incident. The bank promised to improve their cyber-security and ensure this kind of bank heist is prevented in the future.
</p>
<p>
<b>The Fed</b>
</p>
<p>
The immediate result of the breach for the New York Fed is a claim from the Bangladesh Bank for payment of lost funds and a potential lawsuit.
</p>
<p>
The Fed focused security resources on other priorities, such as preventing money-laundering and enforcing U.S. economic sanctions, officials with knowledge of the bank's security operations told Reuters. Fed officials took some comfort in the fact that SWIFT's security software had never been cracked.
</p>
<p>
The Bengladesh heist forced the Fed to invest massively in Fraud prevention solutions and better transaction monitoring systems.
</p>
<p>
<b>Philrem services</b>
</p>
<p>
The Philippine central bank has revoked the license of a remittance company that anti-money laundering investigators said was used to transfer some of the 81 million USD hackers looted from the Bangladesh central bank.
<br>
The Anti-Money Laundering Council (AMLC) issued a complaint against Philrem Service Corporation on April 28, accusing it of creating a fog around transactions and washing the stolen funds via a web of transfers and currency conversions through Philippine bank accounts, before moving the cash through casinos in Manila and junket operators.
</p>
<p>
<b>The Philippines</b>
</p>
<p>
The Philippines' involvement in the 100 million USD Bangladesh Bank heist, which has risked its return to the FATF gray list, showed the urgency of putting more teeth into the Anti-Money Laundering Act (AMLA).
</p>
<p>
The law, which was first introduced in 2001, left casinos out of the list of entities required to report suspicious transactions to the AMLC. There were efforts in the Senate to include this provision in the amended AMLA in 2013, but
this was blocked by some lawmakers and casinos lobbies.
</p>
<a name="sec5"></a>
<h2>5. Conclusion</h2>
<p>
If the hackers had indeed managed to get away with the terrifyingly large amount of 1 billion USD, this would have easily been the biggest bank heist in history, not to mention cyber heist.
</p>
<p>
Interestingly, these kinds of attacks will be increasingly common and if banks aren't updating their security processes and maintaining their network infrastructures, and the success rates of these attacks will only go up. Worse still, if hackers have access to banks and can manipulate funds, any businesses that partner with those banks is also at risk.
</p>
<p>
Imagine the following, if the worm had functionned correctly and not blocked the printer, if the Deutsche bank didn't find the typo, if the Fed didn't become suspicious because of the Jupiter keyword, the attack might have been a complete success. Not only the attackers would have successfully withdrawn almost one billion US dollars from the Bangladesh bank VOSTRO account at the Fed, but the attack might have been noted only weeks or months after the facts.
</p>
<p>
Finally, imagine that the same attack succeeds against a american or an european bank. In the US and in Europe, the SWIFT interfaces are integrated in an STP (Straight Through Processing) way. There is no such thing as manual reconciliation from some papers printed on a printer. The handling of confirmations and position reconciliation is mostly completely automated.
<br>
As such, the same attack succeeding in Europe for instance might take months to be discovered and uncovered, only a the moment the big position reconciliations between NOSTRO and VOSTRO accounts in correspondent banks are triggered.
</p>
<p>
And this is where it gets really funny. Everybody always had the illusion that SWIFT was so secure, so sure. It gave banking institutions worldwide the illusion that everything related to SWIFT is just as secure. But if the network itself is pretty secure indeed, the specific bridges and interfaces linking the Banking Information Systems to SWIFT can be very weak, as shown by the Bangladesh Heist.
<br>
Today european and US banking institutions and central banks are very worried and investigating transaction monitoring and security solutions to prevent such misadventure to happen to them.
<br>
Again the same attack in Europe would be a much bigger disaster.
</p>
<p>
Now another funny story to conclude this article: imagine a similar hack between two banks in Europe and imagine that one of them suspects something ... They would use SWIFT again to reconcile their views of the truth (MT109 and MT999).
<br>
These messages can be hacked just as well, in which case the theft may remain uncovered for months.
<br>
This is really hilarious.
</p>
<p>
(This article is available as a slideshare presentation here
<a href="https://www.slideshare.net/JrmeKehrli/deciphering-the-bengladesh-bank-heist">https://www.slideshare.net/JrmeKehrli/deciphering-the-bengladesh-bank-heist</a>)
</p>
<div class="centering">
<div class="centered" style="border: 1px solid #AAAAAA; width: 700px; text-align: left;" >
<p>
<b>Edit Sep 3rd, 2020</b>
</p>
<p>
At the time of writing of this article, it was a common belief in the group of people I was discussing this with that the main worm used for the heist (EvtDiag filename) was a custom version of the Dridex worm.
<br>
It has been brought to my attention that nothing confirm this hypothesis, on the contrary multiple experts and studies show EvtDiag had nothing to do with Dridex.
<br>
So even though some other sources are clearly pointing Dridex as (one of) the the worm(s) used in the attack (e.g. <a href="http://www.straitstimes.com/business/dridex-malware-linked-to-bangladesh-heist">http://www.straitstimes.com/business/dridex-malware-linked-to-bangladesh-heist</a>), this should be taken cautiously.
</p>
</div>
</div>
https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana2
ELK-MS - ElasticSearch/LogStash/Kibana - Mesos/Spark : a lightweight and efficient alternative to the Hadoop Stack - part III : so why is it cool ?
Jerome Kehrli
2017-08-30T16:43:44-04:00
2017-10-02T12:15:12-04:00
<p>
So, finally the conclusion of this serie of three articles, the big conclusion, where I intend to present why this ELK-MS, ElasticSearch/LogStash/Kibana - Mesos/Spark, stack is cool.
<br>
Without any more waiting, let's give the big conclusion right away, using ElasticSearch, Mesos and Spark can really distribute and scale the processing the way we want and very easily <b>scale the processing linearly with the amount of data to process</b>.
<br>
And this, exactly this and nothing else, is very precisely what we want from a Big Data Processing cluster.
</p>
<p>
At the end of the day, we want a system that books a lot of the resources of the cluster for a job that should process a lot of data and only a small subset of these resources for a job that works on a small subset of data.
<br>
And this is precisely what one can achieve pretty easily with the ELK-MS stack, in an almost natural and straightforward way.
<br>
I will present why and how in this article.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/big_archi_data.png">
<img class="centered" style="width: 400px; " alt="ELK-MS Data Architecture" src="https://www.niceideas.ch/es_spark/images/big_archi_data.png" />
</a>
</div>
<br>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana">The first article - ELK-MS - part I : setup the cluster</a> in this serie presents the ELK-MS stack and how to set up a test cluster using the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS.tar.gz">niceideas ELK-MS package</a>.
</p>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1">The second article - ELK-MS - part II : assessing behaviour</a> presents a few concerns, assesses the expected behaviour using the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS_TEST.tar.gz">niceideas ELK-MS TEST package</a> and discusses challenges and constraints in this ELK-MS environment.
</p>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana2">This third and last article - ELK-MS - part III : so why is it cool?</a> presents, as indicated, why this ELK-MS stack is really really cool and works great.
</p>
<p>
This article assumes a basic understanding of Big Data / NoSQL technologies in general by the reader.
</p>
<!-- ELK-MS - ElasticSearch/LogStash/Kibana - Mesos/Spark : a lightweight and efficient alternative to the Hadoop Stack - part III : so why is it cool ? -->
<p><i>
<b>Edited 2017-10-30</b>: I was using ES 5.0.0 with Spark 2.2.0 at the time of writing the initial version of this article.
<br>
With ElasticSearch 6.x, and ES-Hadoop 6.x, the game changes a little. The Spark 2.2.0 Dynamic allocation system is now perfectly compatible with the way ES-Hadoop 6.x enforces data locality optimization and everything works just as expected.
</i>
</p>
<p>
So, finally the conclusion of this serie of three articles, the big conclusion, where I intend to present why this ELK-MS, ElasticSearch/LogStash/Kibana - Mesos/Spark, stack is really really cool.
<br>
Without any more waiting, let's give the big conclusion right away, using ElasticSearch, Mesos and Spark can really distribute and scale the processing the way we want and out of the box /(using Dynamic Allocation) <b>scale the processing linearly with the amount of data to process</b>.
<br>
And this, exactly this and nothing else, is very precisely what we want from a Big Data Processing cluster.
</p>
<p>
At the end of the day, we want a system that books a lot of the resources of the cluster for a job that should process a lot of data and only a small subset of these resources for a job that works on a small subset of data, with a strong enforcement of data locality optimization.
<br>
And this is precisely what one can achieve pretty easily with the ELK-MS stack, in an almost natural and straightforward way.
<br>
I will present why and how in this article.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/big_archi_data.png">
<img class="centered" style="width: 400px; " alt="ELK-MS Data Architecture" src="https://www.niceideas.ch/es_spark/images/big_archi_data.png" />
</a>
</div>
<br>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana">The first article - ELK-MS - part I : setup the cluster</a> in this serie presents the ELK-MS stack and how to set up a test cluster using the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS.tar.gz">niceideas ELK-MS package</a>.
</p>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1">The second article - ELK-MS - part II : assessing behaviour</a> presents a few concerns, assesses the expected behaviour using the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS_TEST.tar.gz">niceideas ELK-MS TEST package</a> and discusses challenges and constraints in this ELK-MS environment.
</p>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana2">This third and last article - ELK-MS - part III : so why is it cool?</a> presents, as indicated, why this ELK-MS stack is really really cool and works great.
</p>
<p>
This article assumes a basic understanding of Big Data / NoSQL technologies in general by the reader.
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. Introduction</a></li>
<li><a href="#sec2">2. Data locality and workload distribution</a></li>
<li><a href="#sec3">3. Examples</a></li>
<li><a href="#sec4">4. Conclusion</a></li>
</ul>
<a name="sec1"></a>
<h2>1. Introduction </h2>
<p>
The reader might want to refer to the <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana#sec1">Introduction of the first article in the serie</a> as well as <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1#sec1">the introduction of the second article</a>.
</p>
<p>
Summarizing them, this series of article is about presenting and assessing the ELK-MS stack, the tests done using the test cluster and present the conclusion, in terms of constraints as well as key lessons.
<br>
The second article was presenting the technical constraints coming from integrating Spark with ElasticSearch through the ES-Hadoop connector when running Spark on Mesos.
<br>
In this second article I focused a lot on what was not working and what were the constraints. A reader might have had the impression that these constraints could prevent a wide range of use cases on the ELK-MS stack. I want to address this fear in this third article since this is all but true, Spark on Mesos using data from ElasticSearch is really a pretty versatile environment and can address most if not all data analysis requirements.
</p>
<p>
In this last article, I will present how one can use a sound approach regarding data distribution in ElasticSearch to drive the distribution of the workload on the Spark cluster.
<br>
And it turns out that it's pretty straightforward to come up with a simple, efficient and natural approach to control the workload distribution using ElasticSearch, Spark and Mesos.
</p>
<p>
<b>ES index layout strategies</b>
</p>
<p>
The parameters that architects and developers need to tune to control the data distribution on ElasticSearch, which, in turn, controls the workload distribution on spark, are as follows:
</p>
<ul>
<li>The <b>index splitting</b> strategy</li>
<li>The <b>index sharding</b> strategy</li>
<li>The <b>replication</b> strategy (factor)</li>
<li>The <b>sharding key</b></li>
</ul>
<p>
<b>Spark aspects</b>
</p>
<p>
Then on the spark side the only important aspect is to use a proper version of ES-Hadoop supporting the Dynamic Allocation System without compromising data locality optimization (i.e ES-Hadoop >= 6.x for Spark 2.2)
</p>
<p>
But before digging into this, and if that is not already done before, I can only strongly recommend reading the <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana">the first article</a> in this serie, related presenting the ELK-MS stack and <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1">the second article</a> which presents the conclusions required to understand what will follow.
</p>
<a name="sec2"></a>
<h2>2. Data locality and workload distribution</h2>
<p>
What has been presented in <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1#sec3">the conclusion section of the ELK-MS part II article</a> is summarized hereunder:
</p>
<ul>
<li>
Fine Grained scheduling mode of spark jobs by Mesos screws performances up to an unacceptable level. ELK-MS need to stick to <b>Coarse-Grained</b> scheduling mode.
</li>
<li>
ES-Hadoop is able to enforce data-locality optimization under nominal situations. Under a heavily loaded cluster, data-locality optimization can be compromised for two reasons:
<ul>
<li>
If the local Mesos / Spark node to a specific ES node is not available after the configured waiting time, the processing will be moved to another free Mesos / Spark node.
</li>
<li>
ElasticSearch can well decide to serve the request from another node should the local ES node be busy at the time it is being requested by the local spark node.
</li>
</ul>
</li>
<li>
With ES-Hadoop 5.x, Dynamic allocation was messing up data locality optimization between ES and Spark. As such only Static allocation was usable and it was required to limit artificially the amount of nodes for a given job in good correspondance to the amount of shards in ES (usage of property <code>spark.cores.max</code> to limit the amount of spark executors and the <code>search_shards</code> API in ES to find out about the amount of shards to be processed)
<br>
<b>But now with ES-Hadoop 6.x, Dynamic allocation doesn't interfere with data locality optimization and everything works well out of the box.</b>
</li>
<li>
Re-distributing the data on the cluster after the initial partitioning decision is only done by spark under specific circumstances.
</li>
</ul>
<p>
<b>ES-Hadoop drives spark partitioning strategy</b>
</p>
<p>
So what happens with ES-Hadoop 6.x and dynamic allocation is that ElasticSearch sharding strategy drives the partitionning strategy of corresponding data frames in Spark. With Data Locality Optimization kicking in, even with Dynamic Allocation enabled, The Spark / Mesos cluster will do its best to create the Spark partitions on the nodes where the ES shards are located.
<br>
And this really works just out of the box.
</p>
<p>
<b>Eventually, there will be just as many executors booked by Mesos / Spark on the cluster as is requiredto handle every ES shars in a dedicated, co-located partition within Spark.</b>
</p>
<a name="sec3"></a>
<h2>3. Examples</h2>
<p>
In order to illustrate why I believe that in fact the way ELK-MS behaves when it comes to distributing the workload following the distribution of the data is <i>efficient and natural</i>, we'll use the examples below.
</p>
<p>
Imagine the following situation: the ELK-MS test cluster contains 6 nodes with similar configurations. The dataset to be stored is called <code>dataset</code> and contains 2 months of data.
<br>
In ElasticSearch the indexing settings are as follows:
</p>
<ul>
<li>
The <b>Index splitting strategy</b> is <b>by month</b>. This is not strictly an ElasticSearch setting, this is configured in Logstash or any other data ingestion tool.
<br>
As a matter of fact, whenever one wants to store temporal data in ElasticSearch (timeseries), one naturally considers splitting the index by year, month or even day depending on the size of the dataset.
</li>
<li>
The <b>sharding strategy</b> consists in creating 3 shards.
</li>
<li>
The <b>replication strategy</b> consists in creating 2 replicas (meaning 1 primary shard and 2 replicas).
</li>
<li>
We do not care about configuring the <i>sharding key</i> any differently than the default for now (a few words on the sharding key configuration are given in the conclusion).
</li>
</ul>
<p>
<b>Initial situation</b>
</p>
<p>
We can imagine that the above situation ends up in the following data layout on the cluster. (One should note though that this is not very realistic since ES would likely not split both month this way when it comes to storing replicas):
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/big_archi_data_empty.png">
<img class="centered" style="width: 500px; " alt="Data Architecture nominal situation" src="https://www.niceideas.ch/es_spark/images/big_archi_data_empty.png" />
</a>
</div>
<br>
<p>
<b>Working on a small subset of data for one month</b>
</p>
<p>
Now let's imagine that we write a processing script in spark that fetches a small subset of the data of one month, June 2017, so [A] here.
</p>
<p>
In addition, imagine that the filter ends up identifying precisely the data from a single shard of the index. Spark / Mesos would create in this case a single spark partition on the node co-located to the ES shard.
</p>
<p>
The processing happens this way in this case:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/big_archi_data_subset.png">
<img class="centered" style="width: 500px; " alt="Data Architecture working on subset" src="https://www.niceideas.ch/es_spark/images/big_archi_data_subset.png" />
</a>
</div>
<br>
<p>
Since only one shard needs to be read from ElasticSearch, ES-Hadoop will drive the creation of a single partition in the resulting DataFrame (or RDD), which in turn will cause Spark to request a single task in one executor, the one local to the ES shard.
</p>
<p>
So what actually happens is that working on a single shard located on single ES node will actually drive spark in a way to make it work on one single node as well.
<br>
Using replicas has the benefits to give the Mesos / Spark cluster some choice in regards to which this node should be. This is especially important if the cluster is somewhat loaded.
</p>
<p>
<b>Working on a single month of data</b>
</p>
<p>
In this second example, the processing script works on a single month of data, the full month of June 2017, so all shards of [A] here.
</p>
<p>
This will drive Spark to create 3 corresponding partitions on the Mesos / Spark cluster.
<br>
The processing works as follows in this case:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/big_archi_data_1_month.png">
<img class="centered" style="width: 500px; " alt="Data Architecture working on full month" src="https://www.niceideas.ch/es_spark/images/big_archi_data_1_month.png" />
</a>
</div>
<br>
<p>
Three shards from ES need to be fetched to Spark. ES-Hadoop will create 3 partitions which leads to 3 tasks to be dispatched on the Spark processing stage. These 3 tasks will be executed on the 3 local ES nodes owning the shards.
</p>
<p>
Again, distributing the input data on one third of the ES cluster on one side, and limiting's Spark resources to the actual number of nodes required on the other side, leads to one third of the Spark cluster to be used for the spark processing.
<br>
In this case, the ElasticSearch data distribution strategy drives the workload distribution on spark.
<br>
Again replication is useful to ensure a successful distribution even under a loaded cluster.
</p>
<p>
<b>Working on the whole period</b>
</p>
<p>
This will drive spark to create partitions on all nodes of the cluster.
<br>
The processing happens this way:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/big_archi_data.png">
<img class="centered" style="width: 500px; " alt="Data Architecture working on both months" src="https://www.niceideas.ch/es_spark/images/big_archi_data.png" />
</a>
</div>
<br>
<p>
When working on the whole period, it happens fortunately in this case that we end up fetching shards from the whole ES cluster, in this case the whole spark cluster will be used to distribute the processing workload, since each and every local spark node will need to work on the local ES shard.
</p>
<p>
Again, one last time, <b>the ElasticSearch data distribution strategy drives the workload distribution in good understanding to the data distribution, enforcing data-locality optimization.</b>
</p>
<a name="sec4"></a>
<h2>4. Conclusion</h2>
<p>
In conclusion, having the ElasticSearch data distribution strategy driving the processing distribution on the Mesos / Spark cluster, thanks to the ES-Hadoop connector requirements given to spark, makes a lot of sense if you think of it.
</p>
<p>
First it's <b>simple and consistent</b>. One can understand how the first stages of processing will occur within spark by simply looking at the data distribution using for instance Cerebro. Everything is well predictable and straightforward to assess.
</p>
<p>
But more importantly, it's <b>efficient</b> since, well, whenever we store data in ElasticSearch, we think of the distribution strategy, in terms of index splitting, sharding and replication precisely for the single purpose of performance.
<br>
Creating too many indexes and shards, more that the amount of nodes, would be pretty stupid since having more than X shards to read per node, where X is the amount of CPUs available to ES on a node, leads to poor performances. As such, the highest limit is the amount of CPUs in the cluster. Isn't it fortunate that this is also the limits we want in such case for our spark processing cluster?
<br>
On the other hand, when one wants to store a tiny dataset, a single index and a single shard is sufficient. In this case, a processing on this dataset would also use a single node in the spark cluster. Again that is precisely what we want.
</p>
<p>
In the end, one <i>"simply"</i> needs to optimize his ElasticSearch cluster and the spark processing will be optimized accordingly.
<br>
Eventually, the processing distribution will scale linearly with the data distribution. As such, it's <b>a very natural approach</b> in addition to being simple and efficient.
</p>
<p>
Summing things up, the spark processing workload distribution being driven by the ElasticSearch data distribution, both are impacted by the following parameters of an ES index:
</p>
<ul>
<li>The <b>index splitting</b> strategy</li>
<li>The <b>index sharding</b> strategy</li>
<li>The <b>replication</b> strategy (factor)</li>
<li>The <b>sharding key</b></li>
</ul>
<p>
The sharding key is not very important unless one has to implement a lot of joins in his processing scripts. In this case, one should carefully look at the various situations of these joins and find out which property is used most often as join key.
<br>
The sharding key should be this very same join key, thus enabling spark to implement the joins with best data locality, most of the time on the local node, since all shards with same sharding key end up on same node.
<br>
This may be the topic of another article on the subject, but likely not soon ... since, after so much writing, I need to focus on something else than Spark and ElasticSearch for a little while ...
</p>
<p>
As a last word on this topic for now, I would like to emphasize that not only this ELK-MS is working cool, in a simple, natural, efficient and performing way, but in addition all the UI consoles (Cerebro, Kibana, Mesos Console, Spark History Server) are state of the art, the Spark APIs is brilliantly designed and implemented, ElasticSearch itself in addition answers a whole range of use cases on its own, etc.
<br>
This stack is simply so amazingly cool.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1
ELK-MS - ElasticSearch/LogStash/Kibana - Mesos/Spark : a lightweight and efficient alternative to the Hadoop Stack - part II : assessing behaviour
Jerome Kehrli
2017-08-23T17:30:37-04:00
2020-07-21T18:21:58-04:00
<p>
This article is the second article in my serie of two articles presenting the ELK-MS Stack and test cluster.
</p>
<p>
ELK-MS stands for ElasticSearch/LogStash/Kibana - Mesos/Spark. The ELK-MS stack is a simple, lightweight, efficient, low-latency and performing alternative to the Hadoop stack providing state of the art Data Analytics features.
</p>
<p>
ELK-MS is especially interesting for people that don't want to settle down for anything but the best regarding Big Data Analytics functionalities but yet don't want to deploy a full-blend Hadoop distribution, for instance from Cloudera or HortonWorks.
<br>
Again, I am not saying that Cloudera and HortonWorks' Hadoops distributions are not good. <i>Au contraire</i>, they are awesome and really simplifies the overwhelming burden of configuring and maintaining the set of software components they provide.
<br>
But there is definitely room for something lighter and simpler in terms of deployment and complexity.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/application_architecture.png">
<img class="centered" style="width: 600px;" alt="ELK-MS Application Architecture" src="https://www.niceideas.ch/es_spark/images/application_architecture.png" />
</a>
</div>
<br>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana">The first article - entitled - ELK-MS - part I : setup the cluster</a> in this serie presents the ELK-MS stack and how to set up a test cluster using the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS.tar.gz">niceideas ELK-MS package</a>.
</p>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1">This second article - ELK-MS - part II : assessing behaviour</a> presents a few concerns, assesses the expected behaviour using the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS_TEST.tar.gz">niceideas ELK-MS TEST package</a> and discusses the challenges and constraints on this ELK-MS environment.
</p>
<p>
The conclusions of this serie of articles are presented in <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana2">the third and last article - ELK-MS - part III : so why is it cool?</a> which presents, as the name suggests, why this ELK-MS stack is really really cool and works great.
</p>
<p>
This article assumes a basic understanding of Big Data / NoSQL technologies in general by the reader.
</p>
<!-- ELK-MS - ElasticSearch/LogStash/Kibana - Mesos/Spark : a lightweight and efficient alternative to the Hadoop Stack - part II : assessing behaviour -->
<style>
.container-wrapper {
width: 100%;
float: left;
}
.container {
width: 100%;
margin-left: auto;
margin-right: auto;
}
.container figure {
float: left;
max-width: 23%;
min-width: 160px;
margin: 0 0 10px 2%;
}
/*.container figure:nth-child(1),
.container figure:nth-child(3) {
margin-left: 0;
}
*/
.container_inner {
height: 100px;
max-height: 100px;
overflow-y: hidden;
}
.container figure a img, .container figure img {
max-width: 100%;
float: left;
clip: rect(0,100px,100px,0);
position: relative;
width: 100%;
}
</style>
<p><i>
<b>Edited 2017-10-30</b>: I was using ES 5.0.0 with Spark 2.2.0 at the time of writing the initial version of this article.
<br>
With ElasticSearch 6.x, and ES-Hadoop 6.x, the game changes a little. The Spark 2.2.0 Dynamic allocation system is now perfectly compatible with the way ES-Hadoop 6.x enforces data locality optimization and everything works just as expected.
</i>
</p>
<p>
This article is the second article in my serie of two articles presenting the ELK-MS Stack and test cluster.
</p>
<p>
ELK-MS stands for ElasticSearch/LogStash/Kibana - Mesos/Spark. The ELK-MS stack is a simple, lightweight, efficient, low-latency and performing alternative to the Hadoop stack providing state of the art Data Analytics features.
</p>
<p>
ELK-MS is especially interesting for people that don't want to settle down for anything but the best regarding Big Data Analytics functionalities but yet don't want to deploy a full-blend Hadoop distribution, for instance from Cloudera or HortonWorks.
<br>
Again, I am not saying that Cloudera and HortonWorks' Hadoops distributions are not good. <i>Au contraire</i>, they are awesome and really simplifies the overwhelming burden of configuring and maintaining the set of software components they provide.
<br>
But there is definitely room for something lighter and simpler in terms of deployment and complexity.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/application_architecture.png">
<img class="centered" style="width: 600px;" alt="ELK-MS Application Architecture" src="https://www.niceideas.ch/es_spark/images/application_architecture.png" />
</a>
</div>
<br>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana">The first article - entitled - ELK-MS - part I : setup the cluster</a> in this serie presents the ELK-MS stack and how to set up a test cluster using the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS.tar.gz">niceideas ELK-MS package</a>.
</p>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1">This second article - ELK-MS - part II : assessing behaviour</a> presents a few concerns, assesses the expected behaviour using the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS_TEST.tar.gz">niceideas ELK-MS TEST package</a> and discusses the challenges and constraints on this ELK-MS environment.
</p>
<p>
The conclusions of this serie of articles are presented in <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana2">the third and last article - ELK-MS - part III : so why is it cool?</a> which presents, as the name suggests, why this ELK-MS stack is really really cool and works great.
</p>
<p>
This article assumes a basic understanding of Big Data / NoSQL technologies in general by the reader.
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. Introduction</a></li>
<li><a href="#sec2">2. Testing framework</a>
<ul>
<li><a href="#sec21">2.1 niceideas ELK-MS TEST</a></li>
<li><a href="#sec21">2.2 Test Scenario Script</a></li>
<li><a href="#sec23">2.3 Used Dataset</a></li>
<li><a href="#sec24">2.4 Test purposes</a></li>
</ul>
</li>
<li><a href="#sec3">3. Conclusions from assessment tests</a>
<ul>
<li><a href="#sec31">3.1 ES-Hadoop and Data-locality enforcement </a></li>
<li><a href="#sec32">3.2 Spark coarse grained scheduling by Mesos vs. Fine Grained</a></li>
<li><a href="#sec33">3.3 Spark Static Resource Allocation vs. Dynamic Allocation</a>
<ul>
<li><a href="#sec331">3.3.1 ES-Hadoop 5.x</a></li>
<li><a href="#sec332">3.3.2 ES-Hadoop 6.x</a></li>
</ul>
</li>
<li><a href="#sec34">3.4 Latency regarding Python instantiation</a></li>
<li><a href="#sec35">3.5 Other ES-Hadoop concerns</a></li>
<li><a href="#sec36">3.6 Other concerns</a></li>
</ul>
</li>
<li><a href="#sec4">4. Further work</a></li>
<li><a href="#sec5">5. Details of Tests</a>
<ul>
<li><a href="#sec51">5.1 Nominal Tests</a>
<ul>
<li><a href="#sec511">5.1.1 Legacy RDD API on bank dataset</a></li>
<li><a href="#sec512">5.1.2 Legacy DataFrame API on bank dataset</a></li>
<li><a href="#sec513">5.1.3 DataFrame API on bank dataset</a></li>
<li><a href="#sec514">5.1.4 DataFrame API on Apache-logs dataset</a></li>
<li><a href="#sec515">5.1.5 DataFrame API on Shakespeare dataset</a></li>
</ul>
</li>
<li><a href="#sec52">5.2 Data-Locality tests</a>
<ul>
<li><a href="#sec521">5.2.1 Bank dataset with 1 shard</a></li>
<li><a href="#sec522">5.2.2 Bank dataset with 2 shards</a></li>
<li><a href="#sec523">5.2.3 Bank dataset with 3 shards</a></li>
<li><a href="#sec524">5.2.4 Bank dataset with 1 shard and replicas</a></li>
<li><a href="#sec525">5.2.5 Testing repartitioning</a></li>
</ul>
</li>
<li><a href="#sec53">5.3 Aggregation tests</a>
<ul>
<li><a href="#sec531">5.3.1 ES-side Aggregations</a></li>
<li><a href="#sec532">5.3.2 Spark-side Aggregations</a></li>
</ul>
</li>
<li><a href="#sec54">5.4 Join test</a></li>
<li><a href="#sec55">5.5 Concurrency test</a></li>
</ul>
</li>
<li><a href="#sec6">6. References</a></li>
</ul>
<br>
<a name="sec1"></a>
<h2>1. Introduction </h2>
<p>
The reader might want to refer to the <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana#sec1">Introduction of the first article in the serie</a>.
</p>
<p>
Summarizing it, this article is about assessing the behaviour of the ELK-MS Stack using the test cluster introduced in the first article.
<br>
Especially two questions need to be answered:
</p>
<ol>
<li>
First, how does data-locality optimization work using ES-Hadoop to read data from ElasticSearch to Spark? On a large cluster, achieving Data Locality is <i>the sinews of war</i>. Before considering the ELK-MS stack as an actual alternative to a more standard Hadoop stack, assessing the sound behaviour and a good respect to data locality of the software stack is not optional.
</li>
<li>
Second, how does Mesos schedule spark executors and how does it impact data-locality? Mesos needs to be an effective alternative to YARN when it comes to dispatching spark executors while still taking into account data-locality.
</li>
</ol>
<p>
These are the two main objectives of the tests for which I am reporting the conclusions hereafter, as well as a few other points.
<br>
This article is not about testing Spark or Mesos themselves, it's really about testing how ElasticSearch / Mesos / Spark behave all together to support the application architecture from the schema above.
</p>
<p>
In addition, in contrary to the state of the art on Spark, in my current company, we are not going to be using Java or Scala to implement the spark processing logic, we are going to use python.
<br>
The reason for this is simple: our Data Scientists know python, period. They do not know Java, they are not willing to learn Scala. Our Data Scientist know R and python and as such as Head of R&D I have made python our standard language for our Data Analytics algorithms (not that I don't like R, <i>au contraire</i>, but I believe python is at the right intersection between Data Science and Engineering).
<br>
Choosing python as processing language has an impact when it comes to programming Spark, the support of python is, as a matter of fact, a little under the support of Scala and Java.
</p>
<p>
Now all of the above give this article is rationality: programming an ElasticSearch / Mesos / Spark Task with python is something that suffers from really little documentation available.
<br>
In the previous article I wanted to present how to set things up as well as share my setup tools and in this article I want to present how to use it, it's behaviour and share some short sample programs in the form of my tests package.
</p>
<a name="sec2"></a>
<h2>2. Testing Framework </h2>
<p>
I would summarize the specificities of the usage of Spark in my current company as follows:
</p>
<ul>
<li>Data analytics use cases are implemented in pyspark and python scripts and not native Scala or Java APIs</li>
<li>The input data and results are stored in ElasticSearch, not in HDFS</li>
<li>Spark runs on Mesos and not the more standard YARN on Hadoop.</li>
</ul>
<p>
So I needed a way to test and assess that all of this is working as expected and that the behaviour of the Mesos/Spark stack, both from the perspective of concurrency and respect of data-locality in between ES nodes and Spark nodes, is sound.
<br>
This is the objective of the <i>niceideas_ELK-MS-TEST</i> framework.
</p>
<p>
I am presenting this framework, the approach and the tests it contains herunder.
<br>
The test framework is <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS_TEST.tar.gz">available for download here</a>.
</p>
<a name="sec21"></a>
<h3>2.1 niceideas ELK-MS TEST</h3>
<p>
The <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS_TEST.tar.gz">niceideas ELK-MS TEST package</a> structure, after being properly extracted in a local folder, is as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/testing.png">
<img class="centered" style="width: 450px;" alt="ELK-MS TEST Structure" src="https://www.niceideas.ch/es_spark/images/testing.png" />
</a>
</div>
<br>
<ul>
<li>
<code>./vm_execute.sh</code>: this is the script one calls on the host machine to launch a test. The test to be executed should be given as argument.
</li>
<li>
<code>./tests/*</code>: the test scenario scripts.
</li>
</ul>
<p>
Executing a test on the ELK-MS Test cluster, is simply done, for instance, by the following command:
</p>
<pre>
badtrash@badbook:/data/niceideas_ELK-MS-TEST$ ./vm_execute.sh scenarii/5_concurrency_1_swissdata_df.sh
</pre>
<p>
This would execute the test 5_1 and show the spark driver logs on the console.
</p>
<a name="sec2"></a>
<h3>2.2 Test Scenario Script</h3>
<p>
Each and every test scenario script has the very same structure :
</p>
<ol>
<li>Create an ad'hoc shell script taking care of downloading a data set and loading in into ElasticSearch</li>
<li>Execute that Data Loading Shell script</li>
<li>Create an ad'hoc python script taking care of implementing the Spark processing</li>
<li>Execute the Data Processing Python script</li>
</ol>
<p>
For instance, a test scenario <code>X_test_Y_variant.sh</code> would have following structure:
</p>
<pre>
#!/bin/bash
# 1. Create Data Ingestion script
# -----------------------------------------------------------------------------
cat > X_test_Y_variant_do.sh <<- "EOF"
#!/bin/bash
# echo commands
set -x
<span style="color: red;"># ...
# Various shell commands to proceed with loading the data in ES
# ...</span>
# turn off command echoing
set +x
EOF
# 2. Exexcute Data Ingestion Script
# -----------------------------------------------------------------------------
bash X_test_Y_variant_do.sh
if [[ $? != 0 ]]; then
echo "Script execution failed. See previous logs"
exit -1
fi
# 3. Create pyspark script
# -----------------------------------------------------------------------------
cat > X_test_Y_variant.py <<- "EOF"
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext
# Spark configuration
conf = SparkConf().setAppName("ESTest_X_Y")
# SparkContext and SQLContext
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)
<span style="color: red;"># ...
# Various spark processing commands
# ...</span>
EOF
# 4. Exexcute pyspark script
# -----------------------------------------------------------------------------
spark-submit X_test_Y_variant.py
if [[ $? != 0 ]]; then
echo "Script execution failed. See previous logs"
exit -1
fi
</pre>
<p>
This key point of these scripts is that they are self contained and idempotent. They make no assumption about the state of the ELK-MS cluster before and they always start by cleaning all the data before reloading the data required for the tests.
</p>
<a name="sec23"></a>
<h3>2.3 Used Dataset</h3>
<p>
All the tests scenarii from the <i>niceideas ELK-MS TEST</i> package used either one of the following datasets:
</p>
<ul>
<li>
<b>Bank Dataset</b>: from <a href="https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip">https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip</a> - a dataset of financial accounts with owner and balance information.
</li>
<li>
<b>Shakespeare Dataset</b>: from <a href="https://download.elastic.co/demos/kibana/gettingstarted/shakespeare.json">https://download.elastic.co/demos/kibana/gettingstarted/shakespeare.json</a> - the complete work of Shakespeare, every line of every speech.
</li>
<li>
<b>Apache logs Dataset</b>: from <a href="https://download.elastic.co/demos/kibana/gettingstarted/logs.jsonl.gz">https://download.elastic.co/demos/kibana/gettingstarted/logs.jsonl.gz</a> - a set or an apache web server log files.
</li>
<li>
<b>Swiss AirBnB</b>: two datasets in fact:
<ul>
<li>
from <a href="http://niceideas.ch/mes/swissairbnb/tomslee_airbnb_switzerland_1451_2017-07-11.csv">http://niceideas.ch/mes/swissairbnb/tomslee_airbnb_switzerland_1451_2017-07-11.csv</a> : the list of AirBnB offers in Switzerland as of July 2017.
</li>
<li>
from <a href="http://niceideas.ch/mes/swissairbnb/swisscitiespop.txt">http://niceideas.ch/mes/swissairbnb/swisscitiespop.txt</a> : the list of swiss cities with population and geoloc information.
</ul>
</li>
</ul>
<p>
The last dataset, related to swiss AirBnB offers and cities information is required to test the behaviour of joins on Spark.
<br>
The other datasets represent different volumes and enable us to test various aspects.
</p>
<a name="sec24"></a>
<h3>2.4 Test purposes</h3>
<p>
The tests from the <i>niceideas ELK-MS TEST</i> package are presented in details with all results in section <a href="#sec5">5. Details of Tests</a>.
<br>
Before presenting the conclusions inferred from these tests in the next section, this is a short summary of the purpose of every family of tests:
</p>
<ol>
<li>
<b>Nominal tests</b> - assess how the various kinds of APIs of Spark are used to read data from ES: the RDD API, the legacy DataFrame API (SQLContext) and the new DataFrame API (SQLSession).
</li>
<li>
<b>Data-locality tests</b> - assess how data-locality optimization between ES and Spark works and to what extent.
</li>
<li>
<b>Aggregation tests</b> - how aggregation on ES data works.
</li>
<li>
<b>Join tests</b> - how joining two data frames coming from ES works.
</li>
<li>
<b>Concurrency tests</b> - how does Mesos / Spark behave when running several jobs at a time.
</li>
</ol>
<p>
Again, the section <a href="#sec5">5. Details of Tests</a> presents each and every test in details, along with the screenshots of Cerebro, Spark History Server, the logs of the spark driver, etc.
</p>
<a name="sec3"></a>
<h2>3. Conclusions from assessment tests</h2>
<p>
I am reporting in this section already the conclusions that can be taken from the tests executed in the scope of this work.
<br>
The conclusions and important information are presented in this early section already, preventing the reader from the requirement to read all the individual tests presented in details in the next section.
</p>
<a name="sec31"></a>
<h3>3.1 ES-Hadoop and Data-locality enforcement </h3>
<p>
<b>Data locality</b>
</p>
<p>
Data locality is how close data is to the code processing it. There are several levels of locality based on the data’s current location. In order from closest to farthest:
</p>
<ul>
<li>
<b><code>PROCESS_LOCAL</code></b> data is in the same JVM as the running code. This is the best locality possible
</li>
<li>
<b><code>NODE_LOCAL</code></b> data is on the same node. Examples might be in HDFS on the same node, or in another executor on the same node. This is a little slower than PROCESS_LOCAL because the data has to travel between processes
</li>
<li>
<b><code>NO_PREF</code></b> data is accessed equally quickly from anywhere and has no locality preference
</li>
<li>
<b><code>RACK_LOCAL</code></b> data is on the same rack of servers. Data is on a different server on the same rack so needs to be sent over the network, typically through a single switch
</li>
<li>
<b><code>ANY</code></b> data is elsewhere on the network and not in the same rack
Spark prefers to schedule all tasks at the best locality level, but this is not always possible. In situations where there is no unprocessed data on any idle executor, Spark switches to lower locality levels. There are two options: a) wait until a busy CPU frees up to start a task on data on the same server, or b) immediately start a new task in a farther away place that requires moving data there.
</li>
</ul>
<p>
What Spark typically does is wait a bit in the hopes that a busy CPU frees up. Once that timeout expires, it starts moving the data from far away to the free CPU.
<br>
The setting indicating how long it should wait before moving the processing elsewhere is <code>spark.locality.wait=10s</code>.
</p>
<p>
<b>ES-Hadoop</b>
</p>
<p>
Data-locality enforcement works amazingly under nominal conditions. ES-Hadoop makes the Spark scheduler understand the topology of the shards on the ES cluster and Spark dispatches the processing accordingly. Mesos doesn't interfere in this regards.
</p>
<p>
But again, it works only under nominal conditions.
<br>
As indicated above, there can be several factors compromising Data-locality:
</p>
<ol>
<li>
First, imagine that at resource allocation time the Mesos cluster is heavily loaded. Spark will wait for <code>spark.locality.wait=10s</code> trying to get the processing executed on the node where ES stored the target data shard.
<br>
But if in this period the node doesn't become free, spark will move the processing elsewhere.
</li>
<li>
The second case it not anymore related to spark, but to ElasticSearch. Imagine that at the very moment the spark executor submits the request to ES (through the ES-Hadoop connector), the co-located ES node is busy doing something else (answering another request, indexing some data, etc.).
<br>
In this case, ES will delegate the answering of the request to another node and local data-locality is broken.
</ol>
<a name="sec32"></a>
<h3>3.2 Spark coarse grained scheduling by Mesos vs. Fine Grained </h3>
<p>
In <i>Coarse Grained</i> scheduling mode, the default, Mesos considers spark only at the scale of the required <i>spark executor</i> processes. All Mesos knows about spark is the executor processes on the nodes they are running. Mesos knows nothing of Spark's jobs internals such as stages and tasks.
<br>
In addition, static allocation makes Mesos Job pretty easy: try to allocate as many resources from the cluster to spark executors for pending jobs as are available. This has the following consequences:
</p>
<ol>
<li>
First, if a job is submitted to the cluster at a moment when the cluster is completely free, the job will be allocated the whole cluster. If another job comes even only just a few seconds after, it will still need to wait for the cluster to be freed by the first job, and that will happen only when the first job completes.
</li>
<li>
Second, if several jobs are waiting to be executed, when the cluster is freed, Mesos will allocate the cluster resources evenly to each and every job. Now imagine that all these jobs are short-lived jobs and only one of them is a long-lived job. At allocation time (static allocation), that long-lived job got only a small portion of the cluster. Even if very soon the cluster becomes free, that job will still need to complete its execution on his small portion, making most of the cluster unused.
</ol>
<p>
Historically, Mesos on Spark can benefit from a <i>Fine Grained</i> scheduling mode instead, where Mesos will schedule not only spark executors on nodes in a rough fashion but really each and every individual spark task instead.
<br>
In regards to data-locality optimization, this doesn't seem to have any impact.
<br>
In regards to performance on the other hand, <i>Fine Grained</i> scheduling mode really messes performances completely.
</p>
<p>
The thing is that Mesos requires quite some time to negotiate with the resources providers. If that negotiation happens for every individual spark tasks, a huge amount of time is lost and eventually the impact on performance is not acceptable.
</p>
<p>
For this reason (and others), the <i>Fine Grained</i> scheduling mode is deprecated: <a href="https://issues.apache.org/jira/browse/SPARK-11857">https://issues.apache.org/jira/browse/SPARK-11857</a>
</p>
<a name="sec33"></a>
<h3>3.3 Spark Static Resource Allocation vs. Dynamic Allocation </h3>
<p>
By default, Spark's scheduler uses a <b>Static Resource Allocation</b> system. This means that, at the job (or driver) initialization time, Spark, with the help of Mesos in this case, will decide what resource from the Mesos cluster can be allocated to the job. This decision is static, meaning that once decided the set of resources allocated to the job will never change in its whole life regardless of what happens on the cluster (other / additional nodes becoming free, etc.)
<br>
This has the consequences listed above in the previous section, the whole cluster is allocated to a single job, further jobs need to wait, etc. and as such it's not very optimal.
</p>
<p>
Now of course Spark provides a solution to this, the <b>Dynamic Allocation</b> System.
</p>
<p>
<b>And this is where Spark gets really cool.</b> With Dynamic Allocation, the Spark / Mesos cluster is evenly shared between multiples jobs requesting execution on the cluster regardless of the time of appearance of the jobs.
<br>
And with ES-Hadoop 6.x, the Dynamic allocation system is perfectly able to respect the locality requirements communicated by the elastic-search spark connector and respects them as much as possible
</p>
<a name="sec331"></a>
<h3>3.3.1 ES-Hadoop 5.x and Spark 2.2</h3>
<p>
With ES-Hadoop version 5.x, the way the elasticsearch-spark connector was enforcing data locality was incompatible with Spark 2.2.0 and as such
But unfortunately, when using Dynamic Allocation, Spark simply doesn't take into consideration ES-Hadoop's requirements regarding data locality optimization anymore.
</p>
<p>
Without going into details, the problem comes from the fact that ES-Hadoop makes spark request as many executors as shards and indicates as <i>preferred location</i> the nodes owning the ES shards.
<br>
But Dynamic allocation screws all of this by allocating executors only one after the other (more or less) and only after monitoring evolutions of the job processing needs and the amount of tasks created. In no way does the dynamic allocation system give any consideration for ES-Hadoop requirements.
</p>
<a name="sec332"></a>
<h3>3.3.2 ES-Hadoop 6.x</h3>
<p>
As indicated in the release notes of the ElasticSearch-Hadoop connector 6.0.0, the ElasticTeam has added support for Spark 2.2.0. This support has fixed the messing up with Dynamic Allocation problem that was suffering ES-Hadoop 5.x.
</p>
<p>
Now even with Dynamic Allocation properly enables, which is a requirement for us in order to optimize the Mesos Cluster resources consumption, Data Locality os optimized and properly enforced everywhen possible.
</p>
<a name="sec34"></a>
<h3>3.4 Latency regarding Python instantiation</h3>
<p>
Executing some tasks in Python takes time in comparison to executing tasks natively in Java or Scala. The problem is that spark tasks in python require to launch the individual task processing in seperate process than the Spark JVM. Only Java and Scala Spark processings run natively in the Spark JVM.
</p>
<p>
This problem is not necessarily a big deal since the DataFrame or RDD APIs exposed to python pyspark scripts are actually implemented by Scala code underneath, they resolve to native Scala code.
<br>
There is one noticeable exception in this regards: UDF (User Defined functions) implemented in python. While this is very possible, it should be avoided at all cost.
<br>
One can very well still use pyspark but write UDF in Scala.
</p>
<ul>
<li>An explanation of this problem : <a href="https://fr.slideshare.net/SparkSummit/getting-the-best-performance-with-pyspark">https://fr.slideshare.net/SparkSummit/getting-the-best-performance-with-pyspark</a>
</li>
<li>UDF in Scala : <a href="http://aseigneurin.github.io/2016/09/01/spark-calling-scala-code-from-pyspark.html">http://aseigneurin.github.io/2016/09/01/spark-calling-scala-code-from-pyspark.html</a>
</li>
</ul>
<a name="sec35"></a>
<h3>3.5 Other ES-Hadoop concerns</h3>
<p>
<b>Repartitioning</b>
</p>
<p>
I couldn't find a way to make repartitioning work the way I want, meaning re-distributing the data on the cluster in order to scale out the further workload.
<br>
I am not saying there is no way, just that I haven't found one so far.
</p>
<p>
As such, a sound approach regarding initial sharding in ES should be adopted. One should take into consideration that <i>a priori</i>, initial sharding may well drive the way Spark will be able to scale the processing out on the cluster.
<br>
While creating by default one shard per node in the cluster would definitely be overkill, the general idea should tend in this direction.
</p>
<p>
<b>ES level aggregations</b>
</p>
<p>
It's simply impossible to forge a query from Spark to ElasticSearch through ES-Hadoop that would make ElasticSearch compute aggregation and returning them instead of the raw data.
<br>
Such advanced querying features are not available from spark.
</p>
<p>
The need is well identified but it remains <i>Work in Progress</i> at the moment: <a href="https://github.com/elastic/elasticsearch-hadoop/issues/276 ">https://github.com/elastic/elasticsearch-hadoop/issues/276</a>.
</p>
<a name="sec36"></a>
<h3>3.6 Other concerns</h3>
<p>
<b>Spark History Server</b>
</p>
<p>
Running Spark in Mesos, there is no long-lived Spark process. Spark executors are created when required by Mesos and the Mesos master and slave processes are the only long lived process on the cluster in this regards.
</p>
<p>
As such, the Spark Application UI (on ports 4040, 4041, etc.) only live for the time of the Spark processing. When the job is finished, the Spark UI application vanishes.
</p>
<p>
For this reason, Spark provides an History server. The installation and operation of the History Server is presented in <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana">the first article of this serie : ELK-MS - part I : setup the cluster</a>.
</p>
<p>
Interestingly, that history server supports the same JSON / REST API that the usual Spark Console, with only a very few limitations.
<br>
For instance, one can use the REST API to discover about the Application-ID of running jobs in order to kill them (whenever required). For this, simply list the jobs and find out about those that have <code>"endTimeEpoch" : -1</code>, meaning the application is still alive:
</p>
<pre>
curl -XGET http://192.168.10.10:18080/api/v1/applications
</pre>
<p>
<b>Limitations of the ELK-MS stack</b>
</p>
<p>
As stated in the previous article, ElasticSearch is not a distributed filesystem, it's a document-oriented NoSQL database.
</p>
<p>
There are situations where a distributed filesystem provides interesting possibilities. Those are not provided by the ELK-MS stack as is. It would be interesting to test Ceph on Mesos for this. See <a href="http://tracker.ceph.com/projects/ceph/wiki/Ceph-mesos"> http://tracker.ceph.com/projects/ceph/wiki/Ceph-mesos</a>.
</p>
<a name="sec4"></a>
<h2>4. Further work</h2>
<p>
I am still considering some next steps on the topic of the ELK-MS stack testing since there are still a few things I would like to test ot assess:
</p>
<p>
In a raw fashion:
</p>
<ul>
<li>
Find out about how to set maximum nodes booked by Mesos for a single spark job in order to avoid fully booking the cluster.
</li>
<li>
ElasticSearch on mesos
<ul>
<li>
This seems quite obvious. I expect the overall cluster performance to be way better if Mesos and ES don't compete with each other for hardware resources on nodes.
</li>
<li>
There are workaround of course, such as configuring Mesos to avoid using all the CPUs of a node. But that will never be as efficient as letting Mesos distribute the global workload.
</li>
</ul>
</li>
<li>
Find a way for repartitioning to work the way I intend it: data should get redistributed across the cluster!
</li>
<li>
Give Spark Streaming a try to reduce latency.
<ul>
<li>
<a href="https://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html#spark-streaming">https://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html#spark-streaming</a>
</li>
</ul>
</li>
<li>
Try FAIR Spark scheduler and play with it.
<ul>
<li>
I got satisfying results using spark FIFO scheduler in terms of concurrency and haven't seen the need to change to FAIR.
</li>
<li>
It really seems Mesos takes care of everything and I do really not see what the FAIR scheduler can change but I want to be sure.
</li>
<li>
There are some chances that this makes me rewrite this whole article ... in another article.
</li>
</ul>
</li>
<li>
Ceph integration on Mesos for binary files processing.
<ul>
<li>
How to integrate Ceph and spark ? Here as well very little documentation seems to be available.
</li>
<li>
I found pretty much only this : <a href="https://indico.cern.ch/event/524549/contributions/2185930/attachments/1290231/1921189/2016.06.13_-_Spark_on_Ceph.pdf">https://indico.cern.ch/event/524549/contributions/2185930/attachments/1290231/1921189/2016.06.13_-_Spark_on_Ceph.pdf</a>
</li>
</ul>
</li>
<li>
What about HDFS on Mesos ?
<ul>
<li>
I would want to give it a try even though I am really rather considering Ceph for the use cases ElasticSearch forbids me to address.
</li>
<li>
The thing is that Ceph integrates much better in the UNIX unified filesystem than HDFS
</li>
<li>
Even though there is an approach to reach same level of integration with HDFS based on Fuse <a href="https://wiki.apache.org/hadoop/MountableHDFS">https://wiki.apache.org/hadoop/MountableHDFS</a>. But that is still limited (doesn't support ownership informations for now)
</li>
</ul>
</li>
</ul>
<a name="sec5"></a>
<h2>5. Details of Tests</h2>
<p>
This very big section now presents each and every tests in details, along with the results in the form the the logs of the script (data feeding and spark driver logs), the screenshots of the UI applications (Cerebro, Mesos Console, Spark History Server).
</p>
<p>
The conclusions from the individual tests have been reported in the global <a href="#sec3">3. Conclusions from assessment tests</a> section above.
</p>
<a name="sec51"></a>
<h3>5.1 Nominal Tests</h3>
<p>
<b>Nominal tests</b> - assess how the various kinds of APIs of Spark are used to read data from ES: the RDD API, the legacy DataFrame API (SQLContext) and the new DataFrame API (SQLSession).
</p>
<a name="sec511"></a>
<h4>5.1.1 Legacy RDD API on bank dataset</h4>
<p>
<b>Test details</b>
</p>
<ul>
<li><b>Test Script</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/tests/scenarii/1_nominal_1_test_bank_rdd_legacy.sh"><code>1_nominal_1_test_bank_rdd_legacy.sh</code></a></li>
<li><b>Input Dataset</b>: Bank Dataset from <a href="https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip">https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip</a></li>
<li><b>Purpose</b>: see how Spark's RDD API can be used to fetch data from ElasticSearch and how sharding in ES impacts executors layout on the cluster
</ul>
<p>
<b>Relevant portion of spark Script</b>
</p>
<pre>
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext
# Spark configuration
conf = SparkConf().setAppName("ESTest_1_1")
# SparkContext and SQLContext
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)
# Simplest possible query
q = "?q=*"
es_read_conf = {
"es.resource" : "bank",
"es.query" : q
}
es_rdd = sc.newAPIHadoopRDD(
inputFormatClass="org.elasticsearch.hadoop.mr.EsInputFormat",
keyClass="org.apache.hadoop.io.NullWritable",
valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",
conf=es_read_conf)
es_df = sqlContext.createDataFrame(es_rdd)
# I need to collect the result to show them on the console
data_list = es_df.collect()
print ("Printing 10 first results")
for x in data_list[0:10]:
print x
# Print count : THIS IS FUNNY :
# it relaunches the whole Distributed Data Frame Processing
print ("Fetched %s accounts (re-computed)") % es_df.count()
# Print count
print ("Fetched %s accounts (from collected list)") % len (data_list)
</pre>
<p>
<b>Results</b>
</p>
<ul>
<li><b>Logs of the Spark Driver</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/1_nominal_1_test_bank_rdd_legacy.log">1_nominal_1_test_bank_rdd_legacy.log</a>
<li><b>Screenshots from the various admin console after the test execution</b>:
</ul>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_1_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_1_1.png' />
</div>
</a>
<figcaption><b>Test 1-1 / Dataset in ES</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_1_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_1_1.png' />
</div>
</a>
<figcaption><b>Test 1-1 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_1.png' />
</div>
</a>
<figcaption><b>Test 1-1 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_1_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_1_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 1-1 / Job 0 / Stage 0</b></figcaption>
</figure>
</div>
</div>
<p>
<b>Conclusions</b>
</p>
<ul>
<li>
Spark can read data from ElasticSearch using the ES-Hadoop connector and the RDD API really out of the box.
<br>
One just needs to configure a few settings to the <code>newAPIHadoopRDD</code> API:
<ul>
<li><code>inputFormatClass="org.elasticsearch.hadoop.mr.EsInputFormat"</code></li>
<li><code>keyClass="org.apache.hadoop.io.NullWritable"</code></li>
<li><code>valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable"</code></li>
</ul>
</li>
<li>
Mesos spreads the workload on the cluster efficiently.
<ul>
<li>This test was run alone on the cluster</li>
<li>2 nodes are sufficient to run the job since, thanks to replicas two nodes have actually all the shards </li>
<li>Mesos creates a dedicated spark executor on each of the 2 nodes</li>
<li>Sparks then successfully distribute the RDD on the 2 executors</li>
</ul>
</li>
<li>
Data-locality optimization works out of the box.
<ul>
<li>There are 5 shards in ElasticSearch, which, with replicas, are well spread on the cluster</li>
<li>Mesos / Spark dispatches the workload efficiently since it creates 5 RDD partitions for the 5 shards, each and every of them respecting data locality (<code>NODE_LOCAL</code>) and as such respecting the requirements given by the ES-Hadoop connector.
</li>
</ul>
</li>
</ul>
<a name="sec512"></a>
<h4>5.1.2 Legacy DataFrame API on bank dataset</h4>
<p>
<b>Test details</b>
</p>
<ul>
<li><b>Test Script</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/tests/scenarii/1_nominal_2_test_bank_df_legacy.sh"><code>1_nominal_2_test_bank_df_legacy.sh</code></a></li>
<li><b>Input Dataset</b>: Bank Dataset from <a href="https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip">https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip</a></li>
<li><b>Purpose</b>: see how Spark's legacy DataFrame API can be used to fetch data from ElasticSearch and how sharding in ES impacts executors layout on the cluster
</ul>
<p>
<b>Relevant portion of spark Script</b>
</p>
<pre>
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext
# Spark configuration
conf = SparkConf().setAppName("ESTest_1_2")
## !!! Caution : this is pre 2.0 API !!!
# SparkContext and SQLContext
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)
es_df = sqlContext.read \
.format("org.elasticsearch.spark.sql") \
.options(pushdown=True) \
.load("bank") \
.where("gender='F'")
# I need to collect the result to show them on the console
data_list = es_df.collect()
print ("Printing 10 first results")
for x in data_list[0:10]:
print x
# Print count : THIS IS FUNNY :
# it relaunches the whole Distributed Data Frame Processing
print ("Fetched %s women accounts (re-computed)") % es_df.count()
# Print count
print ("Fetched %s women accounts (from collected list)") % len (data_list)
</pre>
<p>
<b>Results</b>
</p>
<ul>
<li><b>Logs of the Spark Driver</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/1_nominal_2_test_bank_df_legacy.log">1_nominal_2_test_bank_df_legacy.log</a>
<li><b>Screenshots from the various admin console after the test execution</b>:
</ul>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_1_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_1_2.png' />
</div>
</a>
<figcaption><b>Test 1-2 / Dataset in ES</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_1_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_1_2.png' />
</div>
</a>
<figcaption><b>Test 1-2 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_2.png' />
</div>
</a>
<figcaption><b>Test 1-2 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_2_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_2_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 1-2 / Job 0 / Stage 0</b></figcaption>
</figure>
</div>
</div>
<p>
<b>Conclusions</b>
</p>
<ul>
<li>
Spark can read data from ElasticSearch using the ES-Hadoop connector and the Legacy DataFrame API (SQLContext) really out of the box.
<br>
The single configuration required is <code>format("org.elasticsearch.spark.sql") on the <code>SQLContext</code> API</code>
</li>
<li>
Here as well, the Dynamic allocation system allocates nodes to the job one after the other.
<br>
After two nodes alolocated to the job, all shards (thx to replicas) become available locally and data localiy optimization can be satisfied without any other node required. The job executes on these 2 nodes.
</li>
<li>
In this case, as seen on <a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_2_job_0_stage_0.png">result_1_2_job_0_stage_0.png</a>, spark successfully respects data locality as well.
</li>
</ul>
<a name="sec513"></a>
<h4>5.1.3 DataFrame API on bank dataset</h4>
<p>
<b>Test details</b>
</p>
<ul>
<li><b>Test Script</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/tests/scenarii/1_nominal_3_test_bank_df.sh"><code>1_nominal_3_test_bank_df.sh</code></a></li>
<li><b>Input Dataset</b>: Bank Dataset from <a href="https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip">https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip</a></li>
<li><b>Purpose</b>: see how Spark's New (>= 2.0) DataFrame API can be used to fetch data from ElasticSearch and how sharding in ES impacts executors layout on the cluster
</ul>
<p>
<b>Relevant portion of spark Script</b>
</p>
<pre>
from pyspark.conf import SparkConf
from pyspark.sql import SQLContext, SparkSession
# Spark configuration
conf = SparkConf().setAppName("ESTest_1_3")
# Spark SQL Session
ss = SparkSession.builder \
.config(conf=conf) \
.getOrCreate()
es_df = ss.read \
.format("org.elasticsearch.spark.sql") \
.options(pushdown=True) \
.load("bank") \
.where("gender='F'")
# I need to collect the result to show them on the console
data_list = es_df.collect()
print ("Printing 10 first results")
for x in data_list[0:10]:
print x
# Print count : THIS IS FUNNY :
# it relaunches the whole Distributed Data Frame Processing
print ("Fetched %s women accounts (re-computed)") % es_df.count()
# Print count
print ("Fetched %s women accounts (from collected list)") % len (data_list)
</pre>
<p>
<b>Results</b>
</p>
<ul>
<li><b>Logs of the Spark Driver</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/1_nominal_3_test_bank_df.log">1_nominal_3_test_bank_df.log</a>
<li><b>Screenshots from the various admin console after the test execution</b>:
</ul>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_1_3.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_1_3.png' />
</div>
</a>
<figcaption><b>Test 1-3 / Dataset in ES</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_1_3.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_1_3.png' />
</div>
</a>
<figcaption><b>Test 1-3 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_3.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_3.png' />
</div>
</a>
<figcaption><b>Test 1-3 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_3_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_3_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 1-3 / Job 0 / Stage 0</b></figcaption>
</figure>
</div>
</div>
<p>
<b>Conclusions</b>
</p>
<ul>
<li>
Spark can read data from ElasticSearch using the ES-Hadoop connector and the New DataFrame API (SQLSession) really out of the box.
<br>
The single configuration required here as well is <code>format("org.elasticsearch.spark.sql") on the <code>SQLSession</code> API</code>
</li>
<li>
Nothing specific to report regarding the other aspects: Mesos and Spark's dynamic allocation system distribute the workload as expected, still creates a dedicated Spark Executor for 2 nodes of the cluster which is sufficient (thx replicas), Spark respects data locality strictly, etc.
</li>
</ul>
<a name="sec514"></a>
<h4>5.1.4 DataFrame API on Apache-logs dataset</h4>
<p>
<b>Test details</b>
</p>
<ul>
<li><b>Test Script</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/tests/scenarii/1_nominal_4_test_apache-logs_df.sh"><code>1_nominal_4_test_apache-logs_df.sh</code></a></li>
<li><b>Input Dataset</b>: Apache Logs Dataset from <a href="https://download.elastic.co/demos/kibana/gettingstarted/logs.jsonl.gz">https://download.elastic.co/demos/kibana/gettingstarted/logs.jsonl.gz</a></li>
<li><b>Purpose</b>: see how Spark's New (>= 2.0) DataFrame API can be used to fetch data from ElasticSearch from another dataset and how sharding in ES impacts executors layout on the cluster
</ul>
<p>
<b>Relevant portion of spark Script</b>
</p>
<pre>
from pyspark.conf import SparkConf
from pyspark.sql import SQLContext, SparkSession
# Spark configuration
conf = SparkConf().setAppName("ESTest_1_4")
# es.read.field.exclude (default empty) :
# Fields/properties that are discarded when reading the documents
# from Elasticsearch
conf.set ("es.read.field.exclude", "relatedContent")
# es.read.field.as.array.include (default empty) :
# Fields/properties that should be considered as arrays/lists
conf.set ("es.read.field.as.array.include", "@tags,headings,links")
# Spark SQL Session
ss = SparkSession.builder \
.config(conf=conf) \
.getOrCreate()
# Query configuration only (cannot pass any ES conf here :-( )
es_query_conf= {
"pushdown": True
}
es_df = ss.read \
.format("org.elasticsearch.spark.sql") \
.options(conf=es_query_conf) \
.load("apache-logs-*")
# I need to collect the result to show them on the console
data_list = es_df.collect()
print ("Printing 10 first results")
for x in data_list[0:10]:
print x
# Print count : THIS IS FUNNY :
# it relaunches the whole Distributed Data Frame Processing
print ("Fetched %s logs (re-computed)") % es_df.count()
# Print count
print ("Fetched %s logs (from collected list)") % len (data_list)
</pre>
<p>
<b>Results</b>
</p>
<ul>
<li><b>Logs of the Spark Driver</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/1_nominal_4_test_apache-logs_df.log">1_nominal_4_test_apache-logs_df.log</a>
<li><b>Screenshots from the various admin console after the test execution</b>:
</ul>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_1_4.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_1_4.png' />
</div>
</a>
<figcaption><b>Test 1-4 / Dataset in ES</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_1_4.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_1_4.png' />
</div>
</a>
<figcaption><b>Test 1-4 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_4.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_4.png' />
</div>
</a>
<figcaption><b>Test 1-4 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_4_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_4_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 1-4 / Job 0 / Stage 0</b></figcaption>
</figure>
</div>
</div>
<p>
<b>Conclusions</b>
</p>
<p>
N.D (nothing to declare). Everything works as expected (see previous tests results from the <a href="#sec51">1 Nominal Tests</a> family).
<br>
Interestingly here, the workload justifies the booking of the three nodes of te cluster, which is successfully achieved since the job runs alone on the cluster.
</p>
<a name="sec515"></a>
<h4>5.1.5 DataFrame API on Shakespeare dataset</h4>
<p>
<b>Test details</b>
</p>
<ul>
<li><b>Test Script</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/tests/scenarii/1_nominal_5_test_shakespeare.sh"><code>1_nominal_5_test_shakespeare.sh</code></a></li>
<li><b>Input Dataset</b>: Shakespeare's Works Dataset from <a href="https://download.elastic.co/demos/kibana/gettingstarted/shakespeare.json">https://download.elastic.co/demos/kibana/gettingstarted/shakespeare.json</a></li>
<li><b>Purpose</b>: see how Spark's New (>= 2.0) DataFrame API can be used to fetch data from ElasticSearch from another dataset and how sharding in ES impacts executors layout on the cluster
</ul>
<p>
<b>Relevant portion of spark Script</b>
</p>
<pre>
from pyspark.conf import SparkConf
from pyspark.sql import SQLContext, SparkSession
# Spark configuration
conf = SparkConf().setAppName("ESTest_1_5")
# Spark SQL Session
ss = SparkSession.builder \
.config(conf=conf) \
.getOrCreate()
# Query configuration only (cannot pass any ES conf here :-( )
es_query_conf= {
"pushdown": True
}
es_df = ss.read \
.format("org.elasticsearch.spark.sql") \
.options(conf=es_query_conf) \
.load("shakespeare*")
# Collect result to the driver
data_list = es_df.collect()
print ("Printing 10 first results")
for x in data_list[0:10]:
print x
# Print count : THIS IS FUNNY :
# it relaunches the whole Distributed Data Frame Processing
print ("Fetched %s logs (re-computed)") % es_df.count()
# Print count
print ("Fetched %s logs (from collected list)") % len (data_list)
</pre>
<p>
<b>Results</b>
</p>
<ul>
<li><b>Logs of the Spark Driver</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/1_nominal_5_test_shakespeare.log">1_nominal_5_test_shakespeare.log</a>
<li><b>Screenshots from the various admin console after the test execution</b>:
</ul>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_1_5.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_1_5.png' />
</div>
</a>
<figcaption><b>Test 1-5 / Dataset in ES</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_1_5.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_1_5.png' />
</div>
</a>
<figcaption><b>Test 1-5 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_5.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_5.png' />
</div>
</a>
<figcaption><b>Test 1-5 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_5_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_1_5_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 1-5 / Job 0 / Stage 0</b></figcaption>
</figure>
</div>
</div>
<p>
<b>Conclusions</b>
</p>
<p>
N.D (nothing to declare). Everything works as expected (see previous tests results from the <a href="#sec51">1 Nominal Tests</a> family).
<br>
This time, however, due to the lack of replicas, the three nodes are actually required to satisfy data localiy optimization. The allocation of the 2 nodes to the job happens successfully again since the job runs alone.
</p>
<a name="sec52"></a>
<h3>5.2 Data-locality tests</h3>
<p>
<b>Data-locality tests</b> - assess how data-locality optimization between ES and Spark works and to what extent.
</p>
<a name="sec521"></a>
<h4>5.2.1 Bank dataset with 1 shard</h4>
<p>
<b>Test details</b>
</p>
<ul>
<li><b>Test Script</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/tests/scenarii/2_collocation_1_bank_one_shard.sh"><code>2_collocation_1_bank_one_shard.sh</code></a></li>
<li><b>Input Dataset</b>: Bank Dataset from <a href="https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip">https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip</a></li>
<li><b>Purpose</b>: Assess how data-locality works when using a dataset with a single shard on a single node of the cluster, see what decisions will Mesos / Spark take from ES-Hadoop's requirements.
</ul>
<p>
<b>Expected Behaviour</b>
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/system_archi_data_2_1.png">
<img class="centered" style="width: 600px;" alt="Data Architecture Test 2_1" src="https://www.niceideas.ch/es_spark/images/system_archi_data_2_1.png" />
</a>
</div>
<br>
<p>
<b>Relevant portion of spark Script</b>
</p>
<pre>
from pyspark.conf import SparkConf
from pyspark.sql import SQLContext, SparkSession
# Spark configuration
conf = SparkConf().setAppName("ESTest_2_1")
# Spark SQL Session
ss = SparkSession.builder \
.config(conf=conf) \
.getOrCreate()
# (1)
es_df = ss.read \
.format("org.elasticsearch.spark.sql") \
.options(pushdown=True) \
.load("bank") \
.where("gender='F'")
# (2) Print size of every partition on nodes
def f(iterable):
# need to convert iterable to list to have len()
print("Fetched % rows on node") % len(list(iterable))
es_df.foreachPartition(f)
# (3) I need to collect the result to show them on the console
data_list = es_df.collect()
print ("Printing 10 first results")
for x in data_list[0:10]:
print x
# Print count
print ("Fetched %s women accounts (from collected list)") % len (data_list)
</pre>
<p>
<b>Results</b>
</p>
<ul>
<li><b>Logs of the Spark Driver</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/2_collocation_1_bank_one_shard.log">2_collocation_1_bank_one_shard.log</a>
<li><b>Screenshots from the various admin console after the test execution</b>:
</ul>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_2_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_2_1.png' />
</div>
</a>
<figcaption><b>Test 2-1 / Dataset in ES</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_2_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_2_1.png' />
</div>
</a>
<figcaption><b>Test 2-1 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_1.png' />
</div>
</a>
<figcaption><b>Test 2-1 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_1_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_1_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 2-1 / Job 0 / Stage 0</b></figcaption>
</figure>
</div>
</div>
<p>
<b>Conclusions</b>
</p>
<ul>
<li>
<b>In terms of workload distribution, this is where Dynamic Allocation is really cool.</b> Since the single shard is on a single node, the Spark Dynamic Allocation System, with the help of Mesos, takes care of booking that single node as well for the Spark processing job
</li>
<li>
As a sidenote, using static allocation here, mesos Spark would have booked the whole cluster for the job, which would have been far from optimal in terms of workload distribution. Since the cluster is fully available, Mesos would have booked it all for the Job to come. But eventually 2 of the 3 spark executors won't be used at all.
<br>
That wouldn't have been a big deal since this test is running alone. But if some more jobs are added to the cluster and requests an executor, they will have to wait for that first job to be finished before they can share the cluster among them.
</li>
<li>Data-locality works as expected. The single shard is located on <code>192.168.10.12</code> and both the driver logs and the Spark Console for Job 0 / Stage 0 confirms that the co-located Spark executor has been the only one processing the data.
</li>
</ul>
<a name="sec522"></a>
<h4>5.2.2 Bank dataset with 2 shards</h4>
<p>
<b>Test details</b>
</p>
<ul>
<li><b>Test Script</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/tests/scenarii/2_collocation_2_bank_two_shards.sh"><code>2_collocation_2_bank_two_shards.sh</code></a></li>
<li><b>Input Dataset</b>: Bank Dataset from <a href="https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip">https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip</a></li>
<li><b>Purpose</b>: Assess how data-locality optimization works when using a dataset with a two shards on two nodes of the cluster, see what decisions will Mesos / Spark take from ES-Hadoop's requirements.
</ul>
<p>
<b>Expected Behaviour</b>
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/system_archi_data_2_2.png">
<img class="centered" style="width: 600px;" alt="Data Architecture Test 2_2" src="https://www.niceideas.ch/es_spark/images/system_archi_data_2_2.png" />
</a>
</div>
<br>
<p>
<b>Relevant portion of spark Script</b>
</p>
<pre>
from pyspark.conf import SparkConf
from pyspark.sql import SQLContext, SparkSession
# Spark configuration
conf = SparkConf().setAppName("ESTest_2_2")
# Spark SQL Session
ss = SparkSession.builder \
.config(conf=conf) \
.getOrCreate()
# (1)
es_df = ss.read \
.format("org.elasticsearch.spark.sql") \
.options(pushdown=True) \
.load("bank") \
.where("gender='F'")
# (2) Print size of every partition on nodes
def f(iterable):
# need to convert iterable to list to have len()
print("Fetched % rows on node") % len(list(iterable))
es_df.foreachPartition(f)
# (3) I need to collect the result to show them on the console
data_list = es_df.collect()
print ("Printing 10 first results")
for x in data_list[0:10]:
print x
# Print count
print ("Fetched %s women accounts (from collected list)") % len (data_list)
</pre>
<p>
<b>Results</b>
</p>
<ul>
<li><b>Logs of the Spark Driver</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/2_collocation_2_bank_two_shards.log">2_collocation_2_bank_two_shards.log</a>
<li><b>Screenshots from the various admin console after the test execution</b>:
</ul>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_2_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_2_2.png' />
</div>
</a>
<figcaption><b>Test 2-2 / Dataset in ES</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_2_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_2_2.png' />
</div>
</a>
<figcaption><b>Test 2-2 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_2.png' />
</div>
</a>
<figcaption><b>Test 2-2 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_2_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_2_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 2-2 / Job 0 / Stage 0</b></figcaption>
</figure>
</div>
</div>
<p>
<b>Conclusions</b>
</p>
<ul>
<li>
Same remark as above regarding workload distribution.
<br>
<b>Dynamic Allocation is really cool.</b> Since the two shards are on two nodes, the Spark Dynamic Allocation System, with the help of Mesos, takes care of booking the two corresponding nodes as well for the Spark processing job.
</li>
</li>
<li>Data locality works as expected. The two shards are on <code>192.168.10.10</code> and <code>192.168.10.12</code> and both the driver logs and the Spark Console for Job 0 / Stage 0 confirms that both co-located Spark executor have been used to process the 2 shards.
<br>
The tasks have been executed with <code>NODE_LOCAL</code> locality level.
</li>
</ul>
<a name="sec523"></a>
<h4>5.2.3 Bank dataset with 3 shards</h4>
<p>
<b>Test details</b>
</p>
<ul>
<li><b>Test Script</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/tests/scenarii/2_collocation_3_bank_three_shards.sh"><code>2_collocation_3_bank_three_shards.sh</code></a></li>
<li><b>Input Dataset</b>: Bank Dataset from <a href="https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip">https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip</a></li>
<li><b>Purpose</b>: Assess how data-locality optimization works when using a dataset with a three shards on three nodes of the cluster, see what decisions will Mesos / Spark take from ES-Hadoop's requirements.
</ul>
<p>
<b>Expected Behaviour</b>
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/system_archi_data_2_3.png">
<img class="centered" style="width: 600px; " alt="Data Architecture Test 2_3" src="https://www.niceideas.ch/es_spark/images/system_archi_data_2_3.png" />
</a>
</div>
<br>
<p>
<b>Relevant portion of spark Script</b>
</p>
<pre>
from pyspark.conf import SparkConf
from pyspark.sql import SQLContext, SparkSession
# Spark configuration
conf = SparkConf().setAppName("ESTest_2_3")
# Spark SQL Session
ss = SparkSession.builder \
.config(conf=conf) \
.getOrCreate()
# (1)
es_df = ss.read \
.format("org.elasticsearch.spark.sql") \
.options(pushdown=True) \
.load("bank") \
.where("gender='F'")
# (2) Print size of every partition on nodes
def f(iterable):
# need to convert iterable to list to have len()
print("Fetched % rows on node") % len(list(iterable))
es_df.foreachPartition(f)
# (3) I need to collect the result to show them on the console
data_list = es_df.collect()
print ("Printing 10 first results")
for x in data_list[0:10]:
print x
# Print count
print ("Fetched %s women accounts (from collected list)") % len (data_list)
</pre>
<p>
<b>Results</b>
</p>
<ul>
<li><b>Logs of the Spark Driver</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/2_collocation_3_bank_three_shards.log">2_collocation_3_bank_three_shards.log</a>
<li><b>Screenshots from the various admin console after the test execution</b>:
</ul>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_2_3.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_2_3.png' />
</div>
</a>
<figcaption><b>Test 2-3 / Dataset in ES</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_2_3.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_2_3.png' />
</div>
</a>
<figcaption><b>Test 2-3 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_3.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_3.png' />
</div>
</a>
<figcaption><b>Test 2-3 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_3_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_3_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 2-3 / Job 0 / Stage 0</b></figcaption>
</figure>
</div>
</div>
<p>
<b>Conclusions</b>
</p>
<ul>
<li>
Three shards on three nodes so three nodes booked for processing, everything works as expected.
</li>
<li>Data locality works as expected. The 3 spark executors <i>consumes</i> data from their co-located shards. This is confirmed by everything behaving as expected as can be seen in the driver logs or in the Spark Application UI.
</li>
</ul>
<a name="sec524"></a>
<h4>5.2.4 Bank dataset with 1 shard and replicas</h4>
<p>
<b>Test details</b>
</p>
<ul>
<li><b>Test Script</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/tests/scenarii/2_collocation_4_bank_one_shard_with_replicas.sh"><code>2_collocation_4_bank_one_shard_with_replicas.sh</code></a></li>
<li><b>Input Dataset</b>: Bank Dataset from <a href="https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip">https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip</a></li>
<li><b>Purpose</b>: Assess how data-locality works when using a dataset with one shard and two replicas on three nodes of the cluster, see what decisions will Mesos / Spark take from ES-Hadoop's requirements.
</ul>
<p>
<b>Expected Behaviour</b>
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/system_archi_data_2_4.png">
<img class="centered" style="width: 600px;" alt="Data Architecture Test 2_4" src="https://www.niceideas.ch/es_spark/images/system_archi_data_2_4.png" />
</a>
</div>
<br>
<p>
<b>Relevant portion of spark Script</b>
</p>
<pre>
from pyspark.conf import SparkConf
from pyspark.sql import SQLContext, SparkSession
# Spark configuration
conf = SparkConf().setAppName("ESTest_2_4")
# FIXME GET RID OF THESE TESTS
# Trying some ES settings
# conf.set ("spark.es.input.max.docs.per.partition", 100)
# That doesn't really help => it split the dataframe on several nodes indeed
# but it doesn't impact the fetching
# Spark SQL Session
ss = SparkSession.builder \
.config(conf=conf) \
.getOrCreate()
# (1)
es_df = ss.read \
.format("org.elasticsearch.spark.sql") \
.options(pushdown=True) \
.load("bank") \
.where("gender='F'")
# (2) Print size of every partition on nodes
def f(iterable):
# need to convert iterable to list to have len()
print("Fetched % rows on node") % len(list(iterable))
es_df.foreachPartition(f)
# (3) I need to collect the result to show them on the console
data_list = es_df.collect()
print ("Printing 10 first results")
for x in data_list[0:10]:
print x
# Print count
print ("Fetched %s women accounts (from collected list)") % len (data_list)
</pre>
<p>
<b>Results</b>
</p>
<ul>
<li><b>Logs of the Spark Driver</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/2_collocation_4_bank_one_shard_with_replicas.log">2_collocation_4_bank_one_shard_with_replicas.log</a>
<li><b>Screenshots from the various admin console after the test execution</b>:
</ul>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_2_4.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_2_4.png' />
</div>
</a>
<figcaption><b>Test 2-4 / Dataset in ES</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_2_4.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_2_4.png' />
</div>
</a>
<figcaption><b>Test 2-4 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_4.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_4.png' />
</div>
</a>
<figcaption><b>Test 2-4 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_4_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_4_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 2-4 / Job 0 / Stage 0</b></figcaption>
</figure>
</div>
</div>
<p>
<b>Conclusions</b>
</p>
<ul>
<li>
In this case, it's really as if each and every ES node of the cluster has a copy of the data. ElasticSearch makes no distinction when it comes to serve requests between primary shards and secondary shards (replicas).
</li>
<li>
Which node will finally execute the processing is really random. Out of several executions I always ended up having a different Spark node executing the whole processing. Data co-locality still kicks in and one single node still does the whole processing every time
</li>
<li>
<b>Important note: All of the above works under normal behaviour. Under a heavily loaded cluster, the results can be significantly different. </b>
<br>
By running different scenarii under different conditions, I have been able to determine 2 different situations in addition to the nominal one (the one on a free cluster):
<ul>
<li>
First, It can happen that Mesos tries to distribute a specific spark processing part to the Spark executor co-located to the ES shard.
<br>
But then, when the Spark processing finally queries that local node to get the shard, it can well happen that this ES node is busy answering a different request from a different client application.
<br>
In this case, that local ES node will report itself as busy and will ask another node from the ES cluster to server the request.
<br>
So even though initially Mesos / Spark distributed the workload to the local node to the shard in ES, <b>eventually the request will be served by another distant node from te cluster</b>.
</li>
<li>
Second, it can also happen that all nodes co-located to the ES node owning all shards (primary and replicas) are busy.
<br>
In this case, Mesos / Spark will only wait a few seconds expecting of this node to become free, and if that fails to happen, <b>eventually a different Mesos node will run the processing, indifferently for data locality</b>.
</li>
</ul>
</li>
<li>
The difference here is that the existence of replicas suddenly <b>gives ElasticSearch the choice</b>
<br>
ElasticSearch has the choice to answer and serve the data from another node than the local node if suddenly the local node is busy!
</li>
<li>
In addition, Mesos / Spark will only wait <code>spark.locality.wait=10s</code> to try to make the specific processing part local to the ES node owning a shard (or a replica BTW). If none of these nodes (owning one of the primary shard or replicas) becomes free and available for that amount of time, then Mesos will distribute the workload to another available node from the Mesos cluster.
</li>
</ul>
<a name="sec525"></a>
<h4>5.2.5 Testing repartitioning</h4>
<p>
<b>Test details</b>
</p>
<ul>
<li><b>Test Script</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/tests/scenarii/2_collocation_5_bank_one_shard_repartition_NOT_WORKING.sh"><code>2_collocation_5_bank_one_shard_repartition_NOT_WORKING.sh</code></a></li>
<li><b>Input Dataset</b>: Bank Dataset from <a href="https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip">https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip</a></li>
<li><b>Purpose</b>: Assess how one can redistribute the data on the cluster after loading data from a sub-set of the cluster nodes such as, for instance, only one node, see how Spark can redistribute the data evenly on the cluster after having loaded an unbalanced data.
</ul>
<p>
<b>Expected Behaviour</b>
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/system_archi_data_2_5.png">
<img class="centered" style="width: 600px;" alt="Data Architecture Test 2_5" src="https://www.niceideas.ch/es_spark/images/system_archi_data_2_5.png" />
</a>
</div>
<br>
<p>
<b>Relevant portion of spark Script</b>
</p>
<pre>
from pyspark.conf import SparkConf
from pyspark.sql import SQLContext, SparkSession
# Spark configuration
conf = SparkConf().setAppName("ESTest_2_5")
# Spark SQL Session
ss = SparkSession.builder \
.config(conf=conf) \
.getOrCreate()
# (1)
es_df = ss.read \
.format("org.elasticsearch.spark.sql") \
.options(pushdown=True) \
.load("bank") \
.where("gender='F'")
# Print size of every partition on nodes
def f(iterable):
# need to convert iterable to list to have len()
print("A - % rows stored on node") % len(list(iterable))
es_df.foreachPartition(f)
# Doesn't help
#es_df2 = es_df.coalesce(1)
## Print size of every partition on nodes
#es_df2.foreachPartition(f)
# (2)
es_df3 = es_df.repartition(4 * 3)
# Print size of every partition on nodes
es_df3.foreachPartition(f)
# (3) I need to collect the result to show them on the console
data_list = es_df3.collect()
print ("Printing 10 first results")
for x in data_list[0:10]:
print x
# Print count
print ("Fetched %s women accounts (from collected list)") % len (data_list)
# Print
print (ss._jsc.sc().getExecutorMemoryStatus().size())
</pre>
<p>
<b>Results</b>
</p>
<ul>
<li><b>Logs of the Spark Driver</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/2_collocation_5_bank_one_shard_repartition_NOT_WORKING.log">2_collocation_5_bank_one_shard_repartition.log</a>
<li><b>Screenshots from the various admin console after the test execution</b>:
</ul>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_2_5.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_2_5.png' />
</div>
</a>
<figcaption><b>Test 2-5 / Dataset in ES</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_2_5.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_2_5.png' />
</div>
</a>
<figcaption><b>Test 2-5 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_5.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_5.png' />
</div>
</a>
<figcaption><b>Test 2-5 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_5_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_2_5_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 2-5 / Job 0 / Stage 0</b></figcaption>
</figure>
</div>
</div>
<p>
<b>Conclusions</b>
</p>
<ul>
<li><b>I haven't been able to make repartitioning work the way I intended it to work</b>.
<ul>
<li>Eventually all of my tests led to the underlying RDD being repartitioned, but all the partitions remain local to the initially owning node</li>
<li>I never managed to find a way to make the Spark cluster redistribute the different partitions to the various Spark executors from the cluster</li>
</ul>
<li>
I don't know if that comes from Spark somehow <i>knowing</i> that it doesn't need to do that for the post-processing to be done efficiently.
</li>
<li>
Long story short, I have no real conclusions in this regards, reason why the above schema is crossed by an X.
</li>
</ul>
<a name="sec53"></a>
<h3>5.3 Aggregation tests</h3>
<p>
<b>Aggregation tests</b> - assess how aggregation on ES data works.
</p>
<a name="sec531"></a>
<h4>5.3.1 ES-side Aggregations</h4>
<p>
<b>Test details</b>
</p>
<ul>
<li><b>Test Script</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/tests/scenarii/3_aggregation_1_es_shakespeare_rdd_legacy_NOT_WORKING.sh"><code>3_aggregation_1_es_shakespeare_rdd_legacy_NOT_WORKING.sh</code></a></li>
<li><b>Input Dataset</b>: Shakespeare's Works Dataset from <a href="https://download.elastic.co/demos/kibana/gettingstarted/shakespeare.json">https://download.elastic.co/demos/kibana/gettingstarted/shakespeare.json</a></li>
<li><b>Purpose</b>: see how Spark can exploit native ElasticSearch features such as ES-side aggregations instead of performing aggregations on its own.
</ul>
<p>
<b>Expected Behaviour</b>
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/system_archi_data_3_1.png">
<img class="centered" style="width: 600px;" alt="Data Architecture Test 3_1" src="https://www.niceideas.ch/es_spark/images/system_archi_data_3_1.png" />
</a>
</div>
<br>
<p>
<b>Relevant portion of spark Script</b>
</p>
<pre>
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext
# Spark configuration
conf = SparkConf().setAppName("ESTest_3_1")
# SparkContext and SQLContext
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)
# -> query dsl
es_aggregations_query = '''
{
"query" : { "match_all": {} },
"size" : 0,
"aggregations" : {
"play_name": {
"terms": {
"field" : "play_name"
}
}
}
}
'''
es_read_conf = {
"es.resource" : "shakespeare",
"es.endpoint" : "_search",
"es.query" : es_aggregations_query
}
# (1)
es_rdd = sc.newAPIHadoopRDD(
inputFormatClass="org.elasticsearch.hadoop.mr.EsInputFormat",
keyClass="org.apache.hadoop.io.NullWritable",
valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",
conf=es_read_conf)
es_df = sqlContext.createDataFrame(es_rdd)
# I need to collect the result
data_list = es_df.collect()
print ("Printing 10 first results")
for x in data_list[0:10]:
print x
# Print count
print ("Fetched %s rows (from collected list)") % len (data_list)
</pre>
<p>
<b>Results</b>
</p>
<ul>
<li><b>Logs of the Spark Driver</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/3_aggregation_1_es_shakespeare_rdd_legacy_NOT_WORKING.log">3_aggregation_1_es_shakespeare_rdd_legacy.log</a>
<li><b>Screenshots from the various admin console after the test execution</b>:
</ul>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_3_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_3_1.png' />
</div>
</a>
<figcaption><b>Test 3-1 / Dataset in ES</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_3_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_3_1.png' />
</div>
</a>
<figcaption><b>Test 3-1 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_3_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_3_1.png' />
</div>
</a>
<figcaption><b>Test 3-1 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_3_1_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_3_1_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 3-1 / Job 0 / Stage 0</b></figcaption>
</figure>
</div>
</div>
<p>
<b>Conclusions</b>
</p>
<ul>
<li>
<b>There is simply no way at the moment to submit specific requests, such as aggregation requests, from spark to ElasticSearch using the ES-Hadoop connector.</b>
</li>
<li>
The need is well identified but the solution is still work in progress: <a href="https://github.com/elastic/elasticsearch-hadoop/issues/276">https://github.com/elastic/elasticsearch-hadoop/issues/276</a>
</li>
<li>
Since it's impossible to make this work as expected, the schematic above is crossed with an X.
</li>
</ul>
<a name="sec532"></a>
<h4>5.3.2 Spark-side Aggregations</h4>
<p>
<b>Test details</b>
</p>
<ul>
<li><b>Test Script</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/tests/scenarii/3_aggregation_2_spark_shakespeare.sh"><code>3_aggregation_2_spark_shakespeare.sh</code></a></li>
<li><b>Input Dataset</b>: Shakespeare's Works Dataset from <a href="https://download.elastic.co/demos/kibana/gettingstarted/shakespeare.json">https://download.elastic.co/demos/kibana/gettingstarted/shakespeare.json</a></li>
<li><b>Purpose</b>: see how Spark performs aggregations on its own.
</ul>
<p>
<b>Expected Behaviour</b>
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/system_archi_data_3_2.png">
<img style="width: 600px;" alt="Data Architecture Test 3_2" src="https://www.niceideas.ch/es_spark/images/system_archi_data_3_2.png" />
</a>
</div>
<br>
<p>
<b>Relevant portion of spark Script</b>
</p>
<pre>
from pyspark.conf import SparkConf
from pyspark.sql import SQLContext, SparkSession
# Spark configuration
conf = SparkConf().setAppName("ESTest_3_2")
# Every time there is a shuffle, Spark needs to decide how many partitions will
# the shuffle RDD have.
# 2 times the amount of CPUS on the cluster is a good value (default is 200)
conf.set("spark.sql.shuffle.partitions", "12")
# Spark SQL Session
ss = SparkSession.builder \
.config(conf=conf) \
.getOrCreate()
# Query configuration only (cannot pass any ES conf here :-( )
es_query_conf= {
"pushdown": True
}
# (1)
es_df = ss.read \
.format("org.elasticsearch.spark.sql") \
.options(conf=es_query_conf) \
.load("shakespeare*")
# (2) Compute aggregates : I want the count of lines per book
agg_df = es_df.groupBy(es_df.play_name).count()
# (3) Collect result to the driver
data_list = agg_df.collect()
print ("Printing 10 first results")
for x in data_list[0:10]:
print x
# Print count
print ("Fetched %s rows (from collected list)") % len (data_list)
</pre>
<p>
<b>Results</b>
</p>
<ul>
<li><b>Logs of the Spark Driver</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/3_aggregation_2_spark_shakespeare.log">3_aggregation_2_spark_shakespeare.log</a>
<li><b>Screenshots from the various admin console after the test execution</b>:
</ul>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_3_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_3_2.png' />
</div>
</a>
<figcaption><b>Test 3-2 / Dataset in ES</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_3_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_3_2.png' />
</div>
</a>
<figcaption><b>Test 3-2 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_3_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_3_2.png' />
</div>
</a>
<figcaption><b>Test 3-2 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_3_2_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_3_2_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 3-2 / Job 0 / Stage 0</b></figcaption>
</figure>
</div>
</div>
<p>
<b>Conclusions</b>
</p>
<ul>
<li>There aren't a lof of things to conclude here. We can just mention that everything works as expected and return the user to the Data Flow schematic above.</li>
<li>Data locality kicks-in, etc.
</ul>
<a name="sec54"></a>
<h3>5.4 Join test</h3>
<p>
<b>Test details</b>
</p>
<ul>
<li><b>Test Script</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/tests/scenarii/4_join_1_swissdata_df.sh"><code>4_join_1_swissdata_df.sh</code></a></li>
<li><b>Input Dataset</b>: two datasets in fact:
<ul>
<li>
from <a href="http://niceideas.ch/mes/swissairbnb/tomslee_airbnb_switzerland_1451_2017-07-11.csv">http://niceideas.ch/mes/swissairbnb/tomslee_airbnb_switzerland_1451_2017-07-11.csv</a> : the list of AirBnB offers in Switzerland as of July 2017.
</li>
<li>
from <a href="http://niceideas.ch/mes/swissairbnb/swisscitiespop.txt">http://niceideas.ch/mes/swissairbnb/swisscitiespop.txt</a> : the list of swiss cities with population and geoloc information.
</ul>
</li>
<li><b>Purpose</b>: see how the ELK-MS stack behaves when its has several datasets to load from ES into Spark and then join.
</ul>
<p>
<b>Expected Behaviour</b>
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/system_archi_data_4_1.png">
<img class="centered" style="width: 600px;" alt="Data Architecture Test 4_1" src="https://www.niceideas.ch/es_spark/images/system_archi_data_4_1.png" />
</a>
</div>
<br>
<p>
<b>Relevant portion of spark Script</b>
</p>
<pre>
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext, SparkSession
import pyspark.sql.functions as F
# Spark configuration
# all these options can be given to the command line to spark-submit
# (they would need to be prefixed by "spark.")
conf = SparkConf().setAppName("ESTest_4_1")
# Every time there is a shuffle, Spark needs to decide how many partitions will
# the shuffle RDD have.
# 2 times the amount of CPUS oi nthe cluster is a good value (default is 200)
conf.set("spark.sql.shuffle.partitions", "12")
# Spark SQL Session
ss = SparkSession.builder \
.config(conf=conf) \
.getOrCreate()
# Query configuration only (cannot pass any ES conf here :-( )
es_query_conf= {
"pushdown": True
}
# (1).1 Read city and population
citypop_df = ss.read \
.format("org.elasticsearch.spark.sql") \
.options(conf=es_query_conf) \
.load("swiss-citypop") \
.alias("citypop_df")
# (1).2. Read airbnb offers
airbnb_df = ss.read \
.format("org.elasticsearch.spark.sql") \
.options(conf=es_query_conf) \
.load("swiss-airbnb") \
.alias("airbnb_df")
# (2) Join on city
joint_df = airbnb_df \
.join( \
citypop_df, \
(F.lower(airbnb_df.city) == F.lower(citypop_df.accent_city)), \
"left_outer" \
) \
.select( \
'room_id', 'airbnb_df.country', 'airbnb_df.city', \
'room_type', 'bedrooms', 'bathrooms', 'price', 'reviews', \
'overall_satisfaction', \
'airbnb_df.latitude', 'airbnb_df.longitude', \
'citypop_df.latitude', 'citypop_df.longitude', 'population', \
'region' \
)
# (3) Collect result to the driver
data_list = joint_df.collect()
print ("Printing 10 first results")
for x in data_list[0:10]:
print x
# Print count
print ("Computed %s positions (from collected list)") % len (data_list)
</pre>
<p>
<b>Results</b>
</p>
<ul>
<li><b>Logs of the Spark Driver</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/4_join_1_swissdata_df.log">4_join_1_swissdata_df.log</a>
<li><b>Screenshots from the various admin console after the test execution</b>:
</ul>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_4_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_cerebro/result_4_1.png' />
</div>
</a>
<figcaption><b>Test 4-1 / Dataset in ES</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_4_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_4_1.png' />
</div>
</a>
<figcaption><b>Test 4-1 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_4_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_4_1.png' />
</div>
</a>
<figcaption><b>Test 4-1 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_4_1_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_4_1_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 4-1 / Job 0 / Stage 0</b></figcaption>
</figure>
</div>
</div>
<br>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_4_1_job_0_stage_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_4_1_job_0_stage_1.png' />
</div>
</a>
<figcaption><b>Test 4-1 / Job 0 / Stage 1</b></figcaption>
</figure>
</div>
</div>
<p>
<b>Conclusions</b>
</p>
<ul>
<li>
Here as well there aren't a log of things to conclude. Everything works just as expected.
</li>
<li>
Data locality kicks-in both at the ES data fetching side and on the private Spark Side for the join.
</li>
</ul>
<a name="sec55"></a>
<h3>5.5 Concurrency test</h3>
<p>
<b>Test details</b>
</p>
<ul>
<li><b>Test Script</b>: <a href="https://www.niceideas.ch/es_spark/sandbox/tests/scenarii/5_concurrency_1_swissdata_df.sh"><code>5_concurrency_1_swissdata_df.sh</code></a></li>
<li><b>Input Dataset</b>: two datasets in fact:
<ul>
<li>
from <a href="http://niceideas.ch/mes/swissairbnb/tomslee_airbnb_switzerland_1451_2017-07-11.csv">http://niceideas.ch/mes/swissairbnb/tomslee_airbnb_switzerland_1451_2017-07-11.csv</a> : the list of AirBnB offers in Switzerland as of July 2017.
</li>
<li>
from <a href="http://niceideas.ch/mes/swissairbnb/swisscitiespop.txt">http://niceideas.ch/mes/swissairbnb/swisscitiespop.txt</a> : the list of swiss cities with population and geoloc information.
</ul>
</li>
<li><b>Purpose</b>: see how the ELK-MS stack behaves when submitting several jobs at the same time to the cluster and what happens in terms of concurrency.
</ul>
<p>
<b>The Spark Script</b>
</p>
<p>
The concurrency tests simply executes four times in parallel the scenario inspired from <a href="#sec532">5.3.2 Spark-side Aggregations</a> as follows:
</p>
<pre>
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext, SparkSession
import pyspark.sql.functions as F
# Spark configuration
conf = SparkConf()
# Every time there is a shuffle, Spark needs to decide how many partitions will
# the shuffle RDD have.
# 2 times the amount of CPUS on the cluster is a good value (default is 200)
conf.set("spark.sql.shuffle.partitions", "12")
# Spark SQL Session
ss = SparkSession.builder \
.config(conf=conf) \
.getOrCreate()
# Query configuration only (cannot pass any ES conf here :-( )
es_query_conf= {
"pushdown": True
}
# 1. Read city and population
citypop_df = ss.read \
.format("org.elasticsearch.spark.sql") \
.options(conf=es_query_conf) \
.load("swiss-citypop") \
.alias("citypop_df")
# 2. Read airbnb offers
airbnb_df = ss.read \
.format("org.elasticsearch.spark.sql") \
.options(conf=es_query_conf) \
.load("swiss-airbnb") \
.alias("airbnb_df")
# 3. Join on city
joint_df = airbnb_df \
.join( \
citypop_df, \
(F.lower(airbnb_df.city) == F.lower(citypop_df.accent_city)), \
"left_outer" \
) \
.select( \
'room_id', 'airbnb_df.country', 'airbnb_df.city', \
'room_type', 'bedrooms', 'bathrooms', 'price', 'reviews', \
'overall_satisfaction', \
'airbnb_df.latitude', 'airbnb_df.longitude', \
'citypop_df.latitude', 'citypop_df.longitude', 'population', \
'region' \
)
# Collect result to the driver
data_list = joint_df.collect()
print ("Printing 10 first results")
for x in data_list[0:10]:
print x
# Print count
print ("Computed %s positions (from collected list)") % len (data_list)
</pre>
<p>
<b>Results</b>
</p>
<ul>
<li><b>The various logs</b>:
<ul>
<li>Logs of the script : <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/5_concurrency_1_swissdata_df.log">5_concurrency_1_swissdata_df.log</a></li>
<li>Process P1 logs : <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/log_5_concurrency_1_swissdata_1.log">log_5_concurrency_1_swissdata_1.log</a></li>
<li>Process P2 logs : <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/log_5_concurrency_1_swissdata_2.log">log_5_concurrency_1_swissdata_2.log</a></li>
<li>Process P3 logs : <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/log_5_concurrency_1_swissdata_3.log">log_5_concurrency_1_swissdata_3.log</a></li>
<li>Process P4 logs : <a href="https://www.niceideas.ch/es_spark/sandbox/result_logs/log_5_concurrency_1_swissdata_4.log">log_5_concurrency_1_swissdata_4.log</a></li>
</ul>
<li><b>Screenshots from the various admin console after the test execution</b>:
</ul>
<p>
First the mesos console showing the completion of the 4 jobs:
</p>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_5_1_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_5_1_1.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P1 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_5_1_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_5_1_2.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P2 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_5_1_3.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_5_1_3.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P3 / Job Completion in mesos</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_5_1_4.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_mesos/mesos_fwk_5_1_4.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P4 / Job Completion in mesos</b></figcaption>
</figure>
</div>
</div>
<p>
Overview of the 4 processes in Spark console, the specific view of each of the process:
</p>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_1.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P1 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_2.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P2 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_3.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_3.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P3 / Process Overview on Spark</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_4.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_4.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P4 / Process Overview on Spark</b></figcaption>
</figure>
</div>
</div>
<p>
Focusing on Job 1 (P1), each and every relevant views from the Spark Application UI:
</p>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_1_job_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_1_job_0.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P1 / Job 0</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_1_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_1_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P1 / Job 0 / Stage 0</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_1_job_0_stage_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_1_job_0_stage_1.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P1 / Job 0 / Stage 1</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_1_job_0_stage_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_1_job_0_stage_2.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P1 / Job 0 / Stage 2</b></figcaption>
</figure>
</div>
</div>
<p>
Focusing on Job 2 (P2), each and every relevant views from the Spark Application UI:
</p>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_2_job_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_2_job_0.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P2 / Job 0</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_2_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_2_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P2 / Job 0 / Stage 0</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_2_job_0_stage_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_2_job_0_stage_1.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P2 / Job 0 / Stage 1</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_2_job_0_stage_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_2_job_0_stage_2.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P2 / Job 0 / Stage 2</b></figcaption>
</figure>
</div>
</div>
<p>
Focusing on Job 3 (P3), each and every relevant views from the Spark Application UI:
</p>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_3_job_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_3_job_0.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P3 / Job 0</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_3_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_3_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P3 / Job 0 / Stage 0</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_3_job_0_stage_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_3_job_0_stage_1.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P3 / Job 0 / Stage 1</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_3_job_0_stage_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_3_job_0_stage_2.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P3 / Job 0 / Stage 2</b></figcaption>
</figure>
</div>
</div>
<p>
Focusing on Job 4 (P4), each and every relevant views from the Spark Application UI:
</p>
<div class="container-wrapper">
<div class="container">
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_4_job_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_4_job_0.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P4 / Job 0</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_4_job_0_stage_0.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_4_job_0_stage_0.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P4 / Job 0 / Stage 0</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_4_job_0_stage_1.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_4_job_0_stage_1.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P4 / Job 0 / Stage 1</b></figcaption>
</figure>
<figure>
<a style="border: 0px none;" href="https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_4_job_0_stage_2.png">
<div class="container_inner">
<img src='https://www.niceideas.ch/es_spark/sandbox/result_spark/result_5_1_4_job_0_stage_2.png' />
</div>
</a>
<figcaption><b>Test 5-1 / P4 / Job 0 / Stage 2</b></figcaption>
</figure>
</div>
</div>
<p>
<b>Conclusions</b>
</p>
<ul>
<li>
Before everything else let's mention that this test has been executed, first, using the FIFO scheduler (<code>spark.scheduler.mode=FIFO</code>) and second, using the <b>Dynamic allocation system</b>
</li>
<li>
Dynamic allocation seems to work a little slower that static allocation in this case.
</li>
<li>
With static allocation (only actual way on ES-Hadoop 5.x), what happens is that the first job that is prepared by the drivre a tiny little bit before the 3 others will get the shole cluster, and only whenever that first job is done, the three next sones will get an even share of the cluster, i.e one node each and complete almost at the same time.
</li>
<li>
With dynamic allocation, the cluster is well shares among jobs. Once in a while a job may get an additional executor and another job will need to wait but all in all the 4 jobs really run together on the three nodes.
</li>
</ol>
</li>
<li>
<p>
In terms of concurrency, we can see on the following image that the cluster is used quite effectively, looking at the CPU consumption on the host machine:
</p><br>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/concurrency_performance.png">
<img class="centered" style="width: 800px; " alt="Concurrency test - view of CPU" src='https://www.niceideas.ch/es_spark/images/concurrency_performance.png' />
</a>
<span class="centered">
(Note : each and every of the 3 VMs can use up to 2 CPUS of the host which has 4 CPUs in total)
</span>
<br>
</div>
</li>
<li>
Also, all my tests, including this one has been executed using <b>Coarse Grained Scheduling Mode</b> (<code>spark.mesos.coarse=true</code>).
<ul>
<li>
One might think that using Fine Grained Mode, things would be more efficient since each and ever task would be distributed on the cluster at will and we wouldn't end up in the <i>static topology</i> described above.
</li>
<li>
But unfortunately, Mesos latency when it comes to negotiating resources really messes up performances. The dynamic dispatching of tasks works well, but the overall processes performances are screwed by the time Mesos requires for negotiation.
<br>
In the ends, Fine Grained Scheduling mode kills performance of the whole cluster down.
</li>
<li>
I have executed this very same test using <code>spark.mesos.coarse=false</code> and the dropdown in terms of cluster usage efficiency is seen by looking at the <a href="https://www.niceideas.ch/es_spark/images/concurrency_performance_FINE-GRAINED.png">CPU consumption on the host machine for test 5 - 1 using Fine Grained Mode</a>
</ul>
</li>
<li>
In regards to data locality, since the 3 last processes get one single node of the cluster each, only one third of the tasks will execute with locality level <code>NODE_LOCAL</code>. Two thirds of them will require to fetch data from the network.
</li>
</ul>
<a name="sec6"></a>
<h2>6. References</h2>
<p>
<b>Spark and mesos</b>
</p>
<ul>
<li>
<a href="https://spark.apache.org/docs/latest/running-on-mesos.html">https://spark.apache.org/docs/latest/running-on-mesos.html (specific spark mesos configuration)</a>
</li>
</ul>
<p>
<b>ES Hadoop doc</b>
</p>
<ul>
<li>
Spark : <a href="https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html">https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html</a>
</li>
<li>
Configuration : <a href="https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#_querying">https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#_querying</a>
</li>
</ul>
<p>
<b>Pyspark.sql doc</b>
</p>
<ul>
<li>
2.2 : <a href="https://spark.apache.org/docs/2.2.0/api/python/pyspark.sql.html">https://spark.apache.org/docs/2.2.0/api/python/pyspark.sql.html</a>
</li>
<li>
1.6 : <a href="https://spark.apache.org/docs/1.6.2/api/python/pyspark.sql.html">https://spark.apache.org/docs/1.6.2/api/python/pyspark.sql.html</a>
</li>
</ul>
<p>
<b>Spark Doc</b>
</p>
<ul>
<li>
Configuration : <a href="https://spark.apache.org/docs/latest/configuration.html">https://spark.apache.org/docs/latest/configuration.html</a>
</li>
<li>
Spark history server : <a href="https://spark.apache.org/docs/latest/monitoring.html">https://spark.apache.org/docs/latest/monitoring.html</a>
</li>
<li>
Dynamic resource allocation : <a href="https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation">https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation</a>
</li>
</ul>
<p>
<b>Other Pyspark specificities</b>
</p>
<ul>
<li>
Pyspark RDD API : <a href="https://spark.apache.org/docs/2.2.0/api/python/pyspark.html#pyspark.RDD">https://spark.apache.org/docs/2.2.0/api/python/pyspark.html#pyspark.RDD</a>
</li>
<li>
Pyspark performance : <a href="https://fr.slideshare.net/SparkSummit/getting-the-best-performance-with-pyspark">https://fr.slideshare.net/SparkSummit/getting-the-best-performance-with-pyspark</a>
</li>
</ul>
https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana
ELK-MS - ElasticSearch/LogStash/Kibana - Mesos/Spark : a lightweight and efficient alternative to the Hadoop Stack - part I : setup the cluster
Jerome Kehrli
2017-08-23T17:29:12-04:00
2020-07-22T03:11:59-04:00
<p>
In my current company, we implement heavy Data Analytics algorithms and use cases for our customers. Historically, these heavy computations were taking a whole lot of different forms, mostly custom computation scripts in python or else using RDBMS databases to store data and results.
<br>
A few years ago, we started to hit the limits of what we were able to achieve using traditional architectures and had to move both our storage and processing layers to NoSQL / Big Data technologies.
</p>
<p>
We considered a whole lot of different approaches, but eventually, and contrary to what I expected first, we didn't settle for a standard Hadoop stack. We are using ElasticSearch as key storage backend and Apache Spark as processing backend.
<br>
Now of course we were initially still considering a Hadoop stack for the single purpose of using YARN as resource management layer for Spark ... until we discovered Apache Mesos.
</p>
<p>
Today this state of the art ELK-MS - for ElasticSearch/Logstash/Kibana - Mesos/Spark stack performs amazingly and I believe it to be a really <b>lightweight, efficient, low latency and performing</b> alternative to a plain old Hadoop Stack.
<br>
I am writing a serie of two articles to present this stack and why it's cool.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/technical_archi_simple.png">
<img class="centered" style="width: 500px;" alt="ELK-MS Simple Technical Architecture" src="https://www.niceideas.ch/es_spark/images/technical_archi_simple.png" />
</a>
</div>
<br>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana">This first article - ELK-MS - part I : setup the cluster</a> in this serie presents the ELK-MS stack and how to set up a test cluster using the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS.tar.gz">niceideas ELK-MS package</a>.
</p>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1">The second article - ELK-MS - part II : assessing behaviour</a> presents a few concerns, assesses the expected behaviour using the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS_TEST.tar.gz">niceideas ELK-MS TEST package</a> and discusses challenges and constraints in this ELK-MS environment.
</p>
<p>
The conclusions of this serie of articles are presented in <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana2">the third and last article - ELK-MS - part III : so why is it cool?</a> which presents, as the name suggests, why this ELK-MS stack is really really cool and works great.
</p>
<p>
This article assumes a basic understanding of Hadoop and Big Data / NoSQL technologies in general by the reader.
</p>
<!-- ELK-MS - ElasticSearch/LogStash/Kibana - Mesos/Spark : a lightweight and efficient alternative to the Hadoop Stack - part I : setup the cluster -->
<p><i>
<b>Edited 2017-10-30</b>: I was using ES 5.0.0 with Spark 2.2.0 at the time of writing the initial version of this article.
<br>
With ElasticSearch 6.x, and ES-Hadoop 6.x, the game changes a little. The Spark 2.2.0 Dynamic allocation system is now perfectly compatible with the way ES-Hadoop 6.x enforces data locality optimization and everything works just as expected.
</i>
</p>
<p>
In my current company, we implement heavy Data Analytics algorithms and use cases for our customers. Historically, these heavy computations were taking a whole lot of different forms, mostly custom computation scripts in python or else using RDBMS databases to store data and results.
<br>
A few years ago, we started to hit the limits of what we were able to achieve using traditional architectures and had to move both our storage and processing layers to NoSQL / Big Data technologies.
</p>
<p>
We considered a whole lot of different approaches, but eventually, and contrary to what I expected first, we didn't settle for a standard Hadoop stack. We are using ElasticSearch as key storage backend and Apache Spark as processing backend.
<br>
Now of course we were initially still considering a Hadoop stack for the single purpose of using YARN as resource management layer for Spark ... until we discovered Apache Mesos.
</p>
<p>
Today this state of the art ELK-MS - for ElasticSearch/Logstash/Kibana - Mesos/Spark stack performs amazingly and I believe it to be a really <b>lightweight, efficient, low latency and performing</b> alternative to a plain old Hadoop Stack.
<br>
I am writing a serie of two articles to present this stack and why it's cool.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/technical_archi_simple.png">
<img class="centered" style="width: 500px;" alt="ELK-MS Simple Technical Architecture" src="https://www.niceideas.ch/es_spark/images/technical_archi_simple.png" />
</a>
</div>
<br>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana">This first article - ELK-MS - part I : setup the cluster</a> in this serie presents the ELK-MS stack and how to set up a test cluster using the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS.tar.gz">niceideas ELK-MS package</a>.
</p>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1">The second article - ELK-MS - part II : assessing behaviour</a> presents a few concerns, assesses the expected behaviour using the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS_TEST.tar.gz">niceideas ELK-MS TEST package</a> and discusses challenges and constraints in this ELK-MS environment.
</p>
<p>
The conclusions of this serie of articles are presented in <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana2">the third and last article - ELK-MS - part III : so why is it cool?</a> which presents, as the name suggests, why this ELK-MS stack is really really cool and works great.
</p>
<p>
This article assumes a basic understanding of Hadoop and Big Data / NoSQL technologies in general by the reader.
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. Introduction</a>
<ul>
<li><a href="#sec11">1.1 Rationality</a></li>
<li><a href="#sec12">1.2 Purpose of this serie of articles</a></li>
</ul>
</li>
<li><a href="#sec2">2. Target Architecture</a>
<ul>
<li><a href="#sec21">2.1 Technical Architecture</a></li>
<li><a href="#sec22">2.2 Components</a>
<ul>
<li><a href="#sec221">2.2.1 ElasticSearch</a></li>
<li><a href="#sec222">2.2.2 Logstash</a></li>
<li><a href="#sec223">2.2.3 Kibana</a></li>
<li><a href="#sec224">2.2.4 Cerebro</a></li>
<li><a href="#sec225">2.2.5 Spark</a></li>
<li><a href="#sec226">2.2.6 Mesos</a></li>
<li><a href="#sec227">2.2.7 Spark on Mesos specificities</a></li>
</ul>
</li>
<li><a href="#sec23">2.3 Making it work together : ES-Hadoop</a></li>
<li><a href="#sec24">2.4 Application Architecture</a></li>
</ul>
</li>
<li><a href="#sec3">3. niceideas ELK-MS</a>
<ul>
<li><a href="#sec31">3.1 System and principles</a></li>
<li><a href="#sec32">3.2 The build system</a>
<ul>
<li><a href="#sec321">3.2.1 Required Tools</a></li>
<li><a href="#sec322">3.2.2 Build System Project Layout</a></li>
</ul>
</li>
<li><a href="#sec33">3.3 Calling the build system and results</a></li>
<li><a href="#sec34">3.4 Testing the System</a></li>
<li><a href="#sec35">3.5 Tips & Tricks</a></li>
</ul>
</li>
<li><a href="#sec4">4. Noteworthy configuration elements</a>
<ul>
<li><a href="#sec41">4.1 NTP</a></li>
<li><a href="#sec42">4.2 Zookeeper</a></li>
<li><a href="#sec43">4.3 Elasticsearch</a></li>
<li><a href="#sec44">4.4 Logstash, Kibana, Cerebro</a></li>
<li><a href="#sec45">4.5 Mesos</a></li>
<li><a href="#sec46">4.6 Spark</a></li>
<li><a href="#sec47">4.7 ES-Hadoop</a></li>
</ul>
</li>
<li><a href="#sec4">5. Conclusion</a></li>
</ul>
<a name="sec1"></a>
<h2>1. Introduction </h2>
<p>
Actually deploying a whole Hadoop stack is, let's say, at least <i>heavy</i>. Having HDFS, YARN, the Map Reduce framework and maybe Tez up and running is one thing, and it's maybe not that complicated, sure.
</p>
<p>
But with such a <i>vanilla</i> stack you're not going very far. You'll at least add the following minimal set of software components: <a href="http://sqoop.apache.org/">Apache Sqoop</a> for importing data in your HDFS cluster, <a href="https://pig.apache.org/">Apache Pig</a> for processing this data, <a href="https://hive.apache.org/">Apache Hive</a> for querying it. But yeah, then, Hive is so slow for small queries returning small datasets, you'll likely add Stinger ... and then a whole lot of other components.
<br>
Now setting all of these software components up and running and tuning them well is a real hassle so one might consider a HortonWorks or Cloudera distribution instead, and this is where it gets really heavy.
<br>
Don't get me wrong, both HortonWorks and Cloudera are doing an amazing job and their distributions are awesome.
<br>
But I am working in a context where we want something lighter, something more efficient, something easier to set up, master and monitor.
</p>
<p>
In addition, HDFS is great. But it's really only about distributed storage of data. Vanilla Hadoop doesn't really provide anything on top of this data aside from MapReduce. On the other hand, the NoSQL landscape is filled with plenty of solutions achieving the same resilience and performance than HDFS but providing advanced data querying features on top of this data.
<br>
Among all these solutions, <a href="https://www.elastic.co/products/elasticsearch">ElasticSearch</a> is the <i>one stop shop</i> for our use cases. It fulfills 100% of our requirements and provides us out of the box with all the querying features we require (and some striking advanced features).
<br>
Using ElasticSearch for our data storage needs, we have no usage whatsoever for HDFS.
<br>
In addition, ElasticSearch comes out of the box with a pretty awesome replacement of Sqoop: <a href="https://www.elastic.co/products/logstash">Logstash</a> and a brilliant Data Visualization tool that has no free alternative in the Hadoop world: <a href="https://www.elastic.co/products/kibana">Kibana</a>.
</p>
<p>
Now regarding Data Processing, here as well we found our <i>one stop shop</i> in the form of <a href="https://spark.apache.org/">Apache Spark</a>. Spark is a (vary) fast and general engine for large-scale data processing. At the ground of our processing needs, there is not one single use case we cannot map easily and naturally to Spark'S API, either using low level RDDs or using the DataFrame API (SparkSQL).
</p>
<p>
Now Spark requires some external scheduler and resources manager. It can run without it of course but fails in achieving concurrency when doing so.
<br>
We were seriously considering deploying Hadoop and YARN for this until we discovered <a href="http://mesos.apache.org/">Apache Mesos</a>. Mesos is a distributed systems kernel built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides Spark with API’s for resource management and scheduling across entire Data-center and cloud environments.
</p>
<a name="sec11"></a>
<h3>1.1 Rationality </h3>
<p>
I call the software stack formed by the above components the ELK-MS stack, for <i>ElasticSearch/LogStash/Kibana - Mesos/Spark</i>.
</p>
<p>
<b>The ELK-MS stack is a simple, lightweight, efficient, low-latency and performing alternative to the Hadoop stack providing state of the art Data Analytics features</b>:
</p>
<ul>
<li>
<b>lightweight</b> : ELK-MS is lightweight both in terms of <i>setup</i> and <i>runtime</i>.
<br>
In terms of setup, the distributed storage engine, ElasticSearch, the resource manager, Mesos, and the distributed processing engine, spark, are amazingly easy to setup and configure. They really work almost out of the box and only very few configuration properties have to be set when it comes to configuring resources in Mesos, honestly trying to optimize anything other than the default value really tends to worsen things.
<br>
In terms of runtime, ElasticSearch, Mesos and some components of Spark, the only long-running daemons have a very low memory footprint under low workload. Now of course, both ElasticSearch and Spark have pretty heavy memory needs when working.
</li>
<li>
<b>efficient</b> : ElasticSearch, in contrary to HDFS, is not just a wide and simple distributed storage engine. ElasticSearch is in addition a real-time querying engine. It provides pretty advanced features such as aggregations and, up to a certain level, even distributed processing (scripted fields or else). With ELK-MS, the storage layer itself provides basic data analytics features.
<br>
In addition, Spark supports through the RDD API most if not all of what we can achieve using low-level Map Reduce. It obviously also supports plain old MapReduce. But the really striking feature of Spark is the DataFrame API and SparqSQL.
</li>
<li>
<b>low-latency</b> : Spark is by design much faster than Hadoop. In addition, jobs on spark can be implemented in such as way that the processing time and job initialization time is much shorter than on Hadoop MapReduce (Tez makes things more even on Hadoop though). <br>
But there again Spark has a joker: the Spark Streaming extension.
</li>
<li>
<b>performing</b> : in addition to the above, both ElasticSearch and Spark share a common gene, not necessarily widely spread among the NoSQL landscape: the capacity to benefit as much from a big cluster with thousands of nodes than from a big machine with a hundreds of processor.
<br>
Spark and ElasticSearch are very good on a large cluster of small machines (and to be honest, the scaling out is really the preferred way to achieve optimal performance with both).
<br>
<b>But in contrary to Hadoop, both Spark and ElasticSearch also works pretty good on a single fat machine with hundreds of processors</b>, able to benefit from the multi-processor architecture of one single machine.
</li>
</ul>
<p>
The <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1#sec3">conclusions of the behaviour assessment tests</a>, at the end of the second article, as well as The <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana2">conclusion of this serie of articles</a> give some more leads on why the ELK-MS stack is cool.
</p>
<p>
For these reasons, we are extensively using the ELK-MS stack for our Data Analytics needs in my current company.
</p>
<a name="sec12"></a>
<h3>1.2 Purpose of this serie of articles</h3>
<p>
Setting up the ELK-MS stack in a nominal working mode is easy, but still requires a few steps. In addition, when assessing the stack and for testing purpose, I needed a way to setup a cluster and test key features such as optimization of data-locality between ElasticSearch and Spark.
</p>
<p>
I have written a set of scripts taking care of the nominal setup and a test framework based on <a href="https://www.vagrantup.com/">Vagrant</a> and <a href="https://www.virtualbox.org/">VirtualBox</a>.
</p>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana">This first article - ELK-MS - part I : setup the cluster</a> in this serie presents the ELK-MS stack and how to set up a test cluster using the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS.tar.gz">niceideas ELK-MS package</a>.
</p>
<p>
<a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1">The second article - ELK-MS - part II : assessing behaviour</a> presents a few concerns, assesses the expected behaviour using the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS_TEST.tar.gz">niceideas ELK-MS TEST package</a> and discusses challenges and constraints in this ELK-MS environment.
</p>
<p>
The conclusions of this serie of articles are presented in <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana2">the third and last article - ELK-MS - part III : so why is it cool?</a> which presents, as the name suggests, why this ELK-MS stack is really really cool and works great.
</p>
<a name="sec2"></a>
<h2>2. Target Architecture </h2>
<p>
Before presenting the components and some noteworthy configuration aspects, let's dig into the architecture of the ELK-MS stack.
</p>
<a name="sec21"></a>
<h3>2.1 Technical Architecture</h3>
<p>
The <b>technical architecture</b> of the ELK-MS stack is as follows
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/technical_archi_master.png">
<img class="centered" style="width: 600px; " alt="ELK-MS Technical Architecture" src="https://www.niceideas.ch/es_spark/images/technical_archi_master.png" />
</a>
</div>
<br>
<p>
The components in grey are provided out of the box at OS level by <b>Debian Stretch</b> distribution.
<br>
The components in yellow are provided by Elastic in the ELK Stack.
<br>
Mesos is in light red.
<br>
The components in blue are from the Spark Framework.
</p>
<p>
Let's present all these components.
</p>
<a name="sec22"></a>
<h3>2.2 Components</h3>
<p>
This section presents the most essential components of the ELK-MS stack.
</p>
<a name="sec221"></a>
<h4>2.2.1 ElasticSearch</h4>
<table style="border: 0px none;">
<tr>
<td style="width: 100px; border: 0px none;">
<img style="width: 100px; min-width: 100px; border: 0px none;" alt="ElasticSearch Logo" src="https://www.niceideas.ch/es_spark/images/elasticsearch_logo.png" />
</td>
<td style="width: 100%; border: 0px none;">
<p>
From <a href="https://www.elastic.co/products/elasticsearch">ElasticSearch's web site</a> : "<i>ElasticSearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.</i>"
</p>
<p>
ElasticSearch is a NoSQL Document-oriented database benefitting from the NoSQL Genes: data distribution by sharding (partitioning) and replication. It can run on all kind of hardware, from a big fat hundred CPUs machine to a multi-data centers cluster of commodity hardware.
<br>
The native document storage format is JSON.
</p>
<p>
ElasticSearch support real-time querying of data and advanced analytics features such as aggregation, scripted fields, advanced memory management models and even some support for MapReduce directly in ElasticSearch's engine.
</p>
</td>
</tr>
</table>
<a name="sec222"></a>
<h4>2.2.2 Logstash</h4>
<table style="border: 0px none;">
<tr>
<td style="width: 100px; border: 0px none;">
<img style="width: 100px; min-width: 100px; border: 0px none;" alt="Logstash Logo" src="https://www.niceideas.ch/es_spark/images/logstash_logo.png" />
</td>
<td style="width: 100%; border: 0px none;">
<p>
From <a href="https://www.elastic.co/products/logstash">Logstash's web site</a> : "<i>Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite "stash." (Ours is Elasticsearch, naturally.).</i>"
</p>
<p>
Logstash is really the equivalent of Sqoop in the Elastic world. It's a largely configurable data processing engine whose primary intent is to feed ElasticSearch with data that can come from pretty much all imaginable data sources and formats. Of course Logstash can also output data to a very extended set of sinks in addition to ElasticSearch.
<br>
It's easily extendable through plugins which are straightforward to build, should the 200 provided plugins not be sufficient.
</p>
<p>
Logstash can also be distributed just as ElasticSearch, enabling not only to scale out the data ingestion processing but also enabling smart co-location strategies with ElasticSearch.
</td>
</tr>
</table>
<a name="sec223"></a>
<h4>2.2.3 Kibana</h4>
<table style="border: 0px none;">
<tr>
<td style="width: 100px; border: 0px none;">
<img style="width: 100px; min-width: 100px; border: 0px none;" alt="Kibana Logo" src="https://www.niceideas.ch/es_spark/images/kibana_logo.png" />
</td>
<td style="width: 100%; border: 0px none;">
<p>
From <a href="https://www.elastic.co/products/kibana">Kibana's web site</a> : "<i>Kibana lets you visualize your ElasticSearch data and navigate the Elastic Stack, so you can do anything from learning why you're getting paged at 2:00 a.m. to understanding the impact rain might have on your quarterly numbers.</i>"
</p>
<p>
Kibana core ships with the classics: histograms, line graphs, pie charts, sunbursts, and more. They leverage the full aggregation capabilities of ElasticSearch.
<br>
Kibana as well is easily extendable and integrating any kind of native D3.js visualization is usually done in a few hours of coding.
</p>
<p>
<b>In the context of ELK-MS</b>, Kibana is an amazing addition to ElasticSearch, since we can write Spark programs that work with data from ES but also stores their results in ES. As such, Kibana can be used out of the box to visualize not only the input data but also the results of the Spark scripts.
</td>
</tr>
</table>
<a name="sec224"></a>
<h4>2.2.4 Cerebro</h4>
<table style="border: 0px none;">
<tr>
<td style="width: 100px; border: 0px none;">
<img style="width: 100px; min-width: 100px; border: 0px none;" alt="Cerebro Logo" src="https://www.niceideas.ch/es_spark/images/cerebro_logo.png" />
</td>
<td style="width: 100%; border: 0px none;">
<p>
From <a href="https://github.com/lmenezes/cerebro">Cerebro's web site</a> : "<i>Cerebro is an open source(MIT License) ElasticSearch web admin tool built using Scala, Play Framework, AngularJS and Bootstrap..</i>"
</p>
<p>
Cerebro is the one-stop-shop, little and simple but efficient, monitoring and administration tool for ElasticSearch.
</p>
<p>
Cerebro is a must have with ElasticSearch since working only with the REST API to understand ElasticSearch's topology and perform most trivial administration tasks (such as defining mapping templates, etc.) is a real hassle.
<br>
Cerebro is far from perfect but really does the job.
</p>
</td>
</tr>
</table>
<a name="sec225"></a>
<h4>2.2.5 Spark</h4>
<table style="border: 0px none;">
<tr>
<td style="width: 100px; border: 0px none;">
<img style="width: 100px; min-width: 100px; border: 0px none;" alt="Spark Logo" src="https://www.niceideas.ch/es_spark/images/spark_logo.png" />
</td>
<td style="width: 100%; border: 0px none;">
<p>
From <a href="https://spark.apache.org/">Spark's web site</a> : "<i>Apache Spark is a fast and general engine for large-scale data processing.</i>"
</p>
<p>
From <a href="https://en.wikipedia.org/wiki/Apache_Spark">Wikipedia's Spark article</a>: "<i>Apache Spark provides programmers with an application programming
interface centered on a data structure called the resilient distributed dataset (RDD), a read-only multiset of data items distributed over a cluster of machines,
that is maintained in a fault-tolerant way.</i>
<br>
<i>It was developed in response to limitations in the MapReduce cluster computing paradigm, which forces a particular linear dataflow structure on distributed programs: MapReduce programs read input data from disk, map a function across the data, reduce the results of the map, and store reduction results on disk. Spark's RDDs function as a working set for distributed programs that offers a (deliberately) restricted form of distributed shared memory.</i>
</p>
<p>
<i>The availability of RDDs facilitates the implementation of both iterative algorithms, that visit their dataset multiple times in a loop, and interactive/exploratory data analysis, i.e., the repeated database-style querying of data. The latency of such applications (compared to a MapReduce implementation, as was common in Apache Hadoop stacks) may be reduced by several orders of magnitude.</i>"
</p>
</td>
</tr>
</table>
<a name="sec226"></a>
<h4>2.2.6 Mesos</h4>
<table style="border: 0px none;">
<tr>
<td style="width: 100px; border: 0px none;">
<img style="width: 100px; min-width: 100px; border: 0px none;" alt="Mesos Logo" src="https://www.niceideas.ch/es_spark/images/mesos_logo.png" />
</td>
<td style="width: 100%; border: 0px none;">
<p>
From <a href="http://mesos.apache.org/">Mesos' web site</a> : "<i>Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.</i>
<br>
<i>Mesos is a distributed systems kernel, built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos
kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elasticsearch) with API’s for
resource management and scheduling across entire Datacenter and cloud environments.</i>"
</p>
<p>
<b>In the context of ELK-MS</b>, and in a general way when considering running Spark in a production environment, Mesos is the way to go if one doesn't want to deploy a full Hadoop stack to support Spark. In the end, it appears that Mesos performs amazingly, both by suffering only form a very small memory footprint on the cluster and by being incredibly easy to setup and administer.
</p>
</td>
</tr>
</table>
<a name="sec227"></a>
<h4>2.2.7 Spark on Mesos specificities</h4>
<p>
Happily Spark and Mesos, both products from the Apache fundation, know about each other and are designed to work with each other.
<br>
There are some specificities though when it comes to run Spark on Mesos as opposed to running Spark on the more usual YARN, as explained below,
</p>
<b>Spark Mesos Dispatcher</b>
<table style="border: 0px none;">
<tr>
<td style="width: 100px; border: 0px none;">
<img style="width: 100px; min-width: 100px; border: 0px none;" alt="Spark Mesos Dispatcher Logo" src="https://www.niceideas.ch/es_spark/images/spark-mesos-dispatcher_logo.png" />
</td>
<td style="width: 100%; border: 0px none;">
<p>
Interestingly, mesos handles spark workers in a pretty amazing way. Not only does Mesos consider <i>node locality</i> requirements between spark and ElasticSearch, but Mesos also provides required retry policies and else.
</p>
<p>
When launching a spark job, there is nevertheless one <i>Single Point of Failure</i> that remains: the spark driver that lives outside of the Mesos/Spark cluster, on the machine it is launched by the user or the driving process.
</p>
<p>
For this reason, spark provides the <b>Spark Mesos Dispatcher</b> that can be used to <b>dispatch the Spark Driver itself on the Mesos/Spark cluster</b>.
<br>
Using the <i>Spark Mesos Dispatcher</i>, the driver itself, just as the spark processing is balanced on the cluster to an available node and can be supervised (retried, monitored in terms of memory consumption, etc.).
</p>
<p>
The <i>Spark Mesos Dispatcher</i> addresses the single weakness of a spark process: the driver that can crash or exhaust resources and handles it just as any other bit of spark processing.
</p>
</td>
</tr>
</table>
<p>
<b>Spark History Server</b>
</p>
<p>
In contrary to spark running in standalone mode, when spark runs on Mesos, it has no long life running backend that the user can use to interact with when Spark is not actually executing a job.
<br>
Mesos takes care of creating and dispatching Spark workers when required. When no Spark job is being executed, there is no spark process somewhere one can interact with to query, for instance, the results of a previous job.
</p>
<p>
Happily spark provides a solution out of the box for this : the <b>spark History Server</b>.
<br>
The Spark History Server is a lightweight process that presents the results stored in the Spark <i>Event Log folder</i>, that folder on the filesystem where Spark stores consolidated results from the various workers.
<br>
The documentation of Spark is very unclear about this, but since only the spark driver stores consolidated results in the <i>event log folder</i>, if all drivers are launched on the same machine (for instance the Mesos master machine), there is no specific needs for HDFS.
</p>
<p>
One should note that running the Spark History Server without HDFS to store the event log can be a problem if one uses the <i>Spark Mesos Dispatcher</i> to distribute the driver program itself on the Mesos cluster. In this case using a common NFS share for instance would solve the problem.
</p>
<a name="sec23"></a>
<h3>2.3. Making it work together : ES-Hadoop</h3>
<table style="border: 0px none;">
<tr>
<td style="width: 100px; border: 0px none;">
<img style="width: 100px; min-width: 100px; border: 0px none;" alt="ES-Hadoop Logo" src="https://www.niceideas.ch/es_spark/images/es-hadoop_logo.png" />
</td>
<td style="width: 100%; border: 0px none;">
<p>
From <a href="https://www.elastic.co/products/hadoop">ES-Hadoop's web site</a> : "<i>Connect the massive data storage and deep processing power of Hadoop with the real-time search and analytics of Elasticsearch. The Elasticsearch-Hadoop (ES-Hadoop) connector lets you get quick insight from your big data and makes working in the Hadoop ecosystem even better.</i>"
</p>
<p>
Initially ES-Hadoop contains the set of classes implementing the connectors for pretty much all <i>de facto "standards"</i> components of a full hadoop stack, such as Hive, Pig, Spark, etc.
<br>
Interestingly, as far as Spark is concerned, Spark can perfectly use ES-Hadoop to load from or store data to ElasticSearch <b>outside of an Hadoop stack</b>. In fact, the spark connector from the ES-Hadoop library has no dependency on a Hadoop stack whatsoever.
</p>
<p>
<b>In the context of ELK-MS, the ES-Hadoop connector is one of the most important components.</b> When one considers building a large collocated ES / Mesos / Spark cluster and execute tasks requiring to fetch large datasets from ES to Spark, the <b>data-locality knowledge</b> supported by ES-Hadoop is utmost important. The <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1">second article of this serie</a> is largely devoted to assessing how and how far the optimization of data-locality works.
</p>
<p>
When launching a job using Spark, the connector determines the locations of the shards in ElasticSearch that it will be targeting and creates a partition per shard (or even further to allow for greater parallelism). Each of these partition definitions carries with it the index name, the shard id, the slice id and the addresses of the machines where it can find this data on locally. It then relies on Spark's task scheduling to achieve data locality.
<br>
Spark will stand up a task for each of the input partitions, and each reading task is pinned to a node that is hosting the shards. This just means that the task will always try to read from that node first, but will target other nodes if that node fails the processing or fails from becoming available before the timeout.
</p>
</td>
</tr>
</table>
<a name="sec24"></a>
<h3>2.4 Application Architecture</h3>
<p>
Typical Data Flows on the ELK-MS platform is illustrated by the following <b>Application Architecture</b> schema:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/application_architecture.png">
<img class="centered" style="width: 600px;" alt="ELK-MS Application Architecture" src="https://www.niceideas.ch/es_spark/images/application_architecture.png" />
</a>
</div>
<br>
<p>
The tests presented <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1">"in the second article in this serie: ELK-MS - part II : assessing behaviour"</a> are intended to assess the well behaviour of this application architecture.
</p>
<a name="sec3"></a>
<h2>3. niceideas ELK-MS</h2>
<p>
So. Again as stated in introduction before, when playing with ES / Mesos / Spark, it happened quite fast that I got two urgent needs:
</p>
<ol>
<li>
First, I needed a <b>reference for configuring the various software</b> so that they work well together. Instead of writing pages of documentation indicating the settings to tune, I ended up putting all of that in setup scripts aimed at helping me re-apply the configuration at will.
</li>
<li>
Second, I needed a <b>test cluster</b> allowing me to assess how various key features were working, among which ensuring optimization of data-locality was one of the most important.
</li>
</ol>
<p>
In the end I wrote a set of scripts using Vagrant and VirtualBox aimed at making it possible for me to rebuild the test cluster and reapply the configuration at will. I packaged all these scripts together and call this package the <b>niceideas_ELK-MS</b> package.
</p>
<p>
This package is <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS.tar.gz">available for download here</a>.
</p>
<a name="sec31"></a>
<h3>3.1 System and principles</h3>
<p>
The <b>System Architecture</b> of the ELK-MS platform as build by the <i>niceideas_ELK-MS</i> package is as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/system_archi.png">
<img class="centered" style="width: 400px;" alt="ELK-MS System Architecture" src="https://www.niceideas.ch/es_spark/images/system_archi.png" />
</a>
</div>
<br>
<ul>
<li>
The master node, <b>MES Master</b> (for Mesos/Elasticsearch/Spark) is called <code>mes_master</code>. It contains the full stack of software including the management UIs. The Master node is also a data node.
</li>
<li>
The two data nodes, <b>MES Slave X</b> are called <code>mes_node1</code> and <code>mes_node2</code>. They only provide an ElasticSearch instance and a Mesos worker instance to drive Spark Executors.
</li>
</ul>
<p>
Having two possible <i>Mesos Masters</i> is not considered for now but the technical stack is deployed with this possibility wide open by using zookeeper to manage mesos masters.
</p>
<a name="sec32"></a>
<h3>3.2 The build system</h3>
<p>
The remainder of this section is a description of the <i>niceideas_ELK-MS</i> package build system and a presentation of the expected results.
</p>
<a name="sec321"></a>
<h4>3.2.1 Required Tools</h4>
<p>
First the build system is really intended to work on Linux, but would work as well on Windows except that vagrant commands need to be called directly.
</p>
<p>
But before digging into this, the following tools need to be installed and properly working on the host machine where the ELK-MS test cluster has to be built:
</p>
<ul>
<li>
<b>VirtualBox</b>: is an x86 and AMD64/Intel64 virtualization solution.
<br>
The <i>niceideas_ELK-MS</i> package will build a cluster of nodes taking the form of VMs running on the host machine (the user computer).
</li>
<li>
<b>Vagrant</b>: is a tool for building and managing virtual machine environments in a single workflow.
<br>
The <i>niceideas_ELK-MS</i> package uses Vagrant to build and manage the VMs without any user interaction required and to drive the provisioning scripts execution.
</li>
<li>
<b>vagrant-reload</b> vagrant plugin is require to reload the machines after some changes applied by the provisionning scripts requiring a VM reboot.
<br>
See <a href="https://github.com/aidanns/vagrant-reload/blob/master/README.md">https://github.com/aidanns/vagrant-reload/blob/master/README.md</a>.
</li>
</ul>
<a name="sec322"></a>
<h4>3.2.2 Build System Project Layout</h4>
<p>
The <i>niceideas_ELK-MS</i> package structure, after being properly extracted in a local folder, is as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/provisionning.png">
<img class="centered" style="width: 300px;" alt="ELK-MS Provisionning" src="https://www.niceideas.ch/es_spark/images/provisionning.png" />
</a>
</div>
<br>
<ul>
<li>
<code>./setup.sh</code>: basically takes care of everything by calling vagrant to build the 3 VMs
</li>
<li>
<code>./vagrant/VagrantFile</code>: vagrant definition file to define the 3 VMs and the provisioning scripts
</li>
<li>
<code>./provisionning/*</code>: the provisoning scripts. The entry point is <code>setup.sh</code> that calls each and every other script.
</li>
</ul>
<p>
<b>Rationality</b>
</p>
<p>
In a DevOps world, there are better tools than shell scripts to proceed with VM or machine provisioning, such as Ansible, Chef, Puppet, etc.
<br>
But in my case, I want it to be possible for me to go on any VM, any machine and re-apply my configuration to Spark, Mesos, ElasticSearch or else by simply calling a shell script with a few arguments.
<br>
So even though there are more efficient alternatives, I kept shell scripts here for the sake of simplicity.
</p>
<p>
<b>Building the ELK-MS test cluster on Windows</b>
</p>
<p>
With VirtualBox and Vagrant properly installed on Windows, nothing should prevent someone from building the cluster on Windows.
<br>
But in this case, of course, the root scripts <code>setup.sh</code>, <code>start_cluster.sh</code>, <code>stop_cluster.sh</code> are not usable (or else cygwin ? MingW ?).
</p>
<p>
In this case, the user should call vagrant manually to build the 3 VMs <code>mes_master</code>, <code>mes_node1</code> and <code>mes_node2</code> as follows:
</p>
<pre>
c:\niceideas_ELK-MS\vagrant> vagrant up mes_master
...
c:\niceideas_ELK-MS\vagrant> vagrant up mes_node1
...
c:\niceideas_ELK-MS\vagrant> vagrant up mes_node2
...
</pre>
<a name="sec33"></a>
<h3>3.3 Calling the build system and results</h3>
<p>
Again, calling the build system to fully build the cluster, on Linux, is as simple as:
</p>
<pre>
badtrash@badbook:/data/niceideas_ELK-MS/setup$ ./setup.sh
</pre>
<p>
A full dump of the result of the <code>setup.sh</code> script is <a href="http://www.niceideas.ch/es_spark/setuplog.html">available here</a>.
</p>
<a name="sec34"></a>
<h3>3.4 Testing the System</h3>
<p>
After calling the <code>setup.sh</code> script above, the 3 VMs are properly created, as one can check in VirtualBox:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/virtualbox_manager.png">
<img class="centered" style="width: 500px;" alt="The three VMs in VirtualBox Manager" src="https://www.niceideas.ch/es_spark/images/virtualbox_manager.png" />
</a>
</div>
<br>
<p>
In addition, the 4 UI applications should be available at following addresses (caution, the links below return to your cluster, not niceideas.ch):
</p>
<ul>
<li>
<a href="http://192.168.10.10:9000/#/overview?host=http:%2F%2Flocalhost:9200">Cerebro on http://192.168.10.10:9000/</a>
</li>
<li>
<a href="http://192.168.10.10:5050/">Mesos Console on http://192.168.10.10:5050/</a>
</li>
<li>
<a href="http://192.168.10.10:18080/">Spark History Server on http://192.168.10.10:18080/</a>
</li>
<li>
<a href="http://192.168.10.10:5601/">Kibana on http://192.168.10.10:5601/</a>
</li>
</ul>
<p>
<b>Cerebro:</b> (<a href="http://192.168.10.10:9000/#/overview?host=http:%2F%2Flocalhost:9200">http://192.168.10.10:9000/</a>)
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/cerebro_empty.png">
<img class="centered" style="width: 750px; " alt="Cerebro - ES management" src="https://www.niceideas.ch/es_spark/images/cerebro_empty.png" />
</a>
</div>
<br>
<p>
(One can see the 3 nodes available)
</p>
<p>
<b>Mesos:</b> (<a href="http://192.168.10.10:5050/">http://192.168.10.10:5050/</a>)
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/mesos_agents.png">
<img class="centered" style="width: 750px; " alt="Mesos - the 3 agents" src="https://www.niceideas.ch/es_spark/images/mesos_agents.png" />
</a>
</div>
<br>
<p>
(One can see the 3 nodes available)
</p>
<p>
<b>Spark History Server:</b> (<a href="http://192.168.10.10:18080/">http://192.168.10.10:18080/</a>)
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/history_server.png">
<img class="centered" style="width: 750px; " alt="Spark History Server" src="https://www.niceideas.ch/es_spark/images/history_server.png" />
</a>
</div>
<br>
<p>
<b>Kibana:</b> (<a href="http://192.168.10.10:5601/">http://192.168.10.10:5601/</a>)
</p>
<div class="centering">
<a href="https://www.niceideas.ch/es_spark/images/kibana_empty.png">
<img class="centered" style="width: 750px; " alt="Spark History Server" src="https://www.niceideas.ch/es_spark/images/kibana_empty.png" />
</a>
</div>
<br>
<a name="sec36"></a>
<h3>3.5 Tips & Tricks</h3>
<p>
<b>This closes the presentation of the <i>niceideas_ELK-MS</i> package. The remainder of this article gives some hints regarding the configuration of the different software components</b>.
<br>
Readers interested in understanding what the build system of <i>niceideas_ELK-MS</i> presented above does without the hassle of analyzing the setup scripts can continue reading hereunder.
<br>
Reader interested only in understanding the cluster layout and the concerns of the ES / Spark integration can move to <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1">the second article in this serie: ELK-MS - part II : assessing behaviour</a>.
</p>
<p>
<b>Killing a stuck job</b>
</p>
<p>
Once in a while, for various reasons, a job gets stuck. In this case the easiest way to kill it is using the Spark Web console.
<br>
But wait, hold on, you just said above that such a console is not available when running through Mesos ?
<br>
Well actually, the Spark console is available as long as the spark job is alive ... which is the case, happily, when a spark job is stuck.
</p>
<p>
So one can follow the link provided by Mesos on the Spark console and use the usual <i>kill</i> link from there.
</p>
<p>
<b>Spark fine grained scheduling by Mesos</b>
</p>
<p>
When reading about Mesos <i>fine grained</i> scheduling of spark job, one might think it makes sense to give it a try ... don't!
</p>
<p>
Spark <i>fine grained</i> scheduling by Mesos is really really messed up.
<br>
One might believe that it helps concurrency and better resource allocation but it really doesn't, In practice what happens is that an amazing proportion of time is lost scheduling all the individual spark tasks, plus it often compromises co-location of data between ES and Spark.
</p>
<p>
It's even deprecated in the latest spark versions.
<br>
More information in this regards is available here: <a href="https://issues.apache.org/jira/browse/SPARK-11857">https://issues.apache.org/jira/browse/SPARK-11857</a>.
</p>
<a name="sec4"></a>
<h2>4. Noteworthy configuration elements</h2>
<p>
The below presents the important configuration aspects taken care of by the provisioning scripts.
</p>
<a name="sec41"></a>
<h3>4.1 NTP</h3>
<p>
Related scripts from the <i>niceideas_ELK-MS</i> package are as follows:
</p>
<ul>
<li>Configuration script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/setupNTP.sh"><code></code>setupNTP.sh</a></li>
</ul>
<p>
Just as with every big data or NoSQL cluster, having a shared common understanding of time in the cluster is key. So NTP needs to be properly set up.
</p>
<p>
<b>On master <code>mes_master</code></b>
</p>
<p>
Sample portion from <code>/etc/ntp.conf</code>:
</p>
<pre>
pool ntp1.hetzner.de iburst
pool ntp2.hetzner.com iburst
pool ntp3.hetzner.net iburst
</pre>
<p>
<b>On slaves <code>mes_node1</code> and <code>mes_node2</code></b>
</p>
<p>
Sample portion from <code>/etc/ntp.conf</code>:
</p>
<pre>
server 192.168.10.10
#enabling mes_master to set time
restrict 192.168.10.10 mask 255.255.255.255 nomodify notrap nopeer noquery
#disable maximum offset of 1000 seconds
tinker panic 0
</pre>
<a name="sec42"></a>
<h3>4.2 Zookeeper</h3>
<p>
Related scripts from the <i>niceideas_ELK-MS</i> package are as follows:
</p>
<ul>
<li>Configuration script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/setupZookeeper.sh"><code>setupZookeeper.sh</code></a></li>
</ul>
<p>
Zookeeper is really only required when considering several Mesos master since in this case we need the <i>quorum</i> feature of zookeeper to proceed with proper election of the master and to track their state.
<br>
At the moment ELK-MS has only one Mesos master, but we'll make it production and HA ready by already setting up and using zookeeper.
</p>
<p>
<b>On master <code>mes_master</code></b>
</p>
<p>
Sample portion from <code>/etc/zookeeper/conf/zoo.cfg</code>:
</p>
<pre>
server.1=192.168.10.10:2888:3888
</pre>
<p>
In addition, we need to set zookeeper master id in <code>/etc/zookeeper/conf/myid</code>.
<br>
Let's just put a single character "1" in it for now.
</p>
<a name="sec43"></a>
<h3>4.3 Elasticsearch</h3>
<p>
Related scripts from the <i>niceideas_ELK-MS</i> package are as follows:
</p>
<ul>
<li>Installation script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/installElasticSearch.sh"><code>installElasticSearch.sh</code></a></li>
<li>Configuration script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/setupElasticSearch.sh"><code>setupElasticSearch.sh</code></a></li>
<li>Systemd service file: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/utils/elasticsearch.service"><code>elasticsearch.service</code></a></li>
</ul>
<p>
There's not a whole lot of things to configure in ES. The installation and setup scripts are really just created a dedicated users, a whole bunch of folders and simlinks, etc.
<br>
The only important configuration elements are as follows:
</p>
<p>
<b>On master <code>mes_master</code></b>
</p>
<p>
Sample portion from <code>/usr/local/lib/elasticsearch-6.0.0/config/elasticsearch.yml </code>:
</p>
<pre>
# name of the cluster (has to be common)
cluster.name: mes-es-cluster
# name of the node (has to be unique)
node.name: mes_master
# Bind on all interfaces (internal and external)
network.host: 0.0.0.0
# We're good with one node
discovery.zen.minimum_master_nodes: 1
#If you set a network.host that results in multiple bind addresses
#yet rely on a specific address for node-to-node communication, you
#should explicitly set network.publish_host
network.publish_host: 192.168.10.10
</pre>
<p>
<b>On slaves <code>mes_node1</code> and <code>mes_node2</code></b>
</p>
<p>
Sample portion from <code>/usr/local/lib/elasticsearch-6.0.0/config/elasticsearch.yml </code>:
</p>
<pre>
# name of the cluster (has to be common)
cluster.name: mes-es-cluster
# name of the node (has to be unique, this is for node1)
node.name: mes_node1
# Bind on all interfaces (internal and external)
network.host: 0.0.0.0
# We're good with one node
discovery.zen.minimum_master_nodes: 1
# enabling discovery of master
discovery.zen.ping.unicast.hosts: ["192.168.10.10"]
#If you set a network.host that results in multiple bind addresses
#yet rely on a specific address for node-to-node communication, you
#should explicitly set network.publish_host
# (this is for node1)
network.publish_host: 192.168.10.11
</pre>
<a name="sec44"></a>
<h3>4.4 Logstash, Kibana, Cerebro</h3>
<p>
Related scripts from the <i>niceideas_ELK-MS</i> package are as follows:
</p>
<ul>
<li>Logstash Installation script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/installLogstash.sh"><code>installLogstash.sh</code></a></li>
<li>Logstash Configuration script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/setupLogstash.sh"><code>setupLogstash.sh</code></a></li>
</ul>
<ul>
<li>Cerebro Installation script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/installCerebro.sh"><code>installCerebro.sh</code></a></li>
<li>Cerebro Configuration script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/setupCerebro.sh"><code>setupCerebro.sh</code></a></li>
<li>Cerebro Systemd service file: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/utils/cerebro.service"><code>cerebro.service</code></a></li>
</ul>
<ul>
<li>Kibana Installation script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/installKibana.sh"><code>installKibana.sh</code></a></li>
<li>Kibana Configuration script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/setupKibana.sh"><code>setupKibana.sh</code></a></li>
<li>Kibana Systemd service file: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/utils/kibana.service"><code>kibana.service</code></a></li>
</ul>
<p>
There is really nothing specific to report in terms of configuration for these 3 tools.
</p>
<a name="sec45"></a>
<h3>4.5 Mesos</h3>
<p>
Related scripts from the <i>niceideas_ELK-MS</i> package are as follows:
</p>
<ul>
<li>Installation script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/installMesos.sh"><code>installMesos.sh</code></a></li>
<li>Configuration script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/setupMesos.sh"><code>setupMesos.sh</code></a></li>
<li>Mesos startup script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/utils/mesos-init-wrapper.sh"><code>mesos-init-wrapper.sh</code></a></li>
<li>Mesos Master Systemd startup file: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/utils/mesos-master.service"><code>mesos-master.service</code></a></li>
<li>Mesos Slave Systemd startup file: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/utils/mesos-slave.service"><code>mesos-slave.service</code></a></li>
</ul>
<p>
The noteworthy configuration aspects are as follows.
</p>
<p>
<b>On both master and slaves</b>
</p>
<p>
The file <code>/usr/local/etc/mesos/mesos-env.sh</code> contains common configuration for both mesos-master and mesos-slave.
<br>
So we should create this file on every node of the cluster.
</p>
<pre>
#Working configuration
export MESOS_log_dir=/var/log/mesos
#Specify a human readable name for the cluster
export MESOS_cluster=mes_cluster
#Avoid issues with systems that have multiple ethernet interfaces when the Master
#or Slave registers with a loopback or otherwise undesirable interface.
# (This is for master, put IP of the node)
export MESOS_ip=192.168.10.10
#By default, the Master will use the system hostname which can result in issues
#in the event the system name isn't resolvable via your DNS server.
# (This is for master, put IP of the node)
export MESOS_hostname=192.168.10.10
</pre>
<p>
Then, the file <code>/usr/local/etc/mesos/mesos-slave-env.sh</code> configures mesos-slave.
<br>
Since we run a mesos-slave process on the mes_master machine as well, we define this file on every node of the cluster as well.
</p>
<pre>
#Path of the slave work directory.
#This is where executor sandboxes will be placed, as well as the agent's checkpointed state.
export MESOS_work_dir=/var/lib/mesos/slave
#we need the Slave to discover the Master.
#This is accomplished by updating the master argument to the master Zookeeper URL
export MESOS_master=zk://$MASTER_IP:2181/mesos
</pre>
<p>
<b>On master <code>mes_master</code> only:</b>
</p>
<p>
The mesos-master process is configured by <code>/usr/local/etc/mesos/mesos-master-env.sh</code>:
</p>
<pre>
#Path of the master work directory.
#This is where the persistent information of the cluster will be stored
export MESOS_work_dir=/var/lib/mesos/master
#Specify the master Zookeeper URL which the Mesos Master will register with
export MESOS_zk=zk://$192.168.10.10:2181/mesos
# Change quorum for a greater value if one has more than one master
# (only 1 in our case)
export MESOS_quorum=1
</pre>
<a name="sec46"></a>
<h3>4.6 Spark</h3>
<p>
Related scripts from the <i>niceideas_ELK-MS</i> package are as follows:
</p>
<ul>
<li>Installation script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/installSpark.sh"><code>installSpark.sh</code></a></li>
<li>Configuration script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/setupSpark.sh"><code>setupSpark.sh</code></a></li>
<li>Dynamic Allocation Configuration script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/setupSparkDynamicAllocation.sh"><code>setupSparkDynamicAllocation.sh</code></a></li>
<li>Spark History Server start wrapper: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/utils/start-spark-history-server-wrapper.sh"><code>start-spark-history-server-wrapper.sh</code></a></li>
<li>Spark History Server Systemd startup file: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/utils/spark-history-server.service"><code>spark-history-server.service</code></a></li>
<li>Spark Mesos Dispatcher start wrapper: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/utils/start-spark-mesos-dispatcher-wrapper.sh"><code>start-spark-mesos-dispatcher-wrapper.sh</code></a></li>
<li>Spark Mesos Dispatcher Systemd startup file: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/utils/spark-mesos-dispatcher.service"><code>spark-mesos-dispatcher.service</code></a></li>
</ul>
<p>
Aside from the specific startup wrappers and systemd service configuration files required for the <i>Spark History Server</i> and the <i>Spark Mesos Dispatcher</i>, the noteworthy configuration elements are as follows.
</p>
<p>
<b>On both master and slaves</b>
</p>
<p>
The file <code>/usr/local/lib/spark-2.2.0/conf/spark-env.sh</code> defines common environment variables required by spark workers and drivers.
<br>
So we should create this file on every node of the cluster (in addition the master also executes spark workers).
</p>
<pre>
#point to your libmesos.so if you use Mesos
export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/mesos-1.3.0/lib/libmesos.so
#Important configuration directories
export SPARK_CONF_DIR=/usr/local/lib/spark-2.2.0/conf
export SPARK_LOG_DIR=/usr/local/lib/spark-2.2.0/logs
</pre>
<p>
The file <code>/usr/local/lib/spark-2.2.0/conf/spark-defaults.conf</code> defines common configuration properties required by spark workers and drivers.
<br>
So we should create this file on every node of the cluster (since the master as well executes spark workers).
</p>
<pre>
#Finding the mesos master through zookeeper
spark.master=mesos://zk://$MASTER_IP:2181/mesos
#Activating EventLog stuff (required by history server)
spark.eventLog.enabled=true
spark.eventLog.dir=/var/lib/spark/eventlog
#Default serializer
spark.serializer=org.apache.spark.serializer.KryoSerializer
#Limiting the driver (client) memory
spark.driver.memory=800m
#Settings required for Spark driver distribution over mesos cluster
#(Cluster Mode through Mesos Dispatcher)
spark.mesos.executor.home=/usr/local/lib/spark-2.2.0/
#If set to true, runs over Mesos clusters in coarse-grained sharing mode,
#where Spark acquires one long-lived Mesos task on each machine.
#If set to false, runs over Mesos cluster in fine-grained sharing mode,
#where one Mesos task is created per Spark task.
#(Fine grained mode is deprecated and one should consider dynamic allocation
#instead)
spark.mesos.coarse=true
#ElasticSearch setting (first node to be reached => can use localhost everywhere)
spark.es.nodes=localhost
spark.es.port=9200
es.nodes.data.only=false
#The scheduling mode between jobs submitted to the same SparkContext.
#Can be FIFO or FAIR. FAIR Seem not to work well with mesos
#(FIFO is the default BTW ...)
spark.scheduler.mode=FIFO
#How long to wait to launch a data-local task before giving up
#and launching it on a less-local node.
spark.locality.wait=20s
# Configuring dynamic allocation
# (See Spark configuration page online for more information)
spark.dynamicAllocation.enabled=true
#(Caution here : small values cause issues. I have executors killed with 10s for instance)
spark.dynamicAllocation.executorIdleTimeout=120s
spark.dynamicAllocation.cachedExecutorIdleTimeout=300s
# Configuring spark shuffle service (required for dynamic allocation)
spark.shuffle.service.enabled=true
</pre>
<p>
<b>On master <code>mes_master</code> only</b>
</p>
<p>
In the very same file <code>/usr/local/lib/spark-2.2.0/conf/spark-defaults.conf</code>, we add what is required for Spark History Server:
</p>
<pre>
#For the filesystem history provider,
#the directory containing application event logs to load.
spark.history.fs.logDirectory=file:///var/lib/spark/eventlog
#The period at which to check for new or updated logs in the log directory.
spark.history.fs.update.interval=5s
</pre>
<a name="sec47"></a>
<h3>4.7 ES-Hadoop</h3>
<p>
Related scripts from the <i>niceideas_ELK-MS</i> package are as follows:
</p>
<ul>
<li>Installation script: <a href="https://www.niceideas.ch/es_spark/sandbox/setup/provisionning/installESHadoop.sh"><code>installESHadoop.sh</code></a></li>
</ul>
<p>
Nothing specific to report, basically the only thing to be done to install ES-Hadoop is to copy the spark driver <code>elasticsearch-spark-20_2.11-6.0.0.jar</code> to the spark jars folder <code>/usr/local/lib/spark-2.2.0/jars/</code>.
</p>
<a name="sec5"></a>
<h2>5. Conclusion</h2>
<p>
With all of the information above, you should be able to set up your own ElasticSearch / Mesos / Spark Cluster in no time.
<br>
Or simply use the <a href="https://www.niceideas.ch/es_spark/sandbox/niceideas_ELK-MS.tar.gz">niceideas_ELK-MS</a> package to build a test cluster using one single command.
</p>
<p>
Now the next article in this serie, <a href="https://www.niceideas.ch/roller2/badtrash/entry/elk-ms-elasticsearch-logstash-kibana1">ELK-MS - part II : assessing behaviour</a> will present the tests I did on this test cluster and the conclusions in terms of behaviour assessment.
</p>
<p>
I'm already telling you the big conclusion: Using ElasticSearch / Mesos / Spark for your Big Data Analytics needs is mind-joggling. It works really amazingly and supports a striking range of use cases while being a hundred times lighter than a plain old Hadoop stack both to setup and to operate.
</p>
<p>
Kuddos to the folks at Apache and at Elastic for making this possible.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/periodic-table-of-agile-principles
Periodic Table of Agile Principles and Practices
Jerome Kehrli
2017-06-29T17:19:29-04:00
2017-09-14T03:37:38-04:00
<p>
After writing <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-planning-tools-and-processes">my previous article</a>, I wondered how I could represent on a single schematic all the <i>Agile Principles and Practices</i> from the methods I am following, XP, Scrum, Lean Startup, DevOps and others.
<br>
I found that the approach I used in <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-planning-tools-and-processes#sec5"> in a former schematic</a> - a graph of relationship between practices - is not optimal. It already looks ugly with only a few practices and using the same approach for the whole set of them would make it nothing but a mess.
</p>
<p>
So I had to come up with something else, something better.
<br>
Recently I fell by chance on the <a href="https://en.wikipedia.org/wiki/Periodic_table">Periodic Table of the Elements</a>... Long time no see... Remembering my physics lessons in University, I always loved that table. I remembered spending hours understanding the layout and admiring the beauty of its natural simplicity.
<br>
So I had the idea of trying the same layout, not the same approach since both are not comparable, really only the same layout for <i>Agile Principles and Practices</i>.
</p>
<p>
The result is hereunder: <b>The Periodic Table of Agile Principles and Practices</b>:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/78e8967b-a55b-4639-9ae1-bad60b0befda">
<img class="centered" style="width: 750px; " alt="Periodic Table of Agile Principles and Practices" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/e1e282ca-a48e-456d-b876-4734563573a6" />
</a>
</div>
<br>
<p>
The layout principle is and the description of the principles and practices is explained hereafter.
</p>
<!-- Periodic Table of Agile Principles and Practices -->
<p>
After writing <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-planning-tools-and-processes">my previous article</a>, I wondered how I could represent on a single schematic all the <i>Agile Principles and Practices</i> from the methods I am following, XP, Scrum, Lean Startup, DevOps and others.
<br>
I found that the approach I used in <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-planning-tools-and-processes#sec5"> in a former schematic</a> - a graph of relationship between practices - is not optimal. It already looks ugly with only a few practices and using the same approach for the whole set of them would make it nothing but a mess.
</p>
<p>
So I had to come up with something else, something better.
<br>
Recently I fell by chance on the <a href="https://en.wikipedia.org/wiki/Periodic_table">Periodic Table of the Elements</a>... Long time no see... Remembering my physics lessons in University, I always loved that table. I remembered spending hours understanding the layout and admiring the beauty of its natural simplicity.
<br>
So I had the idea of trying the same layout, not the same approach since both are not comparable, really only the same layout for <i>Agile Principles and Practices</i>.
</p>
<p>
The result is hereunder: <b>The Periodic Table of Agile Principles and Practices</b>:
</p>
<p>
(This article is available as a PDF document here <a href="https://www.niceideas.ch/Agile_table.pdf">https://www.niceideas.ch/Agile_table.pdf</a> and a slideshare presentation there <a href="https://www.slideshare.net/JrmeKehrli/periodic-table-of-agile-principles-and-practices">https://www.slideshare.net/JrmeKehrli/periodic-table-of-agile-principles-and-practices</a>)
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/78e8967b-a55b-4639-9ae1-bad60b0befda">
<img class="centered" style="width: 750px; " alt="Periodic Table of Agile Principles and Practices" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/e1e282ca-a48e-456d-b876-4734563573a6" />
</a>
</div>
<br>
<p>
The layout principle is and the description of the principles and practices is explained hereafter.
</p>
<h2>Layout Principle</h2>
<ul>
<li>
The <b>Origin Method</b> such as XP, Scrum, DevOps, etc is indicated by the color as well as the name of the method on the top-right corner.
</li>
<li>
The <b>Category</b>, <i>Principle</i> or <i>Practice</i> is indicated by the shape: rectangle or round corners.
</li>
<li>
The number represents the <b>Complexity</b> expressed as the number of dependencies.
</li>
<li>
The <b>team or committee</b> concerned with the principle or practice is indicated as note on the bottom-right corner.
</li>
<li>
The <b>horizontal dimension</b> is related to the complexity. The more on the right is an element, the higher its complexity.
</li>
<li>
The <b>vertical dimension</b> is related to classifying principles and practices more organization or more related to engineering, in specific layers related to the category or team they apply to.
</li>
</ul>
<p>
This is best presented as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/7a36d6d6-446b-4834-b087-57dbfe1c91a6">
<img class="centered" style="width: 900px;" alt="Periodic Table of Agile Principles and Practices - Explanations" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/230510f9-15fd-46ef-9625-f504ecd249f7" />
</a>
</div>
<br>
<h2>Remarks</h2>
<ul>
<li>
Interestingly, but not surprisingly, scrum is really in the middle of the schematic, underlying the fact that it impacts as well development principles and the development team organization.
</li>
<li>
XP is really everywhere down the line.
</li>
<li>
Product Development is really about Product Management in the Agile world.
</li>
<li>
DevOps is more related to development practices than everything else.
</li>
</ul>
<p>
The next part of this article describes each and every principle and practice.
</p>
<h2>XP</h2>
<style>
table.pte {
border: 0px none;
border-collapse: collapse;
}
div.entryContent table.pte tr td,
table.pte tr td {
padding-top: 10px;
border: 0px none;
border-bottom: 1px solid #888888;
}
table.pte tr td img {
display: inline;
}
td.desc_right {
padding-left: 10px;
}
.pte-image-big {
width: 90px;
min-width: 90px;
max-width: 90px;
}
.pte-image-small {
width: 30px;
min-width: 30px;
max-width: 30px;
}
</style>
<a name="Sn"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Sn.png"
alt="Sn : Simple Design"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Sn : Simple Design
</b><br>
A simple design always takes less time to finish than a complex one. So always do the simplest thing that could possibly work next. If you find something that is complex replace it with something simple. It's always faster and cheaper to replace complex code now, before you waste a lot more time on it.
</p>
<p>
Depends on
<a href="#Td"><img src="https://www.niceideas.ch/agile_table/Td.png" alt="Td" class="pte-image-small" /></a>,
<a href="#Rf"><img src="https://www.niceideas.ch/agile_table/Rf.png" alt="Rf" class="pte-image-small" /></a>,
<a href="#Su"><img src="https://www.niceideas.ch/agile_table/Su.png" alt="Su" class="pte-image-small" /></a>,
<a href="#Mt"><img src="https://www.niceideas.ch/agile_table/Mt.png" alt="Mt" class="pte-image-small" /></a>,
<a href="#Si"><img src="https://www.niceideas.ch/agile_table/Si.png" alt="Si" class="pte-image-small" /></a>,
<a href="#Am"><img src="https://www.niceideas.ch/agile_table/Am.png" alt="Am" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Mt"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Mt.png"
alt="Mt : Metaphor"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Mt : Metaphor
</b><br>
System Metaphor is itself a metaphor for a simple design with certain qualities. The most important quality is being able to explain the system design to new people without resorting to dumping huge documents on them. A design should have a structure that helps new people begin contributing quickly. The second quality is a design that makes naming classes and methods consistent.
</p>
<p>
Depends on
<a href="#Sn"><img src="https://www.niceideas.ch/agile_table/Sn.png" alt="Sn" class="pte-image-small" /></a>,
<a href="#Rf"><img src="https://www.niceideas.ch/agile_table/Rf.png" alt="Rf" class="pte-image-small" /></a>,
<a href="#Oc"><img src="https://www.niceideas.ch/agile_table/Oc.png" alt="Oc" class="pte-image-small" /></a>,
<a href="#Am"><img src="https://www.niceideas.ch/agile_table/Am.png" alt="Am" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Td"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Td.png"
alt="Td : TDD = Test Driven Development "
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Td : TDD = Test Driven Development
</b><br>
Test-driven development is a software development process that relies on the repetition of a very short development cycle: requirements are turned into very specific test cases, then the software is improved to pass the new tests, only. This is opposed to software development that allows software to be added that is not proven to meet requirements.
</p>
<p>
Depends on
<a href="#Wt"><img src="https://www.niceideas.ch/agile_table/Wt.png" alt="Wt" class="pte-image-small" /></a>,
<a href="#Ci"><img src="https://www.niceideas.ch/agile_table/Ci.png" alt="Ci" class="pte-image-small" /></a>,
<a href="#Cs"><img src="https://www.niceideas.ch/agile_table/Cs.png" alt="Cs" class="pte-image-small" /></a>,
<a href="#Sc"><img src="https://www.niceideas.ch/agile_table/Sc.png" alt="Sc" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Oc"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Oc.png"
alt="Oc : Onsite Customer"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Oc : Onsite Customer
</b><br>
One of the few requirements of extreme programming (XP) is to have the customer available. Not only to help the development team, but to be a part of it as well. All phases of an XP project require communication with the customer, preferably face to face, on site. It's best to simply assign one or more customers to the development team.
</p>
</td>
</tr>
</table>
<a name="Rf"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Rf.png"
alt="Rf : Refactoring"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Rf : Refactoring
</b><br>
We computer programmers hold onto our software designs long after they have become unwieldy. We continue to use and reuse code that is no longer maintainable because it still works in some way and we are afraid to modify it. But is it really cost effective to do so? Extreme Programming (XP) takes the stance that it is not. When we remove redundancy, eliminate unused functionality, and rejuvenate obsolete designs we are refactoring. Refactoring throughout the entire project life cycle saves time and increases quality.
<br>
Refactor mercilessly to keep the design simple as you go and to avoid needless clutter and complexity. Keep your code clean and concise so it is easier to understand, modify, and extend
</p>
<p>
Depends on
<a href="#Td"><img src="https://www.niceideas.ch/agile_table/Td.png" alt="Td" class="pte-image-small" /></a>,
<a href="#Sn"><img src="https://www.niceideas.ch/agile_table/Sn.png" alt="Sn" class="pte-image-small" /></a>,
<a href="#Mt"><img src="https://www.niceideas.ch/agile_table/Mt.png" alt="Mt" class="pte-image-small" /></a>,
<a href="#Cs"><img src="https://www.niceideas.ch/agile_table/Cs.png" alt="Cs" class="pte-image-small" /></a>,
<a href="#Ci"><img src="https://www.niceideas.ch/agile_table/Ci.png" alt="Ci" class="pte-image-small" /></a>,
<a href="#Am"><img src="https://www.niceideas.ch/agile_table/Am.png" alt="Am" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Cs"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Cs.png"
alt="Cs : Coding Standards"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Cs : Coding Standards
</b><br>
Code must be formatted to agreed coding standards. Coding standards keep the code consistent and easy for the entire team to read and refactor. Code that looks the same encourages collective ownership.
</p>
</td>
</tr>
</table>
<a name="Su"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Su.png"
alt="Su : Sustainable Pace"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Su : Sustainable Pace
</b><br>
To set your pace you need to take your iteration ends seriously. You want the most completed, tested, integrated, production ready software you can get each iteration. Incomplete or buggy software represents an unknown amount of future effort, so you can't measure it. If it looks like you will not be able to get everything finished by iteration end have an iteration planning meeting and re-scope the iteration to maximize your project velocity. Even if there is only one day left in the iteration it is better to get the entire team re-focused on a single completed task than many incomplete ones.
</p>
</td>
</tr>
</table>
<a name="Wt"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Wt.png"
alt="Wt : Whole Team"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Wt : Whole Team
</b><br>
All the contributors to an XP project sit together, members of a whole team. The team shares the project goals and the responsibility for achieving them. This team must include a business representative, the "Customer" who provides the requirements, sets the priorities, and steers the project
</p>
</td>
</tr>
</table>
<a name="Ci"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Ci.png"
alt="Ci : Continuous Integration"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Ci : Continuous Integration
</b><br>
Developers should be integrating and commiting code into the code repository every few hours, when ever possible. In any case never hold onto changes for more than a day. Continuous integration often avoids diverging or fragmented development efforts, where developers are not communicating with each other about what can be re-used, or what could be shared. Everyone needs to work with the latest version. Changes should not be made to obsolete code causing integration headaches.
</p>
<p>
Depends on
<a href="#Td"><img src="https://www.niceideas.ch/agile_table/Td.png" alt="Td" class="pte-image-small" /></a>,
<a href="#Cs"><img src="https://www.niceideas.ch/agile_table/Cs.png" alt="Cs" class="pte-image-small" /></a>,
<a href="#Co"><img src="https://www.niceideas.ch/agile_table/Co.png" alt="Co" class="pte-image-small" /></a>,
<a href="#Sc"><img src="https://www.niceideas.ch/agile_table/Sc.png" alt="Sc" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Co"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Co.png"
alt="Co : Collective Ownership"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Co : Collective Ownership
</b><br>
Collective Ownership encourages everyone to contribute new ideas to all segments of the project. Any developer can change any line of code to add functionality, fix bugs, improve designs or refactor. No one person becomes a bottle neck for changes.
</p>
</td>
</tr>
</table>
<a name="Cr"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Cr.png"
alt="Cr : Code Review"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Cr : Code Review
</b><br>
Code review is increasingly favored over strict Pair Programming as initially requires by the XP Method. The problem with Pair programming is that it cannot fitr everybody.
<br>
Code reviews are considered important by many large-process gurus. They are intended to ensure conformance to standards, and more importantly, intended to ensure that the code is clear, efficient, works, and has QWAN. They also intended to help disseminate knowledge about the code to the rest of the team.
</p>
<p>
Depends on
<a href="#Sn"><img src="https://www.niceideas.ch/agile_table/Sn.png" alt="Sn" class="pte-image-small" /></a>,
<a href="#Cs"><img src="https://www.niceideas.ch/agile_table/Cs.png" alt="Cs" class="pte-image-small" /></a>,
<a href="#Sc"><img src="https://www.niceideas.ch/agile_table/Sc.png" alt="Sc" class="pte-image-small" /></a>,
<a href="#Am"><img src="https://www.niceideas.ch/agile_table/Am.png" alt="Am" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Pg"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Pg.png"
alt="Pg : Planning Game"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Pg : Planning Game
</b><br>
The main planning process within extreme programming is called the Planning Game. The game is a meeting that occurs once per iteration, typically once a week. The planning process is divided into two parts: Release Planning and Sprint Planning.
</p>
<p>
Depends on
<a href="#Sr"><img src="https://www.niceideas.ch/agile_table/Sr.png" alt="Sr" class="pte-image-small" /></a>,
<a href="#Oc"><img src="https://www.niceideas.ch/agile_table/Oc.png" alt="Oc" class="pte-image-small" /></a>,
<a href="#Po"><img src="https://www.niceideas.ch/agile_table/Po.png" alt="Po" class="pte-image-small" /></a>,
<a href="#Pm"><img src="https://www.niceideas.ch/agile_table/Pm.png" alt="Pm" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Sr"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Sr.png"
alt="Sr : Small Releases"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Sr : Small Releases
</b><br>
The development team needs to release iterative versions of the system to the customers often. Some teams deploy new software into production every day. At the very least you will want to get new software into production every week or two. At the end of every iteration you will have tested, working, production ready software to demonstrate to your customers. The decision to put it into production is theirs.
</p>
</td>
</tr>
</table>
<a name="Sc"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Sc.png"
alt="Sc : Source Code Management"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Sc : Source Code Management
</b><br>
A component of software configuration management, version control, also known as revision control or source control, is the management of changes to documents, computer programs, large web sites, and other collections of information. Changes are usually identified by a number or letter code, termed the "revision number", "revision level", or simply "revision". For example, an initial set of files is "revision 1". When the first change is made, the resulting set is "revision 2", and so on. Each revision is associated with a timestamp and the person making the change. Revisions can be compared, restored, and with some types of files, merged.
</p>
</td>
</tr>
</table>
<a name="Bs"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Bs.png"
alt="Bs : Boyscout Rule"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Bs : Boyscout Rule
</b><br>
The Boy Scouts have a rule: "Always leave the campground cleaner than you found it." If you find a mess on the ground, you clean it up regardless of who might have made the mess. You intentionally improve the environment for the next group of campers. Actually the original form of that rule, written by Robert Stephenson Smyth Baden-Powell, the father of scouting, was "Try and leave this world a little better than you found it."
<br>
What if we followed a similar rule in our code: "Always check a module in cleaner than when you checked it out." No matter who the original author was, what if we always made some effort, no matter how small, to improve the module. What would be the result?
</p>
<p>
Depends on
<a href="#Rf"><img src="https://www.niceideas.ch/agile_table/Rf.png" alt="Rf" class="pte-image-small" /></a>,
<a href="#Td"><img src="https://www.niceideas.ch/agile_table/Td.png" alt="Td" class="pte-image-small" /></a>,
<a href="#Sn"><img src="https://www.niceideas.ch/agile_table/Sn.png" alt="Sn" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="No"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/No.png"
alt="No : No premature optimization"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
No : No premature optimization
</b><br>
In Donald Knuth's paper "Structured Programming With GoTo Statements", he wrote: "Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%."
</p>
</td>
</tr>
</table>
<a name="At"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/At.png"
alt="At : Acceptance testing"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
At : Acceptance testing
</b><br>
Acceptance tests are created from user stories. During an iteration the user stories selected during the iteration planning meeting will be translated into acceptance tests. The customer specifies scenarios to test when a user story has been correctly implemented. A story can have one or many acceptance tests, what ever it takes to ensure the functionality works.
</p>
<p>
Depends on
<a href="#Oc"><img src="https://www.niceideas.ch/agile_table/Oc.png" alt="Oc" class="pte-image-small" /></a>,
<a href="#Sr"><img src="https://www.niceideas.ch/agile_table/Sr.png" alt="Sr" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Ac"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Ac.png"
alt="Ac : Automated Tests Coverage"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Ac : Automated Tests Coverage
</b><br>
Code Coverage is a measurement of how many lines/blocks/arcs of your code are executed while the automated tests are running.
<br>
Code coverage on every dimension should be above possible to 80% (the famous 80/20) rule and close to 100% (TDD).
</p>
<p>
Depends on
<a href="#Ci"><img src="https://www.niceideas.ch/agile_table/Ci.png" alt="Ci" class="pte-image-small" /></a>,
<a href="#Td"><img src="https://www.niceideas.ch/agile_table/Td.png" alt="Td" class="pte-image-small" /></a>,
<a href="#Am"><img src="https://www.niceideas.ch/agile_table/Am.png" alt="Am" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<h2>Scrum</h2>
<a name="Sp"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Sp.png"
alt="Sp : Sprint"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Sp : Sprint
</b><br>
A Sprint is a time-box of one month or less during which a "Done", useable, and potentially releasable product Increment is created. Sprints best have consistent durations throughout a development effort. A new Sprint starts immediately after the conclusion of the previous Sprint.
</p>
<p>
Depends on
<a href="#Sr"><img src="https://www.niceideas.ch/agile_table/Sr.png" alt="Sr" class="pte-image-small" /></a>,
<a href="#Sl"><img src="https://www.niceideas.ch/agile_table/Sl.png" alt="Sl" class="pte-image-small" /></a>,
<a href="#So"><img src="https://www.niceideas.ch/agile_table/So.png" alt="So" class="pte-image-small" /></a>,
<a href="#Sb"><img src="https://www.niceideas.ch/agile_table/Sb.png" alt="Sb" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="In"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/In.png"
alt="In : Product Increment"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
In : Product Increment (Shippable Product)
</b><br>
In Scrum, the Development Team delivers each Sprint a Product Increment.
<br>
The increment must consist of thoroughly tested code that has been built into an executable, and the user operation of the functionality is documented either in Help files or user documentation. These requirements are documented in the Definition of Done.
<br>
If everything works fine and the Development Team has estimated well, the Product Increment includes all items, which were planned in the Sprint Backlog, tested and documented.
</p>
<p>
Depends on
<a href="#Sr"><img src="https://www.niceideas.ch/agile_table/Sr.png" alt="Sr" class="pte-image-small" /></a>,
<a href="#Sp"><img src="https://www.niceideas.ch/agile_table/Sp.png" alt="Sp" class="pte-image-small" /></a>,
<a href="#Pb"><img src="https://www.niceideas.ch/agile_table/Pb.png" alt="Pb" class="pte-image-small" /></a>,
<a href="#Ci"><img src="https://www.niceideas.ch/agile_table/Ci.png" alt="Ci" class="pte-image-small" /></a>,
<a href="#Cd"><img src="https://www.niceideas.ch/agile_table/Cd.png" alt="Cd" class="pte-image-small" /></a>,
<a href="#Ft"><img src="https://www.niceideas.ch/agile_table/Ft.png" alt="Ft" class="pte-image-small" /></a>,
<a href="#Pt"><img src="https://www.niceideas.ch/agile_table/Pt.png" alt="Pt" class="pte-image-small" /></a>,
<a href="#Ff"><img src="https://www.niceideas.ch/agile_table/Ff.png" alt="Ff" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Sl"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Sl.png"
alt="Sl : Sprint Planning"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Sl : Sprint Planning
</b><br>
In Scrum, the sprint planning meeting is attended by the product owner, ScrumMaster and the entire Scrum team. Outside stakeholders may attend by invitation of the team, although this is rare in most companies.
<br>
During the sprint planning meeting, the product owner describes the highest priority features to the team. The team asks enough questions that they can turn a high-level user story of the product backlog into the more detailed tasks of the sprint backlog.
</p>
<p>
Depends on
<a href="#Pg"><img src="https://www.niceideas.ch/agile_table/Pg.png" alt="Pg" class="pte-image-small" /></a>,
<a href="#Po"><img src="https://www.niceideas.ch/agile_table/Po.png" alt="Po" class="pte-image-small" /></a>,
<a href="#Tv"><img src="https://www.niceideas.ch/agile_table/Tv.png" alt="Tv" class="pte-image-small" /></a>,
<a href="#Us"><img src="https://www.niceideas.ch/agile_table/Us.png" alt="Us" class="pte-image-small" /></a>,
<a href="#Pm"><img src="https://www.niceideas.ch/agile_table/Pm.png" alt="Pm" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="So"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/So.png"
alt="So : Sprint Retrospective"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
So : Sprint Retrospective
</b><br>
No matter how good a Scrum team is, there is always opportunity to improve. Although a good Scrum team will be constantly looking for improvement opportunities, the team should set aside a brief, dedicated period at the end of each sprint to deliberately reflect on how they are doing and to find ways to improve. This occurs during the sprint retrospective.
<br>
The sprint retrospective is usually the last thing done in a sprint. Many teams will do it immediately after the sprint review. The entire team, including both the ScrumMaster and the product owner should participate. You can schedule a scrum retrospective for up to an hour, which is usually quite sufficient. However, occasionally a hot topic will arise or a team conflict will escalate and the retrospective could take significantly longer.
</p>
<p>
Depends on
<a href="#Kb"><img src="https://www.niceideas.ch/agile_table/Kb.png" alt="Kb" class="pte-image-small" /></a>,
<a href="#Wh"><img src="https://www.niceideas.ch/agile_table/Wh.png" alt="Wh" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Sb"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Sb.png"
alt="Sb : Sprint Backlog"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Sb : Sprint Backlog
</b><br>
The sprint backlog is a list of tasks identified by the Scrum team to be completed during the Scrum sprint. During the sprint planning meeting, the team selects some number of product backlog items, usually in the form of user stories, and identifies the tasks necessary to complete each user story. Most teams also estimate how many hours each task will take someone on the team to complete.
</p>
<p>
Depends on
<a href="#Pb"><img src="https://www.niceideas.ch/agile_table/Pb.png" alt="Pb" class="pte-image-small" /></a>,
<a href="#Sp"><img src="https://www.niceideas.ch/agile_table/Sp.png" alt="Sb" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Pb"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Pb.png"
alt="Pb : Product Backlog "
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Pb : Product Backlog
</b><br>
The agile product backlog in Scrum is a prioritized features list, containing short descriptions of all functionality desired in the product. When applying Scrum, it's not necessary to start a project with a lengthy, upfront effort to document all requirements. Typically, a Scrum team and its product owner begin by writing down everything they can think of for agile backlog prioritization. This agile product backlog is almost always more than enough for a first sprint. The Scrum product backlog is then allowed to grow and change as more is learned about the product and its customers.
</p>
<p>
Depends on
<a href="#Sg"><img src="https://www.niceideas.ch/agile_table/Sg.png" alt="Sg" class="pte-image-small" /></a>,
<a href="#Pm"><img src="https://www.niceideas.ch/agile_table/Pm.png" alt="Pm" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Sd"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Sd.png"
alt="Sd : Sprint Demo"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Sd : Sprint Demo
</b><br>
In Scrum, each sprint is required to deliver a potentially shippable product increment. This means that at the end of each sprint, the team has produced a coded, tested and usable piece of software.
<br>
So at the end of each sprint, a sprint review meeting is held. During this meeting, the Scrum team shows what they accomplished during the sprint. Typically this takes the form of a demo of the new features.
</p>
<p>
Depends on
<a href="#Sp"><img src="https://www.niceideas.ch/agile_table/Sp.png" alt="Sp" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Po"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Po.png"
alt="Po : Product Owner"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Po : Product Owner
</b><br>
The Scrum product owner is typically a project's key stakeholder. Part of the product owner responsibilities is to have a vision of what he or she wishes to build, and convey that vision to the scrum team. This is key to successfully starting any agile software development project. The agile product owner does this in part through the product backlog, which is a prioritized features list for the product.
<br>
The product owner is commonly a lead user of the system or someone from marketing, product management or anyone with a solid understanding of users, the market place, the competition and of future trends for the domain or type of system being developed.
</p>
<p>
Depends on
<a href="#Oc"><img src="https://www.niceideas.ch/agile_table/Oc.png" alt="Oc" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Ds"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Ds.png"
alt="Ds : Daily Scrum"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Ds : Daily Scrum
</b><br>
In Scrum, on each day of a sprint, the team holds a daily scrum meeting called the "daily scrum." Meetings are typically held in the same location and at the same time each day. Ideally, a daily scrum meeting is held in the morning, as it helps set the context for the coming day's work. These scrum meetings are strictly time-boxed to 15 minutes. This keeps the discussion brisk but relevant.
</p>
</td>
</tr>
</table>
<a name="Sm"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Sm.png"
alt="Sm : Scrum Master"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Sm : Scrum Master
</b><br>
What is a Scrum Master? The ScrumMaster is responsible for making sure a Scrum team lives by the values and practices of Scrum. The ScrumMaster is often considered a coach for the team, helping the team do the best work it possibly can. The ScrumMaster can also be thought of as a process owner for the team, creating a balance with the project's key stakeholder, who is referred to as the product owner.
<br>
The ScrumMaster does anything possible to help the team perform at their highest level. This involves removing any impediments to progress, facilitating meetings, and doing things like working with the product owner to make sure the product backlog is in good shape and ready for the next sprint. The ScrumMaster role is commonly filled by a former project manager or a technical team leader but can be anyone.
</p>
</td>
</tr>
</table>
<a name="Do"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Do.png"
alt="Do: Definition of Done"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Do: Definition of Done
</b><br>
Definition of Done is a simple list of activities (writing code, coding comments, unit testing, integration testing, release notes, design documents, etc.) that add verifiable/demonstrable value to the product. Focusing on value-added steps allows the team to focus on what must be completed in order to build software while eliminating wasteful activities that only complicate software development efforts.
</p>
<p>
Depends on
<a href="#Cs"><img src="https://www.niceideas.ch/agile_table/Cs.png" alt="Cs" class="pte-image-small" /></a>,
<a href="#Cr"><img src="https://www.niceideas.ch/agile_table/Cr.png" alt="Cr" class="pte-image-small" /></a>,
<a href="#Td"><img src="https://www.niceideas.ch/agile_table/Td.png" alt="Td" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Pp"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Pp.png"
alt="Pp : Planning Poker"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Pp : Planning Poker
</b><br>
Planning Poker is an agile estimating and planning technique that is consensus based. To start a poker planning session, the product owner or customer reads an agile user story or describes a feature to the estimators.
<br>
Each estimator is holding a deck of Planning Poker cards with values like 0, 1, 2, 3, 5, 8, 13, 20, 40 and 100, which is the sequence we recommend. The values represent the number of story points, ideal days, or other units in which the team estimates.
<br>
The estimators discuss the feature, asking questions of the product owner as needed. When the feature has been fully discussed, each estimator privately selects one card to represent his or her estimate. All cards are then revealed at the same time.
<br>
If all estimators selected the same value, that becomes the estimate. If not, the estimators discuss their estimates. The high and low estimators should especially share their reasons. After further discussion, each estimator reselects an estimate card, and all cards are again revealed at the same time.
<br>
The poker planning process is repeated until consensus is achieved or until the estimators decide that agile estimating and planning of a particular item needs to be deferred until additional information can be acquired.
</p>
<p>
Depends on
<a href="#Es"><img src="https://www.niceideas.ch/agile_table/Es.png" alt="Es" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Es"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Es.png"
alt="Es : Estimations in Story Points"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Es : Estimations in Story Points
</b><br>
Story points are a unit of measure for expressing an estimate of the overall effort that will be required to fully implement a product backlog item or any other piece of work.
<br>
When we estimate with story points, we assign a point value to each item. The raw values we assign are unimportant. What matters are the relative values. A story that is assigned a 2 should be twice as much as a story that is assigned a 1. It should also be two-thirds of a story that is estimated as 3 story points.
<br>
Instead of assigning 1, 2 and 3, that team could instead have assigned 100, 200 and 300. Or 1 million, 2 million and 3 million. It is the ratios that matter, not the actual numbers.
</p>
</td>
</tr>
</table>
<a name="Tv"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Tv.png"
alt="Tv : Team Velocity"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Tv : Team Velocity
</b><br>
Velocity is simply a metric based on the completed items in a sprint by a single team. The metric is completely subjective to that specific team, and should never be extrapolated for any other comparison.
<br>
Velocity is a reflective metric gathered from the sprint throughput of a stable team. Usually, a velocity metric is not considered valid until several sprints have been completed.
</p>
<p>
Depends on
<a href="#Es"><img src="https://www.niceideas.ch/agile_table/Es.png" alt="Es" class="pte-image-small" /></a>,
<a href="#Sp"><img src="https://www.niceideas.ch/agile_table/Sp.png" alt="Sp" class="pte-image-small" /></a>,
<a href="#So"><img src="https://www.niceideas.ch/agile_table/So.png" alt="So" class="pte-image-small" /></a>,
<a href="#Su"><img src="https://www.niceideas.ch/agile_table/Su.png" alt="Su" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<h2>Product Development </h2>
<a name="Us"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Us.png"
alt="Us : User Stories"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Us : User Stories
</b><br>
In software development and product management, a user story is an informal, natural language description of one or more features of a software system. User stories are often written from the perspective of an end user or user of a system. They are often recorded on index cards, on Post-it notes, or in project management software. Depending on the project, user stories may be written by various stakeholders including clients, users, managers or development team members.
</p>
<p>
Depends on
<a href="#Oc"><img src="https://www.niceideas.ch/agile_table/Oc.png" alt="Oc" class="pte-image-small" /></a>,
<a href="#Po"><img src="https://www.niceideas.ch/agile_table/Po.png" alt="Po" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Sg"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Sg.png"
alt="Sg : Story Mapping"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Sg : Story Mapping
</b><br>
Story mapping consists of ordering user stories along two independent dimensions. The "map" arranges user activities along the horizontal axis in rough order of priority (or "the order in which you would describe activities to explain the behaviour of the system"). Down the vertical axis, it represents increasing sophistication of the implementation.
<br>
Given a story map so arranged, the first horizontal row represents a "walking skeleton", a barebone but usable version of the product. Working through successive rows fleshes out the product with additional functionality.
</p>
<p>
Depends on
<a href="#Oc"><img src="https://www.niceideas.ch/agile_table/Oc.png" alt="Oc" class="pte-image-small" /></a>,
<a href="#Po"><img src="https://www.niceideas.ch/agile_table/Po.png" alt="Po" class="pte-image-small" /></a>,
<a href="#Us"><img src="https://www.niceideas.ch/agile_table/Us.png" alt="Us" class="pte-image-small" /></a>,
<a href="#Pv"><img src="https://www.niceideas.ch/agile_table/Pv.png" alt="Pv" class="pte-image-small" /></a>,
<a href="#Pm"><img src="https://www.niceideas.ch/agile_table/Pm.png" alt="Pm" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Cc"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Cc.png"
alt="Cc : 3 C's - Card, conversation, confirmation"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Cc : 3 C's - Card, conversation, confirmation
</b><br>
"Card, Conversation, Confirmation"; this formula (from Ron Jeffries) captures the components of a User Story:
<br>
a <b>"Card"</b> (or often a Post-It note), a physical token giving tangible and durable form to what would otherwise only be an abstraction;
<br>
a <b>"conversation"</b> taking place at different time and places during a project between the various people concerned by a given feature of a software product: customers, users, developers, testers; this conversation is largely verbal but most often supplemented by documentation;
<br>
the <b>"confirmation"</b>, finally, the more formal the better, that the objectives the conversation revolved around have been reached.
</p>
<p>
Depends on
<a href="#Us"><img src="https://www.niceideas.ch/agile_table/Us.png" alt="Us" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Pv"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Pv.png"
alt="Pv : Product Vision"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Pv : Product Vision (elevator Pitch)
</b><br>
Every Scrum project needs a product vision that acts as the project's true north, sets the direction and guides the Scrum team. It is the overarching goal everyone must share – Product Owner, ScrumMaster, team, management, customers and other stakeholders. As Ken Schwaber puts it: "The minimum plan necessary to start a Scrum project consists of a vision and a Product Backlog. The vision describes why the project is being undertaken and what the desired end state is."
</p>
<p>
Depends on
<a href="#Pm"><img src="https://www.niceideas.ch/agile_table/Pm.png" alt="Pm" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Iv"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Iv.png"
alt="Iv : INVEST"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Iv : INVEST
</b><br>
The INVEST mnemonic for agile software projects was created by Bill Wake as a reminder of the characteristics of a good quality User Story:
<br>
<b>Independent</b>: The User Story should be self-contained, in a way that there is no inherent dependency on another PBI;
<br>
<b>Negotiable</b>: User Stories are not explicit contracts and should leave space for discussion;
<br>
<b>Valuable</b>: A User Story must deliver value to the stakeholders;
<br>
<b>Estimatable</b>: You must always be able to estimate the size of a User Story;
<br>
<b>Small</b>: User Storys should not be so big as to become impossible to plan/task/prioritize with a certain level of accuracy;
<br>
<b>Testable</b>The User Story or its related description must provide the necessary information to make test development possible.
</p>
<p>
Depends on
<a href="#Us"><img src="https://www.niceideas.ch/agile_table/Us.png" alt="Us" class="pte-image-small" /></a>,
<a href="#Oc"><img src="https://www.niceideas.ch/agile_table/Oc.png" alt="Oc" class="pte-image-small" /></a>,
<a href="#Po"><img src="https://www.niceideas.ch/agile_table/Po.png" alt="Po" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<h2>DevOps</h2>
<a name="Ff"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Ff.png"
alt="Ff : Feature Flipping"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Ff : Feature Flipping
</b><br>
Feature flipping is a technique in software development that attempts to provide an alternative to maintaining multiple source-code branches (known as feature branches), such that the feature can be tested, even before it is completed and ready for release. Feature flipping is used to hide, enable or disable the features, during run time. For example, during the development process, the developer can enable the feature for testing and disable it for remaining users
</p>
<p>
Depends on
<a href="#Cm"><img src="https://www.niceideas.ch/agile_table/Cm.png" alt="Cm" class="pte-image-small" /></a>,
<a href="#Am"><img src="https://www.niceideas.ch/agile_table/Am.png" alt="Am" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Cd"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Cd.png"
alt="Cd : Continuous Delivery"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Cd : Continuous Delivery
</b><br>
Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing, and releasing software faster and more frequently. The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.
</p>
<p>
Depends on
<a href="#Ci"><img src="https://www.niceideas.ch/agile_table/Ci.png" alt="Ci" class="pte-image-small" /></a>,
<a href="#Td"><img src="https://www.niceideas.ch/agile_table/Td.png" alt="Td" class="pte-image-small" /></a>,
<a href="#Ap"><img src="https://www.niceideas.ch/agile_table/Ap.png" alt="Ap" class="pte-image-small" /></a>,
<a href="#In"><img src="https://www.niceideas.ch/agile_table/In.png" alt="In" class="pte-image-small" /></a>,
<a href="#Sr"><img src="https://www.niceideas.ch/agile_table/Sr.png" alt="Sr" class="pte-image-small" /></a>,
<a href="#Ic"><img src="https://www.niceideas.ch/agile_table/Ic.png" alt="Ic" class="pte-image-small" /></a>,
<a href="#Zd"><img src="https://www.niceideas.ch/agile_table/Zd.png" alt="Zd" class="pte-image-small" /></a>,
<a href="#Vc"><img src="https://www.niceideas.ch/agile_table/Vc.png" alt="Vc" class="pte-image-small" /></a>,
<a href="#Bp"><img src="https://www.niceideas.ch/agile_table/Bp.png" alt="Bp" class="pte-image-small" /></a>,
<a href="#Ar"><img src="https://www.niceideas.ch/agile_table/Ar.png" alt="Ar" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Ap"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Ap.png"
alt="Ap : Automated Provisioning"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Ap : Automated Provisioning
</b><br>
(Infrastructure as Code) Server provisioning is a set of actions to prepare a server with appropriate systems, data and software, and make it ready for network operation. Typical tasks when provisioning a server are: select a server from a pool of available servers, load the appropriate software (operating system, device drivers, middleware, and applications), appropriately customize and configure the system and the software to create or change a boot image for this server, and then change its parameters, such as IP address, IP Gateway to find associated network and storage resources (sometimes separated as resource provisioning) to audit the system
<br>
With DevOps and Automated Provisioning, this whole configuration pipeline should be completely automated and executable in one-click, either automatically or on-demand.
</p>
<p>
Depends on
<a href="#Ic"><img src="https://www.niceideas.ch/agile_table/Ic.png" alt="Ic" class="pte-image-small" /></a>,
<a href="#Cm"><img src="https://www.niceideas.ch/agile_table/Cm.png" alt="Cm" class="pte-image-small" /></a>,
<a href="#Vc"><img src="https://www.niceideas.ch/agile_table/Vc.png" alt="Vc" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Ic"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Ic.png"
alt="Ic : Infrastructure Continuous Integration"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Ic : Infrastructure Continuous Integration
</b><br>
(Infrastructure as Code) Infrastructure Continuous Integration consists in leveraging Continuous Integration techniques to Infrastructure components.
<br>
The continuous integration system is necessarily complex, spanning the development, test and staging environments. The continuous integration build should continuously build and test the provisioning, configuring and maintaining of the various infrastructure components.
</p>
<p>
Depends on
<a href="#Ci"><img src="https://www.niceideas.ch/agile_table/Ci.png" alt="Ci" class="pte-image-small" /></a>,
<a href="#Cm"><img src="https://www.niceideas.ch/agile_table/Cm.png" alt="Cm" class="pte-image-small" /></a>,
<a href="#Ap"><img src="https://www.niceideas.ch/agile_table/Ap.png" alt="Ap" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Zd"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Zd.png"
alt="Zd : Zero Downtime Deployments"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Zd : Zero Downtime Deployments
</b><br>
A Zero Downtime Deployment consists in redeploying (typically for a software upgrade) a production system without any downtime appearing to end users. To achieve such lofty goals, redundancy becomes a critical requirement at every level of your infrastructure. There are various techniques involved such a canari release or blue-green deployments.
</p>
<p>
Depends on
<a href="#Cm"><img src="https://www.niceideas.ch/agile_table/Cm.png" alt="Cm" class="pte-image-small" /></a>,
<a href="#Am"><img src="https://www.niceideas.ch/agile_table/Am.png" alt="Am" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Cm"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Cm.png"
alt="Cm : Configuration Management"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Cm : Configuration Management
</b><br>
Configuration management is a class of tool supporting the automation of the configuration of a system, platform or software. It typically consists in <i>define-with-code</i> the various config elements that prepare a provisioned compute resource (like a server or AWS Ec2 instance) for service (installing software, setting up users, configuring services, placing files with template-defined variables, defining external config resources like DNS records in a relevant zone).
</p>
<p>
Depends on
<a href="#Am"><img src="https://www.niceideas.ch/agile_table/Am.png" alt="Am" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Vc"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Vc.png"
alt="Vc : Virtualization and Containers"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Vc : Virtualization and Containers
</b><br>
Hardware virtualization or platform virtualization refers to the creation of a virtual machine that acts like a real computer with an operating system. Software executed on these virtual machines is separated from the underlying hardware resources.
<br>
Containerization - also called container-based virtualization and application containerization - is an OS-level virtualization method for deploying and running distributed applications without launching an entire VM for each application. Instead, multiple isolated systems, called containers, are run on a single control host and access a single kernel.
</p>
</td>
</tr>
</table>
<a name="Bp"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Bp.png"
alt="Bp : Build Pipelines"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Bp : Build Pipelines
</b><br>
Build pipelines are integrated views of downstream and upstream build jobs on a build server. Build pipelines are requires to automated all the various tasks towards continuous delivery such as : provisionning of the environment, build of the various software (with compilation, tests, packaging, etc.), deployment of the software components, applying configuration and testing the deployed platform.
</p>
<p>
Depends on
<a href="#Ic"><img src="https://www.niceideas.ch/agile_table/Ic.png" alt="Ic" class="pte-image-small" /></a>,
<a href="#Ci"><img src="https://www.niceideas.ch/agile_table/Ci.png" alt="Ci" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Ar"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Ar.png"
alt="Ar : Automated Releases"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Ar : Automated Releases
</b><br>
Release Automation consists in automating all the various steps required to release a new version of a software: building, testing, tagging, branching and deploying the binaries to a Binary management tools.
</p>
<p>
Depends on
<a href="#Bp"><img src="https://www.niceideas.ch/agile_table/Bp.png" alt="Bp" class="pte-image-small" /></a>,
<a href="#Bm"><img src="https://www.niceideas.ch/agile_table/Bm.png" alt="Bm" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="St"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/St.png"
alt="St : Share the tools"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
St : Share the tools
</b><br>
Share the tools is a DevOps principles aimed at leveraging both Dev and Ops tools and practices to the other side of the wall. Developers should leverage their automation and building tool to Infrastructure Automation, Provisionning and Testing. Ops should share the production monitoring concerns with developers.
</p>
<p>
Depends on
<a href="#Am"><img src="https://www.niceideas.ch/agile_table/Am.png" alt="Am" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Os"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Os.png"
alt="Os : Operators are stakeholders"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Os : Operators are stakeholders
</b><br>
Operators as stakeholders is a DevOps principle stating that Operators should be considered the other users of the platform. They should be fully integrated in the Software Development Process.
<br>
At specification time, operators should give their non-functional requirements just as business users give their functional requirement. Such non-functional requirements should be handled with same important and priority by the development team.
<br>
At implementation time, operators should provide feedback and non-functional tests specifications continuously just as business users provides feedback on functional features.
</p>
<p>
Depends on
<a href="#Pm"><img src="https://www.niceideas.ch/agile_table/Pm.png" alt="Pm" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Or"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Or.png"
alt="Or : Operators in Rituals"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Or : Operators in Rituals
</b><br>
Operators in Rituals is a DevOps principle stating that operators should be integrated in the Development Team Rituals such as the Sprint Planning and Sprint Retrospective and represent non-functional constraints during these rituals just as the Product Owner represents the functional interests.
</p>
<p>
Depends on
<a href="#Sl"><img src="https://www.niceideas.ch/agile_table/Sl.png" alt="Sl" class="pte-image-small" /></a>,
<a href="#So"><img src="https://www.niceideas.ch/agile_table/So.png" alt="So" class="pte-image-small" /></a>,
<a href="#Ds"><img src="https://www.niceideas.ch/agile_table/Ds.png" alt="Ds" class="pte-image-small" /></a>,
<a href="#Cd"><img src="https://www.niceideas.ch/agile_table/Cd.png" alt="Cd" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Bm"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Bm.png"
alt="Bm : Binaries Management"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Bm : Binaries Management
</b><br>
A binary repository manager is a software tool designed to optimize the download and storage of binary files used and produced in software development. It centralizes the management of all the binary artifacts generated and used by the organization to overcome the complexity arising from the diversity of binary artifact types, their position in the overall workflow and the dependencies between them.
<br>
A binary repository is a software repository for packages, artifacts and their corresponding metadata. It can be used to store binary files produced by an organization itself, such as product releases and nightly product builds, or for third party binaries which must be treated differently for both technical and legal reasons.
</p>
</td>
</tr>
</table>
<h2>Lean Startup</h2>
<a name="Fl"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Fl.png"
alt="Fl : Feedback Loop"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Fl : Feedback Loop
</b><br>
The <i>Build-Measure-Learn</i> feedback loop is one of the central principles of Lean Startup Method.
<br>
A startup is to find a successful revenue model that can be developed with further investment. Build-Measure-Learn is a framework for establishing – and continuously improving – the effectiveness of new products, services and ideas quickly and cost-effectively.
<br>
In practice, the model involves a cycle of creating and testing hypotheses by building something small for potential customers to try, measuring their reactions, and learning from the results.
</p>
<p>
Depends on
<a href="#Sd"><img src="https://www.niceideas.ch/agile_table/Sd.png" alt="Sd" class="pte-image-small" /></a>,
<a href="#Cd"><img src="https://www.niceideas.ch/agile_table/Cd.png" alt="Cd" class="pte-image-small" /></a>,
<a href="#Oc"><img src="https://www.niceideas.ch/agile_table/Oc.png" alt="Oc" class="pte-image-small" /></a>,
<a href="#Pm"><img src="https://www.niceideas.ch/agile_table/Pm.png" alt="Pm" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Ft"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Ft.png"
alt="Ft : feature Teams"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Ft : feature Teams
</b><br>
A <i>feature team</i> is a long-lived, cross-functional, cross-component team that completes many end-to-end customer features—one by one. It is opposed to the traditional approach of <i>Component Team</i> where a team is specialized on an individual software components and maintains it over several projects at the same time.
<br>
The Feature team approach seeks to avoid the bottlenecks usually appearing with Component Teams.
</p>
</td>
</tr>
</table>
<a name="Fa"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Fa.png"
alt="Fa : Fail Fast"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Fa : Fail Fast
</b><br>
Fail fast means getting out of planning mode and into testing mode, eventually for every critical component of your model of change. Customer development is the process that embodies this principle and helps you determine which hypotheses to start with and which are the most critical for your new idea.
<br>
An important goal of the philosophy is to cut losses when testing reveals something isn't working and quickly try something else, a concept known as pivoting.
</p>
<p>
Depends on
<a href="#Pm"><img src="https://www.niceideas.ch/agile_table/Pm.png" alt="Pm" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Mv"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Mv.png"
alt="Mv : MVP"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Mv : MVP
</b><br>
In product development, the minimum viable product (MVP) is a product with just enough features to satisfy early customers, and to provide feedback for future development
</p>
<p>
Depends on
<a href="#Pm"><img src="https://www.niceideas.ch/agile_table/Pm.png" alt="Pm" class="pte-image-small" /></a>,
<a href="#Am"><img src="https://www.niceideas.ch/agile_table/Am.png" alt="Am" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Gb"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Gb.png"
alt="Gb : Get Out of the building"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Gb : Get Out of the building
</b><br>
If you are pre-Product/Market Fit and you aren't actually "Getting out of the Building" (actually talking to your customers), you aren't doing Customer Development, and your startup isn't a Lean Startup.
<br>
Again: If you aren't actually talking to your customers, you aren't doing Customer Development.
</p>
</td>
</tr>
</table>
<a name="Pt"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Pt.png"
alt="Pt : Pizza Teams"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Pt : Pizza Teams
</b><br>
The idea of a "two pizza team" was coined by Jeff Bezo, founder of Amazon.com. If you can't feed a team with two pizzas, it's too large. That limits a task force to five to seven people, depending on their appetites."
<br>
The underlying idea is that as a team's size grows, the amount of one-on-one communication channels tend to explode.
<br>
Beyond ten, communication loses efficiency, cohesion diminishes, parasitism behaviors and power struggles appear, and the performance of the team decreases very rapidly with the number of members.
</p>
</td>
</tr>
</table>
<a name="As"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/As.png"
alt="As : Actionable Metrics"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
As : Actionable Metrics
</b><br>
The only metrics that entrepreneurs should invest energy in collecting are those that help them make decisions. Actionable Metrics are opposed to Vanity Metrics.
<br>
This is a precision of another fundamental Lean Startup practice which is "Obsession of Measure" stating that everything should be measured and no decision should be taken in the company if it is not supported by a Key Process Indicator or a Key Risk Indicator.
</p>
<p>
Depends on
<a href="#Pm"><img src="https://www.niceideas.ch/agile_table/Pm.png" alt="Pm" class="pte-image-small" /></a>,
<a href="#Am"><img src="https://www.niceideas.ch/agile_table/Am.png" alt="Am" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Bb"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Bb.png"
alt="Bb : Build vs. Buy"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Bb : Build vs. Buy
</b><br>
This is a fundamental principle of the Lean Startup and the web giants : favor as much as possible building your own software, your own feature instead of buying a third party software or library.
<br>
When initiating a startup, having to pay fees to third party corporations before reaching a sustainable growth is suicidal.
</p>
<p>
Depends on
<a href="#Pm"><img src="https://www.niceideas.ch/agile_table/Pm.png" alt="Pm" class="pte-image-small" /></a>,
<a href="#Am"><img src="https://www.niceideas.ch/agile_table/Am.png" alt="Am" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Ab"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Ab.png"
alt="Ab : A/B Testing"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Ab : A/B Testing
</b><br>
In marketing and business intelligence, A/B testing is a term for a controlled experiment with two variants, A and B. It can be considered as a form of statistical hypothesis testing with two variants leading to the technical term, two-sample hypothesis testing, used in the field of statistics
</p>
<p>
Depends on
<a href="#Pm"><img src="https://www.niceideas.ch/agile_table/Pm.png" alt="Pm" class="pte-image-small" /></a>,
<a href="#Am"><img src="https://www.niceideas.ch/agile_table/Am.png" alt="Am" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<h2>Kanban</h2>
<a name="Ko"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Ko.png"
alt="Ko : Kanban Board"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Ko : Kanban Board
</b><br>
A Kanban board is a work and workflow visualization tool that enables you to optimize the flow of your work. Physical Kanban boards typically use sticky notes on a whiteboard to communicate status, progress, and issues.
<br>
An agile corporation should use a KanBan board to monitor all its processes.
<br>
A development team will typically use a Kanban board to monitor the Sprint backlog completion during a sprint.
</p>
</td>
</tr>
</table>
<h2>Kaizen </h2>
<a name="Kb"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Kb.png"
alt="Kb : Kaizen Burst"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Kb : Kaizen Burst
</b><br>
The Kaizen burst is a specific Kaizen process integrated the the development rituals. In Agile Software Development, it is really integrated in the Sprint Retrospective. This idea is to identify in a visual way (with a post-it on a board for instance) the weaknesses or problems in the development practices or processes. These boxes are called Kaizen burst.
<br>
Theses boxes are commented as actions are taken towards improvement and eventually removed when the weakness has been addressed or the problem solved.
</p>
<p>
Depends on
<a href="#So"><img src="https://www.niceideas.ch/agile_table/So.png" alt="So" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<a name="Wh"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Wh.png"
alt="Wh : 5 Why"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Wh : 5 Why
</b><br>
5 Whys is an iterative interrogative technique used to explore the cause-and-effect relationships underlying a particular problem.
<br>
The primary goal of the technique is to determine the root cause of a defect or problem by repeating the question "Why?" Each answer forms the basis of the next question. The "5" in the name derives from an anecdotal observation on the number of iterations needed to resolve the problem.
</p>
<p>
Depends on
<a href="#So"><img src="https://www.niceideas.ch/agile_table/So.png" alt="So" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<h2>FDD (Feature Driven Development)</h2>
<a name="Si"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Si.png"
alt="Si : SOLID principles"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Si : SOLID principles
</b><br>
In computer programming, the term SOLID is a mnemonic acronym for five design principles intended to make software designs more understandable, flexible and maintainable. The principles are a subset of many principles promoted by Robert C. Martin.
<br>
Though they apply to any object-oriented design, the SOLID principles can also form a core philosophy for methodologies such as agile development or Adaptive Software Development.
<br>
The 5 principles are as follows:
<br>
<b>SRP</b> : Single responsibility principle - a class should have only a single responsibility (i.e. only one potential change in the software's specification should be able to affect the specification of the class)
<br>
<b>OCP</b> : Open/closed principle - "software entities ... should be open for extension, but closed for modification."
<br>
<b>LSP</b> : Liskov substitution principle - "objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program."
<br>
<b>ISP</b> : Interface segregation principle - "many client-specific interfaces are better than one general-purpose interface."
<br>
<b>DIP</b> : Dependency inversion principle - one should "depend upon abstractions, not concretions."
</p>
<p>
Depends on
<a href="#Am"><img src="https://www.niceideas.ch/agile_table/Am.png" alt="Am" class="pte-image-small" /></a>
</p>
</td>
</tr>
</table>
<h2>DAD</h2>
<a name="Pm"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Pm.png"
alt="Pm : Product Management Committee"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Pm : Product Management Committee
</b><br>
The Product Management Committee is both a team and a ritual that enforces a smart approach to product management.
<br>
Product Management consists in identifying and evolving your organization’s business vision; identifying and prioritizing potential products/solutions to support that vision; identifying, prioritizing, and allocating features to products under development; managing functional dependencies between products; and marketing those products to their potential customers.
<br>
The Product Management Committee is the weekly (or bi-weekly) ritual enforcing and supporting this process with the required role attending the committee. It is led by the product Owner which has more a role of facilitator and arbitrator that a formal decision role. The Product Owner represents the PMC to the development team.
</p>
</td>
</tr>
</table>
<a name="Am"></a>
<table class="pte noborder">
<tr>
<td>
<img src="https://www.niceideas.ch/agile_table/Am.png"
alt="Am : Architecture Committee"
class="pte-image-big" />
</td>
<td class="desc_right">
<p>
<b>
Am : Architecture Committee
</b><br>
The Architecture Committee is responsible to analyze user stories and define Development Tasks. Every story should be specified, designed and discussed. Screen mockups if applicable should be drawn, acceptance criteria agreed, etc.
<br>
Since the Architecture Committee is also responsible for estimating Stories, it's important that representatives of the Development Team, not only the Tech Leads and the Architects, but simple developers as well, take part in it.
</p>
</td>
</tr>
</table>
<br>
<p>
(This article is available as a PDF document here <a href="https://www.niceideas.ch/Agile_table.pdf">https://www.niceideas.ch/Agile_table.pdf</a> and a slideshare presentation there <a href="https://www.slideshare.net/JrmeKehrli/periodic-table-of-agile-principles-and-practices">https://www.slideshare.net/JrmeKehrli/periodic-table-of-agile-principles-and-practices</a>)
</p>
https://www.niceideas.ch/roller2/badtrash/entry/agile-planning-tools-and-processes
Agile Planning : tools and processes
Jerome Kehrli
2017-06-14T14:42:48-04:00
2017-09-13T16:48:48-04:00
<!-- Agile Planning : tools and processes -->
<p>
All the work on Agility in the Software Engineering Business in the past 20 years, initiated by Kent Beck, Ward Cunningham and Ron Jeffries, comes from the finding that traditional engineering methodologies apply only poorly to the Software Engineering business.
</p>
<p>
If you think about it, we are building bridges from the early stages of the Roman Empire, three thousand years ago. We are building heavy mechanical machinery for almost three hundred years. But we are really writing software for only fifty years.
<br>
In addition, designing a bridge or a mechanical machine is a lot more concrete than designing a Software. When an engineering team has to work on the very initial stage of the design of a bridge or mechanical machine, everyone in the team can picture the result in his mind in a few minutes and breaking it down to a set of single Components can be done almost visually in one's mind.
</p>
<p>
A software, on the other hand, is a lot more abstract. This has the consequence that a software is <a href="https://www.niceideas.ch/roller2/badtrash/entry/funny-developer-tale">much harder to describe than any other engineering product which leads to many levels of misunderstanding</a>.
</p>
<p>
The <a href="https://en.wikipedia.org/wiki/Waterfall_model">waterfall model</a> of Project Management in Software Engineering really originates in the manufacturing and construction industries.
<br>
Unfortunately, for the reasons mentionned above, despite being so widely used in the industry, it applies only pretty poorly to the Software Engineering business. <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-software-development-lessons-learned#sec11">Most important problems</a> it suffers from are as follows:
</p>
<ul>
<li>
<b>Incomplete or moving specification:</b> due to the abstract nature of software, it's impossible for business experts and business analysts to get it right the first time.
</li>
<li>
<b>The tunnel effect:</b> we live in a very fast evolving world and businesses need to adapt all the time. The software delivered after 2 years of heavy development will fulfill (hardly, but let's admit it) the requirements that were true two years ago, not anymore today.
</li>
<li>
<b>Drop of Quality to meet deadlines:</b> An engineering project is always late, always. Things are just a lot worst with software.
</li>
<li>
<b>Heightened tensions between teams:</b> The misunderstanding between teams leads to tensions, and it most of the time turns pretty ugly pretty quick.
</li>
</ul>
<p>
So again, some 20 years ago, Beck, Cunningham and Jeffries started to formalize some of the practices they were successfully using to address the uncertainties, the overwhelming abstraction and the misunderstandings inherent to software development. They formalized it as the <a href="https://en.wikipedia.org/wiki/Extreme_programming">eXtreme Programming</a> methodology.
</p>
<p>
A few years later, the same guys, with some other pretty well known Software Engineers, such as Alistair Cockburn and Martin Fowler, gathered together in a resort in Utah and wrote the <a href="https://en.wikipedia.org/wiki/Agile_software_development#The_Agile_Manifesto">Manifesto for Agile Software Development</a> in which they shared the essential principles and practices they were successfully using to address problems with more traditional and heavyweight software development methodologies.
</p>
<p>
Today, <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-landscape">Agility is a lot of things</a> and the set of principles of practices in the whole Agile family is very large. Unfortunately, most of them require a lot of experience to be understood and then applied successfully within an organization.
</p>
<p>
Unfortunately, the complexity of embracing a sound Agile Software Development Methodology and the required level of maturity a team has to have to benefit from its advantages is really completely underestimated.
<br>
I cannot remember the number of times I heard a team pretending it was an Agile team because it was doing a Stand up in the morning and deployed Jenkins to run the unit tests at every commit. But yeah, honestly I cannot blame them. It is actually difficult to understand Agile Principles and Practices when one never suffered from the very drawbacks and problems they are addressing.
</p>
<p>
I myself am not an agilist. Agility is not a passion, neither something that thrills me nor something that I love studying in my free time. Agility is to me simply a necessity. I discovered and applied Agile Principles and practices out of necessity and urgency, to address specific issues and problems I was facing with the way my teams were developing software.
</p>
<p>
The latest problem I focused on was Planning. Waterfall and RUP focus a lot on planning and are often mentioned to be superior to Agile methods when it comes to forecasting and planning.
<br>
I believe that this is true when Agility is embraced only incompletely. As a matter of fact, I believe that Agility leads to much better and much more reliable forecasts than traditional methods mostly because:
</p>
<ul>
<li>With Agility, it becomes easy to update and adapt Planning and forecasts to always match the evolving reality and the changes in direction and priority.</li>
<li>When embracing agility as a whole, the tools put in the hands of Managers and Executive are first much simpler and second more accurate than traditional planning tools.</li>
</ul>
<p>
In this article, I intend to present the fundamentals, the roles, the processes, the rituals and the values that I believe a team would need to embrace to achieve success down the line in Agile Software Development Management - Product Management, Team Management and Project Management - with the ultimate goal of making planning and forecasting as simple and efficient as it can be.
<br>
All of this is a reflection of the tools, principles and practices we have embraced or are introducing in my current company.
</p>
<!-- Agile Planning : tools and processes -->
<p>
All the work on Agility in the Software Engineering Business in the past 20 years, initiated by Kent Beck, Ward Cunningham and Ron Jeffries, comes from the finding that traditional engineering methodologies apply only poorly to the Software Engineering business.
</p>
<p>
If you think about it, we are building bridges from the early stages of the Roman Empire, three thousand years ago. We are building heavy mechanical machinery for almost three hundred years. But we are really writing software for only fifty years.
<br>
In addition, designing a bridge or a mechanical machine is a lot more concrete than designing a Software. When an engineering team has to work on the very initial stage of the design of a bridge or mechanical machine, everyone in the team can picture the result in his mind in a few minutes and breaking it down to a set of single Components can be done almost visually in one's mind.
</p>
<p>
A software, on the other hand, is a lot more abstract. This has the consequence that a software is <a href="https://www.niceideas.ch/roller2/badtrash/entry/funny-developer-tale">much harder to describe than any other engineering product which leads to many levels of misunderstanding</a>.
</p>
<p>
The <a href="https://en.wikipedia.org/wiki/Waterfall_model">waterfall model</a> of Project Management in Software Engineering really originates in the manufacturing and construction industries.
<br>
Unfortunately, for the reasons mentionned above, despite being so widely used in the industry, it applies only pretty poorly to the Software Engineering business. <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-software-development-lessons-learned#sec11">Most important problems</a> it suffers from are as follows:
</p>
<ul>
<li>
<b>Incomplete or moving specification:</b> due to the abstract nature of software, it's impossible for business experts and business analysts to get it right the first time.
</li>
<li>
<b>The tunnel effect:</b> we live in a very fast evolving world and businesses need to adapt all the time. The software delivered after 2 years of heavy development will fulfill (hardly, but let's admit it) the requirements that were true two years ago, not anymore today.
</li>
<li>
<b>Drop of Quality to meet deadlines:</b> An engineering project is always late, always. Things are just a lot worst with software.
</li>
<li>
<b>Heightened tensions between teams:</b> The misunderstanding between teams leads to tensions, and it most of the time turns pretty ugly pretty quick.
</li>
</ul>
<p>
So again, some 20 years ago, Beck, Cunningham and Jeffries started to formalize some of the practices they were successfully using to address the uncertainties, the overwhelming abstraction and the misunderstandings inherent to software development. They formalized it as the <a href="https://en.wikipedia.org/wiki/Extreme_programming">eXtreme Programming</a> methodology.
</p>
<p>
A few years later, the same guys, with some other pretty well known Software Engineers, such as Alistair Cockburn and Martin Fowler, gathered together in a resort in Utah and wrote the <a href="https://en.wikipedia.org/wiki/Agile_software_development#The_Agile_Manifesto">Manifesto for Agile Software Development</a> in which they shared the essential principles and practices they were successfully using to address problems with more traditional and heavyweight software development methodologies.
</p>
<p>
Today, <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-landscape">Agility is a lot of things</a> and the set of principles of practices in the whole Agile family is very large. Unfortunately, most of them require a lot of experience to be understood and then applied successfully within an organization.
</p>
<p>
Unfortunately, the complexity of embracing a sound Agile Software Development Methodology and the required level of maturity a team has to have to benefit from its advantages is really completely underestimated.
<br>
I cannot remember the number of times I heard a team pretending it was an Agile team because it was doing a Stand up in the morning and deployed Jenkins to run the unit tests at every commit. But yeah, honestly I cannot blame them. It is actually difficult to understand Agile Principles and Practices when one never suffered from the very drawbacks and problems they are addressing.
</p>
<p>
I myself am not an agilist. Agility is not a passion, neither something that thrills me nor something that I love studying in my free time. Agility is to me simply a necessity. I discovered and applied Agile Principles and practices out of necessity and urgency, to address specific issues and problems I was facing with the way my teams were developing software.
</p>
<p>
The latest problem I focused on was Planning. Waterfall and RUP focus a lot on planning and are often mentioned to be superior to Agile methods when it comes to forecasting and planning.
<br>
I believe that this is true when Agility is embraced only incompletely. As a matter of fact, I believe that Agility leads to much better and much more reliable forecasts than traditional methods mostly because:
</p>
<ul>
<li>With Agility, it becomes easy to update and adapt Planning and forecasts to always match the evolving reality and the changes in direction and priority.</li>
<li>When embracing agility as a whole, the tools put in the hands of Managers and Executive are first much simpler and second more accurate than traditional planning tools.</li>
</ul>
<p>
In this article, I intend to present the fundamentals, the roles, the processes, the rituals and the values that I believe a team would need to embrace to achieve success down the line in Agile Software Development Management - Product Management, Team Management and Project Management - with the ultimate goal of making planning and forecasting as simple and efficient as it can be.
<br>
All of this is a reflection of the tools, principles and practices we have embraced or are introducing in my current company.
</p>
<!-- HERE -->
<p>
This article is available as a slideshare presentation here : <a href="https://www.slideshare.net/JrmeKehrli/agility-and-planning-tools-and-processes">https://www.slideshare.net/JrmeKehrli/agility-and-planning-tools-and-processes</a>.
</p>
<p>
Also, you can read a PDF version of this article here : <a href="https://www.niceideas.ch/Agile_Planning.pdf">https://www.niceideas.ch/Agile_Planning.pdf</a>.
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. Introduction</a></li>
<li><a href="#sec2">2. The Fundamentals</a>
<ul>
<li><a href="#sec21">2.1 eXtreme Programming</a></li>
<li><a href="#sec22">2.2 Scrum</a></li>
<li><a href="#sec23">2.3 DevOps</a></li>
<li><a href="#sec24">2.4 Lean Startup</a></li>
<li><a href="#sec25">2.5 Visual Management and Kanban</a>
<ul>
<li><a href="#sec251">2.5.1 Story Map</a></li>
<li><a href="#sec252">2.5.2 Product Backlog</a></li>
<li><a href="#sec253">2.5.3 Kanban Board</a></li>
<li><a href="#sec254">2.5.4 User Stories</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#sec3">3. Principles</a>
<ul>
<li><a href="#sec31">3.1 The Tools</a></li>
<li><a href="#sec32">3.2 The Organization</a>
<ul>
<li><a href="#sec321">3.2.1 Requires roles</a></li>
<li><a href="#sec322">3.2.2 Required committees and teams</a></li>
</ul>
</li>
<li><a href="#sec33">3.3 The Processes</a>
<ul>
<li><a href="#sec331">3.3.1 Design Process</a></li>
<li><a href="#sec332">3.3.2 Estimation Process</a></li>
<li><a href="#sec333">3.3.3 Product Kanban Board Maintenance Process</a></li>
<li><a href="#sec334">3.3.4 Story Map and Backlog synchronization Process</a></li>
<li><a href="#sec335">3.3.5 Forecasting</a></li>
<li><a href="#sec336">3.3.6 Development Process: Scrum</a></li>
</ul>
</li>
<li><a href="#sec34">3.4 The Rituals</a>
<ul>
<li><a href="#sec341">3.4.1 Product Management Committee</a></li>
<li><a href="#sec342">3.4.2 Architecture Committee</a></li>
<li><a href="#sec343">3.4.3 Sprint Management Committee</a></li>
<li><a href="#sec344">3.4.4 Development Team - Daily Scrum</a></li>
</ul>
</li>
<li><a href="#sec35">3.5 The Values</a></li>
</ul>
</li>
<li><a href="#sec4">4. Overview of the whole process</a></li>
<li><a href="#sec5">5. Return on Practices</a></li>
<li><a href="#sec5">6. Conclusion</a></li>
</ul>
<a name="sec1"></a>
<h2>1. Introduction </h2>
<p>
As stated in my abstract above, embracing sound Agile principles and applying relevant Agile practices is all but easy.
<br>
First, out of all the Agile methods available and described and the overwhelming set of practices and principles, an organization needs to understand which makes sense to it. Adopting a method, a set or principles or practices blindly, because the paper said it was good, or because the Scrum Master believes it is <i>state of the art</i> makes only little sense.
<br>
The set of methods described nowadays is pretty huge and unfortunately, each and every of these practices make sense whenever a team, an organization or a whole corporation suffers from a drawback or an issue it addresses or simply benefits from its advantages.
</p>
<p>
The whole set of Agile methods along with their principles and practices are brilliantly represented by Chris Web on the following infographic:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d028b3c0-5332-4ae7-a91a-dc21b7551299">
<img class="centered" style="width: 600px;" alt="The Agile Lansdcape from Deloitte" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d028b3c0-5332-4ae7-a91a-dc21b7551299" />
</a>
<div class="centered">
[Click to enlarge]<br>
(Source : Christopher Webb - LAST Conference 2016 Agile Landscape - <a href="https://www.slideshare.net/ChrisWebb6/last-conference-2016-agile-landscape-presentation-v1">https://www.slideshare.net/ChrisWebb6/last-conference-2016-agile-landscape-presentation-v1</a>)
</div>
</div>
<br>
<p>
Junior teams should go with a <i>base method</i> that makes sense to it, such as <a href="https://en.wikipedia.org/wiki/Scrum_(software_development)">Scrum</a> or <a href="https://en.wikipedia.org/wiki/Kanban_(development)">Kanban</a> <b>while remembering that none of it makes sense without a strict respect to the whole set of <a href="https://en.wikipedia.org/wiki/Extreme_programming">XP principles and practices</a></b>.
</p>
<p>
More experienced teams will likely come up with their own methodology, cleverly built from the principles and practices of several underlying methods.
</p>
<p>
Again, in my opinion <b>XP is the most fundamental building block on which all the rest is built</b>, not a method among others.
<br>
I often read papers online presenting XP as one Agile Software Development Method among others. My point of view is very different. I strongly believe - and experience everyday - that XP proposes the fundamental principles and practices on which are built all the other methods.
<br>
Without a thorough adoption of XP principles and practices, one cannot benefit from the full advantages of Agility. In addition, some principles and practices proposes by other methods such as DevOps, leverage on some XP principles and practices but never voids them.
</p>
<p>
When explaining this, I like to recover this schema I wrote a few years ago when I was doing consulting missions around Agility and Digital Transformation:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/61280dad-cbae-4bfe-bcaa-a75389e3b1e5">
<img class="centered" style="width: 550px; " alt="Pyramid of Agile Practices" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/61280dad-cbae-4bfe-bcaa-a75389e3b1e5" />
</a>
</div>
<br>
<p>
This reads as follows:
</p>
<ul>
<li>Without a proper understanding and adoption of eXtreme Programming values, principles and practices, moving towards <i>Agile Software Development</i> will be difficult.</li>
<li>Without Agility throughout the IT processes, both on the development side (Agile) and on the Production side (DevOps), embracing Lean Startup practices and raising Agility above the IT Department will be difficult.</li>
<li>Without a sound understanding of the Lean Startup Philosophy and practices and a company-wide Agile process (such as a company wide Kanban), transforming the company to an Agile Corporation will be difficult. </li>
<li>Finally, only Agile Corporations can really imagine successfully achieving a Digital Transformation</li>
</ul>
<p>
But then again, referring to Chris Webb's Agile Landscape, picking up the practices that make sense and have an added value in any context is the choice of every organization. Every different mature agile organization will use a slightly different set of practices than every other.
</p>
<p>
I will now be presenting the fundamental set of practices I deem important when it comes to successfully embracing Agile Planning and Agile Software Development.
</p>
<a name="sec2"></a>
<h2>2. The Fundamentals </h2>
<p>
The set of practices I deem essential to embrace <i>Agile Planning</i> comes from the following methods: XP, Scrum, Kanban, DevOps, Lean Startup and a lot of Visual Management tricks.
</p>
<a name="sec21"></a>
<h3>2.1 eXtreme Programming </h3>
<p>
e<b>X</b>treme <b>P</b>rogramming (XP) is the most fundamental software development method from the Agile tree that focuses on the implementation of an application, without neglecting the project management aspect. XP is suitable for small teams with changing needs. XP pushes to extreme levels simple principles.
</p>
<p>
The <i>eXtreme programming</i> method was invented by Kent Beck, Ward Cunningham, Ron Jeffries and Palleja Xavier during their work on the project <i>C3</i>. C3 was the calculation of compensation project at Chrysler.
<br>
Kent Beck, project manager in March 1996, began to refine the development method used on the project. It was officially born in October 1999 with Kent Beck's <i>Extreme Programming Explained</i> book.
</p>
<p>
In the book Extreme Programming Explained, the method is defined as:
</p>
<ul>
<li>An attempt to reconcile the human with productivity;</li>
<li>A mechanism to facilitate social change;</li>
<li>A way of improvement;</li>
<li>A style of development;</li>
<li>A discipline in the development of computer applications.</li>
</ul>
<p>
Its main goal is to reduce the costs of change. In traditional methods, needs are defined and often fixed at the start of the IT project, which increases the subsequent costs of modifications. XP is committed to making the project more flexible and open to change by introducing core <b>values</b>, <b>principles</b> and <b>practices</b>:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/660a5518-bd24-4f68-b50d-2efc2790f248">
<img class="centered" style="width: 700px; " alt="extreme Programming in a Nutshell" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/660a5518-bd24-4f68-b50d-2efc2790f248" />
</a>
</div>
<br>
<p>
The principles of this method are not new: they have existed in the software industry for decades and in management methods for even longer. The originality of the method is to push them to the extreme:
</p>
<ul>
<li>Since the code review is a good practice, it will be done permanently (by a binomial);</li>
<li>Since the tests are useful, they will be done systematically before each implementation;</li>
<li>Since the design is important, it will be done throughout the project (refactoring);</li>
<li>Since simplicity makes it possible to advance faster, we will always choose the simplest solution;</li>
<li>Since understanding is important, we will define and evolve metaphors together;</li>
<li>Since the integration of the modifications is crucial, we will do it several times a day;</li>
<li>Since the needs evolve rapidly, we will make cycles of development very rapid to adapt to the change.</li>
</ul>
<p>
The practices listed by the eXtreme Programming method form the fundamental Software Engineering Practices of Agility.
<br>
Interestingly, one cannot pick up a subset of these practices and believe that it should work. Kent Beck uses the following schematic to illustrate how these practices work together and depend on each others:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/ffb1d3a9-7fef-42f5-9a73-f51d6f97d2f8">
<img class="centered" style="width: 600px; " alt="XP Practices" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/ffb1d3a9-7fef-42f5-9a73-f51d6f97d2f8" />
</a>
</div>
<br>
<p>
All of this makes a lot of sense if you think of it: doing refactorings without TDD would be suicidal, Continuous Integration without TDD as well, Testing without simple design is complicated, Simple Design is enforced by TDD, etc.
</p>
<a name="sec22"></a>
<h3>2.2 Scrum </h3>
<p>
Scrum is a schematic organization of complex product development. It is defined by its creators as an <i>"iterative holistic framework that focuses on common goals by delivering productive and creative products of the highest possible value"</i>
</p>
<p>
This organizational scheme is based on the division of a project into time boxes, called <i>"sprints"</i>. A sprint can last between a few days and a month (with a preference for two weeks).
<br>
Each sprint starts with an estimate followed by operational planning. The sprint ends with a demonstration of what has been completed.
<br>
Before starting a new sprint, the team makes a retrospective. This technique analyzes the progress of the completed sprint, in order to improve its practices (Continuous Improvement / Kaizen).
<br>
The work flow of the development team is facilitated by its self-organization, so there should be no formal Project Manager but a Team Leader instead with a coaching role more than a management role.
</p>
<p>
The Scrum process can be represented as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/0d0237c0-20c6-47d2-8114-b85066c6e789">
<img class="centered" style="width: 700px;" alt="Overview of Scrum process" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/0d0237c0-20c6-47d2-8114-b85066c6e789" />
</a>
</div>
<br>
<p>
Some more information about scrum is available in <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-software-development-lessons-learned#sec13">a previous article here</a>.
</p>
<p>
<b>Working with Story Points</b>
</p>
<p>
In waterfall, managers determine a team member's workload capacity in terms of time. Managers ask selected developers to estimate how long they anticipate certain tasks will take and then assign work based on that team member's total available time. In waterfall, tests are done after coding by specific job titles rather than written in conjunction with the code.
<br>
The downsides of waterfall are well known: work is always late, there are always quality problems, some people are always waiting for other people, and there's always a last minute crunch to meet the deadline.
</p>
<p>
Scrum teams take a radically different approach.
</p>
<ul>
<li>First of all, entire Scrum teams, rather than individuals, take on the work. The whole team is responsible for each Product Backlog Item. The whole team is responsible for a tested product. There's no "my work" vs. "your work." So we focus on collective effort per Product Backlog Item rather than individual effort per task. </li>
<li>Second, Scrum teams prefer to compare items to each other, or estimate them in relative units rather than absolute time units. <b>Ultimately this produces better forecasts.</b> </li>
<li>Thirdly, Scrum teams break customer-visible requirements into the smallest possible stories, reducing risk dramatically. When there's too much work for 7 people, we organize into feature teams to eliminate dependencies.</li>
</ul>
<p>
<b>Planning poker</b>, also called Scrum poker, is a consensus-based, gamified technique for estimating, mostly used to estimate effort or relative size of development goals in software development.
<br>
In planning poker, members of the group make estimates by playing numbered cards face-down to the table, instead of speaking them aloud. The cards are revealed, and the estimates are then discussed. By hiding the figures in this way, the group can avoid the cognitive bias of anchoring, where the first number spoken aloud sets a precedent for subsequent estimates.
</p>
<p>
The cards in the deck have numbers on them. A typical deck has cards showing the Fibonacci sequence including a zero: 0, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89; other decks use similar progressions.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/54d9699f-11cf-40ee-bc89-f7ceeab60db2">
<img class="centered" style="width: 200px; " alt="Planning Poker" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/54d9699f-11cf-40ee-bc89-f7ceeab60db2" />
</a>
</div>
<br>
<p>
The reason to use planning poker is to avoid the influence of the other participants. If a number is spoken, it can sound like a suggestion and influence the other participants' sizing. Planning poker should force people to think independently and propose their numbers simultaneously. This is accomplished by requiring that all team members disclose their estimates simultaneously. Individuals show their cards at once, inspiring the term "<i>planning poker</i>."
</p>
<p>
In Scrum these numbers are called <b>Story Points</b> - or <b>SP</b>.
</p>
<a name="sec23"></a>
<h3>2.3 DevOps </h3>
<p>
DevOps is a methodology capturing the practices adopted from the very start by the web giants who had a unique opportunity as well as a strong requirement to invent new ways of working due to the very nature of their business: the need to evolve their systems at an unprecedented pace as well as extend them and their business sometimes on a daily basis.
</p>
<p>
DevOps is not a question of tools, or mastering chef or docker. DevOps is a methodology, a set of principles and practices that help both developers and operators reach their goals while maximizing value delivery to the customers or the users as well as the quality of these deliverables.
</p>
<p>
The problem comes from the fact that developers and operators - while both required by corporations with large IT departments - have very different objectives.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/472d552f-71ea-4640-a815-026e18cd865e">
<img class="centered" style="width: 450px; " alt="Wall of Confusion" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/472d552f-71ea-4640-a815-026e18cd865e" />
</a>
</div>
<br>
<p>
DevOps consists mostly in extending agile development practices by further streamlining the movement of software change thru the build, validate, deploy and delivery stages, while empowering cross-functional teams with full ownership of software applications - from design thru production support.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/77393bca-f284-443d-a48d-b1fadbc97789">
<img class="centered" style="width: 500px; " alt="DevOps" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/77393bca-f284-443d-a48d-b1fadbc97789" />
</a>
</div>
<br>
<p>
DevOps encourages <b>communication</b>, <b>collaboration</b>, <b>integration</b> and <b>automation</b> among software developers and IT operators in order to improve both the speed and quality of delivering software.
</p>
<p>
DevOps teams focus on standardizing development environments and automating delivery processes to improve delivery predictability, efficiency, security and maintainability. The DevOps ideals provide developers more control of the production environment and a better understanding of the production infrastructure.
<br>
DevOps encourages empowering teams with the autonomy to build, validate, deliver and support their own applications.
</p>
<p>
<b>So what are the core principles ?</b>
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/f37a1f42-86dd-4fef-a13e-e6b043c9c478">
<img class="centered" style="width: 450px; " alt="DevOps principles and practices" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/f37a1f42-86dd-4fef-a13e-e6b043c9c478" />
</a>
</div>
<br>
<p>
I presented these principles and practices in details in <a href="https://www.niceideas.ch/roller2/badtrash/entry/devops-explained">my previous article dedicated to devops.</a>.
</p>
<p>
DevOps is a revolution that aims at addressing the wall of confusion between development teams and operation teams in big corporations having large IT departments where these roles are traditionally well separated and isolated.
</p>
<a name="sec24"></a>
<h3>2.4 Lean Startup </h3>
<p>
Some years ago, Eric Ries, Steve Blank and others initiated <i>The Lean Startup</i> movement. The Lean Startup is a movement, an inspiration, a set of principles and practices that any entrepreneur initiating a startup would be well advised to follow.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/8b002746-dd21-4668-970e-dafcaa864567">
<img class="centered" style="width: 500px; " alt="Lean Startup Movement" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/8b002746-dd21-4668-970e-dafcaa864567" />
</a>
</div>
<br>
<p>
In my opinion, the most fundamental aspect of Lean Startup is the <i>Build-Measure-Learn</i> loop.
<br>
The fundamental activity of a startup is to turn ideas into products, measure how customers respond, and then learn whether to pivot or persevere. All successful startup processes should be geared to accelerate that feedback loop.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/4b4e8fe7-e841-46cc-996b-6b00df12b175">
<img class="centered" style="width: 650px; " alt="Build-Measure-Learn" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/4b4e8fe7-e841-46cc-996b-6b00df12b175" />
</a>
</div>
<br>
<p>
The five-part version of the Build-Measure-Learn schema helps us see that the real intent of building is to test <i>"ideas"</i> - not just to build blindly without an objective.
<br>
The need for "data" indicates that after we measure our experiments we'll use the data to further refine our learning. And the new learning will influence our next ideas. So we can see that the goal of Build-Measure-Learn isn't just to build things, the goal is to build things to validate or invalidate the initial idea.
</p>
<p>
<b>The four steps to the Epiphany</b>
</p>
<p>
Shortly put, Steve Blank proposes that companies need a <b>Customer Development process</b> that complements, or even in large portions replaces, their <i>Product Development Process</i>. The <i>Customer Development process</i> goes directly to the theory of <a href="https://en.wikipedia.org/wiki/Product/market_fit">Product/Market Fit</a>.
<br>
In "<i>The four steps to the Epiphany</i>", Steve Blank provides a roadmap for how to get to Product/Market Fit.
</p>
<p>
The four stages the <i>Customer Development Model</i> are: customer discovery, customer validation, customer creation, and company creation.
</p>
<ol>
<li><b>Customer discovery</b>: understanding customer problems and needs</li>
<li><b>Customer validation</b>: developing a sales model that can be replicated</li>
<li><b>Customer creation / Get new Customers</b>: creating and driving end user demand</li>
<li><b>Customer building / Company Creation</b>: transitioning from learning to executing</li>
</ol>
<p>
We can represent them as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/2f0ac2c3-13b4-4a24-a2fd-61ef32a66941">
<img class="centered" style="width: 750px; " alt="Lean Startup Practices" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/2f0ac2c3-13b4-4a24-a2fd-61ef32a66941" />
</a>
</div>
<br>
<p>
I added to the schema above the most essential principles and practices introduced and discussed by <i>the Lean Startup</i> approach.
</p>
<p>
I discussed these principles and practices in length in <a href="https://www.niceideas.ch/roller2/badtrash/entry/lean-startup-a-focus-on">a previous article on this blog</a>.
</p>
<a name="sec25"></a>
<h3>2.5 Visual Management and Kanban </h3>
<p>
Visual Management is an English terminology that combines several <i>Lean management</i> concepts centered on visual perception. The aim is to put the information and its context in order to make the work and the decision-making obvious.
</p>
<p>
Visual Management is an answer to the well known credo <i>"You can't manage what you can't see"</i>. It finds its root in <i>Obeya War Rooms</i>:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/01215572-e424-4fdf-8715-5cd4af1997bf">
<img class="centered" style="width: 700px; " alt="Toyota Obeya war Room" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/01215572-e424-4fdf-8715-5cd4af1997bf" />
</a><br>
<div class="centered">
(Source : <a href="http://alexsibaja.blogspot.ch/2014/08/obeya-war-room-powerful-visual.html">http://alexsibaja.blogspot.ch/2014/08/obeya-war-room-powerful-visual.html</a>)
</div>
</div>
<br>
<p>
Obeya (from Japanese <i>"large room"</i> or <i>"war room"</i>) refers to a form of project management used in many Asian companies, initially and including Toyota, and is a component of <i>lean manufacturing</i> and in particular the Toyota Production System.
<br>
During the product and process development, all individuals involved in managerial planning meet in a <i>"great room"</i> to speed communication and decision-making. This is intended to reduce <i>"departmental thinking"</i> and improve on methods like email and social networking. The Obeya can be understood as a team spirit improvement tool at an administrative level.
</p>
<p>
Nowadays, visual management is very much linked to <i>Lean Management</i> and Lean Startup, but IMHO it's really a field on its own. In the field of <b>Agile Planning</b>, I believe that Visual Management with sound tools and approaches is not optional.
<br>
At the end of the day, as we ill see, a good Project Management tool is a tool than enables anyone in the company to understand what is achievable in a given time or what time is required to deliver a given scope within a few minutes. And <b>nothing competes with Visual Tools</b> in this regards.
</p>
<p>
I will introduce here the fundamental tools I believe an Agile team should consider when it comes to Visual Management:
</p>
<a name="sec251"></a>
<h4>2.5.1 Story Map</h4>
<p>
The purpose of the Story Map is that arranging user stories into a helpful shape - a map - is usually deemed as most appropriate.
<br>
A Story Map is a visual management tool aimed at presenting the situation of the Software or the features to be implemented in a clear and graphical way. A Story Map is composed by user stories (see below).
</p>
<p>
A small story map may look like something like this:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d68d069a-2c6b-4771-b0d1-1f26fc3ec528">
<img class="centered" style="width: 900px; " alt="Story map Principle" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d68d069a-2c6b-4771-b0d1-1f26fc3ec528" />
</a>
</div>
<br>
<p>
At the top of the map are "big stories." We call them <b>themes</b>. A theme is sort of a big thing that people do - something that has lots of steps, and doesn't always have a precise workflow. A theme is a big category containing actual user stories grouped in <b>Epics</b>.
</p>
<p>
Epics are big user stories such as the one mentioned in example above. They usually involve a lot of development and cannot be considered as is in an actual product backlog. For this reason, Epics are split in a sub-set of <b>stories</b>, more precise and concrete that are candidate to be put in an actual product backlog.
</p>
<p>
I presented more information on Story Maps in <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-software-development-lessons-learned#sec32">a previous article here</a>.
<br>
For the moment, let's just remember that there is an important notion of <b>priority</b> on the vertical scale: the lower a story, the lesser its priority.
<br>
There is also a les obvious notion of priority horizontally: stories on the left should be implemented first since they have a greater value than the stories on the right, but all of that of course with respect of the more important vertical priority.
<br>
Long story short: the development team needs to implement all the stories of a row, from left to right, before it can consider the stories of the next row.
</p>
<p>
An pretty good and straightforward example of a Story Map related to an <i>email client application</i>:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d68d069a-2c6b-4771-b0d1-1f26fc3ec528">
<img class="centered" style="width: 750px; " alt="Story map Demo" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d68d069a-2c6b-4771-b0d1-1f26fc3ec528" />
</a>
</div>
<br>
<p>
And a real world example built during an real life Workshop:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/378db356-dcaf-44fb-b495-73571ecd8865">
<img class="centered" style="width: 750px; " alt="Story map Real World" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/378db356-dcaf-44fb-b495-73571ecd8865" />
</a><br>
<div class="centered">
(Copyright OCTO Technology / <i>Unfortunately I haven't been able to recover the source</i>)
</div>
</div>
<br>
<p>
A story Map is usually a visual tool, laid down on the wall of a shared meeting room or even the development team open-space. Distributed teams may consider digital tools but a physical, real and visual map on a wall is way better.
</p>
<a name="sec252"></a>
<h4>2.5.2 Product Backlog</h4>
<p>
The product backlog is the tool used by the Development tool to track the tasks to be implemented. These development tasks should be linked to a User Story on the Story Map.
<br>
As such, the product backlog should be seen as a much more detailed and technical version of the Story Map.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/7ad71564-b3e2-4d28-b507-d2e2f8c624e8">
<img class="centered" style="width: 600px; " alt="Product Backlog" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/7ad71564-b3e2-4d28-b507-d2e2f8c624e8" />
</a>
</div>
<br>
<p>
The product backlog shows the same releases than the Story Map. The development tasks in the current sprints should have a more detailed form than the development tasks not analyzed yet during <i>Sprint Planning</i>.
<br>
In a general way, the product Backlog should be kept synchronized with the Story Map and the reverse is true as well. Every User Story on the Map is broken down in development tasks in the Product Backlog and all tasks in the backlog should be attached to a User Story on the Map.
</p>
<p>
Their difference is as follows:
</p>
<ul>
<li><b>Story Map</b> : The Story Map is a management tool. It is a visual tool used by the <i>Product Management Team</i> to drive the high level development of the product and to defined releases and priorities. </li>
<li><b>Product Backlog</b> : The product Backlog is a technical project management tool, not a visual management tool. Its is usually supported by a digital tool (such as Jira or Redmine) and aims at organizing at a fine level the development team activities.</li>
</ul>
<p>
Some important constraints should be noted right away:
</p>
<ul>
<li>Each and every developer activity, not matter how quick and small, should be well identified by a development task in the product backlog.</li>
<li>Each and every development task should be linked to a User Story on the Story Map. I cannot stress enough how much this is important.</li>
</ul>
<a name="sec253"></a>
<h4>2.5.3 Kanban Board</h4>
<p>
Kanban is model for introducing change through incremental improvements. One can apply Kanban principles to any process one is already running.
</p>
<p>
In Kanban, one organizes the work on a Kanban board. The board has states as columns, which every work item passes through - from left to right. One pull work items along through the [<i>in progress</i>], [<i>testing</i>], [<i>ready for release</i>], and [<i>released</i>] columns (examples). And you may have various swim lanes - horizontal "<i>pipelines</i>" for different types of work.
<br>
The only management criteria introduced by Kanban is the so called "<i>Work In Progress</i>" or WIP. By managing WIP you can optimize flow of work items. Besides visualizing work on a Kanban board and monitoring WIP, nothing else needs to be changed to get started with Kanban.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/b42428b2-749a-4efa-850f-b5736da52171">
<img class="centered" style="width: 600px;" alt="Kanban board" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b42428b2-749a-4efa-850f-b5736da52171" />
</a>
</div>
<br>
<p>
Kanban boards can be mixed with Story Maps to follow the development of the tasks scheduled for next releases as far as their delivery on the current development version of the product.
<br>
In this case, the left-most column of the Kanban board becomes the Story Map containing the Stories to be developed while the right-most column of the Kanban board contains the User Stories identifying features already provided by the product.
<br>
I myself call such a mix of Story Map and Kanban a <b>Product Kanban Board</b>.
</p>
<p>
An real-world example of such a mix of Story Maps and Kanban boards could be as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/3948a797-359c-45c7-ba70-5494574772bc">
<img class="centered" style="width: 800px; " alt="Kanban and Story Maps - Visual Management" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/3948a797-359c-45c7-ba70-5494574772bc" />
</a>
</div>
<br>
<a name="sec254"></a>
<h4>2.5.4 User Stories</h4>
<p>
User stories are short, simple descriptions of a feature told from the perspective of the person who desires the new capability, usually a user or customer of the system.
</p>
<p>
They typically follow a simple template:
</p>
<div class="centering">
<div class="centered">
<i>As a <type of user>, I want <some goal> so that <some reason>.</i>
</div>
</div>
<br>
<p>
User stories are often written on sticky notes and arranged on walls or tables to facilitate planning and discussion.
<br>
As such, they strongly shift the focus from writing about features to discussing them. In fact, these discussions are more important than whatever text is written.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/35e0a8d6-521d-4775-818a-6df27e653f20">
<img class="centered" style="width: 200px; " alt="User Story" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/35e0a8d6-521d-4775-818a-6df27e653f20" />
</a>
</div>
<br>
<p>
It's the product owner's responsibility to make sure a product backlog of agile user stories exists, but that doesn't mean that the product owner is the one who writes them. Over the course of a good agile project, you should expect to have user story examples written by each team member.
<br>
Also, note that who writes a user story is far less important than who is involved in the discussions of it.
</p>
<p>
Some example stories for different application contexts:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d40939a0-bf16-46eb-9c78-b0e922d62fd0">
<img class="centered" style="width: 500px; " alt="Story Examples" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d40939a0-bf16-46eb-9c78-b0e922d62fd0" />
</a>
</div>
<br>
<p>
User Stories are used to track existing features as well as features to be developed on a mix of Story Map and Kanban, the <b>Product Kanban Board</b>.
</p>
<a name="sec3"></a>
<h2>3. Principles </h2>
<p>
Having covered the fundamentals, we will now go through the principles required for <b>Agile Planning</b> and see how the principles and practices introduced in the previous section should be used to achieve <i>reliable forecasts and planning</i> with Agile methodologies.
</p>
<p>
We should now discover:
</p>
<ul>
<li><b>The tools</b>, mostly visual management tools that the organization should adopt.</li>
<li><b>The Organization</b> to be put in place with required roles and committees.</li>
<li><b>The processes</b> that should be respected and that will lead to accurate estimations and forecasts.</li>
<li><b>The Rituals</b> supporting the processes.
<li><b>The Values</b> the team has to embrace to successfully run the processes and deploy the required practices.</li>
</ul>
<a name="sec31"></a>
<h3>3.1 The tools </h3>
<p>
The tools that the organization should adopt are as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/7cc8dcbc-ca33-4f38-bc26-2a3100a9b5c2">
<img class="centered" style="width: 900px; " alt="Agile Planning Tools" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/7cc8dcbc-ca33-4f38-bc26-2a3100a9b5c2" />
</a>
</div>
<br>
<p>
I believe that I introduced these tools in length in the section <a href="#sec25">2.5 Visual Management and Kanban</a> so I won't be adding a lot. We will see in the next section related to processes how these tools are used and how they complement each other by addressing different needs.
</p>
<a name="sec32"></a>
<h3>3.2 The Organization </h3>
<p>
The organization to put in place consists in identifying <b>roles</b> as well as <b>committees and teams</b>.
</p>
<a name="sec321"></a>
<h4>3.2.1 Required roles</h4>
<p>
The required roles are as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/8554fd26-4b6d-46fc-956b-7fba19b6135f">
<img class="centered" style="width: 500px; " alt="Required roles" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/8554fd26-4b6d-46fc-956b-7fba19b6135f" />
</a>
</div>
<br>
<ul>
<li>
<b>Team Leader</b> : The Team Leader animates the Team rituals (such as Sprint Planning, Sprint Retrospective, Daily scrum) and acts as a coach and a mentor to the development team. He is not a manager, he is a leader (Lead by Example, Management 3.0, etc.). He also represents the development team in other rituals (PMC).
<br>
At the end of the day, the Team Leader should not be made responsible for neither the team successes nor the team failures, the whole team should be accountable for this.
<br>
If the team leader is solely responsible for the Team's performance, then we will be tempted to shortcut quality or mess some rituals to speed up the pace and successfully respect some artificial deadline or else. When that is the case, the team requires a <i>Scrum Master</i> who should guarantee the Scrum rituals and processes are well respected.
<br>
In my opinion it makes a lot more sense to avoid such situation by making sure everyone in the team is accountable for the team performance and also responsible for the proper respect of the defined Agile processes and rituals. In this case, the Team Leader becomes an arbitrator, a facilitator, a coach and a support, not a manager. At the end of the day, management is too important to be left to managers ;-)
</li>
<li>
<b>Architect</b> : The Architect (or architects) should be the most experienced developer(s), the one(s) with the biggest technical and functional knowledge. There can be several architects, a lead architect, a technical architect, etc. This doesn't really matter.
<br>
The important thing is that the architect should be entitled to take architecture decision by still referring to the whole team as much as possible. The architect leads the Architecture Committee where architecture decisions are taken.
<br>
The architect, with the help of the tech leads, provides guidance and support to developers. he is also responsible to check the Code Quality, leading the code reviews, and ensure the sticking to Code conventions, etc.
</li>
<li>
<b>Tech Leads and Developers</b> : The tech leads and developers form the core of the development team, they eventually develop the software.
<br>
Tech Leads are coaches and supports to developers and represent them in the Architecture Committee.
</li>
<li>
<b>Product Owner </b>: The product Owner represents the stakeholders and drives priorities in good understanding with the market and customer needs. He is not a leader, he is an arbitrator and acts as the bridge between the business requirements and the development team.
<br>
I can only recommend the reader to watch the magnificent video <a href="https://www.youtube.com/watch?v=vkYEqz_MA5Y">"Agile Product Ownership in a Nutshell" from Henrik Kniberg</a>.
</li>
<li>
<b>Business representatives </b>: Business representatives (sales, customers, etc.) have to be involved in the Product Management Committee by the product Owner whenever required.
</li>
</ul>
<p>
<b>Why bother ?</b>
</p>
<p>
Roles are required mostly for two reasons : <b>efficiency</b> and <b>focus</b>:
</p>
<ul>
<li>
<b>Efficiency</b>: roles are required to avoid having the whole organization meeting all the time at every meeting for every possible concern.
</li>
<li>
<b>Focus</b>: every role owner should acts as required by his role and put himself in the right mindset for every ritual. Rituals are eventually a role playing game.
<br>
Roles are not functions ! We are not speaking hierarchy here, it's more a question of role play : when someone is assigned a role, his objective is to act and challenge the matters being discussed in correspondence with his role !
</li>
</ul>
<p>
As an important note, roles can well be shared. A same co-worker can have multiple roles if required, even though it would be better to avoid this.
</p>
<a name="sec322"></a>
<h4>3.2.2 Required Committees and teams</h4>
<p>
Required committees and teams are as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/71b8bcdd-d97a-46bd-a1c9-4942874f6481">
<img class="centered" style="width: 720px; " alt="Required Committees and teams" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/71b8bcdd-d97a-46bd-a1c9-4942874f6481" />
</a>
</div>
<br>
<ul>
<li>
<b>Development team</b> : The development team is responsible to develop the software. It is composed by Developers, Tech Leads, Architects and the Team Leader. At the end of the day, they're all developers and even the Team Leader should be able to spend a ratio of his time developing the Software (Lead by Example). Its essential ritual is the daily scrum every day.
</li>
<li>
<b>Product Management Committee</b> : The Product Management Committee - or PMC - is composed by the Development Team Leader, The Architect(s) and The Product Owner. The Product Owner should convoke business representatives as required. The PMC is responsible for identifying the new features to be added to the product and prioritize them. It should take place every week or every two weeks at most.
<br>
The PMC identifies new features as User Stories and Uses the Story Map to track them and prioritize them. Priorities are redefined and adapted as Stories Estimations (in Story Points) are refined. This process is explained later. Priorities should be set in respect to <b>the value</b> and <b>the cost</b> (in SP) of each and every story.
</li>
<li>
<b>Architecture Committee</b> : The Architecture Committee is composed by the Team Leader, The Architect(s), The Product Owner, the Tech Leads and representatives of the Development team.
<br>
The Architecture Committee is responsible to analyze user stories and define Development Tasks. Every story should be specified, designed and discussed. Screen mockups if applicable should be drawn, acceptance criteria agreed, etc.
<br>
Since the Architecture Committee is also responsible for estimating Stories, it's important that representatives of the Development Team, not only the Tech Leads and the Architects, but simple developers as well, take part in it. Ideally, there should be a rotation and at every meeting a different couple of developers should be convoked. This is required to have everyone agreeing on the estimations.
<br>
The Architecture Committee should take place every week or every two weeks at most as well and ideally not long after the PMC.
</li>
<li>
<b>Sprint Management Committee</b> : The Sprint Management Committee is basically composed by the Development Team plus the Product Owner.
<br>
During Sprint Planning, the Sprint Management Committee discusses the implementation concerns of the tasks specified by the Architecture Committee and challenge the estimations if required. The Development Tasks defined by the Architecture Committee are detailed as much as possible.
<br>
During Sprint retrospective, the Sprint Management Committee discussed the issues and drawbacks encountered during former sprint and agrees on an action plan to address them.
</li>
</ul>
<a name="sec33"></a>
<h3>3.3 The Processes </h3>
<p>
I will be presenting now the various processes that are required to achieve the ultimate goal of Agile Planning : reliable forecasts and planning.
</p>
<a name="sec331"></a>
<h4>3.3.1 Design Process</h4>
<p>
The <i>Design process</i> consists in breaking a <i>User Story</i> identified by the PMC into <i>Development tasks</i> that developers can understand and work on.
<br>
It can be illustrated as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/623b344c-56ab-4842-9706-413f07534944">
<img class="centered" style="width: 900px; " alt="Design Process" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/623b344c-56ab-4842-9706-413f07534944" />
</a>
</div>
<br>
<p>
<b>A. Identification of User Stories</b>
</p>
<p>
The <b>PMC</b> produces a <b>User Story</b> laid down on the Story Map.
</p>
<p>
<b>B. From User Stories to Development Stories</b>
</p>
<p>
The <b>Architecture Committee</b> analyzes every new story and for each of them it creates a <b>Development Story</b> on the Product Backlog.
</p>
<p>
Such a <i>Development Story</i> is not anymore a simple post-it in a Story Map, it is a <i>digital User story</i> created in the backlog management tool such as Jira or Redmine. The Development Story is specified and design. It should contain:
</p>
<ul>
<li>The initial user story from the Story Map as it was expressed at that time.</li>
<li>A complete description of the purpose and intents of the feature.</li>
<li>A complete description of the expected behaviour from all perspectives: user, system, etc.</li>
<li>Mock-ups of screens and front-end behaviours as well as validations to be performed on the front-end.</li>
<li>A precise and exhaustive list and description of all business rules.</li>
<li>A list and description of the data to be manipulated.</li>
<li>Several examples of source data or actions and expected results.</li>
<li>Acceptance criteria (functional and non-functional) and a complete testing procedure.</li>
<li>The list of documents - technical and functional - that will need to be updated or adapted and how.</li>
</ul>
<p>
<b>C. From Development Stories to Development Tasks</b>
</p>
<p>
The <b> Architecture Committee</b> also breaks the <b>Development Story</b> down in several <b>Development Tasks</b>.
<br>
Development tasks should be split by logical or functional units or layers. For instance, one task could be related to the GUI while another one could be related to the database changes, etc. But if it is possible, it is always better not to split them by layer but rather vertically by sub-feature.
<br>
What should never be done is splitting a Story in tasks by the type of job, for instance development, unit test, integration tests. That should never ever be done. A developer, or a couple of developers should always implement a sub-feature entirely, with all the required tests, functional tests, migration scripts, documentation updates, etc.
</p>
<p>
<b>D. From Development Tasks to Detailed Tasks</b>
</p>
<p>
The <b>Sprint Management Committee</b>, during <i>Sprint Planning</i> recovers all these <b>Development Tasks</b> and analyzes them further.
</p>
<p>
The questions to be answered at this time are:
</p>
<ul>
<li>Are all the information provided by the Architecture Committee clear enough or are some precisions required?</li>
<li>Is there any unforeseen impact on other parts of the software?</li>
<li>Is there any tool or specific environment setup or configuration required to implement these tasks?</li>
<li>etc.</li>
</ul>
<p>
Specifically the developers that were not present at Architecture Committee when a task has been designed should challenge it and make sure they understand not only what need to be done but really also how to do it precisely.
<br>
At this stage, the new findings should lead to a refinement of the initial estimations agreed by the Architecture Committee.
</p>
<p>
In addition, at this stage the Development team discovers about the secondary aspects identified by the Architecture Committee such as documentation to be updated or adapted, automated tests to be implemented or adapted, etc. and mentions in a detailed way each and every step to be done in the Detailed tasks.
</p>
<a name="sec332"></a>
<h4>3.3.2 Estimation Process</h4>
<p>
What we want eventually, is a Story Map containing estimations for all the Stories that have been analyzed by the Architecture Committee.
<br>
The result we want to achieve here can be represented as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/c1d1253c-b607-48c6-8940-d8a9197f42c4">
<img class="centered" style="width: 600px; " alt="Story Map with estimations" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/c1d1253c-b607-48c6-8940-d8a9197f42c4" />
</a>
</div>
<br>
<p>
Each and every story that has been broken down by the Architecture Committee and created in the Product Backlog is clearly identified: it has an estimation expressed as a total number of Story Points.
<br>
That number corresponds to the total of the estimations in SP of the individual <i>Development Tasks</i> underneath.
</p>
<p>
<b>A. Initial Estimations</b>
</p>
<p>
At this stage, The <b>Architecture Committee</b> is in charge of the Initial Estimations.
<br>
After a Story has been broken down in tasks, each and every of these tasks is estimated by the Committee using the <i>Planning Poker</i> approach.
<br>
The sum of the estimations of every individual tasks is reported on both the Development Story (Product Backlog) and the User Story (Story Map):
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/3d966d2e-c07e-4f8a-96c0-d7215235cff0">
<img class="centered" style="width: 900px; " alt="Initial Estimations" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/3d966d2e-c07e-4f8a-96c0-d7215235cff0" />
</a>
</div>
<br>
<p>
<b>B. Refined Estimations</b>
</p>
<p>
When the <b>Sprint Management Committee</b> recovers the <b>Development Tasks</b> to refine them, there might be new impacts discovered, new unforeseen refactorings required, etc.
</p>
<p>
The Sprint Management Committee should challenge the initial estimations with their new findings and adapt the estimations accordingly.
<br>
Again, these new <i>Refined Estimations</i> should be reported on both the Development Story (Product Backlog) and the User Story (Story Map):
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/dc66f36c-1f9c-4975-8b08-37720b496b3d">
<img class="centered" style="width: 900px; " alt="Refined Estimations" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/dc66f36c-1f9c-4975-8b08-37720b496b3d" />
</a>
</div>
<br>
<p>
<b>C. Final Estimations</b>
</p>
<p>
Eventually, during the sprint, it can happen that a developer discovers that a task will take a bigger time than expected, or, in the contrary, a much shorter time.
<br>
Reporting such changes in estimations at this very late stage is maybe not important for Scrum, since the sprint is already filled, but it's important for both the Sprint Management Committee and the Architecture Committee to be notified about them in order to improve the way they do estimations.
<br>
As part of Continuous Improvement (Kaizen), the Architecture Committee needs to identify where the gap comes from and try to have more accurate estimations next time.
</p>
<p>
So even at this stage, when a developer discovers gaps or shortcut, it's important that any impact in terms of estimation is reported as far as to the Story Map:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/6159e36f-dcd8-4d03-af65-419e562badfe">
<img class="centered" style="width: 900px; " alt="Final Estimations" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/6159e36f-dcd8-4d03-af65-419e562badfe" />
</a>
</div>
<br>
<p>
<b>Why bother ?</b>
</p>
<p>
The management tool is the story map, not the product backlog. The product backlog is a technical tool to organize the development activities. It's not a management tool.
</p>
<p>
The Product Management Committee should be able to decide about priorities using solely the Story Map. In addition, it should be possible to forecast a delivery date using solely the Story Map.
<br>
For this reason, the Story Map should contain as up to date as possible estimations.
</p>
<p>
Everyone in the company should be able to take is little calculator, go in front of the story map and know precisely when a task will be delivered.
<br>
We'll see how soon !
</p>
<p>
<b>What about updating estimations after the task has been completed and we know how much time we spent on it ?</b>
</p>
<p>
One needs to understand what we're trying to achieve here.
</p>
<p>
We're trying to continuously improve our ability to come up with accurate and reliable estimations based on the information we have. When we estimate tasks at <i>ARCHCOM</i> or <i>Sprint Planning</i>, we only have analysis information at our disposal, we have no clue about any post-implementation information such as the actual time that will be spent on the task.
<br>
As such, while it is very important to improve our ability to estimate using analysis information (as done at ARCHCOM), it makes no sense to update estimations after implementation since actual implementation time is an information we will never have before implementing the task.
</p>
<p>
Again, we want to improve our ability to estimate using the information we have. And actual implementation time is an information we don't have so it's useless in regards to improving the estimation process and as such doesn't trigger any estimation update.
</p>
<p>
In addition, the estimation process is a comparison game, not an evaluation game (or less). An Estimation in SP should have no clear relationship with actual implementation time, for many reasons, among them the fact the different developers have different capacity. A 10 SP task is always a 10 SP task, for every developer. But it may well represent 4 days of work for a junior developer and 2 days of work for a senior developer.
<br>
This aspect is a very important reason behind the rationality to think in terms of SP instead of Man/Days. And of course SP should be a measure of the whole team capacity, not individuals.
</p>
<p>
This is why we don't bother updating estimations after actual implementation. Nevertheless, we should still use that knowledge to improve our estimations, but actually trying to update the estimation in SP makes no sense.
</p>
<a name="sec333"></a>
<h4>3.3.3 Product Kanban Board Maintenance Process</h4>
<p>
Maintaining the <i>Product Kanban Board</i> (Mix of Story Map and Kanban Board) as up to date as possible with latest activities of the development team as well as the latest estimations is important.
<br>
Again, The <i>Product Kanban Board</i> is the only tool that should be required by the Product Management Committee to come up with estimations and forecasts.
</p>
<p>
We will now see how this <i>Product Kanban Board</i> should be maintained throughout the sprints and how it is used.
</p>
<p>
<b>A. Initial Stage: before the first sprint of the nest release</b>
</p>
<p>
We start with a Board of the following shape:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a4abc1b6-d804-4a30-95ab-be9ac45ae7b2">
<img class="centered" style="width: 900px; " alt="Product Kanban - Initial Stage" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a4abc1b6-d804-4a30-95ab-be9ac45ae7b2" />
</a>
</div>
<br>
<p>
The boxes in blue indicate how a <i>User Story</i> is moved across the board when it advances in the analysis and development process:
</p>
<ul>
<li>First, when the Architecture Committee has done analyzing and breaking down the Story, the estimation it came up with is reported on the User Story in the violet pellet.</li>
<li>Then, A Story is moved to <b>Implementation / Doing</b> when a first of its development tasks is being implemented in the current sprint</li>
<li>It is moved to <b>Implementation / Done</b> when the last of its development tasks is done being implemented (meaning completely done : with automated tests, IT tests, etc. At this stage it's simply waiting the next <i>continuous deliver</i> build to be available on Test environment for acceptance tests.</li>
<li>When the <i>Continuous Delivery</i> build has been executed, the Story is moved to <b>Testing</b>.</li>
<li>When the product Owner either tested the Story (or delegated such tests) and accepts the results, the Story is moved to <b>Done</b></li>
</ul>
<p>
The Story Map on the left is a pretty standard Story Map, where releases are identified.
</p>
<p>
The Story Map on the right, on the other hand, drops the notion of releases completely. It identifies the features as they are available as a whole in the current development version of the product, regardless of both past releases and releases to come.
<br>
A story identifying a new feature is simply added to it to capture the fact that the feature is now available on the development version.
<br>
On the other hand, a story identifying a modification of an existing feature should be <b>merged with the original story</b>, potentially leading to a new story, corresponding to the new way of expressing the feature.
</p>
<p>
<b>B. During the first sprint</b>
</p>
<p>
During the first sprint after this initial stage, the Kanban board in the middle identifies the Stories that are being worked on and their status:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/4e2f2347-c769-4552-a72f-eaca4f9000c4">
<img class="centered" style="width: 900px; " alt="Product Kanban - First Sprint" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/4e2f2347-c769-4552-a72f-eaca4f9000c4" />
</a>
</div>
<br>
<p>
<b>C. During the second sprint</b>
</p>
<p>
After first sprint, developed stories are put on the Story Map on the right and a next set of Stories are being developed:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/315440d4-522c-49ca-b048-224d13efd5fc">
<img class="centered" style="width: 900px; " alt="Product Kanban - Next Sprint" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/315440d4-522c-49ca-b048-224d13efd5fc" />
</a>
</div>
<br>
<p>
<b>D. After the first release</b>
</p>
<p>
After the first release, we can see that all the tasks from the first release of the Story Map on the left have been moved to the Story Map on the right.
<br>
The Story Map on the left is adapted and the next releases are shifted up.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/655b8fc2-cb7d-49df-8097-cd8894671257">
<img class="centered" style="width: 900px; " alt="Product Kanban - After first release" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/655b8fc2-cb7d-49df-8097-cd8894671257" />
</a>
</div>
<br>
<p>
Notes:
</p>
<ul>
<li>Actual releases <b>will</b> differ: we can release potentially at every end of Sprint. Releases identified on the Story Map on the left will likely be broken down in smaller releases.</li>
<li>Again, one should embrace <b>Continuous Delivery</b>: The development Team releases at every end of sprint. Making it a customer release is a Product Management Decision</li>
<li>One should consider <a href="https://www.niceideas.ch/roller2/badtrash/entry/devops-explained#sec35">feature flipping</a> in order not to compromise a potential release with a story that would not have been completely implemented in one sprint.</li>
</ul>
<p>
<b>E. No notion of release in <i>Done</i></b> (Right Story Map)
</p>
<p>
The Story Map on the right shouldn't have any notion of releases. It represents the Product as is the current development version and it makes no sense anymore remembering there which task has been developed in which release.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/e521062a-cf09-4d17-b44f-70fcf75957ff">
<img class="centered" style="width: 900px; " alt="Product Kanban - No releases in done" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/e521062a-cf09-4d17-b44f-70fcf75957ff" />
</a>
</div>
<br>
<p>
Also, User stories on the right may need to be merged whenever they relate to the same feature.
</p>
<a name="sec334"></a>
<h4>3.3.4 Story Map and Backlog synchronization Process</h4>
<p>
The priorities of the <i>Development Tasks</i> on the <i>Product Backlog</i> should match and follow the priorities of the <i>User Stories</i> on the <i>Story Map</i>.
</p>
<p>
The Story Map drives the priorities. The Product Management Committee uses estimations and updates provided by the Architecture Committee and the Development Team to adapt the priorities of the stories on the Story Map and move them accordingly.
<br>
When a story priority changes, the priorities of the corresponding Development Tasks on the Product Backlog should be changed in order to reflect the new priority of the User Story.
</p>
<p>
The principle is as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/c340b3b7-042e-4dc6-a5d9-fe232b3ffbc2">
<img class="centered" style="width: 900px; " alt="Backlog and Story Map synchronization principle" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/c340b3b7-042e-4dc6-a5d9-fe232b3ffbc2" />
</a>
</div>
<br>
<p>
In terms of process, things occur this way:
</p>
<ol>
<li>The <i>Architecture Committee</i> takes Stories created by the Product Management Committee, designs them and estimates them.</li>
<li>The <i>Product Management Committee</i> learns about Stories Estimations and re-prioritizes the Story Map accordingly</li>
<li>The <i>Architecture Committee</i> synchronizes the priorities of the corresponding Development Tasks.</li>
</ol>
<p>
This can be represented this way:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/f4d13c78-7e20-44e0-87f5-8f2c186c7b75">
<img class="centered" style="width: 900px; " alt="Backlog and Story Map synchronization process" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/f4d13c78-7e20-44e0-87f5-8f2c186c7b75" />
</a>
</div>
<br>
<p>
Let's see now how all of this is used to be able to achieve its ultimate objective : reliable planning and forecasting.
</p>
<a name="sec335"></a>
<h4>3.3.5 Forecasting</h4>
<p>
So ... forecasting, finally.
<br>
At the end of the day, pretty much everything I have presented above, all these tools, charts and processes are deployed towards this ultimate objective: doing planning and being able to produce accurate forecasts.
</p>
<p>
If one respects well the processes presented above and use the tools the right ways, one should end up with the Story Map presented in <a href="#sec332">3.3.2 Estimation Process</a>, hence Stories that hold the indication of a pretty accurate estimation in Story Points.
</p>
<p>
In addition, a story map holds an important notion of priority: the development team needs to implement all the stories of a row, from left to right, before it can consider the stories of the next row.
</p>
<p>
So how does one know when a story will be implemented by the development team? The answer is simple: when all stories of the previous rows as well as all stories on the left on the same row are implemented.
<br>
From there, calculating the amount of Story Points to be developed before a specific story can be implemented is straightforward:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/fa67f626-9e00-4454-86d8-ea938003c1d0">
<img class="centered" style="width: 600px; " alt="Forecasting" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/fa67f626-9e00-4454-86d8-ea938003c1d0" />
</a>
</div>
<br>
<p>
Recovering the example introduced in <a href="#sec332">3.3.2 Estimation Process</a>, if we want to know when the Story with the blue box around it, we have first to know how many story points have to be implemented first, 1750 SP in this example.
</p>
<p>
Based on this, we know that this story will be <b>delivered once all the stories before it will be implemented plus this story as well</b>, hence 1750 + 100 SP = 1850 SP.
</p>
<p>
<b>Estimating a delivery date</b>
</p>
<p>
In order to estimate a delivery date for that story, we need to know how much time is required to deliver these 1850 SP.
<br>
Here comes the notion of Sprint capacity, or rather Sprint velocity. Strictly speaking, Agilists speak of capacity when reasoning of man days and Sprint velocity when reasoning in Story Points.
<br>
I myself use Sprint capacity for both cases.
</p>
<p>
Computing Sprint velocity requires to have all the practices described in introduction in place for several Sprints. I will come back on practices in the next chapters so I'm leaving them aside for now.
<br>
If the Agile Team is mature in regards to its practices, it can compute the Sprint Capacity be looking at the range of Story Points achieved during 5 last sprints:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/91f84cf5-d6b9-4769-ae04-f4c739c3f3da">
<img class="centered" style="width: 800px;" alt="Uncertainty adressed by range of SP" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/91f84cf5-d6b9-4769-ae04-f4c739c3f3da" />
</a>
</div>
<br>
<p>
We don't use the most extreme, minimum and maximum values. Extreme values most of the time explain themselves by external factors: people get sick, leaves on holidays, tasks are sometimes finished in next sprint, etc.
<br>
Instead, out of five sprints, we'll use the second maximum value and the last-but-one value.
</p>
<p>
We use this range, and not a single value of average or median, to address a fundamental aspect of software engineering: the uncertainty.
<br>
The range gives us a lower value and an upper value which we will use as follows.
</p>
<ul>
<li>
<b>In case of <i>fixed time</i></b>, when we have a fixed delivery date, the lower and upper values give us the minimum or maximum set of features we can have implemented at that date.
</li>
<li>
<b>In case of <i>fixed scope</i></b>, when we have to release a version of the software with a given set of features, the lower and upper values will give us the earliest date and the latest date at which we can release.
</li>
</ul>
<p>
As a sidenote, when we count Story Points implemented in a sprint, we should focus on developer tasks, not User Stories, since User Stories are too coarse grained.
<br>
A User story can well take several sprints to be completed. A developer task within one of these stories should not. Tasks should be designed in such a way that they are small enough to always fit a sprint.
</p>
<p>
Recovering the example above, let's imagine we are want to achieve a fixed scope, we want to know, using the Story Map as it is, how much time will be required to implement these 1850 Story Points.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/433558bd-c825-492f-9cd2-c83c2fba059d">
<img class="centered" style="width: 700px; " alt="Burndown Forecast" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/433558bd-c825-492f-9cd2-c83c2fba059d" />
</a>
</div>
<br>
<ul>
<li>Using the lower limit of 128 SP per sprint, it would take us 15 sprints to complete the scope, hence 30 weeks or 6.7 months</li>
<li>Using the upper limit of 138 SP per sprint, it would take us 14 sprints to complete the scope, hence 28 weeks or 6.2 months</li>
</ul>
<p>
Based on this, the PMC or the Product Owner can communicate to the stakeholders that the feature would be release not before 6 months but before 7 months.
</p>
<a name="sec336"></a>
<h4>3.3.6 Development process: Scrum</h4>
<p>
I said enough about Scrum in both <a href="#sec22">this article</a> and <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-software-development-lessons-learned#sec13">my previous article</a>.
<br>
Let me just introduce this chart that does a great job in introducing the notion of <b>Product Increment</b> as a shippable version of the product since we adopt Continuous Delivery:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/b67a03fb-8033-4b6a-9f9f-648a6b4edd96">
<img class="centered" style="width: 700px;" alt="Scrum Process" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b67a03fb-8033-4b6a-9f9f-648a6b4edd96" />
</a>
</div>
<br>
<p>
This allows me to present the last tool I mentioned in the introduction of this sprint, which is the <b>Sprint Kanban Board</b>:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/66b435d4-08e1-4391-84cb-c4f7dd0b49f5">
<img class="centered" style="width: 700px; " alt="Sprint Kanban" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/66b435d4-08e1-4391-84cb-c4f7dd0b49f5" />
</a>
</div>
<br>
<p>
The sprint Kanban board is used to track the progress of tasks within the sprint and enables to organize developer activities.
</p>
<p>
Some people use extensively <i>burndown charts</i> to track the proper progress of a sprint or the product backlog towards a specific release as a whole. I myself never find it so useful. I really get all I want to know about how a release or a specific sprint is doing by using the Product Backlog, the Product Kanban Board or the Sprint Kanban.
</p>
<p>
<b>Commitment</b>
</p>
<p>
A very important aspect when it comes to the Scrum sprint, which is as well an important value the team has to commit to, is the commitment of the team to close the Sprint scope at the end of the Sprint, whatever it takes.
</p>
<p>
Postponing tasks from sprint to sprint is a nightmare to manage and ruins forecasting. The development team has to progress estimating tasks, trying to be as accurate as possible, and be realistic when planning the sprint and feeding.
<br>
To be honest it takes quite some sprints to find right way both for estimating tasks and feeding the backlog. But after these first initiation sprints, the development team <b> has to commit to the sprint scope at all cost, whatever it takes</b>.
</p>
<p>
The only answer when discovering at the end of the sprint that the scope won't be completed without overtime is first to work as much as is required to complete it and second to identify how to improve the estimation process and the sprint feeding process to avoid this situation in the future (Kaizen burst).
<br>
It's is never an acceptable answer to simply postpone some tasks to the next sprint.
</p>
<a name="sec34"></a>
<h3>3.4 The Rituals </h3>
<p>
Rituals of the various teams are as follows.
<br>
Committees are rituals by themselves, the difference between a team and a committee is that a committee gathers solely for a specific ritual
</p>
<a name="sec341"></a>
<h4>3.4.1 Product Management Committee</h4>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/cbd5a9a0-6786-43d7-9731-066f834c4770">
<img class="centered" style="width: 200px; " alt="Product Management Committee" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/cbd5a9a0-6786-43d7-9731-066f834c4770" />
</a>
</div>
<br>
<p>
The <i>Product Management Committee</i> gather every X weeks. It really depends of the corporation, the size of the team, the rate at which new functional requirements appear. Every few 2 weeks should be sufficient in general, otherwise the frequency can increase as far as every week.
</p>
<p>
The duties of the <i>Product Management Committee</i> are as follows:
</p>
<p>
<b>Story Mapping</b>
</p>
<ul>
<li>Identification of new needs and requirements (also technical and technological !)</li>
<li>Breakdown of these requirements in User Stories</li>
<li><i>"Guessing"</i> of an Initial Priority of a User Story based on Value (and foreseen size)</li>
</ul>
<p>
<b>Maintenance (update) of Priorities</b>
</p>
<ul>
<li>Setting of Actual Priorities based on Estimations from Architecture Committee</li>
<li>Review of priorities of Whole Story Map after update of estimations
<ul>
<li>From Sprint Management Committee</li>
<li>From Development Team</li>
</ul>
</li>
</ul>
<a name="sec342"></a>
<h4>3.4.2 Architecture Committee</h4>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/be8600bc-85ae-48d9-b6d3-a81a2d35554d">
<img class="centered" style="width: 200px; " alt="Architecture Committee" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/be8600bc-85ae-48d9-b6d3-a81a2d35554d" />
</a>
</div>
<br>
<p>
The <i>Architecture Committee</i> also gather every X weeks. It should meet at least few minutes (coffee break) but not more than one or two days after <i>Product Management Committee</i>.
<br>
The Architecture Committee recovers the last User Stories designed at PMC and synchronizes the Product Backlog with the Story Map. Stories are specified, designed and broken downs in Development Tasks.
</p>
<p>
The duties of the <i>Architecture Committee</i> are as follows:
</p>
<p>
<b>Specification and Design of User Stories</b>
</p>
<ul>
<li>Specification of functional and non-functional requirements</li>
<li>Identification of business rules</li>
<li>Identification of Acceptance criteria</li>
<li>Design of GUI </li>
<li>Architecture and Design of Software</li>
<li>Identification of documents and procedures to be updated / adapted</li>
<li>Identification of automated tests to be implemented</li>
</ul>
<p>
<b>Estimation of User Stories</b>
</p>
<ul>
<li>Breakdown in individual Development Tasks
<ul>
<li>This needs to be done sufficiently in advance</li>
</ul>
</li>
<li>Estimation of Development Tasks</li>
<li>Computing of total Estimation and reporting on User Story</li>
<li>Continuous Improvement: understanding of gaps in estimation after notification of Sprint Committee and how to improve</li>
</ul>
<p>
<b>Software Architecture</b>
</p>
<ul>
<li>Identification and maintenance of Coding Standards and Architecture Standards</li>
<li>Review of ad'hoc architecture topics</li>
</ul>
<a name="sec343"></a>
<h4>3.4.3 Sprint Management Committee</h4>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/df862c29-1b79-4c8a-b253-7a900ffbda37">
<img class="centered" style="width: 200px; " alt="Sprint Mangement Committee" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/df862c29-1b79-4c8a-b253-7a900ffbda37" />
</a>
</div>
<br>
<p>
The <i>Sprint Management Committee</i> gathers at every beginning and end of sprint.
<br>
A sprint starts with the <i>Sprint Planning</i> and ends with <i>Sprint Demo</i> and <i>Sprint Retrospective</i>:
</p>
<p>
<b>Sprint Planning</b>
</p>
<ul>
<li>Discuss Development Tasks to ensure whole team has a clear view of what needs to be done → Detailed Tasks</li>
<li><b>Definition of done</b>: list exhaustively the form of automated tests to be implemented as well as the documentation to be updated and the scope of these changes.</li>
<li>Review and challenge estimations of Detailed Tasks. Update estimation of User Story accordingly</li>
<li>Feed the Sprint Backlog with such Detailed Tasks until Sprint Capacity is reached</li>
</ul>
<p>
<b>Sprint Retro</b>
</p>
<ul>
<li>Review Tasks not completed and create task identifying GAP for next Sprint. Update estimations.</li>
<li>Review SP achieved during sprint and review Sprint Capacity</li>
<li>Discuss issues encountered during Sprint and identify action points. Update processes and rituals accordingly</li>
<li>Continuous Improvement: understanding of gaps in tasks and estimations and how to improve</li>
</ul>
<p>
<b>Sprint Demo</b>
</p>
<ul>
<li>End of Sprint / really optional with Continuous Delivery and Continuous Acceptance Tests</li>
<li>Present sprint developments and integrate feedback. Create new tasks and update estimations.</li>
</ul>
<a name="sec344"></a>
<h4>3.4.4 Development Team - Daily Scrum</h4>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/2895b34e-0cd6-4e6c-9a85-4b94723b8f60">
<img class="centered" style="width: 160px; " alt="Development Team" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/2895b34e-0cd6-4e6c-9a85-4b94723b8f60" />
</a>
</div>
<br>
<p>
The daily scrum happens every day, ideally early in the moment, at the time all the team is in the office.
<br>
The scope of the daily scrum is as follows:
</p>
<p>
<b>Round table - every team member presents:</b>
</p>
<ul>
<li>Past or current development task</li>
<li>Status on that task and precise progress</li>
<li>Next steps</li>
<li>Next task if former is completed</li>
<li>Identification of unforeseen GAPS and adaptation of estimations</li>
</ul>
<p>
<b>Identification of challenges, issues and support needs</b>
</p>
<ul>
<li>Scheduling of ad'hoc meeting and required attendees to discuss specific issues</li>
</ul>
<a name="sec35"></a>
<h3>3.5 The Values </h3>
<p>
<b>Sticking rituals, respecting principles and enforcing practices is difficult.</b>
</p>
<ul>
<li>It's difficult to ensure and behaves in such a way that breaking the build (failing tests) is an exception.</li>
<li>It's difficult to respect the boyscout rule.</li>
<li>It's a lot more difficult to design things carefully and stick to the KISS principle.</li>
<li>It's difficult and a lot of work to keep the Story Map and Product Backlog in sync and up-to date with the reality.</li>
<li>It's difficult to stick to the TDD approach.</li>
<li>It's difficult not to squeeze the Kaizen phase at the end of every meeting and being objective when it comes to analyzing strengths and weaknesses.</li>
</ul>
<p>
All of this make two Agile values especially important: <b>Discipline</b> and <b>courage</b>.
<br>
Both are utmost important and essential to address these difficulties.
</p>
<p>
Sticking to the Scrum rituals, enforcing TDD and other XP principles and practices require courage and discipline. It also requires a lot of discipline to Maintain and synchronize the Product Backlog and the Story Map.
<br>
Updating the estimations of the User Stories continuously as the understanding of the work to be done progresses also takes a lot of discipline.
</p>
<p>
Finally, discipline and courage are enforced by a strict definition of the processes and rituals and a proper maintenance of this definition as the culture and practices evolve.
<br>
At the end of the day, defining these committees and rituals is all about that. Why are all these committees / teams / rituals required if eventually a person can have several roles? Because they enforce discipline: they are scheduled and have precise agendas.
</p>
<a name="sec4"></a>
<h2>4. Overview of the whole process </h2>
<p>
The whole process looks as follows:
</p>
<ul>
<li>
Product Management Committee (X-Weekly)
<ul>
<li><b>1</b> Identification of a new User Story</li>
<li><b>2</b> Initial foreseen priority (i.e. release) depending on value and initial estimation (oral)</li>
</ul>
</li>
<li>
Architecture Committee (X-Weekly)
<ul>
<li><b>3</b> Design and specification by architecture committee : Story → Development Story → Task</li>
<li><b>4</b> Estimation of individual tasks</li>
<li><b>5</b> Computation of total SP and setting of size of Development Story and User Story</li>
<li><b>6</b> Re-prioritization (based on new estimation)</li>
</ul>
</li>
<li>
Sprint Planning + Sprint retrospective (Sprintly)
<ul>
<li><b>7</b> Review of TaskS and discussion : Task → Detailed Task</li>
<li><b>8</b> Adaptation of Estimation on TaskS</li>
<li><b>9</b> Update of Total Size of Development Story and User Story</li>
<li><b>10</b> Notification to Architecture Committee (Kaizen / Sprint retrospective)</li>
</ul>
</li>
<li>
Daily Scrum
<ul>
<li><b>11</b> Identification of Gap on Task</li>
<li><b>12</b> Adaptation of Estimation on Task</li>
<li><b>13</b> Update of Total Size of Development Story and User Story</li>
<li><b>14</b> Notification to Architecture Committee (Kaizen / Sprint retrospective)</li>
</ul>
</li>
</ul>
<p>
In a graphical way:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/6e3fc346-7277-45ce-af68-bea91cba88d5">
<img class="centered" style="width: 900px;" alt="Overview of the whole process" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/6e3fc346-7277-45ce-af68-bea91cba88d5" />
</a>
</div>
<br>
<a name="sec5"></a>
<h2>5. Return on Practices</h2>
<p>
As stated a lot of times in this article, all of this, reliable planning and true agility, require a strong commitment of the team to Agile practices and principles.
</p>
<p>
One cannot apply only a small subset of the Agile Practices and believe he will achieve true agility and Reliable Agile Planning.
<br>
The Agile practices I listed in introduction form a package with strong dependencies between each other.
</p>
<p>
IMHO the dependencies are as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/509a7199-f9b7-41d4-8ee3-e1970fb11ece">
<img class="centered" style="width: 800px; " alt="Agile Planning Practices dependencies" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/509a7199-f9b7-41d4-8ee3-e1970fb11ece" />
</a>
</div>
<br>
<p>
An arrow denotes a dependency between two practices.
</p>
<p>
Explanations of a few of these dependencies:
</p>
<ul>
<li>
You cannot imagine <b>reliable planning and forecasting</b> if you don't provide the management with appropriate tools : <b>Story Map</b> and <b>Kanban boards</b>.
<br>
Also, it's going to be difficult without a proper technical tool for the development team: <b>The Product Backlog</b>.
<br>
Finally, it obviously requires <b>Reliable Estimations</b>.
</li>
<li>
<b>Reliable estimations</B> need to have manageable and well <b>planned sprints</b>. 1 week sprints are too small, a lot can happen in 1 week while 3 weeks are too big in my opinion, the fluctuations are too important. I strongly believe that 2 weeks sprints is the right size when it comes to having an accurate and reliable Sprint Capacity (or Velocity) in <b>SP</b>.
<br>
With 2 weeks sprints only, the development team cannot afford spending time on releasing the <b>Shippable Product</b>, releasing should be a completely automated procedure and in this regards <b>Continuous Delivery</b> is not optional.
</li>
<li>
Then achieving <b>Continuous Delivery</b> requires a lot of things and a good mastery of common XP and DevOps Practices.
</li>
</ul>
<a name="sec6"></a>
<h2>6. Conclusion</h2>
<p>
Management needs a management tool to take enlightened decision. The product backlog should not be a management tool, it's really rather the development team's internal business. The Story Map, on the other hand, is a simple, visual and effective management tool.
<br>
All the rituals and processes introduced in this article are deployed towards the same ultimate goal: <b>enabling the management to use the Story Map as a management tool for planning and forecasting.</b> In addition, the specific form of Story Map introduced here, the <b>Product Kanban Board</b>, becomes also a Project Management Tool aimed to tracking the progresses of the development team.
</p>
<p>
The difficulty, the reason why it requires a strict enforcement of processes and rituals, is to synchronize the Story Map and the Product Backlog.
<br>
Since the development team works mostly with the Product Backlog, the later has eventually the accurate and realistic information about size and time of deployment, through the notion of Story Points.
<br>
But this is in no help for the management, hence the reason why it is required to backfeed the estimations put in the Product Backlog to the Story Map.
</p>
<p>
Eventually, if these processes and rituals are respected and well applied, anyone in the company can come in front of the <i>Product Kanban Board</i> with a little calculator and compute the delivery date (or rather the range) for any given story.
<br>
Anyone can use the Story Map to compute how much work can be done for any given date, or what time is required to deliver a specific scope.
</p>
<p>
All of this with a simple calculator and a few seconds, without Excel, without any Internet connection, without any complicated tool nor any pile of paper, just a calculator ... or a brilliant mind.
</p>
<p>
Now having said that, I would like to conclude this article by mentioning that the processes and tools I am presenting here work for us. They may not work as is for another organization. It's up to every organization to discover and find the practices and principles that best fit its needs and individuals.
<br>
As an example, the association of two Story Maps, the <i>"to do"</i> on the left and the <i>"done"</i> on the right of a Kanban Board for the needs of both Product and Project Management is a really personal recipe. While I myself got the idea from another organization, I haven't seen that often.
<br>
This shows in my opinion the very best qualities of an agilist: the curiosity to discover new ways of working and the courage to try them or invent them.
</p>
<p>
This article is available as a slideshare presentation here : <a href="https://www.slideshare.net/JrmeKehrli/agility-and-planning-tools-and-processes">https://www.slideshare.net/JrmeKehrli/agility-and-planning-tools-and-processes</a>.
</p>
<p>
Also, you can read a PDF version of this article here : <a href="https://www.niceideas.ch/Agile_Planning.pdf">https://www.niceideas.ch/Agile_Planning.pdf</a>.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/bytecode-manipulation-with-javassist-for1
Bytecode manipulation with Javassist for fun and profit part II: Generating toString and getter/setters using bytecode manipulation
Jerome Kehrli
2017-04-24T16:38:42-04:00
2017-09-13T16:41:56-04:00
<p>
Following my first article on Bytecode manipulation with Javassist presented there: <a href="https://www.niceideas.ch/roller2/badtrash/entry/bytecode-manipulation-with-javassist-for">Bytecode manipulation with Javassist for fun and profit part I: Implementing a lightweight IoC container in 300 lines of code</a>, I am here presenting another example of Bytecode manipulation with Javassist: generating <code>toString</code> method as well as property getters and setters with Javassist.
</p>
<p>
While the former example was oriented towards understanding how Javassist and bytecode manipulation comes in help with implementing IoC concerns, such as what is done by the spring framework of the pico IoC container, this new example is oriented towards generating <i>boilerplate code</i>, in a similar way to what <a href="https://projectlombok.org/">Project Lombok</a> is doing.
<br>
As a matter of fact, generating boilerplate code is another very sound use case for bytecode manipulation.
</p>
<p>
Boilerplate code refers to portions of code that have to be included or written in the same way in many places with little or no alteration.
<br>
The term is often used when referring to languages that are considered verbose, i.e. the programmer must write a lot of code to do minimal job. And Java is unfortunately a clear winner in this regards.
<br>
Avoiding boilerplate code is one of the main reasons (but by far not the only one of course !) why developers are moving away from Java in favor of other JVM languages such as Scala.
</p>
<p>
In addition, as a reminder, a sound understanding of the Java Bytecode and the way to manipulate it are strong prerequisites to software analytics tools, mocking libraries, profilers, etc. Bytecode manipulation is a key possibility in this regards, thanks to the JVM and the fact that bytecode is interpreted.
<br>
Traditionally, bytecode manipulation libraries suffer from complicated approaches and techniques. Javassist, however, proposes a natural, simple and efficient approach bringing bytecode manipulation possibilities to everyone.
</p>
<p>
So in this second example about Javassist we'll see how to implement typical <i>Lombok</i> features using Javassist, in a few dozen lines of code.
</p>
<!-- Bytecode manipulation with Javassist for fun and profit part II: Generating toString and getter/setters using bytecode manipulation -->
<p>
Following my first article on Bytecode manipulation with Javassist presented there: <a href="https://www.niceideas.ch/roller2/badtrash/entry/bytecode-manipulation-with-javassist-for">Bytecode manipulation with Javassist for fun and profit part I: Implementing a lightweight IoC container in 300 lines of code</a>, I am here presenting another example of Bytecode manipulation with Javassist: generating <code>toString</code> method as well as property getters and setters with Javassist.
</p>
<p>
While the former example was oriented towards understanding how Javassist and bytecode manipulation comes in help with implementing IoC concerns, such as what is done by the spring framework of the pico IoC container, this new example is oriented towards generating <i>boilerplate code</i>, in a similar way to what <a href="https://projectlombok.org/">Project Lombok</a> is doing.
<br>
As a matter of fact, generating boilerplate code is another very sound use case for bytecode manipulation.
</p>
<p>
Boilerplate code refers to portions of code that have to be included or written in the same way in many places with little or no alteration.
<br>
The term is often used when referring to languages that are considered verbose, i.e. the programmer must write a lot of code to do minimal job. And Java is unfortunately a clear winner in this regards.
<br>
Avoiding boilerplate code is one of the main reasons (but by far not the only one of course !) why developers are moving away from Java in favor of other JVM languages such as Scala.
</p>
<p>
In addition, as a reminder, a sound understanding of the Java Bytecode and the way to manipulate it are strong prerequisites to software analytics tools, mocking libraries, profilers, etc. Bytecode manipulation is a key possibility in this regards, thanks to the JVM and the fact that bytecode is interpreted.
<br>
Traditionally, bytecode manipulation libraries suffer from complicated approaches and techniques. Javassist, however, proposes a natural, simple and efficient approach bringing bytecode manipulation possibilities to everyone.
</p>
<p>
So in this second example about Javassist we'll see how to implement typical <i>Lombok</i> features using Javassist, in a few dozen lines of code.
</p>
<p>
Part of this article is available as a slideshare presentation here: <a href="https://www.slideshare.net/JrmeKehrli/bytecode-manipulation-with-javassist-for-fun-and-profit">https://www.slideshare.net/JrmeKehrli/bytecode-manipulation-with-javassist-for-fun-and-profit</a>.
</p>
<p>
You might want to have a look at the first article in this serie available here : <a href="https://www.niceideas.ch/roller2/badtrash/entry/bytecode-manipulation-with-javassist-for">Bytecode manipulation with Javassist for fun and profit part I: Implementing a lightweight IoC container in 300 lines of code</a>.
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. Introduction / Purpose </a></li>
<li><a href="#sec2">2. Javassist</a></li>
<li><a href="#sec3">3. Java Instrumentation framework and applications</a>
<ul>
<li><a href="#sec31">3.1 Lombok</a>
<ul>
<li><a href="#sec311">3.1.1 Example class</a></li>
<li><a href="#sec312">3.1.2 Without Lombok</a></li>
<li><a href="#sec313">3.1.3 With Lombok</a></li>
<li><a href="#sec314">3.1.4 Bytecode Manipulation vs. Code Generation</a></li>
<li><a href="#sec315">3.1.5 Just a note on concerns</a></li>
</ul>
</li>
<li><a href="#sec32">3.2 Java agents and the linkage problem</a>
<ul>
<li><a href="#sec321">3.2.1 Overcoming the Linkage problem with Java Agents</a></li>
<li><a href="#sec322">3.2.2 ClassFileTransformer</a></li>
<li><a href="#sec323">3.2.3 Caution</a></li>
<li><a href="#sec324">3.2.4 Simple Example</a></li>
<li><a href="#sec325">3.2.5 Invoking the Agent</a></li>
<li><a href="#sec326">3.2.6 Workaround</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#sec4">4.0 BCG: a simple approach for generating boilerplate code using Javassist</a>
<ul>
<li><a href="#sec41">4.1 Principle</a></li>
<li><a href="#sec42">4.2 Design</a></li>
<li><a href="#sec43">4.3 Implementation</a>
<ul>
<li><a href="#sec431">4.3.1 The code of the Agent</a></li>
<li><a href="#sec432">4.3.2 Interface ClassTransformer</a></li>
<li><a href="#sec433">4.3.3 Common Abstraction</a></li>
<li><a href="#sec434">4.3.4 The set of Class Transformers</a></li>
<li><a href="#sec435">4.3.5 Test Class Example</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#sec5">5. Conclusion</a></li>
</ul>
<a name="sec1"></a>
<h2>1. Introduction / Purpose </h2>
<p>
<i>I am giving only a few lines here and return the user to <a href="https://www.niceideas.ch/roller2/badtrash/entry/bytecode-manipulation-with-javassist-for#sec1">my first article on this topic</a> for the complete introduction to bytecode manipulation.</i>
</p>
<p>
Bytecode manipulation consists in modifying the classes - represented by bytecode - compiled by the Java compiler, at runtime. It is used extensively for instance by frameworks such as Spring (IoC) and Hibernate (ORM) to inject dynamic behaviour to Java objects at runtime.
<br>
Bytecode manipulation is traditionally difficult. Out of all the various libraries and tools to achieve it, Javassist stands out due to its natural and simple yet efficient approach to it.
</p>
<a name="sec2"></a>
<h2>2. Javassist</h2>
<p>
<i>I am giving only a few lines here and return the user to <a href="https://www.niceideas.ch/roller2/badtrash/entry/bytecode-manipulation-with-javassist-for#sec2">my first article on this topic</a> for the complete introduction to Javassist.</i>
</p>
<p>
Quoting <a href="">Wikipedia</a>:
</p>
<p>
Javassist (Java programming assistant) is a Java library providing a means to manipulate the Java bytecode of an application.
</p>
<p>
Javassist enables Java programs to define a new class at runtime and to modify a class file when the JVM loads it. Unlike other similar bytecode editors, Javassist provides two levels of API: source level and bytecode level. Using the source-level API, programmers can edit a class file without knowledge of the specifications of the Java bytecode; the whole API is designed with only the vocabulary of the Java language. Programmers can even specify inserted bytecode in the form of Java source text; Javassist compiles it on the fly. On the other hand, the bytecode-level API allows the users to directly edit a class file as other editors.
</p>
<a name="sec3"></a>
<h2>3. Java Instrumentation framework and applications</h2>
<p>
Java 5 was the first version seeing the proper implementation of JSR-163 (Java Platform Profiling Architecture API) support including a bytecode instrumentation mechanism through the introduction of the Java Programming Language Instrumentation Services (JPLIS). At first that JSR only mentioned native (C) interfaces but evolved fast towards a pretty convenient Java API.
<br>
This was an interesting breakthrough since it allowed, with the help of an agent, to modify the content of a class bytecode inherent to the methods of a class in such a way as to modify its behavior at runtime.
</p>
<p>
The key point of the JSR-163 is JVMTI. JVMTI - or Java Virtual Machine Tool Interface - allows a program to inspect the state and to control the execution of applications running in the Java Virtual Machine. JVMTI is designed to provide an <i>Application Programming Interface</i> (API) for the development of tools that need access to the state of the JVM. Examples for such tools are debuggers, profilers or runtime boilerplate code generator.
</p>
<p>
The scope of this article is to focus on presenting how a boilerplate code generator can benefit from JVMTI and bytecode manipulation to achieve unprecedented comfort and easiness of programming than ever before on the JVM, unless considering a different language than Java, such as Scala or Clojure.
</p>
<p>
As a way to illustrate Boilerplate code concerns, We will focus on <i>Project Lombok</i> and its features and try to see how to reproduce some of them using Javassist.
</p>
<a name="sec31"></a>
<h3>3.1 Lombok</h3>
<p>
Project Lombok is a <i>Boilerplate code</i> generator that addresses one of the most frequent criticism against the Java Programming Language: the volume of this type of code that is found in most projects.
<br>
<i>Boilerplate code</i> is a term used to describe code that is repeated in many parts of an application with only slight contextual changes and with little added value.
<br>
Project Lombok reduces the need of some of the worst offenders by replacing each of them with a simple annotation. It then takes care of generating the boilerplate code at runtime
</p>
<p>
Project Lombok also integrates well in IDEs making it possible to use the usual IDE features such as refactoring and usage analysis pretty transparently.
</p>
<p>
Importantly in our context, since we will focus below on reproducing Lombok features using runtime bytecode generation, I have to mention right away that Lombok doesn't just generate Java sources or bytecode: it transforms the <i>Abstract Syntax Tree</i> (AST), by modifying its structure at compile-time.
<br>
The AST is a tree representation of the parsed source code, created by the compiler, similar to the DOM tree model of an XML file. By modifying (or transforming) the AST, Lombok keeps the source code trim and free of bloat, unlike plain-text code-generation.
<br>
Lombok's generated code is also visible to classes within the same compilation unit, unlike direct bytecode manipulation.
<br>
<i>Our approach, as we will see below, using runtime bytecode manipulation shall not benefit from the same comfort and is hence less efficient than what Lombok is doing.</i>
</p>
<a name="sec311"></a>
<h4>3.1.1 Example class</h4>
<p>
Let's see an example. Imagine the following Java POJO:
</p>
<pre>
<b>public class</b> DataExample {
<b>private final</b> String name;
<b>private int</b> age;
<b>private double</b> score;
<b>private</b> String[] tags;
}
</pre>
<p>
Typical boilerplate code involved when considering such a POJO are:
</p>
<ul>
<li>Getters and Setters for all private fields, making them <i>JavaBean properties</i></li>
<li>A nice <code>toString</code> method giving the values of its properties when an object is output on the console</li>
<li>Consistent <code>hashCode</code> and <code>equals</code> methods enabling to compare and manipulate two different objects with same values</li>
<li>A default constructor without any argument (Javabean standard)</li>
<li>An <i>all args</i> constructor taking all values as argument to build the instance</li>
</ul>
<a name="sec312"></a>
<h4>3.1.2 Without Lombok</h4>
<p>
Without Lombok, writing all this code is a nightmare, and the simple class above becomes as follows:
</p>
<pre>
<b>import</b> java.util.Arrays;
<b>public class</b> DataExample {
<b>private final</b> String name;
<b>private int</b> age;
<b>private double</b> score;
<b>private</b> String[] tags;
<b>public</b> DataExample(String name) {
<b>this</b>.name = name;
}
<b>public</b> DataExample(String name, <b>int</b> age, <b>double</b> score, String[] tags) {
<b>this</b>.name = name;
<b>this</b>.age = age;
<b>this</b>.score = score;
<b>this</b>.tags = tags;
}
<b>public</b> String getName() {
<b>return this</b>.name;
}
<b>void</b> setAge(<b>int</b> age) {
<b>this</b>.age = age;
}
<b>public int</b> getAge() {
<b>return this</b>.age;
}
<b>public void</b> setScore(<b>double</b> score) {
<b>this</b>.score = score;
}
<b>public double</b> getScore() {
<b>return this</b>.score;
}
<b>public</b> String[] getTags() {
<b>return this</b>.tags;
}
<b>public void</b> setTags(String[] tags) {
<b>this</b>.tags = tags;
}
@Override
<b>public</b> String toString() {
<b>return</b> <span style="color: blue;">"DataExample("</span> + <b>this</b>.getName() +
<span style="color: blue;">", "</span> + <b>this</b>.getAge() +
<span style="color: blue;">", "</span> + <b>this</b>.getScore() +
<span style="color: blue;">", "</span> + Arrays.deepToString(<b>this</b>.getTags()) + <span style="color: blue;">")"</span>;
}
@Override
<b>public boolean</b> equals(Object o) {
<b>if</b> (o == <b>this</b>) <b>return true</b>;
<b>if</b> (!(o <b>instanceof</b> DataExample)) <b>return false</b>;
DataExample other = (DataExample) o;
<b>if</b> (<b>this</b>.getName() == <b>null</b> ?
other.getName() != <b>null</b> :
!<b>this</b>.getName().equals(other.getName()))
<b>return false</b>;
<b>if</b> (<b>this</b>.getAge() != other.getAge()) <b>return false</b>;
<b>if</b> (Double.compare(<b>this</b>.getScore(), other.getScore()) != 0) <b>return false</b>;
<b>if</b> (!Arrays.deepEquals(<b>this</b>.getTags(), other.getTags())) <b>return false</b>;
<b>return true</b>;
}
@Override
<b>public int</b> hashCode() {
<b>final int</b> PRIME = 59;
<b>int</b> result = 1;
<b>final long</b> temp1 = Double.doubleToLongBits(<b>this</b>.getScore());
result = (result*PRIME) + (<b>this</b>.getName() == <b>null</b> ? 43 : <b>this</b>.getName().hashCode());
result = (result*PRIME) + <b>this</b>.getAge();
result = (result*PRIME) + (<b>int</b>)(temp1 ^ (temp1 >>> 32));
result = (result*PRIME) + Arrays.deepHashCode(<b>this</b>.getTags());
<b>return</b> result;
}
}
</pre>
<p>
One needs to understand that while the features implemented by the code above are pretty useful and very important, the code itself has no added value whatsoever. Most IDEs generate this code for you using some right-click here and there. A machine can write this code for you, can you imagine this? ... And yet with java it HAS to be written. This makes no sense.
</p>
<p>
So without Lombok, for a 4 properties, 5 lines of code class ... One has to write more than 60 lines of boilerplate code, that's a ratio of [Boilerplate code / Useful Code] of more than 1600% !!!
<br>
This is the main rationality behind project Lombok.
</p>
<a name="sec313"></a>
<h4>3.1.3 With Lombok</h4>
<p>
With Lombok we can use the following set of annotations on top of the class to generate the very same boilerplate code that we had to write on our own above:
</p>
<pre>
@Getter
@Setter
@ToString
@EqualsAndHashCode
@RequiredArgsConstructor
@AllArgsConstructor
<b>public class</b> DataExample {
<b>private final</b> String name;
<b>private int</b> age;
<b>private double</b> score;
<b>private</b> String[] tags;
}
</pre>
<p>
All these annotations are straightforward to understand so I won't be describing them any further.
</p>
<p>
You might want to look at the <a href="https://projectlombok.org/features/index.html">Lombok documentation</a> to learn more about these annotations.
</p>
<p>
With Lombok, we get a much better ratio of boilerplate code concerns vs. useful code of 100% instead of more than 1600% without Lombok.
</p>
<a name="sec314"></a>
<h4>3.1.4 Lombok Apporoach (AST transformation) vs. Bytecode Manipulation vs. Code Generation</h4>
<p>
<b>Code generation</b>
</p>
<p>
Of course there are alternatives to Lombok. For instance, most if not all IDEs enable the developer to generate these <i>boilerplate</i> methods in the class source file just as Lombok does it.
<br>
But generating this code in the class source file is IMHO not the right approach. At the end of the day, I believe that <b>generated code</b> - i.e. code that can be written by a machine - <b>should simply not be written</b>! If a machine can write this code then this code has no added value at all! I do not want to see such code in my class files, I do not want to be aware of it.
<br>
Lombok generates this code transparently and seamlessly. By modifying (or transforming) the AST, Lombok keeps your source code trim and free of bloat, unlike plain-text code-generation. With Lombok, I do not need to be aware of this code, it's not polluting my <i>code coverage</i> computation (in Sonar for instance or else) and everything behaves <i>as if</i> this code was actually written except that one doesn't see it and doesn't need to care about it.
</p>
<p>
<b>Bytecode Manipulation</b>
</p>
<p>
Bytecode manipulation is preferable over Code generation IMHO in regards to generating (at least some of the) boilerplate code. For the same reason indicated above: bytecode manipulation makes the boilerplate code transparent and avoids it polluting my source code.
<br>
It is however not as efficient as AST Transformation such as Lombok is doing. Lombok's generated code is also visible to classes within the same compilation unit, unlike direct bytecode manipulation.
</p>
<p>
For instance when it comes to generating getters and setters, Lombok makes them visible at compile time, making it possible for client code to use them without them appearing in the Java Bean class.
<br>
Using bytecode manipulation to generate getters and setters, they become invisible to the compiler and cannot be used by another class, except using runtime reflection, which may be fine for some use cases (hibernate, etc) but not for most of them.
</p>
<p>
<b>AST Transformation</b>
</p>
<p>
More information on Lombok's internal is availanle there: <a href="https://www.ibm.com/developerworks/library/j-lombok/">Custom AST transformations with Project Lombok</a>.
</p>
<a name="sec315"></a>
<h4>3.1.5 Just a note on concerns</h4>
<p>
I sometime hear (... read) arguments online against the use of Lombok. Most of the time, people against Lombok complain about the <i>magic</i> added by Lombok.
</p>
<p>
It is true that Lombok adds a lot of <i>magic</i> to the application. But that magic is related to boilerplate code with no added value and easy to debug and understand in case of any doubt. In addition, it is very well recognized by most common IDEs such as eclipse and IDEA using well known or even official plugins.
</p>
<a name="sec32"></a>
<h3>3.2 Java agents and the linkage problem</h3>
<p>
Javassist cannot modify a class after it has been loaded by a classloader ... as far as this classloader is concerned.
<br>
Whenever one tries to modify a class already loaded by the referenced classloader, that attempt to call <code>pool.makeClass( ... )</code> will fail and complain that class is <i>frozen</i> (i.e. already created via <code>toClass()</code>.
<br>
Being able to do that would require to <i>unload</i> the class first from the reference Classloader.
</p>
<p>
The problem here is that one cannot unload a single class from a ClassLoader. A class may be unloaded if it and its ClassLoader became unreachable but since every class refers to its loader that implies that all classes loaded by this loader must have to become unreachable too.
<br>
Of course one can (re-)create the class using a different ClassLoader but that would require to make the whole program use that new Classloader and this becomes fairly complicated.
<br>
At the end of the day, that would well require reloading the whole application and initializing everything all over again. This makes no sense.
</p>
<p>
Let's just accept here that a class cannot be changed by Javassist once it has been already loaded by the Classloader.
</p>
<a name="sec321"></a>
<h4>3.2.1 Overcoming the Linkage problem with Java Agents</h4>
<p>
The only (easy) way to overcome this problem is to change the class implementation using bytecode manipulation before the class is loaded by any Classloader. And happily, as often in the Java World, the JVM provides a mechanism for this, the JPLIS - <i>Java Programming Language Instrumentation Services</i> - and the concept of <i>Java Agent</i>.
</p>
<p>
In its essence, a Java agent is a regular Java class which follows a set of strict conventions. The agent class must implement a <code>public static void premain(String agentArgs, Instrumentation inst)</code> method which becomes an agent entry point (similar to the main method for regular Java applications).
</p>
<p>
Once the Java Virtual Machine (JVM) has initialized, each such <code>premain(String agentArgs, Instrumentation inst)</code> method of every agent will be called in the order the agents were specified on JVM start. When this initialization step is done, the real Java application main method will be called.
</p>
<p>
The instrumentation capabilities of Java agents are truly unlimited. Most noticeable one is the ability to redefine classes at run-time. The redefinition may change method bodies, the constant pool and attributes. The redefinition must not add, remove or rename fields or methods, change the signatures of methods, or change inheritance.
</p>
<p>
Please notice that re-transformed or redefined class bytecode is not checked, verified and installed just after the transformations or redefinitions have been applied. If the resulting bytecode is erroneous or not correct, the exception will be thrown and that may crash JVM completely.
</p>
<a name="sec322"></a>
<h4>3.2.2 ClassFileTransformer</h4>
<p>
A Java agent <code>premain</code> method takes the Instrumentation entry point - class <code>java.lang.instrument.Instrumentation</code> - as argument.
</p>
<p>
The Instrumentation entry point provides several commodity methods to check the possibilities of the JVM but the most important API of the <code>java.lang.instrument.Instrumentation</code> class is the method <code>void addTransformer(ClassFileTransformer transformer);</code> that enable the developer to register several <code>java.lang.instrument.ClassFileTransformer</code>.
</p>
<p>
The <code>java.lang.instrument.ClassFileTransformer</code> interface defines one single method <code>byte[] transform(...)</code> that is responsible to apply transformations (as far as complete rewriting if required) the java classes being loaded by the JVM.
<br>
The <code>transform(...)</code> method is called for each and every class being loaded by a classloader. Both the class being loaded and the classloader actually loading it as well as other information are given in argument.
</p>
<p>
The <code>transform(...)</code> method is the ideal place where bytecode manipulation libraries can be used to modify classes just before they are loaded by the classloader.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/62b1c423-2028-4ef3-ab25-37ac8bbaf7ab">
<img class="centered" style="width: 450px;" alt="Java Agent Behaviour" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/62b1c423-2028-4ef3-ab25-37ac8bbaf7ab" />
</a><br>
<div class="centered">
(Source : <a href="http://www.barcelonajug.org/2015/04/java-agents.html">http://www.barcelonajug.org/2015/04/java-agents.html</a>)
</div>
</div>
<br>
<a name="sec323"></a>
<h4>3.2.3 Caution</h4>
<p>
As a sidenote, the Java agent class may also have a <code>public static void agentmain(String agentArgs, Instrumentation inst)</code> method which is used when the agent is started after JVM startup.
</p>
<p>
A common practice when developing agents is to implement both the <code>agentmain</code> and <code>premain</code> methods and delegate one to the other. See <a href="#sec431">implementation of Java Agent</a> of BCG below.
</p>
<a name="sec324"></a>
<h4>3.2.4 Simple Example</h4>
<pre>
<b>package</b> ch.niceideas.common.agent;
<b>import</b> java.lang.instrument.ClassFileTransformer;
<b>import</b> java.lang.instrument.IllegalClassFormatException;
<b>import</b> java.lang.instrument.Instrumentation;
<b>import</b> java.security.ProtectionDomain;
<b>public class</b> ClassLoadingLoggingAgent {
<b>public static void</b> premain(String agentArgument, Instrumentation instrumentation){
System.out.println(<span style="color: blue;">"Hello, Agent [ "</span> + agentArgument + <span style="color: blue;">" ]"</span>);
instrumentation.addTransformer (<b>new</b> ClassFileTransformer() {
@Override
<b>public byte</b>[] transform(
ClassLoader loader,
String className,
Class<?> classBeingRedefined,
ProtectionDomain protectionDomain,
<b>byte</b>[] classfileBuffer) <b>throws</b> IllegalClassFormatException {
<span style="color: green;">// Transform is called just before class loading occurs :-)</span>
System.out.println(<span style="color: blue;">"Class being loaded : "</span> + className);
<span style="color: green;">// No transformation ...</span>
<b>return</b> classfileBuffer;
}
});
}
}
</pre>
<p>
Let's now see how to invoke a simple program using this simple agent.
</p>
<a name="sec325"></a>
<h4>3.2.5 Invoking the Agent</h4>
<p>
When running from the command line, the Java agent could be passed to JVM instance using <code>-javaagent</code> argument which has following semantic <code>-javaagent:<path-to-jar>[=options]</code>.
</p>
<p>
A java agent needs to be packaged in a jar file and that jar file needs to have a specific and proper <code>MANIFEST.MF</code> file indicating the class containing the <code>premain</code> method.
</p>
<p>
A proper manifest file for the agent above should be packages within the jar archive containing the agent classes under <code>META-INF/MANIFEST.MF</code> and would be as follows:
</p>
<pre>
Manifest-Version: 1.0
Premain-Class: ch.niceideas.common.agent.ClassLoadingLoggingAgent
</pre>
<p>
Now let's imagine we invoke our agent on a simple program defined as follows:
</p>
<pre>
<b>package</b> ch.niceideas.common.enhancer;
<b>public class</b> TestMain {
<b>public static void</b> main (String args[]) {
System.out.println (<span style="color: blue;">"Program Main"</span>);
}
}
</pre>
<p>
Sample result on this simple program is as follows:
</p>
<pre>
badtrash@badbook:/data/work/niceideas-commons/target/test-classes$ <span style="color: DarkBlue;">java</span> \
<span style="color: DarkBlue;">-javaagent:</span>/home/badtrash/ClassLoadingLoggingAgent.jar<span style="color: DarkRed;">=007</span> \
ch.niceideas.common.enhancer.TestMain
Hello, Agent [ 007 ]
Class being loaded : java/lang/invoke/MethodHandleImpl
Class being loaded : java/lang/invoke/MethodHandleImpl$1
Class being loaded : java/lang/invoke/MethodHandleImpl$2
Class being loaded : java/util/function/Function
Class being loaded : java/lang/invoke/MethodHandleImpl$3
Class being loaded : java/lang/invoke/MethodHandleImpl$4
Class being loaded : java/lang/ClassValue
Class being loaded : java/lang/ClassValue$Entry
Class being loaded : java/lang/ClassValue$Identity
Class being loaded : java/lang/ClassValue$Version
Class being loaded : java/lang/invoke/MemberName$Factory
Class being loaded : java/lang/invoke/MethodHandleStatics
Class being loaded : java/lang/invoke/MethodHandleStatics$1
Class being loaded : sun/launcher/LauncherHelper
Class being loaded : java/util/concurrent/ConcurrentHashMap$ForwardingNode
Class being loaded : sun/misc/URLClassPath$FileLoader$1
Class being loaded : java/lang/Package
Class being loaded : java/io/FileInputStream$1
Class being loaded : ch/niceideas/common/enhancer/TestMain
Class being loaded : sun/launcher/LauncherHelper$FXHelper
Class being loaded : java/lang/Class$MethodArray
Class being loaded : java/lang/Void
Program Main
Class being loaded : java/lang/Shutdown
Class being loaded : java/lang/Shutdown$Lock
</pre>
<a name="sec326"></a>
<h4>3.2.6 Workaround</h4>
<p>
As a sidenote, and to conclude this section, let's just mention that using a java agent to inject behaviour at runtime using bytecode manipulation is not always a requirement, it depends eventually on the use case.
<br>
A pretty common approach favored over java agents usage is the subclassing approach. It consists in defining a new class as a subclass of the class to be enhanced and injecting the new behaviour to that subclass instead.
<br>
This is a pretty straightforward approach and prevents the usage of a java agent since we don't care whether or not the initial class has already be loaded. Since we define a new class, the subclass, we're good to go no matter what happens with the initial class.
</p>
<p>
I have given an example of this approach in my previous article as <a href="https://www.niceideas.ch/roller2/badtrash/entry/bytecode-manipulation-with-javassist-for#sec41">described here</a>.
</p>
<p>
In the case of boilerplate code generation such as done by Lombok, using an agent is pretty much the only way. The Value Objects or Java Beans enhanced this way can well be used later by the running program for ORM concerns or IoC concerns. These other frameworks, such as hibernate or spring, very often use the subclassing approach to inject their own behaviour.
<br>
If a programmer attempts to use the subclass trick to inject his own behaviour and then use his class with hibernate for instance, his changes will likely be ignored by hibernate or spring that will generate their own subclass (using CGLIB or Javassist) that can conflict the developer's own subclass. In this case enhancing the class itself is a way simpler approach.
</p>
<p>
Finally, using a java agent is a convenient way to avoid situations where the developer attempts to enhance a class that has already been loaded by the classloader. But that is not necessarily always required and the developer can well choose to implement his own control over the application lifecycle to ensure he has the chance to modify classes before the application attempts to load them.
<br>
But that is impossible when using annotations of course and hence frameworks such as Lombok using annotations extensively have no other choice than using a java agent.
</p>
<a name="sec4"></a>
<h2>4. BCG: a simple approach for generating boilerplate code using Javassist</h2>
<p>
Now back on the Javassist topic.
</p>
<p>
The purpose of this article is to give a second example of a sound Javassist use case: the generation of boilerplate code using bytecode manipulation, just as project Lombok is doing.
<br>
In fact, I will present here the BCG tool that mimic Lombok and re-implement two features of the Lombok feature set.
</p>
<p>
I am presenting here the few dozen lines of code on the BCG tool - BCG for <i>Boilerplate Code Generator</i>.
<br>
BCG is a simple tool that uses Javassist and implement a Java agent to key Lombok features:
</p>
<ol>
<li><code>toString()</code> method generation</li>
<li>property getters and setters generation</li>
</ol>
<p>
Note that BCG is not a production tool or anything like it, it is really just a Javassist example and intended to demonstrate how straightforward, simple and efficient it would be to re-implement Lombok features using Javassist ... Should one want to do that, which is not likely since Lombok is working so cool and so easily extendable.
</p>
<p>
As mentioned above, we will only be mimicking project Lombok here using bytecode manipulation. We are not implementing these features the same way Lombok is doing. Lombok is working at compile-time using AST Transformation. We will be working at runtime using bytecode manipulation.
</p>
<a name="sec41"></a>
<h3>4.1 Principle </h3>
<p>
We want to be able to implement <i>transformers</i> that take care of performing one specific modification to target classes and activated by the presence of one specific annotation on these classes.
</p>
<p>
The key idea is to implement a <i>Java Agent</i> that analyze each and every class just before is loaded by the classloader and verifies if this class needs to be transformed.
<br>
We want to implement <i>Transformers</i> that recognize classes declaring a specific annotation and proceed with the transformation of these classes.
<br>
We want the system to be easily extendable with new transformers.
</p>
<a name="sec42"></a>
<h3>4.2 Design </h3>
<p>
The design of BCG is as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/928b7d05-c842-400b-a285-297b795c5e4b">
<img class="centered" style="width: 600px;" alt="Boilerplate code generator design" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/928b7d05-c842-400b-a285-297b795c5e4b" />
</a>
</div>
<br>
<p>
Principal components are as follows:
</p>
<ul>
<li>
<b><code>EnhancerAgent</code></b> : This is a JVM Agent that implements classes transformation from recognized annotations.
<br>
The EnhancerAgent is called before the application starts and registers a <code>java.lang.instrument.ClassFileTransformer</code> that enhances classes declaring specific annotations before they are loaded by classloader(s).
<br>
The <code>java.lang.instrument.ClassFileTransformer</code> here is a simple anonymous adapter.
</li>
<li>
<b><code>ClassTransformer</code></b> : This is an interface implemented by actual Class Transformers. A Class Transformer transforms Java classes declaring the recognized
annotation(s) using Javassist.
</li>
<li>
<b><code>AbstractTransformer</code></b> : This is the base class for all Transformers. It provides commodity routines for ClassTransformers and simplifies registration API.
</li>
</ul>
<p>
Then all actual transformers extend <code>AbstractTransformer</code> and simply declare the annotation they recognize.
</p>
<a name="sec43"></a>
<h3>4.3 Implementation</h3>
<p>
The source code of all classes and interfaces from the design above is given below.
</p>
<p>
(In all snippets of code from now on, <code><span style="color: DarkRed">I will be coloring relevant Javassist API calls in dark red</span></code>)
</p>
<a name="sec431"></a>
<h4>4.3.1 The code of the Agent</h4>
<p>
<b>[Class <code>EnhancerAgent</code>]</b>
</p>
<pre>
<b>package</b> ch.niceideas.common.enhancer;
<b>import</b> ch.niceideas.common.enhancer.impls.CountInstanceTransformer;
<b>import</b> ch.niceideas.common.enhancer.impls.DataTransformer;
<b>import</b> ch.niceideas.common.enhancer.impls.ToStringTransformer;
<b>import</b> <span style="color: DarkRed">javassist.CannotCompileException</span>;
<b>import</b> <span style="color: DarkRed">javassist.ClassPool</span>;
<b>import</b> <span style="color: DarkRed">javassist.CtClass</span>;
<b>import</b> java.io.IOException;
<b>import</b> java.lang.annotation.Annotation;
<b>import</b> java.lang.instrument.ClassFileTransformer;
<b>import</b> java.lang.instrument.IllegalClassFormatException;
<b>import</b> java.lang.instrument.Instrumentation;
<b>import</b> java.security.ProtectionDomain;
<b>import</b> java.util.ArrayList;
<b>import</b> java.util.List;
<span style="color: green;">/**
* This is a JVM Agent that implements classes transformation from recognized
* annotations.
* <p />
*
* The EnhancerAgent is called before the application starts and registers a
* <code>java.lang.instrument.ClassFileTransformer</code> that enhances
* classes declaring specific annotations before they are loaded
* by classloader(s).
* <p />
*
* The <code>java.lang.instrument.ClassFileTransformer</code> here is a simple
* anonymous adapter.
*/</span>
<b>public class</b> EnhancerAgent {
<b>private static</b> ClassTransformer[] transformers = <b>null</b>;
<span style="color: green;">// for now I don't have any better way than declaring all transformers here</span>
<b>static</b> {
transformers = <b>new</b> ClassTransformer[] {
<b>new</b> CountInstanceTransformer(),
<b>new</b> ToStringTransformer(),
<b>new</b> DataTransformer()
};
}
<span style="color: green;">// Java Agent API</span>
<b>public static void</b> premain(String agentArgs, Instrumentation inst) {
agentmain (agentArgs, inst);
}
<span style="color: green;">// API used when agent invoked after JVM Startup</span>
<b>public static void</b> agentmain(String agentArgs, Instrumentation inst) {
inst.addTransformer(<b>new</b> ClassFileTransformer() {
@Override
<b>public byte</b>[] transform (
ClassLoader loader,
String className,
Class<?> classBeingRedefined,
ProtectionDomain protectionDomain,
<b>byte</b>[] classfileBuffer)
<b>throws</b> IllegalClassFormatException {
<span style="color: green;">// Can return null if no transformation is performed</span>
<b>byte</b>[] transformedClass = <b>null</b>;
<span style="color: DarkRed">CtClass</span> cl = <b>null</b>;
<span style="color: DarkRed">ClassPool</span> pool = <span style="color: DarkRed">ClassPool.getDefault</span>();
<b>try</b> {
cl = pool.<span style="color: DarkRed">makeClass</span>(<b>new</b> java.io.ByteArrayInputStream(classfileBuffer));
<b>for</b> (ClassTransformer transformer : transformers) {
<b>if</b> (transformer.accepts (cl)) {
transformer.transform (cl);
System.out.println (<span style="color: blue;">"Transformed class "</span> + cl.getName()
+ <span style="color: blue;">" with "</span> + transformer.getClass().getSimpleName());
}
}
<span style="color: green;">// Generate changed bytecode</span>
transformedClass = cl.<span style="color: DarkRed">toBytecode</span>();
} <b>catch</b> (IOException | <span style="color: DarkRed">CannotCompileException</span> e) {
e.printStackTrace();
} <b>finally</b> {
<b>if</b> (cl != <b>null</b>) {
cl.<span style="color: DarkRed">detach</span>();
}
}
<b>return</b> transformedClass;
}
});
}
}
</pre>
<a name="sec432"></a>
<h4>4.3.2 Interface ClassTransformer</h4>
<p>
Transformers implement this interface:
</p>
<p>
<b>[Class <code>ClassTransformer</code>]</b>
</p>
<pre>
<b>package</b> ch.niceideas.common.enhancer;
<b>import</b> <span style="color: DarkRed">javassist.CtClass</span>;
<span style="color: green;">/**
* A class Transformer transforms Java classes declaring the recognized
* annotation(s) using Javassist.
*/</span>
<b>public interface</b> ClassTransformer {
<span style="color: green;">/**
* Used by the EnhancerAgent to know whether this class accepts the supported anotation
*
* @param cl the class to test
* @return true if the passed annotation is accepted
*/</span>
<b>boolean</b> accepts(<span style="color: DarkRed">CtClass</span> cl);
<span style="color: green;">/**
* Proceed with the transformation of the javassist loaded class given as argument
*
* @param cl the javassist loaded class to be transformed
*/</span>
<b>void</b> transform(<span style="color: DarkRed">CtClass</span> cl);
}
</pre>
<a name="sec433"></a>
<h4>4.3.3 Common Abstraction</h4>
<p>
And an abstract class provides some binding commodities to transformers
</p>
<p>
<b>[Class <code>AbstractTransformer</code>]</b>
</p>
<pre>
<b>package</b> ch.niceideas.common.enhancer.impls;
<b>import</b> ch.niceideas.common.enhancer.ClassTransformer;
<b>import</b> <span style="color: DarkRed">javassist.CtClass</span>;
<b>import</b> java.lang.annotation.Annotation;
<span style="color: green;">/**
* Base class for ClassTransformers.
* <br />
* Provides commodity routines for ClassTransformers and simplifies registration API.
*/</span>
<b>public abstract class</b> AbstractTransformer <b>implements</b> ClassTransformer {
@Override
<b>public final boolean</b> accepts(<span style="color: DarkRed">CtClass</span> cl) {
<b>return</b> cl.<span style="color: DarkRed">hasAnnotation</span>(getAnnotationClass());
}
<span style="color: green;">/**
* Classes that wants to be transformed by this transformer needs to declare
* this annotation.
*
* @return the type of the annotation accepted by this transformer.
*/</span>
<b>protected abstract</b> Class<? <b>extends</b> Annotation> getAnnotationClass();
@Override
<b>public abstract void</b> transform(<span style="color: DarkRed">CtClass</span> cl);
}
</pre>
<a name="sec434"></a>
<h4>4.3.4 The set of Class Transformers</h4>
<p>
A first Class Transformer : outputs the count of instances of classes declaring the <code>@CountInstance</code> annotation.
</p>
<p>
<b>[Class <code>CountInstanceTransformer</code>]</b>
</p>
<pre>
<b>package</b> ch.niceideas.common.enhancer.impls;
<b>import</b> ch.niceideas.common.enhancer.ClassTransformer;
<b>import</b> ch.niceideas.common.enhancer.annotations.CountInstance;
<b>import</b> <span style="color: DarkRed">javassist.CannotCompileException</span>;
<b>import</b> <span style="color: DarkRed">javassist.CtBehavior</span>;
<b>import</b> <span style="color: DarkRed">javassist.CtClass</span>;
<b>import</b> <span style="color: DarkRed">javassist.CtField</span>;
<b>import</b> java.lang.annotation.Annotation;
<span style="color: green;">/**
* This transformer accepts classe declaring the "@CountInstance" annotation.
* <br>
* It enhances the class with an instanceCounter and outputs the value of ths
* instanceCounter everytime an instance is built.
*/</span>
<b>public class</b> CountInstanceTransformer <b>extends</b> AbstractTransformer
<b>implements</b> ClassTransformer {
@Override
<b>protected</b> Class<? <b>extends</b> Annotation> getAnnotationClass() {
<b>return</b> CountInstance.<b>class</b>;
}
@Override
<b>public void</b> transform(<span style="color: DarkRed">CtClass</span> cl) {
<b>try</b> {
<b>if</b> (!cl.<span style="color: DarkRed">isInterface</span>()) {
<span style="color: green;">// Add a static field in the class</span>
<span style="color: DarkRed">CtField</span> field = <span style="color: DarkRed">CtField.make</span>(<span style="color: blue;">"private static long _instanceCount;"</span>, cl);
cl.<span style="color: DarkRed">addField</span>(field);
<span style="color: DarkRed">CtBehavior</span>[] constructors = cl.<span style="color: DarkRed">getDeclaredConstructors</span>();
<b>for</b> (<b>int</b> i = 0; i < constructors.length; i++) {
<span style="color: green;">// Increment counter and output it</span>
constructors[i].<span style="color: DarkRed">insertAfter</span>(<span style="color: blue;">"_instanceCount++;"</span>);
constructors[i].<span style="color: DarkRed">insertAfter</span>(<span style="color: blue;">"System.out.println(\""</span>
+ cl.<span style="color: DarkRed">getName</span>() + <span style="color: blue;">" : \" + _instanceCount);"</span>);
}
}
} <b>catch</b> (<span style="color: DarkRed">CannotCompileException</span> e) {
e.printStackTrace();
<b>throw new</b> RuntimeException (e);
}
}
}
</pre>
<p>
The <code>CountInstanceTransformer</code> accepts classes declaring the <code>CountInstance</code> annotation :
</p>
<pre>
<b>package</b> ch.niceideas.common.enhancer.annotations;
<span style="color: green;">/**
* Classes declaring this annotation will have an instancecounter which value is
* output everytime an instance is constructed
*/</span>
<b>public @interface</b> CountInstance {
}
</pre>
<p>
Second Class Transformer: generates getters and setters for classes declaring the <code>@Data</code> annotation.
</p>
<p>
<b>[Class <code>DataTransformer</code>]</b>
</p>
<pre>
<b>package</b> ch.niceideas.common.enhancer.impls;
<b>import</b> ch.niceideas.common.enhancer.ClassTransformer;
<b>import</b> ch.niceideas.common.enhancer.annotations.Data;
<b>import</b> <span style="color: DarkRed">javassist.*</span>;
<b>import</b> java.lang.annotation.Annotation;
<span style="color: green;">/**
* This transformer accepts classes declaring the "@Data" annotation.
* <br />
* It generates a getters and setters dynamically for every field of the class
* if they do not already exist
*/</span>
<b>public class</b> DataTransformer <b>extends</b> AbstractTransformer <b>implements</b> ClassTransformer {
@Override
<b>protected</b> Class<? <b>extends</b> Annotation> getAnnotationClass() {
<b>return</b> Data.<b>class</b>;
}
@Override
<b>public void</b> transform(<span style="color: DarkRed">CtClass</span> cl) {
<b>try</b> {
<b>if</b> (!cl.<span style="color: DarkRed">isInterface</span>()) {
<b>for</b> (<span style="color: DarkRed">CtField</span> field : cl.<span style="color: DarkRed">getDeclaredFields</span>()) {
String camelCaseField = field.getName().substring(0, 1).toUpperCase()
+ field.getName().substring(1);
if (!hasMethod(<span style="color: blue;">"get"</span> + camelCaseField, cl)) {
cl.<span style="color: DarkRed">addMethod</span>(<span style="color: DarkRed">CtNewMethod.getter</span>(<span style="color: blue;">"get"</span> + camelCaseField, field));
}
if (!hasMethod(<span style="color: blue;">"set"</span> + camelCaseField, cl)) {
cl.<span style="color: DarkRed">addMethod</span>(<span style="color: DarkRed">CtNewMethod.setter</span>(<span style="color: blue;">"set"</span> + camelCaseField, field));
}
}
}
} <b>catch</b> (<span style="color: DarkRed">CannotCompileException</span> e) {
e.printStackTrace();
<b>throw new</b> RuntimeException(e);
}
}
<span style="color: green;">/** javassist has unfortunately no hasMethod API */</span>
<b>private static boolean</b> hasMethod (String methodName, <span style="color: DarkRed">CtClass</span> cl) {
<b>try</b> {
cl.<span style="color: DarkRed">getDeclaredMethod</span>(methodName);
<b>return true</b>;
} <b>catch</b> (<span style="color: DarkRed">NotFoundException</span> e) {
<b>return false</b>;
}
}
}
</pre>
<p>
The <code>DataTransformer</code> accepts classes declaring the <code>Data</code> annotation :
</p>
<pre>
<b>package</b> ch.niceideas.common.enhancer.annotations;
<span style="color: green;">/**
* Classes declaring this annotation will have getters and setters automatically
* generated
*/</span>
<b>public @interface</b> Data {
}
</pre>
<p>
Third Class Transformer : generates the <code>toString</code> method for classes declaring the <code>@ToString</code> annotation.
</p>
<p>
<b>[Class <code>DataTransformer</code>]</b>
</p>
<pre>
<b>package</b> ch.niceideas.common.enhancer.impls;
<b>import</b> ch.niceideas.common.enhancer.ClassTransformer;
<b>import</b> ch.niceideas.common.enhancer.annotations.ToString;
<b>import</b> <span style="color: DarkRed">javassist.*</span>;
<b>import</b> java.lang.annotation.Annotation;
<span style="color: green;">/**
* This transformer accepts classes declaring the "@ToString" annotation.
* <br />
* It generates a toString method dynamically. The toString method is generated using
* bytecode manipulation and avoids reflection.
*/</span>
<b>public class</b> ToStringTransformer <b>extends</b> AbstractTransformer <b>implements</b> ClassTransformer {
@Override
<b>protected</b> Class<? extends Annotation> getAnnotationClass() {
<b>return</b> ToString.<b>class</b>;
}
@Override
<b>public void</b> transform(<span style="color: DarkRed">CtClass</span> cl) {
<b>try</b> {
<b>if</b> (!cl.<span style="color: DarkRed">isInterface</span>()) {
StringBuilder bb = <b>new</b> StringBuilder(<span style="color: blue;">"{\n"</span>);
bb.append(<span style="color: blue;">" StringBuilder sb = new StringBuilder(\""</span>
+ cl.getName() + <span style="color: blue;">"\");\n"</span>);
bb.append(<span style="color: blue;">" sb.append(\"[\");\n"</span>);
<b>for</b> (<span style="color: DarkRed">CtField</span> field : cl.<span style="color: DarkRed">getDeclaredFields</span>()) {
field.<span style="color: DarkRed">setModifiers</span>(Modifier.PUBLIC); <span style="color: green;">// hacky hack</span>
bb.append(<span style="color: blue;">" sb.append(\""</span> + field.<span style="color: DarkRed">getName</span>() + <span style="color: blue;">"\");\n"</span>);
bb.append(<span style="color: blue;">" sb.append(\"=\");\n"</span>);
bb.append(<span style="color: blue;">" sb.append(this."</span> + field.<span style="color: DarkRed">getName</span>() + <span style="color: blue;">");\n"</span>);
bb.append(<span style="color: blue;">" sb.append(\" \");\n"</span>);
}
bb.append(<span style="color: blue;">" sb.append(\"]\");\n"</span>);
bb.append(<span style="color: blue;">" return sb.toString();\n"</span>);
bb.append(<span style="color: blue;">"}"</span>);
<b>try</b> {
<span style="color: DarkRed">CtMethod</span> toStringMethod = cl.<span style="color: DarkRed">getDeclaredMethod</span>(<span style="color: blue;">"toString"</span>);
toStringMethod.<span style="color: DarkRed">setBody</span>(bb.toString());
} <b>catch</b> (<span style="color: DarkRed">NotFoundException</span> e) {
<span style="color: DarkRed">CtMethod</span> newMethod = <span style="color: DarkRed">CtNewMethod.make</span>(<span style="color: blue;">"public String toString() \n"</span>
+ bb.toString(), cl);
cl.<span style="color: DarkRed">addMethod</span>(newMethod);
}
}
} <b>catch</b> (<span style="color: DarkRed">CannotCompileException</span> e) {
e.printStackTrace();
<b>throw new</b> RuntimeException (e);
}
}
}
</pre>
<p>
The <code>ToStringTransformer</code> accepts classes declaring the <code>ToString</code> annotation :
</p>
<pre>
<b>package</b> ch.niceideas.common.enhancer.annotations;
<span style="color: green;">/**
* Classes declaring this annotation will have a toString method generated
* automagically
*/</span>
<b>public @interface</b> ToString {
}
</pre>
<a name="sec435"></a>
<h4>4.3.5 Test Class Example</h4>
<p>
A test class for the <code>DataTransformer</code> for instance, with the usage of a nice library to test agents: ElectronicArts AgentLoader.
</p>
<p>
EA Agent Loader is a collection of utilities for java agent developers. It allows programmers to write and test their java agents using dynamic agent loading (without using the -javaagent jvm parameter).
</p>
<pre>
<b>package</b> ch.niceideas.common.enhancer;
<b>import</b> ch.niceideas.common.enhancer.testData.TestData;
<b>import</b> com.ea.agentloader.AgentLoader;
<b>import</b> junit.framework.TestCase;
<b>import</b> org.apache.log4j.Logger;
<b>import</b> org.junit.Before;
<b>import</b> org.junit.Test;
<b>import</b> java.lang.reflect.Method;
<b>public class</b> DataTest <b>extends</b> TestCase{
<b>private static final</b> Logger logger = Logger.getLogger(DataTest.<b>class</b>);
@Before
<b>public void</b> setUp() <b>throws</b> Exception {
AgentLoader.loadAgentClass(EnhancerAgent.<b>class</b>.getName(), <span style="color: blue;">""</span>);
}
@Test
<b>public void</b> testDataTransformer() <b>throws</b> Exception {
TestData testData = <b>new</b> TestData();
Method getI = TestData.<b>class</b>.getDeclaredMethod(<span style="color: blue;">"getI"</span>);
assertEquals (0, getI.invoke(testData));
Method getMyString = TestData.<b>class</b>.getDeclaredMethod(<span style="color: blue;">"getMyString"</span>);
assertEquals (<span style="color: blue;">"abc"</span>, getMyString.invoke(testData));
Method getValue = TestData.<b>class</b>.getDeclaredMethod(<span style="color: blue;">"getValue"</span>);
assertEquals (-1L, getValue.invoke(testData));
Method setI = TestData.<b>class</b>.getDeclaredMethod(<span style="color: blue;">"setI"</span>, <b>int</b>.<b>class</b>);
setI.invoke(testData, 9);
assertEquals (9, getI.invoke(testData));
Method setMyString = TestData.<b>class</b>.getDeclaredMethod(<span style="color: blue;">"setMyString"</span>, String.<b>class</b>);
setMyString.invoke(testData, <span style="color: blue;">"xyz"</span>);
assertEquals (<span style="color: blue;">"xyz"</span>, getMyString.invoke(testData));
Method setValue = TestData.<b>class</b>.getDeclaredMethod(<span style="color: blue;">"setValue"</span>, <b>long</b>.<b>class</b>);
setValue.invoke(testData, 999L);
assertEquals (999L, getValue.invoke(testData));
}
}
</pre>
<p>
This test case uses the following test data :
</p>
<pre>
<b>package</b> ch.niceideas.common.enhancer.testData;
<b>import</b> ch.niceideas.common.enhancer.annotations.Data;
<span style="color: green;">/**
* a test Data for the DataTest test case
*/</span>
@Data
<b>public class</b> TestData {
<b>private int</b> i = 0;
<b>private</b> String myString = <span style="color: blue;">"abc"</span>;
<b>private</b> long value = -1;
}
</pre>
<a name="sec5"></a>
<h2>5. Conclusion </h2>
<p>
Again. Bytecode manipulation opens a whole lot of new possibilities for the JVM and is key to address one of the biggest weakness of the JVM: the overwhelming verbosity of the language. The approaches and techniques presented above are extensively used by so many libraries and frameworks that have become the facto standard nowadays: AspectJ, Spring, Hibernate, Jprofiler, etc.
<br>
For that reason, and because it's really a lot of fun, one might find bytecode manipulation a pretty valuable mechanism to master.
</p>
<p>
In addition, even though Lombok uses for very good reasons a different technique (AST Transformation), I find it astonishing to see how bytecode manipulation enables a developer to mimic its features in so few lines of code.
</p>
<p>
One can use bytecode manipulation to perform many tasks that would be difficult or impossible to do otherwise, and once one learns it, the sky is the limit.
</p>
<p>
Javassist put this power in hands of every Java developer in a simple, intuitive and efficient way.
</p>
<p>
You might want to have a look at the first article in this serie available here : <a href="https://www.niceideas.ch/roller2/badtrash/entry/bytecode-manipulation-with-javassist-for">Bytecode manipulation with Javassist for fun and profit part I: Implementing a lightweight IoC container in 300 lines of code</a>.
</p>
<p>
Part of this article is available as a slideshare presentation here: <a href="https://www.slideshare.net/JrmeKehrli/bytecode-manipulation-with-javassist-for-fun-and-profit">https://www.slideshare.net/JrmeKehrli/bytecode-manipulation-with-javassist-for-fun-and-profit</a>.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/the-digitalization-challenge-and-opportunities
The Digitalization - Challenge and opportunities for financial institutions
Jerome Kehrli
2017-03-21T16:52:55-04:00
2017-09-13T16:42:30-04:00
<p>
A few weeks ago, I did a speech about <i>the Digitalization</i> and its impact on financial institutions, both in terms of <i>challenges</i> and <i>opportunities</i> in the context of my role as Head of R&D in my current company.
<br>
I am reporting here my speech as an article.
</p>
<p>
Even though the Digitalization and its impacts is something so widely discussed and studied nowadays, even in banking institutions, I still find it puzzling that so many of them struggle following the pace.
<br>
Having said that, many others on the other hand have well understood how much technology is about to disrupt the banking business just as Uber has disrupted the transportation business and AirBnB the lodging business and many good and enlightening initiatives start to flourish in the news.
</p>
<p>
But still, it seems to me that most innovations in banking are really coming from small players or even startups - think of fintechs - instead of coming from the big players of the banking industry. For instance, paying everything with a cellphone is a thing for a few years now in many African countries while it's not at all in Europe, even in Switzerland, THE country of banking.
<br>
Especially in Switzerland, financial institutions struggle keeping up with evolution of their business coming from the digitalization on one side and the regulatory pressure as well as the reduction of the margins on the other side.
<br>
Discussing this very matter further exceeds the scope of this article of course but I want to report below my speech notes and present what I see as the most important challenges and opportunities for the banking industry coming from the digitalization.
</p>
<!-- The Digitalization - Challenge and opportunities for financial institutions -->
<p>
A few weeks ago, I did a speech about <i>the Digitalization</i> and its impact on financial institutions, both in terms of <i>challenges</i> and <i>opportunities</i> in the context of my role as Head of R&D in my current company.
<br>
I am reporting here my speech as an article.
</p>
<p>
Even though the Digitalization and its impacts is something so widely discussed and studied nowadays, even in banking institutions, I still find it puzzling that so many of them struggle following the pace.
<br>
Having said that, many others on the other hand have well understood how much technology is about to disrupt the banking business just as Uber has disrupted the transportation business and AirBnB the lodging business and many good and enlightening initiatives start to flourish in the news.
</p>
<p>
But still, it seems to me that most innovations in banking are really coming from small players or even startups - think of fintechs - instead of coming from the big players of the banking industry. For instance, paying everything with a cellphone is a thing for a few years now in many African countries while it's not at all in Europe, even in Switzerland, THE country of banking.
<br>
Especially in Switzerland, financial institutions struggle keeping up with evolution of their business coming from the digitalization on one side and the regulatory pressure as well as the reduction of the margins on the other side.
<br>
Discussing this very matter further exceeds the scope of this article of course but I want to report below my speech notes and present what I see as the most important challenges and opportunities for the banking industry coming from the digitalization.
</p>
<p>
Part of this article is available as a slideshare presentation here: <a href="https://www.slideshare.net/JrmeKehrli/digitalization-a-challenge-and-an-opportunity-for-banks">https://www.slideshare.net/JrmeKehrli/digitalization-a-challenge-and-an-opportunity-for-banks</a>.
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. Introduction</a>
<ul>
<li><a href="#sec11">1.1 An evolving society</a></li>
<li><a href="#sec12">1.2 Some Facts</a></li>
<li><a href="#sec13">1.3 More Facts</a></li>
<li><a href="#sec14">1.4 Today</a></li>
<li><a href="#sec15">1.5 Tomorrow</a></li>
<li><a href="#sec16">1.6 Big Data</a></li>
<li><a href="#sec17">1.7 Digitalization</a></li>
</ul>
</li>
<li><a href="#sec2">2. Financial Services ARE impacted !</a>
<ul>
<li><a href="#sec21">2.1 The rise of Mobile Banking</a></li>
<li><a href="#sec22">2.2 The rise of Alternative Financing</a></li>
<li><a href="#sec23">2.3 A whole new world for banking</a></li>
<li><a href="#sec24">2.4 Adapt ... or Vanish</a></li>
</ul>
</li>
<li><a href="#sec3">3. Some definitions</a>
<ul>
<li><a href="#sec31">3.1 Digitalization</a></li>
<li><a href="#sec32">3.2 Digital Transformation</a></li>
</ul>
</li>
<li><a href="#sec4">4. Challenges and Opportunities for Banking Institutions</a>
<ul>
<li><a href="#sec41">4.1 Challenges</a></li>
<li><a href="#sec42">4.2 Opportunities</a></li>
</ul>
</li>
<li><a href="#sec5">5. Conclusion</a></li>
</ul>
<a name="sec1"></a>
<h2>1. Introduction</h2>
<a name="sec11"></a>
<h3>1.1 An evolving society</h3>
<p>
Our society is evolving.
</p>
<p>
Yesterday - in 2008, we were amazed by the first smartphones. Today they have almost become a part of ourselves. We cannot go without them anymore.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/fd28f608-8d61-4a72-a2e9-55ce5e6e2501">
<img class="centered" style="width: 720px; " alt="Kanban States" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/fd28f608-8d61-4a72-a2e9-55ce5e6e2501" />
</a>
</div>
<br>
<p>
Today everybody has in his pocket a computer that is more powerful by several order of magnitudes that the computer that sent the first people to the moon.
<br>
Later, 30 years ago, we had computers still way less powerful than an iPhone that were fitting an entire room, today an iPhone stands in our pocket.
</p>
<p>
Is it the biggest invention of the decade? Likely, but the previous decade, not the current one. I'll get back to that.
</p>
<p>
Nowadays, new technologies emerge first in the consumer market and then spread into business. New solutions emerge every month and corporations cannot keep up the pace.
<br>
This new reality has a name: it's the consumerization.
</p>
<p>
The consumerization has a consequence. Increasingly, the trend is to hire employees with their devices and applications. This is the BYOD for "Bring Your Own Device" trend. It comes from the fact that employees are more comfortable and more efficient with their own devices.
</p>
<p>
The direct consequence of the consumerization is a use of a mix of professional and personal tools by employees (Office Suite, Gmail, Google+, Twitter, Facebook, Dropbox, Evernote, ...)
<br>
Nowadays many companies are still blocking access to these personal tools from their employees, mostly financial institutions I have to say.
<br>
Tomorrow, that won't be possible anymore.
</p>
<p>
People are used to be connected all the time, with highly efficient devices on highly responsive services, everywhere and for every possible need.
</p>
<a name="sec12"></a>
<h3>1.2 Some Facts</h3>
<p>
Global sales of PCs never really exploded.
<br>
On the other hand, Global sales of smartphones and tablets explodes.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/b82a3de1-3b4d-4111-87f5-218d543ad6e5">
<img class="centered" style="width: 600px;" alt="Global Unit Shipments" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b82a3de1-3b4d-4111-87f5-218d543ad6e5" />
</a>
</div>
<br>
<p>
Global Mobile traffic went from 1% in 2009 to 4% in 2010 and 12% in 2012. Today it reaches 40%. In 2020, Global Mobile Traffic will exceed fix PCs Internet Traffic.
</p>
<p>
This may be hard to understand for people in Europe or the US for instance. But look at India for instance: the wired telecommunication infrastructure there could never be developed as it could be in Europe or in the US. There, the mobile traffic already exceeded the Desktop traffic in 2012.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a2c7a97c-5f3f-4599-b96f-7d5bcae45b57">
<img class="centered" style="width: 650px; " alt="Global Internet Traffic Flow Ratio" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a2c7a97c-5f3f-4599-b96f-7d5bcae45b57" />
</a>
</div>
<br>
<p>
Global sales of smartphones and tablets has exploded!
</p>
<p>
In 2017, over 3 billion people will be connected all the time, everywhere and for every kind of needs.
</p>
<a name="sec13"></a>
<h3>1.3 More Facts</h3>
<p>
We look at our smartphones 150 times a day.
<br>
We are using our smartphones all the time, even when watching another media.
</p>
<p>
Even when watching TV, we cannot refrain from using a connected device at the same time, either a smartphone or a tablet.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/bd785924-4aaa-40b0-b8ec-76a8b84a4a39">
<img class="centered" style="width: 550px; " alt="Tablets and Smartphone while watching TV" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/bd785924-4aaa-40b0-b8ec-76a8b84a4a39" />
</a>
</div>
<br>
<p>
As a funny note, men and women are using their smartphone or tablet while watching TV for significantly similar reasons.
<br>
But there are 2 exceptions:
</p>
<ul>
<li>Looking at sport results on an iPad while watching TV seems to be rather a man thing.</li>
<li>On the other hand, looking at Facebook feeds while watching TV seems to be rather a woman thing.</li>
</ul>
<a name="sec14"></a>
<h3>1.4 Today</h3>
<p>
I cannot stress enough how much what we are experiencing from a few years is important and what it means in terms of change of society.
</p>
<div class="centering">
<a style="border: 0px none;" href="https://www.niceideas.ch/roller2/badtrash/mediaresource/2698bfab-bf8c-4887-9f0b-8d57f8a57c9f">
<img class="centered" style="width: 550px; border: 0px none;" alt="Today" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/2698bfab-bf8c-4887-9f0b-8d57f8a57c9f" />
</a>
</div>
<br>
<p>
Today, <b>we are inter-connected on different kind of medias, during a continuous time and for every possible need</b>. This has become a part of the human behaviour. In a few years (OK maybe a little more) the majority of the workforce will be composed by millennials, by people almost born with an iPhone.
<br>
At that moment, exchanging data on Internet all the time and for every possible need will seem to people to be as natural as breathing.
</p>
<p>
But this is today ...
</p>
<a name="sec15"></a>
<h3>1.5 Tomorrow</h3>
<p>
Tomorrow there will be dozens of billions of additional sources, in the form of smart devices connected on internet and exchanging data.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/51fd431e-f4a4-4212-9957-546776d0e3f5">
<img class="centered" style="width: 550px;" alt="Tomorrow" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/51fd431e-f4a4-4212-9957-546776d0e3f5" />
</a>
</div>
<br>
<p>
The <i>Internet of Things</i> - or IoT - refers to "<i>uniquely identifiable objects and their interconnection on internet, as well as their automatic exchange of information with third party services</i>".
</p>
<p>
Dr. Henrik Christensen, Professor of Computer Science and the Chair of Robotics at the Georgia Institute of Technology, said, not long ago: <i>"My current prediction is that kids born today will never have to drive a car"</i>.
</p>
<p>
There are 3 billion people connected in 2017 and exchanging data on Internet.
<br>
Gartner thinks there will be 26 billion devices on the Internet of Things by 2020.
<br>
ABI Research is even more optimistic and claims that 30 billion devices will be wirelessly connected to the Internet of Things by 2020.
</p>
<div class="centering">
<div class="centered">
<b>
The internet of things is the coming big thing!
</b>
</div>
</div>
<br>
<p>
The "<i>internet of people</i>" and the "<i>Internet of Things</i>" form the "<i>Internet of Everything</i>".
</p>
<p>
Cisco defines the Internet of Everything (IoE) as follows <i>"The Internet of Everything brings together people, process, data, and things to make networked connections more relevant and valuable than ever before-turning information into actions that create new capabilities, richer experiences, and unprecedented economic."</i>
</p>
<p>
The Internet of Everything is the coming evolution from the interconnection of people and objects, always, all the time, everywhere and for every possible need.
</p>
<a name="sec16"></a>
<h3>1.6 Big Data</h3>
<p>
Since we started estimating and measuring the amount of produced data until 2003, 5 exabytes (5 billions gigabytes) have been produced.
<br>
In 2011, this quantity was generated in 2 days (think of facebook, twitter, google search logs, financial transaction logs, etc.).
<br>
In 2014, this quantity is generated in 10 minutes.
<br>
Today it is likely generated in a few minutes.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/882990de-6894-43d4-9f0e-ef2f97ff3742">
<img class="centered" style="width: 700px;" alt="In 60 seconds" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/882990de-6894-43d4-9f0e-ef2f97ff3742" />
</a>
</div>
<br>
<p>
Not only do we generate more and more data but today, thanks to <b>Big Data technologies</b>, we have the means and the technology to analyze, exploit and mine this data to extract meaningful business insights
</p>
<p>
The data generated by the company's own systems, such as logs, user and customer activity tails, etc. can be a very interesting source of information regarding customer behaviours, profiles, trends, desires, etc.
<br>
But <i>Big Data</i> becomes really relevant only when one also considers the <i>data external to the corporation</i>, such as facebook status feeds, twitter logs, linkedin news, etc.
</p>
<p>
Today, a whole lot of additional sources are available to corporations to gather business insights related to market trends.
</p>
<a name="sec17"></a>
<h3>1.7 Digitalization</h3>
<p>
To summarize. what we are experiencing today:
</p>
<ul>
<li>the digitalization of the masses,</li>
<li>the era of power,</li>
<li>the availability of massive amount of data,</li>
<li>more importantly, the ability to analyze and use this data,</li>
<li>the always and everywhere interconnection of people,</li>
<li>the internet of things and</li>
<li>the coming internet of everything,</li>
</ul>
<p>
must lead corporation to adapt.
</p>
<div class="centering">
<div class="centered" style="width: 350px; border: 1px solid #555555;"><b>
The digitalization urges corporations to
<br>
search for new business models and
<br>
rethink their operating model
</b>
</div>
</div>
<br>
<a name="sec2"></a>
<h2>2. Financial Services ARE impacted !</h2>
<p>
Now when I run this speech in financial institutions, it happens sometimes that I hear comments such as <i>"Yeah, well, all of this makes surely a lot of sense for fancy internet companies. But we're a bank here. We're doing serious business, we're not Facebook."</i>
</p>
<p>
I'm always puzzled by this kind of reactions because in my opinion, serious businesses such as banking institutions are in contrary on the front line when it comes to meeting the digitalization challenges.
<br>
It's now wonder fintechs have become such a thing and are increasingly eating the banking business.
</p>
<p>
Think of millennials, think of these people that are almost born with a tablet or a smartphone in their hands.
<br>
My father, used to go to the banking institutions that was closest from where he lived and where he worked. That's how he made his choice.
<br>
I myself, I have chosen my banking institutions at the time I was a student. My choice was driven by the conditions that banks were giving to students, such as a free credit card, no additional costs, etc. Then I simply remain loyal to my first choice.
<br>
Millennials choice will be different. They will choose the bank that provides them with the best online experience. That will be their main driver. For these people born with twitter, facebook and all these fancy online services, it will simply seem impossible to have to physically go to a branch to perform whatever operation they will need related to their banking account, including its initial opening.
</p>
<p>
Let me give you some examples ...
</p>
<p>
<a name="sec21"></a>
<h3>2.1 The rise of Mobile Banking</h3>
<p>
Only twenty years ago, we had to move to a physical location - a branch - to perform any kind of financial transaction, such as simply paying a phone bill.
<br>
What is the situation today?
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/2c1ba975-0ad1-4c84-b634-8d50bfc12314">
<img class="centered" style="width: 500px;" alt="Wells Fargo is closing branches" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/2c1ba975-0ad1-4c84-b634-8d50bfc12314" />
</a><br>
<div class="centered">
(Source : <a href="http://money.cnn.com/2017/01/13/investing/wells-fargo-branch-closures/">http://money.cnn.com/2017/01/13/investing/wells-fargo-branch-closures/</a>)
</div>
</div>
<br>
<p>
Wells Fargo announced plans earlier this year to close over 400 branches in the US. The tendency of big financial institutions is to close physical branches at an unprecedented pace, as a reflection of people's preferences for online and mobile banking.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/5ae3da06-709c-437d-8e10-738430033a81">
<img class="centered" style="width: 500px;" alt="Mobile Banking is killing the branch" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/5ae3da06-709c-437d-8e10-738430033a81" />
</a><br>
<div class="centered">
(Source : <a href="https://thefinanser.com/2013/06/more-debate-about-the-shift-to-mobile.html/">https://thefinanser.com/2013/06/more-debate-about-the-shift-to-mobile.html/</a>)
</div>
</div>
<br>
<p>
In online banking there is also a clear tendency from 2012 onward. Mobile banking usage skyrocks while fixed Internet banking is stagnating or even slightly shrinking.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/95814438-6f2f-449a-a890-025e6d49a3b5">
<img class="centered" style="width: 650px;" alt="Goldman Sachs and End of Traditional Banking" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/95814438-6f2f-449a-a890-025e6d49a3b5" />
</a><br>
<div class="centered">
(Source : <a href="https://www.forbes.com/sites/oliviergarret/2017/02/22/goldman-sachs-recent-move-marks-the-end-of-traditional-banking/#32be5e5069aa">https://www.forbes.com/sites/oliviergarret/2017/02/22/goldman-sachs-recent-move-marks-the-end-of-traditional-banking/#32be5e5069aa</a>)
</div>
</div>
<br>
<p>
On one hand, rising compliance costs and restrictive regulations is the new normal. This forces banks to increase operational efficiency at all costs, which is pretty difficult when regulation related costs tend to explode.
<br>On the other hand, reduced margins and increased costs is forcing banking institutions to adapt. Growing the Investment management business line is a relevant approach of course but diversifying earnings with new retail banking initiatives aimed at ensuring a first place on the online banking market is mandatory.
</p>
<div class="centering">
<div class="centered">
<b>The digitalization is shaking the fundamentals of the banking business.</b>
</div>
</div>
<br>
<a name="sec22"></a>
<h3>2.2 The rise of Alternative Financing</h3>
<p>
Here as well, twenty years ago there was no fintech, no crowdfunding.
<br>
Today the banking business is increasingly eaten by <i>fancy technology companies</i> (fintechs).
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/0eaabe23-cf77-4390-a961-87c4ae61fbcd">
<img class="centered" style="width: 650px;" alt="Crowdfunding disrupts Traditional Banking" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/0eaabe23-cf77-4390-a961-87c4ae61fbcd" />
</a><br>
<div class="centered">
(Source : <a href="http://www.mergersandinquisitions.com/future-of-investment-banking-2015/">http://www.mergersandinquisitions.com/future-of-investment-banking-2015/</a>)
</div>
</div>
<br>
<p>
Just as technology as disrupted the transportation business with Uber, the lodging business with AirBnB, the consumer lending business with so many peer-to-peer lending platforms available, Technology is about to disrupt investment banking.
<br>
This is happening.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/87f8d9e9-6959-41b2-90e3-3e211fb8894b">
<img class="centered" style="width: 600px;" alt="Alternatives Financing Models Volumes" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/87f8d9e9-6959-41b2-90e3-3e211fb8894b" />
</a><br>
<div class="centered">
(Source : <a href="https://www.crowdfundinsider.com/2015/02/63013-moving-mainstream-centre-for-alternative-finance-of-cambridge-judge-business-school-publishes-european-alternative-finance-benchmarking-report/">https://www.crowdfundinsider.com/2015/02/63013-moving-mainstream-centre-for-alternative-finance-of-cambridge-judge-business-school-publishes-european-alternative-finance-benchmarking-report/</a>)
</div>
</div>
<br>
<p>
Alternative financing models are progressing throughout the business lines. Peer-2-Peer consumer lending, Crowdfunding, Peer-2-Peer business lending, all of them are exploding over the past years.
<br>
This chart shows the situation in Europe but the worldwide situation is pretty similar.
<br>
The important information here is that the volume of peer-to-peer and crowd financing is doubling every year since 2012.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/0fffce6c-b045-4018-9322-76d1f8d6393a">
<img class="centered" style="width: 750px;" alt="Alternative Financing Landscape" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/0fffce6c-b045-4018-9322-76d1f8d6393a" />
</a><br>
<div class="centered">
(Source : <a href="http://www.kaplancollectionagency.com/resource-center/alternative-financing-landscape/">http://www.kaplancollectionagency.com/resource-center/alternative-financing-landscape/</a>)
</div>
</div>
<br>
<p>
This is the landscape of alternative financing firms and startups. More companies are appearing every month, almost every week.
<br>
If you look at the global fintech landscape, you can multiply the count of companies here by 20.
</p>
<p>
Fintechs are eating the banking business.
<br>
Banking institutions need to adapt, or eventually a lot of them will disappear.
</p>
<a name="sec23"></a>
<h3>2.3 A whole new world for banking</h3>
<p>
Back in 2010 nobody ever heard that word. Only very high tech or finance specialists were aware of any crypto-currency.
<br>
Today everybody knows bitcoin and most people heard about blockchain.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/be08346a-0205-42d8-b466-80cd7fde8e0f">
<img class="centered" style="width: 750px;" alt="Bitcoin Transactions" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/be08346a-0205-42d8-b466-80cd7fde8e0f" />
</a><br>
<div class="centered">
(Source : <a href="https://blog.bitpay.com/understanding-bitcoins-growth-in-2015/">https://blog.bitpay.com/understanding-bitcoins-growth-in-2015/</a>)
</div>
</div>
<br>
<p>
While the blockchain technology is not ready yet to completely replace the trust third party, it has the potential to disrupt the very root of the worldwide financial system.
<br>
Happily here financial institutions have understood this from the beginning and a lot of blockchain initiatives nowadays are led by big financial institutions.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/7cb67999-0d3f-4d45-be4f-9f799748d7fb">
<img class="centered" style="width: 500px;" alt="Blockchain disrupt banking" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/7cb67999-0d3f-4d45-be4f-9f799748d7fb" />
</a><br>
<div class="centered">
(Source : <a href="http://www.cxotoday.com/story/blockchain-to-disrupt-banking-industry-by-2020-infosys/">http://www.cxotoday.com/story/blockchain-to-disrupt-banking-industry-by-2020-infosys/</a>)
</div>
</div>
<br>
<p>
I focused a lot on how the digitalization is challenging the banking business.
<br>
But the blockchain technology is a good example that there are opportunities as well.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/483cf503-86d0-46aa-a26c-34096fc734cf">
<img class="centered" style="width: 600px;" alt="Royal Bank Scotland has bough Watson " src="https://www.niceideas.ch/roller2/badtrash/mediaresource/483cf503-86d0-46aa-a26c-34096fc734cf" />
</a><br>
<div class="centered">
(Source : <a href="http://www.prnewswire.com/news-releases/royal-bank-of-scotland-engages-ibm-watson-for-cognitive-insights-to-better-serve-customers-300340461.html">http://www.prnewswire.com/news-releases/royal-bank-of-scotland-engages-ibm-watson-for-cognitive-insights-to-better-serve-customers-300340461.html</a>)
</div>
</div>
<br>
<p>
Another interesting example. Royal Bank of Scotland engaged IBM Watson to take care of the simplest customer requests coming from online channels, as a way to enhance operational efficiency.
<br>
I myself am not necessarily a big fan of IBM or Watson but this is actually a pretty sound use case for Watson.
<br>
And in any case it's a brilliant example of how technology can help banking institutions increase operational efficiency.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d39deab7-6173-4305-898a-5f4b1934ea76">
<img class="centered" style="width: 500px;" alt="Twitter to predict Stock Market" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d39deab7-6173-4305-898a-5f4b1934ea76" />
</a><br>
<div class="centered">
(Source : <a href="http://www.dailymail.co.uk/sciencetech/article-2120416/Twitter-predicts-stock-prices-accurately-investment-tactic-say-scientists.html">http://www.dailymail.co.uk/sciencetech/article-2120416/Twitter-predicts-stock-prices-accurately-investment-tactic-say-scientists.html</a>)
</div>
</div>
<br>
<p>
A team in the university of California has designed a model aimed at predicting stock prices evolution by using statistics of tweets, sentiment analysis and relationship discovery algorithms.
<br>
Not only they are able to predict stock prices evolution one day in advance but they came up with a model much more accurate that pretty much every other initiative so far.
</p>
<p>
Technology, here Big Data, also offers unprecedented opportunities for financial institutions.
</p>
<a name="sec24"></a>
<h3>2.4 Adapt ... or Vanish</h3>
<p>
Again, both the evolution of means and the evolution of behaviours induced by the new technologies such as
</p>
<ul>
<li>the digitalization of the masses,</li>
<li>the Big Data technologies,</li>
<li>the internet of everything,</li>
<li>etc.</li>
</ul>
<p>
have strong consequences on corporations and the society in general.
</p>
<div class="centering">
<div class="centered" style="width: 350px; border: 1px solid #555555;"><b>
The digitalization
<br>
shakes the fundamentals of our society,
<br>
shocks our references and
<br>
revolutionizes our business models
</b>
</div>
</div>
<p>
Corporations need to adapt. The time is now or never.
</p>
<a name="sec3"></a>
<h2>3. Some definitions</h2>
<p>
I think the reader should now have a grasp on what I mean with the term <i>"digitalization"</i> so it's a good time to give a few formal definitions.
</p>
<a name="sec31"></a>
<h3>3.1 Digitalization</h3>
<p>
A first definition that I think is good, from Business Dictionary:
</p>
<div class="centering">
<div class="centered" style="width: 350px; border: 1px solid #555555;">
"The digitalization is
<br>
the <b>integration of digital technologies</b>
<br>
into <b>everyday life</b>
<br>
by the <b>digitization of everything</b>
<br>
that can be digitized"
</div>
<br>
<div class="centered">
(Source : BusinessDictionary - <a href="http://www.businessdictionary.com/definition/digitalization.html">http://www.businessdictionary.com/definition/digitalization.html</a>)
</div>
</div>
<br>
<p>
This is happening today: everything that can be digitized, either is digitized or is getting digitized.
<br>
I realized recently that I myself haven't written anything down on paper for a pretty long time.
<br>
I use Internet to do my payments, book my holidays or business trips, search for a phone number.
<br>
I take notes on my laptop or my smartphone.
<br>
I even take my medical appointments using email.
</p>
<p>
The digitalization is the increasing integration of digital technologies into everyday life.
<br>
Corporations need to adapt their <i>business models</i> and <i>operating model</i> to follow this trend. They need to transform their business and make it suited to the digital era.
</p>
<p>
The following is the definition of the digitalization from OCTO Technology. I think it's brilliant.
</p>
<div class="centering">
<div class="centered" style="width: 350px; border: 1px solid #555555;">
"The digitalization is
<br>
the <b>impact</b> on corporations and organizations
<br>
of the fact that
<br>
<b>people</b> and <b>things</b>
<br>
are <b>always</b> and <b>everywhere inter-connected
<br>
for every possible need</b>."
</div>
<div class="centered">
<br>
(Source : OCTO Technology - <a href="http://blog.octo.com/digitalisation-une-definition/">http://blog.octo.com/digitalisation-une-definition/</a>)
</div>
</div>
<br>
<p>
What interests us today is the impact of the digitalization on corporations and in this context the definition from OCTO Technology is crystal clear, accurate and most relevant.
</p>
<p>
Another way to put it would be:
<br>
The digitalization is the impact on enterprises and organizations of the Internet of Everything.
</p>
<a name="sec32"></a>
<h3>3.2 Digital Transformation</h3>
<p>
One definition remains: what is the <i>digital transformation</i>?
<br>
I could not find any easy way to present the notion of digital transformation as a one sentence.
</p>
<p>
Instead, I find the following schematic most relevant in presenting what is the digital transformation.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/c31a6fbd-391e-4992-9f1c-dcbd5a1f62ff">
<img class="centered" style="width: 800px;" alt="Digital Transformation" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/c31a6fbd-391e-4992-9f1c-dcbd5a1f62ff" />
</a>
</div>
<br>
<p>
This square is a corporation. The orange form represent its organization, its processes and its culture.
</p>
<p>
The most recent technological evolutions, influence the society and causes an evolutions both of <b>means</b> and <b>behaviours</b>.
</p>
<ul>
<li>In terms of means, think of the always and everywhere interconnection of people and things, the consummerization, the new businesses such as crowdfunding and crowdsourcing, The availability of massive amount of data and the ability to analyze it, etc.
</li>
<li>
In terms of behaviour, think of the increasing digital literacy of people, the digital natives, all the behaviour changes brought by social networks and the increasingly connected world and real-time communication means. People want everything, now and tailor made.
</li>
</ul>
<p>
This evolution of our society as a whole forces the corporation to transform its operating model, adapt its organization and its culture.
</p>
<div class="centering">
<div class="centered">
<b>
This is the 4th industrial revolution.
</b>
</div>
</div>
<br>
<p>
Corporations need to adapt three most essential aspects:
</p>
<ul>
<li>First, the internal organization of corporations needs to be adapted to match the responsiveness and dynamic required to design products in the digital era.
<br>
Key practices here are Agility and DevOps at every level in the company around the IT organization designing the digital products.
<br>
Management and hierarchy should also be adapted to enable low response time to market events and customer feedbacks.
<br>
Finally, every action and decision within the corporation should be taken with customer centricity in mind.
</li>
<li>
Entering the digital era requires a significant evolution of the culture of the company.
<br>
Lean Startup principle and practices should be embraces and a thorough Customer Development Process should accompany the Product development processes in place.
<br>
Also, in the digital era with shrinking margins and increasing complexity of products and regulations, operational efficiency should be a constant focus.
</li>
<li>
Finally, The marketing approach should evolve to meet the customer expectations in a digital world. Customers expect corporations to meet them where they are, in a mostly online world. Corporations already started the digital marketing move many years ago, but that is not sufficient.
<br>
In this ever more selfish world, people are looking for tailor-made. It's all about me, myself and I. Corporations need to consider this and thanks to a sound adoption of Lean Startup principles such multivariate tests, corporations have the possibility to provide customers with very customized solutions. Even further, it is nowadays common to implicate customers in the design of the product itself and even in the process of searching for new products to develop. Think co-creation and co-innovation.
</li>
</ul>
<p>
Interestingly, just as technology is <i>the driver</i> behind the evolutions forcing corporations to adapt, technology is also <i>the solution</i> to the transformation of corporations.
</p>
<p>
Corporations need to understand that they have no choice. They need to digitalize significant portions of their business and understand the central place that technology has to take.
<br>
Corporations that still believe today that IT and Technology is a center of cost instead of the key vector for innovation will eventually disappear. Whatever the industry, IT and Technology must be considered as a key investment and the most important vector of innovation.
</p>
<a name="sec4"></a>
<h2>4. Challenges and Opportunities for Banking Institutions</h2>
<p>
So all this definition part has been pretty generic.
<br>
I would want now to focus a little bit more on the digital transformation of financial institutions.
</p>
<p>
I would like to consider the digital transformations on two perspective:
</p>
<ul>
<li>First perspective concerns the challenges that the evolution of means and behaviour from the digital era is causing to financial institutions</li>
<li>Second perspective is the opportunities that the digital era is offering to financial institutions.</li>
</ul>
<a name="sec41"></a>
<h3>4.1 Challenges</h3>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a26eac9f-2809-4d72-a84a-0a95f30561f5">
<img class="centered" style="width: 750px;" alt="Challenges" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a26eac9f-2809-4d72-a84a-0a95f30561f5" />
</a>
</div>
<br>
<p>
<b>Competitiveness</b>
</p>
<p>
In the digital ear, comparisons and advising web sites flourish and new comparison services appear almost every month.
<br>
As an example, my father still buys his consumer electronic devices from the physical store closest from where he lives.
<br>
I myself, I am using an aggregation and comparison web site to buy my devices from the online shop offering me the cheapest price.
<br>
The same evolution that impacted consumer goods will eventually apply to absolutely every business including the banking business.
<br>
Banking institutions need to adopt a fair price policy and emphasize clarity and simplicity when designing their products.
</p>
<p>
<b>Customer satisfaction</b>
</p>
<p>
In a digital world, people wants everything immediately. In addition, reputation is very important and can be harmed in no time. People suffering from stolen credit cards will express themselves on social network and can harm the reputation of an institution that dealt badly with such a situation in no time.
</p>
<p>
<b>Customer centricity</b>
</p>
<p>
Today more than ever replacing enhancing a thorough product development process with a customer development process, meeting the customer where he is, focusing on needs and demands should be the core focus of financial institutions.
</p>
<p>
<b>Marketing and branding</b>
</p>
<p>
It's all about innovation and reputation. Design the best products, the most innovative ones, implement striking online and mobile services, communicate about them on the right channels and the chances they become viral are important.
<br>
At the same time, an anecdotal fact, discussed widely on the internet and becoming viral, can cause a lot more harm to a company than a bad balance sheet.
<br>
Again, making a difference in the digital era is all about innovation and reputation.
</p>
<p>
<b>Operation Efficiency</b>
</p>
<p>
With shrinking margins and increasing product development costs, corporations need to rethink the way they work. Tracking and eliminating waste should be part of the business processes used to run the company, not a side activity to be done once in a while when shareholders complain.
<br>
The key leads here are Process automation and the reduction of intervention delays.
</p>
<p>
<b>Risk Management and Mitigation</b>
</p>
<p>
In a digital world, the attack surface for cyber-criminal or simply thefts is much larger than in the traditional world. New channels, especially digital channels, to access the banking institutions products come with higher risks.
<br>
While a sampling approach to control and audit could be sufficient before, it is not the case anymore with the Digitalization. Banking institutions need to move their controls towards continuous, automated, comprehensive and real-time control and audit approaches.
<br>
State of the art fraud detection systems are not optional anymore.
</p>
<a name="sec42"></a>
<h3>4.2 Opportunities</h3>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/bb8db4df-9b84-4b2f-8e1d-b84254d61ce0">
<img class="centered" style="width: 750px;" alt="Opportunities" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/bb8db4df-9b84-4b2f-8e1d-b84254d61ce0" />
</a>
</div>
<br>
<p>
<b>Competitiveness</b>
</p>
<p>
In a digital world, technology can help putting in places platforms for <i>co-creation</i> and <i>co-innovation</i>, implicating the customers in both the identification of new products and their definition, which are ultimately the best way to handle innovation and the requirement for tailor made. In addition, digital products more than ever before can be made very customized.
</p>
<p>
<b>Customer satisfaction</b>
</p>
<p>
When processes and products are digitalized, achieving 24/7 availability is straightforward. A computer doesn't sleep. In addition, communication about fair price strategies is easier when the catalog of products is available online. Finally, even customer follow-up processes can be automated.
</p>
<p>
<b>Customer centricity</b>
</p>
<p>
The digitalization requires financial institutions to meet the customer on his preferred channels, but is also provides them with the means to do so. Getting online and digital, from a purely technological perspective, is not difficult. Changing the organization of the corporation to achieve the digitalization is the difficult part.
</p>
<p>
<b>Marketing and branding</b>
</p>
<p>
It's all about innovation and reputation.
<br>
The digital world offers unprecedented opportunities to convince a customer to buy a product. Think of trial systems, demo systems, sandboxes...
<br>
In addition, digital marketing is now a field on its own.
</p>
<p>
<b>Operation Efficiency</b>
</p>
<p>
Technology is also key to achieve <i>Operational Efficiency</i>. New products or technologies aimed at moving to a paperless corporation and de-materializing processes appear every month, not to say every week. Think of digital signatures, responsive interfaces, etc.
</p>
<p>
<b>Risk Management and Mitigation</b>
</p>
<p>
Here as well, most recent technologies such as Big Data Analytics, Machine Learning and real-time processing systems offer unprecedented opportunities to move towards continuous, automated, comprehensive and real-time control and audit approaches.
<br>
In addition, web technologies have significantly progressed over the past 10 years making it possible to build responsive dashboards to monitor <i>Key Performance Indicators</i> and <i>Key Risk Indicators</i> as well as state of the art Data Discovery Platforms.
</p>
<a name="sec5"></a>
<h2>5. Conclusion</h2>
<p>
I am presenting above the <i>challenges</i> and <i>opportunities</i> for banking institutions coming from the <i>Digitalization</i>. But again that is really one side of the medal, on the other side there is the increasing regulatory pressure and the shrinking margins.
</p>
<p>
With the digitalization, new opportunities for growth and innovation are emerging. Many banking institutions have started to move towards accomplishing the digital transformation in many aspects of their business. But the target seems to be still so far.
<br>
Others are struggling to identify the path and are put in danger by smaller actors or fintechs that are increasingly eating the banking business. They need to understand that the moment is now or never.
</p>
<p>
Part of this article is available as a slideshare presentation here: <a href="https://www.slideshare.net/JrmeKehrli/digitalization-a-challenge-and-an-opportunity-for-banks">https://www.slideshare.net/JrmeKehrli/digitalization-a-challenge-and-an-opportunity-for-banks</a>.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/agile-landscape
Agile Landscape from Deloitte
Jerome Kehrli
2017-03-02T17:51:12-05:00
2017-09-13T16:40:41-04:00
<!-- Agile Landscape from Deloitte -->
<p>
I've seen this infographic from Christopher Webb at Deloitte (at the time) recently.
<br>
This is the most brilliant infographic I've seen for years.
</p>
<p>
Christopher Webb presents here a pretty extended set of <i>Agile Practices</i> associated to their respective <i>frameworks</i>. The practices presented are a collection of all Agile practices down the line, related to engineering but also management, product identification, design, operation, etc.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d028b3c0-5332-4ae7-a91a-dc21b7551299">
<img class="centered" style="width: 600px;" alt="The Agile Lansdcape from Deloitte" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d028b3c0-5332-4ae7-a91a-dc21b7551299" />
</a><br>
<div class="centered">
[Click to enlarge]<br>
(Source : Christopher Webb - LAST Conference 2016 Agile Landscape - <a href="https://www.slideshare.net/ChrisWebb6/last-conference-2016-agile-landscape-presentation-v1">https://www.slideshare.net/ChrisWebb6/last-conference-2016-agile-landscape-presentation-v1</a>)
</div>
</div>
<br>
<p>
I find this infographic brilliant since its the first time I see a "<i>one ring to rule them all</i>" view of what I consider should be the practices towards scaling Agility at the level of the whole IT Organization.
</p>
<p>
Very often, when we think of Agility, we limit our consideration to solely the Software Build Process.
<br>
But Agility is more than that. And I believe an Agile corporation should embrace also <b>Agile Design</b>, <b>Agile Operations</b> and <b>Agile Management</b>.
<br>
This infographic does a great job in presenting how these frameworks enrich and complements each others towards scaling Agility at the level of the whole IT Organization.
</p>
<p>
To be honest there are even many more frameworks that those indicated on this infographic and Chris Webb is presenting some additional - reaching 43 in total - in <a href="https://www.slideshare.net/ChrisWebb6/last-conference-2016-agile-landscape-presentation-v1">his presentation</a>.
<br>
But I believe he did a great job in presenting the most essential ones and presenting how these practices, principles and framework work together to achieve the ultimate goal of every corporation: skyrocketing employee productivity and happiness, maximizing customer satisfaction and blowing operational efficiency up.
</p>
<p>
Now I would want to present why I think considering Agility down the line in each and every aspect around the engineering team and how these frameworks completing each other are important.
</p>
<!-- Agile Landscape from Deloitte -->
<p>
I've seen this infographic from Christopher Webb at Deloitte (at the time) recently.
<br>
This is the most brilliant infographic I've seen for years.
</p>
<p>
Christopher Webb presents here a pretty extended set of <i>Agile Practices</i> associated to their respective <i>frameworks</i>. The practices presented are a collection of all Agile practices down the line, related to engineering but also management, product identification, design, operation, etc.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d028b3c0-5332-4ae7-a91a-dc21b7551299">
<img class="centered" style="width: 600px;" alt="The Agile Lansdcape from Deloitte" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d028b3c0-5332-4ae7-a91a-dc21b7551299" />
</a><br>
<div class="centered">
[Click to enlarge]<br>
(Source : Christopher Webb - LAST Conference 2016 Agile Landscape - <a href="https://www.slideshare.net/ChrisWebb6/last-conference-2016-agile-landscape-presentation-v1">https://www.slideshare.net/ChrisWebb6/last-conference-2016-agile-landscape-presentation-v1</a>)
</div>
</div>
<br>
<p>
I find this infographic brilliant since its the first time I see a "<i>one ring to rule them all</i>" view of what I consider should be the practices towards scaling Agility at the level of the whole IT Organization.
</p>
<p>
Very often, when we think of Agility, we limit our consideration to solely the Software Build Process.
<br>
But Agility is more than that. And I believe an Agile corporation should embrace also <b>Agile Design</b>, <b>Agile Operations</b> and <b>Agile Management</b>.
<br>
This infographic does a great job in presenting how these frameworks enrich and complements each others towards scaling Agility at the level of the whole IT Organization.
</p>
<p>
To be honest there are even many more frameworks that those indicated on this infographic and Chris Webb is presenting some additional - reaching 43 in total - in <a href="https://www.slideshare.net/ChrisWebb6/last-conference-2016-agile-landscape-presentation-v1">his presentation</a>.
<br>
But I believe he did a great job in presenting the most essential ones and presenting how these practices, principles and framework work together to achieve the ultimate goal of every corporation: skyrocketing employee productivity and happiness, maximizing customer satisfaction and blowing operational efficiency up.
</p>
<p>
Now I would want to present why I think considering Agility down the line in each and every aspect around the engineering team and how these frameworks completing each other are important.
</p>
<h3>1. Agile Design</h3>
<p>
Normally I am a little sensitive with formal meaning of the word <i>design</i> in software engineering.
</p>
<p>
But for once I'll make an exception.
<br>
So for once, by <i>design</i> here, I mean the largest possible definition of the term, encompassing as much the discovery of the key features as well as the architecture of the system to be implemented.
</p>
<p>
Agility in identifying beforehand the product to be implemented and its key features is a must.
<br>
Later when the rough form of the product is identified, the process consists in having a Vision workshop to align the stakeholders on the product vision, then Story Mapping workshops, all of these emphasizing Agility, Adaptation and lightweight processes in comparison to the tons of documents produced by more traditional methods.
</p>
<p>
This is pretty well covered in the infographic above and Design thinking covers all the practices that seem key to me such from the light <i>Business Model Canvas</i> to <i>Product Vision</i> definition workshops and <i>Story Mapping</i> workshops.
</p>
<p>
At the end of the day, Agility is mostly about the capacity to <i>adapt</i> and <i>react</i> to changing requirements and changing priorities. Enforcing thorough product identification and feature design phases before actually initiating the development of an <b>MVP</b> aimed at validating (or contradicting) the hypothesis makes little sense in my opinion.
<br>
One important framework to consider here is the <i>Lean</i> approach and the <a href="https://www.niceideas.ch/roller2/badtrash/entry/lean-startup-a-focus-on">Lean Startup Practices</a>.
</p>
<p>
At the end of the day, <i>Agile Software Development</i> methodologies cannot deploy their full potential if the company itself is not Agile.
</p>
<h3>2. Agile Development</h3>
<p>
At the root of everything there is XP. <i>eXtreme Programming</i> was mostly initiated by Kent Beck, strong from his experience on the C3 project. Kent Beck hardly invented a lot of things but rather took some practices more or less used previously in the industry and took them to <i>extreme</i> levels.
</p>
<p>
Agile Software Development is really built on top of XP genes. Today XP is considered just another Agile Software Development Framework, but I don't share that view. To me, XP and the related practices form the most fundamental core of Agile Software Development Methodologies.
<br>
XP Practices take a form or another in the various Agile Frameworks such as RDD, Scrum, Kanban, Scrumban, etc. In some of them some core XP practices are not mentioned; not because they should not be applied, but really because they're nowadays considered so natural that they're assumed. Think for instance of TDD (Unit Tests first), Continuous Integration, Simple Metaphor (Meaningful Naming, Domain Driven Design, Design patterns), etc.
</p>
<p>
I discussed in <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-software-development-lessons-learned">a previous article on this this blog </a> the software development methodology we are using in my current company and interestingly all of our practices are pretty well identified on the infographic above.
</p>
<h3>3. Agile Operation</h3>
<p>
Agile operation is really about <b>DevOps</b>.
</p>
<p>
I developed in length in a dedicated article on this very blog what DevOps is and why it's important so I let the reader refer to <a href="https://www.niceideas.ch/roller2/badtrash/entry/devops-explained">this article</a>.
</p>
<p>
Let's just mention that here as well it is hard for the development team to leverage its Agile practices if the other departments of the corporation - and out of those the operation is crucial - have not embraced Agility.
</p>
<h3>4. Agile Management</h3>
<p>
Agile management is about Leadership and Leadership pursues the goal of growing and transforming organizations into great places to work for, where people are engaged, the work is improved and customers are simply delighted.
</p>
<p>
Agile Management is a lot about Management 3.0.
<br>
Management 1.0 was about doing the wrong thing, by treating people like cogs in a system. Management 2.0 was about doing the right think wrong, with understanding the goals and having good intentions, but using old-fashioned top-down initiatives.
<br>
Management 3.0 is about doing the right thing for the team, involving everyone in improving the system and fostering innovation.
</p>
<p>
Agile Management is about making the components of the Agile corporation collaborate together towards anticipating changes and adapt smoothly and flawlessly.
<br>
There are three most essential vectors:
</p>
<ul>
<li><b>Collective Intelligence</b>: which is key to address and control the increasing complexity of organizations and businesses and based on having everyone in the company taking part in the continuous improvement processes</li>
<li><b>Optimal use of Technology</b> : Technology is an amazing vector of efficiency in regards to tools supporting the organization</li>
<li><b>A sound adoption of Continuous Improvement Processes</b> : making the organization identify and build on its strength while continuously addressing its weaknesses to adapt itself continuously.</li>
</ul>
<p>
Agile Management values individual and interactions over formal processes and hierarchy. It really conists in empowering people and making the organization a place where they can develop themselves with passion and energy, leveraging their capacity for both action and innovation.
<br>
Now of course this needs to be driven and Agile Management encourages continuous feedback in the form, for instance, of O3s - <i>One-On-One</i> - on a regular basis where both the employee and the manager can provide feedback on the organization, respectively the performance of the employee.
</p>
<p>
Managing performance in this sense is identifying the strengths of the employee, which we should leverage, and the weaknesses, which we should address and improve.
</p>
<p>
Empowering people is a key practice since, at the end of the day, Management is too important to be left to Managers ;-)
<br>
Agility is about adaptation but also about efficiency and quality (think XP practices here) and Agile Management is about putting practices in place aimed at making engineers give the best they can and participate at every level in the success of the company.
</p>
<p>
I would conclude this section by giving my favorite definition of management:
</p>
<div class="centering">
<div class="centered">
<i>"Hire great people, and then get the hell out of their way."</i>
</div>
</div>
<br>
<h3>5. Conclusion</h3>
<p>
This infographic is an awesome view of what we have achieved over the last 10 to 15 years in regards to understanding of how to design, engineer, build and manage better.
<br>
I believe finding better ways of working should be an everyday concern for organizations, from startups to international corporations.
</p>
<p>
Quoting <a href="http://www.ge.com/annual00/download/images/GEannual00.pdf">Jack Welsh</a>:
</p>
<div class="centering">
<div class="centered">
<i>"If the rate of change on the outside exceeds the rate of change on the inside, the end is near."</i>
</div>
</div>
<br>
<p>
My personal pick-up is:
</p>
<ul>
<li>Lean (Startup) - See <a href="https://www.niceideas.ch/roller2/badtrash/entry/lean-startup-a-focus-on">my article on Lean Startup</a></li>
<li>DevOps - See <a href="https://www.niceideas.ch/roller2/badtrash/entry/devops-explained">my article on DevOps</a></li>
<li>XP, Scrum and Kanban (Agile Development) - See <a href="https://www.niceideas.ch/roller2/badtrash/entry/agile-software-development-lessons-learned">my article on agility</a></li>
<li>Management 3.0 - Empowering and energizing people, Developing competences, Aligning teams, Continuous Improvement</li>
<li>Kaizen (of course)</li>
</ul>
<p>
I have no experience on <i>Scaling Agile</i> frameworks for now. It's becoming a pretty hot topic in my current company though and I'll revert with an article on this blog when I have some.
<br>
My preference would go to LeSS I think, since it seems more natural to me. But that is just a pretty initial opinion, and it may change ...
</p>
https://www.niceideas.ch/roller2/badtrash/entry/bytecode-manipulation-with-javassist-for
Bytecode manipulation with Javassist for fun and profit part I: Implementing a lightweight IoC container in 300 lines of code
Jerome Kehrli
2017-02-13T15:30:33-05:00
2017-09-13T16:42:14-04:00
<p>
Java bytecode is the form of instructions that the JVM executes.
<br>
A Java programmer, normally, does not need to be aware of how Java bytecode works.
</p>
<p>
Understanding the bytecode, however, is essential to the areas of tooling and program analysis, where the applications can modify the bytecode to adjust the behavior according to the application's domain. Profilers, mocking tools, AOP, ORM frameworks, IoC Containers, boilerplate code generators, etc. require to understand Java bytecode thoroughly and come up with means of manipulating it at runtime.
<br>
Each and every of these advanced features of what is nowadays standard approaches when programming with Java require a sound understanding of the Java bytecode, not to mention completely new languages running on the JVM such as Scala or Clojure.
</p>
<p>
Bytecode manipulation is not easy though ... except with Javassist.
<br>
Of all the libraries and tools providing advanced bytecode manipulation features, Javassist is the easiest to use and the quickest to master. It takes a few minutes to every initiated Java developer to understand and be able to use Javassist efficiently. And mastering bytecode manipulation, opens a whole new world of approaches and possibilities.
</p>
<p>
The goal of this article is to present Javassist in the light of a concrete use case: the implementation in a little more than 300 lines of code of a lightweight, simple but cute IoC Container: SCIF - Simple and Cute IoC Framework.
</p>
<!-- Bytecode manipulation with Javassist for fun and profit part I: Implementing a lightweight IoC container in 300 lines of code -->
<p>
Java bytecode is the form of instructions that the JVM executes.
<br>
A Java programmer, normally, does not need to be aware of how Java bytecode works.
</p>
<p>
Understanding the bytecode, however, is essential to the areas of tooling and program analysis, where the applications can modify the bytecode to adjust the behavior according to the application's domain. Profilers, mocking tools, AOP, ORM frameworks, IoC Containers, boilerplate code generators, etc. require to understand Java bytecode thoroughly and come up with means of manipulating it at runtime.
<br>
Each and every of these advanced features of what is nowadays standard approaches when programming with Java require a sound understanding of the Java bytecode, not to mention completely new languages running on the JVM such as Scala or Clojure.
</p>
<p>
Bytecode manipulation is not easy though ... except with Javassist.
<br>
Of all the libraries and tools providing advanced bytecode manipulation features, Javassist is the easiest to use and the quickest to master. It takes a few minutes to every initiated Java developer to understand and be able to use Javassist efficiently. And mastering bytecode manipulation, opens a whole new world of approaches and possibilities.
</p>
<p>
The goal of this article is to present Javassist in the light of a concrete use case: the implementation in a little more than 300 lines of code of a lightweight, simple but cute IoC Container: SCIF - Simple and Cute IoC Framework.
</p>
<p>
A new version of <a href="https://www.niceideas.ch/roller2/badtrash/entry/comet-having-fun-with-the#sec31">comet-tennis demo app</a> with the SCIF framework integrated is available <a download="comet_tennis_src_0.4.tar.gz" type="application/tar+gzip" href="https://www.niceideas.ch/roller2/badtrash/mediaresource/2eeba15c-dc3c-4546-adf8-a41cd4b31932">here</a>.
</p>
<p>
Part of this article is available as a slideshare presentation here: <a href="https://www.slideshare.net/JrmeKehrli/bytecode-manipulation-with-javassist-for-fun-and-profit">https://www.slideshare.net/JrmeKehrli/bytecode-manipulation-with-javassist-for-fun-and-profit</a>.
</p>
<p>
You might also want to have a look at the second article in this serie available here : <a href="https://www.niceideas.ch/roller2/badtrash/entry/bytecode-manipulation-with-javassist-for1">Bytecode manipulation with Javassist for fun and profit part II: Generating toString and getter/setters using bytecode manipulation</a>.
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. Introduction / Purpose </a>
<ul>
<li><a href="#sec11">1.1 Runtime Reflection</a></li>
<li><a href="#sec12">1.2 Bytcode manipulation</a></li>
</ul>
</li>
<li><a href="#sec2">2. Javassist</a>
<ul>
<li><a href="#sec21">2.1 Javassist prupose and behaviour</a></li>
<li><a href="#sec22">2.2 A gentle example with Javassist</a></li>
</ul>
</li>
<li><a href="#sec3">3. IoC</a>
<ul>
<li><a href="#sec31">3.1 IoC history</a></li>
<li><a href="#sec32">3.2 IoC Principle</a></li>
<li><a href="#sec33">3.2 IoC frameworks (Spring / Pico / Google)</a></li>
</ul>
</li>
<li><a href="#sec4">4.0 SCIF : Simple and Cute IoC Framework</a>
<ul>
<li><a href="#sec41">4.1 Principle</a></li>
<li><a href="#sec42">4.2 Design</a></li>
<li><a href="#sec43">4.3 Some focus on code</a></li>
<li><a href="#sec44">4.4 DemoApp : Comet Tennis</a></li>
</ul>
</li>
<li><a href="#sec5">5. Conclusion</a></li>
</ul>
<a name="sec1"></a>
<h2>1. Introduction / Purpose </h2>
<p>
Bytecode manipulation consists in modifying the classes - represented by bytecode - compiled by the Java compiler, at runtime. It is used extensively for instance by frameworks such as Spring (IoC) and Hibernate (ORM) to inject dynamic behaviour to Java objects at runtime.
<br>
But first let's look at a very summarized view of the Java toolchain to remind a few concepts:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/583331b9-53d5-45a5-bb37-0277e7b58b95">
<img class="centered" style="width: 500px; " alt="Java Toolchain" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/583331b9-53d5-45a5-bb37-0277e7b58b95" />
</a>
</div>
<br>
<p>
Java source files are compiled to Java class files by the Java Compiler. These Java classes take the form of <i>bytecode</i>. This bytecode is loaded by the JVM to execute the Java program.
<br>
In principle the bytecode is read only and cannot be change once loaded. That is true, but:
</p>
<ul>
<li>The java classes bytecode can be modified before being loaded by the classloader through the usage of an <i>agent</i></li>
<li>Classes bytecode can be modified at runtime without an agent as long as the class has not been loaded yet by a classloader.</li>
<li>Classes can be generated entirely dynamically at runtime using bytecode manipulation techniques</li>
</ul>
<p>
In this article we'll dig into the library <a href="http://jboss-javassist.github.io/javassist/">Javassist</a> which is a bytecode manipulation framework that can help in achieving all of the above mechanisms.
<br>
But before, let's describe three different, unrelated but complementary techniques: <i>Introspection</i>, <i>Reflection</i> and <i>Bytecode Manipulation</i>.
</p>
<a name="sec11"></a>
<h3>1.1 Runtime Reflection </h3>
<p>
<i>Runtime reflection</i> is the ability of a computer program to examine, introspect, and modify its own structure and behavior at runtime.
</p>
<p>
Reflection is commonly used by programs which require the ability to examine or modify the runtime behavior of applications running in the Java virtual machine.
<br>
Reflection is a powerful technique and can enable applications to perform operations which would otherwise be impossible.
</p>
<p>
The ability to examine and manipulate a Java class from within itself may not sound like very much, but in other programming languages this feature simply doesn't exist. For example, there is no way in a Pascal, C, or C++ program to obtain detailed information about the functions defined within that program.
</p>
<p>
In java there is no specific <b>introspection</b> API available natively. <i>Introspection</i> is performed as well using the <b>Java Reflection API</b>.
</p>
<p>
Still, conceptually, introspection and reflection are different things:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/dc4da8de-0885-4470-b0ae-0f392de9c4f3">
<img class="centered" style="width: 600px;" alt="Introspection and Reflection" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/dc4da8de-0885-4470-b0ae-0f392de9c4f3" />
</a>
</div>
<br>
<p>
<b>Type introspection</b> is the ability of a program to examine the type or properties of an object at runtime.
<br>
This is short example of Type Introspection in Java, where we discover the fields of an object and show their values dynamically. Again, Introspection in Java is really done using the <i>Reflection API</i>
</p>
<pre>
<b>import</b> java.lang.reflect.Field;
<b>public class</b> TestIntrospection {
<b>public static class</b> TestData {
<b>private int</b> i = <span style="color: blue;">0</span>;
<b>private</b> String myString = <span style="color: blue;">"abc"</span>;
<b>private long</b> value = <span style="color: blue;">-1</span>;
}
<span style="color: green;">// test Introspection</span>
<b>public static void</b> main (String[] args) {
<b>try</b> {
<span style="color: green;">// Using Introspection, we really don't care of the actual type</span>
Object td = <b>new</b> TestData();
<span style="color: green;">// List fields of TestData and get their values</span>
<b>for</b> (Field field : td.getClass().getDeclaredFields()) {
field.setAccessible(<b>true</b>); <span style="color: green;">// just make private fields accessible</span>
System.out.println (field.getName() + <span style="color: blue;">"="</span> + field.get(td));
}
} <b>catch</b> (IllegalAccessException e) {
e.printStackTrace();
}
}
}
</pre>
<p>
<b>Runtime Reflection</b> is a native feature in the Java programming language. It allows an executing Java program to examine or "<i>introspect</i>" upon itself, and manipulate internal properties of the program.
<br>
A short example could be as follows, where we change the value of a field using reflection:
</p>
<pre>
<b>import</b> java.lang.reflect.Field;
<b>public class</b> TestReflection {
<b>public static class</b> TestData {
<b>private int</b> i = <span style="color: blue;">0</span>;
<b>private</b> String myString = <span style="color: blue;">"abc"</span>;
<b>private long</b> value = <span style="color: blue;">-1</span>;
}
<span style="color: green;">// test Reflection</span>
<b>public static void</b> main (String[] args) {
<b>try</b> {
<span style="color: green;">// Using Reflection, we really don't care of the actual type</span>
Object td = <b>new</b> TestData();
<span style="color: green;">// Change the value of the field myString</span>
Field myStringField = td.getClass().getDeclaredField(<span style="color: blue;">"myString"</span>);
myStringField.setAccessible(<b>true</b>); <span style="color: green;">// just make private fields accessible</span>
myStringField.set(td, <span style="color: blue;">"xyz"</span>);
System.out.println (myStringField.getName() + <span style="color: blue;">"="</span> + myStringField.get(td));
} <b>catch</b> (NoSuchFieldException | IllegalAccessException e) {
e.printStackTrace();
}
}
}
</pre>
<p>
<i>Runtime Reflection</i> is a very powerful feature of the JVM.
<br>
I wrote a <a href="https://www.niceideas.ch/roller2/badtrash/entry/java-create-enum-instances-dynamically">previous article</a> on this very blog showing how to dynamically add values to an <code>Enum Type</code> in Java using Runtime Reflection.
</p>
<p>
<b>Why are those important in the scope of bytecode manipulation and Javassist ?</b>
</p>
<p>
<i>Runtime Reflection</i> is important in our context for two reasons:
</p>
<ol>
<li>First, because Javassist attempts to keep an API as close as possible to the Java Runtime Reflection API as a way to appear as natural as possible to Java developers.</li>
<li>Second, and this is maybe more important, because behaviour injected in Java Classes using bytecode manipulation is not known by the compiler. Thus, it is sometimes only available through runtime reflection.</li>
</ol>
<a name="sec12"></a>
<h3>1.2 Bytcode manipulation</h3>
<p>
Bytecode manipulation allows the developer to express instructions in a format that is directly understood by the Java Virtual Machine, without passing from source code to bytecode through compiler.
<br>
Bytecode is somewhat similar to assembly code directly interpretable by the CPU. But with Java the bytecode is, first, interpreted by a Virtual Machine, the JVM, and second much more understandable that assembly code.
</p>
<p>
One might wonder why one would want to get interested in bytecode manipulation and generation. As a matter of fact, every java developer has likely already been using bytecode manipulation all over the place without knowing it.
<br>
Since the JVM can modify bytecode and use new bytecode while it is running, this generates a whole new universe of languages and tools that by far surpasses the initial intent of the Java language.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/18bc7da9-9a4d-40ab-8a0a-6430fafa6e3a">
<img class="centered" style="width: 600px;" alt="Bytecode Manipulation" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/18bc7da9-9a4d-40ab-8a0a-6430fafa6e3a" />
</a>
</div>
<br>
<p>
<b>Bytecode manipulation use cases</b>
</p>
<p>
Some examples are:
</p>
<ul>
<li>ORM frameworks such as <a href="http://hibernate.org/">Hibernate</a> use bytecode manipulation to inject, for instance, relationship management code (lazy loading, etc.) inside mapped entities.</li>
<li><a href="http://findbugs.sourceforge.net/">FindBugs</a> inspects bytecode for static code analysis</li>
<li>Languages like <a href="http://www.groovy-lang.org/">Groovy</a>, <a href="https://www.scala-lang.org/">Scala</a>, <a href="https://clojure.org/">Clojure</a> generate bytecode from different source code.</li>
<li>IoC frameworks such as <a href="https://spring.io/">Spring</a> use it to seamlessly weave your application lifecycle together</li>
<li>language extensions like <a href="http://www.eclipse.org/aspectj/">AspectJ</a> can augment the capabilities of Java by modifying the classes that the Java compiler generated</li>
<li>etc.</li>
</ul>
<p>
The Java platform provides you with many ways to work with bytecode, for instance:
</p>
<ul>
<li>One can write his own compiler for any kind of new and crazy language</li>
<li>One can generate on the fly sub-classes of already loaded classes and use them instead of original classes to get additional behaviour</li>
<li>One can write an instrumentation agent that plugs right into the JVM and modifies behaviour of classes before they are loaded by the classloader</li>
<li>etc.</li>
</ul>
<p>
With so many options available, one of these will certainly fit any experiment that one wants to play around with. With bytecode manipulation, one really gets the whole power of the JVM for free and the capacity to slot in any idea exactly where it's needed while reusing the rest of the Java platform.
</p>
<p>
From my perspective, this is what excites me the most, as a developer I can really focus on my crazy idea that is not supported by the Java language and I don't have to write an entire platform to make it come to life.
<br>
Certainly this has been one of the key areas why the Java community has constantly been experimenting with new ways to push the programming toolset further.
</p>
<p>
This article wont present the details of the Java bytecode any further. We'll focus instead on high level libraries aimed at manipulating the Java Bytecode.
<br>
Should you be interested in the low level details, I can only recommend that you read <a href="https://niceideas.ch//mastering-java-bytecode.pdf">this excellent paper from ZeroTunaround</a>, the guys behind JRebel.
</p>
<p>
<b>Most common bytecode manipulation libraries</b>
</p>
<p>
As a matter of fact, while Runtime Reflection is supported natively by the JVM, <b>Bytecode manipulation</b>, on the other hand, is fairly difficult to achieve without the usage of <i>a specific library</i>
</p>
<p>
The most common bytecode manipulation libraries in Java are as follows:
</p>
<ul>
<li><a href="http://asm.ow2.org/">ASM</a> s a project of the OW2 Consortium. It provides a simple API for decomposing, modifying, and recomposing binary Java classes. ASM exposes the internal aggregate components of a given Java class through its visitor oriented API. ASM also provides, on top of this visitor API, a tree API that represents classes as object constructs. Both APIs can be used for modifying the binary bytecode, as well as generating new bytecode</li>
<li><a href="https://commons.apache.org/proper/commons-bcel/">BCEL</a> provides a simple library that exposes the internal aggregate components of a given Java class through its API as object constructs (as opposed to the disassembly of the lower-level opcodes). These objects also expose operations for modifying the binary bytecode, as well as generating new bytecode (via injection of new code into the existing code, or through generation of new classes altogether).</li>
<li><a href="https://github.com/cglib/cglib/wiki">CGLIB</a> is a powerful, high performance and quality Code Generation Library, it is used to extend JAVA classes and implements interfaces at runtime. CGLIB is really oriented towards implementing new classes at runtime, as opposed to modifying existing bytecode such as other libraries.</li>
<li><a href="http://jboss-javassist.github.io/javassist/">Javassist</a> is a Java library providing a means to manipulate the Java bytecode of an application. In this sense Javassist provides the support for structural reflection, i.e. the ability to change the implementation of a class at run time.</li>
</ul>
<p>
Javassist is much easier to use that lower level libraries such as BCEL or ASM. It is also less limited and more powerful than CGLIB.
</p>
<p>
This chart shows in addition the AspectJ framework, as a way to have the user get an understanding of the level of abstraction provided by these tools:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/08231a71-0633-4a02-934f-a968d20cbe69">
<img class="centered" style="width: 500px;" alt="Bytecode Manipulation frameworks" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/08231a71-0633-4a02-934f-a968d20cbe69" />
</a>
</div>
<br>
<p>
Now, as you might have guessed from its title, this rest of this article will focus on Javassist.
</p>
<a name="sec2"></a>
<h2>2. Javassist</h2>
<p>
From the <a href="http://jboss-javassist.github.io/javassist/">Javassist web site:</a>
</p>
<p>
"Javassist (Java Programming Assistant) makes Java bytecode manipulation simple. It is a class library for editing bytecode in Java; it enables Java programs to define a new class at runtime and to modify a class file when the JVM loads it.
<br>
Unlike other similar bytecode editors, Javassist provides two levels of API: source level and bytecode level. If the users use the source-level API, they can edit a class file without knowledge of the specifications of the Java bytecode.
<br>
The whole API is designed with only the vocabulary of the Java language. You can even specify inserted bytecode in the form of source text; Javassist compiles it on the fly. On the other hand, the bytecode-level API allows the users to directly edit a class file as other editors."
</p>
<p>
The fact that Javassist is presented above as being able to modify classes at loading time is not a limitation of the Javassist framework itself, but rather a consequence from the linking system of the JVM. Once a class has already been loaded, changing it would result in a Linkage Error (unless the JVM is launched with the JPDA [Java Platform Debugger Architecture] enabled, which would make a class dynamically reloadable, but that is another story).
<br>
Interestingly, Javasssist is perfectly able to modify a class long after the application has started as long as that specific class has not been loaded.
<br>
This is just to emphasize that Javassist can perfectly be used to modify classes at runtime and not only at "<i>pre-main</i>" time by the usage of a JVM agent, even though this suffers from great constraints.
</p>
<p>
In my opinion, the great strength of Javassist over its competitors is that Javassist enables the user to generate bytecode on the fly from actual Java code given to it in the form of a string by calling the Java Compiler on the fly on such strings.
<br>
And that is freaking awesome.
</p>
<a name="sec21"></a>
<h3>2.1 Javassist purpose and behaviour</h3>
<p>
Javassist provides the developer with a high level API around classes, methods, fields, etc. aimed at making it as easy as possible to change the implementation of existing classes or even implement completely new classes, dynamically, at runtime, using bytecode manipulation.
</p>
<p>
(The following is explained in more details in the <a href="http://jboss-javassist.github.io/javassist/tutorial/tutorial.html">official Javassist tutorial</a>.)
</p>
<p>
The most important elements of the Javassist API are presented on the schema below:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/10735f16-d4b1-4bc8-8a4e-e70761f85b4b">
<img class="centered" style="width: 500px;" alt="Javassist API" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/10735f16-d4b1-4bc8-8a4e-e70761f85b4b" />
</a>
</div>
<br>
<p>
The <code>class Javassist.CtClass</code> is an abstract representation of a class file. A <code>CtClass</code> (compile-time class) object is a handle for dealing with a class file. The following program is a very simple example:
</p>
<p>
(In all examples of code from now on, <code><span style="color: DarkRed">I will be coloring relevant Javassist API calls in dark red</span></code>)
</p>
<pre>
<span style="color: DarkRed">ClassPool</span> pool = <span style="color: DarkRed">ClassPool.getDefault()</span>;
<span style="color: DarkRed">CtClass</span> cc = pool.<span style="color: DarkRed">get</span>(<span style="color: blue;">"test.Rectangle"</span>);
cc.<span style="color: DarkRed">setSuperclass</span>(pool.<span style="color: DarkRed">get</span>(<span style="color: blue;">"test.Point"</span>));
cc.<span style="color: DarkRed">writeFile()</span>;
</pre>
<p>
<b><code>ClassPool</code></b>
</p>
<p>
This program first obtains a <code>ClassPool</code> object, which controls bytecode modification with Javassist. The <code>ClassPool</code> object is a container of <code>CtClass</code> object representing a class file. It reads a class file on demand for constructing a <code>CtClass</code> object and records the constructed object for responding later accesses.
</p>
<p>
To modify the definition of a class, the users must first obtain from a <code>ClassPool</code> object a reference to a <code>CtClass</code> object representing that <code>class.get()</code> in <code>ClassPool</code> is used for this purpose.
<br>
In the case of the program shown above, the <code>CtClass</code> object representing a class <code>test.Rectangle</code> is obtained from the <code>ClassPool</code> object and it is assigned to a variable <code>cc</code>. The <code>ClassPool</code> object returned by <code>getDefault()</code> searches the default system search path.
</p>
<p>
<b><code>CtClass</code></b>
</p>
<p>
The <code>CtClass</code> object obtained from a <code>ClassPool</code> object can be modified.
<br>
In the example above, it is modified so that the superclass of <code>test.Rectangle</code> is changed into a class <code>test.Point</code>. This change is reflected on the original class file when <code>writeFile()</code> in <code>CtClass()</code> is finally called.
</p>
<p>
<code>writeFile()</code> translates the <code>CtClass</code> object into a class file and writes it on a local disk. Javassist also provides a method for directly obtaining the modified bytecode. To obtain the bytecode, call <code>toBytecode()</code>:
</p>
<pre>
<b>byte</b>[] b = cc.<span style="color: DarkRed">toBytecode</span>();
</pre>
<p>
(Bear in mind that this is especially useful when implementing a Java agent)
</p>
<p>
You can directly load the <code>CtClass</code> as well:
</p>
<pre>
Class clazz = cc.<span style="color: DarkRed">toClass</span>();
</pre>
<p>
A class can be returned to the pool, making it available to the classloader and hence the whole application:
</p>
<pre>
pool.<span style="color: DarkRed">toClass</span>(cc, Thread.currentThread().getContextClassLoader(), <b>null</b>);
</pre>
<p>
<b>Defining a new class</b>
</p>
<p>
To define a new class from scratch, <code>makeClass()</code> must be called on a <code>ClassPool</code>.
</p>
<pre>
<span style="color: DarkRed">ClassPool</span> pool = <span style="color: DarkRed">ClassPool.getDefault</span>();
<span style="color: DarkRed">CtClass</span> cc = pool.<span style="color: DarkRed">makeClass</span>(<span style="color: blue;">"Circle"</span>);
cc.<span style="color: DarkRed">setSuperclass</span>(pool.<span style="color: DarkRed">get</span>(<span style="color: blue;">"test.Point"</span>));
</pre>
<p>
This program defines a class <code>Circle</code> including no members except those inherited by the parent class <code>Point</code>. Member methods of <code>Circle</code> can afterwards be created with factory methods declared in <code>CtNewMethod</code> and appended to <code>Circle</code> with <code>addMethod()</code> in <code>CtClass</code>.
<br>
<code>makeClass()</code> cannot create a new interface; <code>makeInterface()</code> in <code>ClassPool</code> can do. Member methods in an interface can be created with <code>abstractMethod()</code> in <code>CtNewMethod</code>. Note that an interface method is an abstract method.
</p>
<p>
<b>Implementing / Modifying a class</b>
</p>
<p>
Methods are represented by <code>CtMethod</code> objects. <code>CtMethod</code> provides several methods for modifying the definition of the method. Note that if a method is inherited from a super class, then the same <code>CtMethod</code> object that represents the inherited method represents the method declared in that super class. A <code>CtMethod</code> object corresponds to every method declaration.
<br>
Constructors are represented by their very own type in Javassist : <code>CtConstructor</code>. Both <code>CtMethod</code> and <code>CtConstructor</code> extends the same base class and have a lot of their API in common.
</p>
<p>
Javassist does not allow to remove a method or field, but it allows to change the name. So if a method is not necessary any more, it should be renamed and changed to be a private method by calling <code>setName()</code> and <code>setModifiers()</code> declared in CtMethod, for instance to hide it. But beware of linkage errors at runtime if you mess with a method used by another class.
</p>
<p>
<code>CtMethod</code> and <code>CtConstructor</code> can be used to completely implement / rewrite a constructor or a method from scratch. They also provide methods <code>insertBefore()</code>, <code>insertAfter()</code>, and <code>addCatch()</code>. They are used for inserting a code fragment into the body of an existing method.
</p>
<p>
When implementing or rewriting completely a method from scratch, using <code>CtNewMethod.make()</code> is in my opinion the most convenient approach. It enables the developer to <b>implement a method by providing Java Source Code</b> syntax in a simple string.
<br>
For instance:
</p>
<pre>
<span style="color: DarkRed">CtClass</span> point = <span style="color: DarkRed">ClassPool.getDefault().get</span>(<span style="color: blue;">"Point"</span>);
<span style="color: DarkRed">CtMethod</span> m = <span style="color: DarkRed">CtNewMethod.make</span>(
<span style="color: blue;">"public int xmove(int dx) { x += dx; }"</span>,
point);
point.<span style="color: DarkRed">addMethod</span>(m);
</pre>
<p>
<code>CtNewMethod</code> provides a lot of high level methods for implementing getters, setters and other commodity methods directly, sometimes even without having to bother providing an implementation on your own.
</p>
<p>
Some pretty complete information in this regards is available on the <a href="http://jboss-javassist.github.io/javassist/tutorial/tutorial2.html">Second official javassist tutorial</a>.
</p>
<a name="sec22"></a>
<h3>2.2 A gentle example with Javassist</h3>
<p>
We'll see now a simple and yet complete example using Javassist. We'll implement the getters and setters for the fields of the class <code>TestData</code> introduced <a href="#sec11">here</a> using bytecode manipulation.
<br>
Then we'll test the getter and setter for the field <code>myString</code>. Since these getters and setters are injected using bytecode manipulation at runtime, we'll have to use runtime reflection to call them:
</p>
<p>
(Reminder, <code><span style="color: DarkRed">I am coloring relevant Javassist API calls in dark red</span></code>)
</p>
<pre>
<b>import</b> javassist.*;
<b>import</b> java.lang.reflect.InvocationTargetException;
<b>import</b> java.lang.reflect.Method;
<b>public class</b> TestJavassist {
<b>public static class</b> TestData {
<b>private int</b> i = <span style="color: blue;">0</span>;
<b>private</b> String myString = <span style="color: blue;">"abc"</span>;
<b>private long</b> value = <span style="color: blue;">-1</span>;
}
<span style="color: green;">// test Javassist</span>
<b>public static void</b> main (String[] args) {
<b>try</b> {
<span style="color: DarkRed">ClassPool</span> cp = <span style="color: DarkRed">ClassPool.getDefault</span>();
<span style="color: DarkRed">CtClass</span> clazz = cp.<span style="color: DarkRed">get</span>(<span style="color: blue;">"ch.niceideas.common.utils.TestJavassist$TestData"</span>);
<b>for</b> (<span style="color: DarkRed">CtField</span> field : clazz.<span style="color: DarkRed">getDeclaredFields</span>()) {
String camelCaseField = field.getName().substring(<span style="color: blue;">0</span>, <span style="color: blue;">1</span>).toUpperCase()
+ field.getName().substring(<span style="color: blue;">1</span>);
<span style="color: green;">// We don't need to mess with implementation here. CtnewMethod has a</span>
<span style="color: green;">// commodity method to implement a getter directly</span>
<span style="color: DarkRed">CtMethod</span> fieldGetter = <span style="color: DarkRed">CtNewMethod.getter</span>(<span style="color: blue;">"get"</span> + camelCaseField, field);
clazz.<span style="color: DarkRed">addMethod</span>(fieldGetter);
<span style="color: green;">// Just for the sake of an example, we'll define the setter by actually</span>
<span style="color: green;">// providing the implementation, not using the commodity method offered</span>
<span style="color: green;">// by CtNewMethod</span>
<span style="color: DarkRed">CtMethod</span> fieldSetter = <span style="color: DarkRed">CtNewMethod.make</span>(
<span style="color: blue;">"public void set"</span> + camelCaseField + <span style="color: blue;">" \n"</span> +
<span style="color: blue;">" ("</span> + field.getType().getName() + <span style="color: blue;">" param) { \n"</span> +
<span style="color: blue;">" this."</span> + field.getName() + <span style="color: blue;">" = param; \n"</span> +
<span style="color: blue;">"}"</span>,
clazz);
clazz.<span style="color: DarkRed">addMethod</span>(fieldSetter);
}
<span style="color: green;">// Save class and make it available</span>
cp.<span style="color: DarkRed">toClass</span>(clazz, Thread.currentThread().getContextClassLoader(), <b>null</b>);
<span style="color: green;">// Now instantiate a new TestData</span>
TestData td = <b>new</b> TestData();
<span style="color: green;">// Get the value of the field 'myString' using the newly defined getter</span>
Method getter = td.getClass().getDeclaredMethod(<span style="color: blue;">"getMyString"</span>);
System.out.println (getter.invoke(td));
<span style="color: green;">// Change the value of field 'myString' using newly defined setter</span>
Method setter = td.getClass().getDeclaredMethod(<span style="color: blue;">"setMyString"</span>, String.<b>class</b>);
setter.invoke(td, <span style="color: blue;">"xyz"</span>);
<span style="color: green;">// Get the value again</span>
System.out.println (getter.invoke(td));
} <b>catch</b> ( NotFoundException | CannotCompileException | NoSuchMethodException
| IllegalAccessException | InvocationTargetException e) {
e.printStackTrace();
}
}
}
</pre>
<a name="sec3"></a>
<h2>3. IoC</h2>
<p>
OK. Since the example code I want to implement below with Javassist is a simple IoC container, I guess I should present what IoC actually is beforehand.
</p>
<p>
<b>Inversion of Control</b> is a design pattern related to lifecycle management of components in an application benefiting from a services architecture.
<br>
In such an application, business components are usually implemented in the form of various services, such as business services, business managers, DAOs, etc. The main class delegates specific business concerns to business services, which delegate finer aspects in their turn to managers, which further delegate various business of technical aspects to smaller managers, or DAOs, adapters. etc.
</p>
<p>
These various services need to know about each others to be able to call each others. Managing the construction and instantiation of these services is called <i>components lifecycle management</i>.
</p>
<p>
Very often, business services are stateless components, not keeping any state in instance variables or else. Traditionally, for a very long time, these stateless services have been implemented as <i>singletons</i>. For a very long time this was a very convenient approach since the main singleton simply needs to <i>get</i> the other singletons it was using, which in turn simply needed to get the other singletons they were using, and so on.
<br>
By separating the instantiation of singletons and their initialization in two stages, cycles could be handled easily and everyone was happy.
<br>
But with the rise of XP and unit testing, singleton-based applications were suffering from a very critical drawback:
</p>
<p>
Singletons were enforcing strict dependencies on other service implementation at compile time, making it pretty impossible to replace the dependencies by mock objects or stubs as required for efficiently unit testing a specific service.
<br>
Using singletons, testing a specific service often meant to be required to completely build and initialize the whole application, which can well turn into a nightmare.
</p>
<p>
Java EE was of course not an answer to this problem since it required to have a Java EE container, which is even more of a nightmare (well let's not get me started on Java EE, shall we ?).
</p>
<p>
<b>Inversion of Control</b>
</p>
<p>
Inversion of control is initially mostly an answer to this problem, as a way to increase the modularity of the application and making it more extensible, and more importantly testable in an easier way by removing the strict dependencies between components.
<br>
The key idea is to delegate the management of the lifecycle of the components and the injection of each component's dependencies to a framework, or rather a container, borrowing the term from Java EE, called here a <b>lightweight container</b>.
</p>
<p>
Instead of every component getting each other references on the singletons or specific instances by building them, the container takes care of <b>instantiating the components</b>, managing their lifecycle in the required scope, and <b>injecting their dependencies</b> at runtime.
<br>
Injecting the dependencies at runtime, with a configurable approach, using a configuration file, annotations or even a dedicated API, opens the possibility to inject a different implementation of a service depending on the context, as long as it respects the required interface.
<br>
For instance, injecting a <i>mock object</i> instead of the real deal for unit testing becomes straightforward.
</p>
<p>
<i>Inversion of Control</i> and <i>Dependency Injection</i> are two different things - and yet strongly related to each other - often confused in some documentation:
</p>
<ul>
<li><b>Inversion of Control</b> - IoC : is the name of the design pattern, the approach. It is considered a design pattern, which, in my opinion, is wrong! IoC is an <i>architecture pattern</i>. But yeah that is really no big deal.</li>
<li><b>Dependency Injection</b> - DI : is the name of a technique, a mechanism on which IoC often relies to take place. It consists in injecting the components required by a specific component at runtime, based on some configuration rules. DI is really just one aspect of IoC.
</ul>
<a name="sec31"></a>
<h3>3.1 IoC history</h3>
<p>
Inversion of Control, as a term, was popularized in 1998 by the Apache Avalon team trying to engineer a "<i>Java Apache Server Framework</i>" for the growing set of server side Java components and tools at Apache.<br>
To the Avalon team, it was clear that components receiving various aspects of component assembly, configuration and lifecycle was a superior design to those components going at getting the same themselves.
</p>
<p>
Later, the authors of the "<i>Java Open Source Programming</i>" book wrote XWork and WebWork2 to support their forthcoming book. Their concepts were very much like those from IoC/Avalon, but dependencies were passed into the component via setters. The need for those dependencies was declared in some accompanying XML.
<br>
That was actually the first IoC framework close to the form they have today.
</p>
<p>
In 2002, Rod Johnson, leader of the Spring Framework wrote the book "<i>Expert One-on-One : J2EE Design and Development</i>" that also discussed the concepts of setter injection, and introduced the codeline that ultimately became the Spring Framework at SourceForge in February of 2003.
<br>
I myself discovered the concept in 2003 when reading that book, following the advise of Mr. Parick Gras, which I take the opportunity to thank a lot for this here.
</p>
<p>
This whole history is presented in details on <a href="http://picocontainer.com/inversion-of-control-history.html">PicoContainer / Inversion of Control History</a> and can be represented as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/859f9a17-9503-486d-b5ca-310cf6c3e4ca">
<img class="centered" style="width: 691px;" alt="IoC history timeline" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/859f9a17-9503-486d-b5ca-310cf6c3e4ca" />
</a><br>
<div class="centered">
(Source : <a href="http://picocontainer.com/inversion-of-control-history.html">PicoContainer / Inversion of Control History</a>)
</div>
</div>
<br>
<a name="sec32"></a>
<h3>3.2 IoC Principle</h3>
<p>
As stated above, in a usual application, the lifecycle of components starts with a main component (or class) that either creates the other services it requires or get their singletons.
<br>
These other components, in their turn, create or get references on their own dependencies, and so on.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a8688128-c6be-4993-9328-1e8506ca2bc7">
<img class="centered" style="width: 300px;" alt="Usual Singleton Lookup" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a8688128-c6be-4993-9328-1e8506ca2bc7" />
</a>
</div>
<br>
<p>
With IoC, a container, called lightweight container - as opposed to Java EE craps that are very heavy (and very bad) containers - takes care of instantiating and managing the lifecycle of the components as well as, more importantly, injecting the dependencies in every component.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/585582be-b4c9-4577-a905-1f8029321f95">
<img class="centered" style="width: 405px;" alt="IoC injection" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/585582be-b4c9-4577-a905-1f8029321f95" />
</a>
</div>
<br>
<a name="sec33"></a>
<h3>3.3 Various frameworks</h3>
<p>
The most important IoC containers today are the following:
</p>
<ul>
<li>The <a href="https://spring.io/">Spring Framework</a> is an application framework and inversion of control container for the Java platform. The core of spring is really about IoC and components management but nowadays there is a complete ecosystem of tools and side frameworks around spring core aimed at developing web application, ORM concerns, etc.</li>
<li>The <a href="http://picocontainer.com/introduction.html">Pico Container</a> is a very lightweight IoC Container and only that. Unlike spring, it is designed to remain small and simple and targets only IoC concerns, nothing else. It is not heavily maintained.</li>
<li><a href="http://tapestry.apache.org/">Apache Tapestry</a> is an open-source component-oriented Java web application framework conceptually similar to JavaServer Faces and Apache Wicket. It provides IoC concerns in addition to the web application framework.</li>
<li><a href="https://github.com/google/guice/wiki/GettingStarted">Google Guice</a> is an open source software framework for the Java platform released by Google. It provides support for dependency injection using annotations to configure Java objects.</li>
</ul>
<a name="sec4"></a>
<h2>4. SCIF : Simple and Cute IoC Framework </h2>
<p>
The rest of this article is dedicated to present the implementation of a very simple IoC framework using Javassist, rather as a way to illustrate how easy and straightforward that is with Javassist than for any other reason :-)
<br>
Implementing Dependency Injection is actually a state-of-the-art use case for Javassist and a nice way to present the possibilities and whereabouts of bytecode manipulation.
<br>
We'll see now how to use Javassist in the light of a concrete use case: the implementation in a little more than 300 lines of code of a lightweight, simple but cute IoC Container: SCIF - Simple and Cute IoC Framework.
</p>
<a name="sec41"></a>
<h3>4.1 Principle </h3>
<p>
SCIF - the system we want to build - is an MVP (Minimum Viable Product). We want it to implement <i>Dependency Injection</i> in its simplest form:
</p>
<ul>
<li>Services are managed by the framework and stored in a Service <i>Registry</i> </li>
<li>Services should declare the annotation <code>@Service</code><li> to be discovered by the framework. The framework searches for services declaring this annotation in the classpath.</li>
<li>Dependencies are identified in services using the annotation <code>@Resource</code>. The framework analyze services to discover about their dependencies at runtime.</li>
<li>If <code>@Resource</code> is declared on a field, the framework injects the dependency directly, at build time.</li>
<li>If <code>@Resource</code> is declared on a getter, the framework uses bytecode manipulation to override the getter in a subclass and implement <i>lazy loading</i> of the dependency.
</ul>
<p>
In case of getter (property) injection instead of field injection, SCIF is forced to generate a sub-class of the initial class and override the getter in that sub-class to implement lazy-loading.
<br>
This is a consequence of the usage of an annotation to identify the services to be enhanced: in order for the framework to be able to query the annotation on a class, that class needs unfortunately to be loaded by the Classloader. Javassist is not able to change a class once that class has been loaded (well at least not easily).
<br>
Changing a class that is already loaded leads unusually to a linkage error and that is forbidden without usage of very advanced techniques, too difficult for this simple framework.
</p>
<p>
<b>Example</b>
</p>
<p>
The code below presents a <code>ServiceA</code> having two dependencies: <code>ServiceB</code> and <code>ServiceC</code>.
<br>
The first dependency, <code>ServiceB</code> is declared by <code>ServiceA</code> on the field itself, using the annotation <code>@Resource</code>.
<br>
The second dependency, <code>ServiceC</code> is declared on the getter, indicating the will of the developer to benefit from lazy loading.
</p>
<p>
The example below illustrates in <code><span style="color: #CB0000;">red</span></code> the code or behaviour that should be injected at runtime by SCIF.
</p>
<pre>
<b>import</b> ch.niceideas.common.service.Registry;
<b>import</b> ch.niceideas.common.service.Service;
<b>import</b> javax.annotation.Resource;
<span style="color: green;">/** A Business Service */</span>
@Service
<b>public class</b> ServiceA {
@Resource <span style="color: green;">// here we inject the dependency on the field</span>
<b>private</b> ServiceB serviceB = <span style="color: #CB0000;">service_injected_by_reflection;</span>
<b>private</b> ServiceC serviceC;
<b>public</b> ServiceB getServiceB() { <b>return</b> serviceB; }
<b>public void</b> setServiceB(ServiceB serviceB) { <b>this</b>.serviceB = serviceB; }
@Resource <span style="color: green;">// here we inject the dependency on the property (getter)</span>
<b>public</b> ServiceC getServiceC() { <b>return</b> serviceC; }
<b>public void</b> setServiceC(ServiceC serviceC) { <b>this</b>.serviceC = serviceC; }
<span style="color: #CB0000;">
// we want to use javassist to generate a sub-class on the fly, at runtime, to handle
// the lazy loading of ServiceC at runtime
<b>public static class</b> javassist_sub <b>extends</b> ServiceA {
<b>private</b> Registry registry = injected_by_reflection;
<b>public</b> ServiceC getServiceC() {
ServiceC retObject = <b>super</b>.getServiceC();
if (retObject == <b>null</b>) {
retObject = (ServiceC) registry.getService(
ServiceC.<b>class</b>.getCanonicalName());
<b>super</b>.setServiceC(retObject);
}
<b>return</b> retObject;
}
}</span>
}
</pre>
<p>
Now let's see what is the design of the SCIF framework enabling this behaviour.
</p>
<a name="sec42"></a>
<h3>4.2 Design </h3>
<p>
SCIF is implemented by the following fundamental classes:
</p>
<ol>
<li>
<code><b>Registry</b></code> : a Registry is a service manager. It stores services following a specific scope passed to the <code>storeService</code> method.
<br>
Stored services can be retrieved regardless of the scope. They are searched in the smallest scope first and then in larger scopes.
</li>
<li>
<code><b>StaticRegistry</b></code> : a static registry stores services in a static map.
<br>
Only the <code>APPLICATION</code> scope is supported. Attempting to store a service in another scope results in an exception.
</li>
<li>
<code><b>RegistryInitializer</b></code> : the RegistryInitializer is the most important component of SCIF.
<br>
It is responsible for:
<br>
<ul>
<li>Searching the classpath for classes declaring the @Service annotation</li>
<li>Injecting dependencies in the various form supported by the IoC Framework:
<ul>
<li><b>Field injection</b>. This is done using simply runtime reflection.</li>
<li><b>Method (getter) injection</b>. This is done by generating dynamically a subclass that takes care of the
<i>Lazy Loading</i> of the dependency.
</li>
</ul>
</li>
</ul>
</li>
<li><code><b>@Service</b></code> : this annotation identifies services to be searched for in the classpath.</li>
<li><code><b>@Resource</b></code> : this annotation identifies dependencies to be injected either at field level or getter (property) level.</li>
</ol>
<p>
The design is as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/07d50408-4f22-42ae-aed8-f33c15f76b69">
<img class="centered" style="width: 750px;" alt="SCIF Design" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/07d50408-4f22-42ae-aed8-f33c15f76b69" />
</a>
</div>
<br>
<p>
<b>Rules regarding <code>@Resource</code> handling</b>
</p>
<p>
The <code>RegistryInitializer</code> analyzes the classes annotated with <code>@Service</code> and handles <code>@Resource</code> annotations the following way:
</p>
<ul>
<li>If <code>@Resource</code> is declared on a field: in this case there is actually no need for bytecode manipulation. When the <code>RegistryInitializer</code> analyzes the services, it simply injects the reference to the field.</li>
<li>If <code>@Resource</code> is declared on a getter, and a corresponding setter is found: in this case, the system assumes he can use the setter to implement a cache for lazy loading. A subclass is created which overrides the getter to implement lazy loading. When calling the overriden getter, the latest will first call the original getter to see if the result service is already available. If that is the case, that service is returned. if it is not the case, the system will get the target service from the registry and use the corresponding setter to store it before returning it.</li>
<lI>If <code>@Resource</code> is declared on arbitrary method or on a getter without a corresponding setter: in this case the system cannot assume he can use the underlying field (through the setter) as a cache, and simply returns the target service from the registry at every call.</li>
</ul>
<p>
The <code>RegistryInitializer</code> uses the following tools / libraries:
</p>
<ul>
<li><code>ReflectionUtils</code> : a little helper class simplifying some operations related to Runtime Reflection.</li>
<li><code>Reflections</code> (org.reflection.Reflections) : a mandatory package when it comes to analyzing the classpath looking for classes declaring a certain annotation.</li>
<li><code>Javassist</code> : the bytecode manipulation library.</li>
</ul>
<a name="sec43"></a>
<h3>4.3 Some focus on code</h3>
<p>
We'll see below the most important pieces of code of the SCIF Framework.
</p>
<p>
(Reminder, <code><span style="color: DarkRed">I am coloring relevant Javassist API calls in dark red</span></code>)
</p>
<p>
It all starts with the method <code>RegistryInitializer.init()</code> that takes care of the whole shebang:
</p>
<pre>
<span style="color: green;">/**
* Initialize a registry from the root package (prefix) given as argument.
* <br />
* The RegistryInitializer will search for classes annotated with @Service and add them
* to the returned registry.
*
* @param rootPackage
* @return A Registry containing all discovered services
* @throws RegistryInitializationException in case of any error
*/</span>
<b>public static</b> Registry init(String rootPackage) <b>throws</b> RegistryInitializationException {
<b>try</b> {
Reflections reflections = <b>new</b> Reflections(rootPackage);
Set<Class<?>> annotated = reflections.getTypesAnnotatedWith(Service.class);
<span style="color: green;">// Create service wrappers</span>
Map<Class<?>, ServiceWrapper> wrappers = initServiceWrappers(annotated);
<span style="color: green;">// Build set of methods and fields to be analyzed</span>
analyzeWrappers(wrappers);
<span style="color: green;">// Now the complicated part, overwrite getters / methods using Javassist,</span>
<span style="color: green;">// dynamically creating a subclass with bytecode manipulation</span>
enhancePropertyGetters(wrappers);
StaticRegistry registry = <b>new</b> StaticRegistry();
<span style="color: green;">// Instantiate all the services and store them in Registry
// with class name as service name</span>
initializeRegistry(wrappers, registry);
<span style="color: green;">// Then do the easy thing : inject service on dependencies expressed on fields</span>
proceedFieldInjection(wrappers, registry);
<b>return</b> registry;
} <b>catch</b> (ReflectiveOperationException | NotFoundException | CannotCompileException e) {
logger.error (e, e);
<b>throw</b> new RegistryInitializationException (e.getMessage(), e);
}
}
</pre>
<p>
The really interesting call here is <code>enhancePropertyGetters(wrappers);</code>. This is where we use bytecode manipulation to generate the subclass dynamically and override the getter of the method declaring the <code>@Resource</code> annotation.
<br>
We won't present the other methods but we'll see the listing of this <code>enhancePropertyGetters()</code> method:
</p>
<pre>
<b>private static void</b> enhancePropertyGetters(Map<Class<?>, ServiceWrapper> wrappers)
<b>throws</b> NotFoundException, CannotCompileException, ClassNotFoundException {
<span style="color: DarkRed">ClassPool.doPruning</span> = <b>true</b>;
<span style="color: DarkRed">ClassPool</span> pool = <span style="color: DarkRed">ClassPool.getDefault</span>();
pool.<span style="color: DarkRed">appendClassPath</span>(<b>new</b> <span style="color: DarkRed">LoaderClassPath</span>(
Thread.currentThread().getContextClassLoader()));
<b>for</b> (ServiceWrapper wrapper : wrappers.values()) {
<b>if</b> (wrapper.getMethodsToEnhance().size() > <span style="color: blue;">0</span>) {
<span style="color: DarkRed">CtClass</span> superClazz = pool.<span style="color: DarkRed">get</span>(wrapper.getServiceName());
<span style="color: green;">// Unfortunately I need to go with the sub-class approach
// I cannot change the original class since it has already been loaded and
// javassist cannot change a class that is already loaded (that would require
// changing linking and javassist cannot do that)</span>
<span style="color: DarkRed">CtClass</span> clazz = pool.<span style="color: DarkRed">makeClass</span>(wrapper.getServiceName() + <span style="color: blue;">"$javassist_sub"</span>);
clazz.<span style="color: DarkRed">stopPruning</span>(<b>true</b>);
clazz.<span style="color: DarkRed">setSuperclass</span>(superClazz);
clazz.<span style="color: DarkRed">setModifiers</span>(Modifier.PUBLIC);
...
<span style="color: green;">// Add registry on class if not already one. The field might already have been</span>
<span style="color: green;">// added on a parent class. If this is the case, don't add it again</span>
injectRegistryField(pool, clazz);
<span style="color: green;">// Proceed with method modification</span>
<b>for</b> (Method method : wrapper.getMethodsToEnhance()) {
<span style="color: green;">// Various cases :</span>
<span style="color: green;">// 1. Method doesn't have the form of a getter</span>
<b>if</b> (!method.getName().startsWith(<span style="color: blue;">"get"</span>)) {
<span style="color: green;">// => Just override method so it returns the service from registry</span>
overrideMethod(clazz, method);
}
<span style="color: green;">// 2. Method is a getter</span>
<b>else</b> {
<b>try</b> {
Method setter = ReflectionUtils.getSetter(wrapper.getServiceClass(),
ReflectionUtils.getPropertyName(method),
method.getReturnType());
<span style="color: green;">// 2.2 setter is found
// => Lazy loading : use underlying field as cache.
// If it it set, do nothing.
// If is is null, look for service and attach it
// Needs to override the getter for doing my business and
// delegate to the setter for setting the field</span>
overrideGetter(clazz, method, setter);
} <b>catch</b> (NoSuchMethodException e) {
logger.debug (e, e);
<span style="color: green;">// 2.1 No setter could be found</span>
<span style="color: green;">// => Just rewrite method so it returns the service
// from the registry</span>
overrideMethod(clazz, method);
}
}
}
...
<span style="color: green;">// make new subclass available to classloader</span>
pool.<span style="color: DarkRed">toClass</span>(clazz, Thread.currentThread().getContextClassLoader(), <b>null</b>);
clazz.<span style="color: DarkRed">stopPruning</span>(<b>false</b>);
<span style="color: green;">// use the new subclass instead of the original class from now on</span>
Class<?> subClazz = Class.forName(clazz.getName());
wrapper.overrideClass(subClazz);
}
}
}
</pre>
<p>
In the code above, the interesting calls are <code>overrideGetter()</code>, <code>overrideMethod()</code> and <code>injectRegistryField()</code> since these are the methods where bytecode manipulation occurs.
<br>
Let's look at these methods:
</p>
<pre>
<b>private static void</b> overrideGetter(<span style="color: DarkRed">CtClass</span> clazz, Method getter, Method setter)
<b>throws</b> CannotCompileException {
String targetService = getter.getReturnType().getCanonicalName();
<span style="color: DarkRed">CtMethod</span> newMethod = <span style="color: DarkRed">CtNewMethod.make</span>(
<span style="color: blue;">"public "</span> + targetService + <span style="color: blue;">" "</span> + getter.getName() + <span style="color: blue;">"() { \n"</span> +
<span style="color: blue;">""</span> +
<span style="color: blue;">" "</span> + targetService + <span style="color: blue;">" retObject = super."</span> + getter.getName() + <span style="color: blue;">"(); "</span>+
<span style="color: blue;">" if (retObject == null) {"</span> +
<span style="color: blue;">" retObject = ("</span> + targetService + <span style="color: blue;">") \n"</span> +
<span style="color: blue;">" getRegistry().getService(\""</span> + targetService + <span style="color: blue;">"\"); "</span> +
<span style="color: blue;">" super."</span> + setter.getName() + <span style="color: blue;">"(retObject);"</span> +
<span style="color: blue;">" }"</span> +
<span style="color: blue;">" return retObject;"</span> +
<span style="color: blue;">""</span> +
<span style="color: blue;">"}"</span>,
clazz);
clazz.<span style="color: DarkRed">addMethod</span>(newMethod);
}
<b>private static void</b> overrideMethod(<span style="color: DarkRed">CtClass</span> clazz, Method method)
<b>throws</b> CannotCompileException {
String targetService = method.getReturnType().getCanonicalName();
<span style="color: DarkRed">CtMethod</span> newMethod = <span style="color: DarkRed">CtNewMethod.make</span>(
<span style="color: blue;">"public "</span> + targetService + <span style="color: blue;">" "</span> + method.getName() + <span style="color: blue;">"() { \n"</span> +
<span style="color: blue;">""</span> +
<span style="color: blue;">" "</span> + targetService + <span style="color: blue;">" retObject = \n"</span> +
<span style="color: blue;">" ("</span> + targetService + <span style="color: blue;">")\n"</span> +
<span style="color: blue;">" getRegistry().getService(\""</span> + targetService + <span style="color: blue;">"\"); "</span> +
<span style="color: blue;">" return retObject;"</span> +
<span style="color: blue;">""</span> +
<span style="color: blue;">"}"</span>,
clazz);
clazz.<span style="color: DarkRed">addMethod</span>(newMethod);
}
<span style="color: green;">/**
* Inject a field to store the registry in the target clazz as well as a getter
* to retrieve that registry.
*
* @param pool the Javassist ClassPool to be used
* @param clazz the class to be modified this way
* @throws NotFoundException
* @throws CannotCompileException
*/</span>
<b>public static void</b> injectRegistryField(<span style="color: DarkRed">ClassPool</span> pool, <span style="color: DarkRed">CtClass</span> clazz)
<b>throws</b> NotFoundException, CannotCompileException {
<span style="color: DarkRed">CtField</span> registryField = null;
<b>try</b> {
registryField = clazz.<span style="color: DarkRed">getField</span>(<span style="color: blue;">"registry"</span>);
} <b>catch</b> (<span style="color: DarkRed">NotFoundException</span> e) {
<span style="color: green;">// ignored</span>
}
<b>if</b> (registryField == <b>null</b>) {
<span style="color: DarkRed">CtClass</span> registryClass = pool.<span style="color: DarkRed">get</span>(Registry.<b>class</b>.getName());
registryField = <b>new</b> <span style="color: DarkRed">CtField</span>(registryClass, <span style="color: blue;">"registry"</span>, clazz);
registryField.<span style="color: DarkRed">setModifiers</span>(Modifier.setPrivate(Modifier.STATIC));
clazz.<span style="color: DarkRed">addField</span>(registryField, <span style="color: blue;">"null"</span>);
<span style="color: DarkRed">CtMethod</span> registryGetter = <span style="color: DarkRed">CtNewMethod.getter</span>(<span style="color: blue;">"getRegistry"</span>, registryField);
registryGetter.<span style="color: DarkRed">setModifiers</span>(Modifier.PUBLIC);
clazz.<span style="color: DarkRed">addMethod</span>(registryGetter);
<span style="color: DarkRed">CtMethod</span> registrySetter = <span style="color: DarkRed">CtNewMethod.make</span>(
<span style="color: blue;">"public static void setRegistry ("</span> + Registry.<b>class</b>.getName() +
<span style="color: blue;">" holder) { "</span> +
<span style="color: blue;">" registry = holder; "</span> +
<span style="color: blue;">"} "</span>,
clazz);
clazz.<span style="color: DarkRed">addMethod</span>(registrySetter);
}
}
</pre>
<p>
We've seen the most important pieces of code of the SCIF Framework above.
<br>
The framework itself is available for download in the next section.
</p>
<a name="sec44"></a>
<h3>4.4 DemoApp : Comet Tennis</h3>
<p>
I integrated the SCIF framework in the comet-tennis demo application. It's a small application I wrote initially here : <a href="https://www.niceideas.ch/roller2/badtrash/entry/comet-having-fun-with-the#sec31">comet-tennis</a>.
<br>
This application uses a few services and the idea here is to use the SCIF framework to manage these services and inject their dependencies.
</p>
<p>
The new package with the SCIF framework integrated is available <a download="comet_tennis_src_0.4.tar.gz" type="application/tar+gzip" href="https://www.niceideas.ch/roller2/badtrash/mediaresource/2eeba15c-dc3c-4546-adf8-a41cd4b31932">here</a>.
</p>
<a name="sec5"></a>
<h2>5. Conclusion </h2>
<p>
Bytecode manipulation is a lot of fun and opens a whole new world of possibilities on the JVM. It's the only way to implement advanced tooling such as IoC Containers, ORM frameworks, boilerplate code generators, etc.
<br>
Normally, bytecode manipulation is something rather pretty difficult to achieve ... except with Javassist.
<br>
Javassist makes bytecode manipulation so easy and straightforward. The ability to write dynamically in simple strings actual java source code and add it on the fly as bytecode to classes being manipulated is striking. Javassist is in my opinion the simplest way to perform bytecode manipulation in Java.
</p>
<p>
I covered <a href="#sec12">above</a> some use cases for bytecode manipulation, there are many others, for instance tampering with licence checking systems of non-free software (Hush. I said nothing)
</p>
<p>
In my career, I have encountered many situations where I wish afterwards I have known Javassist since it would have been pretty helpful. Let me mention two:
</p>
<ol>
<li>
Some 15 years ago, I was working on a pretty big J2EE Websphere application with a lot of EJBS. Tracking the user flow in the distributed system was a nightmare due to the complexity of the business processes and the business rules, so we ended up adding logging information each and every time a business method was entered and left, such as <code>log.debug ("ENTER - business method")</code> and <code>log.debug ("LEAVE - business method")</code>.
<br>
In regards to troubleshooting, this may sound stupid but it ended up being not only pretty convenient but rather really our single and only way to figure what was going on in some situations in such a enormous software.
<br>
Adding these two lines of code (plus a few <code>try { ... } finally { ... }</code> statements to make sure the leaving trace was always output made us add thousands of lines of code to the application ... which could have been replaced by a few lines of code java agent and some javassist magic.
</li>
<li>
Some 10 years ago, I was working for a banking institution on a big Java application making an extensive use of Hibernate. The problem there is that we were trying to map a nice and meaningful business model to a legacy data model. With a lot of hibernate tricks we pretty much succeeded in achieving the mapping, using a lot of custom and pretty tricky code in hibernate session listeners to handle the relationships that hibernate as not able to handle natively.
<br>
There as well, we ended up writing thousands of lines of specific glue code in hibernate listeners which we could have replaced by a pretty simple Javassist base framework to complement the missing features of hibernate.
</li>
</ol>
<p>
You might want to have a look at the second article in this serie available here : <a href="https://www.niceideas.ch/roller2/badtrash/entry/bytecode-manipulation-with-javassist-for1">Bytecode manipulation with Javassist for fun and profit part II: Generating toString and getter/setters using bytecode manipulation</a>.
</p>
<p>
Part of this article is available as a slideshare presentation here: <a href="https://www.slideshare.net/JrmeKehrli/bytecode-manipulation-with-javassist-for-fun-and-profit">https://www.slideshare.net/JrmeKehrli/bytecode-manipulation-with-javassist-for-fun-and-profit</a>.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/lean-startup-a-focus-on
The Lean Startup - A focus on Practices
Jerome Kehrli
2017-01-28T05:05:07-05:00
2020-07-22T03:14:48-04:00
<!-- The Lean Startup - A focus on Practices -->
<p>
A few years ago, I worked intensively on a pet project: AirXCell (long gone ...)
<br>
What was at first some framework and tool I had to write to work on my Master Thesis dedicated to Quantitative Research in finance, became after a few months somewhat my most essential focus in life.
<br>
Initially it was really intended to be only a tool providing me with a way to have a Graphical User Interface on top of all these smart calculations I was doing in R. After my master thesis, I surprised myself to continue to work on it, improving it a little here and a little there. I kept on doing that until the moment I figured I was dedicated several hours to it every day after my day job.
<br>
Pretty soon, I figured I was really holding an interesting software and I became convinced I could make something out of it and eventually, why not, start a company.
</p>
<p>
And of course I did it all wrong.
</p>
<p>
Instead of <b>finding out first if there was a need and a market for it</b>, and then <i>what should I really build to answer this need</i>, I spent hours every day and most of my week-ends developing it further towards what I was convinced was the minimum set of feature it should hold before I actually try to meet some potential customers to tell them about it.
<br>
So I did that for more than a year and a half until I came close to burn-out and send it all to hell.
</p>
<p>
Now the project hasn't evolve for three years. The thing is that I just don't want to hear about it anymore. I burnt myself and I am just disgusted about it. Honestly it is pretty likely that at the time of reading this article, the link above is not even reachable anymore.
<br>
When I think of the amount of time I <strike>invested</strike> wasted in it, and the fact that even now, three years after, I still just don't want to hear anything about this project anymore, I feel so ashamed. Ashamed that I didn't take a leap backwards, read a few books about startup creation, and maybe, who knows, discover <I>The Lean Startup</i> movement before.
<br>
Even now, I still never met any potential customer, any market representative. Even worst: I'm still pretty convinced that there is a need and a market for such a tool. But I'll never know for sure.
</p>
<p>
Such stories, and even worst, stories of startups burning millions of dollars for nothing in the end, happen every day, still today.
</p>
<p>
Some years ago, Eric Ries, Steve Blank and others initiated <b><i>The Lean Startup</i></b> movement. <i>The Lean Startup</i> is a movement, an inspiration, a set of principles and practices that any entrepreneur initiating a startup would be well advised to follow.
<br>
Projecting myself into it, I think that if I had read Ries' book before, or even better Blank's book, I would maybe own my own company today, around AirXCell or another product, instead of being disgusted and honestly not considering it for the near future.
<br>
In addition to giving a pretty important set of principles when it comes to creating and running a startup, <i>The Lean Startup</i> also implies an extended set of Engineering practices, especially software engineering practices.
</p>
<p>
This article focuses on presenting and detailing these <b>Software Engineering Practices from the Lean Startup Movement</b> since, in the end, I believe they can benefit from any kind company, from initiating startup to well established companies with Software Development Activities.
<br>
By Software Engineering practices, I mean software development practices of course but not only. Engineering is also about analyzing the features to be implemented, understanding the customer need and building a successful product, not just writing code.
</p>
<!-- The Lean Startup - A focus on Practices -->
<p>
A few years ago, I worked intensively on a pet project : AirXCell (long gone ...)
<br>
What was at first some framework and tool I had to write to work on my Master Thesis dedicated to Quantitative Research in finance, became after a few months somewhat my most essential focus in life.
<br>
Initially it was really intended to be only a tool providing me with a way to have a Graphical User Interface on top of all these smart calculations I was doing in R. After my master thesis, I surprised myself to continue to work on it, improving it a little here and a little there. I kept on doing that until the moment I figured I was dedicated several hours to it every day after my day job.
<br>
Pretty soon, I figured I was really holding an interesting software and I became convinced I could make something out of it and eventually, why not, start a company.
</p>
<p>
And of course I did it all wrong.
</p>
<p>
Instead of <b>finding out first if there was a need and a market for it</b>, and then <i>what should I really build to answer this need</i>, I spent hours every day and most of my week-ends developing it further towards what I was convinced was the minimum set of feature it should hold before I actually try to meet some potential customers to tell them about it.
<br>
So I did that for more than a year and a half until I came close to burn-out and send it all to hell.
</p>
<p>
Now the project hasn't evolve for three years. The thing is that I just don't want to hear about it anymore. I burnt myself and I am just disgusted about it. Honestly it is pretty likely that at the time of reading this article, the link above is not even reachable anymore.
<br>
When I think of the amount of time I <strike>invested</strike> wasted in it, and the fact that even now, three years after, I still just don't want to hear anything about this project anymore, I feel so ashamed. Ashamed that I didn't take a leap backwards, read a few books about startup creation, and maybe, who knows, discover <I>The Lean Startup</i> movement before.
<br>
Even now, I still never met any potential customer, any market representative. Even worst: I'm still pretty convinced that there is a need and a market for such a tool. But I'll never know for sure.
</p>
<p>
Such stories, and even worst, stories of startups burning millions of dollars for nothing in the end, happen every day, still today.
</p>
<p>
Some years ago, Eric Ries, Steve Blank and others initiated <b><i>The Lean Startup</i></b> movement. <i>The Lean Startup</i> is a movement, an inspiration, a set of principles and practices that any entrepreneur initiating a startup would be well advised to follow.
<br>
Projecting myself into it, I think that if I had read Ries' book before, or even better Blank's book, I would maybe own my own company today, around AirXCell or another product, instead of being disgusted and honestly not considering it for the near future.
<br>
In addition to giving a pretty important set of principles when it comes to creating and running a startup, <i>The Lean Startup</i> also implies an extended set of Engineering practices, especially software engineering practices.
</p>
<p>
This article focuses on presenting and detailing these <b>Software Engineering Practices from the Lean Startup Movement</b> since, in the end, I believe they can benefit from any kind company, from initiating startup to well established companies with Software Development Activities.
<br>
By Software Engineering practices, I mean software development practices of course but not only. Engineering is also about analyzing the features to be implemented, understanding the customer need and building a successful product, not just writing code.
</p>
<p>
Part of this article is available as a slideshare presentation here :
<a href="http://www.slideshare.net/JrmeKehrli/lean-startup-72100971">http://www.slideshare.net/JrmeKehrli/lean-startup-72100971</a> as well as a PDF document here : <a href="https://www.niceideas.ch/lean-startup.pdf">https://www.niceideas.ch/lean-startup.pdf</a>.
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. The Lean Startup</a>
<ul>
<li><a href="#sec11">1.1 Origins </a></li>
<li><a href="#sec12">1.2 The movement</a></li>
<li><a href="#sec13">1.3 Principles </a></li>
<li><a href="#sec14">1.4 The Feedback Loop </a></li>
<li><a href="#sec15">1.5 Business Model Canvas and Lean Canvas </a></li>
<li><a href="#sec16">1.6 Customer Development</a></li>
</ul>
</li>
<li><a href="#sec2">2. The four steps to the Epiphany</a>
<ul>
<li><a href="#sec21">2.1 Overview</a></li>
<li><a href="#sec22">2.2 A 4 steps process</a></li>
</ul>
</li>
<li><a href="#sec3">3. Lean startup practices</a>
<ul>
<li><a href="#sec31">3.1 Customer Discovery</a>
<ul>
<li><a href="#sec311">3.1.1 Get out of the building</a></li>
<li><a href="#sec312">3.1.2 Problem interview</a></li>
<li><a href="#sec313">3.1.3 Solution interview</a></li>
</ul>
</li>
<li><a href="#sec32">3.2 Customer Validation</a>
<ul>
<li><a href="#sec321">3.2.1 MVP</a></li>
<li><a href="#sec322">3.2.2 Fail Fast</a></li>
</ul>
</li>
<li><a href="#sec33">3.3 Re-adapt the product</a>
<ul>
<li><a href="#sec331">3.3.1 Metrics Obsession</a></li>
<li><a href="#sec332">3.3.2 Pivot</a></li>
</ul>
</li>
<li><a href="#sec34">3.4 Get new customers</a>
<ul>
<li><a href="#sec341">3.4.1 Pizza Teams</a></li>
<li><a href="#sec342">3.4.2 Feature Teams</a></li>
<li><a href="#sec343">3.4.3 Build vs. Buy</a></li>
<li><a href="#sec344">3.4.4 A/B Testing</a></li>
<li><a href="#sec345">3.4.5 Scaling Agile</a></li>
</ul>
</li>
<li><a href="#sec35">3.5 Company creation</a></li>
</ul>
</li>
<li><a href="#sec4">4. Conclusions</a></li>
</ul>
<a name="sec1"></a>
<h2>1. The Lean Startup</h2>
<p>
<b>The Lean Startup</b> is today a movement, initiated and supported by some key people that I'll present below.
<br>
But it's also a framework, an inspiration, an approach, a methodology with a set of fundamental principles and practices for helping entrepreneurs increase their odds of building a successful startup.
<br>
Lean Startup cannot be thought as a set of tactics or steps. Don't expect any checklist (well, at least not only checklists) or any recipe to be applied blindly.
</p>
<p>
The approach is built around two main objectives:
</p>
<ol>
<li>Teaching entrepreneurs how to drive a startup through the process of steering (<i>Build-Measure-Learn</i> feedback loop). </li>
<li>Enabling entrepreneurs to scale and grow the business with maximum acceleration</li>
</ol>
<p>
<b>Lean Startup Practices</b>
</p>
<p>
The Lean Startup methodology can be divided in two sets of practices:
</p>
<ol>
<li>The <b>steering practices</b> : designed to minimize the total time through the Build-Measure-Learn feedback loop and </li>
<li>The <b>acceleration practices</b> : which allow Lean Startups to grow without sacrificing the startup's speed and agility</li>
</ol>
<p>
This is developed further in <a href="#sec2">2. The four steps to the Epiphany</a>.
</p>
<a name="sec11"></a>
<h3>1.1 Origins </h3>
<p>
<b>The Lean Movement</b>
</p>
<p>
<b><a href="https://en.wikipedia.org/wiki/Lean_thinking">Lean thinking</a></b> is a <b>business methodology</b> that aims to provide a new way to think about how to organize human activities to deliver more benefits to society and value to individuals while <b>eliminating waste</b>.
<br>
Lean thinking is a <b>new way of thinking any activity</b> and seeing the waste inadvertently generated by the way the process is organized
</p>
<p>
The aim of lean thinking is to create a <b>lean enterprise</b>, one that <b>sustains growth</b> by aligning customer satisfaction with employee satisfaction, and that <b>offers innovative products</b> or services profitably while <b>minimizing unnecessary over-costs</b> to customers, suppliers and the environment.
</p>
<p>
The Lean Movement finds its roots in Toyotism and values <b>performance</b> and <b>continuous improvement</b>. The Lean Movement really rose in the early 90's and the lean tradition has adopted a number of practices from Toyota's own learning curve.
<br> Some worth to mention:
</p>
<ul>
<li><b><a href="https://en.wikipedia.org/wiki/Kaizen">Kaizen</a></b> (Continuous Improvement) : is a strategy where employees at all levels of a company work together pro-actively to achieve regular, incremental improvements to the manufacturing process. The point of Kaizen is that improvement is a normal part of the job, not something to be done "when there is time left after having done everything else", that should involve the company as a whole, from the CEO to the assembly line workers.</li>
<li><b><a href="https://en.wikipedia.org/wiki/Kanban">Kanban</a></b> (Visual Billboard) : is a scheduling system and visual management tool used in <i>Lean Manufacturing</i> to signal steps in their manufacturing process. The system's highly visual nature allows teams to communicate more easily on what work needed to be done and when. It also standardizes cues and refines processes, which helps to reduce waste and maximize value.</li>
</ul>
<p>
Plus strong emphasizes on <i>Autonomation</i>, <i>Visualization</i>, etc.
</p>
<p>
<b>The Lean Startup</b>
</p>
<p>
The author, I should say <i>initial author</i>, of the Lean Startup methodology, Eric Ries, explains in his book "<i>The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses</i>", that traditional management practices and ideas are not adequate to tackle the entrepreneurial challenges of startups.
</p>
<p>
By exploring and studying new and existing approaches, Ries found that adapting <i>Lean thinking</i> to the context of entrepreneurship would allow to discern between <i>value-creating activities</i> versus <i>waste</i>.
</p>
<p>
Thus, Ries, decided to apply lean thinking to the process of innovation. After its initial development and some refinement, as he states, <b>the Lean Startup</b> represents a new approach to creating continuous innovation that builds on many previous management and product development ideas, including lean manufacturing, design thinking, customer development, and agile development.
</p>
<a name="sec12"></a>
<h3>1.2 The movement </h3>
<p>
I would highly recommend this enlightening article - <a href="http://www.salimvirani.com/the-history-of-leanstartup-and-how-to-make-sense-of-it-all/">The History Of Lean Startup</a> - that does a pretty great job explaining how and why the following guys got together and initiated the <i>Lean Startup Movement</i> (aside from a few things I do not agree with).
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/8b002746-dd21-4668-970e-dafcaa864567">
<img class="centered" style="width: 500px; " alt="Lean Startup Movement" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/8b002746-dd21-4668-970e-dafcaa864567" />
</a>
</div>
<br>
<p>
Blank, Ries, Osterwalder and Maurya are the founders or initiators of the <i>Lean Startup Movement</i>. Eric Ries is considered as the leader of the movement, while Steve Blank considers himself as its godfather.
<br>
Osterwalder and Maurya's work on business models is considered to fill a gap in Ries and Blank's work on processes, principles and practices. In Steve Blank's "The four Steps the the Epiphany", the business model section is a vague single page.
<br>
Furtherly, Maurya's "<i>Running Lean</i>" magnificently completes Blank's work on <i>Customer Development</i>. We'll get to that.
</p>
<a name="sec13"></a>
<h3>1.3 Principles </h3>
<p>
In my opinion, the most fundamental aspect of Lean Startup is the <i>Build-Measure-Learn</i> loop, or, in the context of the <i>Customer Development Process</i>, the <i>Customer Discovery - Customer Validation - Re-adapt the product</i> loop. <br>
The idea is to be able to loop in laboratory mode, mostly with prototypes and interviews, in an iterative research process, with as little costs as possible, about the product to be developed. A startup should spend as little investment as possible in terms of product development as long as it has no certainty in regards to the customer needs, the right product to be developed, the potential market, etc.
<br>
This is really key, before hiring employees and starting to develop a product, the entrepreneur should have certainty about the product to be developed and its market.
<br>
Premature scaling is the immediate cause of the <a href="https://steveblank.com/2009/09/07/the-customer-development-manifesto-the-death-spiral-part-3/">Death Spiral</a>.
</p>
<p>
Before digging any further into this, below are the essential principles that characterize <i>The Lean Startup</i> approach, as reported by Eric Ries' book.
</p>
<p>
<b>Entrepreneurs are everywhere</b>
</p>
<p>
You don't have to work in a garage to be in a startup. The concept of entrepreneurship includes anyone who works within Eric Ries' definition of a startup, which I like very much BTW.
<br>
His definition is as follows :
</p>
<div class="centering">
<div class="centered">
<b>A startup is a human institution designed to create new products and services under conditions of extreme uncertainty.</b>
</div>
</div>
<br>
<p>
That means entrepreneurs are everywhere and the Lean Startup approach can work in any size company, even a very large enterprise, in any sector or industry.
</p>
<p>
<b>Entrepreneurship is management</b>
</p>
<p>
A startup is an institution, not just a product, and so it requires a new kind of management specifically geared to its context of extreme uncertainty.
<br>
In fact, Ries believes "<i>entrepreneur</i>" should be considered a job title in all modern companies that depend on innovation for their future growth
</p>
<p>
<b>Validated learnings</b>
</p>
<p>
Startups exist not just to make stuff, make money, or even serve customers. They exist to learn how to build a sustainable business. This learning can be validated scientifically by running frequent experiments that allow entrepreneurs to test each element of their vision.
</p>
<p>
<b>Innovation accounting</b>
</p>
<p>
To improve entrepreneurial outcomes and hold innovators accountable, we need to focus on the boring stuff: how to measure progress, how to set up milestones, and how to prioritize work. This requires a new kind of accounting designed for startups-and the people who hold them accountable.
</p>
<p>
<b>Build-Measure-Learn</b>
</p>
<p>
The fundamental activity of a startup is to turn ideas into products, measure how customers respond, and then learn whether to <b>pivot or persevere</b>. All successful startup processes should be geared to accelerate that <b>feedback loop</b>.
</p>
<a name="sec14"></a>
<h3>1.4 The Feedback Loop</h3>
<p>
The feedback loop is represented as below.
<br>
The five-part version of the <i>Build-Measure-Learn</i> schema helps us see that the real intent of building is to test "<i>ideas</i>" - not just to build blindly without an objective.
<br>
The need for "<i>data</i>" indicates that after we measure our experiments we'll use the data to further refine our learning. And the new learning will influence our next ideas. So we can see that the goal of Build-Measure-Learn isn't just to build things, the goal is to build things to validate or invalidate the initial idea.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/4b4e8fe7-e841-46cc-996b-6b00df12b175">
<img class="centered" style="width: 650px;" alt="Build-Measure-Learn" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/4b4e8fe7-e841-46cc-996b-6b00df12b175" />
</a>
</div>
<br>
<p>
Again, the goal of <i>Build-Measure-Learn</i> is not to build a final product to ship or even to build a prototype of a product, but to <b>maximize learning</b> through incremental and iterative engineering.
<br>
In this case, learning can be about product features, customer needs, distribution channels, the right pricing strategy, etc.
<br>
The "<i>build</i>" step refers to building an <a href="#sec321"><b>MVP</b></a> (Minimal Viable Product).
<br>
It's critical here to understand that an MVP does not mean <i>the product with fewer features</i>. Instead, an MVP should be seen as the simplest thing that you can show to customers to get the most learning at that point in time. Early on in a startup, an MVP could well simply be a set of Powerpoint slides with some fancy animations, or whatever is sufficient to demonstrate a set of features to customers and get feedback from it. Each time one builds an MVP one should also define precisely what one wants to test/measure.
<br>
Later, as more is learned, the MVP goes from low-fidelity to higher fidelity, but the goal continues to be to maximize learning not to build a beta/fully featured prototype of a product or a feature.
</p>
<p>
In the end, the <i>Build-Measure-Learn</i> framework lets startups be fast, agile and efficient.
</p>
<a name="sec15"></a>
<h3>1.5 Business Model Canvas and Lean Canvas </h3>
<p>
Evolution on Business Models and the relative processes were surprisingly missing or poorly addressed from Ries' and Blank's initial work.
<br>
Fortunately, Osterwalder and Maurya caught up and filled the gap.
</p>
<p>
<b>Business Model Canvas</b>
</p>
<p>
The <a href="https://en.wikipedia.org/wiki/Business_Model_Canvas">Business Model Canvas</a> is a strategic management template invented by Alexander Osterwalder and Yves Pigneur for developing new business models or documenting existing ones.
<br>
It is a visual chart with elements describing a company's value proposition, infrastructure, customers, and finances. It assists companies in aligning their activities by illustrating potential trade-offs.
</p>
<p>
<b>Lean Canvas</b>
</p>
<p>
The Lean Canvas is a version of the Business Model Canvas adapted by Ash Maurya specifically for startups. The Lean Canvas focuses on addressing broad customer problems and solutions and delivering them to customer segments through a unique value proposition.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/4446245e-b296-4c9f-84d0-c6a5fb1bd747">
<img class="centered" style="width: 500px; " alt="Lean Canvas" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/4446245e-b296-4c9f-84d0-c6a5fb1bd747" />
</a>
</div>
<br>
<p>
So how should one use the Lean Canvas?
</p>
<ol>
<li><b>Customer Segment and Problem</b><br>
Both Customer Segment and Problem sections should be filled in together.
<br>
Fill in the list of potential <i>customers</i> and <i>users</i> of your product, distinguish customers (willing to pay) clearly from users, then refine each and every identified customer segment. Be careful not no try to focus on a too broad segment at first, think of facebook whose first segment was only Harvard students.
<br>
Fill in carefully the problem encountered by your identified customers.
</li>
<li><b>UVP - Unique Value Proposition</b><br>
The UVP is the unique characteristic of your product or your service making it different from what is already available on the market an that makes it worth the consideration of your customers. Focus on the main problem you are solving and what makes your solution different.
</li>
<li><b>Solution</b><br>
Filling this is initially is tricky, since knowing about the solution for real requires trial and error, build-measure-learn loop, etc. In an initial stage one shouldn't try to be to precise here and keep things pretty open.
</li>
<li><b>Channels</b><br>
This consists in answering: how should you get in touch with your users and customers ? How do you get them to know about your product ? Indicate clearly your communication channels.
</li>
<li><b>Revenue Stream and Cost Structure</b><br>
Both these sections should also be filled in together.
<br>
At first, at the time of the initial stage of the startup, this should really be focused on the costs and revenues related to launching the MVP (how to interview 50 customers ? Whats the initial burn rate ? etc.)
<br>
Later this should evolve towards an initial startup structure and focus on identifying the <i>break-even</i> point by answering the question : how many customers are required to cover my costs ?
</li>
<li><b>Key Metrics</b><br>
Ash Maurya refers to Dave McClure Pirate Metrics to identify the relevant KPIs to be followed : <br>
Aquisition - How do user find you ?<br>
Activation - Do user have a great first experience ?<br>
Retention - Do users come back ?<br>
Revenue - How do you make money ?<br>
Referral - Do users tell others ?
</li>
<li><b>Unfair Advantage</b><br>
This consists in indicating the adoption barriers as well as the competitive advantages of your solution. An <i>unfair advantage</i> is defined as something that cannot be copied easily neither bought.
</li>
</ol>
<p>
<b>Lean Startup : test your plan !</b>
</p>
<p>
Using the new "Build - Measure - Learn" diagram, the question then becomes, "What hypotheses should I test?". This is precisely the purpose of the initial Lean Canvas,
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/9400cf47-0a75-4f96-87ed-662a381ae070">
<img class="centered" style="width: 750px; " alt="Canvas Principle" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/9400cf47-0a75-4f96-87ed-662a381ae070" />
</a>
</div>
<br>
<p>
And it brings us to another definition of a startup:
</p>
<div class="centering">
<div class="centered"><b>
A startup is a temporary organization designed to <i>search</i> for a repeatable and scalable business model.
</b></div>
</div>
<br>
<p>
And once these hypotheses fill the Lean Canvas (Or Business Model Canvas), the key approach is to <b>run experiments</b>. This leads us to the next section.
</p>
<a name="sec16"></a>
<h3>1.6 Customer Development</h3>
<p>
The Customer Development process is a simple methodology for taking new venture hypotheses and getting out of the building to test them. <i>Customer discovery</i> (see below) captures the founders' vision and turns it into a series of business model hypotheses. Then it develops a series of experiments to test customer reactions to those hypotheses and turn them into facts. The experiments can be a series of questions you ask customers. Though, most often an MVP to help potential customers understand your solution accompanies the questions.
</p>
<p>
Startups are building an MVP to learn the most they can, not to get a prototype!
</p>
<p>
The goal of designing these experiments and minimal viable products is not to get data. The data is not the endpoint. Anyone can collect data. <b>The goal is to get insight</b>. The entire point of getting out of the building is to inform the founder's vision.
<br>
The insight may come from analyzing customer answers, but it also may come from interpreting the data in a new way or completely ignoring it when realizing that the idea is related to a completely new and disruptive market that even doesn't exist yet.
</p>
<p>
<b>Customer Development instead of Product Development</b>
</p>
<p>
More startup fail from a <i>lack of customers</i> rather than from a failure of Product Development.
</p>
<p>
The Customer Development model delineates all the customer-related activities in the early stage of a company into their own processes and groups them into four easy-to-understand steps: <i>Customer Discovery</i>, <i>Customer Validation</i>, <i>Customer Creation</i>, and <i>Company Building</i>.
<br>
These steps mesh seamlessly and support a startup's ongoing product development activities. Each step results in specific deliverables and involves specific practices.
</p>
<p>
As its name should communicate, the Customer Development model focuses on developing customers for the product or service your startup is building. Customer Development is really about finding a market for your product. It is built upon the idea that the founder has an idea but he doesn't know if the clients he imagines will buy. He needs to check this point and it is better if he does it soon.
</p>
<a name="sec2"></a>
<h2>2. The four steps to the Epiphany</h2>
<p>
Shortly put, Steve Blank proposes that companies need a <b>Customer Development process</b> that complements, or even in large portions replaces, their <i>Product Development Process</i>. The <i>Customer Development process</i> goes directly to the theory of <a href="https://en.wikipedia.org/wiki/Product/market_fit">Product/Market Fit</a>.
<br>
In "<i>The four steps to the Epiphany</i>", Steve Blank provides a roadmap for how to get to Product/Market Fit.
</p>
<a name="sec21"></a>
<h3>2.1 Overview</h3>
<p>
<b>The Path to Disaster: The Product Development Model</b>
</p>
<p>
The traditional product development model has four stages:
</p>
<ol>
<li>concept/seed, </li>
<li>product development, </li>
<li>beta test, </li>
<li>and launch.</li>
</ol>
<p>
That product development model, when applied to startups, suffers from a lot of flaws. They basically boil down to:
</p>
<ul>
<li>Customers were nowhere in that flow chart</li>
<li>The flow chart was strictly linear</li>
<li>Emphasis on execution over learning</li>
<li>Lack of meaningful milestones for sales/marketing</li>
<li>Treating all startups alike</li>
</ul>
<p>
What's the alternative? Before we get to that, one final topic is the <i>technology life cycle adoption curve</i>, i.e. adoption happens in phases of early adopters (tech enthusiasts, visionaries), mainstream (pragmatists, conservatives), and skeptics.
<br>
Between each category is a <i>chasm</i>, the largest is between the early adopters and the mainstream.
<br>
Crossing the chasm is a success problem. But you're not there yet, "customer development" lives in the realm of the early adopter.
</p>
<p>
<b>The Path to Epiphany: The Customer Development Model</b>
</p>
<p>
Most startups lack a process for discovering their markets, locating their first customers, validating their assumptions, and growing their business.
<br>
The <b>Customer Development Model</b> creates the process for these goals.
</p>
<a name="sec22"></a>
<h3>2.2 A 4 steps process</h3>
<p>
The four stages the <i>Customer Development Model</i> are: customer discovery, customer validation, customer creation, and company creation.
</p>
<ol>
<li><b>Customer discovery</b>: understanding customer problems and needs</li>
<li><b>Customer validation</b>: developing a sales model that can be replicated</li>
<li><b>Customer creation / Get new Customers</b>: creating and driving end user demand</li>
<li><b>Customer building / Company Creation</b>: transitioning from learning to executing</li>
</ol>
<p>
We can represent them as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/9f0c263d-9308-4431-929d-7c7111eee8bf">
<img class="centered" style="width: 750px; " alt="Customer Development" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/9f0c263d-9308-4431-929d-7c7111eee8bf" />
</a>
</div>
<br>
<p>
I won't go any further in this article in describing these steps, their purpose and reasons.
<br>
To be honest Blank's book is pretty heavy and not very accessible. Happily Blank's did a lot of presentations around his book that one can find on youtube or elsewhere. In addition, there are a lot of excellent summaries and text explanations available online on Blank's book and I let the reader refer to this material should he want more information.
</p>
<p>
Instead, I want to focus in this article on the <b>Software Engineering Practices</b> inferred from The Lean Startup approach, since, again, I believe they are very important for any kind of corporation with an important Software Development activity.
<br>
And yet again, Software Engineering practices go beyond solely Software Development practices, but cover every activity in the company aimed at identifying and developing the product.
</p>
<a name="sec3"></a>
<h2>3. Lean startup practices</h2>
<p>
So I want to present the most essentials principles and practices introduced and discussed by <i>the Lean Startup</i> approach.
<br>
These principles and practices are presented on the following schema attached to the stages of the <i>Customer Development</i> process where I think they make more sense:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/2f0ac2c3-13b4-4a24-a2fd-61ef32a66941">
<img class="centered" style="width: 750px; " alt="Lean Startup Practices" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/2f0ac2c3-13b4-4a24-a2fd-61ef32a66941" />
</a>
</div>
<br>
<p>
<b>Important notes</b>
</p>
<ul>
<li>I attached the practices to the step where I think they make more sense, where I think they bring the most added value or should be introduced. But bear in mind that such a <i>categorization</i> is highly subjective and questionable. If you yourself believe some practices should be attached to another step, well just leave a comment and move on.</li>
<li>Also, there are other practices of course. I mention here and will be discussing below the ones that seem the most appealing to me, myself and I. Again my selection is highly subjective and personal. If you think I am missing something important, just leave a comment and move on.
</ul>
<p>
The rest of this paper intends to describe all these engineering - mostly software engineering - practices since, again, at the end of the day, I strongly believe that they form the most essential legacy of the Lean Startup movement and that they can benefit any kind of company, not only startups.
</p>
<a name="sec31"></a>
<h3>3.1 Customer Discovery</h3>
<p>
Customer Discovery, focuses on understanding customer problems and needs. Its really about searching for the <i>Product-Solution Fit</i>, turning the founders' initial hypotheses about their market and customers into facts.
<br>
The Problem-Solution Fit occurs when entrepreneurs identify relevant insights that can be addressed with a suggested solution. As Osterwalder describes it, this fit happens when there is evidence that customers care about certain problems that need to be solved or needs, and, there is a value proposition designed that addresses those needs.
</p>
<p>
In Customer Discovery the startup aims at understanding customer problems and needs and, also, to ideate potential solutions that could be valuable based on the findings. Similarly, Osterwalder calls these problems and needs as <i>jobs, pains and gains</i>.
</p>
<p>
The three practices I want to emphasize at this stage are as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/6438fafe-c2e2-42f0-8d82-31183b686c4b">
<img class="centered" style="width: 650px; " alt="Customer Discovery" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/6438fafe-c2e2-42f0-8d82-31183b686c4b" />
</a>
</div>
<br>
<a name="sec311"></a>
<h4>3.1.1 Get out of the building</h4>
<p>
<b>If you're not Getting out of the Building, you're not doing Customer Development and Lean Startup.</b>
<br>
There are no facts inside the building, only opinions.
</p>
<p>
If you aren't actually talking to your customers, you aren't doing Customer Development. And talking here is really speaking, with your mouth. Preferably in-person, but if not, a video call would work as well, messaging or emailing doesn't.
</p>
<p>
As Steve Blank said "<i>One good customer development interview is better for learning about your customers / product / problem / solution / market than five surveys with 10'000 statistically significant responses.</i>"
</p>
<p>
The problem here is that tech people, especially software engineers, try to avoid going out of the building as much as possible. But this is so important. Engineers need to fight against their nature and get out of the building and talk to customers as much as possible; find out who they are, how they work, what they need and what your startup needs to do, to build and then sell its solution.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/81e8236c-b6f1-4eae-b7c2-97ecf5225245">
<img class="centered" style="width: 250px; " alt="Keep Calm and get out of the building" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/81e8236c-b6f1-4eae-b7c2-97ecf5225245" />
</a>
</div>
<br>
<p>
In fact, so many engineers, just as myself, spent months of working on a prototype or even a complete solution, sometimes for several years, before actually meeting a first potential customer, and discovering the hard way that all this work has been for nothing.
<br>
As hard as it is, Engineers should not work one one single line of code, even not one single powerpoint presentation before having met at least a twenty potential customers or representatives and conducted formal <a href="#sec312">Problem interviews</a>.
<br>
After that, it's still not a question of writing lines of code, it's a question of investing a few hours - not more ! - in designing a demonstrable prototype for the next set of interviews, the <a href="#sec313">Solution interviews</a>. That prototype doesn't need to be actually working, it should only be demonstrable. A powerpoint presentation with clickable animations works perfectly!
</p>
<p>
Again, getting out of the building is not getting in the parking lot, it's really about getting in front of the customer.
<br>
At the end of the day, it's about <i>Customer Discovery</i>. And <i>Customer Discovery</i> is not sales, it's a lot of listening, a lot of understanding, not a lot of talking.
</p>
<p>
A difficulty that people always imagine is that young entrepreneurs with an idea believe that they don't know anybody, so how to figure out who to talk to ?
<br>
But at the time of Linkedin, facebook, twitter, it's hard to believe one cannot find a hundred of people to have a conversation with.
</p>
<p>
And when having a conversation with one of them, whatever else one's asking (<a href="#sec312">3.1.2 Problem interview</a>, <a href="#sec313">3.1.3 Solution interview</a>), one should ask two very important final questions:
</p>
<ol>
<li>
"<i>Who else should I be talking to ?</i>"
<br>
And because you're a pushy entrepreneur, when they give you those names, you should ask "<i>Do you mind if I sit here while you email them introducing me ?</i>"
</li>
<li>
"<i>What should I have really asked you ?</i>"
<br>
And sometimes that gets into another half hour related to what the customer is <i>really</i> worried about, what's really the customer's problem.
</ol>
<p>
Customer Discovery becomes really easy once you realize you don't need to get the world's best first interview.
<br>
In fact its the sum of these data points over time, it's not one's just going to be doing one and one wants to call on the highest level of the organization.
<br>
In fact you actually never want to call on the highest level of the organization because you're not selling yet, you don't know enough.
<br>
What one actually wants is to understand enough about the customers, their problems and how they're solving it today, and whether one's solution is something they would want to consider.
</p>
<p>
A few hints in regards to how to get out of the building:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/4c4c8421-b5ee-4b59-abb3-1198bdf3f9ac">
<img class="centered" style="width: 400px;" alt="Get out of the building" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/4c4c8421-b5ee-4b59-abb3-1198bdf3f9ac" />
</a>
</div>
<br>
<a name="sec312"></a>
<h4>3.1.2 Problem interview</h4>
<p>
Problem Interview is Ash Maurya's term for the interview you conduct to validate whether or not you have a real problem that your target audience has.
</p>
<p>
In the Problem Interview, you want to find out 3 things:
</p>
<ol>
<li><b>Problem</b> - What are you solving? - How do customers rank the top 3 problems?</li>
<li><b>Existing Alternatives</b> - Who is your competition? - How do customers solve these problems today?</li>
<li><b>Customer Segments</b> - Who has the pain? - Is this a viable customer segment?</li>
</ol>
<p>
Talking to people is hard, and talking to people in person is even harder. The best way to do this is building a script and sticking to it. Also don't tweak your script until you've done enough interviews so that your responses are consistent.
<br>
The main point is to collect the information that you will need to validate your problem, and to do it face-to-face, either in-person or by video call. It's actually important to see people and be able to study their body language as well.
</p>
<p>
The interview script - at least the initial you should follow until you have enough experience to build yours - is as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/b54ec66a-fda7-4938-94b4-10456335746d">
<img class="centered" style="width: 500px; " alt="Problem Interview" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b54ec66a-fda7-4938-94b4-10456335746d" />
</a>
</div>
<br>
<p>
If you have to remember just three rules for problem interviews here they are:
</p>
<ol>
<li>Do not talk about your business idea or product. You are here to understand a problem, not imagine or sell a solution yet.</li>
<li>Ask about past events and behaviours</li>
<li>No leading question, learn from the customer</li>
</ol>
<p>
After every interview, take a leap backwards, analyze the answers, make sure you understand everything correctly and synthesize the results.
<br>
After a few dozen of interviews, you should be a able to make yourself a clear understanding of the problem and initiate a few ideas regarding the solution to it.
<br>
Finding and validating your solution brings us to the next topic: the <i>Solution Interview</i>.
</p>
<p>
And what if a customer tells you that the issues you thought are important really aren't? Learn that you have gained important data.
</p>
<a name="sec313"></a>
<h4>3.1.3 Solution interview</h4>
<p>
In the Solution Interview, you want to find out three things:
</p>
<ol>
<li><b>Early Adopters</b> - Who has this problem? - How do we identify an early adopter?</li>
<li><b>Solution</b> - How will you solve the problems? - What features do you need to build?</li>
<li><b>Pricing/Revenue</b> - What is the pricing model? - Will customers pay for it?</li>
</ol>
<p>
The key point here is to understand how to come up with a solution fitting the problem, step by step getting to the right track with your prototype and also understanding what could be a pricing model.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/9ef49888-50eb-4778-bc52-3f4584eeabf7">
<img class="centered" style="width: 500px; " alt="Solution Interview" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/9ef49888-50eb-4778-bc52-3f4584eeabf7" />
</a>
</div>
<br>
<p>
A <i>demo</i> is actually important. Many products are too hard to understand without some kind of demo. If a picture is worth a thousand words, a demonstration is probably worth a million.
</p>
<p>
Identifying early adopters is also key.
<br>
Think of something: if one of the guys you meet tells you that you definitely hold something, ask him if he would want to buy it. If he says he would definitely buy it when it's ready and available, ask him if he would commit to this. If he says he commits to this, ask him if he would be ready to pay half of it now and have it when its ready, thus becoming a partner or an investor.
<br>
If you find ten persons committing on already paying for the solution you draw, you may not even need to search for investors, you already have them. And that is the very best proof you can find that your solution is actually something.
<br>
And customers or partners are actually the best possible type of investors.
</p>
<a name="sec32"></a>
<h3>3.2 Customer Validation</h3>
<p>
The second step of the Customer Development model, <i>Customer Validation</i>, focuses on developing a sales model that can be replicated. The sales model is validated by running experiments to test if customers value how the startup's products and services are responding to the customer problems and needs identified during the previous step.
<br>
If customers show no interest, then the startup can <a href="#sec332">pivot</a> to search for a better business model.
</p>
<p>
Customer Validation needs to happen to validate if the customers really care about the products and services that could be valuable to them. This second step is hence really about <i>Product-Market Fit</i> which occurs when there is a sales model that works, when customers think the proposed solution is valuable to them. This should be proven by evidence that customers care about the products and services that conform the value proposition.
</p>
<p>
Blank believes that <i>product-market fit</i> needs to happen before moving from Customer Validation to Customer Creation (or the <i>Search Phase</i> to the <i>Execution Phase</i>).
</p>
<p>
The two practices I want to emphasize at this stage are as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/87f50709-5984-4ac4-adf1-afb6935c4b9d">
<img class="centered" style="width: 650px;" alt="Customer Validation" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/87f50709-5984-4ac4-adf1-afb6935c4b9d" />
</a>
</div>
<br>
<a name="sec321"></a>
<h4>3.2.1 MVP</h4>
<p>
The <b>Minimum Viable Product</b> is an engineering product with just the set of features required to gather <i>validated learnings</i> about it - or some of its features - and its continuous development.
<br>
This notion of <i>Minimum Feature Set</i> is key in the MVP approach.
</p>
<p>
The key idea is that it makes really no sense developing a full and finalized product without actually knowing what will be the market reception and if all of it is actually worth the development costs.
<br>
Gathering insights and directions from an MVP avoids investing too much in a product based on wrong assumptions. Even further, The <i>Lean Startup</i> methodology seeks to avoid assumptions at all costs, see <a href="#sec14">1.4 The Feedback Loop </a> and <a href="#sec331">3.3.1 Metrics Obsession</a>.
</p>
<p>
The <i>Minimum Viable Product</i> should have just that set of initial features strictly required to have a valid product, usable for its very initial intent, and nothing more. In addition these features should be as minimalist as possible but without compromising the overall <i>User Experience</i>. A car should move, a balloon should be round and bounce, etc.
<br>
when adopting an MVP approach, the MVP is typically put at disposal at first only to <i>early adopters</i>, these customers that may be somewhat forgiving for the "naked" aspect of the product and more importantly that would be willing to give feedback and help steer the product development further.
</p>
<p>
Eric Ries defines the MVP as:
</p>
<div class="centering">
<div class="centered">
<b>
"The minimum viable product is that version of a new product a team uses to collect the maximum amount of validated learning about customers with the least effort."
</b>
</div>
</div>
<br>
<p>
The definition's use of the words <i>maximum</i> and <i>minimum</i> means it is really not formulaic. In practice, it requires a lot of judgement and experience to figure out, for any given context, what MVP makes sense.
</p>
<p>
The following chart is pretty helpful in understanding why both terms <i>minimum</i> and <i>viable</i> are equally important and why designing an MVP is actually difficult:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/abd771cd-b5a4-4a74-91c4-03951407517f">
<img class="centered" style="width: 320px; " alt="MVP" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/abd771cd-b5a4-4a74-91c4-03951407517f" />
</a>
</div>
<br>
<p>
When applied to a new feature of any existing prodUct instead of a brand new product, the MVP approach is in my opinion somewhat different. It consists of implementing the feature itself not completely; rather, a mock-up or even some animation simulating the new feature should be provided.
<br>
The mock-up or links should be properly instrumented so that all user reactions are recorded and measured in order to get insights on the actual demand of the feature and the best form it should take (<a href="#sec331">Measure Obsession</a>),
<br>
This is called a <b>deploy first, code later</b> method.
</p>
<p>
<a href="http://www.expressiveproductdesign.com/minimal-viable-product-mvp/">Fred Voorhorst' work</a> does a pretty good job in explaining what an MVP is:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/756e983e-5fd7-4dc1-a395-f5f1a69747f8">
<img class="centered" style="width: 700px; " alt="MVP - How-To" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/756e983e-5fd7-4dc1-a395-f5f1a69747f8" />
</a><br>
<div class="centered">
(Fred Voorhorst - Expressive Product Design - <a href="http://www.expressiveproductdesign.com/minimal-viable-product-mvp/">http://www.expressiveproductdesign.com/minimal-viable-product-mvp/</a>)
</div>
</div>
<br>
<p>
Developing an MVP is most definitely not the same as developing a sequence of elements which maybe, eventually combine into a product. A single wheel is not of much interest to a user wanting a personal transporter like a car, as illustrated by the first line.
<br>
Instead, developing an MVP is about developing the vision. This is not the same as developing a sequence of intermediate visions, especially not, if these are valuable products by themselves. As an example, a skateboard will likely neither interest someone in search for a car, as illustrated by the second line.
</p>
<p>
Developing an MVP means developing a sequence of prototypes through which you explore what is key for your product idea and what can be omitted.
</p>
<a name="sec322"></a>
<h4>3.2.2 Fail Fast</h4>
<p>
The key point of the "<b>fail fast</b>" principle is to quickly abandon ideas that aren't working. And the big dfficulty of course is not giving up too soon on an idea that could potentially be working. should one find the right channel, the right approach.
<br>
Fail fast means getting out of planning mode and into testing mode, eventually for every component, every single feature, every idea around your product or model of change. <i>Customer development</i> is the process that embodies this principle and helps you determine which hypotheses to start with and which are the most critical for your new idea.
</p>
<p>
It really is OK to fail if one knows the reason of the failure, and that is where most people go wrong. Once a site or a product fails then one needs to analyse why it bombed. It's only then that one can learn from it.
<br>
The key aspect here is really learning. And learning comes from experimenting, <b>trying things, <a href="#sec331">measuring</a> their success and <a href="#sec33">adapting</a></b>.
<br>
An entrepreneur should really be a pathologist investigating a death and finding the cause of the failure. Understanding the cause of a failure can only work if the appropriate measures and metrics around the experiment are in place.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/559f0efe-95eb-4e4e-bdf1-3f6bb998932a">
<img class="centered" style="width: 350px; " alt="Success - what it really looks like" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/559f0efe-95eb-4e4e-bdf1-3f6bb998932a" />
</a>
</div>
<br>
<p>
Now failing is OK as long as we learn from it and as long as we <b>fail as fast as possible</b>. Again, the whole <i>lean</i> idea is to avoid waste as much as possible and there's no greater waste than keeping investing on something that can ultimately not work. Failing as fast as possible, adapting the product, <a href="#sec332">pivoting</a> the startup towards its next approach as soon as possible is key.
<br>
But then again, the big difficulty is not to give up too soon on something that could possible work.
</p>
<div class="centering">
<div class="centered">
<b>
Fail fast, <br>
Learn faster, <br>
Succeed sooner !
</b>
</div>
</div>
<br>
<p>
So how do you know when to turn, when to drop an approach and adapt your solution ? How can you know it's not too soon?
</p>
<p>
<a href="#sec331">Measure, measure, measure</a> of course!
</p>
<p>
The testing of new concepts, failing, and building on failures are necessary when creating a great product.
<br>
The adage, "<i>If you can't measure it, you can't manage it</i>" is often used in management and is very important in <i>The Lean Startup</i> approach. By analyzing data, results can be measured, key lessons learned, and better initiatives employed.
</p>
<a name="sec33"></a>
<h3>3.3 Re-adapt the product</h3>
<p>
Customer development isn't predictable; you don't know what you're going to learn until you start. You'll need the ability to think on your feet and adapt as you uncover new information.
<br>
Adapting, in my opinion, is really re-adapting the product to the new situation, to the new knowledge you gained from the previous steps. And re-adapting the product, your solution, your approach is pivoting.
</p>
<p>
But I want to emphasize here that pivoting, or re-adapting the product, should only happen with the right data, the precise insights that give a clear new direction. Metrics and insight are essential.
</p>
<p>
The key practices here are as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/e39e7b10-58dc-412f-ac22-a2580de1ec96">
<img class="centered" style="width: 650px;" alt="Readapt Product" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/e39e7b10-58dc-412f-ac22-a2580de1ec96" />
</a>
</div>
<br>
<a name="sec331"></a>
<h4>3.3.1 Metrics Obsession</h4>
<p>
In the <i>build-measure-learn</i> loop, there is measure ... <i>The Lean Startup</i> makes from measuring everything an actual obsession. And I believe that this is a damn' good thing.
<br>
Think of it: what if you have an idea regarding a new feature or an evolution of your product and you don't already have the metrics that can help you take a sound and enlightened decision? You'll need to introduce the new measure and wait until you get the data. Waiting is not good for startups.
</p>
<p>
This is why I like thinking of it as a <b>Metrics Obsession</b>. Measure everything, everything you can think of!
<br>
And repeat a hundred times:
</p>
<div class="centering">
<div class="centered">
<b>
I will never ever again think that <br>
Instead I will <i>measure</i> that ...
</b>
</div>
</div>
<br>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/83f8b089-da6c-4f7f-b3ee-64ca3cdd46b3">
<img class="centered" style="width: 500px; " alt="Measure Obsession" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/83f8b089-da6c-4f7f-b3ee-64ca3cdd46b3" />
</a>
</div>
<br>
<p>
Or as Edward Deming said :
</p>
<div class="centering">
<div class="centered">
<b>
"In god we trust, all others must bring data"
</b>
</div>
</div>
<br>
<p>
Imagine you work on a webite. You should enhance your backend to measure, at least: amount of times a page has been displayed, count of users and different users dispalying the pages, amount of times a link or button has been clicked, by who it has been clicked, how much time after the containing page has been displayed, what is the user think time between 2 actions, what is the path of navigation from each and every user (actually build the graph and the counts along the branches), etc.
<br>
Measure everything! Don't hesitate to measure something you do not see any use for now. Sooner or later you will find a usage for that metrics, and that day, you better have it.
</p>
<p>
<b>How to choose good metrics ?</b>
</p>
<p>
Honestly there is no magic silver bullet and it can in fact be pretty difficult to pick up the right metric that would be most helpful to validate a certain hypothesis.
<br>
However, metrics should at all cost respect the three A's. Good metrics
</p>
<ul>
<li>are <b>actionable</b>,</li>
<li>can be <b>audited</b></li>
<li> are <b>accessible</b></li>
</ul>
<p>
An <b>actionable metric</b> is one that ties specific and repeatable actions to observed results. The <i>actionable</i> property of picked up metrics is important since it prevents the entrepreuneur from distorting the reality to his own vision. We speak of <i>Actionable vs. Vanity</i> Metrics.
<br>
Meaningless metrics such as "How many visitors ?", "How many followers ?" are vanity metrics and are useless.
</p>
<p>
Ultimately, your metrics should be useful to <b>measure progress against your own questions</b>.
</p>
<a name="sec332"></a>
<h4>3.3.2 Pivot</h4>
<p>
In the process of learning by iterations, a startup can discover through field returns with real customers that its product is either not adapted to the identified need, that it does not meet that need.
<br>
However, during this learning process, the startup may have identified another need (often related to the first product) or another way to answer the original need.
<br>
When the startup changes its product to meet either this new need or the former need in a different way, it is said to have performed a <b>Pivot</b>.
<br>
A startup can <i>pivot</i> several times during its existence.
</p>
<p>
A <i>pivot</i> is ultimately a <b>change in strategy</b> without <i>a change in vision</i>.
<br>
It is defined as a structured course correction designed <b>to test a new fundamental hypothesis</b> about the product, business model and engine of growth.
</p>
<p>
The vision is important. A startup is created because the founder has a vision and the startup is really built and organized around this vision. If the feedback from the field compromises the vision, the startup doesn't need to pivot, it needs to resign, cease its activities and another startup, another organization aligned to the new vision should prehaps be created.
</p>
<p>
There are various kind of pivots:
</p>
<ul>
<li><b>Zoom-In :</b> a single feature becomes the whole product </li>
<li><b>Zoom-Out :</b> the whole initial product becomes a feature of a new product </li>
<li><b>Customer segment :</b> Good product, bad customer segment </li>
<li><b>Customer need :</b> Repositioning, designing a completely new product (still sticking to the vision)</li>
<li><b>Platform : </b> Change from an application to a platform, or vice versa</li>
<li>Many others ...</li>
</ul>
<p>
<b>Pivot or Persevere</b>
</p>
<p>
Since entrepreneurs are typically emotionally attached to their product ideas, there is a tendency to hang in there too long. This wastes time and money. The pivot or persevere process forces a non-emotional review of the hypothesis.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/0ea0e3ba-a1ce-42cb-a805-c14c5a757537">
<img class="centered" style="width: 360px;" alt="Pivot or Persevere" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/0ea0e3ba-a1ce-42cb-a805-c14c5a757537" />
</a>
</div>
<br>
<p>
Unsuprisingly, knowing when to pivot is an art, not a science. It requires to be well thought through and can be pretty complicated to manage.
<br>
At the end of the day, knowing when to pivot or persevere requires experience and, more importantly, metrics: proper performance indicators giving the entrepreneur clear insights about the market reception of the product and the fitting of customer needs.
</p>
<p>
One thing seems pretty clear though, if it becomes clear to everyone in the company that another approach would better suit the customer needs, the startup needs to pivot, and fast.
</p>
<a name="sec34"></a>
<h3>3.4 Get new customers</h3>
<p>
The third step, the Customer Creation step, to "<i>start building end user demand to scale the business</i>", is the precursor to achieve <i>Business Model Fit</i>. Therefore, the Business Model Fit stage can be understood as validating the value for the company, where as the product-market fit focuses on validating the value for the customer.
</p>
<p>
The set of practices I deem important here are as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/d4a47435-9f33-4b76-a62f-b813e4349577">
<img class="centered" style="width: 650px; " alt="Get New Customers" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/d4a47435-9f33-4b76-a62f-b813e4349577" />
</a>
</div>
<br>
<p>
Again, attaching some of these practices here or in the next and last step can be subjective. In my opiniom, the startup needs to embrace this Lean and Agile principles and practices before it attempts to scale its organization, hence the reason why I considere these practices at this stage.
</p>
<a name="sec341"></a>
<h4>3.4.1 Pizza Teams</h4>
<p>
Jeff Bezos, Amazon's founder and CEO, always said that a team size shouldn't be larger than what two pizzas can feed, two american pizzas, not italian, needless to say.
<br>
This makes it 7 +/- 2 co-workers inside an Agile Team.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/c7fdfc67-0145-4cea-b6a6-3af9f714e19a">
<img class="centered" style="width: 500px;" alt="2 Pizzas Team" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/c7fdfc67-0145-4cea-b6a6-3af9f714e19a" />
</a>
</div>
<br>
<p>
More communication isn't necessarily the solution to communication problems - it's how it is carried out. Compare the interactions at a small dinner - or pizza - party with a larger gathering like a wedding. As group size grows, you simply can't have as meaningful of a conversation with every person, which is why people start clumping off into smaller clusters to chat.
<br>
For Bezos, small teams make it easier to communicate more effectively rather than more, to stay decentralized and moving fast, and encourage high autonomy and innovation. Here's the science behind why the two-pizza team rule works.
</p>
<p>
As team size grows, <b>the amount of one-on-one communication channels tend to explode</b>, following the formula to compute number of links between people which is <code> n ( n - 1) / 2 </code>.
<br>
This is <code>O(n<sup>2</sup>)</code> (Hello Engineers) and is really a <i>combinatorial explosion</i>.
<br>
If you take a basic two-pizza team size of, say, 6. That's 15 links between everyone. Double that group for a team of 12. That shoots up to 66 links.
<br>
The cost of coordinating, communicating, and relating with each other explodes to such a degree that it lowers individual and team productivity.
</p>
<p>
Under five co-workers, the team becomes fragile to external events and lacks creativity.
<br>
Beyond ten, communication loses efficiency, cohesion diminishes, parasitism behaviors and power struggles appear, and the performance of the team decreases very rapidly with the number of members.
</p>
<p>
The right size for an Agile Team is 7 +/- 2 persons.
</p>
<a name="sec342"></a>
<h4>3.4.2 Feature Teams</h4>
<p>
Let's first have a look at what is the other model: <i>Component Teams</i>.
</p>
<p>
<b>Component Teams</b>
</p>
<p>
<i>Components Teams</i> are the usual, the legacy model. In large IT oranizations, there is usually a development team dedicated to the front-end, the Graphical User Interface, another team dedicated to developing the Java (Or Cobol :-) backend, a team responsible to design and maintain the database, etc.
<br>
A Component Team is defined as a development Team whose primary area of concern is restricted to a specific component, or a set of components from a specific layer or tiers, of the system.
<br>
Prior to Agile, most large-scale systems were developed following the component team approach and the development teams were organized around components and subsystems.
</p>
<p>
The most essential drawback of <i>Component Teams</i> is obvious : most new features are spread among several components, creating dependencies that require cooperation between these teams. This is a continuing drag on velocity, as the individual teams spend much of their time discussing dependencies between teams and testing, assessing, fixing behaviour across components rather than delivering end user value as efficiently as possible.
<br>
An important direct consequence of this dependency is that any given feature can only be delivered as fast as can be delivered the component changes by the slowest (or most overloaded) component team.
</p>
<p>
<b>Feature Teams</b>
</p>
<p>
As such, in an Agile Organization, where the whole company is organized around Feature backlogs or Kanban, it makes a lot more sense to organize the various development teams in <b>Feature Teams</b>.
<br>
<i>Feature teams</i> are organized around user-centered functionality. Each and every team, is capable of delivering end-to-end user value throughout the software stack. Feature teams operate primarily with user stories, refactors and spikes. However, technical stories may also occasionally occur in their backlog.
<br>
A feature team is defined as a long-lived, cross-functional, cross-component team that completes many end-to-end customer features, one by one.
</p>
<p>
More Information on Feature Teams:
</p>
<ul>
<li><a href="http://www.scaledagileframework.com/features-and-components/"> From SAFe - Scaled Agile Framework</a></li>
<li><a href="https://less.works/less/structure/feature-teams.html">From LeSS - Large Scale Scrum framework</a></li>
</ul>
<p>
The difference between both models is well illustrated this way:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/cdee6c64-8084-4f23-86bd-79e30d784f0a">
<img class="centered" style="width: 700px; " alt="Component Team vs. Feature Teams" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/cdee6c64-8084-4f23-86bd-79e30d784f0a" />
</a><br>
<div class="centered">
(Source : <a href="https://less.works/less/structure/feature-teams.html">https://less.works/less/structure/feature-teams.html</a>)
</div>
</div>
<br>
<p>
A pretty good summary of the most essential differences between both models is available on the LeSS web site:
</p>
<div class="centering">
<div class="centered">
<table class="nicewithborder">
<thead>
<tr>
<th style="text-align: center; background-color: #DDDDDD;">component team</th>
<th style="text-align: center; background-color: #DDDDDD;">feature team</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center">optimized for delivering the <b>maximum number of lines of code</b></td>
<td style="text-align: center">optimized for delivering the <b>maximum customer value</b></td>
</tr>
<tr>
<td style="text-align: center">focus on increased individual productivity by implementing 'easy' lower-value features</td>
<td style="text-align: center">focus on high-value features and system productivity (value throughput)</td>
</tr>
<tr>
<td style="text-align: center">responsible for only part of a customer-centric feature</td>
<td style="text-align: center">responsible for complete customer-centric feature</td>
</tr>
<tr>
<td style="text-align: center">traditional way of organizing teams - follows Conway's law</td>
<td style="text-align: center">'modern' way of organizing teams - avoids Conway's law</td>
</tr>
<tr>
<td style="text-align: center">leads to 'invented' work and a forever-growing organization</td>
<td style="text-align: center">leads to customer focus, visibility, and smaller organizations</td>
</tr>
<tr>
<td style="text-align: center">dependencies between teams leads to additional planning</td>
<td style="text-align: center"><b>minimizes dependencies between teams to increase flexibility</b></td>
</tr>
<tr>
<td style="text-align: center">focus on single specialization</td>
<td style="text-align: center">focus on multiple specializations</td>
</tr>
<tr>
<td style="text-align: center">individual/team code ownership</td>
<td style="text-align: center"><b>shared product code ownership</b></td>
</tr>
<tr>
<td style="text-align: center">clear individual responsibilities</td>
<td style="text-align: center"><b>shared team responsibilities</b></td>
</tr>
<tr>
<td style="text-align: center">results in 'waterfall' development</td>
<td style="text-align: center"><b>supports iterative development</b></td>
</tr>
<tr>
<td style="text-align: center">exploits existing expertise; lower level of learning new skills</td>
<td style="text-align: center">exploits flexibility; continuous and broad learning</td>
</tr>
<tr>
<td style="text-align: center">works with sloppy engineering practices-effects are localized</td>
<td style="text-align: center">requires skilled engineering practices-effects are broadly visible</td>
</tr>
<tr>
<td style="text-align: center">contrary to belief, often leads to low-quality code in component</td>
<td style="text-align: center"><b>provides a motivation to make code easy to maintain and test</b></td>
</tr>
<tr>
<td style="text-align: center">seemingly easy to implement</td>
<td style="text-align: center">seemingly difficult to implement</td>
</tr>
</tbody>
</table>
<br>
(Source : <a href="https://less.works/less/structure/feature-teams.html">https://less.works/less/structure/feature-teams.html</a>)
</div>
</div>
<br>
<p>
The Analogy with a Star Trek team makes suprisingly and funnily a lot of sense.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/c91988a2-6aa1-4beb-8bdb-f62277d0decf">
<img class="centered" style="width: 500px; " alt="Star Trek" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/c91988a2-6aa1-4beb-8bdb-f62277d0decf" />
</a>
</div>
<br>
<p>
Think of a Star Trek spaceship. The crew is constituted by Commanding Officers, Medical Officers, Medical Staff, Engineering Officers, Engineering Staff, Science Officers, Scientists, etc.
<br>
These different functions, competencies and responsibilities are grouped together to work towards a common objective, its continuing mission: <i>to explore strange new worlds, to seek out new life and new civilizations, to boldly go where no one has gone before</i>.
</p>
<p>
Now imagine if Starfleet had instead put all the Commanding Officers in one ship, all medical staff in another ship, and so on. It would have been pretty difficult to make those ships actually do anything significant, don't you think ?
<br>
This is precisely the situation of <i>Component Teams</i>.
<br>
Just as with a Star Trek Ship, it makes a lot more sense to put all the required competencies together in a team (or ship) and assign them a clear objective, implementing that feature throughout the technology and software stack.
</p>
<a name="sec343"></a>
<h4>3.4.3 Build vs. Buy</h4>
<p>
This dilemma is as old as the world of computers: is it better to invest in developing a software that is best suited to your needs or should you rely on a software package or third party product that embed the capitalization and R&D of <b>another</b> software editor in order to - <b>apparently</b> - speed up your time to market ?
</p>
<p>
In order to be as efficient as possible on the build-measure-learn loop, it is essential to master your development process. For this reason, <i>tailor made</i> solutions are better because the adoption of a third party software package often requires to invest a lot of resources not in the development of your product, but instead in the development of workarounds, hacks and patchs to correct all the points on which the software package is poorly adapted to the specific and precise behavior required by your own product feature.
</p>
<p>
In the case of a startup, this aspect is catastrophic. Investing in the development of hacks and workarounds around a third party product, a product that one has in addition to pay for, sometimes depending on the number of machine or users, instead of developing the startup's core business, should just not happen.
</p>
<p>
This cost aspect is particularly critical of course when scaling the solution. When one multiplies the processors and the servers, the invoice climbs very quickly and not necessarily linearly, and the costs become very visible, no matter whether it is a business software package or an infrastructure brick.
</p>
<p>
This is precisely one of the arguments that led LinkedIn to gradually replace Oracle with a home solution: Voldemor.
</p>
<p>
Most technologies that make the buzz today in the world of high performance architectures are the result of developments made by the Web Giants that have been released as Open Source: Cassandra, developed by Facebook, Hadoop and HBase inspired by Google and developed at Yahoo, Voldemort by LinkedIn, etc.
</p>
<p>
<b>Open-Source software is cool</b>
</p>
<p>
Of course the cost problem doesn't apply to Open-Source and free to use software. In addition, instead of developing workarounds and patches around Open-Source Software, you can instead change its source, fork it and maintain your different baseline while still benefiting frome the developments made on the official baseline by merging it frequently.
</p>
<p>
At the end of the day, integrating an Open-Source software, in contrary to Editor / Closed Source Software, is pretty closed to developing it on your own, as long as you have the competencies to maintain it on your own should you need to.
<br>
Open-Source software is cool, go for it!
</p>
<a name="sec344"></a>
<h4>3.4.4 A/B Testing</h4>
<p>
A/B testing is a marketing technique that consists in proposing several variants of the same object that differ according to a single criterion (for example, the color of a package) in order to determine the version which lead to the best apprciation and acceptance from consumers.
<br>
A / B testing is used to qualify all kinds of multivariate tests.
</p>
<p>
An A/B test evaluates the respective performance of one or more partially or totally different versions of the same product or functionality by comparing them to the original version. The test consists in creating modified versions of the functionality by modifying as many elements as desired.
<br>
The idea is to split the visitors into two groups (hence the name A / B) and to present to each group a different version of the functionality or the product. Then, we should follow the path of the two groups, their appreciation of the functionality by means of ad'hoc metrics, and we consider which of the two variants gives the best result with respect to a given objective.
</p>
<p>
For instance, in order to tests if a <i>trial first approach</i> is more appealing and leads eventually to more sales than a mandatory buying:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/6899c72d-0165-4704-84ae-6ffad63f0d84">
<img class="centered" style="width: 450px;" alt="A/B Testing" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/6899c72d-0165-4704-84ae-6ffad63f0d84" />
</a>
</div>
<br>
<p>
The A/B test enables to validate very quickly the idea of introducing a trial period for a feature or a product.
</p>
<a name="sec345"></a>
<h4>3.4.5 Scaling Agile</h4>
<p>
Transforming a startup into a company, changing and scaling its organization is a unique, and yet challenging, opportunity to make it an agile organization keeping the <i>lean</i> genes on which it has been built.
<br>
The <i>agile</i> aspect here is essential and the approach here actually has a name: <b>Scaling Agile</b>.
</p>
<p>
<i>Scrum</i> and <i>Kanban</i> are two agile frameworks often used at the team level. Over the past decade, as they gained popularity, the industry has begun to adapt and use Agile in larger companies. Two methods (amongst others) emerged to facilitate this process: <a href="http://less.works/"><b>LeSS</b></a> (Large Scale Scrum) and <a href="http://www.scaledagileframework.com/"><b>SAFe</b></a> (Scaled Agile Framework). Both are excellent starting points for using Agile on a large scale within a company.
</p>
<p>
Both approaches differ a little but also have a lot in common: they consist of scaling agility first among multiple agile team within the R&D or Engineering department and then around it, by having the whole company organizing its activities in an agile way and centered on the engineering team, the product development team.
<br>
I won't be describing these both approaches any further here and I let the reader refere to both links above.
</p>
<p>
I just want to emphasize how important I believe that is. Scaling Agile is key in aligning business and IT engagement models.
</p>
<a name="sec35"></a>
<h3>3.5 Company creation</h3>
<p>
Company creation is the end phase, when all assumptions have been confirmed or adapted, when the product is build in an acceptable form, when the break-even pointit reached, and the startup should evolve to a corporation. When that moment is reached, startups must begin the transition from the temporary organization designed to search a business model to a structure focused on executing a validated model.
</p>
<p>
Company creation happens at the moment the company can transition from its informal, learning and discovery-oriented Customer Development team (startup, temporary organization) into formal departments with VPs of Sales, Marketing and Business Development.
<br>
At that moment, these executives should focus on building mission-oriented departments that can exploit the company's early market success.
</p>
<p>
This is a change of bracket. We think of <i>Company Creation</i> since it is really a question of creating a company, from what was "only" a startup. The temporary organization should evolve towards a sustainable and viable organization.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/c2fe6011-91cb-417e-88de-2fd047fc5e3a">
<img class="centered" style="width: 500px; " alt="Scale-Up" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/c2fe6011-91cb-417e-88de-2fd047fc5e3a" />
</a>
</div>
<br>
<p>
Describing anything further in regards to <i>Company Creation</i> exceeds the scope of this article focused on <i>Lean Startup Practices</i>.
<br>
I can only recommend reading Steve Blank's article on the subject (or the big chapter in the <i>"Four Steps to the Epihpany"</i>):
</p>
<ul>
<li><a href="https://steveblank.com/2010/01/14/a-startup-is-not-a-smaller-version-of-a-large-company/">A Startup is Not a Smaller Version of a Large Company</a></li>
<li><a href="https://steveblank.com/2009/12/21/the-elves-leave-middle-earth-%E2%80%93-soda%E2%80%99s-are-no-longer-free/">The Elves Leave Middle Earth - Sodas Are No Longer Free</a></li>
<li><a href="https://steveblank.com/2010/09/20/the-peter-pan-syndrome-%E2%80%93-the-startup-to-company-transition/">The Peter Pan Syndrome - The Startup to Company Transition</a></li>
<li><a href="https://steveblank.com/2015/02/12/what-do-i-do-now/">What Do I Do Now? The Startup Lifecycle</a></li>
</ul>
<a name="sec4"></a>
<h2>4. Conclusions</h2>
<p>
The Lean Startup is not dogmatic. It is first and foremost a question of being aware that the market and the customer are not in the architecture meetings, marketing plans, sales projections or key feature discussions.
</p>
<p>
Bearing this in mind, you will see assumptions everywhere. The key approach then consists in putting in place a discipline of validation of the hypotheses while keeping as key principle to validate the minimum of functionalities at any given time.
</p>
<p>
Before doing any line of code, the main questions to ask revolve around the triplet : <i>Client / Problem / Solution</i>
<br>
Do you really have a problem that is worth resolving? Is your solution the right one for your customer? Is he likely to buy it? For how much ? All the means are good to remove these hypotheses: interviews, market studies, models, whatever you can think of.
</p>
<p>
The next step is to know if the model that you came up with and have been able to test on a smaller scale is really repeatable and extensible.
<br>
How to put a product they have never heard of in the hands of the customers ? Will they understand it as well with its use and benefits ?
</p>
<p>
The Lean Startup is not an approach to be reserved only to mainstream websites or fancy internet products. Innovating by validating hypotheses as quickly as possible and limiting financial investment is obviously a logic that can be transposed to any type of engineering project, even if it is internal.
<br>
I am convinced that the practices and principles from <b>the Lean Startup</b> approach should be more widely used to avoid so many projects burning so much money and effort before being simply dropped.
</p>
<p>
Part of this article is available as a slideshare presentation here :
<a href="http://www.slideshare.net/JrmeKehrli/lean-startup-72100971">http://www.slideshare.net/JrmeKehrli/lean-startup-72100971</a> as well as a PDF document here : <a href="https://www.niceideas.ch/lean-startup.pdf">https://www.niceideas.ch/lean-startup.pdf</a>.
</p>
https://www.niceideas.ch/roller2/badtrash/entry/devops-explained
DevOps explained
Jerome Kehrli
2017-01-04T15:56:41-05:00
2017-11-19T14:34:21-05:00
<!-- DevOps explained -->
<p>
So ... I've read a lot of things recently on DevOps, a lot of very interesting things ... and, unfortunately, some pretty stupid as well. It seems a lot of people are increasingly considering that DevOps is resumed to mastering <code>chef</code>, <code>puppet</code> or docker containers. This really bothers me. DevOps is so much more than any tool such as puppet or docker.
</p>
<p>
This could even make me angry. DevOps seems to me so important. I've spent 15 years working in the engineering business for very big institutions, mostly big financial institutions. DevOps is a very key methodology bringing principles and practices that address precisely the biggest problem, the saddest factor of failure of software development projects in such institutions : the <i>wall of confusion</i> between developers and operators.
</p>
<p>
Don't get me wrong, in most of these big institutions being still far from a large and sound adoption of an Agile Development Methodology beyond some XP practices, there are many other reasons explaining the failure or slippage of software development projects.
<br>
But the <i>wall of confusion</i> is by far, in my opinion, the most frustrating, time consuming, and, well, quite stupid, problem they are facing.
</p>
<p>
So yeah... Instead of getting angry I figured I'd rather present here in a concrete and as precise as possible article what DevOps is and what it brings. Long story short, DevOps is not a set of tools. <b>DevOps is a methodology</b> proposing a set of <b>principles and practices</b>, period. The tools, or rather the toolchain - since the collection of tools supporting these practices can be quite extended - are only intended to support the practices.
<br>
In the end, these tools don't matter. The DevOps toolchains are today very different than they were two years ago and will be very different in two years. Again, this doesn't matter. What matters is a sound understanding of the principles and practices.
</p>
<p>
Presenting a specific toolchain is not the scope of this article, I won't mention any. There are many articles out there focusing on DevOps toolchains. I want here to take a leap backwards and present the principles and practices, their fundamental purpose since, in the end, this is what seems most important to me.
</p>
<p>
DevOps is a methodology capturing the practices adopted from the very start by the web giants who had a unique opportunity as well as a strong requirement to invent new ways of working due to the very nature of their business: the need to evolve their systems at an unprecedented pace as well as extend them and their business sometimes on a daily basis.
<br>
While DevOps makes obviously a critical sense for startups, I believe that the big corporations with large and old-fashioned IT departments are actually the ones that can benefit the most from adopting these principles and practices. I will try to explain why and how in this article.
</p>
<!-- DevOps explained -->
<p>
So ... I've read a lot of things recently on DevOps, a lot of very interesting things ... and, unfortunately, some pretty stupid as well. It seems a lot of people are increasingly considering that DevOps is resumed to mastering <code>chef</code>, <code>puppet</code> or docker containers. This really bothers me. DevOps is so much more than any tool such as puppet or docker.
</p>
<p>
This could even make me angry. DevOps seems to me so important. I've spent 15 years working in the engineering business for very big institutions, mostly big financial institutions. DevOps is a very key methodology bringing principles and practices that address precisely the biggest problem, the saddest factor of failure of software development projects in such institutions : the <i>wall of confusion</i> between developers and operators.
</p>
<p>
Don't get me wrong, in most of these big institutions being still far from a large and sound adoption of an Agile Development Methodology beyond some XP practices, there are many other reasons explaining the failure or slippage of software development projects.
<br>
But the <i>wall of confusion</i> is by far, in my opinion, the most frustrating, time consuming, and, well, quite stupid, problem they are facing.
</p>
<p>
So yeah... Instead of getting angry I figured I'd rather present here in a concrete and as precise as possible article what DevOps is and what it brings. Long story short, DevOps is not a set of tools. <b>DevOps is a methodology</b> proposing a set of <b>principles and practices</b>, period. The tools, or rather the toolchain - since the collection of tools supporting these practices can be quite extended - are only intended to support the practices.
<br>
In the end, these tools don't matter. The DevOps toolchains are today very different than they were two years ago and will be very different in two years. Again, this doesn't matter. What matters is a sound understanding of the principles and practices.
</p>
<p>
Presenting a specific toolchain is not the scope of this article, I won't mention any. There are many articles out there focusing on DevOps toolchains. I want here to take a leap backwards and present the principles and practices, their fundamental purpose since, in the end, this is what seems most important to me.
</p>
<p>
DevOps is a methodology capturing the practices adopted from the very start by the web giants who had a unique opportunity as well as a strong requirement to invent new ways of working due to the very nature of their business: the need to evolve their systems at an unprecedented pace as well as extend them and their business sometimes on a daily basis.
<br>
While DevOps makes obviously a critical sense for startups, I believe that the big corporations with large and old-fashioned IT departments are actually the ones that can benefit the most from adopting these principles and practices. I will try to explain why and how in this article.
</p>
<p>
(This article is available as a PDF document here <a href="https://www.niceideas.ch/devops.pdf">https://www.niceideas.ch/devops.pdf</a> and as a slideshare presentation here
<a href="https://www.slideshare.net/JrmeKehrli/devops-explained-72664261">https://www.slideshare.net/JrmeKehrli/devops-explained-72664261</a>)
</p>
<p>
<b>Summary</b>
</p>
<ul>
<li><a href="#sec1">1. Introduction</a>
<ul>
<li><a href="#sec11">1.1 The management credo </a></li>
<li><a href="#sec12">1.2 a typical IT organization </a></li>
<li><a href="#sec13">1.3 Ops frustration </a></li>
<li><a href="#sec14">1.4 Infrastructure automation </a></li>
<li><a href="#sec15">1.5 DevOps : For once, a magic silver bullet </a></li>
</ul>
</li>
<li><a href="#sec2">2. Infrastructure as Code</a>
<ul>
<li><a href="#sec21">2.1 Overview</a></li>
<li><a href="#sec22">2.2 DevOps Toolchains</a></li>
<li><a href="#sec22">2.3 Benefits</a></li>
</ul>
</li>
<li><a href="#sec3">3. Continuous Delivery</a>
<ul>
<li><a href="#sec31">3.1 Learn from the field</a></li>
<li><a href="#sec32">3.2 Automation</a></li>
<li><a href="#sec33">3.3 Deploy more often</a></li>
<li><a href="#sec34">3.4 Continuous Delivery requirements</a></li>
<li><a href="#sec35">3.5 Zero Downtime Deployments</a></li>
</ul>
</li>
<li><a href="#sec4">4. Collaboration</a>
<ul>
<li><a href="#sec41">4.1 The wall of confusion</a></li>
<li><a href="#sec42">4.2 Software Development Process</a></li>
<li><a href="#sec43">4.3 Share the Tools</a></li>
<li><a href="#sec44">4.4 Work Together</a></li>
</ul>
</li>
<li><a href="#sec5">5. Conclusion</a></li>
</ul>
<a name="sec1"></a>
<h2>1. Introduction</h2>
<p>
DevOps is not a question of tools, or mastering chef or docker. DevOps is a methodology, a set of principles and practices that help both developers and operators reach their goals while maximizing value delivery to the customers or the users as well as the quality of these deliverables.
</p>
<p>
The problem comes from the fact that developers and operators - while both required by corporations with large IT departments - have very different objectives.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/b5394bf3-7943-4b74-843f-bbdc56fd3cd1">
<img class="centered" style="width: 500px; " alt="Developers and Operators" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b5394bf3-7943-4b74-843f-bbdc56fd3cd1" />
</a>
</div>
<br>
<p>
This difference of objectives between developers and operators is called the <b>wall of confusion</b>. We'll see later precisely what that means any why I consider this something big and bad.
</p>
<p>
DevOps is a methodology presenting a set of principles and practices (tools are derived from these practices) aimed at having both these personas working towards an unified and common objective : <b>deliver as much value as possible for the company</b>.
</p>
<p>
And surprisingly, for once, there is a magic silver bullet for this. Very simply, the secret is to <b>bring agility to the production side</b>!
<br>
And that, precisely that and only that, is what DevOps is about !
</p>
<p>
But there are quite a few things I need to present before we can discuss this any further.
</p>
<a name="sec11"></a>
<h3>1.1 The management credo </h3>
<p>
What is the sinews of war of IT Management ? In other words, when it comes to Software Development Projects, what does management want first and foremost ?
</p>
<p>
Any idea ?
</p>
<p>
Let me put you on tracks : what is utmost important when developing a startup ?
</p>
<p>
<b>Improve Time To Market (TTM)</b> of course !
</p>
<p>
The <b>Time To Market</b> or TTM is the length of time it takes from a product being conceived until its being available to users or for sale to customers. TTM is important in industries where products are outmoded quickly.
<br>
In software engineering, where approaches, business and technologies change almost yearly, the TTM is a very important KPI (Key Performance Indicator).
<br>
The TTM is also very often called <b>Lead Time</b>
</p>
<p>
A first problem lays in the fact (as believed by many) that TTM and product quality are opposing attributes of a development process. As we will see below, improving quality (and hence stability) is the objective of operators while reducing lead time (and hence improving TTM) is the objective of developers.
<br>
Let me explain this.
</p>
<p>
An IT organization or department is often judged on these two key KPIs : the quality of the software, where the target is to have as little defects as possible, and the TTM, where the target is to be able to go from business ideas (often given by business users) to production - making the feature available to users or customers - as soon as possible.
<br>
The problem here is that most often these two distinct objectives are supported by two different teams : the <i>developers</i>, building the software, and the <i>operators</i>, running the software.
</p>
<a name="sec12"></a>
<h3>1.2 a typical IT organization </h3>
<p>
A typical IT organization, in a corporation owning an important IT department, looks as follows :
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/3f858c20-5a21-4130-abe5-4af5b93b07db">
<img class="centered" style="width: 900px; " alt="Typical IT Organization" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/3f858c20-5a21-4130-abe5-4af5b93b07db" />
</a>
</div>
<br>
<p>
Mostly for historical reasons (operators come from the hardware and telco business most often), operators are not attached to the same branch than developers. Developers belong to R&D while operators most of the time belong to Infrastructure department (or dedicated operation department).
</p>
<p>
Again, they have different objectives:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/ce967df5-4dca-4f37-a480-b683bd742259">
<img class="centered" style="width: 500px; " alt="Developers and Operators" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/ce967df5-4dca-4f37-a480-b683bd742259" />
</a>
</div>
<br>
<p>
In addition, and as a sidenote, these both teams sometimes even run on different budget. The development team uses the <i>build</i> budget while the operation team uses the <i>run</i> budget. These different budgets and the increasing needs to control and shorten the costs of IT in corporation tend to emphasize the opposition of objectives of the different teams.
<br>
(In parenthesis: nowadays, with the always and everywhere interconnection of people and objects pushing the digitalization of businesses and society in general, the old Plan / Build / Run framework for IT budgeting makes IMHO really no sense anymore, but that is another story)
</p>
<a name="sec13"></a>
<h3>1.3 Ops frustration </h3>
<p>
Now let's focus on operators a little and see, in average, how a typical <i>operation team</i> spends its time:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/60b9fcce-83ef-4639-b03a-34b367088181">
<img class="centered" style="width: 480px; " alt="Operator team time stats" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/60b9fcce-83ef-4639-b03a-34b367088181" />
</a><br>
<div class="centered">
(Source : Study from Deepak Patil [Microsoft Global Foundation Services] in 2006, via James Hamilton [Amazon Web Services] <a href="http://mvdirona.com/jrh/TalksAndPapers/JamesHamilton_POA20090226.pdf">http://mvdirona.com/jrh/TalksAndPapers/JamesHamilton_POA20090226.pdf</a>)
</div>
</div>
<br>
<p>
So almost 50% (47) of total time of Production Teams is dedicated to deployment related topics:
</p>
<ul>
<li>Actually doing deployment or</li>
<li>Fixing problems related to deployments</li>
</ul>
<p>
This is actually a pretty crazy KPI, one that should have been followed much sooner. The truth is, operator teams have been since their inception in the early age of Computer Engineering - 40 years ago, at the time computers were massively introduced in the industry - this kind of hackers running tons of commands manually to perform their tasks. They are used to long checklists of commands or manual processes to perform their duties.
<br>
Somehow, they suffer from the "<i>We always did it like this</i>" syndrome and challenged very little their ways of working over these 40 years.
<br>
But if you think of it, this is really crazy. In average, operators spend almost 50% of their time doing deployment related tasks!
</p>
<p>
This underlines two critical needs for evoluting these processes:
</p>
<ol>
<li>Automate the deployments to reduce the 31% time dedicated to these currently manual tasks.</li>
<li>Industrialize them (just as software development has been industrialized, thanks to XP and Agile) to reduce the 16% related to fixing these deployment related issues.</li>
</ol>
<a name="sec14"></a>
<h3>1.4 Infrastructure automation </h3>
<p>
In this regards, another statistic is pretty enlightening:
</p>
<div class="centering">
<div class="centered">
Probability of succeeding an installation expressed as a function of the number of manual operations
</div>
<br>
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/c4b31869-528a-498a-a86d-61ecb2fce5c2">
<img class="centered" style="width: 700px; " alt="Operator team time stats" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/c4b31869-528a-498a-a86d-61ecb2fce5c2" />
</a>
</div>
<br>
<p>
This is read the following way :
</p>
<ul>
<li>With only 5 manual commands, the probability of succeeding an installation drops to 86% already.</li>
<li>With 55 manual commands, the probability of succeeding an installation drops to 22%.</li>
<li>With 100 manual commands, the probability of succeeding an installation is close to 0! (2%)%</li>
</ul>
<p>
<i>Succeeding the installation</i> means that the software behaves in production as intended. Failing it means something will go wrong and some analysis will be required to understand what went wrong with the installation and some patches will need to be applied or some configuration corrected.
</p>
<p>
<b>So automating all of this and avoiding manual commands at all cost seems to be rather a good idea, doesn't it ?</b>
</p>
<p>
So what's the status in this regards in the industry:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/8b821d6f-2f2d-4df7-b8d7-248958db8701">
<img class="centered" style="width: 500px; " alt="Operator team time stats" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/8b821d6f-2f2d-4df7-b8d7-248958db8701" />
</a><br>
<div class="centered">
(Source : IT Ops & DevOps Productivity Report 2013 - Rebellabs - <a href="http://pages.zeroturnaround.com/rs/zeroturnaround/images/it-ops-devops-productivity-report-2013%20copy.pdf">http://pages.zeroturnaround.com/rs/zeroturnaround/images/it-ops-devops-productivity-report-2013%20copy.pdf</a>)
</div>
</div>
<br>
<p>
(To be perfectly honest, this statistic is pretty old - 2013 - I would expect a little different numbers nowadays)
</p>
<p>
Nonetheless, this gives a pretty good idea of how much is still to be accomplished in regards to Infrastructure automation and how much DevOps principles and practices are very important.
</p>
<p>
Again the web giants had to come up with a new approach, with new practices to address their needs of responsivness. What they started their engineering business in their early days, the practices they put in place is at the root of what is today DevOps.
</p>
<p>
Let's look at where the web giants stand now in this regards. A few examples:
</p>
<ul>
<li>Facebook has thousands of devs and ops, hundreds of thousands of servers. In average, an operator takes care of 500 servers (think automation is optional ?). They do two deployments a day (concept of deployment ring) </li>
<li>Flickr does 10 deployments a day</li>
<li>Netflix designs for failure! The software is designed from the grounds up to tolerate system failures. They test it all the time in production: 65'000 failure tests in production daily by killing random virtual machines ... and measuring that everything still behaves OK.</li>
</ul>
<p>
So what is their secret ?
</p>
<a name="sec15"></a>
<h3>1.5 DevOps : For once, a magic silver bullet </h3>
<p>
The secret is simply to <b>Extend Agility to Production</b>:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/77393bca-f284-443d-a48d-b1fadbc97789">
<img class="centered" style="width: 500px; " alt="DevOps" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/77393bca-f284-443d-a48d-b1fadbc97789" />
</a>
</div>
<br>
<p>
DevOps consists mostly in extending agile development practices by further streamlining the movement of software change thru the build, validate, deploy and delivery stages, while empowering cross-functional teams with full ownership of software applications - from design thru production support.
</p>
<p>
DevOps encourages <b>communication</b>, <b>collaboration</b>, <b>integration</b> and <b>automation</b> among software developers and IT operators in order to improve both the speed and quality of delivering software.
</p>
<p>
DevOps teams focus on standardizing development environments and automating delivery processes to improve delivery predictability, efficiency, security and maintainability. The DevOps ideals provide developers more control of the production environment and a better understanding of the production infrastructure.
<br>
DevOps encourages empowering teams with the autonomy to build, validate, deliver and support their own applications.
</p>
<p>
<b>So what are the core principles ?</b>
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/1524e087-8bff-44c2-a596-7f4597162a6a">
<img class="centered" style="width: 420px; " alt="DevOps Principle" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/1524e087-8bff-44c2-a596-7f4597162a6a" />
</a>
</div>
<br>
<p>
We'll now dig into these 3 essential principles.
</p>
<a name="sec2"></a>
<h2>2. Infrastructure as Code</h2>
<p>
Because humans make mistakes, because the human brain is terribly bad at repetitive tasks, because humans are slow compared to a shell script, and because we are humans after all, we should consider and handle infrastructure concerns just as we handle coding concerns!
</p>
<p>
Infrastructure as code (IaC) is the prerequisite for common DevOps practices such as version control, code review, continuous integration and automated testing. It consists in <b>managing</b> and <b>provisioning</b> computing infrastructure (containers, virtual machines, physical machines, software installation, etc.) and their configuration <b>through machine-processable definition</b> files or scripts, rather than the use of interactive configuration tools and manual commands.
</p>
<p>
I cannot stress enough how much this is a key principle of DevOps. It is really applying software development practices to servers and infrastructure.
<br>
Cloud computing enables complex IT deployments modeled after traditional physical topologies. We can automate the build of complex virtual networks, storage and servers with relative ease. Every aspect of server environments, from the infrastructure down to the operating system settings, can be codified and stored in a version control repository.
</p>
<a name="sec21"></a>
<h3>2.1 Overview</h3>
<p>
In a very summarized way, the levels of infrastructure and operation concerns at which automation should occur is represented on this schema:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/11e52f2c-2287-4304-b14f-67b82b5860fc">
<img class="centered" style="width: 700px; " alt="IaC Overview" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/11e52f2c-2287-4304-b14f-67b82b5860fc" />
</a>
</div>
<br>
<p>
The tools proposed as examples on the schema above are very much oriented towards <i>building</i> the different layers. But a devops toolchain does much more than that.
<br>
I think it's time I tell a little more about the notion of DevOps Toolchains.
</p>
<a name="sec22"></a>
<h3>2.2 DevOps Toolchains</h3>
<p>
Because DevOps is a cultural shift and collaboration between development, operations and testing, there is no single DevOps tool, rather, again, a set ogf them, or <i>DevOps toolchain</i> consisting of multiple tools. Such tools fit into one or more of these categories, which is reflective of the software development and delivery process:
</p>
<ul>
<li><b>Code</b> : Code development and review, version control tools, code merging</li>
<li><b>Build</b> : Continuous integration tools, build status</li>
<li><b>Test</b> : Test and results determine performance</li>
<li><b>Package</b> : Artifact repository, application pre-deployment staging</li>
<li><b>Release</b> : Change management, release approvals, release automation</li>
<li><b>Configure</b> : Infrastructure configuration and management, Infrastructure as Code tools</li>
<li><b>Monitor</b> : Applications performance monitoring, end user experience</li>
</ul>
<p>
Though there are many tools available, certain categories of them are essential in the DevOps toolchain setup for use in an organization.
</p>
<p>
Tools such as Docker (containerization), Jenkins (continuous Integration), Puppet (Infrastructure building) and Vagrant (virtualization platform) among many others are often used and frequently referenced in DevOps tooling discussions as of 2016.
</p>
<p>
<b>Versioning, Continuous Integration and Automated testing of infrastructure components</b>
</p>
<p>
The ability to <b>version</b> the infrastructure - or rather the infrastructure building scripts or configuration files - as well as the ability to <b>automated test</b> it are very important.
<br>
DevOps consists in finally adopting the same practices XP brought 30 years ago to software engineering to the production side.
<br>
Even further, Infrastructure elements should be <b>continuously integrated</b> just as software deliverables.
</p>
<a name="sec23"></a>
<h3>2.3 Benefits</h3>
<p>
There are so many benefits to DevOps. A non-exhaustive list could be as follows:
</p>
<ul>
<li><b>Repeatability and Reliability</b> : building the production machine is now simply running that script or that puppet command. With proper usage of docker containers or vagrant virtual machines, a production machine with the Operating System layer and, of course, all the software properly installed and configured can be set up by typing one single command - <b>One Single Command</b>. And of course this building script or mechanism is continuously integrated upon changes or when being developed, continuously and automatically tested, etc. <br>
Finally we can benefit on the operation side from the same practices we use with success on the software development side, thanks to XP or Agile.</li>
<li><b>Productivity</b> : one click deployment, one click provisioning, one click new environment creation, etc. Again, the whole production environment is set-up using one single command or one click. Now of course that command can well run for hours, but during that time the operator can focus on more interesting things, instead of waiting for a single individual command to complete before typing the next one, and that sometimes for several days...</li>
<li><b>Time to recovery !</b> : one click recovery of the production environment, period.</li>
<li><b>Guarantee that infrastructure is homogeneous</b> : completely eliminating the possibility for an operator to build an environment or install a software slightly differently every time is the only way to guarantee that the infrastructure is perfectly homogeneous and reproducible. Even further, with version control of scripts or puppet configuration files, one can rebuild the production environment precisely as it was last week, last month, or for that particular release of the software.</li>
<li><b>Make sure standards are respected</b> : infrastructure standards are not even required anymore. The standard is the code.</li>
<li><b>Allow developer to do lots of tasks themselves</b> : if developers become themselves suddenly able to re-create the production environment on their own infrastructure by one single click, they become able to do a lot of production related tasks by themselves as well, such as understanding production failures, providing proper configuration, implementing deployment scripts, etc.</li>
</ul>
<p>
These are the few benefits of IaC that I can think of by myself. I bet there are so many much more (suggestions in comments are welcome).
</p>
<a name="sec3"></a>
<h2>3. Continuous Delivery</h2>
<p>
Continuous delivery is an approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing, and releasing software faster and more frequently.
<br>
The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.
</p>
<p>
<b>Important note : Continuous Delivery ≠ Continuous Deployment</b> - continuous delivery is sometimes confused with continuous deployment. Continuous deployment means that every change is automatically deployed to production. Continuous delivery means that the team ensures every change can be deployed to production but may choose not to do it, usually due to business reasons. In order to do continuous deployment one must be doing continuous delivery
</p>
<p>
The key ideas behind continuous deliveries are:
</p>
<ul>
<li><b>The more often you deploy, the more you master the deployment process and the better you automate it</b>. If you have to do something 3 times a day, you <b>will</b> make it bullet proof and reliable soon enough, when you will be fed up of fixing the same issues over and over again.</li>
<li><b>The more often you deploy, the smallest will be the changesets you deploy</b> and hence the smallest will be the risk of something going wrong, or the chances of losing control over the changesets</li>
<li><b>The more often you deploy, the best will be your TTR (Time to Repair / Resolution)</b> and hence the sooner will be the feedback you will get from your business users regarding that feature and the easier it will be to change some things here and there to make it perfectly fit their needs (TTR is very similar to TTM in this regards).</li>
</ul>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/e95d84be-e1de-4991-b4ef-5042df77a096">
<img class="centered" style="width: 520px; " alt="Small changes / More often" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/e95d84be-e1de-4991-b4ef-5042df77a096" />
</a>
<br>
<div class="centered">
(Source : Ops Meta-Metrics: The Currency You Pay For Change - <a href="http://fr.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change-4608108">http://fr.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change-4608108</a>)
</div>
</div>
<br>
<p>
But continuous delivery is more than building a shippable, production-ready version of the product as often as possible. Continuous delivery refers to 3 key practices:
</p>
<ul>
<li>Learn from the fields</li>
<li>Automation</li>
<li>Deploy more often</li>
</ul>
<a name="sec31"></a>
<h3>3.1 Learn from the field</h3>
<p>
Continuous Delivery is key to be able to <b>learn from the field</b>. There is no truth in the development team, the truth lies in the head of the business users. Unfortunately, no one is able to really clearly express his mind, his will in a specification document, no matter the time he dedicates to this task. This is why Agility attempts to put the feature in the hands of the users to get their feedback as soon as possible, at all cost.
<br>
Doing Continuous delivery, as far as continuous deployment, and hence reducing lead time to its minimal possible value, is key to be able to learn the truth from the users, as soon as possible
</p>
<p>
But the truth doesn't come out in the form of a formal user feedback. One should never trust its users or rely on formal feedback to learn from users. One should trust its own measures.
<br>
<b>Measure obsession</b> is a very important notion from the <i>Lean Startup</i> movement but it's also very important in DevOps. One should measure everything! Finding the right metrics enabling the team to learn about the success or failures of an approach, about what would be better and what has the most success can be sometimes tricky. One should always take too many measures instead of missing the one that would enable the team to take an enlightened decision.
</p>
<p>
Don't think, know! And the only way to know is to measure, measure everything: response times, user think times, count of displays, count of API calls, click rate, etc. but not only. Find out about all the metrics that can give you additional insights about the user perception of a feature and measure them, all of them!
</p>
<p>
This can be represented as follows:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/ddfcb1af-1554-42b1-a7a4-1801d41e3822">
<img class="centered" style="width: 350px; " alt="Move Fast" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/ddfcb1af-1554-42b1-a7a4-1801d41e3822" />
</a>
</div>
<br>
<a name="sec32"></a>
<h3>3.2 Automation</h3>
<p>
Automation has already been discussed above in section <a href="#sec2">2. Infrastructure as code</a>.
</p>
<p>
I just want to emphasize here that continuous delivery is impossible without a properly and 100% automation of all infrastructure provisioning and deployment related tasks.
<br>
This is very important, let me repeat it once more: setting up an environment and deploying a production ready version of the software should take one click, one command, it should be entirely automated. Without it, it's impossible to imagine deploying the software several times a day.
</p>
<p>
In section <a href="#sec35">3.5 Zero Downtime Deployments</a> below we will mention additional important techniques helping Continuous Delivery as well.
</p>
<a name="sec33"></a>
<h3>3.3 Deploy more often</h3>
<p>
The DevOps credo is:
</p>
<div class="centering">
<div class="centered">
<span style="font-size: large;"><b><i>"If it hurts, do it more often !"</i></b></span>
</div>
</div>
<br>
<p>
This idea of doing painful things more frequently is very important in agile thinking.
<br>
Automated Testing, refactoring, database migration, specification with customers, planning, releasing - all sorts of activities are done as frequently as possible.
</p>
<p>
There are three good reasons for that:
</p>
<ol>
<li>
Firstly most of these tasks become much more difficult as the amount of work to be done increases, but when broken up into smaller chunks they compose easily.
<br>
Take Database migration for instance: specifying a large database migration involving multiple tables is hard and error prone. But if you take it one small change at a time, it becomes much easier to get each one correct. Furthermore you can string small migrations together easily into a sequence. Thus when one decomposes a large migration into a sequence of little ones, it all becomes much easier to handle. (As a sidenote, this is the essence of database refactoring)
</li>
<li>
The second reason is <i>Feedback</i>. Much of agile thinking is about setting up feedback loops so that we can learn more quickly. Feedback was already an important and explicit value of Extreme Programming. In a complex process, like software development, one has to frequently check where one stands and make course corrections. To do this, one must look for every opportunity to add feedback loops and increase the frequency with which one gets feedback so one can adjust more quickly.
</li>
<li>
The third reason is <i>practice</i>. With any activity, we improve as we do it more often. Practice helps to iron out the kinks in the process, and makes one more familiar with signs of something going wrong. If you reflect on what you are doing, you also come up with ways to improve your practice.
<br>
With software development, there's also in addition the potential for automation. Once one has done something a few times, it's easier to see how to automate it, and more importantly one becomes more motivated to automate it. Automation is especially helpful because it can increase speed and reduce the chance for error.
</li>
</ol>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/aefb0bb2-5f62-423c-a9b3-2de395e04221">
<img class="centered" style="width: 450px; " alt="Master the process" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/aefb0bb2-5f62-423c-a9b3-2de395e04221" />
</a>
</div>
<br>
<p>
Now one question remains : <b>how often to deliver with DevOps ?</b>
</p>
<p>
There is no straight answer to that. It really depends on the product, the team, the market, the company, the users, the operation needs, etc.
<br>
My best answer would be as follows: If you don't deliver at least every 2 weeks - or at the end of your sprint duration period - you do not even do Agile, not to speak of DevOps.
<br>
DevOps encourages to deliver as frequently as possible. In my understanding (please challenge that in the comments if you like), you should train your team to be able to deliver as frequently as possible. A sound approach, the one I'm using with my team is to deliver twice a day on a QA environment. The delivery process is fully automated: twice a day, at noon and at midnight, the machinery starts, builds the software components, runs integration tests, builds the Virtual Machines, start them, deploys the software components, configures them, runs functional tests, etc.
</p>
<a name="sec34"></a>
<h3>3.4 Continuous Delivery requirements</h3>
<p>
What does one need <b>before</b> being able to move to Continuous Delivery?
<br>
My checklist, in a raw fashion :
</p>
<ul>
<li>Continuous integration of both the software components development as well as the platform provisioning and setup.</li>
<li>TDD - Test Driven Development. This is questionable ... But in the end let's face it: TDD is really the single and only way to have an acceptable coverage of the code and branches with unit tests (and unit tests makes is so much easier to fix issues than integration or functional tests).</li>
<li>Code reviews ! At least codereviews ... pair programming would be better of course.</li>
<li>Continuous auditing software - such as Sonar.</li>
<li>Functional testing automation on production-level environment</li>
<li>Strong non-functional testing automation (performance, availability, etc.)</li>
<li>Automated packaging and deployment, independent of target environment</li>
</ul>
<p>
Plus sound software development practices when it comes to managing big features and evolutions, such as <i>Zero Downtime Deployments</i> techniques.
</p>
<a name="sec35"></a>
<h3>3.5 Zero Downtime Deployments</h3>
<div class="centering">
<div class="centered">
<span style="font-size: large;"><b><i>"Zero Downtime Deployment (ZDD) consists in deploying a new version of a system without any interruption of service."</i></b></span>
</div>
</div>
<br>
<p>
ZDD consists in deploying an application in such a way that one introduces a new version of an application to production without making the user see that the application went down in the meantime. From the user's and the company's point of view it's the best possible scenario of deployment since new features can be introduced and bugs can be eliminated without any outage.
</p>
<p>
I'll mention 4 techniques:
</p>
<ol>
<li>Feature Flipping</li>
<li>Dark launch</li>
<li>Blue/Green Deployments</li>
<li>Canari release</li>
</ol>
<p>
<b>Feature flipping</b>
</p>
<p>
Feature flipping allows to enable / disable features while the software is running. It's really straightforward to understand and put in place: simply use a configuration properly to entirely disable a feature from production and only activate it when its completely polished and working well.
</p>
<p>
For instance to disable or activate a feature globally for a whole application:
</p>
<pre>
<span style="color: blue;"><b>if</b></span> Feature.isEnabled('new_awesome_feature')
<span style="color: green;"> # Do something new, cool and awesome</span>
<span style="color: blue;"><b>else</b></span>
<span style="color: green;"> # Do old, same as always stuff</span>
<span style="color: blue;"><b>end</b></span>
</pre>
<p>
Or if one wants to do it on a per-user basis:
</p>
<pre>
<span style="color: blue;"><b>if</b></span> Feature.isEnabled('new_awesome_feature', current_user)
<span style="color: green;"> # Do something new, cool and awesome</span>
<span style="color: blue;"><b>else</b></span>
<span style="color: green;"> # Do old, same as always stuff</span>
<span style="color: blue;"><b>end</b></span>
</pre>
<p>
<b>Dark Launch</b>
</p>
<p>
The idea of <i>Dark Launch</i> is to use production to simulate load!
</p>
<p>
It's difficult to simulate load of a software used by hundreds of millions of people in a testing environment.
<br>
Without realistic load tests, it's impossible to know if infrastructure will stand up to the pressure.
</p>
<p>
Instead of simulating load, why not just deploy the feature to see what happens without disrupting usability?
<br>
Facebook calls this a <i>dark launch</i> of the feature.
</p>
<p>
Let's say you want to turn a static search field used by 500 million people into an autocomplete field so your users don't have to wait as long for the search results. You built a web service for it and want to simulate all those people typing words at once and generating multiple requests to the web service.
<br>
The dark launch strategy is where you would augment the existing form with a hidden background process that sends the entered search keyword to the new autocomplete service multiple times.
<br>
If the web service explodes unexpectedly then no harm is done; the server errors would just be ignored on the web page. But if it does explode then, great, you can tune and refine the service until it holds up.
</p>
<p>
There you have it, a real world load test.
</p>
<p>
<b>Blue/Green Deployments</b>
</p>
<p>
<i>Blue/Green Deployments</i> consists in building a second complete line of production for version N + 1. Both development and operation teams can peacefully build up version N + 1 on this second production line.
<br>
Whenever the version N + 1 is ready to be used, the configuration is changed on the load balancer and users are automatically and transparently redirected to the new version N + 1.
<br>
At this moment, the production line for version N is recovered and used to peacefully build version N + 2.
<br>
And so on.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/1d54c978-4e81-4746-9c3b-9f44459cb98f">
<img class="centered" style="width: 550px; " alt="Blue/Green Deployments" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/1d54c978-4e81-4746-9c3b-9f44459cb98f" />
</a>
<br>
<div class="centered">
(Source : Les Patterns des Géants du Web – Zero Downtime Deployment - <a href="http://blog.octo.com/zero-downtime-deployment/">http://blog.octo.com/zero-downtime-deployment/</a>)
</div>
</div>
<br>
<p>
This is quite effective and easy but the problem is that it requires to double the infrastructure, amount of servers, etc.
<br>
Imagine if Facebook had to maintain a complete second set of its hundreds of thousands of servers.
</p>
<p>
So there is some room for something better.
</p>
<p>
<b>Canari release</b>
</p>
<p>
<i>Canari release</i> is very similar in nature to <i>Blue/Green Deployments</i> but it addresses the problem to have multiple complete production lines.
<br>
The idea is to switch users to the new version in an incremental fashion : as more servers are migrated from the version N line to the version N + 1 line, an equivalent proportion of users are migrated as well.
<br>
This way, the load on every production line matches the amount of servers.
</p>
<p>
At first, only a few servers are migrated to version N + 1 along with a small subset of the users. This also allows to test the new release without risking an impact on all users.
<br>
When all servers have eventually been migrated from line N to line N + 1, the release is finished and everything can start all over again for release N + 2.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/c858f9cd-0e9a-4f9a-bbdf-6eed30713033">
<img class="centered" style="width: 550px; " alt="Canari Deployments" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/c858f9cd-0e9a-4f9a-bbdf-6eed30713033" />
</a>
<br>
<div class="centered">
(Source : Les Patterns des Géants du Web – Zero Downtime Deployment - <a href="http://blog.octo.com/zero-downtime-deployment/">http://blog.octo.com/zero-downtime-deployment/</a>)
</div>
</div>
<br>
<a name="sec4"></a>
<h2>4. Collaboration</h2>
<p>
Agile software development has broken down some of the silos between requirements analysis, testing and development. Deployment, operations and maintenance are other activities which have suffered a similar separation from the rest of the software development process. The DevOps movement is aimed at removing these silos and encouraging collaboration between development and operations.
<br>
Even with the best tools, DevOps is just another buzzword if you don't have the right culture.
</p>
<p>
The primary characteristic of DevOps culture is increased collaboration between the roles of development and operations. There are some important cultural shifts, within teams and at an organizational level, that support this collaboration.
</p>
<p>
This addresses a very important problem that is best illustrated with the following meme:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/83cafdba-bf4a-4476-8d4a-8fd3489351c7">
<img class="centered" style="width: 500px; " alt="Worked in Dev / Ops problem now" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/83cafdba-bf4a-4476-8d4a-8fd3489351c7" />
</a>
<br>
<div class="centered">
(Souce : DevOps Memes @ EMCworld 2015 - <a href="http://fr.slideshare.net/bgracely/devops-memes-emcworld-2015">http://fr.slideshare.net/bgracely/devops-memes-emcworld-2015</a>)
</div>
</div>
<br>
<p>
Team play is so important to DevOps that one could really sum up most of the methodology's goals for improvement with two C's: collaboration and communication. While it takes more than that to truly become a DevOps workplace, any company that has committed to those two concepts is well on its way.
</p>
<p>
But why is it so difficult ?
</p>
<a name="sec41"></a>
<h3>4.1 The wall of confusion</h3>
<p>
Because of the wall of confusion :
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/472d552f-71ea-4640-a815-026e18cd865e">
<img class="centered" style="width: 450px; " alt="Wall of Confusion" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/472d552f-71ea-4640-a815-026e18cd865e" />
</a>
</div>
<br>
<p>
In a traditional development cycle, the development team kicks things off by "throwing" a software release "over the wall" to Operations.
<br>
Operations picks up the release artifacts and begins preparing for their deployment. Operations manually hacks the deployment scripts provided by the developers or, most of the time, maintains their own scripts.
<br>
They also manually edit configuration files to reflect the production environment, which is significantly different than the Development or QA environments.
<br>
At best they are duplicating work that was already done in previous environments, at worst they are about to introduce or uncover new bugs.
</p>
<p>
The IT Operations team then embarks on what they understand to be the currently correct deployment process, which at this point is essentially being performed for the first time due to the script, configuration, process, and environment differences between Development and Operations.
<br>
Of course, somewhere along the way a problem occurs and the developers are called in to help troubleshoot. Operations claims that Development gave them faulty code. Developers respond by pointing out that it worked just fine in their environments, so it must be the case that Operations did something wrong.
<br>
Developers are having a difficult time even diagnosing the problem because the configuration, file locations, and procedure used to get into this state is different then what they expect. Time is running out on the change window and, of course, there isn't a reliable way to roll the environment back to a previously known good state.
</p>
<p>
So what should have been an eventless deployment ended up being an all-hands-on-deck fire drill where a lot of trial and error finally hacked the production environment into a usable state.
<br>
It <b>always</b> happens this way, always.
</p>
<p>
<b>Here comes DevOps</b>
</p>
<p>
DevOps helps to enable IT alignment by aligning development and operations roles and processes in the context of shared business objectives. Both development and operations need to understand that they are part of a unified business process. DevOps thinking ensures that individual decisions and actions strive to support and improve that unified business process, regardless of organizational structure.
</p>
<p>
Even further, as Werner Vogel, CTO of Amazon, said in 2014 :
</p>
<div class="centering">
<div class="centered">
<span style="font-size: large;"><b><i>"You build it, you run it."</i></b></span>
</div>
</div>
<br>
<a name="sec42"></a>
<h3>4.2 Software Development Process</h3>
<p>
Below is a simplified view of how the Agile Software Development Process usually looks like. <br>
Initially the business representatives work with the Product Owner and the Architecture Team to define the software, either through Story Mapping with User stories or with more complete specification.
<br>
Then the development team develops the software in short development sprints, shipping a production ready version of the software to the business users at the end of every sprint in order to capture feedback and get directions as often and as much as possible.<br>
Finally, after every new milestone, the software is deployed for wide usage to all business lines.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/36d0151b-4431-48e6-b10b-27a6ee1bc0b3">
<img class="centered" style="width: 750px; " alt="Software Development Process" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/36d0151b-4431-48e6-b10b-27a6ee1bc0b3" />
</a>
</div>
<br>
<p>
The big change introduced by DevOps is the understanding that <b>operators are the other users of the software !</b> and as such they should be fully integrated in the Software Development Process.
<br>
At specification time, operators should give their non-functional requirements just as business users give their functional requirement. Such non-functional requirements should be handled with same important and priority by the development team.
<br>
At implementation time, operators should provide feedback and non-functional tests specifications continuously just as business users provides feedback on functional features.
<br>
Finally, operators become users of the software just as business users.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/b92539db-d476-45e4-be04-40ed46fc87dd">
<img class="centered" style="width: 750px;" alt="Software Development Process - with Ops" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/b92539db-d476-45e4-be04-40ed46fc87dd" />
</a>
</div>
<br>
<p>
With DevOps, operators become fully integrated in the Software Development Process.
</p>
<a name="sec43"></a>
<h3>4.3 Share the Tools</h3>
<p>
In traditional corporations, teams of operators and teams of developers use specific, dedicated and well separated set of tools.
<br>
Operators usually don't want do know anything about the dev team SCM system as well as continuous integration environment. They perceive this as additional work and fear to be overwhelmed by developer requests if they put their hands on this systems as well. After all, they have well enough to do by taking care of production systems.
<br>
Developers, on their side, usually have no access to production system logs and monitoring tools, sometimes due to lack of will on their side, sometimes for regulation or security concerns.
</p>
<p>
This needs to change! DevOps is here for that.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/a0429129-eba8-4a06-8ff4-bc8412dce4f8">
<img class="centered" style="width: 550px;" alt="Share the tools" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/a0429129-eba8-4a06-8ff4-bc8412dce4f8" />
</a><br>
<div class="centered">
(Source : Mathieu Despriee - OCTO Technology - <a href="http://www.slideshare.net/OCTOTechnology/introduction-to-devops-28779951">Introduction to DevOps</a>)
</div>
</div>
<br>
<p>
One should note that this can be difficult to achieve. For instance for regulation or security reasons, logs may need to be anonymized on the fly, supervision tools need to be secured to avoid an untrained and forbidden developer to actually change something in production, etc. This may take time and cost resources. But the gain in efficiency is way greater that the required investment, and the ROI of this approach for the whole company is striking.
</p>
<a name="sec44"></a>
<h3>4.4 Work Together</h3>
<p>
A fundamental philosophy of DevOps is that developers and operations staff must work closely together on a regular basis.
<br>
An implication is that they must see one other as important stakeholders and actively seek to work together.
</p>
<p>
Inspired from the XP practice "<i>onsite customer</i>", which motivates agile developers to work closely with the business, disciplined agilists take this one step further with the practice of active stakeholder participation, which says that developers should work closely with all of their stakeholders, <b>including operations and support staff</b>.
<br>
This is a two-way street: operations and support staff must also be willing to work closely with developers.
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/ca919943-79b6-49b2-84fe-e76181de9e82">
<img class="centered" style="width: 600px;" alt="Align Development and Operation Teams" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/ca919943-79b6-49b2-84fe-e76181de9e82" />
</a>
</div>
<br>
<p>
In addition, other collaboration leads:
</p>
<ul>
<li>Have operators taking part in Agile rituals (Daily scrum, sprint planning, sprint retro, etc.)</li>
<li>Have devs taking part in production rollouts</li>
<li>Share between Dev and Ops objectives of continuous improvement</li>
</ul>
<a name="sec5"></a>
<h2>5. Conclusion</h2>
<p>
DevOps is a revolution that aims at addressing the <i>wall of confusion</i> between development teams and operation teams in big corporations having large IT departments where these roles are traditionally well separated and isolated.
</p>
<p>
Again, I've spent two thirds of my fifteen years career working for such big institutions, mostly financial institutions, and I have been able to witness this wall of confusion on a daily basis. Some sample things I got to hear:
</p>
<ul>
<li>"<i>It worked fine on my Tomcat. Sorry but I know nothing about your Websphere thing. I really can't help you.</i>" (a dev)</li>
<li>"<i>No we cannot provide you with an extract of this table from the production database. It contains confidential customer-related data.</i>" (an ops)</li>
</ul>
<p>
And many more examples such as those every day .... every day!
</p>
<p>
Happily DevOps is several years old and increasingly even these very traditional corporations are moving in the right direction by adopting DevOps principles and practices. But a lot remains to be done.
</p>
<p>
Now what about smaller corporations that don't necessarily have split functions between developers and operators?
<br>
Adopting DevOps principles and practices, such as deployment automation, continuous delivery and feature flipping still brings a lot.
</p>
<p>
I would summarize DevOps principles this way:
</p>
<div class="centering">
<a href="https://www.niceideas.ch/roller2/badtrash/mediaresource/00e6d23e-d1e7-4dfb-9abd-52d8f4c5673f">
<img class="centered" style="width: 420px; " alt="DevOps Wrap Up" src="https://www.niceideas.ch/roller2/badtrash/mediaresource/00e6d23e-d1e7-4dfb-9abd-52d8f4c5673f" />
</a>
</div>
<br>
<p>
DevOps is simply a step further towards Scaling Agility!
</p>
<p>
(This article is available as a PDF document here <a href="https://www.niceideas.ch/devops.pdf">https://www.niceideas.ch/devops.pdf</a> and as a slideshare presentation here
<a href="https://www.slideshare.net/JrmeKehrli/devops-explained-72664261">https://www.slideshare.net/JrmeKehrli/devops-explained-72664261</a>)
</p>