This post is copied over (and slightly modified) from an old blog post I wrote in 2018 on wordpress.

Early origins

What we intuitively understand as concurrency became quite a complicated concept with the advent of computers. The first generation of machines were invented, and worked as ordained by the theories of computation laid down by Kurt Gödel, Alan Turing, Alonzo Church, and John von Neumann. The first generation vacuum tube behemoths ran their instructions sequentially and churned out results to humans, who for the most part, were relieved that there was a means of automating calculations and processes that they would have otherwise done manually. The second and third generation machines, with their transistors and integrated circuit innards, created large waves of change in the computing world (by and large enabling microcomputers to be inexpensive and accessible by many), but the basic software instructions and processing model remained sequential – feeding a single processor – similar to how a first generation machine would have run its instructions (to be fair, attempts at building super computers using parallelization were undertaken during this time. One such early example was the ILLIAC IV).

Mathematical models of computation

In order to lay down some foundational understanding, and to better appreciate and understand the notion of computation, we divert our attention to a quick overview of abstract models of computations known as Turing machines, Push Down Automata, and Finite State Machines. Abstract machines are the foundations of modern computers. Mathematical models of computation laid the foundations for computers and programming languages, and continue to explore the possibilities and limitations of what computers are capable of.

A finite state automata (or finite state machine – FSM) is an abstract and mathematical model of computation. Very simply put, it models an abstract machine that receives input in the form of a sequence of symbols, and can be only in one state at any given time. Changing from one state to another is known as a transition, and occurs based on the input symbol read from the sequence. From a global perspective of a FSM, the abstract machine only knows the current state it is in, and there is no concept of memory that can hold a list of previous states. Shown below is a rather contrived example of a FSM with three states – A, B, and C.

State A, where the initial arrow is pointing into, is known as the starting or initial state. State C is known as the accepting state, denoted by the double circle. The arrows between the states as well as self referencing from the state, are known as transitions. The transitions are labelled with symbols, denoting the input symbol value that will allow the transition. In the above case there are three states and a set of two input symbols {0, 1}, therefore the FSM is called a three state two symbol FSM.

Given this information, it is very easy to see what will happen if an input sequence such as 1011 is fed into the above FSM. The very initial state will be the starting state A, and from here, we read the first symbol of the input sequence 1011, which is 1. This invokes the FSM to transition state to B, and now we are in state B. The next symbol we read in is 0, which means there will be a transition from B back to A. Likewise, the next symbol is 1, which means the FSM will transition to state B, and the final symbol is 1, whereby the state will transition from B to C.

The important point to note here is, there can be arbitrarily long and complicated sequences of symbols (not limited to binary digits) being read in by the FSM, and when the final symbol of the sequence is read, the FSM could be in state C (the accepting state) or not. If it is in the accepting state after the sequence is completed, the set of symbols is known as a ‘regular language’, and the FSM is said to accept it. vice versa, a language is said to be regular, if there is some FSM that accepts it. This enables us to define that a sequence of symbols such as 1001001001001101 is a regular language to the above FSM, but 100100100100110 is not.

Definition: A regular language is a sequence of symbols that is accepted by an FSM, i.e. if it can terminate in an accepting state.

Determinism and non-determinism

FSMs can be deterministic or non-deterministic. The FSM we looked at just now is an example of a deterministic FSM, where each state can have one and only one transition for an input. In a non-deterministic FSM, an input could change the current state of the FSM by having one transition, more than on transition, or no transition at all. This brings in the notion of non-determinism to the model of FSMs. Given below is an example of a non-deterministic FSM (NFSM).

If you consider the above finite state machine, you would notice, that when in state A, if the input symbol is 0, there will be two state transitions, one to A itself, and another to B – i.e. there are two possible next states. Also, the above NFSM accepts regular languages with a sequence that ends with 010. If the state is in C, and the input symbol is 1, no state transition occurs.

There is another type of NFSM which uses what is known as ‘epsilon transitions’. In a very simple manner of looking at it, an epsilon transition means that the state change can take place without any input symbol being consumed. This will be clearer with the example of a NFSM that uses epsilon transitions as shown below.

In the above NFSM, the transitions labeled with the greek letter epsilon (ε) are the epsilon transitions. State A is the starting state, as well as an accepting state. C and G are accepting states as well. Lets consider what happens when the input set (symbol sequence) consists only of the symbol 0. First, we will be in the starting state A. But as there are two epsilon transitions from A, we will go to the states pointed by these without reading in any input symbol. So we automatically go to states B and H. Now we read in the input symbol, which is 0. In this case only, state H transitions to G, and no state change takes place from B. But as the input sequence is consumed, and the NSFM terminates in an accepting state (G), we can conclude that 0 is a member of a regular language accepted by this NFSM. So is 1, 01, 10, 010, 101, 1010, etc (which can be checked against the above NFSM).

NFSMs are pretty handy when modelling reactive systems. Also, modelling a system using NFSMs is easier than using deterministic FSMs. It would seem that NFSMs are more powerful than DFSMs given the more extensive rule sets of parallel state transitions and epsilon transitions, but it can be proven that for any regular language recognized by a NFSM, there exists an equivalent DFSM that accepts that language (and vice versa). This means that there will always be an equivalent DFSM for any given NFSM. This concept allows us to design or model systems using NFSMs, and later convert it into an equivalent DFSM.

In abstract machines (such as the FSMs we saw above), the power of the machine is another way of saying it can recognize more (regular) languages. There are limitations of FSMs that are addressed by another type of abstract machine known as the pushdown automata (PDA). This can be thought of as a FSM plus a stack that can be used to store information or state in a LIFO (Last In First Out) fashion. This enables the PDA to store bit values in the stack, and come to an acceptance state only if the stack is empty once the input symbol sequence is processed. If you notice, this is a different acceptance criteria from the FSMs we saw earlier. The below image shows a snapshot of a PDA in operation.

If the power of abstract machines are determined by the languages they can recognize, then the mathematical model of computation known as Turing machines are more powerful than both FSMs and PDAs (This is another way of saying that there are sequences of input symbols that cannot be processed by FSMs or PDAs, but which are accepted by Turing machines).

Turing machines

Alan Turing published an article in 1936 titled ‘On Computable Numbers, with an Application to the Entscheidungsproblem‘, in which he first described Turing machines. A Turing machine can be thought of as a FSM with an infinite supply of a tape medium that it can read from and write to. Basically, the Turing machine reads a symbol from the tape which serves as the input symbol, and based on this input symbol, either writes a symbol to the tape, moves the tape left or right by one cell (a cell being one read/write unit of the tape which can contain a symbol), or transition to a new state. This seems a simple and basic way of operation, but Turing machines are powerful because given any algorithm, we can construct a Turing machine that can simulate that algorithm. Also, more interestingly, Turing machines have the innate ability to halt or stop, which gives rise to the halting problem.

You can find an online Turing machine simulator at this link if you would like to experiment with Turing machines (I would suggest going through the tutorials in the simulator, and then trying out with the ‘even number of zeros‘ example which is simple yet explains the fundamentals). Also, this video contains one of the best and shortest explanations of Turing machines I have seen yet.

In modelling computations and abstract machines, Turing machines are the most powerful abstraction we have today, and all out modern computers, program code, and algorithms, are based on it. The reason it is powerful is, anything that a Turing machine can simulate, can be constructed physically in the world today. But something that has no Turing equivalent system, cannot be constructed in reality (i.e. we have never come up with a way of doing computations or any mathematical model, that can do more than what a Turing machine can do – this is why for example, we do not have any system today which can solve the halting problem). In this sense, Turing machines are at the forefront of what is computationally feasible.

Most observations I have made in the past regarding engineering teams, have been of typical six to eight person squads working on applications with web + mobile front ends and multiple backend systems, utilizing a specific branching strategy with code review gates that determine what gets integrated and moves downstream.

I have generally viewed these gate keepers (i.e. the code reviewers) at a high level through five lenses: The programming expert, the domain expert, the clean coder, the hacker, and the historian/custodian. Anyone who reviews code can be viewed as someone with a unique overlap of these five distinct personas/traits. Note that these are not borrowed from anywhere, they are my personal classification of the many engineers I have observed and worked with in the past, and the manner in which they approach reviewing changes in a code base.

The programming expert

The in-depth subject matter expert in the specific programming language and/or tech stack. Knows the ins and outs of the language and its idiomatic usage + application. These engineers may have the potential for unintentionally splitting hair over technicalities, but they have a keen sense + understanding on when the technology is used or applied the wrong way.

The domain expert

Knows the business domain extremely well, and has been exposed to the majority changes in the business that has evolved the organization as well as the system, through time. This engineer is generally a long time team member who would have been present at the inception of the system. He or she will also probably be the person who will be able to explain the meaning + function behind almost every table and column in the database. They can look at a PR and immediately let you know how and why some lines of code are violating a subtle and little known business rule in the system.

The clean coder

The clean code and clean architecture evangelist + advocate. They will scan the code for readability, naming conventions, code + design smells, modules/functions to be refactored, and other deviations from clean code + architecture practices. These engineers provide feedback that helps keep code maintainable, readable, and manageable.

The hacker

The engineer who points out where your code changes are potential vulnerabilities for attack vectors. These engineers are always in threat-modelling mode, and most probably have prior history and experience working in systems that have needed heavy security hardening. These are the people who give key feedback that help close out security loopholes in your code.

The historian/custodian

In most cases a senior engineer with the organization for a long time, who would have been part of initially setting up the project and system. He or she can understand whether a code change is violating established conventions/patterns/practices used by the team, and/or deviates from the system design + architecture. They understand when a change (or an individual approach to implementing/improving something) is going to adversely affect the team and system (A great example borrowed from Kent Beck: In a restaurant, all the cooks know where the knives are. If someone changes the location of the knives, albeit to a better place, it will slow down the productivity of the cooks by impeding on their knowledge and usage of where the knives were by convention - there are better ways to manage this type of improvement).

So, what does this mean for someone who wants to be better equipped at conducting increasingly more effective code reviews? Below is a playbook that I have drawn up over time, and observed to be very effective within engineering teams when applied:

• I have never seen anyone (myself included) who is an expert in all the five personas described. Work towards being an engineer who is a better balanced overlap of all the five personas above. e.g. if you are an in-depth expert in your programming language, try to learn more about the business domain, clean code/architecture practices, OWASP top ten and secure coding, and probe into established team conventions/practices that are more culturally driven and nuanced. The more you immerse yourself in the business + team and pull information from them, the faster you will learn. Expecting to get better at all five traits while cruising by and waiting for information to be pushed to you, is not going to cut it. Communication and proactive involvement are the key game changers here.
• Collaborate with the authors of code changes, and walk through the code change over a call + screen share or shared workspace + monitor. If you are new to the business domain and codebase, this will help you ramp up while at the same time providing the stage to give your feedback + input, as well as clear out any doubts in real-time.
• Together with the team, decide and lay down expectations on how a code change will be structured and presented (e.g. detailed PR description with enough context, PR size should be <250 LOC, screenshots where required, etc.) that will enable reviewing a code change to be an easy, small, focused, and efficient exercise. Define exceptions to this structure, and how they will be managed.
• Automate everything that tends to be nitpicked: tab space convention, which line curly braces should start, formatting, cosmetic concerns, etc. There are many tools out there that can be used to automate a lot of these away.
• Compile a simple code review checklist that will help anyone new to the code base. This will be a common ground level check for verifying core team conventions and rules of thumb such as readable code, meaningful naming, no magic numbers, no commented code, if cyclomatic complexity in a function is >8 then extract and refactor into two functions, etc. This list should ideally have items that cannot be automated.
• Pair up on conducting code reviews - e.g. for a given pull request, pair an engineer weak in the business domain + strong in the programming language, with an engineer strong in the domain + is a historian/custodian of the dev conventions, to review the PR together. Maintain a list that contains pairs of engineers who complement each other’s strengths and can learn from each other, from which you can pick and assign to review code changes.
• Go one step further than pairing to review code - pair program, and/or mob with engineers during development. This (mostly) removes the need for a code review gate and gate keepers, which frees up quite a bit of time and effort. But approach this practice with caution: the caveat here is that you might need a well rounded set of engineers (who collectively display a strong overlap of all five traits/personas) to pair or mob with, in order to have the changes implemented and endorsed real-time through all five lenses described above. If not, you might need a checkpoint again after the change (else the persona representative missing during pairing/mobbing will point out the problems from their perspective, after the change is done and/or has gone into production).

In most teams that I have worked with, the above strategies have worked really well. Mobbing, not so much, as it has been challenging to get a well rounded group (making up a good balance of the five traits) of engineers together at the same time, especially in distributed teams across time zones. And even when we have done so, mobbing to build out features didn’t scale well when the expected feature requirements start growing and queuing up. From where I stand, mobbing is still an exploratory area. Alternatively, in the current teams I work with today, pair programming has worked extremely well and has reduced the cognitive load of changes to be reviewed.

So, what will work for you and your team? Experiment with different strategies from the above list to see how well they work for you, and let me know of any new ones that you have learned or practice in your teams. Above all (and more so if you are a senior engineer, team lead, or people manager), create the psychological safety net where engineers have respect + trust within the team to help each other grow. The more you educate your team and champion engineering excellence, the more they will cultivate new found appreciation + focus towards code reviews.

One of the many problems I’ve observed with how teams adopt SCRUM (or any iterative/incremental development model) is the manner in which a.) planning work, and b.) delivering value is conducted. Teams will generally showcase work status on boards with many swimlanes, a stream of stories moving downstream following the rules of a particular workflow. Planning work generally entails user stories coming in from the business (along with UI/UX mockups and acceptance criteria), and dev teams breaking them down into technical tasks (ideally after drawing up preliminary technical designs) before starting actual development.

When the time comes for a sprint/iteration demo, developers wonder why a lot of things they committed to could not be completed, why features and functionality did not work as expected during the demo, why there is so much rework after the demo (apart from changes cropping up due to user feedback during the demo), and why the heck they don’t have enough time to plan work for the next iteration.

Some key factors that give rise to these symptoms are managers expecting developers to estimate stories and provide a fixed ETA, ineffective planning methods, developers lacking minimal up-front technical designs, skipping due diligence activities that could mitigate technical risk, pressure from the top to rush solutions by taking shortcuts, and remnants of waterfall era mandates of ‘tracking everything you do under an issue/ticket’ (even if it’s a 5 minute fix - it will get dragged to become a two day hiatus with emails, ticket creation, and other wasted ceremonies that provide no value). What I would like to address in this post is how developers plan their work (i.e. break down work into granular tasks) when they get user stories from the business.

During my past tenure at :Different as an engineering team lead, we tweaked some changes in the way engineers plan and breakdown work (given a business roadmap), with profound results. Spearheaded by the engineering leadership of the organization and coupled with buy-in from the engineers, the changes in how work was planned and value delivered, created a new way of looking at delivery expectations, both from an engineering as well as business perspective. It also resulted in better engineering productivity, a reduced bug/failure rate, faster cycle times, users being able to see and test incremental value funneled through the delivery pipeline at a regular cadence, and time-boxed iterative planning activities that did not exhaust the engineering team.

These learnings and takeaways are something that I actively advocate, apply, and continuously improve, within the teams that I work with today. For the past few months at Corzent, the team I work with conducted some interesting experiments, and radically optimized the way the engineering team planned and broke down business expectations, and delivered value to end users.

In the beginning, business requirements/stories were broken down into tasks (in a Kanban workflow), that would be flagged as ‘UI’ tasks, ‘API’ tasks, or ‘Integration’ tasks, and so on. Almost all assignable work items would be horizontal slices of functionality (labelled under the horizontal hierarchy it falls into - UI/API/DB) or an integration/configuration/miscellaneous technical task. The first step we took was to do away with the horizontal slicing of stories into UI/API/Integration/DB tasks, and start breaking down user stories into thin vertical cross sections of end-to-end functionality.

To show how this works, here’s an example of a (highly oversimplified) requirement from the business: “As a user, I want to fill in the application details and save them in the system”

The developers, when planning work, could potentially break this story down into the following three story slices:

1. As a user, when I click the save application button in main view, I should see a modal popup to enter the application details
2. As a user, when I enter the details into the applications detail modal, the entered values should be validated
3. As a user, when I enter valid application details in the modal and click on save, the application should be saved successfully in the system

The above three story slices are extracted from the original story. What should be observed and emphasized is that:

• Each story slice can be implemented by a developer and independently deployed
• The first two stories slices do not implement the save functionality. Clicking the save button might display a ‘work in progress’ or ‘under construction’ message/toast
• The scope of each story slice is added in the description/acceptance-criteria, so that anyone testing understands what needs to be verified when testing the specific story slice
• The cycle time of each story slice is between half to one day - anything larger is further sliced
• Story slices are not technical sub-tasks of a user story - they do not mention any technical work nor give any indication of what part of the system will be touched (i.e. UI/API/DB). The story slice showcases only the value statement to the end user
• Implementing a story slice could result in one or more pull requests, which are linked to the same story slice. The PRs could be for a front end UI client, back end API service, configurations, integration, DB, whatever - anything that needs to get the story slice up and running, and ready to be deployed
• If incremental work-in-progress mapped to story slices should not be visible in production until the feature set is complete, they can be implemented behind a feature flag, which can be toggled on for developer and test deployment environments

A developer would then take a story slice into the Kanban workflow, implement it, raise pull requests for review, and merge + deploy the changes, all within the span of roughly half a day to one day.

Observable outcomes:

• Faster cycle time - implementing small units of value imply small PRs, which in turn makes the review/merge/deploy/feedback loop run faster
• Improved developer productivity and agility - the depth of the problem domain + technical solution of a story slice is far smaller and digestible than a complex business expectation compressed within a single user story
• Users/quality-engineers see changes delivered sooner and at a regular cadence, giving them the ability to provide feedback faster, even if the feature is working partially - this reduces rework stemming from bulk changes
• The business gains predictable insight into the ETA of features and feature-sets, based on monitoring the regular cadence of value being delivered and observed downstream

There were some caveats that we encountered and learned from, two of the more important being:

1. User stories (and hence story slices) can evolve with time. A user story is not a fixed business expectation set in stone, and therefore should be treated as such. A key point that the team discovered was to get into story slicing only after the business problem is clearly understood + clarified with users, any foreseeable system/architectural constraints related to the requirement are addressed, and space for the possibility of the story evolving with time is provided/accepted by engineers and the business.
2. We could not completely avoid stories that are only technical in nature, that are more often than not, sourced by engineering. Whether it’s refactoring, small configurations, or automations, there will be stories that are produced due to engineering necessity not directly related to business user stories/slices. In these cases, we generally prefix the story with [Dev] to denote that it is an isolated technical story - E.g. “[Dev] Remove unused function foo() of class bar, in order service business layer ” (This specific example is again oversimplified and may not need a story/task, but for the sake of demonstrating how we use this…)

On another note, the team also brainstormed and came up with a brilliant way of handling technical tasks (that could potentially impact existing architecture and system design) translated from highly complex business requirements. These type of requirements were not trivial to map out to story slices that could be addressed on their own. The solution was to map out individual story slices (and related technical/dev story slices) as nodes in a directed graph, stemming from root work items/nodes from which all other dependant nodes branch out (e.g. ‘implement feature flag for complex feature X’ would be a root work node for everything else that follows to implement feature X).

The directed edges of the graph specified dependencies (e.g. node A -> node B = work B cannot be started until work A is completed… There were some subtle cases where work B could be started even with work A partially completed, for which we created customized notation to represent this type of partial dependency). The entire graph was also overlaid with a lot of meta-data representing contexts of impacted systems, sub-system boundaries, interactions between components, and external actors/systems/services. The team would grow and evolve this map in cadence with iterative learning and discovery. This solution helped us avoid overlap/duplication between different pieces of work, as well as to discover missing stories/slices that were edge cases and/or hidden in the initial requirements (which were problems we initially had when we tried to do story slicing of highly complex requirements without having a visual aid/map/model, and tried to keep all the complexity in our heads). Now, we confidently move fast across the territory of complex changes required by the business.

Thinking on how we are evolving our planning and work breakdown in my team, I am reminded of some key highlights from Alistair Cockburn’s talk - Getting to the Heart of Agile (I highly recommend watching this talk), where he mentions that the world is always in a constant state of change, and to be agile, entails frequently probing the world and getting a feel for what is changing. The entire span of the value stream of a deliverable or a final end product is injected with thousands of decisions by hundreds of people within an organization. The final outcome is the embodiment of these hundred thousand or so decisions. Statistically, a person can make a mistake in one out of five (or ten) decisions they make. This could be compounded by people waiting on other people’s decisions to make their decisions, in a complex web of decision dependencies. This implies that we are shipping roughly ten thousand mistakes or bad decisions to production, wrapped within the artifacts and services that we deliver.

The engineering practices we adopt and follow should be geared towards shrinking the number of ‘decisions in process’ at a given time, and to discover better ways of probing the real world to get faster insight and feedback. Breaking down stories into story slices was one way we gained this, because for us smaller means better - easy to digest, simple to implement, and fast to validate. This helps us reduce the foot print of mistakes we make in our planning and work breakdown, as well as across the entire end-to-end value stream.

When I was young, I grew up surrounded by old computing and electronics magazines, issues from the 60’s, 70’s, and 80’s, a majority of them being the popular publication creative computing. I used to read them voraciously, during a time when PCs (mostly running Windows 3.1) were just beginning to make an appearance in offices, schools, the occasional library, and to a somewhat rare extent, homes.

I considered myself an antiquarian when it came to these old computing magazines and publications, and enjoyed reading about how the very first home computers looked and operated, about the DEC PDP-11, the early Apple Macintosh, the IBM PC with all the Charlie Chaplin adverts, and how the landscape of personal and home computing was evolving through the past few decades.

Thinking back on all those old computer magazines and articles, what I’m really nostalgic about is my dads Toshiba MSX home computer, which he had bought during the 80’s. The Toshiba MSX home computer was first announced by Microsoft in 1983, and conceived by the then vice-president of Microsoft Japan, Kazuhiko Nishi. The MSX computer was quite famous for being an architecture that major Japanese game corporations wrote games for, before the Nintendo era (the first version of metal gear was written for the MSX architecture). This was the first ever computer which I learned to program on as a young kid, a curious machine (at the time) that was hooked to a CRT television display via RF cable, and programs that were stored on (and loaded from) audio cassette tapes.

The MSX HX-10 model which my dad owned was an amazingly simple machine by today’s standards, having no persistent internal storage or hard drive, instead relying on an external audio cassette recorder to load and store programs on cassette tape (there was a cartridge slot, but audio cassette tapes were widely used as they were a cost effective medium). The HX-10 model was built atop a Zilog-Z80 processor and came with 64Kb of RAM. User space was about half that, giving about 28Kb for the user with the rest used by the system, which was basically what you had to work with. The machine used a version of BASIC known as MSX BASIC which came pre-installed in the ROM, and was loaded into memory when the machine was switched on.

My dad was (and still is) an avid programming and electronics hobbyist, and taught me a lot about coding and computer internals using the MSX HX-10 machine we had at home. I recall him learning and programming certain sprite based game routines in Z80 assembly, as the graphics processing was quite slow when programmed directly in MSX BASIC. Back then, knowing some assembly was pretty much the norm, as it was the only way to write games with fast and smooth graphics on machines like the MSX HX-10. Running into errors or strange issues was an exercise in diligent and patient problem solving, there was no internet available to search for solutions, troubleshooting was through books and manuals.

Thinking back, I now realize that it was a time period where I was so used to the MSX machine, that it was a strange context switch to work on a PC running the Windows OS for the first time. One would assume that it would be a natural and welcome change to switch from the terminal based 80’s machine to a PC running a GUI based operating system, but trust me, I found it very hard to give up on the whirring tape recorder of the MSX machine and the home television as the video display unit.

Learning about computers and programming at a very young age has been influential in what I do today in my career as a developer and programmer. I am still drawn to vintage computers and archaic programming languages in a large part due to my early experience with the Toshiba MSX. That is a time period I always reminisce about, on how it inspired my love of computers and programming.

Image credits (in order of appearance):

1. Charlie Chaplin advert
2. Creative computing cover view
3. Microsoft Multiplan advert
4. Macintosh advert
5. Toshiba MSX unboxed
6. Toshiba MSX rear panel view
7. Toshiba MSX external audio cassette recorder
8. Toshiba MSX start screen
9. Z80 Assembly book
10. Toshiba MSX manual
11. The MSX games book
12. MSX utilities book