Hemant Kumar

Kubernetes certificate based mutual auth with different CAs

Thu, 29 Oct 2020 07:31:00 +0000

Configuring certificate based mutual authentication in Kubernetes using nginx ingress controller is explained pretty well in this post. However, the post assumes that the certificates used for validating the client and the server are issued by the same CA (Certificate Authority). How do you configure client certificate authentication in kubernetes when using client and server certificates issued by different CAs? The current nginx ingress controller docs do not make this absolutely clear either. I recently came across a scenario where we were using our own internal/private CA for issuing client certificates and a publicly trusted CA for server TLS. This post covers configuring kubernetes nginx ingress to use certificates issued by different CAs on the same host to perform mutual authentication.

What is mutual authentication?

Mutual authentication or 2-way authentication is a process in which both the client and server verify each others identity via a Certificate Authority. An X.509 Certificate can provide identity to a machine or a device and enable the independent verification of the issued identity by an external authority such as a CA. Therefore, mutual authentication as defined by codeproject.com is also referred to as certificate based mutual authentication.

Mutual SSL authentication or certificate based mutual authentication refers to two parties authenticating each other through verifying the provided digital certificate so that both parties are assured of the others’ identity.

You can have the client and the server certificates issued by the same CA or as shown below by different CAs.

Configuring mutual authentication

In order to configure mutual authentication for a host in a kubernetes cluster, we are going to run a simple application within kubernetes and ensure it can be accessed publicly over TLS only with a valid client certificate.

There are many ways of setting up a kubernetes cluster but for this exercise we are going to use Azure Kubernetes Service (AKS) and deploy nginx ingress controller to it. We use an ingress controller for routing (layer 7) external traffic to your application running within the AKS cluster and exposing multiple services under the same IP address. The ingress controller deployment on AKS provisions a load balancer in Azure and assigns it a public IP. This allows the nginx controller to be accessed publicly via an EXTERNAL_IP. After deploying the nginx ingress controller, you can get its external IP by

kubectl get svc -n ingress-nginx

NAME                                 TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.0.106.200   13.93.79.119   80:31599/TCP,443:31682/TCP   46d
ingress-nginx-controller-admission   ClusterIP      10.0.101.201   <none>         443/TCP                      46d

We are going to use wildcard DNS service nip.io to give this EXTERNAL_IP a domain name like EXTERNAL_IP.nip.io and configure server TLS for this domain.

Generating the certificates

Usually you are not expected to be in possession of a public CA key and certificate. A Certificate Signing Request (CSR) is sent to a public CA to obtain a globally trusted certificate for securing your assets. However, for demonstration purposes below we will generate two separate CA certificates using OpenSSL and then generate server and client certificates signed by each CA to configure kubernetes ingress.

# Generate a public CA Key and Certificate
$ openssl req -x509 -sha256 -newkey rsa:4096 -days 356 -nodes \
	-keyout public-ca.key \
	-out public-ca.crt \
	-subj '/CN=Public Cert Authority/O=Org Public CA/C=GB'

# Generate the Server Key and Server Certificate and Sign with the public CA Certificate
$ openssl req -new -nodes -newkey rsa:4096 \
	-out server.csr \
	-keyout server.key \
	-subj '/CN={EXTERNAL_IP}.nip.io/O=aks-ingress/C=GB'
$ openssl x509 -req -sha256 -days 365 \
	-in server.csr \
	-CA public-ca.crt -CAkey public-ca.key \
	-set_serial 01 \
	-out server.crt

# Generate an internal CA Key and Certificate
$ openssl req -x509 -sha256 -newkey rsa:4096 -days 356 -nodes \
	-keyout internal-ca.key \
	-out internal-ca.crt \
	-subj '/CN=Internal Cert Authority/O=Org Internal CA/C=GB'

# Generate the Client Key and Client Certificate and Sign with the internal CA Certificate
$ openssl req -new -nodes -newkey rsa:4096 \
	-out client.csr \
	-keyout client.key \
	-subj '/CN=internal-client/O=aks-ingress-client/C=GB'
$ openssl x509 -req -sha256 -days 365 \
	-in client.csr \
	-CA internal-ca.crt -CAkey internal-ca.key \
	-set_serial 02 \
	-out client.crt

Create the kubernetes secrets

Kubermetes requires you to store the certificates as secrets in order for them to be used by nginx ingress controller. We create 2 separate secrets, one for the internal CA certificate to validate client certificates and the other for server TLS for the client to validate server’s identity.

# Add a secret for the internal CA certificate to validate client certs 
kubectl create secret generic internal-ca --from-file=ca.crt=internal-ca.crt

# Add a secret for server TLS (e.g. issued by a public CA) to validate server's identity
kubectl create secret tls server-tls --key server.key --cert server.crt

Deploy the application

Deploy the application pods.

echo "
apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-svc
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: http-svc
  template:
    metadata:
      labels:
        app: http-svc
    spec:
      containers:
      - name: http-svc
        image: gcr.io/kubernetes-e2e-test-images/echoserver:2.1
        ports:
        - containerPort: 8080" | kubectl apply -f -

Expose the pods within the cluster using a Service.

echo "
apiVersion: v1
kind: Service
metadata:
  name: http-svc
  namespace: default
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: http-svc" | kubectl apply -f -

Create the ingress rule

The previous step exposes the service within the kubernetes cluster. To access the service externally we need to create an ingress rule. The ingress rule below sets up TLS and makes the service avaialble on https://{EXTERNAL_IP}.nip.io.

default/internal-ca secret containing the Internal CA certificate is used for the client certificate validation
server-tls secret containing the server certificate is used for server TLS

Please note for the ingress rule to take effect it needs to be created in the same namespace as the service.

echo "
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-tls-verify-client: \"on\"
    nginx.ingress.kubernetes.io/auth-tls-secret: \"default/internal-ca\"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: \"1\"
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: \"true\"
  name: http-svc
  namespace: default
spec:
  rules:
  - host: {EXTERNAL_IP}.nip.io
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /
  tls:
  - hosts:
    - {EXTERNAL_IP}.nip.io
    secretName: server-tls" | kubectl apply -f -

Test the ingress configuration

Sending a request without a client certificate and key should give a 400 error, however the server certificate (issued by the Public CA) validation does succeed as shown below

curl -v -k https://{EXTERNAL_IP}.nip.io

* Server certificate:
*  subject: CN=13.93.79.119.nip.io; O=aks-ingress; C=GB
*  start date: Nov 10 23:19:03 2020 GMT
*  expire date: Nov 10 23:19:03 2021 GMT
*  issuer: CN=Public Cert Authority; O=Org Public CA; C=GB
...
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>
....

Sending a request with the client certificate and key should redirect you to the http-svc:

`curl -v -k https://{EXTERNAL_IP}.nip.io --cert client.crt --key client.key`
...
ssl-client-issuer-dn=C=GB,O=Org Internal CA,CN=Internal Cert Authority
ssl-client-subject-dn=C=GB,O=aks-ingress-client,CN=internal-client
ssl-client-verify=SUCCESS
user-agent=curl/7.58.0
....

Transformation, what the heck?

Tue, 03 Mar 2020 07:31:00 +0000

Throughout my consulting career, I’ve frequently encountered the term “transformation” being thrown around excessively in IT circles. Virtually every organization is undergoing some form of transformation, whether it’s related to agility, digitalisation, or cultural change. To gain a genuine understanding of what transformation entails, it’s crucial to delve deeper and clarify some of the commonly used terminology.

Transformation is a word used to signify intent to make a change, often triggered by an imminent business disaster that is going to affect bottom lines or by a change in the executive. It is generally driven from executives who have a sense of urgency e.g. being driven out of business due to disruption caused by new players in the market. Making things digital is what we did 30 years ago. We replaced tapes with CDs, letters with email and paper based calculations with spreadsheets. Digital transformation is not about making things digital but about modelling your business around technology. Considering technology not merely as a business support function or a commodity available for purchase from an IT vendor, but as a key differentiator to gain deeper insights into customer behaviors and steer your market strategy. This involves modernising your technology stack to unlock data from legacy systems (e.g., through APIs), externalising it to collaborate with partners, and creating novel business opportunities. It also entails providing services to customers anytime, anywhere, and on any device. Additionally, it may involve establishing a data-driven business by combining data from various sources, constructing analytics models for predicting and optimising outcomes, and subsequently transforming the business based on these models to enhance the return on investment from data.

Agile transformation

Driving business change through agile transformation is modelled around aligning people, processes and products.

people - cultivate a supporting environment that emphasizes learning from failures rather than blaming. Setting up teams to succeed by providing them time, space and resources to experiment and learn.
processes and tools - adopt processes and tools that help in continuous improvement and better collaboration by breaking silos within the business
governance - manage investment risk to
- understand where your IT spend is going. Are you tracking cost, time and resource allocation efficiency or tracking speed and value?
- improve delivery assurance by measuring the progress on your plan
customer - manage stakeholder expectations and improve product decision making by factoring in
- opportunity cost - explore multiple options to understand the potential missed opportunities foregone by choosing one option over another.
- cost of delay - ensure product decisions are made not just by understanding the value of something but also its urgency.
money - move from annual investment cycles to beyond budgeting, switch from cost accounting to throughput accounting
organisation - changing the organisation structure and culture to optimize for delivering value to your customers

Simply put agile at scale means breaking up large projects into small pieces, so you can release to the market faster, run experiments, get customer feedback and deliver something that the market wants rather than deliver what you think that they may want. This focus on delivering quickly in small increments reduces risk. An all or nothing approach is required if you are launching a rocket into space, not when you are delivering an improvement over your existing website.

Cultural transformation

Culture is often what people do organically, it is not what you say, it is about what you do. It is about specific behaviours that help in building effective teams e.g.

having a shared purpose
establish safety and openness - ability to express vulnerability and know each others background, goals, strengths and weaknesses
discourage blame - say no to talented jerks - scientific experiments have shown jerks diminish a team’s performance by 30-40%
keep the culture alive (culture capture) - seek and provide constructive feedback. What excites the team, what frustrates them, what is their biggest challenge?

Applying the agile principles in practice without worrying too much about the methodology is critical to being agile rather than doing agile. Too much emphasis on process, stops people from thinking what they are doing and whether they are doing it right. For example, the value of processes like daily standups is their ability to create a safe space for team-members to hold each other accountable against the shared objectives rather than perform a daily morning ninja routine of what I did yesterday and what I will be doing today.

Collaboration

If you want to go fast travel alone, if you want to go far travel together

Organisational structure is important in managing accountability and fixing responsibility within teams but obsession with tightly defined structures and roles ends up creating barriers amongst your teams. The goals across teams sometimes vary vastly and may even be contrary to each other. This can result in organisational silos e.g Development teams want to push new features and functionality quickly to customers where as Operations want the systems to be resilient and can be averse to rapid change. Balancing velocity against risk is a fine art and often safety trumps new thinking. This siloed thinking has a direct effect on the kind of systems built within your organisation.

Opening a dialogue between disparate teams is as much a cultural change as it is an operational one. Having small, self empowered (often collocated) cross functional teams with experts from across the business, drives collective responsibility and ownership of:

issues/problems that the business faces
results and successes
as well as failures

Autonomy and alignment

Everyone wants autonomy, everyone wants to have freedom to do what they like, but there are always boundaries to everyone’s autonomy. In a team, my autonomy might start to step on somebody else’s autonomy, and so we need to form an agreement, expectations.

Don’t mandate how a team should work. Don’t say, “You have to do stand ups. You have to do one weekly iterations. If you want your teams to be very agile and adaptive focus on the outcomes rather than output. We want to make sure that all code has automated tests, we want to make sure that you do frequent releases, and we expect that, but how you would like to do that is up to you and your team. We’re talking about the boundaries of autonomy.

Autonomy may also result in reactive solutions to problems. This is where alignment becomes important, because when you’re in a larger organization, and you’re moving from Start Up to Scale Up, everyone is normally aligned to working on the most urgent thing. In a very early-stage startup, that most urgent thing constantly changes, and as you get bigger and bigger, you need to create levels of alignment.

Embracing failure

Learning is the result of every experiment that you run. Failure is learning.

The impact of a failure isn’t necessarily bad, provided you learn from it without paying a heavy price. You can think of failure as a means to blame someone or an opportunity to expose underlying problems so that they can be fixed. If you can get small incremental changes out to customers frequently, you allow yourselves the ability to fail fast and learn cheaply.

The reason why we couldn’t do it - because of such and such thing e.g. The reason we couldn’t drive customer sales up because our assumptions about customer’s online usage patterns were wrong.

Measuring success

There are various aspects of measuring success - cost, modernisation that leads to improvement in quality and performance.

Being able to identify successes and failures is essential to gathering feedback when developing new processes. Many businesses find it very difficult to visualise information or value flow across the organisation (value stream mapping). Without this information, the only lever left with business managers is cost. The most important question then becomes: Where is my spend going? Whether that spend actually delivers customer value becomes an inessential concern.

Focus on cost utilization means that IT budgeting decisions are based on cost estimation of your portfolio. During the planning phase the budgeting exercise allocates fixed amount of money (in batches) to one or more programes consisting of smaller projects. With the budget allocated and the cost fixed we then expect teams to deliver all the agreed features that we think customers want. However, if customer feedback loops have been established via continuously and incremental delivery methods, they may tell you a very different story about what the customer’s actually want.

Obsession with cost utilization (which is essentially a measure of people busyness) leads to misplaced priorities and bad decisions. Time spent becomes more important than the result. Anything that doesn’t generate revenue becomes a cost centre, IT teams end up being treated as cost functions rather than centres of value addition.

Doug Hubbard in IT measurement inversion suggests that cost has a very limited effect on return on investment whereas the utilization of the system i.e. whether the system is actually going to rollout and whether anyone will use it at all, is the most important factor for ROI analysis.

One way of measuring value of features is using cost of delay - how much is it costing you in per unit time to not deliver a feature? Everything is measurable article explains how to measure intangibles or measure anything

Creating feedback loops

How do we know what we are building is the right thing faster? Whether the assumptions we are making are correct? How do we quickly test those assumptions?

A lean business can cheaply find out whether people are going to use new features in a system by running small experiments.

Tackling the most important problems that add value, not optimizing for the case where we think we are right (For established products, with a well proven business model, two-thirds of the time the features that we want to build have 0 or negative value. In new product development 90% of the time they have no value)
By creating feedback loops to validate assumptions
By delivering in small increments
Enabling experimental approach to product development using a scientific method

One of the ways of achieving this is using hypothesis driven delivery

We believe that [building this feature] [for these people] will achieve [this outcome] We will know we are successful when we see [this signal from the market]

Summary

The path to digital transformation involves dismantling organisational silos by fostering collaboration and establishing cross-functional teams with shared goals. These teams become conduits for rapid experiments and crucial feedback loops, measuring the value derived from these initiatives. Shifting away from traditional budgeting and throughput accounting to reorganize around value streams, flow, and a customer-centric approach is essential. This strategic realignment not only deepens the understanding of customer values but also opens opportunities to rethink outdated processes and create innovative customer experiences.

Six rules for tech leadership

Sun, 01 Mar 2020 07:31:00 +0000

Over the years I have lead some complex technical pieces of work with teams of varied sizes and location. This has involved optimizing working practices and processes of teams, advising management on building engineering capabilities, agile practices, technical governance and continuous delivery. As a technical leader, I have faced numerous challenges while trying to align team objectives, resolve conflicts, highlighting the importance of cross functional requirements to the business and managing technical risk. Learning from my mistakes I have devised a set of rules to help me navigate these challenges. While the six rules I share here are far from exhaustive, I hope they provide value to technology leaders or, at the very least, resonate with their experiences.

Know your goal

“If you don’t know where you want to go, then it doesn’t matter which path you take” - Lewis Carroll

What’s most important to you? This is a tough question to answer, because it requires you to think long and deep. Knowing yourself and knowing exactly what you want to achieve is a continuous process, therefore your answer may change over time. This is what I have found drives me:

Control and flexibility over the kind of work I do and when it is done
Have a positive influence and a wider impact on a number of people (colleagues or customers) based on what I do

Having a clear purpose and repeating it often gives you the strength and the drive to lead by example and take others with you on the journey, even in the face of resistance. It is only when you have a clear understanding of what you want that you are going to be able to articulate it to others. Enabling others can only begin once you know what you want to achieve. People are going to take you seriously only if you are clear in your thinking. Working a job however, requires a focus on immediate priorities, whether they be organisational or team-oriented goals. The key is discovering the optimal balance where your personal objectives align harmoniously with those of the business.

Know your team

Leadership is about enabling others to achieve their true potential by letting them be the best version of themselves. Getting to know your team and the wider business by listening and engaging with them is crucial to be able to connect, build relationships and understand other people’s viewpoints.

“If there is one secret of success, it lies in the ability to get the other person’s point of view and see things from that person’s angle as well as your own” - Henry Ford

Empower others by aligning their individual goals with the team or organizational goals. Team goals often vary vastly and may even be contrary to each other, resulting in silos e.g Development teams want to push new features and functionality quickly to customers where as Operations want the systems to be resilient and can be averse to rapid change. Collaboration, not only within teams but across teams is required to figure out how disparate goals can be met keeping the overall business goals in mind. Some strategies that have worked for me in enabling teams to be self empowered capable of building, running and supporting their apps are

Breaking organizational silos e.g. development & operations - changing the org set up by introducing cross functional teams
Coaching & mentoring dev teams to be focussed on production readiness and support, right from project kick off. Delivering great apps is fine but in order for them to be effective you need the appropriate level of operational excellence, security, reliability, performance efficiency, cost optimization and sustainability in place for them to be successful.
Adopting similar tool set across build, delivery & ops teams that support collaboration e.g. adopting approaches like GitOps
Pair programming within teams to prevent single point of failures.

Get the business on-board

Technical leadership is a lot about meeting, influencing, planning and shaping the technical direction than hands-on work. Your hands-on experience may not be up to date, however you learn a lot about new tools and practices by seeing what the team is doing. Execs and management don’t expect “architect” types to be hands-on. They do expect you to be able to talk authoritatively about technical topics at the high level. For example,

What has proven to be useful approach to cloud migrations - IAAS, PAAS, FAAS etc?
Impact of moving from capex to opex on budgeting (e.g. moving from owning a car to renting a car)
Why invest in adopting continuous integration and continuous delivery?
Sharing pitfalls of serverless
Explaining where container orchestration has gone wrong on projects

This is the kind of stuff that you pick up through having lots of conversations with different people. The key qualification is your recommendations should be aligned to the experiences and wisdom of devs who actually do stuff. The goal is that if the business believes what the “architect” says, it will help the team to deliver as opposed to creating pain for them.

Before making any technology choices, think critically and innovatively - seeing situations in new ways, being able to deal with uncertainty and ambiguity. It is critical to get decisions right early in a project. Ensure the business goals are clear and aligned with the technical direction you want to pursue. While the execs may find your technical proposal amusing, what they are really interested in is:

Cost
Timeline
Resources required
Customer impact

Visualise your tech strategy

Having a visual representation of your technology estate and capabilities allows you to

share technical vision and direction with the entire organisation to get alignment & ensure everybody is pulling in the same direction
identify gaps that could be filled with training
achieve standardisation and assess new technologies for innovation

Light weight documentation that captures key technology decisions and choices helps in providing context not only to your team but also for future teams, who may want to evolve your technology landscape. Architecture decision records is a useful technique that I have used to capture key architecture decisions on projects for visibility within the team and for external oversight.

Most of our time and energy is spent translating complex business rules into code, rather than thinking about the rules themselves. When you’re thinking in terms of a programming language, code constrains your ability to think, it can make you miss the forest for the trees. Therefore it is crucial to think before writing any code and to document your thinking. This could be your business architecture, system architecture or the associated data flows. Whilst documentation is important, it is also important for it to be easily consumable. Using a standard approach like C4 model across all your artifacts for visualising your architecture, goes a long way in communicating and thinking above the code.

Simplify not over engineer

Simplicity in software can be elusive because we often do not make the distinction between essential complexity and accidental complexity. Automation is a noble pursuit, however knowing what to automate and when, determines whether you are simplifying or over engineering. Before you go on an “automate everything” spree it is worth thinking about

What percentage of your software is really complicated? How much of it actually impacts the business? Does the complex part absolutely need to be automated? Can the business deal with it manually or accept it as a risk?
Are you designing a system that can never possibly fail? Trying to handle “every possible thing” that can go wrong in your system leads to accidental complexity. Perform a cost benefit analysis of your design choices and think if all of them are worth investing time and money.
Are you doing premature generalisation? When do you prioritise extensibility and generalisation in your software because each of them has simplicity and cost trade offs?
Do you adopt bleeding edge tech or something that works? Boring is good. Are you experimenting with new tech at the core of your system or at the periphery?
Are you optimising for the things that can be easily seen and measured (e.g. code repetition) while ignoring software complexity that can be hard to measure?
Are you a victim of tech overuse? Technology can solve a lot of problems but overusing it can cause more problems that it can solve. With a hammer everything looks like a nail. You can let humans deal with the edge cases, especially early on in a project.

Adopt tools and tech fit for purpose

“Technology can bring benefits if and only if it diminishes a limitation” - Eliyahu Goldratt

Before adopting any new technology, it is important to do an objective evaluation of:

What is the power of the technology?
What limitation does the technology diminish? How can you prove that the limitation is holding you back? How would you know it was diminishing? What could you measure?
What existing rules enable us to manage this limitation? Do we need to be wedded to those rules? Who owns the rules? Who might be threatened by dismantling them? How can we make it safe to change? How to create a graceful exit?
What new rules will we need? How can we safely exploit this new technology? How do we introduce and institutionalise these new rules across the business?

As a principle - chose the best tools and implementations available over standardising on any one language or platform and resentfully accepting its inherent limitation. Select a tool/programming language keeping business goals in mind to optimise for the right combination of

Cost of the service
- Runtime cost - current vs future, cost variance with scale and additional capabilities
- Cost of operation and support
Fit for purpose
- Minimal lock-in (choice)
- Ease and speed of development (agility)
- Tech capability of the business (support)
- Availability of existing libraries/integrations (flexibility)
Performance & security
Openness - can’t do it alone, learn from shared practices and best ideas in the open source community, no vendor lock-in, contribute & attract talent. e.g. in your area it maybe easier to hire engineers for deploying and securing linux based systems than say windows.

Securing REST APIs

Thu, 10 Aug 2017 19:31:00 +0000

RESTful services are stateless therefore each request needs to be authenticated individually. State in REST terminology means the state of the resource that the API manages, not session state. There maybe good reasons to build a stateful API but that is going against REST principles. It is important to realize that managing sessions is complex and difficult to do securely, as it is prone to replay and impersonation attacks. So what options do we have to secure RESTful services? This post looks into Basic Authentication, MAC (Message Authentication Code), Digital Signatures and OAuth.

Security basics

When looking at any security aspect, often a lot of terms get thrown around which can be overwhelming and also confusing. Therefore, before looking at the nitty-gritty of securing RESTful APIs it is worth getting some security jargon out of the way.

Authentication - Establish the sender’s identity so that receiver knows who they are talking to; e.g. client (user, device, or another service/API) sends credentials (either in plaintext or encrypted) to the server to identify itself.
Authorization - Verification what that the sender has access to. Happens post authentication to determine whether they have access to a certain resource.
Integrity - Ensuring message contents of a request haven’t changed in transit.
Non-repudiation - Ensuring that the sender cannot deny having sent the message; e.g. your bank cannot deny having sent you a bank statement if it has a valid stamp of the bank on it and this could be proved to a third party.
Confidentiality - No one can see the message contents in transit from sender to receiver.

In order to achieve secure communication, be it client to service or service to service, there are fundamentally two problems to solve:

Ensure that the message can only be read by the intended recipient.
Ensure that the message is from a known sender and it has not been modified in transit.

The first problem can be solved using encryption. Encryption is used to achieve confidentiality and means only those with a corresponding secret key can read the message. However encryption alone does not guarantee integrity. The second problem can be solved using cryptography which often uses combination of encryption and hashing to achieve authenticity and integrity in addition to confidentiality.

Symmetric (Private key cryptography)

You share the same secret key between sender and receiver to encrypt and decrypt the message. You can trust the authenticity (from a trusted known sender) of the message, its confidentiality and integrity but non-repudiation cannot be guaranteed. Because the secret key could be shared amongst several participants, there is no single identity attached to the key, therefore the receiver knows it came from a source in possession of the key but doesn’t know which one. The risk of the key falling in the wrong hands is also higher because it needs to be securely shared amongst the participants, often over the internet. Other options include face-to-face meeting or use of a trusted courier but these can often be impractical. Higher the number of participants, higher is the exposure of the key.

Asymmetric (Public key cryptography)

Different key is used between sender and receiver to encrypt and decrypt the message. This gets us beyond the shared key issue in symmetric key cryptography. In order to solve the first secure communication problem mentioned above, when encrypting, you use the receiver’s public key to write (encrypt) the message and the receiver uses their private key to read (decrypt) the message. This establishes confidentiality of the message.

In order to solve the 2nd secure communication problem mentioned above, you use a digital signature. Digitally signing data is equivalent to a physical signature that can only be produced by the signing authority and verified by anyone who has visibility of the signing authority’s signature. Signing uses public key encryption where the sender uses their private key to write message’s signature, and the receiver uses the sender’s public key to check if it’s really from the sender. It is a means of attaching identity to a key. It is discussed in further detail in the section below on message signing using digital signature.

Approaches to securing RESTful APIs

Having covered some security fundamentals, we can now look at the different techniques to secure RESTful APIs. The fundamentals discussed above form the basis of the techniques we are going to look at.

Basic Authentication

The most simple way to authenticate senders is to use HTTP basic authentication. Sender’s credentials (username and password) are base64-encoded and sent across the network unencrypted in an HTTP header.

GET / HTTP/1.1
Host: api.example.com
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=

There are a few issues with HTTP Basic Authentication:

Credentials are sent over the wire and even though they are encoded, they are not encrypted and can easily be converted to plaintext.
Credentials are sent repeatedly, for each request, which widens the attack window.
The password may be stored permanently in the browser, if the user requests. The browser caches the password minimum for the length of the window / process, which can be silently used to make requests to the server e.g. in CSRF attacks.

Using HTTPS can solve the first issue. Even then, the credentials are only protected until SSL/TLS termination. Any internal network routing, logging, etc. can still expose the plaintext credentials. In an enterprise, SSL/TLS termination often occurs much before the request reaches your API server. Does HTTPS protect the credentials in transit? Yes. Is that enough? Usually, No. Basic Authentication with HTTPS provides you confidentiality only for a window during which SSL/TLS is on.

MAC (Message Authentication Code)

Basic Auth over HTTP exposes credentials in transit and does not guarantee integrity of the message. MAC on the other hand is used to send hashed version of credentials and the message using a secret key. It can be used to authenticate a message and verify its integrity. MAC is symmetric, i.e. it uses the same key to produce a MAC value for a message and to verify the MAC value for the message.

For accessing a protected resource

/users/username/account

HMAC (Hash-based message authentication) an implementation of MAC involves calculation of an HMAC value

value = base64encode(hmac("sha256", "secret", "GET+/users/username/account"))

The HMAC value then is sent over as an HTTP header:

GET /users/username/account HTTP/1.1
Host: api.example.com
Authentication: hmac username:[value]

Prevent hash reuse

Hashing the same message repeatedly results in the same HMAC (hash). If the hash falls into the wrong hands, it can be used to make the same request at a later time. Therefore it is important to introduce entropy to the hash generation to prevent a replay attack. This is done by adding a timestamp and nonce to the hash computation. The nonce is a number we only use once and is regenerated on each subsequent request, even if the request is for the same resource.

value = base64encode(hmac("sha256", "secret", "GET+/users/username/account+28jul201712:59:24+123456"))

The additional data - timestamp and nonce is sent to the receiver for reconstructing the hash.

GET /users/username/account HTTP/1.1
Host: example.org
Authentication: hmac username:123456:[value]
Date: 28 jul 2017 12:59:24

The receiver reconstructs the hash with the nonce value and if it doesn’t match the received hash value, discards the message possibly because the hash has been previously used. This ensures each request is only valid once and only once.

If the timestamp is not within a certain range (say 10 minutes) of the receiver’s time, then the receiver can discard the message as it is probably a replay of an earlier message. It is worth noting time-limited authentications can be problematic if the sender and receiver’s time is not synchronized.

Message signing using Digital Signature

Digital signatures use asymmetric public key cryptography to establish authenticity (message sent by a known sender), integrity (message wasn’t tampered with) and non-repudiation (message sent by the sender cannot be denied).

When signing, the sender uses their private key (also called a secret key, abbreviated to sk) to write message’s signature, and the receiver uses the sender’s public key to check if it’s really from the sender. Again, to ensure a message cannot be duplicated and reused in a replay attack a unique identifier for the message like timestamp and nonce is also used while generating the signature.

$Sign(Message, sk) = Signature$ $Verify(Message, Signature, pk)$

When you have verified that the signature against a given message is valid, you can be extremely confident that the only way some one could have produced it is if they knew the private key, associated with the public key that you used to verify the message.

A service when acting as a receiver has a list of public keys for all other services that want to send messages to it. The same service, when acting as the sender can provides its public key to other services that it wants to send messages to.

OAuth 2

OAuth 2 is an open protocol to allow secure authorization in a standard method from web, mobile and desktop applications. It is for delegation of access e.g. you hire a business assistant and delegate her to withdraw money from the business account to fulfill business requests on your behalf. You (the user) have delegated the authority to your assistant (client), however the authorization policy (identity and access control checks performed) to allow the assistant to withdraw money are still enforced by your bank account (resource API), not you.

In techniques mentioned above often it is your resource API that is responsible for establishing the identity of clients and defining access controls. Oauth enables federated security to allow clear separation between your applications and the associated authentication and authorization mechanism. Other identity protocols like SAML and WS-Fed also provide federated security but they are older and relatively more complex than OAuth. Figure below depicts an Oauth2 protocol flow

The separation between application (client) and authorization server means you can either

build out the authorization server as a standalone component which is responsible for obtaining authorization from users and issuing tokens to clients
outsource the authorization server as a service that the user trusts, such as a social identity provider like google or facebook

This allows you to focus on building and scaling your APIs (resource server) independent of authorization.

OAuth 2 has multiple flows called grant types for obtaining an access token. Deciding which grants to implement depends on the type of client applications you support and the experience you want for your users. In essence each flow involves obtaining authorization to get an access token and using the access token to access protected resources. An access token is a JSON Web Token (JWT) encoded in base64URL format that contains a header, payload, and signature. A resource server (API) can validate the access token and can authorize the client (application) to access particular resources based on the scopes and claims in the access token.

OpenId Connect Provider

OAuth is for authorization but applications require to know the users identity too. OpenID Connect adds authentication to OAuth. In OAuth the client applications do not get any information about the user, it is only the resource APIs that get access to the identity data via the access tokens. OpenID Connect defines a standard way of providing this identity data to the client applications by giving them an ID token. Very often an OpenID Connect provider also acts as an OAuth server - which means you can request ID tokens as well as access tokens. This allows client applications to get information about who the user is and when they last authenticated and decide whether to re-authenticate or reject the user. E.g. for high value transactions the client may decide to re-authenticate the user even if they are already logged in.

An OpenId Connect Provider is a REST-like identity layer on top of OAuth 2 that allows clients to verify the identity of the end-user, as well as to obtain basic profile information about the end-user. It provides /.well-known/openid-configuration service discovery endpoint for clients to get information about interacting with the OpenId Connect/OAuth server.

In summary

Before deciding on an API security approach, it is important to understand what are you going to secure and what is the sensitivity of the data being managed? APIs handling things like personal data, medical health records or financial data will need a different security approach than an API handling, say traffic updates. It is also worth defining the scope of your API security. Securing network and server infrastructure for things like intrusion, eves dropping via packet sniffing and physical security often lie outside the scope of API security. Opting for a particular approach may depend on specific security requirements of your application because each technique provides protection on different security aspects. For example basic authentication without HTTPS can provide authenticity but no integrity or confidentiality. MACs may be sufficient for internal APIs (non public facing) serving a few web applications. For highly sensitive data, digital signatures maybe a necessity, but if you are looking for flexibility and performance at scale OAuth 2 Bearer tokens maybe the way to go.

Making sense of Blockchain

Thu, 13 Apr 2017 19:31:00 +0000

Trust is fundamental to commerce. Any business transaction is based upon trust and requires secure way of transferring assets between transacting parties. Banks provide this trust by maintaining a true record of financial transactions. Government agencies provide evidence of land titles, vehicle registration records, health and education records etc by maintaining a transaction log. They provide trust by maintaining a central ledger for recording transactions that can be relied upon to verify each transaction. The onus of maintaining the transactions accurately and securely on the central ledger also lies with the authority owning it. This grants significant responsibility and control to the central authority or intermediary facilitating commerce between transacting parties. The intermediary essentially establishes the rules of commerce that every transacting party must adhere to. While the intermediary often operates effectively, it can occasionally become a single point of failure, as seen in the global financial crash of 2008 where banks were at the epicenter of the economic turmoil.

Can a centralised authority with utmost power be trusted to run and meddle with entire economic systems?
Could the monetary supply and monetary policy be set by a computer where it could not be corrupted by humans, thereby preventing government overreach?
Is the transaction ledger owned by the central authority tamper evident and prevent any illegitimate records being added or updated? Is it independently and transparently verifiable in case of a dispute?
Are the intermediaries efficient in fulfilling monetary transactions? Communication over the internet takes place at a mind boggling rate but systems with layers of middlemen can take days to clear and reconcile.

What is blockchain?

Shortly after the 2008 global financial crisis, a white paper by an unknown entity Satoshi Nakamoto emerged. The paper introduced a new peer to peer financial system where payments are based on cryptographic proof instead of trust, allowing any two willing parties to transact directly with each other without the need for a trusted third party. The system would use a new digital crypto currency called Bitcoin. The technology invented to power this new system was called blockchain. Simply put, a blockchain is a continuously updated decentralized record of who holds what.

Decentralized trust and control

Blockchain transfers control and decision-making from a centralized entity (individual, organization, or group thereof) to a distributed network. Decentralized networks strive to reduce the level of trust that participants must place in one another, and deter their ability to exert authority or control over one another in ways that degrade the functionality of the network. Blockchain attempts to reduce the cost and increase trust in business transactions by using an immutable distributed transaction ledger on peer to peer networks rather than a central ledger. Rather than a single authority like a bank responsible for maintaining transactions, it is now a group of people running blockchain software that do this. They ensure that the information stored in the distributed ledger is immutable and verifiable by applying techniques like cryptography and hashing. The list of transactions, also known as a distributed ledger is available for everyone to see and verify. The distributed nature of the blockchain makes it tamper evident and unhackable because if a block of transactions is messed with, everyone gets to know about it and as long as the bad actors are outnumbered by the good ones it is rejected by the system. Trust is inherently built into the system.

Underlying technology

Asymmetric cryptography using digital signatures A way to verify that a message was sent by the known sender, that the only way some one could have produced it is if they knew the private key, associated with the public key that you used to verify the message.
Peer to peer network A ledger containing the record of all the messages is copied on a group of computers rather than relying on a single authority to maintain the records on a central computer. This decentralization removes the need to trust a single authority but with as many copies of the ledger as the number of computers in the network, which version of the ledger is to be trusted? This is the problem addressed in the original Bitcoin paper. The solution offered is to trust whichever ledger has the most computational work put in to it.
Cryptographic hashing A way to generate a small, unique “fingerprint” for any data allowing quick comparison of large data sets and a secure way to verify data has not been altered. Some computational work must be carried out to generate the fingerprint or hash in the desired format and update the decentralized ledger. This is known as Proof or Work.

How do we ensure the rules are being followed?

Group of people who maintain the distributed ledger are called bookkeepers. Transactions often reach different bookkeepers in different order, depending upon which bookkeeper is online. Bookkeepers need to agree on the order of transactions and rules about money creation, version of software to run and the transaction formats.
Periodically bookkeepers are allowed to add money to their own accounts, thereby creating money out of thin air. But this is only allowed according to very constrained rules. Those rules include a very slow gradual rate of money creation, until no more money can be created.
Math based voting system to determine what the majority thinks. Bitcoin requires bookkeepers to solve a very special math problem to vote. This is called “Proof of work” explained in the Bitcoin white paper “one vote per CPU” instead of one vote per person.
Voting is allowed to happen every 10 minutes to allow all bookkeepers to stay synchronized. Each new group of transactions that gets approved is called a block and these blocks are grouped together in a chain called the blockchain.

Where can blockchain be applied?

You can use the FITS model - Fraud, Intermediaries, Throughput, Stable data to understand the possibility of using blockchain applications in a particular environment. This could include

Fraud - an environment which has a history and likelihood of fraud involved in various transactions, making international payment providers early adopters of blockchain.
Intermediaries or middle men - areas where there are a lot of intermediaries involved who do not provide a lot of value, the application of blockchain can reduce transaction times from days to minutes by taking the middle men out.
Throughput - environments with high throughput or number of transactions per second (tps). Bitcoin, currently can only process 7 transactions per second. Visa processes around 1,700 transactions per second on average, claiming to be able to support 24,000 tps. Mastercard utilizes a network that claims to handle around 5,000 tps. Researchers are working on increasing the Bitcoin throughput.
Stable data - For a blockchain application you do not want volatile data, rather you want things that are going to stay the same for a while e.g. land ownership titles and personal information.

Impact on financial services

Bitcoin is the best-known application of blockchain technology. The prevailing view is that blockchain will cause two main shifts in the way banks do business

The first is in its broad potential to bring financial institutions closer together and make global collaboration easier.
- Due to the lack of trust, very little data is shared amongst financial institutions. Blockchain reduces this trust deficit and can allow seamless transfer of digital assets within a business network and better sharing of data across businesses.
- Creation of secured, shared data with common standards - a public distributed ledger that allows automatic synchronization and removes inefficiencies due to variations in internal processes and data formats within the systems at different institutions.
The second is by creating real efficiencies in the way the bank processes data.
- Simplification of payments infrastructure, the use of smart contracts to standardise post-trade processes without having to rely on a central certifying authority, and efficiently connecting parties in trade finance and syndicated lending by reducing the need for reconciliation at both ends of the business (purchaser and supplier).

Impact on Public Key Infrastructure (PKI)

The most commonly employed approach to Public Key Infrastructures (PKIs) is the Web PKI. It is a Certificate Authority (CA) based system that adopts a centralized trust infrastructure. Communications over the internet are secured through the safe delivery of public keys and the corresponding private keys. PKI has been the backbone of Internet security since its inception through the use of digital certificates.

However, there are problems with centralized PKIs such as CA-based systems. Because of the ability to impersonate another user or a website, CA systems are well-known targets for hackers. If they get hacked you get hacked. By breaching them, the bad guys gain access to a treasure-trove of personal and financial information traveling on the Internet. DigiNotar a Dutch CA whose systems were attacked and many fraudulent certificates issued had to eventually file for bankruptcy. CAs are a single point of failure that can be exploited to compromise encrypted online communication. Blockchain acts as a decentralized, open and transparent key-value store and eliminates traditional PKI vulnerabilities. It is capable of securing data to prevent MITM (Man-in-the-Middle) attacks, and to minimize the power and fragilities of third parties.

Impact on IOT

Today we transact not only with humans but machines and smart devices. How do we trust all the new IOT devices that are coming online? One prediction suggests that by 2020 we will have 7 times more smart devices than we have human beings in the world. That’s about 50 billion devices in 2020 that we’ll have to transact with and that we’ll have to trust. In 1982, a compromised software in the Trans-Siberian pipeline that controlled pump speeds and valve settings, produced pressures far beyond those acceptable to the pipeline joints and welds. This led to a three-kiloton, non-nuclear explosion so big that it was seen from space. There are centralized PKI based solutions to securing IOT device communications, however they are still prone to problems of centralized trust. Blockchain provides a decentralized PKI alternative by adopting given enough eyeballs, all bugs are shallow approach. This allows

Securely sharing any information between machines, including connected vehicles, smart appliances, or manufacturing equipment in a decentralized manner without depending upon a single third party to secure your entire system.
Providing open and transparent scrutiny to the adopted PKI security controls and protocols.
Reacting fast to misuse by revoking certificates by making the process transparent, immutable, and prevent attackers from breaking in, thus effectively avoiding the MITM attacks.

Think Docker, Think Security

Mon, 07 Nov 2016 15:23:00 +0000

Docker allows you to completely abstract the underlying operating system and run your app across multiple platforms (local machine, cloud or on-premise data centre) as long as the destination has the Docker runtime (Docker daemon) running. With Docker, the Continuous Delivery philosophy Build once deploy anywhere really comes to the fore. You build your binary artifact as a Docker image that includes all the application stack and requirements once and deploy the same image to various environments. This ensures the binary is built once and the same source code is promoted in subsequent deployments, allowing agile, continuous application delivery.

I have been using Docker for local development, testing and for running apps in production. Docker is pretty swift to get started with, allows rapid app development, setting up builds and running tests in a repeatable and consistent manner. You can get an application running on your local machine with all its dependencies (web servers, databases) in fairly quick time. Does that mean you ship your machine to production? Probably not! Because you are working with the Docker abstraction, do you need to worry about any underlying security risks, is it all taken care of or should you even care?

Isolation

Docker is a virtualization technique used to create isolated environments called containers for running your applications. A container is quite like a VM but light-weight. It is a bare minimum linux machine with minimum packages installed which means it uses less CPU, less memory and less disk space than a full blown VM. Containers are more like application runtime environments that sit on top of the OS (Docker host) and create an isolated environment in which to run your application.

Docker uses the resource isolation features of the Linux kernel such as Namespaces and cgroups to create the walls between containers and other processes running in the host.

Namesapces control what processes can see. They allow resources to have separate values on the host and in the container; for example PID 1 inside a container is not PID 1 on the host. However not all resources that a container has access to are namespaced i.e they are not isolated on the host and in the containers. Containers running on the same host still share the same operating system kernel and any kernel modules.
cgroups (abbreviated from control groups) control what processes can use by limiting and isolating resource usage (CPU, memory, disk I/O, network, etc.) for a process.

Linux Security Module is another layer of protection that sits on top of namespaces and cgroups.

AppArmor can control and audit various process actions such as file (read, write, execute, etc) and system functions (mount, network tcp, etc). Again, while running a docker container you get a sensible set of defaults:

Preventing writing to /proc/{num}, /proc/sys, /sys
Preventing mount

SELinux provides a mechanism for supporting access control security policies. Uses labels. Defaults for running docker include:

Gets access to everything in /usr and most things in /etc
To give more access: relabel content on the host
To restrict access to things in /usr or things in /etc: relabel them

Seccomp is another security facility in the Linux kernel that allow an application to define what syscalls it allows or denies. Docker’s default seccomp profile is a whitelist which specifies the calls that are allowed. Running docker by default

Prevents ~150 uncommon or dangerous syscalls

Capabilites are a set of privileges that can be independently enabled or disabled for a process to provide or restrict access to the system. Removing capabilities can cause applications to break, therefore deciding which ones to keep and remove is a balancing act. Docker containers run with a sensible subset of default capabilities, e.g. a container will not normally be able to modify capabilities for other containers. Apart from the capabilities removed by default you can remove or add capabilities while running a container.

Security challenges

I was at a GOTO conference in Stockholm earlier in the year and Adrian Mout speaking on Docker security highlighted some security issues mentioned below, that could affect your apps running inside containers. The following is not a comprehensive list but one that should get you thinking.

Kernel exploits: The kernel is shared amongst all the containers and the host. A flaw in the kernel could be exploited by a container process which will bring down the entire host.
Denial of service: All containers share the kernel resources. If one container manages to hog kernel resources like CPU, memory or block I/O, it can starve the other containers on the host for resources, resulting in a denial of service attack.
Container breakouts: Running as root on the container means you will be root on the host. Therefore you need to worry about escalated privileges attack where a user can gain root access on the host due to a vulnerability in the application code. If an attacker gains access to one container that should not allow access to other containers or the host. Therefore it becomes important to run your container with restricted privileges.
Tampered images: You need to be sure that the image that you are running is from a trusted source and not from an attacker who has tricked you. The images that you run should also be up to date, scanned and without any known security vulnerabilities.
Secret Leakages: Secrets for your applications (like API keys and Database passwords) need to be protected from falling into the wrong hands. You need to be very careful before putting these as plain text in source code; for example in a Dockerfile. Once the secret has made it into the Dockerfile, even if you remove the secret later, an attacker can still retrieve it from the history of the Docker image built from the Dockerfile. Ideally using a secret vault for storing application secrets and allowing restricted access to the secret vault serves as a secure mechanism to distribute secrets.

Mitigations

Having gone through key Docker security issues, lets look at some ways to prevent them. CIS Docker Benchmark provides an elaborate list of docker security recommendations. Defence in Depth is a common approach to security that involves building multiple layers of defences in order to hinder attackers. The following mitigations are based on securing the host, container and image in a container based environment.

Host and kernel

Use a good quality supported host system for running containers with regular security updates. Keep the kernel updated with the latest security fixes. The security of the kernel is paramount.
Apart from updating and patching the kernel, it might be worth considering running a hardened kernel using patches such as those provided by grsecurity (https://grsecurity.net/) and PAX (https://pax.grsecurity.net/). This provides extra protection against attackers manipulating program execution by modifying memory (such as buffer overflow attacks).

Run containers with

Least privilege (non admin). When a vulnerability is exploited, it generally provides the attacker with access and privileges equal to those of the application or process that has been compromised. Ensuring that containers operate with the least privileges and access required to get the job done reduces your exposure to risk. A docker container runs as root by default, if there is no user specified. Create a non privileged user and switch to it using a USER statement before an entrypoint script in the Dockerfile.
Limited file system (Readonly access). Prevents attackers from writing a script and tricking your application from running it. This can be done by passing --read-only flag to docker run.
Limited resources (CPU and memory). Limiting memory and CPU prevents against DoS attacks. docker run provides options -m and -c for setting the memory and cpu requirements respectively for running a container.
Limited networking. By default containers running on the same host can talk to each other whether or not ports have been explicitly published or exposed. A container should open only the ports it needs to use in production, to prevent compromised containers from being able to attack other containers.
No access to privileged ports. Only root has access to privileged ports, so if you’re talking to a privileged port you know you’re talking to root.

Docker images

Only run images from trusted parties. Control the inflow of docker images into your development environment. This means using only approved private registries and approved images and versions. As of Docker 1.8 a new security feature was implemented called Docker Content Trust. This feature allows you to verify the authenticity, integrity, and publication date of all Docker images available on the Docker Hub Registry. This feature is not enabled by default but if you enable it by export DOCKER_CONTENT_TRUST=1 Docker notifies you, when you attempt to pull down an image that isn’t signed.
Run regular scans on your docker images for vulnerabilities. Vulnerability management is tricky because source images aren’t always patched. Even if you get the base layer up to date, you probably also have tens or hundreds of other components in your images that aren’t covered by the base layer package manager. Because the environment changes so frequently, traditional approaches to patch management are irrelevant. To stay in front of the problem you have to a) find vulnerabilities as part of the continuous integration (CI) process, and b) use quality gates to prevent the deployment of unsafe and non compliant images in the first place. Docker Hub has its own image scanning tool and there are paid options like Twistlock that provide image vulnerability analysis as well as a container security monitoring.
If needed, remove unwanted packages that your application does not depend upon with major and critical vulnerabilities from the base image to reduce the attack surface.

Services, microservices, bounded context, actors.. the lot

Fri, 30 Oct 2015 15:23:00 +0000

I was at the DDD exchange recently where we had the likes of Udi Dahan, Eric Evans and Scott Wlaschin on the panel. In a post event Q&A session I asked the panel - “Is microservices SOA renamed? “ which triggered an hour long debate. The panelists argued amongst themselves about what exactly a service or a microservice means. By the end of the debate I doubt any one of us was any more wiser. Clearly there was no consensus on the definition of a microservice and what it actually means. It is quite a buzzword these days but none of the panelists could come to a common understanding. This raised a few questions in my head. Disappointed with the expert advice, I decided to look out for a definition of my own.

As what’s needed in such a situation, I looked up the dictionary for the word service - “the action of helping or doing work for someone”. Does a microservice fit into this general definition? In order to come up with a more definitive answer, lets recollect the knowledge that is already out there.

“Microservices aim to do SOA well, it is a specific approach of achieving SOA in the same way as XP and Scrum are specific approaches for Agile software development.” - Sam Newman (Building Microservices)

If microservices is about doing SOA (Service orientated architecture) well, it is probably worth looking at the SOA tenets:

Services are autonomous - Cohesive single responsibility.
Services have explicit boundaries - Loosely coupled, owns its data and business rules.
Service share contract and schema, not Class or Type or a Database
Service compatibility is based upon policy - Explicitly state the constraints (structural and behavioral) which the service imposes on its usage.

These tenets do not appear to be too different from the object oriented principles. According to Alan Kays 1971 description of Smalltalk

An object is a little computer that has its own memory, you send messages to it in order to tell it to do something. It can interact with other objects through messages in order to get that task done.

An object’s private memory provides it the autonomy and message based communication an explicit boundary from other objects. Can we define a service based on the above principles? Lets look at the tenets a bit closely.

What is a Service?

Autonomy of a service suggests it is independent of other services to perform its tasks, therefore in order to be independent it needs to have one and only one well defined responsibility. Uncle Bob has summarised Single Responsibility Principle rather well - “Gather together those things that change for the same reason and separate those things that change for different reasons.” In short a service should not have more than one reason to change.
Boundaries are drawn to restrict free movement and ensure all movement is governed by a set of rules. In the context of a service this restriction is enforced on free movement of data across a service boundary. All data and business rules reside within the service imposing strict restrictions on any movement in and out.
Services interact with other services through a shared contract by sending messages. These messages contain stable data (i.e. immutable, think events). The data going through service boundaries is minimal and very basic.
Usage of a service enforces certain constraints, the incoming messages conform to an expected structure and format.

What a service is NOT?

Anything with the word Service appended to it does not automatically qualify as a service.
A service that has only a function is a function not a service, like calculation, validation (not be confused with DDD’s Domain Services which is a more granular concept). Making it remotely callable through RPC/SOAP still does not make it a service.
A service that only has data is a database not a service. Doing CRUD through REST over HTTP does not change that.

Philip Kruchten’s 4+1 Architecture View Model describes software architecture based on multiple concurrent views.

Defining services involves breaking an overall system into smaller isolated sub systems so that adding features to the overall system requires touching as few sub systems as possible. This decomposition can be at the logical level (business capabilities - the reason for something to exist), component level (dlls, jars, source code repos), process level (web app, http endpoints) or the physical level (machines, hosts). Bounded context in DDD terminology focuses on the logical separation whereas Microservice focuses on the physical separation. The idea of slicing up your system into manageable chunks is the key here.

What is an Actor?

Actors is another decomposition model that allows dividing a system into smaller isolated tasks or actors that can run concurrently. Traditional approaches to concurrency are based on synchronizing shared mutable state which is difficult to get right, as it often involves locking and coordination. Wouldn’t it be better not having to deal with coordinating threads, synchronization and locks? Actors achieve this by avoiding shared state and only mutating internal private state between processing messages. When there are no shared state mutations, synchronization and locking are no more required. When an actor wants to communicate with another actor, it sends a message rather than contacting it directly, all messaging being asynchronous. Apart from concurrency and performance gains there are other benefits with the actor based approach like hot code replacements but asynchronous programming is a whole new ballgame.

Asynchronous message communication allows Actors to embrace latency, potentially at the cost of simplicity. Asynchronous messaging is an event driven model as compared to synchronous model where you have a sequence of events that execute sequentially (in a given order) which in theory is easy (read familiar) to program and reason about. An asynchronous model (event driven) on the other hand, allows you to scale better but you have to find a way to impose ordering on the incoming requests.

In summary

Partitioning can occur at different levels, be it a service, a microservice, an actor or even an object. What we really gain from this partitioning is isolation. We want a small computer with private memory that you can interact with through a contract, without any access to its internal private state. Isolation is easily compromised in objects. We often break encapsulation by sharing an object’s private memory through public getters and setters in languages like Java and C#. Actors enforce this isolation by restricting access to private memory (internal state). Microservies make it even harder to break the isolation by often introducing physical separation and communication over a network.

Each service ends up having its own process or/and network boundary. The contract is a service updates it's shared memory and exposes a mechanism to read from its shared memory but the service itself is the only one that is allowed to write to it.

Turns out, microservices reinforce a lot of the old ideas like SOA and object oriented principles that we have known for sometime. Adding features to your system will invariably involve touching more than one sub system at a given time but depending upon how well the system is isolated, it should involve touching as few sub systems as possible. Isolation gives you all the good stuff - loosely coupled systems, failure isolation, independent evolution and scalability. But distributed mircoservices is not the only way to achieve isolation. Modularised monoliths can very well be isolated too (obviosuly not over a network but in memory), they probably need a bit more discipline.

Problem with mutating state

Tue, 19 May 2015 15:23:00 +0000

The purpose of any computer program is to take some inputs and produce an output. Producing an output causes the program to have an effect which means during its execution cycle, the program changes certain values. A live program models the real world where things change frequently over time and in parallel. But while writing programs we only have a static view of the problem domain and with the best of intentions and tools at hand we try to manage state change in our program as it would happen in the real world. This leaves us having to deal with values that change over time. Are OO languages capable of handling this complexity easily or do we need to look further?

“No man ever steps in the same river twice, for it's not the same river and he's not the same man.” - *Heraclitus*

Rich Hickey in his keynote at the JVM Languages summit talked about value, identity and state. Values are essentially constants that do not change. The flowing water in a river makes the river change constantly over time, this introduces the concept of identity. Identity is a succession of related values where the current one is caused from the previous. State is the value of an identity at a given time.

1; // value (immutable)
int x = 1; // identity (variables are like rivers that change)
x = x + 1; // x changes state (introduces side effect)

An assignment statement however, mutates state and introduces the concept of time. The value of x changes before and after the assignment statement therefore time becomes important. The assignment statement separates the code above the assignment from the code below the assignment because other parts of the system that can access x can now have different views of x depending upon what time it is observed. If x is shared between multiple functions, objects or threads and either has the ability to assign to x, this can lead to side effects.

When you have a function that gives you a side effect then you need another function to undo the side effect. This leads to non deterministic results that introduce accidental complexity (problems which engineers create and fix). For example a program whose output is influenced by the particular order of execution of threads or by a call to gettimeofday or some other non-repeatable thing is generally best considered as non-deterministic. This accidental complexity is inherent in OO languages. When you open a file you must also close it. Functions with side effects are also separated in time - opening a graphical context must precede closing it, using an unmanaged resource in .NET must always precede disposing it. If these functions are not called in the correct order it leads to memory leaks. OO languages support garbage collection to manage some side effects but not all.

Managing side effects

One of the main ideas of Functional programming is to manage side effects. It enables programs to make decisions based on stable values rather than those that change over time (like rivers). Evaluation of a pure function or higher order function has no side effects. f(x) is always the same, no matter what. This leads to referential transparency which implies that provided a set of inputs a function will always result in the same output or it will have the same behaviour.

A pure function is stateless rather than stateful i.e it does not update any shared state/memory. This means that execution of a pure function has no effect on the output of the execution of any other function in your application. However in practice, applications do need to have some side effects.

"In the end, any program must manipulate state. A program that has no side effects whatsoever is a kind of black box. All you can tell is that the box gets hotter" - *Simon Peyton-Jones (Haskell contributor)*

The key is to limit side effects, clearly identify them, and avoid scattering them throughout your application. This can be achieved by having more and more pure functions inside your application that depend only on the input and return a value without:

Accessing global memory/state
Modifying input(s)
Changing shared memory/state

It is interesting to note in an imperative style, the above situations result as a use of the assignment statement. Functional programs attempt to remove the non determinism and complexity in your application by containing side effects. This makes them easier to write and maintain:

Without temporal coupling - order in which functions are called becomes irrelevant.
Fewer concurrency issues due to restricted updates to shared memory/state.
Less time spent in the debugger without having to constantly ask “What is the application state ?”

Time to give up the assignment statement?

The addition of more cores to computer hardware means, the individual throughput of each core goes down but the throughput of the chip with multiple processors goes up, if you can take advantage of the multiple cores. In order to utilise all those cores, we will have to learn to write more pure functions. Our ability to write multi-threaded programs will depend upon the scarce and disciplined use of the assignment statement.