Akka Anti-Patterns, Goodbye: Six Features of Akka 2.6

AKKA ANTI-PATTERNS, GOODBYE:
SIX FEATURES OF AKKA 2.6
WEBINAR
MANUEL BERNHARDT

WHY?
Webinar - https://manuel.bernhardt.io - @elmanu

WHY?
> collected recuring "worst practices"
over the years 1
> ranges from beginner mistakes to
design issues that can affect a
project over years
> most of it has been adressed in Akka
2.6! Let's have a look at 6 of them
1
https://manuel.bernhardt.io/akka-anti-patterns

MANUEL.BERNHARDT.IO
> Helping companies to get started with
reactive systems...
> ... or to solve critical production
issues
> Lightbend consulting and training
partner

AKKA 2.6
> previous major release (2.5.0) in April 2017
> introduces Akka Typed API, but does not deprecate the
Classic one
> Artery as default transport
> many, many improvements and fixes

If you fail to plan, you are
planning to fail!
— Benjamin Franklin

ANTI-PATTERN #1
NOT DEFINING A PROTOCOL

> quite often, actor systems are built without giving
much thought to the message protocol itself
> this works quite well in the beginning, but can become
quite a burden in the long run

HOW THINGS MAY GO WRONG
> tight coupling: message re-use that makes it harder to
evolve over time
> responsibilities: responsibilities that belong together
may be split accross actors over time
> inefficiencies: "organic" growth of an actor system
makes it easy to loose track of how information is
exchanged, can lead to unnecessary message exchanges

It is really difficult to evolve protocols in the classic
API due its dynamic nature / very open definition

"NOT DEFINING A PROTOCOL" IN AKKA 2.6 /
TYPED
⠀

TYPED
> you can't.

TYPED
> you can't.
> Akka Typed forces you to define a protocol
> you need to specify:
> what type of message an actor can handle
> what type of response a message expects

TYPE OF MESSAGE AN ACTOR CAN HANDLE
// marker of the message protocol
sealed trait ConfigurationCommand
// actor supporting this protocol
class Configuration(
context: ActorContext[ConfigurationCommand]
) extends AbstractBehavior[ConfigurationCommand] { ... }

TYPE OF MESSAGE AN ACTOR EXPECTS (1)
// a message of the Configuration protocol
final case class RetrieveConfiguration(
merchantId: MerchantId,
replyTo: ActorRef[ConfigurationResponse]
) extends ConfigurationCommand

TYPE OF MESSAGE AN ACTOR EXPECTS (2)
sealed trait ConfigurationResponse
final case class ConfigurationFound(
configuration: MerchantConfiguration
) extends ConfigurationResponse

HOW DOES THIS CHANGE THE GAME?
> the beginner mistake of sending messages to the wrong
actor can no longer happen
> message paths are now checked by the compiler
> you can infer the protocol flow from the
ActorRef[Message] included in the messages

sealed trait BankingCommand
final case class Login(
credentials: Credentials,
replyTo: ActorRef[LoginResponse]
) extends BankingCommand
sealed trait LoginResponse
final case class Authenticated(
session: ActorRef[SessionCommand]
) extends LoginResponse
sealed trait SessionCommand
final case class CreateTransaction(
transaction: Transaction
) extends SessionCommand

Time is a storm in which we are all
lost.
— William Carlos Williams

ANTI-PATTERN #2
RACE CONDITIONS

> in the actor model, actors handle messages one after
another
> it is safe to use mutable state inside of an actor
> unless you do the following

CLASSIC API
class TransactionManager(processor: ActorRef) extends Actor {
var transactions = List.empty[Transaction]
def receive: Receive = {
case ProcessTransaction(transactionRequest) =>
val response: Future[ProcessingResponse] =
processor ? Process(transactionRequest)
response.map {
// here be dragons!
transactions = response.transaction :: transactions
}
}
}

WHAT'S THE PROBLEM?
> using asynchronous constructs inside of the actor
(single-threaded illusion) is error-prone
> quite easy to make mistakes of this kind (I've seen it
many, many times)
> the recommended way is to use ask in combination with
the pipe pattern, but people may forget / not know

"MISUSING ASK / FUTURES" IN AKKA 2.6 /
TYPED
⠀

"MISUSING ASK / FUTURES" IN AKKA 2.6 /
TYPED
> you can't.

class TransactionManager(
processor: ActorRef[ProcessorCommand]
) extends AbstractBehavior[ManagerCommand] {
var transactions = List.empty[Transaction]
def receive: Receive = {
case ProcessTransaction(transactionRequest) =>
context.ask(processor, Process(transactionRequest)) {
case Success(ProcessingResponse(transaction)) =>
AddTransaction(transaction)
case Failure(_) => TransactionFailed(transactionRequest)
}
case AddTransaction(transaction) =>
transactions = response.transaction :: transactions
case TransactionFailed(request) => // ...
}
}

"...some men aren't looking for
anything logical, like money. They
can't be bought, bullied, reasoned,
or negotiated with. Some men just
want to watch the world burn."
— Alfred Pennyworth, Batman, The Dark Knight

ANTI-PATTERN #3
BLOCKINGWebinar - https://manuel.bernhardt.io - @elmanu

HOW CAN THIS HAPPEN?
> calling a synchronous API
> or calling an API that calls an API that calls an API...
that calls a synchronous API
> explicitly waiting for the completion of a Future
(Await.result)

Akka Anti-Patterns, Goodbye: Six Features of Akka 2.6

BRINGING DOWN AKKA 2.6 BY BLOCKING
⠀

> you can't.

> you can't.
> new internal dispatcher for all of Akka's internal
actors
> you can still starve your own actors, but Akka's
internal mechanisms will continue working

IF YOU REALLY NEED TO BLOCK
blocking-io-dispatcher {
type = Dispatcher
executor = "thread-pool-executor"
thread-pool-executor {
fixed-pool-size = 8
}
throughput = 1
}

IF YOU REALLY NEED TO BLOCK
context.spawn(
blockingBehavior,
"blocking-actor",
DispatcherSelector.fromConfig(
"blocking-io-dispatcher"
)
)

This is the way the world ends
Not with a bang but a whimper.
— T.S. Eliot

ANTI-PATTERN #4
USING JAVA SERIALIZATION

HOW COULD THIS HAPPEN?
> leaving the default on
> ignoring the nagging warning messages in the logs
> affects both serialization of messages over the wire
and for persistence

WHY IS IT BAD?
> performance penalty!
> poor candidate for protocol evolution - message
evolutions result in older components not able to
process them any longer
> persistance - messages and snapshots can't be
processed any longer after changes
> poor security

USING JAVA SERIALIZATION IN AKKA 2.6
⠀

> you can't.

> you can't.
> ... or at least not by default
> Jackson is the new default, with a decently performing
binary serialization format (CBOR), should be good in
most cases

USING JACKSON SERIALIZATION
/** marker interface **/
public interface CborSerializable {}
class Message implements CborSerializable {
public final String id;
public Message(String id) {
this.id = id;
}
}

USING JACKSON SERIALIZATION
akka.actor {
serialization-bindings {
"com.example.CborSerializable" = jackson-cbor
}
}

JACKSON VS. PROTOCOL BUFFERS
> protobuf as a very popular binary serialization in
Akka projects in the past
> it turns out that Jackson is a very decent alternative
and involves much less work than protobuf - also in
terms of evolutions
> definitely worth checking it out in detail before falling
back to protobuf

The desire of excessive power
caused the angels to fall
— Francis Bacon

ANTI-PATTERN #5
USING AKKA REMOTING** DIRECTLY

WHAT'S THE PROBLEM?
> Akka Remoting allows to build pair-to-pair
connections between nodes / Actor Systems
> there's a lot more things to do when it comes to
building stable applications on top of the network
> not trivial

DIRECTLY USING AKKA REMOTING IN AKKA
2.6
⠀

2.6
> you can't.

2.6
> you can't.
> remote watch and deployment will only work when
using the Cluster extension

AKKA CLUSTER
> membership service
> primitives for building distributed
systems: membership, singleton,
sharding, CRDTs, etc.
> check out the articles series2
about
it
2
https://manuel.bernhardt.io/articles/#akka-cluster

There are two ways to be fooled.
One is to believe what isn't true;
the other is to refuse to believe
what is true.
— Soren Kierkegaard

ANTI-PATTERN #6
USING AUTO-DOWNING

GROUP MEMBERSHIP WITH A SINGLE
GROUP IS IMPOSSIBLE WHEN THERE ARE
NODES THAT ARE SUSPECTED OF
HAVING FAILED 3
3
Chandra et al: On the Impossibility of Group Membership (1996)

IT WOULD BE UNWISE TO MAKE
MEMBERSHIP-RELATED DECISIONS
WHILE THERE ARE PROCESSES
SUSPECTED OF HAVING CRASHED

"NETWORK PARTITIONS
DO NOT HAPPEN"
> "we have never observed a partition
in our network"
> "we have been running this
infrastructure for X months now and
nothing ever happened"
> "our network is reliable"

NETWORK PARTITIONS
> more on https://aphyr.com/posts/288-the-network-
is-reliable
> it's not a matter of if, but when.

IN AKKA CLUSTER, THE LEADER CANNOT
TAKE DECISIONS SO LONG AS THERE
ARE MEMBERS THAT ARE FLAGGED AS
UNREACHABLE BY THE FAILURE
DETECTOR

IF YOU USE AUTO-DOWNING IN
PRODUCTION, YOU ARE GUARANTEED TO
GET A SPLIT-BRAIN SCENARIO AT SOME
POINT IN TIME

USING AUTO-DOWNING IN AKKA 2.6
⠀

> you can't.

> you can't.
> it's gone.

> you can't.
> it's gone.
> use a proper split brain resolver / preventer

SPLIT BRAIN RESOLVERS
> Akka Split Brain Resolver by Lightbend
> open-source implementations (at your own risk)
> roll your own (seriously, don't do that)

ANTI-PATTERN #7BEING OUT OF TOUCH WITH THE HARDWARE

WHY IS THIS A BAD THING?

AWS & VCPUS

GCP & VCPUS

EXAMPLE
> kubernetes deployment with 2 CPUs /
container
> default Akka configuration - 16
threads default EC + 16 threads for
the internal dispatcher
=> 32 threads contending for 1 real
core which itself may not be dedicated

BEING OUT OF TOUCH WITH THE HARDWARE
IN AKKA 2.6
⠀

IN AKKA 2.6
> you absolutely can.

IN AKKA 2.6
> you absolutely can.
> Akka can do a lot, but it can't do magic
> learn the basics, know your infrastructure

THANK YOU
> Questions, comments, feedback?
> Contact me at manuel@bernhardt.io /
@elmanu
> Read more about Akka at
https://manuel.bernhardt.io/
articles

Akka Anti-Patterns, Goodbye: Six Features of Akka 2.6

More Related Content

Akka Anti-Patterns, Goodbye: Six Features of Akka 2.6