You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
198 lines
10 KiB
198 lines
10 KiB
<!DOCTYPE html> |
|
<html lang="en"> |
|
<head> |
|
<meta http-equiv="X-UA-Compatible" content="IE=edge"> |
|
<meta http-equiv="content-type" content="text/html; charset=utf-8"> |
|
|
|
<!-- Enable responsiveness on mobile devices--> |
|
<!-- viewport-fit=cover is to support iPhone X rounded corners and notch in landscape--> |
|
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1, viewport-fit=cover"> |
|
|
|
<title>Julio Biason .Me 4.3</title> |
|
|
|
<!-- CSS --> |
|
<link rel="stylesheet" href="https://blog.juliobiason.me/print.css" media="print"> |
|
<link rel="stylesheet" href="https://blog.juliobiason.me/poole.css"> |
|
<link rel="stylesheet" href="https://blog.juliobiason.me/hyde.css"> |
|
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=PT+Sans:400,400italic,700|Abril+Fatface"> |
|
|
|
|
|
|
|
|
|
|
|
</head> |
|
|
|
<body class=" "> |
|
|
|
<div class="sidebar"> |
|
<div class="container sidebar-sticky"> |
|
<div class="sidebar-about"> |
|
|
|
<a href="https://blog.juliobiason.me"><h1>Julio Biason .Me 4.3</h1></a> |
|
|
|
<p class="lead">Old school dev living in a 2.0 dev world</p> |
|
|
|
|
|
</div> |
|
|
|
<ul class="sidebar-nav"> |
|
|
|
|
|
<li class="sidebar-nav-item"><a href="/">English</a></li> |
|
|
|
<li class="sidebar-nav-item"><a href="/pt">Português</a></li> |
|
|
|
<li class="sidebar-nav-item"><a href="/tags">Tags (EN)</a></li> |
|
|
|
<li class="sidebar-nav-item"><a href="/pt/tags">Tags (PT)</a></li> |
|
|
|
|
|
</ul> |
|
</div> |
|
</div> |
|
|
|
|
|
<div class="content container"> |
|
|
|
<div class="post"> |
|
<h1 class="post-title">Self-Healing Microservices</h1> |
|
<span class="post-date"> |
|
2020-01-03 |
|
|
|
<a href="https://blog.juliobiason.me/tags/microservices/">#microservices</a> |
|
|
|
<a href="https://blog.juliobiason.me/tags/healing/">#healing</a> |
|
|
|
<a href="https://blog.juliobiason.me/tags/artifacts/">#artifacts</a> |
|
|
|
</span> |
|
<p>All the <a href="https://blog.juliobiason.me/code/microservices-artifact-input-state/">previous</a> |
|
<a href="https://blog.juliobiason.me/code/microservices-artifact-ejection/">discussions</a> I bought about |
|
microservices was just a prelude to something I still don't have a fixed |
|
solution: how do microservices "heal" themselves in case of missing data?</p> |
|
<span id="continue-reading"></span> |
|
<p>Quick recap before jumping into the problem: Microservices produce artifacts; |
|
artifacts are either send downstream through some message broker to other |
|
microservices or kept in the same microservice for future requests; |
|
microservices can listen to more than one data input to build their artifact.</p> |
|
<p>Previously I mentioned an example of a score microservice that produces an |
|
artifact with the current game score for each team and the names of the |
|
players that scored. This microservice could listen to:</p> |
|
<ol> |
|
<li>The teams queue: this may be needed so we can show the team name or its |
|
abbreviation in the score request; once a team appears in the championship, |
|
the microservice adds it to its state for future reference<sup class="footnote-reference"><a href="#1">1</a></sup>.</li> |
|
<li>The players queue: same as above, so the microservice can return the player |
|
name, nickname, shirt number or something player related if the player |
|
scores; again, it keeps listening to the player queue and stores the |
|
players in its state.</li> |
|
<li>The match queue: if a match will happen, it has to have a score, probably |
|
starting at 0 with no players in the goal list; this is just to avoid any |
|
issues with the services requesting scores of matches that didn't start |
|
or haven't had any goals yet; in any case, the artifact will be ready to be |
|
retrieved.</li> |
|
<li>The narration queue: by listening to the narration queue, the score |
|
microservice will detect any goals, update its state and generate the new |
|
artifact.</li> |
|
</ol> |
|
<p>The keyword to take from the above list is "could": Depending on the way the |
|
microservice <em>and</em> the messages are built, it may not be necessary to have all |
|
this.</p> |
|
<h2 id="using-full-blown-messages">Using full-blown messages</h2> |
|
<p>Let's start with the easiest way to avoid listening to all those queues: |
|
Full-blown messages.</p> |
|
<p>In a full-blown message, all the related data is sent along with the main |
|
information. Using the previous example, the service could listen to just the |
|
match and narration queue, but expect that the "NewMatch" message will contain |
|
the names of the teams, their abbreviation, logo, probably some id and so on; |
|
the same for the "NewNarration" message: it will contain the player name, |
|
nickname, shirt name, player id and so on.</p> |
|
<p>The problem with full-blown messages is that they tend to become bigger and |
|
bigger: As more microservices are plugged in the system, more fields may be |
|
required -- and dropped by services that don't need those fields.</p> |
|
<p>The pro side of full-blown messages is that a microservice will always have |
|
all the information necessary, while keep the number of listening queues low. |
|
This would also help if you just add a new service in the pool: if it starts |
|
with a blank state, it will be able to build all the state from scratch, |
|
'cause all the information is <em>already there</em>.</p> |
|
<h2 id="listen-to-base-queues-request-the-rest">Listen to base queues, request the rest</h2> |
|
<p>Almost like the solution before, the service would listen to the narrations |
|
and matches, but once it detects a missing information (for example, the |
|
narration event says player with ID, but this ID doesn't exist in its state), |
|
the service would request the more "stale" information (players, teams and |
|
products are not added all the time, for example) for some other service and |
|
fill the lacking information in its state.</p> |
|
<p>This means that this microservice now, instead of knowing only about queues, |
|
now has to have information about other services (the ones that process and |
|
store the "stale" data) and their interfaces -- and, in general, it would also |
|
require some service discovery in the system. Those microservices would be |
|
the "two faced" type of microservice, which receives information, store in the |
|
state, build the artifact but also has an interface for it to be retrieved |
|
instead of simply receiving, processing and passing it along. Caching would |
|
also be advised here, so one service can't flood the other requesting the same |
|
data over and over -- and updates from time to time would make sense in some |
|
situations.</p> |
|
<p>The messages are shorter now ('cause you receive only the team/player ID |
|
instead of everything) and retrieval of information happen when necessary, but |
|
where you reduce the number of listeners, you increase the number of requests. |
|
As will full-blown messages, a new service can easily build its own state from |
|
scratch without any issues -- it will do a lot of requests, but it will, |
|
eventually, have all the necessary information.</p> |
|
<h2 id="listen-to-all">Listen to all</h2> |
|
<p>This is exactly same solution as presented in the example above: the |
|
microservice keeps listening to the queues of all related events and build the |
|
state with them.</p> |
|
<p>One problem with this solution: since the queues are asynchronous, |
|
there could be a problem with the ordering of the data, with goals coming |
|
before players (for different reasons). In this case... what would the service |
|
do? Reject the goal in the hopes the player will appear, to avoid any |
|
inconsistencies, and that the message broker requeue the event?</p> |
|
<p>One solution would have services that, along with this one, listen to one |
|
specific data: the score microservice listens to all four, but one |
|
microservice listens only to the player queue. This service would |
|
process things way faster than the score, and serve as sort of "fallback" in |
|
case some data is missing, kinda like the solution above. This will reduce the |
|
traffic in the network, but it'd create have duplicate data in different |
|
services -- although that last point shouldn't be a problem in the first |
|
place.</p> |
|
<p>New services will find it problematic, 'cause although they are receiving the |
|
main data, they are were not alive when the more "stale" data was processed; |
|
they will need to either communicate with other services to get this |
|
information, or someone |
|
will have to manually duplicate the sources.</p> |
|
<h2 id="single-queue">Single queue</h2> |
|
<p>I just describe solutions in which every data has its own queue, but what if |
|
we put <em>all</em> the events in the same queue? This way, order is assured (players |
|
will be queue before the goals, and the services will process players before |
|
they even see there is a goal).</p> |
|
<p>This reduces the number of listeners, but it requires some good |
|
message design, specially in statically typed languages, which usually require |
|
a well-defined structure for serialization and deseralization. </p> |
|
<p>But it solves almost everything else: there is no issue with the processing |
|
order, the number of listeners is low and the messages are small. But it will |
|
also make new services suffer from the lack of stale data, forcing them to |
|
communicate with other services or to have the data manually copied when they |
|
are brought up.</p> |
|
<h1 id="the-best-one">The best one?</h1> |
|
<p>Honestly, I have no idea. I'm siding with "Full-blown messages" simply 'cause |
|
it simplifies the structure of the services, even knowing that network is not |
|
free; if I used some non-statically typed language, I'd probably side with the |
|
single queue one. But, again, I don't think there is any "one size fits all".</p> |
|
<p>Probably there are more architectural options for this, and those are the ones |
|
I can remember discussing with my coworkers.</p> |
|
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup> |
|
<p>It's worth noting that the microservice may simply drop some of the |
|
information. For example, if the artifact produced only requires the |
|
abbreviated name, it full name may be completely dropped from the state.</p> |
|
</div> |
|
|
|
</div> |
|
|
|
|
|
|
|
|
|
</div> |
|
|
|
</body> |
|
|
|
</html>
|
|
|