jbiason
/
Presentations
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
My presentations, using Reveal.js (mostly in Portuguese).
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
431 Commits
1 Branch
0 Tags
126 MiB
 
 
 
 
 
Branch: master
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'master'
${ noResults }
Presentations/mr-banks.html

511 lines
15 KiB
Raw Permalink Blame History

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Mr Banks -- Ou Como Aprendi a Parar de Me Preocupar e Amar o Stream Processing</title>
<meta name="description" content="Colocando uma aplicação Flask em produção em 40 minutos (ou menos)">
<meta name="author" content="Julio Biason">
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no, minimal-ui">
<link rel="stylesheet" href="reveal.js/css/reveal.css">
<link rel="stylesheet" href="reveal.js/css/theme/night.css" id="theme">
<!-- Code syntax highlighting -->
<link rel="stylesheet" href="reveal.js/lib/css/zenburn.css">
<!-- Printing and PDF exports -->
<script>
var link = document.createElement( 'link' );
link.rel = 'stylesheet';
link.type = 'text/css';
link.href = window.location.search.match( /print-pdf/gi ) ? 'css/print/pdf.css' : 'css/print/paper.css';
document.getElementsByTagName( 'head' )[0].appendChild( link );
</script>
<!--[if lt IE 9]>
<script src="lib/js/html5shiv.js"></script>
<![endif]-->
<style type="text/css" media="screen">
.happy {
color: yellow;
}
.reveal section img {
border: none;
}
.reveal ul.empty {
list-style: none outside;
}
li {
display: block;
}
.cursor {
background-color: #666;
color: white;
}
img {
max-height: 90%;
}
</style>
</head>
<body>
<div class="reveal">
<div class="slides">
<section>
<section data-background="_images/unclephil.jpg">
<h1 class="semi-opaque">
Mr Banks
<small class="fragment">Ou Como Aprendi a Parar de Me Preocupar e Amar o Stream Processing </small>
</h1>
</section>
</section>
<section>
<section>
<h2>Batch Processing</h2>
<pre><code>
cat /var/log/messages | grep sync | cut -d\ -f 1 | sort | uniq
</code></pre>
</section>
<section>
<h2>Batch Processing</h2>
<pre><code>
v5 2017-10-17T11:09:59+00:00 7695w 27695w
1462530548 ha 1441740010 servico.clubeportoseguro.com.br
0.064 GET BYPASS 200 - 200 1199 http
/api/produto - -
application/json; charset=utf-8 HTTP/1.1
230 1205 0.047 0.064 0.064 8000
200.230.226.125 17759 - - - - -
- - Jakarta Commons-HttpClient/3.1 - - -
</code></pre>
<aside class="notes">
Esse é uma linha de exemplo dos nossos logs.
Um dos nossos serviços, o Monkeys Collect, captura pacotes de
2000 dessas linhas e coloca no Kafka.
</aside>
</section>
<section>
<h2>Batch Processing</h2>
<ul>
<li>Campos são nomeados</li>
<li>É verificado o status do cache (hit, miss, updating, bypass)</li>
<li>É calculado a banda utilizada (do usuário pra edge, da edge pra origem, da origem de volta pra edge, e da edge pro usuário)</li>
<li>Dados são agrupados por cliente, produto e janela de tempo</li>
</ul>
</section>
<section>
<h2>Batch Processing</h2>
<pre><code>
#!/usr/bin/env python
# -*- encoding: utf-8 -*-o
import collections
def default_data():
return {
'requests': 0,
'user_to_edge': 0,
'edge_to_origin': 0,
'origin_to_edge': 0,
'edge_to_user': 0,
'client': '',
'time': ''
}
def to_bytes(string):
value = 0
for val1 in string.split(','):
for val2 in val1.split(':'):
if val2 != '-':
value += int(val2)
return value
def main():
clients = collections.defaultdict(default_data)
with open('v5sample1.log') as content:
for line in content:
line = line.split('\t')
time = line[1]
client_id = line[2]
client = client_id + time[:16]
upstream_cache_status = line[10].lower()
upstream_bytes_received = to_bytes(line[14])
request_length = to_bytes(line[21])
bytes_sent = to_bytes(line[22])
clients[client]['requests'] += 1
clients[client]['user_to_edge'] += request_length
clients[client]['edge_to_origin'] += request_length \
if upstream_cache_status not in ['hit', 'updating'] \
else 0
clients[client]['origin_to_edge'] += upstream_bytes_received
clients[client]['edge_to_user'] += bytes_sent
clients[client]['client'] = client_id
clients[client]['time'] = 'at11_{min}'.format(min=time[14:16])
if __name__ == "__main__":
main()
</code></pre>
</section>
<section>
<h2>Batch Processing</h2>
<p>Ou seja, se pegássemos cada arquivo e processássemos
linha por linha, teríamos um batch processing.</p>
</section>
<section>
<h2>Batch Processing</h2>
<p>Problemas:</p>
<ul>
<li class="fragment">Os logs não param.</li>
<li class="fragment">São ~25 arquivos/segundo gerados pela CDN. <span class="fragment">(Só HTTP)</span></li>
<li class="fragment">Cada arquivo é de uma máquina, não o agregado geral.</li>
<li class="fragment">"Event time" vs "Processing Time"</li>
</ul>
</section>
</section>
<section>
<section>
<h2>"Os logs não param"</h2>
<img src="_images/unclephil-nonstop.gif" height="200px" style="float:right" alt="The logs never stop"/>
<p class="fragment">Kafka</p>
<img class="fragment" src="_images/unclephil-kafka.png" alt="A simple representation of Kafka" />
</section>
<section>
<p>"São ~25 arquivos/segundo gerados pela CDN."</p>
<p>"Cada arquivo é de uma máquina, não o agregado geral."</p>
<img class="fragment" src="_images/unclephil-victory.gif" alt="Kafka provides solution for those two too" />
</section>
<section>
<p>Se estamos processando os logs continuamente...</p>
<img class="fragment" src="_images/unclephil-continuous.gif" alt="" />
<h2 class="fragment">STREAM PROCESSING!</h2>
</section>
</section>
<section>
<section>
<h2>"Event time" vs "Processing Time"</h2>
<img src="_images/unclephil-time.jpg" height="300px" alt="Taking about Event time vs Processing Time"/>
</section>
<section>
<h2>"Event time" vs "Processing Time"</h2>
<ul>
<li class="fragment">Cada máquina está gerando logs no seu ritmo (máquinas menos acessadas geral logs mais devagar)</li>
<li class="fragment">Monkeys collect aguarda ter ~2000 linhas para enviar para o Kafka</li>
<li class="fragment">Mensagens tem que chegar ao Kafka e então serem processadas</li>
<li class="fragment"><code>v5 2017-10-17T11:09:59+00:00</code></li>
</ul>
</section>
<section>
<h2>"Event time" vs "Processing Time"</h2>
<p>Processing time: A hora que o evento foi <i>processado</i>.</p>
<p>Event time: A hora que o event foi <i>gerado</i>.</p>
</section>
</section>
<section>
<section>
<h2>Flink</h2>
<img src="_images/unclephil-kafkaflink.png" alt="A representation of flink consuming kafka events"/>
</section>
<section>
<h2>Flink</h2>
<ul>
<li>Window: janela de tempo, com um período definido (1h, 1min).</li>
<li>Watermark: tempo a partir da criação da janela para começar a despejar os dados.</li>
<li>Lateness: tempo em que a janela permanece em memória.</li>
<li>Late arrivals: eventos que surgem depois que a janela foi removida.</li>
</ul>
</section>
<section data-transition="fade">
<h2>Flink</h2>
<img src="_images/unclephil-step0.png" alt=""/>
</section>
<section data-transition="fade">
<h2>Flink</h2>
<img src="_images/unclephil-step1.png" alt=""/>
</section>
<section data-transition="fade">
<h2>Flink</h2>
<img src="_images/unclephil-step2.png" alt=""/>
</section>
<section data-transition="fade">
<h2>Flink</h2>
<img src="_images/unclephil-step3.png" alt=""/>
</section>
<section data-transition="fade">
<h2>Flink</h2>
<img src="_images/unclephil-step4.png" alt=""/>
<img class="fragment" src="_images/unclephil-out.gif" alt="" style="float:right" width="240px"/>
</section>
<section data-transition="fade">
<h2>Flink</h2>
<img src="_images/unclephil-step5.png" alt=""/>
</section>
<section data-transition="fade">
<h2>Flink</h2>
<img src="_images/unclephil-step6.png" alt=""/>
</section>
<section data-transition="fade">
<h2>Flink</h2>
<img src="_images/unclephil-step7.png" alt=""/>
<p class="fragment" style="float:right">
<img src="_images/unclephil-out.gif" width="240px" alt=""/>
<br/>
<img src="_images/unclephil-out.gif" width="240px" alt=""/>
</p>
</section>
<section data-transition="fade">
<h2>Flink</h2>
<img src="_images/unclephil-step8.png" alt=""/>
</section>
<section data-transition="fade">
<h2>Flink</h2>
<img src="_images/unclephil-step9.png" alt=""/>
</section>
<section data-transition="fade">
<h2>Flink</h2>
<img src="_images/unclephil-step10.png" alt=""/>
</section>
<section>
<h2>Flink</h2>
<p>"One more thing..."</p>
<img src="_images/unclephil-onemorething.jpg" alt=""/>
</section>
<section>
<h2>Flink</h2>
<p>
Quando os registros são passados para a janela, eles são
agrupados (ao invés de conter todos os registros que se
encaixam na janela, somente os valores agrupados
são guardados).
</p>
</section>
</section>
<section>
<section>
<h2>Mr Banks</h2>
<img src="_images/unclephil-mrbanks.gif" alt=""/>
</section>
<section>
<h2>Mr Banks</h2>
<pre><code>source
.filter(new Selector(processor)).name(s"Selecting ${processor * 100}% messages")
.process(new ProcessMessages(brokenMessageTag)).name("Message Processor")
.flatMap(new MessageSpliter).name("Get Logs") // get the lines in the message
.filter(new LogBrokenFilter).name("Remove broken logs")
.filter(new MissingClientFilter).name("Remove logs without clients")
.flatMap(new MetricExtractor).name("Create metrics")
.assignTimestampsAndWatermarks(new MetricTimestampAndWatermarks(watermarkTime)).name("Watermark")
.keyBy(_.key)
.window(TumblingEventTimeWindows.of(windowTime))
.allowedLateness(latenessTime)
.sideOutputLateData(lateMessageTag)
.reduce(new MetricReducer(), new MetricWindowTimeMatcher()).name("Group metrics")
</code></pre>
</section>
<section>
<h2>Mr Banks</h2>
<pre><code>.filter(new Selector(processor)).name(s"Selecting ${processor * 100}% messages")</code></pre>
</section>
<section>
<h2>Mr Banks</h2>
<pre><code>.process(new ProcessMessages(brokenMessageTag)).name("Message Processor")</code></pre>
</section>
<section>
<h2>Mr Banks</h2>
<pre><code>.flatMap(new MessageSpliter).name("Get Logs")</code></pre>
</section>
<section>
<h2>Mr Banks</h2>
<pre><code>.filter(new LogBrokenFilter).name("Remove broken logs")</code></pre>
</section>
<section>
<h2>Mr Banks</h2>
<pre><code>.filter(new MissingClientFilter).name("Remove logs without clients")</code></pre>
</section>
<section>
<h2>Mr Banks</h2>
<pre><code>.flatMap(new MetricExtractor).name("Create metrics")</code></pre>
</section>
<section>
<h2>Mr Banks</h2>
<pre><code>.assignTimestampsAndWatermarks(new MetricTimestampAndWatermarks(watermarkTime)).name("Watermark")</code></pre>
</section>
<section>
<h2>Mr Banks</h2>
<pre><code>.keyBy(_.key)</code></pre>
</section>
<section>
<h2>Mr Banks</h2>
<pre><code>.window(TumblingEventTimeWindows.of(windowTime))</code></pre>
</section>
<section>
<h2>Mr Banks</h2>
<pre><code>.allowedLateness(latenessTime)</code></pre>
</section>
<section>
<h2>Mr Banks</h2>
<pre><code>.sideOutputLateData(lateMessageTag)</code></pre>
</section>
<section>
<h2>Mr Banks</h2>
<pre><code>.reduce(new MetricReducer(), new MetricWindowTimeMatcher()).name("Group metrics")</code></pre>
</section>
<section>
<h2>Mr Banks</h2>
<img src="_images/unclephil-flinkpipeline.png" alt=""/>
</section>
<section>
<h2>Mr Banks</h2>
<img src="_images/unclephil-flinkpipelinereal.png" alt=""/>
</section>
</section>
<section data-background='_images/thats-all-folks.jpg'>
<section>
<h1 class="fragment semi-opaque">Perguntas?</h1>
</section>
</section>
</div>
</div>
<script src="reveal.js/lib/js/head.min.js"></script>
<script src="reveal.js/js/reveal.js"></script>
<script>
// Full list of configuration options available at:
// https://github.com/hakimel/reveal.js#configuration
Reveal.initialize({
controls: true,
progress: true,
history: true,
center: true,
// showNotes: true,
transition: 'slide', // none/fade/slide/convex/concave/zoom
// Optional reveal.js plugins
dependencies: [
{ src: 'reveal.js/lib/js/classList.js', condition: function() { return !document.body.classList; } },
{ src: 'reveal.js/plugin/markdown/marked.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } },
{ src: 'reveal.js/plugin/markdown/markdown.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } },
{ src: 'reveal.js/plugin/highlight/highlight.js', async: true, callback: function() { hljs.initHighlightingOnLoad(); } },
{ src: 'reveal.js/plugin/zoom-js/zoom.js', async: true },
{ src: 'reveal.js/plugin/notes/notes.js', async: true }
]
});
</script>
</body>
</html>