||5 years ago|
|src/main/java/net/juliobiason/flink||5 years ago|
|.gitignore||5 years ago|
|README.md||5 years ago|
|build.sbt||5 years ago|
An example of Flink Windows and Watermarks
This is a simple example of how Flink deals with Windows and Watermarks. The magic trick here is that there isn't anything programmed to run, you'll have to input all elements by yourself -- and then you'll see what happens.
For this, you'll need two terminal windows: One for your input and another to see the result of Flink.
On the "input" window, run
nc -l 60000; this will start
NetCat on port 60000 on your machine.
On the "output" window, run
sbt "run --hostname 127.0.0.1 --port 60000"; this
will compile the project and start Flink minicluster.
You can pick any port you want, as long as you run both commands with the same value.
SocketFlink will output (on the "output" window) some internal events. You can get more information about those by reading the source code; each event type is based on a Flink function (an operation done in the pipeline) and before each function there is a deeper explanation on what the function does.
The list of event types is:
- SocketFlink interpreted the value you entered and created an event to be processed.
<dt>WATERMARK</dt> <dd>The just created event forced the watermark to be moved. In this example, the watermark will move every time it sees an event that is most recent than any previous one.</dd> <dt>REDUCE</dt> <dd>SocketFlink will group duplicated words in a single element. When this happens, a REDUCE event will appear.</dd> <dt>FIRING</dt> <dd>Flink decided it was time for the event to be expelled from a window.</dd> <dt>SINK</dt> <dd>The fired element in the previous event type reached the sink.</dd>
An example run
||Exactly like before, but this time there was no need to move the watermark (because 2 is not more recent than 2)|
||No magic here, same as before; the thing to note, though, is that the watermark was moved to the second 10, which puts the previous element before it but there is no firing and sinking.|
||Again, no magic. The only thing of note here is that, because our windows are 30 seconds long, this is the last position possible inside the window.|
||No grouping at all: we moved out of the first window AND created a new element (if this was
||Lots going on here: First, when the element is created, it moves the watermark to beyond the start of the previous window. So now the elements in the first window need to be fired and they reach the sink. This is the important part here: the watermark only affects with WINDOWS, not individual elements.|
There is no license for this code. Use it at your own risk. If it breaks, it's your own fault and your own responsability to put it back together. Not suitable for children. Continuous use may cause blindness.