Data Pipelines: Slice and Dice Your Data

so hi everyone my name's Sonic I'm working the company called do stitches which is fully remote company behind twisting twist and yeah nice to see you all in Vilnius in this area and you know like for me literally it's very interesting country like first of all the big fans of basketball which I really like and then they have mill called Zeppelin AI and yeah it has AI in there so double everything like and then like we get in AI you know and as far as the knowledge reduced to any is one of the most like oldest European and like European language right and in death age they were so smart that invented psychics called AI so now they have like a thousand seven with AI side by Barry I anti I like a lot of them it's really cool by the way and I even have a small tip for start-up if you want to name your project make it on this Iranian so you will have AI by default it's really great yeah and like you know at that century they also knew what does mean AI because AI by itself like work means out there so it's probably they knew that is something like terrible and very hard so yeah but and AI it's almost about date approaching like you know and we are all together here because we are processing data and each day we are trying to simplify our life there's a processing way of how we're dealing with data but data could be kind of complex it could have some different issues and here I outline like many of them so basically data could be complex it sometimes it really hard to build build systems which can perform hard like big data structures also data could grow fast and you can have a really big amount of data also like data could change because everything in our real world is change and data could could or should be processed as quickly as possible so yeah and like today I can like propose you to dive into the route and like to try to see why we have these issues like dive to the core of these issues and for this I would like to propose you to solve a very simple task like let's try to read a log files and try to find some substring in each lock record it is a task like you can just read log file and then just check if some string in this log record but here is a problem like you can have a very big amount of data which it will be very hard to produce you can expend hours or days for this yeah and like we already made a very efficient algorithm which like if we don't if you consider that substring can draw record is like very small as the complexity of this algorithm will be om which already is very good like we can't make better we can treat file faster so how we can solve this and my proposal following like let's try to split big file into smaller ones just by chance and let's try to use for example threading or multi processing which we have in Python and yeah the solution will be also quite simple just put all log files passes to Q and Gen just read everything in parallel but image if you decided to calculate some analytics here in this case you need some global object and you need some locking system to prevent some bad stuff which could happen in this training so it's already a bit more complex task then you decided to like commit some other data to data base and you need another thread in current thread with another Q and it's like growing kind of the complexity grown kind of fast and like raise your hands who want to change something here like no much people yeah because the complexity of this kind of approach will grow exponentially and it's really hard to deal with this kind of coal also if you take overflow you can see like a lot of real issues yeah and it's really hard to manage them all so if some error happened in between you will have very bad situation because you need to repeat your code at the beginning and you will have some help stating in your database if you have pictures like this it's also a problem you can't scale more because you used everything what your like machine provides you so here like the full list of problems which you have like first of all we have global objects then it's not support fault tolerance it's not scalable it's really hard to build a complex system and it's really hard to maintain these complex systems and yeah we have another problem like data could be complex like data structure sometimes it really hard to produce in simple way so how we can use how we can solve this so solution is let's try to use queues so basically it's kind of popular approach right now especially on the website like on the web where we can produce some jobs on background and right now we have a lot of different solution like celery Reddy's Kafka ask us like choose you whatever you want but basically this is like the global service which can just help you to communicate with different processes it's like you stand alone queue servers and I like to explain this approach based on rails because it has a very simple data structure so basically imagery we have like protists which called client or producer we just push lock files to your rescue then we have another process will just read rescue and try to produce files one by one if we run work across multiple time we'll have a parallel and what more important like distributed system because like radius communicate with processes using network like and also we will have a full fault tolerance because if something happened there we can just return back our job to the queue so from the Zen side it will look like this we just push files to the radius and then read them using workers if you want to calculate some analytics here you need a database like pick whatever you want but yeah it's still like networking so this approach has some problems and first of all it's a performance problem because you introduced Network here and it's let's say a bit slower than like treading on processing then it's really hard to understand full image of your workers because now like worker it could be like we can compare a worker to micro-service the program with micro-service is not easy to build you know you need to write documentation set up everything and you know for workers it's really easy to create a new one because it's just matter of one decorator so you can generate like hundreds of thousands of them and then you will have a very complex graph with the cycles which is really hard to manage and work with so then we have another problem which is hard to make a change because imagery you have off worker and create a new one we change data structure and if you deploy this in a second you will you can have in queue all data and then new newly deployed worker will fail because he tries to produce it's all data so is this a very light station because in then in the queue you will have old and new data at the same time it's really hard to filter this and then we have an last like very bad situation and when worker could make a side effects or change global state like image we have functions like this it's actually like a worker or salary task we decided to create some analytics here and then we're trying to save this to database and call another worker so if something happened in between we will have situation when we commit to database call another worker but then something fail and worker trying to repeat this dance because all tolerance which we know and he will repeat this task multiple times so we will try to commit data to database multiple time and call another worker also multiple time with which call like data duplication and it's really hard to solve and we can solve this station even with transactions so here the list of Q's problems so basically we have a performance problem it's really hard to understand full picture of how data exchange it's really hard to apply new changes and we have a side effects so here next issue related to data so data could change and it's really important how we can manage this change so how to solve this so basically we have data pipelines today so data pipelines it's a way to connect your workers in the right way so to prevent some issues which we have with the workers and queues and first I want to tell you like the data pipelines it's not ETL so basically you can use data pipelines for make ETL but it's not only in like one way how to use data pipelines like basically detail it means extract transform and load and it's something like I know it may be MVC on the web it's just a concept and data pipelines let's say it's just you whisky so we can compare this stuff here is the same situation like data pipelines by itself it's a way to connect your workers in a serial way so output of an one worker will be input for another but yeah this is what Wikipedia say but I like to say that next element like next worker will be called only in the end of the current one like and we will have this handle gruff like it's like a tree and basically don't have any cycles so it's really easy to understand what's going next and we can very easily understand how data moving through our system and so it allows you to build very like good image of data processing graph so if we're back to our problem with the queue when we change the global state and my side effects if we try to implement data pipelines using queues you can see that we like move worker call to the end of this function and it's important also that in this case we don't call any workers inside other function so they live here so if something happened in between with don't call worker so it don't make any date duplications but we still have some problems with databases because we will try to duplicate data inside database we just for now it's a bit hard to solve using data pipelines so right now we have following tools which help will help you to build data pipelines in air flow and Luigi but again it's just a tools which would help you to like connect your workers in the right way to prevent some issues so in the end like we have following benefits and problems for data pipelines like we have a very good picture of how our data is changed and then we can easily apply change because we can just create a new branch in our tree it's really simple it's maybe not very simple in air flow and Luigi but it's really simple in terms of data pipelines but we have still some problems like performance problem and side-effects so how we can solve a side effects problem and here we have ETL it's basically like a concept like pattern which allows you to extract data or get data from any source you want then transform this data somehow and then love this data to some this basically means that you can solve like a lot of different tasks it's not about only about rocket data from one place to another I really don't like when someone tell you it because a using ATL you can solve a very big range of different tasks so from design side its Harry will look like this we have a tree with different workers and on the root step we have a function which extract data from souther service it can be like requests it can be database file common line whatever you want and then in the middle of this tree we trying to process data to change it somehow filter and only on the Leafs on the end like step of our branch we can't mean this data to database or any other global like object state whatever we want if we try to implement like ETL based on queues it will look like this so we trying to calculate some analytics then in one transaction we trying to call our workers and then like in the worker we make only one transaction to database so if something happened in function called loud database we can repeat this function and everything will be fine we don't duplicate any data so but it's kind of hard to implement such like concept based on queues so that's why like a year ago I start project called stairs and it's a way to implement like queues data pipeline ETL approach in a very simple way to like to have like a very efficient way to process data and like from design side it looks like this so basically you have a producer functions which allows you to extract data from any source then you push data to pipelines where you have like distributed parallel anywhere calculations of your data you can also put data directly to pipelines if you want and then in our pipelines we just call consumers then could change a global state and make side-effects so from a cop cop site it looks like this we just defined stairs up then find the producer which in our case just read a lock file just yeah it's log files Pattinson then we put this file log to pipelines and subscribe to this data some functions each function could be a worker or not this means that it will communicate with another function you know use queue or any other streaming services then we can call like another pipeline or we can call consumer which will be just a standalone worker which can then come in data to database so a consumer side will look like this so basically we can have like a simple consumer like this save to database we'll just insert something to database if it fail it will repeat it what once again so and we have consumer Samsonite consumer eater which can just generate your data in a separate process so for example you can have a neural network in a separate process and just consume data from your data pipelines is directly to your neural network so this is basically like example of how you can use it in a different range of tasks and like this pipelines is really easy to run so you need just one common pipelines run it will run one thread one process which we really easy to manage and deploy for example we can deploy this to AWS lambda it's really easy to do and still of you to do this by default and also you don't need like magic job scheduler which like air flow and which has so it's really easy to work with to start with and deploy so you can like engine should before like you can solve a lot of different tasks using data pipelines and ETL and you can check out like stairs example where I like show how tears could help in the different tasks like data scribe in teaching neural network make some analytics or whatever you want so and here like small example of how we are using data pipelines approach inside to do is so we have AI called Smart scheduled and we have a web server based on goal which then just put jobs events to our data pipeline through rescue and then we just call stairs consumer and then return back data using pops out so this kind of approach allows us to produce like thousand of like requests and it's like quite scalable because you can just call whatever you want like amount of workers for data pipelines so like in the end we have like following situation so if you try to use queues data pipelines tell approach you will have following benefits so basically it's a fault tolerance by default it's really scalable in distributed system you will have a very good picture of how your data change it's really easy to apply new changes and it helps to prevent side effects but we still have kind of important problem which called performance problem and it's really hard to us because we are using like use network and it's kind of yeah if you decided to just produce small records of blocks it will be kind of problem but I want to show to show you some very interesting examples regarding performance like image we have an image which will size like 600 by 600 it's like a lot like a lot of data and let's try to just push this image using a ready skew just push this images to the rails and you can see that we can like push 200 images per second which kind of not very fast but come on it's like a big image but then let's try to just rotate image and you can see that we can irritate only 500 images per second which means if you rotate your image more than 2 times radius will deliver you more jobs that you can produce so that means if you have a hard CPU calculation calculations or just another request to your database whatever you want the radius or another streaming service will provide you more jobs with which you can process but yeah again if you have like a lot of very small items like lock records and spent not much CPU calculation you will have kind of problem but you can solve this using stuff like this so you can use Park which also available in stairs and right now I experimental miss approach error and pison a map which we actually released soon in Python 3 8 and like using this approach I can send between two processes inside one machines about 300 thousand messages per second so if you have a very small items you can solve this by this approach so yeah like and in the end I want to say that if we will solve inter process communication in the right way we will solve a lot of problems because ya can just generate very simple thread and protest applications and it will be very easy to communicate between them inside one machine so yeah and you can check out stairs try to write pull requests try to use it and send me a link to like your project where you use stairs so yeah thank you very much you