Challenges Productionizing Apache Spark Workloads

hi everyone I'm Jaganath way I'm a technical data science practice that's India so today I will be talking about no more struggle with about a sparkman spice Park in in production so I will say my experience that I have worked with almost four to five years in spark and Scala and Python so about me I'm a as I said I'm are taking a lead in data science lead 8xc own labs and prior to that I worked with a nozzle against an excel awkward position I did MS in computer science from Qudsi University in India and this is agenda what we are going to talk about so I will take the context by introducing the primary data structure of spark I already did a frame and data side and then I'll talk about the pragmatics explanation of executed course container state job and tasking spark and I'll talk about the difference between JDBC read that you do and that is parallel or not I will talk about how you can do parallel and purely concurrent JBC right in spark and I will talk about how you can avoid the unnecessary software in joint strategy and after that I will talk about some of the no one is using spark which you cannot do and then I will talk at the end about power of chaos and Delta which just open source one and half month ago so coming to the point yeah so how about para-sports about this park is the unified data processing firm work till now but now it has got edit with the machine learning in AI so you can do spot course box equal you got spark graphics and you go to spark up ml deep so what is oddity oddity is the primary abstraction on the spark called resilient distributed data say to be sketch split it across the cluster and get the parallel execute it so you can apply your lambda function on each split and get processed data it is resilient in immutable means you cannot change thus the format and the data of the RTD because that's how it will be said safe and you can create the compile type safety and strongly type inference with this path means you can create the oddity of integer oddity of string and any primitive type of data that you want it has lazy evolution so when you create your transformation it generate the directed acyclic graph and from that standard executions so saying that if you have the analysis error when you execute your action it will get accepts and means if you are trying to read the file and that file has a wrong format of file doesn't exist those exception you will get at the when you call the X and X 0 so SPARC got so many operations like transformation and X n so spa as a library has support of widely all the functions and available theory like statistical and set theory and you can have the lazy evaluated axons that you can see here so the question is that when to use the RTD ok so if you don't care about the control of data set and you knows that how it looks like you care about low-level API you can use the oddity if you don't care about available lambda function in this but dear sir you can use the oddity if you don't care or schema and the structure of the data you can use the oddity if you don't care about the optimization and performance that you can gain you can use the already and very slow on non jvm languages like Python and ass so far but what I will tell after this will tell you that how it can be faster with Python also and if you don't get about in Edward and inefficiencies you can use the oddities what does it mean if you can see this example at line number 14 so at the line number 15 what I am doing is that I am doing reduced by key so which does the software across the entire cluster and then I am saying the filter okay so the order of the function has not correct here if you can do filter first and then do the use by key you get only those data that you need and then you can suck all that okay so what are the structure API available in the spark reader frame and let us say it okay since data set has no supporting spice park because the dynamic programming nature of the Python and it is already been available with Python so I will skip that here apart from that the one difference that you can see in the RTD and the data frame is that data frame by default you get 200 number of partitions so if you have more number of partisan you can create more number of per ISM and concurrency in a spark so your performance you can boost it out so when you create the idly by default partition is 4 and when you have the data frame the different partition sizes to 200 so if you care about strongly typing ability to use powerful lambda functions if you take care about all function transformation you can use the data set right if you can see this structured API in Apache spark so the if you have syntax error or the notices error you can catch at the same time if you use the sparks equal but if you use the strongly typing you can get at the compile-time itself this is an example that you can read with a data frame here and same example if you use the sick wall you can use this equal here and it is supported on optimized by catalyst engine in spark that does mean you get data frame data set and sequel aft and they generate the unresolved logical plan and it will generate the physical plan and then it will execute the cost model and choose the best physical plan and then it will generate the high level RTD so this oddity then you can see at the end it is not exactly that I was talking about before this is highly object optimized JVM bytecode which executes on top of the JVM so how SPARC executes that so path path uses the Python uses the PI for J for the driver node apart from all the worker node they use the UNIX pipe to communicate with the JVM so this is one example that how you can have data flame optimization so if you use the data frame you get any plan of you medicine that you can see in this example so first you get the logical plan and then if you have to find one is you have parking events and second you have is the employee table so if you know that before you join you can filter it out the data you bring the only Gator that you need and then you join it so sparks supports the predicate push down and and the column pruning so whatever you need it it will join it the best physical plan for your query engine and this is one of the comparison of the data frame in already in terms of performance so if you see little search it is better insulation also so now I will talk about the project called project called Apogee cold outside so can you see the screen here so the library called colas how many people are of you are working with pandas I have [ __ ] with partners so pandas runs on single thread in the Python okay and it tries to fit everything on single set of data block but one in half month ago company called the company behind the spark data break says open source library called colas which is on spell early and there's no need to cast in between pandas dataframe to spark data frame vice-versa this supposed highly native spark EP ice and in terms of performance that you apply is very faster than pandas so in this example you can see here I got spark session then I have find us here then I importing me from data breaks import colas and then I'm creating this Fox session object and then I'm reading the one CSV file and I have time here the how much time it took and then I am using the colas just you need to replace the your PD of Jacob pandas with colas and everything works as it is okay and that data frame is supported natively with spark and then I am saying the how much is this ape so I got 51 500k those and 40 columns here and then I'm selecting this you can directly do when you cows means it can give you the product wise group by here which you can see here how many people of you are working with the one hot encoding when you have have someone tried one hot inking and encoding when you when you have categorical data and when you want to set the value zero one based on the categorical data so if you do hear this play call us don't get the MS final data and the column name product right you can see the one hot encoding here okay so it is supported in this way with the visual way here yeah coming back to the slide so after this I will talk about one more open source project which supports C so when you create the big data architect just what happened the spark is not only used for unified beginner processing but for applying MLA I because when you create your statistical model with a deep longing of tensorflow you think that same process data is available at your data length right so why can't you directly read from pocket why can't you there is it from ever other than generating something else and when you build your architects you understand that that a I and M and will be intricate part of your application or platform so there's no need to be a separate environment separate infrastructure and separate team but same people can work with same infrastructure so coming to the point it's just not only optimization of the library you know the author your DSL code but you have to also optimize the infrastructure so if you understand that the terminology and the internals of this path which helps you to write the code the way it likes so I talked about every internal things like job does mean the each transformation and the xzen that you apply on a spark would create each separate job in a spark stage means a set of tasks in each of which can run parallel using a thread pool executor and task wins the the lowest level of concurrent and parallel execution point here and each stage is split into number of partisan tasks right so each stage will have the data frame and data frame will have n number of partisans so the number of tasks that you can run on the cluster would be stage into number of partition in the stage so in this example you can see that this is a job active job and this is the stage here that you can see inside this stage you would have a number of tasks here and the number of tasks can run parallel you on different executor sites executors nothing but the different various JVM box that you can run here and those are what you are if you learn on top of the yarn or muscles yang will provide you control on Jason in each container will be virtual here and you can see here the parallel task execution in the yarn or spark UI so how it works in if you can see here that maximum allocation because is six year right and if you can see that active nodes are four so maximum allocation into active nodes that does mean total what you go that you got in the yarn that does mean if you see the conjugation that yarn door scheduler dot a location because is one right so minimum is one here you got the maximum is 6 okay so when you submit your spark job you provides the executors and co site so those calls are here if you can't figure in this way in a setting and then you tell when you submit your spark job here if you provide six which will cause for spark job then you can run for Spock job here because 24 by 6 for jobs if you provide what you'll cause one in spark submit that does mean you can run 24 spark jobs parallely right so you have to decide that we job is CPU intensive in job is in memory intensive and with job is a disk intensive also I mean general you you if you have some time and then you would like to pick the first row by sorting that right so if you do in this way which you can see here you sort it because when you do this thing you don't know which but if you want to make it consistent why is not consistent because the duplicate is not consistent across all the partisans you have got here data frame here and it has 400 Paulson is not consistent but if you want to just try and see that how you can make duplicate in this way and consistent you can do data frame not coalesce and provide one so generate only one per user and then apply the duplicate here but what we but you have you will get performance penalty here because if you create one party son you would not get so how you can get this problem solved without getting penalty of the first ones you can use the window function from this Park and you can do window track your poison by and order by update time and then you add one more column and provide your number our window specification where your number is one and drop that value so here this way you can get consistency of the first yes apart from that what kind of tuning can be done is hyper parameter and country person tuning so if you can see the here spark but speculation Falls suspect listen means that identify some of the tasks which are running too late and you start though start that does mean speculation and this is a dynamic location that doesn't mean you allocate resources to your spark job dynamically yeah that's it any questions