Fun with Histograms and the physt Library

I come from Prague and for a long time I was a physicist but about two years ago I switch I'm not ready I don't know whether it was a fortunate step or not I switch to data science and now I'm working in show Max which is a company that provides video streaming in all Africa you can find me on github or Twitter or the best way how to find me is to come in person to Prague where we have pie data meetups I'm one of the organizers so let us speak about histograms so probably all of you know what Instagram is but just to be on the same page I have some like quiz so do you think this is a histogram no it's not because it's a visual representation of some relationship between two variables typically if you have a bar plot of histogram it's just one variable what about this thing is this a histogram almost but from this probably statistical point of view the histogram should shoot a show distribution of some continuous variable well this is a categorical variable sorted in somewhat random order the order is not really random because it's in alphabetic but still from the there is no real like meaning to the alphabet and what about this it's almost a histogram but as you see it's not nice bars not these nice rectangles it's actually kernel density estimate which is very nice and very smooth but it's not histogram so what about this yeah finally this is a histogram and this is the thing I will speak about in my talk why histograms histograms are very nice because when you receive a bunch of data and you want to see very quickly what is like attributes of the data you just rate the histogram and you just know what roughly the distribution of this variable is and it's like really painless the good point which differs histograms for from for example the kernel density estimates is that they are precise and did that compact once you have the bins really fine then the number that is inside the certain bin is really like the exact number of occurrences that you had and it's compact if you want to house some how to export your histograms and or say them then really the only only numbers that you're saving are the bin counts and the edges while for example the kernel density estimate just needs all the other points beyond it and again I'm getting back to step one now you're not in the role of someone who is exploring now you're in the role of someone who is presently presenting your data and the good point is that almost everyone understands visually what Instagram is and you can show it even to people that are not statisticians or physicists or scientists but what is the situation with histogram in Python so if I come to some problem that involves computation in Python I typically start with the numpy library probably everyone does is there someone who does not use numpy here okay not a single hand so you are ashamed probably to admit it publicly so if I have numpy there is a nice function that is called histogram and I just call it and let's see what happens if I put the data which was some kind of distribution of AIDS of people of certain country called fake land so I receive a couple of two things the first thing is an umpire array that contains the the been contents like the frequencies of the occurrences that were find in each bin and the second thing is an array of of edges I mean like the the left and right edges of the bends that is nice but it's probably difficult to work with when you for example want to keep this relation between the two arrays and I don't know we want to modify them then if I want to plot histograms again like the most popular graphical library for visualization in Python is probably matplotlib so I use MATLAB that also has a nice function that is called hist that can produce a boat like this it's either immediately shown to the screen or written to some file or shown in your ipython notebook depending on how you use it and this function also returns something it duplicates the thing that numpy does so again the contents and the edges and also a list of some graphical objects that if I know correctly are the the rectangles that you typically see in the graph so again it's a couple of three arrays but I think that histograms deserve a bit more I think that they should receive some proper object oriented treatment I think that you remember what were the edges what were the values of edges between the bends it was something like 173 point six six five and such so probably I want the computer to provide me with nice bins if I ask him politely I think that it should be able to modify histograms so at new values or anyway just for them just to be dynamic and I think there should be more like boating options so that's how I started Feist this is a project that I started when I still was a physicist and unfortunately I had a lot of time or fortunately and so in my like joke time I was writing this library but since then luckily still I I still have time and the project is not that yet probably now I should just mention that I wanted to enhance matplotlib and I want to enhance numpy I'm working on them based on them and yeah I really like them and I was also partially motivated and inspired by a computing framework from high energy physics that is called truth there are some concepts about object oriented histograms there which I basically take from them speaking just another disclaimer there is a lot of nice histogram in libraries written in Python that you can find on the internet or you can find them linked from the project page which speaking of which you can find Feist on github or using this QR code or in or you can install it just using pipe unfortunately not condom because I'm not friendly with the setup how it is done so maybe later I don't have time for that so I will now speak briefly about the four four four things that I was missing with histograms and just described how thighs do stem wisdom so first of first of all the object representation of the histograms I try to mimic numpy a bit in in the way that if I call some function I receive something and I put the same arguments as I would put in the numpy histogram function I receive an object that contains answer that number I would do so in this if you want to make a one dimensional Instagram I have a function that is called h1 if I call it now again on the same data that I used previously with numpy I receive an object that is of type histogram 1d it's like you see the representation it shows how many bins there are what is the total number that is inside and any information about the type that is used for storing the values now it's a class so it probably has some attributes it has some methods so basically from if you want to get back to what numpy answers you can access the property frequencies that returns exactly what numpy did I have attribute that is called bins that return something different in principle Feist supports also bins that are not consecutive it's probably not the use case for this is not not very broad I think but still it supports so when you get bins from a histogram here you receive both both the left edges and right edges if you want to get something similar to numpy then you call numpy bins or just edges because it's an object you can store more information inside than just some bunch of numbers you can also store some metadata that is then useful when you want to plot plot the histogram for example so in here I store the name I store title and I store the axis names which these are three probably similar things but then if you plot this this histogram you see that the axis name is used for for the axis down in the exercise the title is used at the top of your plot and if you have for example multiple multiple things displayed then the the name is used in the legend of course histogram Inc is not just for one dimension and it's not for the one dimension in also in numpy so you can have more dimensions so let's say that to the eight numbers I also add some more more variables I add weights that are bit correlated and let's say intelligence which hopefully is not correlated with hate although I would be lucky if it's there is some positive correlation and then you can create histograms with multiple dimensions so for example the two dimensional histogram is called is created when calling the method H do you hear you see that another object that is of different class and but it behaves similarly to the histogram one D typically if you want to visually visualize 2d histogram you visualize it using a heat map which you show here and you will see also some more examples later you can create a 3d histogram with a three here I want to show also another thing that if you for example have a panda's data frame and you have properly named columns and then you create a histogram out of it the column names are automatically used for your axis so then if you plot different projections so that the axis are automatically used again it's and now the 3d class is not called histogram 3d it's an a generic histogram and the class that allows you more and more dimensions and 3d visualization is quite complicated unless you create some transparent voxels or use some sliders in your interactive way it's probably difficult to any way to visualize this so it's not supported in fiest anyway so if you have a really many many dimensions you can have as much as many as you want the method is called H and there is just one warning because I'm not using sparse arrays so you can imagine that if you multiply the numbers of bins in each of your directions then very very the the amount of memory grows exponentially with your with your dimensions if you want I've also added a experimental feature that's called a collection that keeps multiple histograms together with the same bins so then you can for example put them together or I don't know just many played with them and at the same time but it's experimental it if it's an object so you can do more things with them there is JSON sterilization of of the histograms if you write your histogram then and you have a face that is not more than about one year old then you can read the file and like like receive the object as it was defined and so if if you have like gigabytes of data and you produce histograms from from them and they are only a couple of kilobytes or megabytes then it's also maybe a good way how to how to send them further now if it's histograms are of course not just like objects from from the asset pythonic point of view they also are some kind of mathematical objects and they can be treated that way so you can for example you would be interested in adding to esta grams together of course you can do it if you have arrays and i'm pi but in here you can just call histogram 1 + histogram two and you receive a combination of them so if you add people from fake lanten tool and you have let's say distribution of AIDS in in some other country that's the union of them what do you think is does has multiplication of histograms have any meaning probably I'm not clear what that would mean so if you try to do that you'll receive error that it probably doesn't make sense of course you can hack it you can take the arrays out of the histogram and multiply them if you like you're really confident that you are doing the right thing so this the operators that are supported is additions of course you must have similar bins it doesn't sense to at histograms with different meaning so in such case of course you also receive an error error subtraction is supported with some warning because like negative frequencies is also some somehow hard to interpret and there the multiplication and division is possible by Skyler if you have a multi-dimensional histograms you can project them into lower dimensions so starting with our 3d histogram that contain two variables you can project them into two so now you have a 2d histogram that you can plot finally if you aim you call you can also project into for example one one dimension another thing that is quite similar to projection but it's not it's actually slicing the histograms and taking pieces out of them you can index the histograms almost like numpy also I think some kind of mass arrays is also supported but now there's there are some issues with that so I would just take two numbers and slices so if I just take a second slice in the first dimension of the 3d histogram I again receive the 2d histogram that I can plot now you see that it's a bit different and then the projection I can like now I'm I have two numbers in the index so I'm just receiving one dimensional histogram like this and I can also like do some funny funny slicing and this as you may imagine then I have slices in all three dimensions so I'm getting back again three histogram now this is like three numbers so it's means that I'm getting to a scalar and the scalar is not just the number it's also like the edges of the bends I've found that a bit more practical than receiving just one number and okay so now the sec topic was that I didn't much like the randomly looking numbers that were presented for me as the beam edges so first of all let's start it's just really oughta matic way which the numpy does and I just do each one without any further arguments and that's what worked I receive it's like the numbers of these numbers but you see that I'm I was not accessing the attribute bins but winning the Inc attribute that leads to to the internal implementation all all like bins along all of the axes are represented as beaming objects there are different Benning cob jects some of them are like really copying with numpy does some of them are not described like like this in the static way but are more dynamic and so on so in this case just again the random numbers but if i won't i won't i know for example that I want veins that are two and a half centimeters wide I called I call each one with two extra arguments one of them is like a string fixed width that's let's say a type of of the Benning and the other argument in this case means like what is the west of the of the the been so when I call it I receive another object that now is like prescribing prescribing like the way how the bins are computed not just two bends themselves and if I look at yeah so and you probably see it from the picture that the bins are aligned with the numbers like 160 and 180 or 200 there is an bit even bore more let's say automatic and probably pleasing way I don't know what width of the bin I want but I know that I want roughly 15 bins let's say it's for some reason I selected this number and so and I list let the computer decide what we'll do it will be I call this algorithm let's say human which is like it's human targeted and it will automatically find for me that like the ideal brainwaves that produces this this amount of mins roughly it should be five centimeters wide and there probably should be nineteen which is like really close to my original fifteen I can go even one step further and just call it human without any more more arguments and in such case it's like select some automatic number that is that is ideal for the number of now of values that you have there is some theory behind it that you can for example find in Wikipedia or numpy documentation it just specifies what is the best number and if I do this I get 10 mins 10 centimeters each that's that was like the nominal numbers but let's say that uh now I'm investigating the number of children each person in my country has you see that this histogram is looking pretty weird and that I have like strange gaps and that's because I have ten histograms for like seven different values which is probably not so fortunate and for this particle case of law integers I have special let's say algorithm that is called integer that like takes the bins centers them around the integers and automatically sets the width to month so now if you have this it's fine if you are having numbers like seven million then probably it's not the way to go because you will have seven million bins and you don't want that and another problem is that if I have a huge range for example I've data about the population of the can all countries in the world 200 numbers starting from a couple of hundreds and going to China's relation of one point something billion and if I want to put them in one histogram and again not specifying any further arguments I received ten bins that one of these bins contains almost any country in the world there is like six six countries that are medium-sized then maybe two over middle of even larger size and then I have India and China in one bin that is not fortunate again so that's why I have exponential exponential burning algorithm that automatically finds the edges that are placed regularly in the logarithmic scale so now this is getting better only the numbers are bit weird again if I want to if I want to have nice bin edges up to now there is no human exponential for that you need to really find your bin weights but again feist helps in that you can specify the number of bins and the range and then it's automatically automatically calculated for you in so in here the numbers are reasonable so that was about the bins and the third main topic is the dynamic behavior of of histograms histograms should not be read-only because let's say that again I have this little country of fake land and I have a nice distribution of eighths of the people there but then I move there and what happens I can either recalculate the histogram again from another array but what I can do is just add one value to my histogram so that's I'm using the left shift left bit shift operator for that or you can if you're not happy with these strange signs you can use a method called fell and you will see that the total number of values in the histogram grew by one and if you remember precisely what were the contents of the bins you see that one one then grew a bit okay that's that's fine but let's say that not me I'm not tall enough but 200 really tall basketball players move in and I try to fill them in the histogram that's easy I don't use fill but I use fill underscore N and I update the histogram but you see one problem I would expect the total number of values to me one hundred one thousand and two hundred and one but the number is a bit less and you probably spot the problem my range is too narrow and the histogram probably would make use of bit of widening this is possible if you if if your binning is set to adaptive which can be done either through the adaptive attribute of the histogram or just when you call the h1 function you can specified it as a keyboard argument then whenever you fill fill it again and you go outside of your range then the wrench is automatically like writing for you you can even start with no bins at all and have just geo values and then like create the histogram on the way yeah and that's another way how to do that is that you if you have to do adaptive histograms that have different ranges but the the ranges are comparable and the bins can be matched against each other you can add them and also then also the range is shifted if you're not satisfied it's just one funny funny like functionality if you think that you have too many bends you can like merge them together so that's then you can have less less bins like for example one bin that's probably the happiest because it's very very very precise and very meaningful seriously so that brings me to the last topic that is just just visual and there is not not so not this not so serious I tried to include some easy plotting in in in Feist and I try to make it a bit general enough for to support different floating libraries most of the functionality is developed for matplotlib but there are some like attempts to bring something to Vega and plotly and volume for some special data I would really nice it would be really nice to support at least one of the interactive libraries either Vega or plotly properly but I don't have enough power to do that so I will show you just a couple of mud pot lip visualizations so the typical if you want to plot something in Feist you just go you can either call the plot method of Instagram or you can use the plot as a proxy in a similar way how pandas does that and then if you take this proxy then you can call the different methods on that that will that will call the proper visualization type so if you want a bar plot you just call the bar bar method if you want a step step plot you go step if you want a scatter scatter a line and so on what you can do is you can show the values so you know I'm getting a bar plot that is the default option and what I'm adding values I change the color I specified line with I can and the error error lines that is that it's given some conditions make sense in this in this like if the the pin contents are really random randomly and probably normally distributed I'm not sure what is the exact condition but if these are fulfilled if it's really random process of generating the variable then these lights have some meaning I can show some some total statistics I can put 2d histograms again the default option is to have hit each map which can have lines don't have lines show the values you can change the color map that is used you can hide the bins that don't contain anything you can make the color map based on logarithmic scale and that's it as I say as I mentioned if you have more dimensions than 2 you're out of luck but you can always you can project or slice into 2d or 1d just just so that this is not just really only theoretical I just show you three simple examples how I really used Feist first this is a this is a project that we did in in one of the hackathons we had like gigabytes of different temperature measurements at different times different days of the year in different places in one of the cities in Czech Republic called Brno and what we were interested in is how the temperature develops during the day based on the months so that was like I think big in the end we created 40 histograms that we were projecting and slicing in which Feist really helped so you see that in January typically the temperature in in Brno is like zero degrees plus or minus something not depending on the era basically while in July the situation is completely different and you have like a huge peak during afternoon hours another thing was again I was not when I was a physicist I was not procrastinating all the time and I really produced some results with this and that was we had neutrons produced in in a beam line and we wanted to like somehow visualize how they how they occur on the surface of some of this beam line and this is where Phi's was helpful and yeah and the last thing I was I did for my friends I said like I chose collected all all the finishing times of people that participated in prac International Matins and like created some distributions from them and using Feist I would say blow like which was it properly and I found some strange interesting things like most people a lot of people wanted to be just below for hours and if you look at the finishing times you just see this pattern that like you have a peak building up towards it for hours and then like in the in the next minute there is a very sudden drop so that means that probably the people achieve their goals or the ones that did not maybe died or I don't know what and and that's basically all and so I wanted to show you that histograms can be represented as objects that they can have nice beam edges that they can be dynamic and then they should be easy to plot and this is the last thing I want to show you do you know what that is do you think it is valid Python syntax actually is and it's some unique coin unicode sign that shows this but unfortunately that's called bar chart and Instagram does not yet have its unique Unicode symbol I would be happy to have it so thank you so any questions