Geodata Processing with Python and JupyterHub

okay thank you for attending in the PI data track I'm talking about geo data processing using Python and Jupiter hub actually I made a type I wanted to write Jupiter lab and I ended up writing Jupiter hub but but no problem because I'm actually using choppy Jupiter app for the same thing so I will talk a little bit about Jupiter Neptune the slides and also a to Peter notebook is available on github the link is here I will also put the link on slack and are already posted on Twitter so you can copy it right through so what is Jupiter help to peter help is basically a server providing jupiter notebooks here is an example of a login screen so i log in first and then something like this shows up I have a chippy to help using Jupiter lab so I can choose different kernels here in this case I have PI sin 3 and armed and I will show during the talk you can also create our own Carlos kernels with specific modules on it and then you can open your data and and all or your to peter notebook files are stored on the server may ask quickly the audience who have you already used to Peter hub before so not too many so I can't really recommend it if you have a set up and you have a good server and that's the point we recently got HPE Apollo 6500 it's extremely expensive this server running with 48 cores and 192 GB of RAM and we also have four and we the Tesla V 100 it's the sm2 version it can be upgraded to eight of these graphics card but for the first tests we only use four and we can do many things with the application deep learning of course and it's it's really really fast with what five over five thousand cuda cores per chip you it's it's astonishing that the drawback of course you see it three hundred watts per GPU so actually we are we are using the heat from the server to heat our building so it's not that bad so let me come to the main topic processing geo data using open source modules I call it open source modules it's very important for me that it's it's open source but what is geo datum geographic data is defined in a iso standard technique committed to eleven series and they do with many documents but the most important part is about ninety percent of government source data as a spatial information or a location component so actually almost all data has location inside and there are many geographic information systems GIS available today and two are special because they support python one is the commercial arc is from ESRI and the second is the open source qjs and you can create modules you can program PI's in there and you can do plugins or direct programming Python in that but today we will not look at J software today we do everything in Jupiter notebooks so let me start what kind of geospatial data I'm talking about usually I make four categories of geospatial data we have vector data you know that for example here we have footprints of buildings we have a raster data today we have I'm from Switzerland by the way and in Switzerland we have at the moment ten centimeter per pixel resolution or two photos of the whole country another category are point clouds and the fourth category would be 3d objects however I'm today only talking about where and roster datum and basically there are two very important libraries for raster and vector data one is Cheadle OG arm Cheadle is the geospatial data abstraction library created by frank for madam in early 2000s and it supports many many different formats there is a Python binding but the problem of the price and binding is it's really C++ like so it's not really pythonic and for that map box created a module called wisteria with which has a really pythonic interface we can redraw so that we look at that in an instant for regular data processing there is gos geometry engine open source it's a C++ library tool and there are also pythonic bindings most important is Shapley we will look at that in an instant too and altogether fire on a shapely build geo pandas geo pandas is basically pandas with support of geographic information so we will look at many modules today I leave that slide let me first show you volume volume is a very easy to use module to just write a map in basically two lines of code okay three lines if you can't import you have a map behind the data we see Open Street Map and we can make a center just actually here we are in this building at the moment and you see location here specified and the zoom level and now let's try to add some vector data to that map before I'm adding that I have to tell you how vector data is defined there are basically there are some standards like the open G's simple feature standard or the OTC zip open C simple feature and open Oh GCC per feature access standard and they define point line strings polygons also polygons with holes how they're done there are many other things I'm not covering here like circles or arcs and so on and the interface is quite easy with this well-known text you just create a point and you make the components inside for example point 10 20 that defines a point in well-known text or we define a multi polygon that's two polygons also with with different the first and the last are the same coordinates it's closed always and the second polygon vertices first and last is the same two so this one defines two polygons and this well-known text can be used more or less directly in Python using shape way we import a geometry and use the point and polygons or multi-point and so on the different standards and you can't just create polygons here with the coordinates you can even display it immediately in the chapito notebook and look at it if you have it gave it as a string like this point 10:28 in a string inside you can just use Shapley wkt load s load string and then you have the polygon inside so I know that a polygon of this way Nia yesterday I found one I googled it actually and found one I will show you after where you find these data sets and then you can just load it from a file as Jim Jason as a cute resin is a different format also containing coordinates polygons it's same principle it's just a different syntax so you have two polygons Multi polygon string line string and so on and then it's coordinates and you can just read it and create a shape using shapely and put it on a map like volume so you see in four lines of code we can put this polygons inside to just use volume futurism and add it to the map okay I will show that life later so you saw some coordinates but how our coordinates defined and that's a really different case usually be used geographic coordinates we have longitude and latitude latitude is the coordinate around let me say y-axis and the longitude is along the x-axis and we see if this so called graticule on a sphere or projected and this would look if you look at a projected earth it would look like this we see we try to reduce the problem from a 3d problem to a 2d problem which also makes it easier but on the other Sun hand we don't have a sphere is not a sphere earth is something like potato and the problem is we have to approximate that and if you take a sphere we are really far off we can if you go to the next and with an airplane and they would take the sphere model I think you would not arrive at the correct approach it would be 60 meters at least wrong and that's not good obviously so we approximate using an ellipsoid and this ellipsoid was defined in 1984 so it's called wgs84 it's a world geodetic system 1984 and the semi-minor axis from the pole to the center is this long and the other ones I mean my axis is this long so we can even calculate the flattening and many other things I'm not going into details with these geodetic systems we also have a second problem we can project the Earth's in many different ways I just put three map projections here Mercator Eckhart Cassini there are hundreds more projections and that's something we have to look at so there is a module called pipe Roche unfortunately due to time reasons we will no not look into map projections too much today but with this pipe Roche we can really use calculate transform from one coordinate system to the other and there are two basically two very important systems one is the two eccentric Cartesian system it's just a Cartesian system X Y Zed where we have the earth but in general if we do these visualizations we use two DS of use a projection system and this projection system wgs84 is you usually they take the epsg code is the european petroleum survey group they have for every projection system a code and the 4326 is for WCS 86 so let's look at this example we are located at this position 5425 you see the problem so what what is 54 the latitude so it's here 54 up and 25 would be around here but you see what I did I took the y-coordinate first and then x-coordinate that's a big problem usually some people say its first latitude and longitude also people say its first longitude and latitude and we even have problems in some price and modules like Shapley for example and Shapley you should lose the XY thing and not yours around so let me show that in a in a demo let me open the the jupiter notebook let's go to the life things now this tube topic can be downloaded on github if you want here are the installation instructions are not going into it also here it's an installation instruction if you want to create your own curl or just with these modules so you can't just for example change colonel and have your own Colonels here okay so let me test if I have to write colonel activate it and I do this volume we see the map is just loaded and in the center that's at the moment the center of the world we are located here and you see again the coordinate first is the latitude and the longitude so in volume it's the lat/long convention okay so let me make it a little bit bigger so you can see it from far behind we can create polygons on the map let me first load this geo chosen polygon from before mr. Chasen load F load string and I can display it as well-known text you will see if you know this way near the border there are not really many points so it's really really simplified it's just for display reasons after we will have one which is much better and I can't display it on a map you see it's really just a couple of points so if you zoom in you see the borders really inaccurate here so but it doesn't matter it's a principle at the moment we see we can display polygons inside okay so I create this point of our location at the moment and we see this is our location at the moment for in geographic coordinates now Shapley has some binary operations contains intersects within touches crosses equals and you can see if for example if this Pike on LT that's our position is within the list when your polygon and we see that's true so we can really make a simple simple test here you may have heard that your pison this year is in Basel Switzerland that's my hometown so let me take the Convention Center point and let me test if this European is within Leishmania and you see no that's false so it's in a different polygon so it's really easy easy-to-use interface for really difficult tasks if you program that yourself you will see testing a point in the polygon it's not just a couple of lines curved so another module is car 2pi and as you saw i use volume for displaying the map with carter pi you can also use maps and display it in different projections and you see it uses map plop plip to display that and if you have really a large amount of data i recommend using this one map so we can just plot all your data inside you can also do some crazy things and change the projection type for example and have a map like this i have more examples in the notebook let me one duel on one last year the Mercator projection known from google maps and so on we see vista antarctic are really really huge and europe in a centrist okay it's a very dangerous map actually the proportions are really really way off so let's go to the next let's go to geo pandas i have a data set to see is 4k a 5k that's a data set CSV file containing all cities with more than 5,000 people living in it it's also some IRA's it's not only cities it's just populated places actually so it's utf-8 heck 8 encoded with a comma and it has no header and to avoid warnings I used a low memory this just pandas standard pandas and you see if I may display the first three we see a really horrible data set with many not number and things I remove some some columns now I just use the name latitude/longitude also the type so I can remove some some things later and the population you see the 5k is not really that strict so it's it's it's not it's not really the best dataset it comes from geo names or you can download it is open data set and it contains really many many cities and its population so I remove sections from the data set some errors which are not interesting for us and so we only have the most important data set inside so next I create the geo pandas dataframe at all before Japan does is basically the same like pandas but it has a geometry column inside and geometry column contains a shapely object so let me say please let me take the longitude latitude create a shapely column and create a generator frame out of it and you see there is a geometrical now and with the well-known text here so we have points mainly I removed the latitude longitude from from the data set we don't need it anymore it's redundant and I can't just take this new data frame and plotted call a green marker size 15 and if I plot it you see all cities on the world with marker 15 and see some areas are really dense and some are not really dense so that's you see the shape of the earth just with with with cities so let's take this our our location again as example and create a column with distance so I want to make every city in this world to distance from this place here exactly this place let me actually let me display it how it looks that's a Geo series this it's just for every for every row in the data set it's the distance in the be careful its degree we are in geographic coordinates and I can create a new row I collect distance I copy this generate a frame and add a new row inside and we call them insert called distance and sort them ascending true and let me display it and here we see the distance from from here to the next cities of course the the next city is Williams because we are in Vilnius and then don't ask me to pronounce but probably you you know that and I guess it's correct or any objections so that's the first ten and I can create a mask now it's a suburb is inside toe it's it's not only City it's um district can be inside too in this in this data set you could filter it out by using the type you see here it's ppl a - there is a whole documentation what is what and you could filter that too okay so if we remember our lives away near polygon this really horrible polygon with only a few points so what I can do is I can create a mask with this I check if all the cities are within least wearing a polygon and create a new data frame I also do a little trick here I just want all places with more than 20,000 people so it's not too many so we can see that's all inside least Wayne not really into this Wayne it's actually inside this simplified polygon so it could be that an April city is inside you too okay I can use volume now I can do a combination I can say okay let me take a leak right markers inside this I just create a map and I use the pandas command apply that just calls of a function and for every row axis one and I can create markers out of it so if these few lines of codes code I can just display all these cities from before on this map and I also made in every marker I used a string with named population so I can click on such a marker and see population and name to see it okay how much time do I have okay so hurry up I do one more diamond it's it's I want to show some life data the USGS has a nice web service with earthquakes it's updated every minute so this is really more that's life I can get all the earthquakes miss magnitude higher than 2.5 I use this one dataset because there are really many many small earthquakes so let me say I just take the 2.5 ones from the last week up to now and download that just here is the euros you can to chasten and you I just download it here as earthquake dot geo Jason I write a binary because it's utf-8 it's I could do it otherwise and then I read it just just download it here I read it with two pandas directly and can open it and you see here we have some IDs magnitude over the penguin and and link and time stamp I also do something magic I just reduce it to two the most important parts time timestamp magnitude place and geometry and let me do histogram quickly that's basic pandas too and we see there is one really big one this must be new I didn't see it when I petted it this morning ok that's that's really it's horrible atheists ok so I can convert that the timestamp with the time zone so let's look when it was in UTC let me plot it first we see if we plot this data set it's not really nice so what I can do is I can read an author a shape file shape and I mentioned before is a standard for vector datum and I can read that and I can plot both are just saved axis object from the countries I just down just created and plot the earthquakes on it and we see I think it's visible on the projector here earthquakes from the last week I didn't I could create different size of markers for different magnitudes that would be a little bit longer code so you would have to plot for every magnitude that I didn't do it now it's actually usually a exercise for the students do that so I don't have it in my standard Jupiter notebook I don't want to show this because it's recorders I could see solutions oh no so we see this one 741 UTC zero this was 75 kilometers from Peru 8.0 these numbers are not really correct it could be less so it's because it's new and complicated thing but it seems really a really strong strong earthquake that took place okay so I think my time is up I didn't show everything but you can try it yourself you have the link to the notebook thank you for attending and I'm here for questions [Applause]