Python Containers: Best Practices

Okay, so it's 2 pm and we can start. Also, before we start, um just uh a message about the egg hunt cards, the ones that the first 50 people who came to this event uh got um just if you are not planning to use it, um then just please give to someone else because we still have a lot of uh prizes. So it would be uh um not good if we are not going to use it. And so for now for today's speaker we have uh Daniel who is a software engineer with experience in both local startups and global VC uh backed companies. And today he's going to present uh Python containers. So the floor is yours and you may start. Thank you. All right. Uh yeah. Hello everyone. Uh thanks for uh coming uh to the session after lunch. This would be around siesta time in my country. So you're welcome to take a nap. Um and today uh we're going to talk about uh containers and best practices uh regarding specifically Python containers. This talk is going to have a a bit of a different spin to uh maybe what I uh originally planned or what other uh talks regarding this topic have. Uh the reason is that uh well if you search for container best practices in uh like the docker documentation literally the first thing that you will get is uh multi-staging and stuff like that. uh this is uh good this is fine and uh you should know about this and it's uh tremendously useful one of the most uh uh useful practices and uh in general best uh uses you can make of uh of uh your container file but I feel like um I feel like it's been uh talked about enough and uh you can easily find information about this so I will have a kind of different spin to this uh I will try in a sort Socratic way where I will play devil's advocate for myself. You will see what I mean in just a minute to advocate for these good practices and not just u uh not just the practices themselves but why uh we need or why we would want to apply them. Uh right now docker file is writing a good docker file is very verbose. Uh this is one of the problems that I uh that I think we have in in this space. Uh I hope that someday it will kind of start going away and more things will start being built into the images or preset into the images so that we won't have to do so much container file manipulation to get to where we want to get at. But at this stage it is still uh very lengthy uh to have uh a clean uh container file. So what I will try to do is partly convince you of the reasons why you would want to do it and also maybe give you some uh tips from uh personal experience or just comment some things that you might not have heard uh elsewhere hopefully so that you can get something out of this that's not you know in the first result of Google search. Um so I have uh essentially two tips uh or two big uh uh topics that I that I want to comment. uh it's only two things but these are I think the two most fundamental uh things that you need to take into account when you are writing your your container files. The first one is applying the principle of lease privilege. This is a very uh a common uh uh phrase or a very common saying in cyber security, but it's also I guess a very common sense uh type of uh uh quote and uh a very common sense principle that you can and and should be using in your uh container definitions. Now, how does this translate into specifically uh container files? You would do this by using the user instruction. So uh you have your uh you would normally want to switch from a root user if you are using it to uh custom one that you have created with specific uh and scope down permissions. Now uh you might argue against this and you would have good reasons to uh this is the first uh character you will meet in this uh presentation. There are two of them. uh thinking emoji will provide uh sort of objections to the things that we will be saying and reasonable objections at that. Uh so this is yeah the devil's advocate and uh thinking emoji will tell us reasons why you might not want to do this. Uh so this is like the little devil on top of your shoulder that tells you why you shouldn't maybe do these things. And uh these objections are partly reasonable. Uh containers were supposed to be uh black boxes of some sort. They are isolated. You normally um don't really care about the underlying operating system because you might just uh send them over uh to be hosted or deployed in a cloud service. Uh so you don't necessarily care about the under underneath system if it leaks. Uh and they are on paper you could think well I don't really care about what's happening inside of the container. uh as long as it stays there uh it's not going to leak out and even if it does it shouldn't be that important. Well, the truth is and this is the second character you will meet uh the nerd emoji uh will provide kind of uh the let's say good per perspective on things and or why you should actually care about these things and take the time to write all of these very annoying lines that kind of bloat your file but uh you should do it anyway. So containers are isolated. Uh that is true to some degree, but it's very easy to leak things. Uh for example, you cannot count out uh on container runtime vulnerabilities. You cannot count out that uh something will be wrong with the kernel and you might have some sort of exploit. It happened before, it might happen again. Uh and you should sort of preemptively protect against those kind of things. Also, if you have um a if you're under a supply chain attack and I think in general we are uh we are playing with fire in in Python in many cases uh with this kind of thing and I think one a big one is probably bound to happen at some point. So if a supply chain attack happens and you install a malicious dependency, uh if that's installed with super user privileges or you have something that has access to all of the context, especially if you use mounts and you're not careful about what kind of volumes you're mounting, you might be well uh asking for trouble. And these malicious dependencies if they are scoped to the let's say if you deploy them within the scope of only your application code well it's still terrible but it's not as terrible as it would be if you just give them permission to anything. Um also you might be breaking some SLAs or something. So uh you know I'll call the police and you're lose your job. No I won't. Um so back to the user um command. So there are uh very broadly speaking there are two types of uh containers. The ones that provide you with an application ready to use let's say airflow uh the airflow container does have an airflow user uh that's preset in the container so that when you run that container you're using the airflow user that does not happen for Python containers or for containers that do not provide directly an application in the first place. The reason is that you are kind of expected to do this yourself. So let's say when you provide this container, you don't know what it's going to be used for and you don't know what scopes it will have in terms of what folders it will access or what you will use it for. So you are kind of expected to be um to create your own users and your own groups to be able to delimmit what your application is going to do in that context. That is not necessarily the case if uh you are using a container for a pre-bundled application that already has its own uh its own boundaries. But you are expected to do this if you deploy an application with uh Python container. So the way you will do this is uh like you see there uh this is uh ripped straight out of the also the the best practice uh documents and uh just to comment you you uh should uh well create the user and the group. It's uh considered by many people good practice if you also assign a UID. Uh you switch to it and you generally use that for all of the different uh operations that you will do that do not strictly require root. Those would be system application installs. Uh and by system application I mean uh package updates or dependencies that come from the operating system. For those you will need root and then what you should do is switch over to root do the operations that you need to do and then switch back uh with the user instruction to the user that you created previously. This is uh almost essential and uh yeah it's uh it's again a bit verbose and it kind of sucks that we have to do this but it's for good reason. So um for extra brownie points if only the user has permissions over the uh applications executable uh uh files that's perfect but at the very least this is the kind of the bare minimum. Uh another kind of tip I have or uh something that you might want to explore is podman. This is a sort of drop in docker replacement. It's uh it's gotten quite better uh the past uh few years. Before it used to be uh it used to not have very good developer experience. Some things would break uh some things were not uh uh as good. Things like networking were uh kind of slow. Now it's gone progressively better. And the trick about Potman or the magic about it is uh I don't want to take too long about this but just so you know Docker has um uh uh Don in the system. So when you start up your uh system if you're using Linux with systemd Docker will have a demon that has super user privileges and then uh the command line interface is a sock it's it connects to the socket for the demon and the demon is the one who actually runs your containers. Uh thus those containers have or are running under a super user name space. With Potman it works a bit differently. So a podman is a process that's started under your user. So under your local uh user that you have in your machine and when you spawn a container it forks off of that process with the same permissions that your user has. So uh this is a good um this is a good excuse to kind of exercise what kind of super user privileges you might be abusing in your applications. uh for example, if you're trying to uh create folders uh with root permissions, you're trying to access system files that you shouldn't be able to, this will not work initially and you will have to fix it. One very common thing that has happened to me thousands of times and I guess maybe to to you too is that you have uh you mount a volume in your uh file system and then you try to lead it for whatever reason and then you see that you don't have permission to do that and you have to use root to delete it. that if that happens it's not necessarily bad but it's a sign that you are uh kind of potentially abusing your super user privileges and uh you know root is creating files under your file system that should be your property let's say uh so you can give it a try uh it's uh it's it has for the most part feature parity to docker you might find quirks here and there but it's it has gotten quite better uh some people think that uh this is uh this has some other uh advantages uh to Docker rel regarding uh open source ability and things like that. That's not true. Docker is open source. So is Potman. They're controlled by different companies. Docker is by Docker. Potman is by Red Hat. So pick the one you like the least. But uh essentially uh they are kind of uh interchangeable for the most part. The second thing that I want to comment is that the smaller your container the better. This is this sounds like a very strong rule of thumb. Uh cuz it sounds very like maximalist. Uh do I always have to make things as small as possible? Well, it turns out that for the most part, yes. It sounds kind of uh uh uh too strong, but but it often ends up being the case. And thinking emoji has some opinions about this. Uh do you really need to care about this? So differences between containers will be in the order of megabytes. So it's not like you will have a container of 1 terabyte and then one with 5 megabytes. Uh besides transfer speeds are pretty high and storage is quite cheap nowadays. So what's the big deal with this and why should you care so much about reducing your container size? It seems like some sort of code golfing uh exercise applied to containerization. It turns out that that's not entirely the case. There are many good reasons why you would want to make your containers as small as you can. Uh the first and most important probably is that the smaller your container, the safer it is kind of by definition. You might have uh heard about the one of the latest exploits that happened uh with uh SSH uh where it had a very big vulnerability. There was another really big CVE with Cups, the printer uh system. Uh and if you're thinking, well, why should I have uh any of that in my container? That's exactly the point. You shouldn't. Uh you don't really need it. And if you can remove those kind of entry points for potential vulnerabilities, your container will by definition and by conception much safer uh and much more secure than one that's bloated with potential uh vulnerability entry points. So that's the most important one, but it's not the only one. Uh it also happens that even though differences in size between containers are relatively small, you get on one hand charged a lot for uh data transfer in many cloud services and especially if you're doing like evil things like uh um uh transferring between availability zones or you're using egress through an ad instead of a VPC endpoint stuff like that you the charges will rack up and you will start to notice um especially if you do things at scale at the same time it's um it's also faster and that also kind of translates to money at some point if your runners are uh if your CI/CD pipeline has to pull a container and uh that container is uh bigger may not matter a second in like one deploy a week but if you're doing multiple deploys uh a day then it might start to rack up and you might start to notice it. So uh it's actually uh more important than it would seem at at at first glance. And the way that you would uh reduce your container size is normally by the base image selection. I will speak uh first about the Python images because those are the ones that are more uh central to us. Uh if I have time, I will digress a little bit about other options you might want to consider. But for now, let's uh speak about the uh the ones that come from the official Docker Hub. So you have three. The first one is the base uh DBN image. Uh so it's the one that doesn't have any suffix. This is okay, but it's probably too bloated. If you read the the Docker Hub uh read and if you check out the description for this image, it will say that they recommend using this. I don't want to be presumptuous enough to like contradict what the people who build this actually say. Uh they they certainly know better than me, but I I think this is too much. I think you will probably not need many of the utilities that this one has. And I think you can for the most part use the slim version without any penalties. If you need something that you don't have in this base image, you can just install it and it's totally fine. It's much better to start with something very small and only install what you need than to start with something bloated and kind of uh also assume or uh rely upon utilities that you're not explicitly calling up also. Um uh also just the size is much bigger. Uh lastly uh the Alpine version is the most controversial of the three. Uh so Alpine uh I think everyone knows at this point but Alpine the the the trick with Alpine is that distribution that doesn't use uh lip uses muscle which is a different uh uh let's say C uh uh variant to to keep it simple it has historically been troublesome with Python but this is one of the things that I sort of want to or I will go a bit against uh the grain in this you will hear many like container experts and a lot of people who work very often with Python uh discourage the usage of alpine uh partly because of many of these caveats some do apply some don't but it has gotten quite way better with time so some software might not work I think this is getting rarer and rarer um it still might happen worst case scenario you can use g- compat but at that stage just don't use albin uh another reason why you might care about this is that the muscle memory allocator is slower uh than lip cy. So if you are doing a lot of operations like uh malo or string length or string copy in C this might actually hit you. It's not a massive performance difference if uh like for normal workloads and besides you shouldn't care that much about it for many regular use cases but it is a fact. This is true and you can benchmark it and it comes out. It still kind of is a bit the case. It's gotten quite better. Uh by the way, so it has improved a lot. 5 years ago or something it was very noticeable. Now uh not so much. Uh lastly, and this is the most important point, some wheels may not be available for Albin. So what happens is that you might install a package that has some uh uh C inside of it and uh the wheel might not be available. So you will have to compile that package when you pull it. This is actually one legitimate reason why you might not want to use this distribution. It's much better to use uh to just grab the wheel uh faster pull times. You don't have to compile anything and uh it's much quicker. So these are all legitimate. However, uh for the most part, it works okay in all other cases and these are starting to become more and more corner cases and less applicable to many um to to many let's say regular use cases. So what you see on the bottom left uh it was uh well known that for some time Alpine had a lot of problems with DNS. A lot of people using Kubernetes uh really complain about this. It's now gotten almost fixed uh or even completely fixed. I I haven't found anything particularly troublesome when using it myself. Uh this is a relatively recent change. Uh the the message you see there is two years ago, but uh there's one below where uh the version in which uh the change got applied to muscle uh sorry to to Alpine which bundled the muscle version was uh kind of a year ago or so. So it's quite a recent change but it is steadily getting better. The availability for wheels is also getting much better. So you will find now that many common packages bundle wheels for uh for muscle. And what you see on the bottom right is a list of dependencies of one of the projects that I uh use in production with an alpin image. And you might think well no [ __ ] there's nothing there. Uh you are basically using an HTTP client and the AWS library and nothing else. But that is kind of the point. uh unless you're doing something that's very Cintensive uh even if you are in some cases but unless you're doing that something uh some heavy data analytics or other very specialized kind of work I think that for regular web development or uh just a generic Python usage al is at a point where you can reliably use it and it is much lighter so it has some caveats and you should take them into account but I feel that people are overly negative from feelings carried from many years ago And we've come to a point where I think it's pretty useful for normal use cases. All right. And to finish, uh this is the kind of last thing I I want to cover. Uh because this is another thing that I think doesn't get just commented uh enough. It's kind of assumed that some people do it, some people don't. Uh updating the base image, should you do it? Uh well, the answer is yes, but the reasoning behind it is uh like there's an argument against it. uh thinking emoji here has a lot of very good points. These are all very legitimate. So uh what I'm referring to in case you you don't visualize it is just calling your package manager saying a update and a upgrade and uh upgrading the packages in your container file definition. There are as you see legitimate reasons against it. It makes your build times longer. It might bust the cache. uh very briefly about the docker caching in case you don't know the the the instructions that come uh uh below a certain point just get thrown out if one of the ones above change that's a simplistic uh explanation but one that suffices for for this use case uh you lose build uh at importency this what do I mean by this like fancy word uh essentially you cannot guarantee when you do this that you will get the same container file in the same code and you will get the same container as an output so you don't have something that will be reliably rebuildable every single time. If you don't do anything that modifies the base operating system and you don't modify the code in your file, the output that you will get, you can more or less count on it being the same. So you will get the same container build. If you do update the package, you are kind of breaking that because you're changing the underlying operating system. How troublesome this is in real life scenarios, I don't think it's very much, but it is something to take into account. And lastly, you can argue and this is a good argument also that the responsibility for providing secure images should lie upon the people who provide those images and not be kind of delegated down to you to keep it updated. It's uh it's a good point but unfortunately that's not the way it is and that you cannot continuously be pushing images uh for every single package update that the image bundles. It would be very impractical. So uh the counterargument is it's very plain and you see it right there. I took this screenshot just the other day, the 313 Python image. Uh, just grabbed it. Uh, no cherry picking here. I just grabbed the latest image and snipped it. As you can see, it has three high vulnerabilities, three medium vulnerabilities, and 144 low vulnerabilities. Now, it is true that some of those might not be fixed already. It is true that um uh some of these might be let's say pre-nown to the release of the image and not all of them will have fixes and it's not a guarantee that all of them will be fixed at this point even however I can almost guarantee that a chunk of those will be solved if you update your packages and you will have a very much lower surface area for errors when you run uh those operations. So just based on the fact that it's a much more secure practice even if you are losing all the things that I mentioned above I think it's very worth it to uh to run this and it should also maybe be something that shouldn't even be uh uh relied upon for you to do it and maybe done some sort of automatic way but that's digressing for now. Uh you should be updating your base image. I think it's a very strong argument uh in view of all the weaknesses that the container can have. So that's my opinion. I think it's a solid one. And that's all I have for uh for today. Basically, I have a one last uh um uh one last slide where uh in case you didn't knew about these kind of commands. Uh you can just read them by yourself. I'm about to go over time so I won't take you longer. Uh so many of you will already know about this. If you don't, uh I guess now you know. Uh this is one of the things that you can surely find uh if you just uh search but uh you can also find a lot of uh kind of sub-optimal stuff. So I think this is uh uh these ones are quite good and quite safe to include in more or less all the um all the containers that you use. So uh yeah, that's it. That's what I have for today. Thank you. [Applause]