I’r going to talk about the tool sets that I’ve been using for the last year and a half called Nicks and how it applies to the whole stack of packaging development deployment, basically like using in Python. We all know it’s a quite a depressing topic, but it’s getting better and one of the main things that I really really really hate about.
It is that we have the setup dot pipe that is dynamic and whatever you do, we have to run this dynamic script at least run the egg info command to get something out of it, and that’s why, for example, we don’t have dependencies on the pipe I And so on, and so on and for example, no J’s community has these J’s simple JSON file that is static. You write it down and, and you can easily parts packages and do stuff with it, but there is hope there is the pep for to 6 meter data 2.
0, which specifies basically the JSON media data for packaging, and hopefully people will then generate this file. Put it into a distribution with with the Python source and we will have static meter data available it’s in the draft mode. So who knows when this will get upstream but yeah? There is hope – and the second point is – we have a lot of legacy infrastructure which is kind of connected to this set of data.
But there is now Python packaging Authority group, that’s working on this and all the contributors, and I think they really deserve an applause for they work. And then there is this scary third part of the problem. These are non Python dependencies and this is this. Is the problem that every community tries to solve by you know, building an infrastructure to package all the other stuff that not just Python, and we all share this goal, but maybe maybe it’s time you know to to to look out and find take something else.
You know nowadays, JavaScript stack is basically inevitable. You have always JavaScript stack in your tool set and you know we can either build all the the the tools in Python to process JavaScript dependencies and so on and so on, or we can take the JavaScript. Our tools said to use, but then we need a tool that will actually package Python in JavaScript right for our application, and then we might even other things.
So so mixed project was basically started 11 years ago and it was developed by L code Ostra. As part of his PhD and the PhD talks about dependency hell and and how to approach that – and it was done in a university in hotel in Holland in a functional language department. So the idea is to take functional language for moral thinking and applied to the packaging problem, and it turns out it’s. It’s really really really fits the problem, so basically mixes two things: it’s a package manager and it’s also the language which we also call mix expressions.
It’s a very minimal language. It’s basically a configuration files plus lambda functions and it’ll be a little bit of other stuff. It’s lazy, evaluator! That’s something that we’re kind of not used in the Python community that much and you have to get used to the whole thing that when you actually something is actually touched, then it’s actually evaluated, and that gives it a really a lot of power for a configuration Which I will show a little bit later, it’s a standalone package manager.
You can install it on any POSIX system, the visual support we have for Linux, Mac and FreeBSD. There could be it could work on Windows if a company will sponsor that work. But currently it’s the support is basically discontinued yeah. This is so so. What is a purely functional language? I will give a very vague description in in sense of the software package, so basically, the software package should be output of a function that is deterministic and it only depends on the functional inputs without any side effects.
So when we describe the packages in Nix the the meter data that we put on those files, that’s the only thing that should affect the package and not nothing else, and we call this purity mix. So there you see for an example package. This is the prefix where the package would get installed. Our slash sneaks story is like a flat repository for the packages, and then you see the cryptographic hash as how one and then the name of the package and the version.
So every package is stored in this separate folder mmm and because we want this result, the outputs to be to be deterministic. We want to make it immutable, so the whole NIC story is mounted as read-only just to be sure that nobody will touch it and all the times times are set to UNIX day time, plus one and so on, and so on and this this hash that you See there is basically the hash of all the inputs, so if we, if this theory works that it’s the the result should depend on it on the inputs, then if we hash it, then we can uniquely identify a package.
So so this is, can you actually see this? It’s a little bit. I hope you can, but this is like an example nginx package, how you would package nginx it’s a little bit simplified at what we currently have in the repositories. So at the top you see this is our anonymous lambda function. It gets high order function, so those are just. Another function was passed in and then we called the standard environment make the duration, which is the main, the main function that does all the heavy lifting and in there you basically see it’s like.
We call it attributes I’d cynics, but this is basically a dictionary and we pass it name version, and then we tell it where to go to download sources. We tell the dependencies which is called built in puts some configure flags and then some just description about the package and that’s and then all this is basically passed to to a bash script. That goes to different phases, and it knows what to do with this meter.
Data and what you see here, this is basically that’s what gets hashed. These are all the inputs to build and nginx. This is all the information we need and, of course, there is a dependency graph of the of the packages or OpenSSL dilip and so on and so on are also written in these enix. So this is like a quick example how how powerful Nix is? If you look, if you go they’ll be back, if you look at the of this this file, we want to overwrite the things.
Basically, the the lambda function at the top and Demeter data by the package. So we want to give a user of this distribution or package repository the power to to change anything. So the top, the top line, basically overrides the lambda function with something new but is probably fell asleep. So basically it overrides the lambda lambda function and we can. We can say: okay, let’s, let’s take another open, SSL and feed it into, and then we get a new engine expeced which, which is a different, open, SSL version mmm, and we can all write a derivation itself and, for example, at the bottom example.
I have I over I’d their source and we can, for example, take nginx from gate and this this is you what you can do in the user space right, because sometimes you you have to change what the upstream does so so to install Nix on your distribution. This is like I mean just from security point of view. People will like go crazy, but you can download the script and see it’s not doing that much and you can run it yourself, but basically this is the easiest way to install it and because everything is stored under slash sneaks, you can just remove slash sneaks and You don’t have your package manager anymore, and you also have to remove the profile and you’re in your user.
Ok, so so this is. This is basically where, where everything comes together and there’s a lot of things to explain around how the NICS works so ok, we say that we have slash, sneak store and inside there are the packages and you can see it’s version and Firefox. So somehow we need to get these file system here here, that you are used to nowadays right and that basically, is then joined together into an user environment which you can see on the right – and this is this – is basically your environment with all the binaries, our Libraries are stored and under slash bin, slash leap and so on, and because we have this set of packages, will it be cool that we can have multiple of those, not just one on the system.
So this is what we have so called profile. Cynics like. We have, then, I will talk later about Nick size, which is distribution, builds on top of the package manager and we have a system profile there, which is basically your distribution. But then each user gets a profile and you can define it can create profiles on the fly per project and each profile has its own life cycle, of how you install packages inside and upgrade and and delete them and uninstall them and and basically then, the profile.
Also has a whole history of what what’s changed. So basically, when you install a package, you would get a new user environment with that binary inside and the profile would get like an another version number in the history and basically the last thing that you actually do something to the package manager. Is that sibling, Nicks vernix profile default will then at at the end of all operation, change to the new user environment and because ceilings are atomic in in POSIX.
That means that we have atomic operations of installing upgrading and uninstalling packages, and then you see on the left, you see that each user, then yeah has its own profile, and that means that users without root access, they can install packages for themselves. Of course, you can disable this if you want, and so on. So one of the really cool features is that Nix is source and binary based collection of packages.
This is very unique and the way that this is actually works is that we have a so called built farm called Hydra, and we built all the packages about all the packages there and basically the because the hash basically provides the basically unique, uniquely identifies the package. You can go as the hijra server or do you have a package with this hash, and if it has this package it will fetch the binary and if not, it will go and compile it.
And this is something that companies they then used to to set up Hydra at their own servers and have their own, like basically continuous integration tool for building the packages, and there is now all in the Nick’s 1.7. I think there is also support for SSH. So you can do the same thing to the SSH protocol, not only HTTP. So just I don’t know if this is going to work, but let’s try it out it’s a little bit yeah.
So basically like this is nixar, so it’s a little bit different than if you only use NYX but like my vim binary, then points to the vim to the vim binary that is stored inside the next or I’ll. Let me show you next or just how did you see that there is a bunch of stuff in there so yeah? This is. This is the whole thing and, for example, if we look at if we look at the linker of vim, you will see that all the dynamic libraries points to precisely one package in the nick store, and that makes it very deterministic that you, you know for sure That if you build this on two different machines, you will get the same result if you use the same source of Nick’s packages.
So right, let’s talk about Python mmm. Of course we have also collection of Python packages, and we have this. Have this function called build fight on package, which is basically a wrapper around make the duration that knows about these two tools and setup tools, and it works a little bit. It’s quite designed and then to make the duration – and this is like, for example, how you would package pillow, provide the dependencies source meter data and build vitam package will know how to run the setup I built and then setup.
I install at the right phases. You can check inside the next packages repository how it works. It’s just like 200 lines for the whole implementation. So so, when you have a lot of packages, for example, I also do a clone development and we have like 250 packages. You don’t want to do it by your hand, so there are basically two two tools for this Python. Tunics is basically just goes. There grabs the tarball gets the hash and splits out a template that is very generic, and if there is something really like non python dependencies, then you will have to fix that on your own and there is this cool tool called pipette unix, which will also be We will be working on during the sprints there’s quite a lot of developers here that tries to handle all the edge cases and automatically basically fetch packages from pypy and then generate these NYX packages.
For you – and we have these tools for note. We have no judge makes and – and we have our and so on – and so on right. So, let’s, let’s packaging, let’s move to development right, so wouldn’t it be cool if we had a tool like virtual environment but on the level on the on the layer of the package manager, not just for Python software, so you would activate environment and you would get Like geeks and all the other non Python, basically dependencies in tools available and that’s what Nick shell does so, basically how Nick shell works is that it will build all the dependencies of your package.
It will source all the information it has about those and instead of actually go and building this package, it will get you into this shell that it’s in the shell that it would actually build the package. So you have everything there available and there is a cool. Well, not really hack, because it’s also meant to use this way. But there is a cool feature that you can say that you’re not building any package.
You said source to no, and then you just provide the build inputs – and you say Nick shell – and you only get these dependencies available in your shell, for example, and in this right. This works on unemployed system, so this POSIX system. So you can give this to developers and they will get always the same environment with the same gates and and so on and so on. There is also a flag called code, pure so by default.
Nick shell will inherit your current environment and you will have all the tools available and pure basically means that it will not do that it. You will have only the tools available that you list in the build inputs. So, let’s I’m sorry for them. I hope you still see something, so this is basically activated Nick shell. I mean I I did this before on my laptop. Otherwise it would go and download those packages from from Hydra, but the network here is a bit flaky.
So, basically now I have it available. Well, let me just see that file yeah. So now I have get available and if I do pure here and then get it will say it’s not available because it will not inherit it, and this is one way to make sure that you have all the tools in your nick shell. So the same thing goes: this is, I use this this trick to actually install media core on centers, because I just didn’t want to bother with Python there.
So I just use the whole Nick’s like a stack of packages and use Nick shell, and then I have everything available in run virtual and install and that’s it. So the same goes if you have a Python package, basically all right. So this is, for example, a Python package of this is one project that I did like. I have gstreamer in there d-bus and all kinds of things that are hard to package normally with Python and, and there is of this code trick, we call we have a like – we have a variable called in Excel, so when you actually run the shell, this will Be true and we can act the extra dependencies in this case and if you only build the package, those dependencies will not get into the derivation.
So so then, okay, then okay, we have this set of packages. How could we extend this idea of a functional language to to the whole operating system and build a distribution on top of it, and it turns out that, yes, this works really nicely and when you think of it, like a configuration, files are basically just one file And software package is a bunch of files. The only difference is that your Linux distribution will package software for you and the configurations.
Files is what you will not really write yourself or change some default, but Nix Nix is basically the language that we have now. So you can use this language on both sides. Hmm so Nick says basically is a stateless. This uses stateless approach to configuration so, for example, in puppet and in chef they have declarative configuration in front, but at the back, basically, they still execute a step-by-step imperative.
They check if nginx is up and if not I mean that Network started and so on, and there are a lot of edge cases to cover here. So a lot of errors that you can hit onto and here basically the way it works is if something changes. I will show later an example of how how to do a system D process and if any parameter to that system D process changes, then it will know that it has to restart or real that process.
So it all boils down to these data going through these functions, Enix and and when something changes it will do an update. So here you can see like a minimum configuration and I just like configure moanin and you then say Nick shires rebuild switch and it would activate and get the machine into this state. So one of the things that’s also good to mention here is that mixes is basically DevOps from the beginning.
You want going chains on configuration files by default. You have one file to specify it. What is what you want, your state machine machine state to be, and you execute it, and then we have a tool that basically does provisioning of cloud servers and so on. On top of that, so, for example, if we wanted to use pyramids, which I’m using my day job, basically we would import that the default that makes file that we were using before for development.
So the project is already packaged, but then we would say package rights tests and we would write the production any file to Nick store. That’s right X is basically a function that will write a configuration files to the next or then we will do a clarity of Li specify. Ok, we have a process system D process that should start with pyramids Ben serve and pass the production in E file and, for example, if the production in e file changes here, then this hash of will change of this service, and it will know it has to Reload or restart it and so on, and then of course, on top of that, we want to use a provisioning control right, and this is like the minimum example how to how to them provision excise machines.
You install Nix ops, you specify, for example, this is a trivial, a trivial machine, so we have like a web server running Apache serving some static files. This is like the physical configuration and then we have the logical, which is basically. Where do we want to deploy it? We say: okay, the backend is VirtualBox, giving me one gigabyte of memory, and then I have like a trivial hetner, because sneaks up supports like Amazon, Pfitzner and now also Google compute engine, it’s experimental bits and all Korean.
And then you would say, create created this configuration and then deploy it and it would actually provision the VirtualBox and and you would have Apache running in your VirtualBox. So I don’t really have a demo for this because it’s going to take a while to actually show it, but just to show just to show you the whole stack. So when I would actually deploy in my projects, then I would have three files, one into full techniques, which is for the map, the development and the building of this project and developing example, machine which defines the physical and then the physical state of machine.
At the end of VirtualBox, or can then also the history, so it’s four and if we look at all those files, first default, it’s a little bit straight, but basically there you have just like build packets, my name sources current, so director II and dependencies pyramid – and This is this is, for example, configuration of machine that would lunch of chromium full screen in kiosk mode and serve the pyramid.
So at the top you say we say: okay, import. The package then enable each server enable display manager, window manager and sets etc, and the desktop manager with provider or all command, which is basically waits three seconds and then run chromium at a local host 8080 and then lower. You see again the configuration of the system D service for pyramids very simple example, and at the bottom you see how we define declaratively, a user called guest that we use for the for this.
For the chromium, graphical interface and it’s. This is basically, then the whole configuration of the Machine, so you can see like you package. The package was about 10 lines. This is about, I don’t know 100 lines or something or 150, and the VirtualBox is is like a few lines also, and this is – and this is the whole packaging development and deployment stack, that you then use to to actually and let’s might still try to so Now it’s tries to download the basic image and it’s yeah.
We can wait for a little while, but there is no point so basically now it would download first the image. Then all the dependencies configure them. It configure launch the VirtualBox copy. All those sneaks store packages inside and then activate, and you were to get the fullscreen Chrome, which we the pyramid, application running. We are having the cycle mixer springe in slovenia and Leblon aware i’m thrown in from the 23rd to 22nd of august.
So this is a great opportunity to meet the developers, talk to them. The decor and the Elco toaster will probably also be there the core developer – and this is this – is basically the image from the last year just to to shameless plugs. I wrote a blog post a little bit more into detail. Why why whining sighs tries to solve this problem in the little better, wait and other other solutions, and I had a talk at forced them about Nick’s OS.
It was more focused on Nick’s eyes, so there is a article on YouTube if you want to read it and, of course, well check it out on Nick’s eyes, dork and you’re welcome on freenode to stop by and say hi and meet the community, no questions yeah. So it looks quite interesting – and I didn’t know Nick so as before, so it looks like my puppet, my vagrant and everything I can throw away and get Nicks and it all runs out of the box.
Why isn’t that like widely adopted? So so far? What do you can you mark out? What are the differences, what are pros and cons, and so on? So the question is: if why did why? Is it not that popular yet yeah it comparison to like vagrant and all the established tools, which are, of course, very different, have different ESL’s and so on? So of course, it would be awesome to have one approach to this kind of problems.
I mean the short answer would be: we need more marketing people. The long answer would be that actually now for about a year or two, we have Nick shops and a Nick shell and so on, and now I think finally, this is this stack is ready to be used and one of the biggest two companies is logic blocks And Zalora, and they have about 100 servers provisioned with this and and the community is really growing the haskell community.
Basically, there were a few blog posts in the last few months explaining how people develop haskell with Nick shires, and it went very viral, and I would love that the same happens to to the Python community. If we actually want to solve these problems – and I think now it’s the time – that’s you know – this is we’ll really see a lot of a lot of new users. You can see that a mailing list IRC and everywhere, and I hope that we get there.
I hope that we get to the same point. It looks really interesting. The you said that it’s supported on POSIX systems include Mac, yeah yeah, and we have we have quite a bunch of unhappy home, Brio X users that ain’t no use sneaks or credit packages. Thank you, but we don’t have that much freebsd users yet so I it’s more of a just one thing. Maybe was not clear what it does like. What are the benefits of it’s not a question.
It’s just as what I miss from the talk. What are the benefits of actually using in starting developing mix, using mix and with Python? It’s you get in our company. We got development environment, which was the kind of switch to continuous development environment like not only for deployments. We use the this thing, but also for continuous development right. So, each time a developer comes like in a snap of a finger, you have a development environment, ready and no virtualization right.
It’s just your system, so it’s kind of a lot different way and quickly getting started with new projects. That was a major boost for us cool thanks hi. Thanks for a great great talk, I had a few questions. One is that you mentioned briefly: binaries find uh. Do you provide, like the sneaks itself, provide binary? So do you expect the one wanting to use binary for deploying stuff with next to like roll, your own binary storage or something? Yes, so the Hydra project? Basically, you can host it yourself and then it will build the binaries of your customized packages or your projects in your company, and you can point that to the official Hydra and to your here, and it will just ask about for binaries and fetch them.
Mine is more like a security related like I say you find you know things that would never happen. You find a bug in open, SSL or something, and you get like like counting, is like. If I run normal like say, depending upon the books, I get the new openness cell, which has to say ABI competitively, and our only thing I need to do is just restart my services for them to end up in the same by using the newest version.
So how do you – because I saw that you linked specifically to those like a certain verses of software? How do you solve this index? Dude, that’s just mean rebuilding everything or, and so um yeah. This is this is one of the problems that we so basically have. If you, if you change the OpenSSL, then he drew has to recompile all the binaries. This is not like the last time that was a hole in the OpenSSL.
I think it took like one day or something and that’s of course, unacceptable, but we have now an option called. I think it’s called security updates or something in exercise, and basically there is a hack around thing. So, as the hash will not change, so you don’t have to recompile everything, so you say the regional, the regional library was open, SSL and the new one is open, SSL, bla, bla, bla and, and that will replace older everything that uses this open, SSL library and It will not go and rebuild everything, that’s needed, and that way you can really really quickly.
You know update your server and because, if you’re using the NYX OS, it will also know that which which processes were using open, SSL and go and restart those. The the hash of a mafia package depends on its inputs, which is say sorry Saudi hash. For your package depends on the inputs, which say for a pure Python package would be and Python itself, so you can rebuild that and get the same version, but is there? Is there anything that also ties it to the version of the next tool chain that you used to build it? Because if there was a new feature introduced in the Nix tool chain or something is that? Is there a way of basically rebuilding a package of like two years ago or something exactly as it was at that time? So if basically Nix upgrades or something yeah yeah, but that’s like the whole Nix is upgraded separately.
So it doesn’t affect the Nix tool chain. But everything else down to the GCC and Julie and so on is basically a dependency then of your Python up package. I’r not sure if that answers your question and so um, I believe the Knicks toolchain sort of, if you build a binary, will change to change the our path for some in binaries etc. But if the so, if they, if that behavior slightly changes, is something like that, can you rebuild its especially diversion of the next tool chain that are used in a part of the hash, just as a as its dependencies are? There’s that part of the dependency? I’r not sure I understand the questions I may be chocolates.
Yeah come to me and we can hear okay any more questions. Then your slides, I missed a version definition when you listed the dependencies where are they defined like so. Basically, this is kind of like Ubuntu style, where the name is. The version is tied to the name and the name. Basically, the version is not important in UNIX at all, because that’s just basically a meter data and when, when the mix packages repository changes, you will just get the new package and inside there.
Of course there is the the version name, but we don’t do any. Like detection or something about the versions, you showed this plan example, so I want some specific like GStreamer version, and it was part of the name yeah. So when we have like, when you have like GStreamer, which you have version 1 and the version before 1, then we have basically two packages and then you can pick which one you want to use and you can always override the source and get another version.
If you want to change the upstream default for your project or server or whatever, what is the difference between Nix and docker? Let’s so, basically, docker tries to isolate the environments from your system right and provides a very nice API. On top of that, while sneaks basically tries to solve the packaging problem and configuration problem, so these are not like, I think, they’ll. Those two things go together, you can use.
Nix is inside the docker. If you want, of course, we’re also using the darker to solve the packaging problem and providing a huge binary blob, but that’s another discussion and in Nix you don’t have to do this. You don’t have this problem, but it’s still nice to have. You know those lightweight containers and to experiments around that and so on. That’s very short answer cool. Oh one make one question: yes, lon! Do you use sneaks in the web development because you showed a lot of stuff about the OS dependencies and OS package dependencies and even if python dependencies, let’s say right now it’s for the backend, but in our company we have a lot of struggle of packaging and Deploying services with a lot of let’s say, for example, JavaScript with Bower and so on.
So how does Nix apply to that? I know that you can declare your own sources and it can be JavaScript sources, but do you have any, for example, JavaScript repository and how does it apply to the package so that, for example, Python code finds those JavaScript libraries and so on? Because this is our crucial problem, the Debian, for example, can handle their own dependencies and it’s fine. You can do you, your own pipe people repository and it’s fine, but gluing beep and bother, for example, that’s a struggle and how does nicks apply to that yeah? That’s! That’s that’s exactly where Nick shines really good, so we have a tool.
Bower tunics. I think it’s called to generate Bower packages to generate nick’s packages from bower upstream in the next packages repository and then you would go in your project. Do the same for all the extra stuff that you want and basically the NICS knows all about about both sets of packages and you have those available and then you have all the Python dependencies available and then you use the DES make derivation and Nick shell to Develop on that and it will make it it will expose those packages for you to use.
It’s really hard to explain this without an example. But there are, there is a blog post. If you, if you google around, you, will see how it’s used for notes packages for Bower, I don’t think it’s, but it’s the same thing I mean it’s just the front end, and this is exactly where Nick’s really shines when you have to combine two two stacks Together, you
Jimmy's Internet Marketing Business Blog 