Building reproducible Development environment
One of the development team’s biggest challenges is having a consistent and reproducible environment between team members.
The bigger the team or more teams working on the same project codebase, the issue is amplified.
Different languages try to manage dependencies using specifically inbuilt language-specific package manager.
Ex. Python it is using by de-facto pip as Python package installer that is using Python Package Index repository. All dependencies can be declared in the requirements.txt file and installed with pip install -r requirements.txt Pypi is limited in managing consistent Python packages and versioning. New Python package managers try to solve the limitation of the pip like Data Science oriented Anaconda or Poetry that bring multiple improvements over pip with dependency management and packaging. Poetry is arguably Python’s most advanced dependency management option today by using a configuration file in toml format and generating project scaffolding, locking dependency in the *.lock file.
Ex. Node.js is another big contender in package management hell. Npm is a default Node.js package manager. Historically, npm lacked a dependency locking mechanism. New projects sprong out to help rectify the lack of functionality of npm like Npmp and Yarn. As I am more familiar with, Yarn was initially developed by Facebook and included lock file support and monorepo support. Npmjs community, in the past few years since the npm version 5 release, got feature parity with yarn, when it comes to lock file support.
Many other languages try to cope with dependency management by introducing specialized tolls.
Context
In my career as development lead, I encountered the same pattern of problems again and again on maintaining stable, repeatable, and consistent codebase and dependency management.
We can categorize the issue in two distinct buckets.
The development team working on the same code base may encounter issues once the dependency version is changed by one of the developers that may break the app that may rely on a specific version only.
You can mitigate this behavior primarily with test coverage or enforcing team practices on who can change the package version and how change is communicated to the rest of the group.
The dependency tree and example will be if the primary package is changed and the new package is dependent on multiple other packages. As a consequence, the new dependencies version may break partially or fully application code.
The only possible mitigation is to have a comprehensive test code coverage, which is not always possible.
The Problem
Let’s define our problem.
We need to have declarative, reliable, and reproducible management development environment managed in the source control system.
Development environment codebase should be independent of the underlying system configuration or environment variable used.
The development environment should be easy to maintain and configure without additional management overhead or limitations on development teams.
The development environment configuration management tools should support monorepo practice and trunk-based development.
The solution should be easily integrated with the CICD pipeline.
Solution
In time I tried multiple options and solutions to maintain a consistent and reproducible environment between development teams. Most of the solutions tried were language-specific, and I could not replicate them on different development languages.
A few years back, I started using NixOS as my primary OS for Laptop and Desktop. The main attraction was the declarative nature of the underlying configuration management language Nix. One of Nix’s tools is nix-shell that provided some ideas on how I could solve consistency and reproducibility issues for developers.
Nix-shell is a big part of Nix, and one of the downsides is it can’t be used stand-alone. As we already decided to use NixOs as our development environment, it was natural to use all the advanced development toolset provided by Nix.
The nix-shell official definition “nix-shell - start an interactive shell based on a Nix expression” is not helpful. Probably a better definition would be “nix-shell command is used to provide an isolated repeatable environment, and it uses Nix expressions for that.”
If you are already running Nix on your Linux or using NixOS, nix-shell is installed as part of your Nix env.
You can use Nix-Shell in three scenarios.
nix-shell packages
As an ad-hoc replacement for package manager (you can run an application without resorting to permanent installation)
Ex. We need a Python interpreter to run a script. But Python is not installed in the system. We can use nix-shell to run the environment and deploy Python only for the duration of the nix-shell running environment. Once we exit from nix-shell Python will not be available anymore.
~> python
python: command not found~> nix-shell --packages python[nix-shell:~]$ python --version
Python 2.7.18[nix-shell:~]$ exit
exit~> python
python: command not found
We can specify the version of the python we want to run.
~> nix-shell -p python39
these 4 paths will be fetched (11.03 MiB download, 53.47 MiB unpacked):
/nix/store/0fgr90x0rsbl9j2rv3935mrm9mfy0zls-tzdata-2020f
/nix/store/2kzvsbp8i7k3kq284cg9bv2zk6iadi2m-expat-2.2.10
/nix/store/ph6hpbx4pr31146wpb72yrk3f3yy0xcs-python3-3.9.4
/nix/store/rxcgs0xi9ngd0zsq47f4g06v2dqpqsr0-libffi-3.3
copying path '/nix/store/0fgr90x0rsbl9j2rv3935mrm9mfy0zls-tzdata-2020f' from 'https://cache.nixos.org'...
copying path '/nix/store/2kzvsbp8i7k3kq284cg9bv2zk6iadi2m-expat-2.2.10' from 'https://cache.nixos.org'...
copying path '/nix/store/rxcgs0xi9ngd0zsq47f4g06v2dqpqsr0-libffi-3.3' from 'https://cache.nixos.org'...
copying path '/nix/store/ph6hpbx4pr31146wpb72yrk3f3yy0xcs-python3-3.9.4' from 'https://cache.nixos.org'...[nix-shell:~]$ python --version
Python 3.9.4[nix-shell:~]$ exit
exit
We can as well directly run, in our case, Python shell by;
~> nix-shell -p python39 --run python
Python 3.9.4 (default, Apr 4 2021, 18:23:51)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>We can run multiple packages, or in the below case, we run Python with the NumPy library already available for Python to consume. We need to specify only NumPy and Python interpreter will be deployed as a dependency.
~> nix-shell -p python3Packages.numpy --run python
these 7 paths will be fetched (24.90 MiB download, 170.80 MiB unpacked):
/nix/store/6fig8w8sir42gsli6zwzlp7njl2j6rbl-blas-3-dev
/nix/store/761cpqh61b4zak4rl0y3sz5pa0954d3c-blas-3
/nix/store/c1lb5svhbkc7nsi8cpk6xbwgbiij38z9-openblas-0.3.13
/nix/store/fm5xm721z0jf65570g78v5xkaa6jy3kd-lapack-3-dev
/nix/store/lr061z7swdw9gfgv8r2x2wzaz582m0xv-lapack-3
/nix/store/p0y5xw5yk57rwlik4yamq9xk7z5s45y6-gfortran-9.3.0-lib
/nix/store/ys7ffnghl2iaza8ly4mjpycfyargxv4z-python3.8-numpy-1.20.2
copying path '/nix/store/p0y5xw5yk57rwlik4yamq9xk7z5s45y6-gfortran-9.3.0-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/c1lb5svhbkc7nsi8cpk6xbwgbiij38z9-openblas-0.3.13' from 'https://cache.nixos.org'...
copying path '/nix/store/761cpqh61b4zak4rl0y3sz5pa0954d3c-blas-3' from 'https://cache.nixos.org'...
copying path '/nix/store/lr061z7swdw9gfgv8r2x2wzaz582m0xv-lapack-3' from 'https://cache.nixos.org'...
copying path '/nix/store/fm5xm721z0jf65570g78v5xkaa6jy3kd-lapack-3-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/6fig8w8sir42gsli6zwzlp7njl2j6rbl-blas-3-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/ys7ffnghl2iaza8ly4mjpycfyargxv4z-python3.8-numpy-1.20.2' from 'https://cache.nixos.org'...
Python 3.8.9 (default, Apr 2 2021, 11:20:07)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>nix-shell shebangs
Another use case is to embed nix-shell as a shell script that will help run on any Nix compatible OS even if the software specified in the script is not installed on the system.
We can use nix-shell shebangs
~> python
python: command not found~> cat test.sh
#! /usr/bin/env nix-shell
#! nix-shell -i python -p python
print "Hello world!"~> chmod +x test.sh~> ./test.sh
Hello world!
shell.nix
The most valuable and essential capability of the nix-shell is to have in your git repository code folder where you keep your sources shell.nix and where we can describe all dependencies and libraries needed for the project. Every time the developer is in the folder where the code resides and runs nix-shell, all project software dependencies will be deployed for the entire user session.
Let’s take the most basic example of the Node.js code base.
~> node --version
node: command not found~> yarn --version
yarn: command not found
We need file shell.nix and populate with;
~> cat shell.nix
let pkgs = import <nixpkgs> {};
in pkgs.mkShell rec {
name = "nodeAapp";
buildInputs = with pkgs; [
nodejs yarn
];
}Now we can run nix-shell.
~> nix-shell
these 6 paths will be fetched (21.79 MiB download, 102.64 MiB unpacked):
/nix/store/d50va99bmlng5926npmxvfmy7ly7qsjs-icu4c-69.1
/nix/store/hahkzr18y0x8vdxn4g2s2igxzga2qlar-openssl-1.1.1k-dev
/nix/store/km96sj9xw1ww7jpz4055dfr92nmmq33i-libuv-1.41.0
/nix/store/m8csnvay8fj9479f4zdqnr29s9dbjdha-yarn-1.22.10
/nix/store/s42x7xflxz9br1axz9i7583m42j17piy-icu4c-69.1-dev
/nix/store/vp3y4rm082app6f5ld4h9dq5z00bbapp-nodejs-14.17.2
copying path '/nix/store/d50va99bmlng5926npmxvfmy7ly7qsjs-icu4c-69.1' from 'https://cache.nixos.org'...
copying path '/nix/store/hahkzr18y0x8vdxn4g2s2igxzga2qlar-openssl-1.1.1k-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/km96sj9xw1ww7jpz4055dfr92nmmq33i-libuv-1.41.0' from 'https://cache.nixos.org'...
copying path '/nix/store/s42x7xflxz9br1axz9i7583m42j17piy-icu4c-69.1-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/vp3y4rm082app6f5ld4h9dq5z00bbapp-nodejs-14.17.2' from 'https://cache.nixos.org'...
copying path '/nix/store/m8csnvay8fj9479f4zdqnr29s9dbjdha-yarn-1.22.10' from 'https://cache.nixos.org'...[nix-shell:~/src/test]$ node --version
v14.17.2[nix-shell:~/src/test]$ yarn --version
1.22.10
We also can specify the version.
shell.nix
let pkgs = import <nixpkgs> {};in pkgs.mkShell rec {
name = "webdev"; buildInputs = with pkgs; [
nodejs-12_x
yarn
];
}
Note: advantage is by developing in monorepo team may have microservices running with different Node.js versions, and this will allow the team members to support multiple version for the project
We can use the shellHook function to extend nix-shell functionality.
shell.nix
with (import <nixpkgs> {});
mkShell {
buildInputs = [
nodejs
yarn
];
shellHook = ''
mkdir -p .nix-node
export NODE_PATH=$PWD/.nix-node
export NPM_CONFIG_PREFIX=$PWD/.nix-node
export PATH=$NODE_PATH/bin:$PATH
'';
}Let’s take a look at the Python example with Pandas and NumPy libraries as dependencies.
shell.nix
with (import <nixpkgs> {});
mkShell {
buildInputs = with python39Packages; [
pip
pandas
numpy
];
shellHook = ''
alias pip="PIP_PREFIX='$HOME/.pip_packages' \pip"
'';
}~> nix-shell
these 23 paths will be fetched (37.14 MiB download, 197.66 MiB unpacked):
/nix/store/06mpf704f7mm0nxvvwciv7vzda1g739g-python3.9-pbr-5.5.1
/nix/store/3fzlg2lvd0vmqyy2va96796w95rqk8jn-python3.9-tables-3.6.1
/nix/store/7hk3s95mllbj2bjy680s5pbq2f657chg-python3.9-et_xmlfile-1.0.1
/nix/store/7iw8n01m46m6n4gxr46y6cnys1r3cp4z-python3.9-xlwt-1.3.0
/nix/store/8fwy5f74b20xh7zyyz875ri64z1nd86d-python3.9-numpy-1.20.2
/nix/store/8lvgm16bifq6a17vhn5fgacrwf06lvwn-python3.9-pip-21.0.1
/nix/store/8nqr6x0w568p9kxr5rz3mghb63f924j2-python3.9-Bottleneck-1.3.2
/nix/store/aiw78m14p2r7xmxdq9z7ij6am3air1nm-python3.9-python-dateutil-2.8.1
/nix/store/dz9f2hjbs16gq2h8v7zs7z23fyh11xcq-python3.9-SQLAlchemy-1.3.23
/nix/store/j4gfk78r1xalkpd1j3195x0v202vqbwi-python3.9-pytz-2021.1
/nix/store/jx9669w7109bkmjj7x249yiflzyl8c87-python3.9-html5lib-1.1
/nix/store/kpqmf12s5j0yzncp0635c07wh2y5lfkf-python3.9-beautifulsoup4-4.9.3
/nix/store/kxrspjijzc81axl0sb968z1s7ix0glvb-python3.9-soupsieve-2.2.1
/nix/store/lpjpm6y207id13d3x7y994mxdvvb2c9b-python3.9-numexpr-2.7.3
/nix/store/mxd4dd79jmldaysvkkl0hrxlk91amlci-python3.9-webencodings-0.5.1
/nix/store/mzbagdp4b6kifr2narz1zp9r5lklnzin-python3.9-openpyxl-3.0.7
/nix/store/n9rxhgjsxlqy9w82v4abhwjwiv4kcp48-python3.9-scipy-1.6.1
/nix/store/nnp5x5vf8algzq5ikwyv7j4dxqc4zrd4-python3.9-mock-4.0.3
/nix/store/qlrpcacwyfdq2sgd6z9wpmm6vzax65l6-python3.9-xlrd-2.0.1
/nix/store/qrfg4g4l8skb6yj1b1gws1xrvjzkf99p-python3.9-jdcal-1.4.1
/nix/store/v7a9l919llggn700sn9mzdvma731akfg-python3.9-lxml-4.6.3
/nix/store/wfvd4l92z04chhd3rdq959npvn3bs9kg-python3.9-six-1.15.0
/nix/store/z3j94srpg9pxljb301mhbdlg0nibddh7-python3.9-pandas-1.2.3
copying path '/nix/store/8fwy5f74b20xh7zyyz875ri64z1nd86d-python3.9-numpy-1.20.2' from 'https://cache.nixos.org'...
copying path '/nix/store/8lvgm16bifq6a17vhn5fgacrwf06lvwn-python3.9-pip-21.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/v7a9l919llggn700sn9mzdvma731akfg-python3.9-lxml-4.6.3' from 'https://cache.nixos.org'...
copying path '/nix/store/dz9f2hjbs16gq2h8v7zs7z23fyh11xcq-python3.9-SQLAlchemy-1.3.23' from 'https://cache.nixos.org'...
copying path '/nix/store/7hk3s95mllbj2bjy680s5pbq2f657chg-python3.9-et_xmlfile-1.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/qrfg4g4l8skb6yj1b1gws1xrvjzkf99p-python3.9-jdcal-1.4.1' from 'https://cache.nixos.org'...
copying path '/nix/store/06mpf704f7mm0nxvvwciv7vzda1g739g-python3.9-pbr-5.5.1' from 'https://cache.nixos.org'...
copying path '/nix/store/j4gfk78r1xalkpd1j3195x0v202vqbwi-python3.9-pytz-2021.1' from 'https://cache.nixos.org'...
copying path '/nix/store/wfvd4l92z04chhd3rdq959npvn3bs9kg-python3.9-six-1.15.0' from 'https://cache.nixos.org'...
copying path '/nix/store/mzbagdp4b6kifr2narz1zp9r5lklnzin-python3.9-openpyxl-3.0.7' from 'https://cache.nixos.org'...
copying path '/nix/store/kxrspjijzc81axl0sb968z1s7ix0glvb-python3.9-soupsieve-2.2.1' from 'https://cache.nixos.org'...
copying path '/nix/store/8nqr6x0w568p9kxr5rz3mghb63f924j2-python3.9-Bottleneck-1.3.2' from 'https://cache.nixos.org'...
copying path '/nix/store/nnp5x5vf8algzq5ikwyv7j4dxqc4zrd4-python3.9-mock-4.0.3' from 'https://cache.nixos.org'...
copying path '/nix/store/lpjpm6y207id13d3x7y994mxdvvb2c9b-python3.9-numexpr-2.7.3' from 'https://cache.nixos.org'...
copying path '/nix/store/aiw78m14p2r7xmxdq9z7ij6am3air1nm-python3.9-python-dateutil-2.8.1' from 'https://cache.nixos.org'...
copying path '/nix/store/kpqmf12s5j0yzncp0635c07wh2y5lfkf-python3.9-beautifulsoup4-4.9.3' from 'https://cache.nixos.org'...
copying path '/nix/store/n9rxhgjsxlqy9w82v4abhwjwiv4kcp48-python3.9-scipy-1.6.1' from 'https://cache.nixos.org'...
copying path '/nix/store/mxd4dd79jmldaysvkkl0hrxlk91amlci-python3.9-webencodings-0.5.1' from 'https://cache.nixos.org'...
copying path '/nix/store/qlrpcacwyfdq2sgd6z9wpmm6vzax65l6-python3.9-xlrd-2.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/3fzlg2lvd0vmqyy2va96796w95rqk8jn-python3.9-tables-3.6.1' from 'https://cache.nixos.org'...
copying path '/nix/store/7iw8n01m46m6n4gxr46y6cnys1r3cp4z-python3.9-xlwt-1.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/jx9669w7109bkmjj7x249yiflzyl8c87-python3.9-html5lib-1.1' from 'https://cache.nixos.org'...
copying path '/nix/store/z3j94srpg9pxljb301mhbdlg0nibddh7-python3.9-pandas-1.2.3' from 'https://cache.nixos.org'...[nix-shell:~/src/test]$ python
Python 3.9.4 (default, Apr 4 2021, 18:23:51)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> import numpy as np
>>>
Python’s case can get tricky as Python provided virtualenv that can be managed by PyPI or poetry. nix-shell as well creates an isolated environment that can easily replace virtualenv.
The dilemma is mostly around how familiar the team is with nix expressions.
If you are starting your journey into Nix you can find a comprehensive guide in NixWiki Python page and use Mach-nix that makes it easy to create and share reproducible python environments or packages. Another option is to use poetry2nix to turn Poetry projects into Nix derivations without writing Nix expressions.
Nix-shell supports the latest nix flake option that will remove dependency on nix-channels and allow a better consistency between nix environments with direct integration with git.
Example from The blog post Practical Nix Flakes
~> nix shell https://github.com/nixos/nixpkgs/archive/nixpkgs-unstable.tar.gz#hello ~> nix shell 'git+https://github.com/nixos/nixpkgs?ref=nixpkgs-unstable#hello'
Nix Wiki provides more in depth examples on NixWiki Flake Page
Our team implemented a native nix shell with flake integration and poetry to manage python package dependencies.
flake.nix
{
description = "A flake using poetry2nix"; nixConfig.bash-prompt = "\[nix-develop\]$ "; inputs.nixpkgs.url = "github:NixOS/nixpkgs";
inputs.utils.url = "github:numtide/flake-utils";
inputs.poetry2nix-src.url = "github:nix-community/poetry2nix"; outputs = {nixpkgs, utils, poetry2nix-src, self}: utils.lib.eachDefaultSystem (system:
let
pkgs = import nixpkgs { inherit system; overlays = [ poetry2nix-src.overlay ]; }; matrixInfraEnv = pkgs.poetry2nix.mkPoetryEnv {
projectDir = ./.;
python = pkgs.python39;
editablePackageSources = {
matrixInfra = ./.;
};
};
in {
devShell = pkgs.mkShell {
buildInputs = [ matrixInfraEnv pkgs.python39Packages.pip ];
};
});
}
To make it easy for the developers, we start using direnv that natively integrates with Nix and Flakes.
Every time the developer enters the folder, all dependencies and environment variables will be deployed without running the nix shell command.
You will need to create a file .envrc where you store your flake.nix, or shell.nix.
# for shell.vin
~> cat .envrc
usen nix# for flake.nix
~> cat .envrc
use flake
To be aware, learning Nix will be a very steep curve. In my opinion, the effort is worth it as it solves so many problems the development community faces daily working in large teams.
Conclusions
Nix-shell supports almost all programming languages under the sky.
you can find an extensive repository with examples in Git repository
Check as well an excellent blog post by Mattia Gheda’s An introduction to nix-shell and his A nix-shell for developing Elixir
We scratch the surface on potential use cases for nix-shell, but this can help with information and use cases to try and make the 1st step into the Nix ecosystem.