Google Cloud “GCP” native NixOS images build

10 min readSep 13, 2021

I am using NixOs as a Development environment that is running on Google Cloud VM.
NixOs official Web site does not provide an official GCP image.
On the NixOs Wiki page you can find an article how you can build your GCP NixOs Image that didn’t work well in the past but now seems all build image issue was resolved and now works well.
NixOS Wiki page article also provides two GCP storage that you can find outdated GCP VM images that seem not actively maintained anymore.

nixos-images > gs://nixos-images
nixos-cloud-images > gs://nixos-images

In the below HowTo, we will cover how you can build your images in GCP by using Google Cloud VM and store the images in your Google storage bucket.

Prepare Google Cloud environment.

Before we can start, you will need to have a Google Cloud Account

We will create a dedicated GCP Project with VM that we will use to build updated NixOs images from the latest nixpkgs builds.

Prerequisites

Before starting
In the 1st step, you will need to install Google Cloud SDK Depending on the OS or Linux Distribution, and please follow installation instructions.

Gcloud authentication

Once installation is complete, you will need to authenticate CLI to be able to access your google cloud resources.

~> gcloud auth loginGo to the following link in your browser:    https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=32542940657.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fappengine.admin+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcompute+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth&state=skHTaEGrhJSYIzDIwt4phrhSzUm97t&prompt=consent&access_type=offline&code_challenge=EcgLC0aZhpefFkL7k6ep-8lvJ1Og8NgCvs9VllOE5lQ&code_challenge_method=S256Enter verification code:

GCP Project creation

Once authenticated, you should be able to access all cloud resources in your google cloud account.
If you are using your account for the 1st time, Google will create a default Project for your account. We will ignore the default project and will make a dedicated project to build NioxOs images.

GCP new Project creation

We will create a dedicated GCP project. I use the name for the project mudrii-nixos

~> gcloud projects create mudrii-nixos 
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/mudrii-nixos].
Waiting for [operations/cp.8533728690478532386] to finish...done. Enabling service [cloudapis.googleapis.com] on project [mudrii-nixos]...
Operation "operations/acf.p2-237810657129-fa1f0fa7-3015-46ff-ab52-f92d7bd3f8df" finished successfully.

Get details on the created project.

~> gcloud projects describe mudrii-nixos
createTime: '2021-09-06T08:57:16.001Z'
lifecycleState: ACTIVE
name: mudrii-nixos
projectId: mudrii-nixos
projectNumber: '111111111111'
~> gcloud projects list
PROJECT_ID             NAME              PROJECT_NUMBER
mudrii-nixos           mudrii-nixos      111111111111

Configure gcloud CLI

Once a new project was created is a good practice to add the newly created project as default for gcloud CLI. every command executed with gcloud of gsutils will perform on the newly created project

~> gcloud config set project mudrii-nixos
Updated property [core/project].

Configure Project billing

One more step is needed to make the GCP project usable we will need to add billing to the project.

List available billing accounts

~> gcloud alpha billing accounts list
ACCOUNT_ID            NAME                OPEN   MASTER_ACCOUNT_ID
ZZZZZ-ZZZZZZZ-ZZZZZZ  MyBill              True

Associate billing account with the project

~> gcloud alpha billing accounts projects link mudrii-nixos --billing-account=ZZZZZ-ZZZZZZZ-ZZZZZZ 
billingAccountName: billingAccounts/ZZZZZ-ZZZZZZZ-ZZZZZZ 
billingEnabled: true
name: projects/mudrii-nixos/billingInfo
projectId: mudrii-nixos

Verify billing account association

~> gcloud alpha billing accounts projects list --billing-account=ZZZZZ-ZZZZZZZ-ZZZZZZ
PROJECT_ID             BILLING_ACCOUNT_ID    BILLING_ENABLED
mudrii-nixos           ZZZZZ-ZZZZZZZ-ZZZZZZ  True

Create and configure Google storage bucket

The next step is to create a Google storage bucket to store generated nixos GCP VM images.

~> gsutil mb gs://nixos-images-gcp
~> gsutil du -s -h gs://nixos-images-gcp

I decided to grant read access to external users to the generated images. If you do not intend to share generated images, you can skip the below step.

~> gsutil iam ch allUsers:objectViewer gs://nixos-images-gcp

Create and Configure GCP networking

By default, Google will create a global VPC and subnets for every region and firewall rules once a project is created.

To make it clean, I decided to create a separate network stack with a subnet and firewall to control the resources better.

Before creating network stuck a good practice to remove all automated default network VPC, subnets, firewall.

List available network resource

List VPC

~> gcloud compute networks list
NAME     SUBNET_MODE  BGP_ROUTING_MODE  IPV4_RANGE  GATEWAY_IPV4
default  AUTO         REGIONAL

List subnets

~> gcloud compute networks subnets list
NAME     REGION                   NETWORK  RANGE
default  us-central1              default  10.128.0.0/20
default  europe-west1             default  10.132.0.0/20
default  us-west1                 default  10.138.0.0/20
default  asia-east1               default  10.140.0.0/20
default  us-east1                 default  10.142.0.0/20
default  asia-northeast1          default  10.146.0.0/20
default  asia-southeast1          default  10.148.0.0/20
default  us-east4                 default  10.150.0.0/20
default  australia-southeast1     default  10.152.0.0/20
default  europe-west2             default  10.154.0.0/20
default  europe-west3             default  10.156.0.0/20
default  southamerica-east1       default  10.158.0.0/20
default  asia-south1              default  10.160.0.0/20
default  northamerica-northeast1  default  10.162.0.0/20
default  europe-west4             default  10.164.0.0/20
default  europe-north1            default  10.166.0.0/20
default  us-west2                 default  10.168.0.0/20
default  asia-east2               default  10.170.0.0/20
default  europe-west6             default  10.172.0.0/20
default  asia-northeast2          default  10.174.0.0/20
default  asia-northeast3          default  10.178.0.0/20
default  us-west3                 default  10.180.0.0/20
default  us-west4                 default  10.182.0.0/20
default  asia-southeast2          default  10.184.0.0/20
default  europe-central2          default  10.186.0.0/20
default  northamerica-northeast2  default  10.188.0.0/20
default  asia-south2              default  10.190.0.0/20
default  australia-southeast2     default  10.192.0.0/20

List firewalls

~> gcloud compute firewall-rules list 
NAME                    NETWORK  DIRECTION  PRIORITY  ALLOW                         DENY  DISABLED
default-allow-icmp      default  INGRESS    65534     icmp                                False
default-allow-internal  default  INGRESS    65534     tcp:0-65535,udp:0-65535,icmp        False
default-allow-rdp       default  INGRESS    65534     tcp:3389                            False
default-allow-ssh       default  INGRESS    65534     tcp:22                              FalseTo show all fields of the firewall, please show in JSON format: --format=json
To show all fields in table format, please see the examples in --help.

Remove default created network resource

Before removing VPC, we need to remove the firewall 1st, and once all firewall rules have been removed, we can remove the default VPC.

Remove default Firewall

~> gcloud compute firewall-rules delete default-allow-internal --quiet
Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-internal].
~> gcloud compute firewall-rules delete default-allow-icmp --quiet 
Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-icmp].
~> gcloud compute firewall-rules delete default-allow-rdp --quiet
Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-rdp].
~> gcloud compute firewall-rules delete default-allow-ssh --quiet
Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-ssh].

Remove default VPC

~> gcloud compute networks delete default --quiet
Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/networks/default].

Create Networking resources

We cleared all default network stuck, and now we can create a new network stuck.

Create VPC

Now we are ready to create a custom VPC.

~> gcloud compute networks create vpc-nixos --subnet-mode=custom 
Created [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/networks/vpc-nixos].
NAME       SUBNET_MODE  BGP_ROUTING_MODE  IPV4_RANGE  GATEWAY_IPV4
vpc-nixos  CUSTOM       REGIONALInstances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:

Create custom subnet

In the below example, I use a narrow IP range in asia-southeast1 “Singapore” region. As I will run a single instance that will generate nixos images, I do not intend to run anything else.

~> gcloud compute networks subnets create subnet-sg --network=vpc-nixos --range=192.168.1.0/24 --region asia-southeast1
Created [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/regions/asia-southeast1/subnetworks/subnet-sg].
NAME         REGION           NETWORK    RANGE
subnet-sg  asia-southeast1  vpc-nixos  192.168.1.0/24

Create a firewall rule to allow ssh

In order to connect to remote VM over ssh we need to open a firewall on port 22.

~> gcloud compute --project=mudrii-nixos firewall-rules create allow-ssh --direction=INGRESS --priority=1000 --network=vpc-nixos --action=ALLOW --rules=tcp:22 --target-tags=allow-ssh
Creating firewall...⠹Created [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/allow-ssh].
Creating firewall...done.
NAME       NETWORK    DIRECTION  PRIORITY  ALLOW   DENY  DISABLED
allow-ssh  vpc-nixos  INGRESS    1000      tcp:22        False

Add image to your VM images repository to be able to create NixOs VM in GCP

To create VM with NixOS installed, we need to have a VM image in our GCP images repositories.

In the below example, I am using one of the images I created previously locate in gs://nixos-images-gcp/

NOTE: make sure you specify your project ID --project=mudrii-nixos

~> gcloud compute images create nixos-21-05-2873 \
--source-uri=gs://nixos-images-gcp/nixos-image-21.05.2873.6bfe71f2a4e-x86_64-linux.raw.tar.gz \
--description=nixos-image-21.05.2873.6bfe71f2a4e \
--family=nixos \
--project=mudrii-nixos

List available images in your image repository

~> gcloud compute images list | grep nixos
nixos-image-21-05-2873-6bfe71f2a4e-x86-64-linux       mudrii-nixos         nixos-image-21-05                             READY

Create NixOS VM

Our network stuck is up a ready we can create a dedicated VM to build nixos GCP images.

Few point to mention.

Vm will be created in the same region where we set our subnet. --zone=asia-southeast1-b
--machine-type=n2d-standard-4 VM AMD EMYC 4 vCPU and 16 GB RAM
--metadata=enable-oslogin=true allow you to ssh into the VM with your gcloud account
--tags=allow-ssh Added tag to the VM to allow open port ssh we configured in the firewall
--boot-disk-size=60GB size of the Disk you may select a lower capacity I find 30 works well too
--boot-disk-type=pd-ssd This is the fastest option local SSD will speed up significantly image generation

~> gcloud beta compute --project=mudrii-nixos instances create nixos-base --zone=asia-southeast1-b --machine-type=n2d-standard-4 --subnet=subnet-sg --network-tier=PREMIUM --metadata=enable-oslogin=true --tags=allow-ssh --image=nixos-21-05-2873 --image-project=mudrii-nixos --boot-disk-size=60GB --boot-disk-type=pd-ssd --boot-disk-device-name=nixos-base
Created [https://www.googleapis.com/compute/beta/projects/mudrii-nixos/zones/asia-southeast1-b/instances/nixos-base].
WARNING: Some requests generated warnings:
 - Disk size: '60 GB' is larger than image size: '3 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 
NAME        ZONE               MACHINE_TYPE    PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
nixos-base  asia-southeast1-b  n2d-standard-4               192.168.1.2  35.188.151.168  RUNNING

SSH into remote

We can ssh directly to the newly created VM with the gcloud command

~> gcloud compute ssh --project=mudrii-nixos nixos-base

Building NixOS image

Once SSH gets into the newly created NixOs Google VM, we can start preparing to build the NixoS CGP image.

Authenticate from VM into Google CLoud

First, we need to become root.

~> sudo -i

We need access to google SDK to upload newly created images into the google bucket created in the previous steps. To authenticate, we need google SDK, and instead of installing, we will use nix-shell.

[root@nixos-base:~]# nix-shell -p google-cloud-sdk git

Once we have access to gcloud binary we can auth into the google cloud

[nix-shell:~]# gcloud auth login
[nix-shell:~]# gcloud projects list
PROJECT_ID    NAME          PROJECT_NUMBER
mudrii-nixos  mudrii-nixos  111111111111
[nix-shell:~]# gcloud config set project mudrii-nixos
Updated property [core/project].
[nix-shell:~]# gcloud config configurations list
NAME     IS_ACTIVE  ACCOUNT            PROJECT       COMPUTE_DEFAULT_ZONE  COMPUTE_DEFAULT_REGION
default  True       my_email@gmail.com  mudrii-nixos

To validate authentication, you can check if you have access to your Gcloud bucket created in the previous step.

NOTE: make sure you add BOTO_CONFIG=/dev/null or you will get an error during the image build

[nix-shell:~]# export BOTO_CONFIG=/dev/null[nix-shell:~]# gsutil ls -l gs://nixos-images-gcp

Update NixOS to the latest version

To create an image with the latest stable NixOs is essential to update the existing version with the latest version.

Check existing NixOs version and Linux Kernel.

[nix-shell:~]# nixos-version
21.05.3001.12eb1d16ae3 (Okapi)
[nix-shell:~]# uname -a
Linux nixos-base.asia-southeast1-b.c.mudrii-nixos.internal 5.10.62 #1-NixOS SMP Fri Sep 3 08:09:31 UTC 2021 x86_64 GNU/Linux

Update NixOs to the latest version

Update 1st nix-channels

[nix-shell:~]# nix-channel --list
nixos https://nixos.org/channels/nixos-21.05
[nix-shell:~]# nix-channel --update 
unpacking channels...
created 1 symlinks in user environment

Update 1st NixOS

[nix-shell:~]# nixos-rebuild switch
building Nix...
building the system configuration...
these derivations will be built:
...
..
.
[nix-shell:~]# nixos-version
21.05.3021.8b0b81dab17 (Okapi)

Or if the version is outdated significantly, run;

[nix-shell:~]# nixos-rebuild switch --upgrade

NixOs cleanup

Before attempting image creation is a good idea to do a cleanup and do a garbage collection on existing update and repair any sha inconsistency that may cause failed image build.

[nix-shell:~]# nix-collect-garbage -d
removing old generations of profile /nix/var/nix/profiles/system
removing generation 4
removing old generations of profile /nix/var/nix/profiles/per-user/root/channels
removing generation 5
finding garbage collector roots...
removing stale link from 
...
..
.
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves 21.60 MiB
167 store paths deleted, 1129.80 MiB freed
[nix-shell:~]# nix-store --gc
finding garbage collector roots...
deleting garbage...
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves 21.60 MiB
0 store paths deleted, 0.00 MiB freed[nix-shell:~]# nix-store --repair --verify --check-contents
reading the Nix store...
checking path existence...
checking hashes...
path '/nix/store/kacsvbh8qjl28izy5g7a8p96z6xdvnik-google-cloud-sdk-340.0.0' was modified! expected hash 'sha256:0dvxzzklaswx0d2svx0nzjilqfmgd2dxffi7hcbz89p7r6w1jab0', got 'sha256:1wpsb25jajbqvsw29jy073w0cr994005xb22pb9p8mkinn9vx8hp'
copying path '/nix/store/kacsvbh8qjl28izy5g7a8p96z6xdvnik-google-cloud-sdk-340.0.0' from 'https://cache.nixos.org'...
[nix-shell:~]# nix-store --optimise
430.37 MiB freed by hard-linking 47107 files

NixOs GCP image build

To build a GCP nixos image, we need to clone the nixpkgs repository where build scripts is locate

[nix-shell:~]# git clone https://github.com/NixOS/nixpkgs.git --depth 1
Cloning into 'nixpkgs'...
remote: Enumerating objects: 45763, done.
remote: Counting objects: 100% (45763/45763), done.
remote: Compressing objects: 100% (29848/29848), done.
remote: Total 45763 (delta 1593), reused 38844 (delta 1293), pack-reused 0
Receiving objects: 100% (45763/45763), 30.46 MiB | 14.19 MiB/s, done.
Resolving deltas: 100% (1593/1593), done.
Updating files: 100% (27800/27800), done.

The final step is to run a script that will build a GCP nixos image based on the underlying system version.

NOTE: Make sure you specify your GCP storage bucket name in BUCKET_NAME=

[nix-shell:~]# BUCKET_NAME=nixos-images-gcp nixpkgs/nixos/maintainers/scripts/gce/create-gce.sh
these paths will be fetched (0.05 MiB download, 0.28 MiB unpacked):
  /nix/store/p5lnl4zr45n7mf9kz9w8yz3rqh001b5c-bash-interactive-4.4-p23-dev
copying path '/nix/store/p5lnl4zr45n7mf9kz9w8yz3rqh001b5c-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
...
..
.
/nix/store/ii2h0jqwfzmzdc6lxyfmg4ia5726r6g6-google-compute-image
gs://nixos-images-gcp/nixos-image-21.05.3021.8b0b81dab17-x86_64-linux.raw.tar.gz

NixOs image build verification

Once the image is created and uploaded into Google storage bucket and in the image repository, we can verify by;

[nix-shell:~]# gsutil ls -l gs://nixos-images-gcp
 434150003  2021-09-12T03:18:26Z  gs://nixos-images-gcp/nixos-image-21.05.3021.8b0b81dab17-x86_64-linux.raw.tar.gz

Verify Image repository

[nix-shell:~]# gcloud compute images list | grep nixos
nixos-image-21-05-3021-8b0b81dab17-x86-64-linux       mudrii-nixos         nixos-image-21-05                             READY

Post-build cleanup

It is good to remove the nixpkgs repository to minimize space and speed on the next build and stop VM to reduce the cost on your cloud bill.

[nix-shell:~]# ls -la
total 44
drwx------  8 root root 4096 Sep 13 11:02 .
drwxr-xr-x 17 root root 4096 Sep  6 10:55 ..
-rw-------  1 root root 2538 Sep 10 10:27 .bash_history
drwxr-xr-x  3 root root 4096 Sep  6 11:07 .cache
drwxr-xr-x  3 root root 4096 Sep  6 11:07 .config
lrwxrwxrwx  1 root root   64 Sep 13 11:02 gce -> /nix/store/ii2h0jqwfzmzdc6lxyfmg4ia5726r6g6-google-compute-image
drwxr-xr-x  3 root root 4096 Sep  6 11:48 .gsutil
-rw-r--r--  1 root root   45 Aug 25 05:05 .nix-channels
drwx------  2 root root 4096 Sep 13 10:30 .nix-defexpr
drwxr-xr-x  9 root root 4096 Sep 13 10:55 nixpkgs
drwx------  2 root root 4096 Sep  6 10:55 .ssh
[nix-shell:~]# rm -rf ~/nixpkgs
[nix-shell:~]# sudo shutdown now
Connection to 35.198.250.170 closed by remote host.
Connection to 35.198.250.170 closed.
ERROR: (gcloud.compute.ssh) [/run/current-system/sw/bin/ssh] exited with return code [255].

Fin

Next time when you want to create a new VM on Google Cloud you can specify in custom images your latest nixos image. Make sure you add --metadata=enable-oslogin=true allow you to ssh into the VM.