For the past year I have been working on-and-off on a project that will sync all my photos and videos to S3. My main goal was to minimize storage costs and provide and off-site backup. It has worked for me, but the process of backing everything up is a very bespoke workflow so it’s probably useful to only me.
I have media coming from an iPhone, a GoPro, an Insta360 and occasionally a couple of old DSLRs, so I need to keep it all organized. My project, an ASP.NET Core app, just provides a simple way to track and upload these files. Part of the workflow is spinning up a couple containers (one for the app and one for PostgreSQL) so that I can back up everything up.
I don’t want to keep a server running or have any additional cloud costs other than just storage so that’s why I run these containers intermittently. It’s not a huge pain since I only do it weekly or bi-weekly, but still sorta annoying to do. Why not try to cut this process down? A couple of minutes is way better than the upwards of 30 minutes it would sometimes take me.
Another thing I also care about is file integrity, so I made sure my app would always generate a SHA256 checksum. This means I can always validate it later. I’m not worried about it on S3, more so on my local drives.
TrueNAS Scale
I began looking at TrueNAS because it uses the ZFS filesystem and can detect corruption. As I read more into it, I saw that the newest version of TrueNAS (Scale) runs kubernetes under the hood meaning it can run containerized apps that can interact with any data stored on the server.
Getting a container running on TrueNAS Scale is very simple. You just upload your container image to an online repo and TrueNAS Scale will pull the image. It will then create a namespace and service in kubernetes to run. I ran my app alongside a PostgreSQL container and the only out-of-the-box issue I ran into was my ASP.NET Core container could not resolve the DNS to get to the Postgres container.
TrueNAS automatically created separate namespaces in k8s for these two containers. There may be a way to get them in the same namespace which might have simplified my issue, but I couldn’t figure out how via the TrueNAS UI. Actually all the kubernetes YAML spec files are auto-generated by TrueNAS - I didn’t need to manage it at all.
Resolving other containers from the ASP.NET Core container
Maybe this is more of a kubernetes issue than a TrueNAS scale issue, but since you don’t manage the k8s spec files at all, debugging is a little more tricky. In my ASP.NET Core config, I set my connection string as:
Host=postgres-ix-chart.ix-postgres.svc.cluster.local;Port=5432;Database=some_db;Username=some_user;Password=some_pass;
This follows the k8s convention to access a service: [service name].[namespace].svc.cluster.local, but my ASP.NET Core container still could not connect to the Postgres container.
I confirmed it to be a DNS issue because the container could not resolve the service domain name using nslookup. Also, the contents of /etc/resolve.conf was only the DNS of my local network, which itself has no idea of the kubernetes cluster.
This meant that the pod was not using the DNS of the cluster, but instead the TrueNAS server itself. I found the fix in the k8s docs, which was to change the DNS policy to use the cluster’s DNS (ClusterFirst). If you were editing the spec file, you would set dnsPolicy: ClusterFirst.
And in TrueNAS, you need to set this dropdown:
The dropdown options are very…verbose, but to be fair, it describes the problem exactly.