Attackers on IT infrastructure have become more sophisticated in recent times. Not only do they ecrypt live data and entire virtual machines, they also have learned to delete or encrypt entire backups too. No matter if your online backups are stored on a local repository or a cloud share. If that happens and you do not have an offline backup like tape or any other air gapped solution you’re up a creek without a paddle. Even worse: If you’re responsible for data backup, you might be facing unpleasant questions by the management.
Now the question is how to protect backup data in case an attacker gets hold on your backup server. Once they get credentials to the server, they can do whatever the backup-admin can do. Deleting backups for example. One first step is to separate your backup systems from your domain. In case the domain administrator account gets compromised there’s still (a little) barrier between attackers and your backup systems. But still, you’ll never know if attackers have knowledge of a zeroday exploit to takeover your backup server. It would be a better approach to protect backups against deletion for a defined timespan. There’s a similar option on AWS S3 buckets, which are usually used as offsite backup copies. Getting back offsite backups takes a considerable amount of time. In case of a crypto attack time is crucial and the clock is ticking. Wouldn’t it be great to have immutable backups on your primary repositories on site? Good news. There’s light at the end of the tunnel (and I promise it’s not the oncoming train).
Immutable Backups
Before we go into detail, we need to clarify what immutable backups mean. The idea is that you write once and then files (backups) are protected for a self defined period of time. Even the backup administrator cannot delete them before the defined timespan has passed.
Veeam Backup & Replication v11 will make use of a native Linux XFS filesystem feature. XFS can set an extended file attribute [i] which will protect the file from renaming, modification, deletion or hard-linking. The key point is that backups are transferred and written with non-privileged accounts and the immutable attribute is set and removed by a privileged service on the repository server which cannot be accessed from outside.
All you need is a Linux repository formatted with XFS and Veeam Backup & Replication v11, which will be released in the near future. (Update: Expected release date will be February 24th 2021)
List of materials
- vSAN 7 Update1
- Veeam Backup 11.0.0.591 BETA 2
- Proxie Server: Ubuntu 20.04 LTS
- Repository Server: Ubuntu 20.04 LTS
All examples shown in this article are from a POC lab environment with focus on the immutable attribute. Other settigs you see may not be best practises for production environments.
Setting up the repository
Basically there’s one new checkmark to set when you’re adding a new backup repository (see red frame in the picrure below). The minimum timespan for immutability is 7 days.
Once you’ve set up your first Linux XFS repository, you can try a test run. You might run into a write permission problem if you’re re-using an existing Linux repository server. But no worries, it’s easy to fix.
In my case the /backup path had been created with different credentials and my backup user “veeam” did not have write permissions on the folder. All you have to do is to transfer ownership to the backup user.
sudo chown -R veeam:veeam /backup
Parameter [-R] sets recursive ownership to all subfolders. Then you specify user:group. In my case the user “veeam” belongs to group “veeam” and the path to take ownership is /backup. Now the backup user veeam can write to the repository.
Investigate results
I’ve created a small test job “Testjob2” and let it run twice. On the first run it created a full backup (vbk) and an incremental backup (vib) on the second run. Let’s have a look on the backup repository server. We’ll enter the /backup directory and check the file attributes.
cd /backup lsattr
We can see that our two backup files have an [i] attribute, which indicates that the’re protected (immutable).
But there’s more to see. Let’s list the content of our /backup directory.
veeam@repo01:/backup/Testjob2$ ls -la
total 6861212
drwxr-xr-x 2 veeam veeam 133 Jan 22 15:09 .
drwxr-xr-x 4 veeam veeam 37 Jan 22 15:01 ..
-rw-r--r-- 1 root root 201 Jan 22 15:09 .veeam.1.lock
-rw-r--r-- 1 veeam veeam 23833 Jan 22 15:09 Testjob2.vbm
-rw-r--r-- 1 veeam veeam 6999216128 Jan 22 15:05 Testjob2D2021-01-22T150152_EC83.vbk
-rw-r--r-- 1 veeam veeam 26636288 Jan 22 15:09 Testjob2D2021-01-22T150855_3C8F.vib
There’s a new file named .veeam.1.lock. Let’s have a look into that file:
As you can see it contains all the retention data for each immutable backup file. Note that the timestamp is identical for both the full backup and the increment, although they have been created at different times.
Trying to delete a backup
Now let’s try to delete our backup from the GUI (Delete from disk).
Veeam will ask for confirmation and tries to delete the files. If you watch the log output, you’ll see that the removal will fail.
The protection is valid for the whole backup chain. That means your latest incremental backup will define the protection of all backup files in the chain, even if your oldest (full) backup is older than the protection period. You cannot remove the full backup, while there are immutable increments that rely on it. Therefore it’s neccessary to have periodic active full backups in the chain.
Let’s verify that by adding an active full backup.
veeam@repo01:/backup/Testjob2$ ls -la
total 13678532
drwxr-xr-x 2 veeam veeam 176 Jan 22 15:50 .
drwxr-xr-x 4 veeam veeam 37 Jan 22 15:01 ..
-rw-r--r-- 1 root root 291 Jan 22 15:50 .veeam.2.lock
-rw-r--r-- 1 veeam veeam 34466 Jan 22 15:50 Testjob2.vbm
-rw-r--r-- 1 veeam veeam 6999216128 Jan 22 15:05 Testjob2D2021-01-22T150152_EC83.vbk
-rw-r--r-- 1 veeam veeam 26636288 Jan 22 15:09 Testjob2D2021-01-22T150855_3C8F.vib
-rw-r--r-- 1 veeam veeam 6980923392 Jan 22 15:49 Testjob2D2021-01-22T154716_B467.vbk
veeam@repo01:/backup/Testjob2$
As you can see the name of the lockfile has changed. We can see that there are two different timestamps for protection (yellow and green boxes). One for the old chain and one for the new.
Take away message: a backup chain cannot be deleted before the youngest increment aged out of the protection period.
Try to remove the attribute on the shell
We’ve shown that backups are protectd against deletion from the backup GUI. What about a non privileged user (veeam) on the shell? The command to remove immutable attribute is:
chattr -i <file>
We’ll try this on the latest full backup file Testjob2D2021-01-22T154716_B467.vbk.
veeam@repo01:/backup/Testjob2$ lsattr
----i--------------- ./Testjob2D2021-01-22T150152_EC83.vbk
----i--------------- ./Testjob2D2021-01-22T150855_3C8F.vib
----i--------------- ./Testjob2D2021-01-22T154716_B467.vbk
-------------------- ./Testjob2.vbm
veeam@repo01:/backup/Testjob2$ chattr -i ./Testjob2D2021-01-22T154716_B467.vbk
chattr: Operation not permitted while setting flags on ./Testjob2D2021-01-22T154716_B467.vbk
veeam@repo01:/backup/Testjob2$
Sorry, no prize for you. That’s what we’ve expected. User veeam is not permitted to change the [i] attribute.
What about sudo?
sudo chattr -i ./Testjob2D2021-01-22T154716_B467.vbk
By executing commands with sudo we elevate user pivileges to root privileges. And of course we have to enter root credentials. We basically turn our non-privileged user into a root user.
veeam@repo01:/backup/Testjob2$ lsattr
----i--------------- ./Testjob2D2021-01-22T150152_EC83.vbk
----i--------------- ./Testjob2D2021-01-22T150855_3C8F.vib
-------------------- ./Testjob2D2021-01-22T154716_B467.vbk
-------------------- ./Testjob2.vbm
Now the [i] attribute has gone from file Testjob2D2021-01-22T154716_B467.vbk.
Is that a problem? Not really, because user root is not permitted to login via SSH. The root user isn’t used for any remote connection. Neither Veeam Backup nor anything else. Anyone who has root credentials can do anything on a Linux system – that’s what root users do.
Restrictions
- When using immutable backups on Linux repositories you cannot have the proxy role on the same server.
- Immutability only works with forward incremental backups and periodic full backup job settings and without rollbacks.
- Using concurrent repositories on the same server require either all repositories to be immutable or all mutable.
- Changes to the backup chain are not supported.
- All repositories of a SOBR must be of the same kind. Either mutable or immutable.
Conclusion
The use of immutable attributes on backup files on Linux repositories is a huge step forward towards backup security. We should re-think our strategy and shift repositories from Windows repository servers towards Linux repositories. There’s no such thing like absolute security, but the more obstacles we put into an attackers way the less likely we’ll become victims.