Hardening your Veeam Backup strategy with immutable repositories on Linux XFS

Attackers on IT infrastructure have become more sophisticated in recent times. Not only do they ecrypt live data and entire virtual machines, they also have learned to delete or encrypt entire backups too. No matter if your online backups are stored on a local repository or a cloud share. If that happens and you do not have an offline backup like tape or any other air gapped solution you’re up a creek without a paddle. Even worse: If you’re responsible for data backup, you might be facing unpleasant questions by the management.

Now the question is how to protect backup data in case an attacker gets hold on your backup server. Once they get credentials to the server, they can do whatever the backup-admin can do. Deleting backups for example. One first step is to separate your backup systems from your domain. In case the domain administrator account gets compromised there’s still (a little barrier) between attackers and your backup systems. But still, you’ll never know if attackers have knowledge of a zeroday exploit to takeover your backup server. It would be a better approach to protect backups against deletion for a defined timespan. There’s a similar option on AWS S3 buckets, which are usually used as offsite backup copies. Getting back offsite backups takes a considerable amount of time. In case of a crypto attack time is crucial and the clock is ticking. Wouldn’t it be great to have immutable backups on your primary repositories on site? Good news. There’s light at the end of the tunnel (and I promise it’s not the oncoming train).

Immutable Backups

Before we go into detail, we need to clarify what immutable backups mean. The idea is that you write once and then files (backups) are protected for a self defined period of time. Even the backup administrator cannot delete them before the defined timespan has passed.

Veeam Backup & Replication v11 will make use of a native Linux XFS filesystem feature. XFS can set an extended file attribute [i] which will protect the file from renaming, modification, deletion or hard-linking. The key point is that backups are transferred and written with non-privileged accounts and the immutable attribute is set and removed by a privileged service on the repository server which cannot be accessed from outside.

All you need is a Linux repository formatted with XFS and Veeam Backup & Replication v11, which will be released in the near future.

Continue reading “Hardening your Veeam Backup strategy with immutable repositories on Linux XFS”

ExaGrid Time-Lock – Who’s (still) afraid of ransomware?

Introduction

Ransomware currently represents one of the most prominent threats to IT infrastructures. Reports of successful attacks are accumulating, the attacks are getting closer. More than 30% of all companies, institutes, universities or public authorities in Germany have already dealt with such attacks. In some cases, a ransom was paid to get access to their own data again.

Even with payment, success is never certain. After all, one negotiates with criminals. Authorities therefore advise against payment.

The essential protective measure against the consequences of such an attack is an up-to-date and consistent backup.

Ransomware vs. Backup

Unfortunately, attackers also know about the importance of backups. The currently circulating malware, such as Emotet or Ryuk, contain code that actively searches for backups on the net. Using previously obtained access data for Active Directory accounts or by attacking via RDP exploits or using the brand-new Zerologon exploit an attempt could be made to take over the systems that operate the data backup in the company or hold the backup data.

The automatic attack is often followed by hackers in the flesh who actively browse the net and try to destroy all backups. This is often an easy task, since backups today are preferably held on hard disk systems, permanently connected to the infrastructure.

The reason is obvious: If all backups are deleted or also encrypted, the compliance of the “customer” to pay his ransom increases by far.

Many approaches have therefore already been conceived to store the backup data out of reach of an attacker. A very simple and secure variant is an Air-Gap – a physical separation of the backup media from the system. For example, LTO tapes can be physically removed from the library.

Without this kind of time-consuming manual extraction – which would also have to be performed daily – the data remains latently vulnerable. It doesn’t matter whether it is stored on disk systems, dedup appliances, tapes in a library or even in an S3 cloud repository.

S3 cloud providers have therefore proposed an API extension called “Immutability” some time ago. With this, at least the backups in the cloud layer can be made immune to changes for a certain time.

Some of these solutions are natively supported by Veeam. Amazon AWS is one of them. Microsoft Azure is currently still missing. Furthermore S3 memory is not suitable for every application. A primary backup with Veeam on S3 is for example not directly possible. The S3 layer is only available as an extension of a scale-out backup repository.

Continue reading “ExaGrid Time-Lock – Who’s (still) afraid of ransomware?”

Veeam Storage Plugin for DataCore – Deepdive

Any questions, remarks or additions: melter[at]idicos.de

SANsymphony meets Veeam Backup and Replication – true love in the end!

In December 2019 the plugin for the popular DataCore SANsymphony SDS was finally released. And it is done in the only proper way: With full support and validation by Veeam.

In this article series we will cover several aspects of the plugin:

Continue reading “Veeam Storage Plugin for DataCore – Deepdive”

Veeam Backup v10 on vSAN 7.0

There have been many new releases in the first quarter of 2020. The long anticipated release of Veeam Backup & Replication version 10, we’ve been waiting for since 2017 and also the latest generation of VMware vSphere. While I had vSAN 7 beta running on my homelab cluster before GA, I’ve worked with Veeam Backup 10 only in customer projects. There’s unfortunately no room for playing with new features unless the customer requests it. One of the new features of Veeam v10 is the ability to use Linux proxies and repositories. With XFS filesystem on the repository you can use the fast clone feature which is similar to ReFS on Windows.

In this tutorial I will show how to:

  • Deploy and size the Veeam server
  • Show base configuration to integrate vCenter
  • Build, configure and deploy a Linux proxy and its integration into backup infrastructure
  • Build, configure and deploy a Linux XFS repository

Using Veeam Backup on a vSAN Cluster has special design requirements. There’s no direct SAN backup on VMware vSAN because there’s neither a SAN, nor a fabric and nor HBAs. There are only two backup methods available: Network Mode (nbd) and Virtual Appliance Mode (hotadd). The latter is recommended for vSAN, but you should deploy one proxy per host to avoid unnecessary traffic on the vSAN interfaces. Hotadd also utilizes Veeam Advanced Data Fetcher (ADF).

Talking about licenses: Having Linux proxies on each host will reduce the cost of Windows licensing. One more reason to play around with this new feature. A Veeam license will be required too, but as a vExpert I can get a NFR (not for resale) license which is valid for one year. Just one of the advantages of being a vExpert. 🙂

Let the games begin. We’ll need a Veeam server that holds the job database and the main application. The proxy and repository role will be kept on individual (Linux) servers.

Continue reading “Veeam Backup v10 on vSAN 7.0”