Heads up! Watch your NIC order when adding more hosts to VCF

VMware Cloud Foundation is a unified SDDC platform for the hybrid cloud. It is based on VMware’s compute, storage, and network virtualization.

VCF can be expanded with more workload domains by adding further hosts, or it can be stretched over two availability zones (AZ). The expansion is initiated by and under control of the SDDC-Manager. The procedure is fairly straightforward and SDDC-Manger does all the configuration tasks in the background, i.e. forming vSAN clusters, networks, kernel ports, vCenters and NSX control planes.

  • setup hosts with ESXi base image
  • confige a management IP address
  • set root credentials
  • configure DNS and NTP
  • import new hosts into SDDC-Manager
  • deploy new WLD

There is a pitfall that can be easily overseen: The order of the new host’s NICs. Before we can import new hosts, we’ll get to see a checklist about the host requirements. The hosts need to have two NICs with at least 10 GBit.

While reading the list there’s a little detail that is often overlooked. Traditional numbering means that both NICs must have numbers vmnic0 and vmnic1. Unfortunately this seems to be hard coded and cannot be changed (as of current version 4.2). To make matters worse, many server systems have onboard 1 GBit network adapters. There’s a KB article that explains how VMware ESXi determines the order in which names are assigned to network devices. It’ll start with onboard NICs and then continues with PCIe cards. As a result you’ll might end up with two 1 GBit onboard NICs as vmnic0 and vmnic1. In this case the bringup of the VCF expansion will fail.

While you can choose NICs during initial VCF bringup, this is not possible during expansion and this time there’s no such thing as a bringup sheet. You can’t select more than two NICs either when using SDDC-Manager. In that case you’ll need to use API-calls.

Workaround

Currently there’s no other way than to disable onboard NICs in the system BIOS. If your desired NICs still show a higher number you’ll need to put the PCIe card into a lane with lower number.

Activate Immutability flag on existing Veeam XFS Repositories

Veeam Backup & Replication v11 is about to be launched on February 24th. Partners and service providers already have access to the RTM build.

One of my favourite features of version 11 are immutable backup files. You can read about it in my recent post “Hardening your Veeam Backup strategy with immutable repositories on Linux XFS“.

Some of you might already use Linux repository servers based on XFS with fast clone. Once you upgrade to Veeam Backup & Replication version 11 you may wish to activate immutable backups. I will explain in this blog post how you can use this new feature on existing XFS repository servers.

Activate immutability

To activate the immutability feature edit your existing backup repository servers and go to step “repository” (picture below). Starting with version 11 there’s a checkbox “make recent backups immutable for [x] days” where 7 is the minimum value to enter. Once you activate the checkbox you will most likely see a message “Unable to set the immutability option: cannot find the transport service on server <your repo server>. Configure the server anyway?“. Click on [Yes].

To make use of the immutability flag you’ll have to reconfigure the transport service on the Linux server.

Continue reading “Activate Immutability flag on existing Veeam XFS Repositories”

Hardening your Veeam Backup strategy with immutable repositories on Linux XFS

Attackers on IT infrastructure have become more sophisticated in recent times. Not only do they ecrypt live data and entire virtual machines, they also have learned to delete or encrypt entire backups too. No matter if your online backups are stored on a local repository or a cloud share. If that happens and you do not have an offline backup like tape or any other air gapped solution you’re up a creek without a paddle. Even worse: If you’re responsible for data backup, you might be facing unpleasant questions by the management.

Now the question is how to protect backup data in case an attacker gets hold on your backup server. Once they get credentials to the server, they can do whatever the backup-admin can do. Deleting backups for example. One first step is to separate your backup systems from your domain. In case the domain administrator account gets compromised there’s still (a little) barrier between attackers and your backup systems. But still, you’ll never know if attackers have knowledge of a zeroday exploit to takeover your backup server. It would be a better approach to protect backups against deletion for a defined timespan. There’s a similar option on AWS S3 buckets, which are usually used as offsite backup copies. Getting back offsite backups takes a considerable amount of time. In case of a crypto attack time is crucial and the clock is ticking. Wouldn’t it be great to have immutable backups on your primary repositories on site? Good news. There’s light at the end of the tunnel (and I promise it’s not the oncoming train).

Immutable Backups

Before we go into detail, we need to clarify what immutable backups mean. The idea is that you write once and then files (backups) are protected for a self defined period of time. Even the backup administrator cannot delete them before the defined timespan has passed.

Veeam Backup & Replication v11 will make use of a native Linux XFS filesystem feature. XFS can set an extended file attribute [i] which will protect the file from renaming, modification, deletion or hard-linking. The key point is that backups are transferred and written with non-privileged accounts and the immutable attribute is set and removed by a privileged service on the repository server which cannot be accessed from outside.

All you need is a Linux repository formatted with XFS and Veeam Backup & Replication v11, which will be released in the near future. (Update: Expected release date will be February 24th 2021)

Continue reading “Hardening your Veeam Backup strategy with immutable repositories on Linux XFS”

2020 – A retrospective view

I don’t think it’s an exaggeration when I say “2020 was a year like no other.” Pretty much everyone in the IT industry and elsewhere will confirm that. I’m taking this last day of 2020 as an opportunity to write my last blog article of the year and review important events for me in it.

January

The year began quite normally, almost like any other. After the turn of the year, tasks and projects for the coming months slowly started. According to experience, it is a quiet month, because many customers usually don’t resume work before the second week.

February

February had some of my personal highlights of the year.

We held a VMUG meeting in Nuremberg on February 13th. Today I have to add that it was an on-site meeting. At that time a matter of course. Today it sounds like a tale from a distant past. I myself had given a presentation there on the topic of “Strategies for proactive error prevention”. A closer look at the products VMware Skyline and Runecast Analyzer. What do they have in common and what are their unique selling points?

A long planned private trip took us to the high latitudes of the Arctic. Spitsbergen in winter. No other event left such deep impressions this year like this one. On February 14th, after 84 days of darkness, the long polar night ends on Spitsbergen and the sun comes above the horizon again for the first time. In mid-April, the sun will not set at all for another 99 days. In the time in between the ice desert shines in wonderful light. We love the Arctic and wanted to experience this natural spectacle and – of course – capture it in pictures.

Polar bear warning applies to the whole archipelago of Svalbard.

Exploring Spitsbergen means being away from civilization for many hours at temperatures down to -38°C. Always accompanied by an armed guide, because the island is polar bear country and you’re required by law to have at least one armed member in your party once leaving the settlement.

We’ve learned, for example, that such simple things like eating can be a real challenge at very low temperatures We’ve also witnessed what it means to be caught in a white-out when the sky merges into the coulour of the ice, visibility is close to zero and everything around you is just white.

Svalbard reindeer

While being abroad I got a pleasant notification in my inbox. I was nominated to be a vExpert for another year.

March

Our stay lasted until the first week of March. But the world we returned to was a different to the one we left. Already on the way we read news about Corona outbreaks and sold-out toilet paper. When we left Germany, Corona was hardly a topic. Now it was THE topic and would remain so for months.

On March 11th, the last German on site VMUG took place at VMware’s premises in Munich. I myself was co-organizer and speaker and had no concerns about the train ride and the meeting at that time. Looking back and with the knowledge of today, it was certainly not the brightest idea. We were unnecessarily putting ourselves at great risk. Washing hands alone is not enough against SARS-CoV-2. Luckily (as far as I know), the event had no further health consequences for anyone involved.

Continue reading “2020 – A retrospective view”