Veeam ReFS Repository on iSCSI Targets

Troubleshooting Repository Deadlocks

With Resilient Filesystem (ReFS) integration into Veeam Availability Suite 9.5 a whole bunch of features was integrated. One of the biggest advantages is ‘Fast Cloning Technology’ which enables synthetic full backups by merely creating pointers to already existing datablocks on the repository.

In a small scale environment I had a hardware repository server (Win 2016) with an iSCSI Volume as repository (ReFS, 64k) as primary backup target. This constellation worked like a Swiss watch. Daily backups ran for months without any trouble. Fast cloning technology enabled weekly synthetic full backups with minimal consumption of extra space.

Recently I’ve added another iSCSI Volume (ReFS, 64k) to be used as repository for backup copies. That’s when the fun began… Continue reading “Veeam ReFS Repository on iSCSI Targets”

Change Brocade FOS default passwords

Brocade FC-Switches are equipped with four default useraccounts: admin, root, user and factory.

By connectiong an SSH session with user ‘root’ and default password ‘fibranne’ you will be prompted to change logins for accounts root, user and factory.

 

This happens at login as long as you did not change default passwords. The process can be started by pressing <ENTER>. If it is skipped by pressing Ctrl-C you will be prompted again at next login of user ‘root’

Show users

userconfig --show -a

This command will show a list aof all local users and their settings.

 

vMotion fails at 21% with error 195887371

How to troubleshoot vMotion issues

Troubleshooting vMotion issues is in most cases a matter of networking issues. I will demonstrate in this case how to trace down the problem and how to find possible culprits.

What’s the problem?

Initiating a host vMotion between esx1 and esx2 passes all pre-checks, but then fails at 21% progress.

Migrate virtual machine:Failed waiting for data. Error 195887371. The ESX hosts failed to connect over the VMotion network.

See the error stack for details on the cause of this problem.
Time: 07.01.2018 19:08:08
Target: WSUS
vCenter Server: vc
Error Stack
Migration [167797862:1515348488969364] failed to connect to remote host <192.168.45.246> from host <10.0.100.102>: Timeout.
vMotion migration [167797862:1515348488969364] vMotion migration [167797862:1515348488969364] stream thread failed to connect to the remote host <192.168.45.246>: The ESX hosts failed to connect over the VMotion network
The vMotion migrations failed because the ESX hosts were not able to connect over the vMotion network. Check the vMotion network settings and physical network configuration. 
Migration [167797862:1515348488969364] failed to connect to remote host <10.0.100.102> from host <192.168.45.246>: Timeout.
vMotion migration [167797862:1515348488969364] failed to create a connection with remote host <10.0.100.102>: The ESX hosts failed to connect over the VMotion network
Failed waiting for data. Error 195887371. The ESX hosts failed to connect over the VMotion network.

Continue reading “vMotion fails at 21% with error 195887371”

Using VM tags to manage backup SLAs

Agile backup job assignment with VM-tags and Veeam Backup

Organizing VMs in backup jobs can be a tedious task. Especially when there is a larger number of VMs and multiple jobs. It might happen that you miss out a VM for a job, or have it doubled.

To check whether a VM is backed up by the corresponding jobs, you either have to go through the settings of every single job or use smart tools like Veeam-One.

There are a couple of ways to add VMs to a backup job. You can choose single VMs by name, or select an entire VM folder, resource-pool or datastore. But one of the most sophisticated and versatile methods is to leverage VM-tags for selection.

What are tags?

A tag behaves like a label or a sticker that you put on a VM. It defines a property or a membership of a given VM. Think of a tag that marks a VM for daily backup. A second tag might mark a VM for hourly or weekly backup. You don’t have to adjust your backup jobs twice a week to remove or add new virtual machines. With VM-tags you don’t have to touch backup jobs at all. Just tell the job once to select all VMs with a specific tag and you’re done.

Even checking job membership for a VM is easier with tags. Just have a look at its tags.

How?

I will now show a simple example how to use tagged VMs in combination with Veeam Backup & Replication.

Continue reading “Using VM tags to manage backup SLAs”

VMware releases patches for Meltdown and Spectre bug

Important patches available

VMware has issued Security Advisories for the recent Meltdown and Spectre bugs to address side-channel analysis due to speculative execution.

I recommend reading a post by Anton Gostev (Veeam), which i reposted yesterday.

It includes patches for VC, ESXi, Workstation and Fusion.

VMSA-2018-0004

VMware vSphere, Workstation and Fusion updates add Hypervisor-Assisted Guest Remediation for speculative execution issue.

There’s also an update to VMSA-2018-0002

VMSA-2018-0002.1

Continue reading “VMware releases patches for Meltdown and Spectre bug”

Resilient Network Infrastructure for Virtualization

Network topology 101 for Virtual Infrastructures

I usually don’t like writing about obvious matters. Yes, fire is hot – night is dark and ice is cold. But in recent times I’ve witnessed some network topology designs (?), that made me frown.

I admit, that in some cases the situation is based on a lack of budget or just structures that have grown over years. I can understand that and it’s no shame. It’s my job to give advices and help to re-design.

No matter how many resources you have – if you use them without thinking, it will never be enough.

On the other hand there are environments who boast with high class components that have cost a fortune and which are organized in such an inefficient way that it almost hurts.

This article is not intended as a networking deep-dive. It’s a shallow 101 about network design that should be common knowledge. It’s a guide for the novice but I’d be happy to get responses by experts too.

The Basics

First let’s start with four simple networking requirements for Virtual Infrastructures.

  • redundancy
  • resiliency
  • bandwidth
  • latency

Continue reading “Resilient Network Infrastructure for Virtualization”

create kernel panic on ESXi

There are situations when you need to check cluster reactions after a ESX host crash. For example to see if HA will start VM on other hosts.

The easiest method is to pull a hosts powercord. But there ar more elegant ways to let a host crash.

Warning! Do not use on productive systems! This is for testing purposes under controlled conditions only. Use at your own risk.

PSOD

You can trigger a Purple-Screen-of-Death (PSOD) by issuing a special command that causes a kernel panic.  Use the VMkernel Sys Info Shell (vsish).

First you need a SSH connection to your host. Change to vsish

vsish
set /reliability/crashMe/Panic

Alternatively you can issue the command together with parameters.

vsish -e set /reliability/crashMe/Panic 1

Your host will end up in a PSOD and can be restarted afterwards.