Guidance on Network Configurations for Azure Stack HCI Cluster

[HasanHasib]

Hello All,

I am going to deploy my very first Azure Stack HCI cluster.

I need guidance on how to assign the IP addresses to the NICs, what should be IP address scheme, and best practices.

Details of network adapters are as follows:

2 NICs of 25 Gbps for compute/Virtualization (VMs) traffic

2 NICs of 25 Gbps for Storage

2 NICs of 1 Gbps for Mgmt

Questions:

(1) How to assign the IP addresses? since it's my first time with the black screen.

(2) How to configure SET for the NICs to be used for virtualization/compute step-by-step?

(3) I know that I don't need to configure SET (Switch Embedded Teaming) for the Storage NICs, but what should I do for the Mgmt NICs? Should I configure SET for them OR should 1 NIC be enough for the Mgmt?

(4) How to configure SET step-by-step?

(5) What is the recommendation for the IP address scheming for all the NICs? Let's say I'm assigning 10.4.8.x to the Mgmt NICs, should I use the same IP address scheme for Storage and Compute NICs as well, or should I assign some different IP schemes like 172.x.x.x

(6) There are 2 NICs for Storage and we are not configuring SET for them... So, just to confirm, will I need to assign 2 IP addresses for Storage on each server?

(7) How to enable the iWARP RDMA for the Storage NICs? I'm thinking to use iWARP RDMA as I read MSFT engineers recommend/prefer iWARP over RoCE. Please confirm my understanding.

(8) For the Mgmt NIC, first I will configure the SET then I will assign the IP address to the SET NIC, just like we used to with the conventional NIC Teaming in Windows Server 2012 R2.

Apologies in advance for such basic questions... It's my first time with Azure Stack HCI...

Thank you very much!

[#2]

This URL provides step wise guidance of setting up AZHCI
https://learn.microsoft.com/en-us/azure-stack/hci/deploy/deployment-tool-install-os

[#3]

Hi @Chandrasekhar_Arya 

I already checked the https://learn.microsoft.com/en-us/azure-stack/hci/deploy/deployment-tool-install-os

It doesn't answer my queries. It just says: "Configure networking as per your environment."

[#4 · Accepted solution]

Hi @HasanHasib,

I would recommend reaching out to a partner that can help you setup the cluster. If you bought a new solution the vendor can definitely help you. Also Windows Admin Center helps out when new to Azure Stack HCI and configures the right settings for you such as SET switch.

(1) How to assign the IP addresses? since it's my first time with the black screen.

You can assign IP adresses using the 'sconfig' screen. When you login, you will end up on that screen automatically and then you type '8' to go to the networking config screen.

(2) How to configure SET for the NICs to be used for virtualization/compute step-by-step?

You can specify creating a SET switch when executing the 'New-VMSwitch' cmdlet, see New-VMSwitch (Hyper-V) | Microsoft Learn

(3) I know that I don't need to configure SET (Switch Embedded Teaming) for the Storage NICs, but what should I do for the Mgmt NICs? Should I configure SET for them OR should 1 NIC be enough for the Mgmt?

Based on your requirements (bandwidth etc) you can go either way. For production I prefer to use two.

(4) How to configure SET step-by-step?

Same answer as (2).

(5) What is the recommendation for the IP address scheming for all the NICs? Let's say I'm assigning 10.4.8.x to the Mgmt NICs, should I use the same IP address scheme for Storage and Compute NICs as well, or should I assign some different IP schemes like 172.x.x.x

You can set it up as you wish depending on your network architecture. Can you stay in the same 'scheme' so that everything is in 10.4.x.x range. So for example:
Management: 10.4.8.x
Storage1: 10.4.9.x
Storage2: 10.4.10.x

Or you can use smaller subnets to save IP addresses.

(6) There are 2 NICs for Storage and we are not configuring SET for them... So, just to confirm, will I need to assign 2 IP addresses for Storage on each server?

Yes.

(7) How to enable the iWARP RDMA for the Storage NICs? I'm thinking to use iWARP RDMA as I read MSFT engineers recommend/prefer iWARP over RoCE. Please confirm my understanding.

This is a setting you enable on the physical network card. Depending on the brand you can set it using Set-NetAdapterAdvancedProperty

(8) For the Mgmt NIC, first I will configure the SET then I will assign the IP address to the SET NIC, just like we used to with the conventional NIC Teaming in Windows Server 2012 R2.

Correct.

--
Darryl van der Peijl
Splitbrain

[#5]

Hi @Darryl van der Peijl 

Thank you very much for to the point answers!

Just 1 more question... Do I need to assign an IP address to the Virtualization NIC as well? In the Windows Server 2012 R2 Hyper-V Failover Cluster, we used to create 2 Teamed NICs. 1 for Mgmt, 2nd for Hyper-V Switch. We just used to assign an IP address to the Mgmt NIC. For the Hyper-V NIC, we wouldn't assign an IP address at the NIC level. It was used for creating the vSwitch and then we used to tag the VLAN ID for the specific VMs.

I am confused here...

Could you please share the best practice?

@thomasmaurer 

@Karl_Wester-Ebbinghaus 

Thank you once again!

[#6]

Hi @HasanHasib,

In the current operating systems we use "Switch embedded teaming". With this technology the teaming and vSwitch functionality is combined. You can choose per vSwitch if you want to create a virtual adapter that will present itself in the OS. If you create 2 vSwitches it could be that it automatically will create 2 virtual adapters as well, in that case you can remove the one that you don't need.

--
Darryl van der Peijl
Splitbrain

[#7]

OT: hi Hasan, to make it more accessible, please use only the tag Azure Stack HCI for AzHCI related questions. Azure Stack is not the same. I do not like this familiar naming as many deciders have problems to separate them and many first responses are "too expensive". It is a bit a drama like with Edge Browser that has a burnt name for many, still.

[#8]

(1) How to assign the IP addresses? since it's my first time with the black screen.

Run Get-netadapter and keep notes of NIC.
use Sconfig to assign IP address


(2) How to configure SET for the NICs to be used for virtualization/compute step-by-step?

If you are using WAC to configure so it will provide you option to configure NIC.
My suggestion create SET (MGMT + Compute) Set2 (only for SMB)
(3) I know that I don't need to configure SET (Switch Embedded Teaming) for the Storage NICs, but what should I do for the Mgmt NICs? Should I configure SET for them OR should 1 NIC be enough for the Mgmt?
i gave answer above.

(4) How to configure SET step-by-step?
You can use WAC or use Powershell Command to configure SET


(5) What is the recommendation for the IP address scheming for all the NICs? Let's say I'm assigning 10.4.8.x to the Mgmt NICs, should I use the same IP address scheme for Storage and Compute NICs as well, or should I assign some different IP schemes like 172.x.x.x
MGMT = 192.168.0.10/26 VLAN 10
Compute = 192.168.1.11/26 VLAN 11
SMB= 192.168.2/11/26 VLAN 12

(6) There are 2 NICs for Storage and we are not configuring SET for them... So, just to confirm, will I need to assign 2 IP addresses for Storage on each server?
yes

(7) How to enable the iWARP RDMA for the Storage NICs? I'm thinking to use iWARP RDMA as I read MSFT engineers recommend/prefer iWARP over RoCE. Please confirm my understanding.

check the documenantion, i personally suggest to use ROCV2 over iWARP

(8) For the Mgmt NIC, first I will configure the SET then I will assign the IP address to the SET NIC, just like we used to with the conventional NIC Teaming in Windows Server 2012 R2.

First assign IP Address and it will create NIC after that

[#9]

Hi there. iWARP or RoCE depends on the NIC vendor. I recommend to avoid Intel NICs, and Broadcom. The best experience will be Mellanox CX5 and newer with and RoCE.

you can use sconfig to set static NIC addresses, Gateway and DNS. Unfortunately this is a one time experience as sconfig has an issue when you try to change the IP address or gateway. it works only great if the NIC has never been configured (DHCP / default)

[#10]

@HasanHasib 
1. assign IP, Gateway on your first management NIC (lowest MACadress) on each nodes
2. for management NICs (for non-converged), and storage NICs (fully converged)
make sure that SR-IOV is in your UEFI and VF are set according to your vendors documentation.  {127 instead 8 (default value)} these settings in UEFI are per NIC, not per NIC Port
reboot

3. use NetworkATC to finalize your basic configuration

install-module networkATC

#assume that mgmt traffic is VLAN 0 or untagged, and you don't want to use RDMA for compute / management, otherwise you need tagged VLAN for RDMA

#your pair of management NIC Ports, they should not be both on the same NIC for true redundancy, e.g. NIC FW updates
$Compute_Adapters = @('Slot 2 Port 1','Slot 7 Port 1') 

$compute_vlan = @(0)

Add-NetIntent -Name Compute -Management -AdapterName $Compute_Adapters -Verbose -StorageVlans $compute_vlan


Strictly avoid to create manual SET switches this and also WAC uses different options that may not be optimal, especially if you care for SR-IOV.

after NetworkATC has created SET Switch "convergedSwitch(Compute)"

and the OS NIC "vEthernet(Compute)"

Check if everything is alright with SR-IOV 

Get-VMSwitch *compute* | select *iov* 
creating the SET Switch it will always use the first NIC in the pair above to take over the IP etc.

example

Name : ConvergedSwitch(compute)
IovEnabled : True
IovSupport : True
IovSupportReasons :
IovQueuePairCount : 130558
IovQueuePairsInUse : 64
IovVirtualFunctionCount : 254
IovVirtualFunctionsInUse : 0

There should be functions available. if it is not true, then there is something wrong in the UEFI settings. In this case you have to remove the intent and the vSwitch and start over.

If you do not need SR-IOV then it is easier. you do not have to alter UEFI NIC settings and can WAC can be used, too.