Easy Networking with Vagrant and VirtualBox

From Wiki4Intranet
Jump to: navigation, search
(Created page with "When we are playing with vagrant boxes on VirtualBox, we should use three network adapters for each box. What, why, what problems and how to deal with it? Lets discuss. === N...")
 
(:)
Line 54: Line 54:
 
;Contra:  
 
;Contra:  
 
* For intercommunication of vboxes private network more useful than public. Because we can manage names/IP-addresses and vbox IPs are independent from unstable DHCP. So public network should be addition, not substitution of private one.
 
* For intercommunication of vboxes private network more useful than public. Because we can manage names/IP-addresses and vbox IPs are independent from unstable DHCP. So public network should be addition, not substitution of private one.
* For bridging with Vagrant we have to know external network adapter name. Yes, it possible to specify Vagrant several network adapters, but impossible to use regexps or wildcards. So impossible to enoumerate
+
* For bridging with Vagrant we have to know external network adapter name. Yes, it possible to specify Vagrant several network adapters, but impossible to use regexps or wildcards. So impossible to enumerate all this stuff<ref>Diffirent rules of different linux distros</ref> like "eth0", "eth41", «enp0s31f6». Same problem with initializing packer-boxes.
* Для бриджевания, надо знать название адаптера. Ну, на самом деле там можно предложнить на выбор несколько названий адаптеров, но сейчас же в каждом дистрибутиве линукса свой принцип формирования имен. И он динамический! Нельзя использовать wildcardы, сказать, блин, забриджуй меня хоть с каким-нибудь первым попавшемся ''работающем'' ethernetом, будь он хоть "eth0", хоть "eth31", хоть «enp0s31f6»<ref>Это вот у меня сейчас тут</ref>, а ведь еще винда у некоторых… Аналогичная проблема при разворачивании сделанных packerом образов на машинах пользователей — там она тоже будет настойчиво просить выбрать адаптер.  
+
* By default, vagrant-born boxes, even with public network adapter not seen from other network serments. That is if vbox have IP=10.200.9.70, it will be seen from 10.200.9.nnn, but unreacheable from 10.200.8.*. WTF??!!
* По умолчанию, созданные vagrant-ом виртуалки, даже забриджованные с реальным сетевым адаптером, почему-то не видны из других сегментов локальной сети! Т.е. если вдруг достался IPшник 10.200.9.70, то виртуалка будет видна из 10.200.9.nnn, но не видна даже из 10.200.8.*.  
+
  
WTF??!!
+
== Solution ==
 +
 
 +
What adapter is best, and how to make all work with couple lines of code without hardcoding IPs and such stuff?
 +
I did not see such solution before.
 +
 
 +
How I make it: we should take all of them and fix their problems.
 +
 
 +
;NAT: Let it be. But only for «vagrant ssh» and «vagrant provision».
 +
 
 +
;Private: We have to dispatch IP-addresses and make something like DNS.
 +
First, lets, install vagrant plugin (from user, not root):
 +
  vagrant plugin install vagrant-hosts
 +
This plugin will authomatically update /etc/hosts for all all boxes from same Vagrantfile.
 +
 
 +
But how we can get unique IPs for each host without some DHCP server or monkey hardcoding?
 +
Let's use CryptoPower — using hash from hostnames, in potential 254 slots we can pack few dozens hash-generated IPs without collisions.
 +
<source lang="ruby">
 +
    start_port = (Digest::SHA256.hexdigest conf.vm.hostname)[1..4].to_i(16)+1000
 +
    subip = (Digest::SHA256.hexdigest conf.vm.hostname)[0..1].to_i(16) % 250 + 2
 +
    conf.vm.network "private_network", ip: "192.168.4.#{subip}"
 +
</source>
 +
 
 +
How to reach v-boxes from host by name?
 +
Unfortunatelly, my solution now not fully authomated:
 +
We have to get list of mapping «name←→IP»:
 +
<code-bash>
 +
>vagrant hosts list                                                                                                               
 +
192.168.4.132 tms-ubuntu.local.com tms-ubuntu                                                                                                                               
 +
192.168.4.110 tms-db-centos-1.local.com tms-db-centos-1                                                                                                                     
 +
192.168.4.212 tms-centos.local.com tms-centos                                                                                                                               
 +
192.168.4.104 tms-ubuntu-new.local.com tms-ubuntu-new                                                                                                                       
 +
192.168.4.62 tms-ubuntu-latest.local.com tms-ubuntu-latest                                                                                                                 
 +
192.168.4.216 tms-monitoring.local.com tms-monitoring                                                                                                                       
 +
192.168.4.97 tms-db-centos-3.local.com tms-db-centos-3                                                                                                                     
 +
192.168.4.166 rhel4packages.local.com rhel4packages                                                                                                                         
 +
192.168.4.21 tms-db-centos-2.local.com tms-db-centos-2                                                                                                                     
 +
192.168.4.226 tms-ubuntu-bridgebag.local.com tms-ubuntu-bridgebag       
 +
</code-bash>
 +
 
 +
Then, arrr, we should paste it to /etc/hosts, from root
 +
sudo mcedit /etc/hosts
 +
 
 +
Very pity, that this can not be done without root rights, so this cannot be automated, without violation of princip "vagrant needs only user rights".
 +
Unfortunately, Linux has no "user level hosts files". Yes, many people wants [https://unix.stackexchange.com/questions/10438/can-i-create-a-user-specific-hosts-file-to-complement-etc-hosts this], and even exists [https://github.com/bAndie91/libnss_homehosts some interesting forks of libnss], but it is not standard it is unlikely that something will change in the near future.
 +
 
 +
But if you know right way — please, [mailto:stas-fomin@yandex.ru let me know].
 +
 
 +
;Public network:
 +
First problem — smartbridging. Lets talk with Virtualbox about avalable bridged interfaces and select adapter from prioritized list of regexp:
 +
<code-python>
 +
    preferred_interfaces = ['eth0.*', 'enp0s.*']
 +
    host_interfaces = %x( VBoxManage list bridgedifs | grep ^Name ).gsub(/Name:\s+/, '').split("\n")
 +
    network_interface_to_use = preferred_interfaces.map{ |pi| host_interfaces.find { |vm| vm =~ /#{Regexp.new(pi)}/ } }.compact[0]
 +
    conf.vm.network "public_network", bridge: network_interface_to_use
 +
</code-python>
 +
 
 +
Why is the problem from accessing from other network segments:
 +
*  Vagrant hardcoded, that NAT-adapter always first (or no NAT, we we force another adapter be first). All bugs about it — wontfix.
 +
** On start/reload/etc first (usually NAT) adapter maked default on routing. So if packets goes from another segment, they actually reach a v-box, but all responses routed to NAT → /dev/null. [https://github.com/mitchellh/vagrant/issues/2389]
 +
 
 +
We should fix it without some additional scripts/settings inside VM. Without hardcoding IPs or vendorlocking to some distros.
 +
 
 +
My solution — lets Vagrant fix all that it breaks.
 +
Lets assume, that last interface will be public,  and make small script on python (no bash-awk unreadable magic) inside ruby-based Vagrant file.
 +
Lets assume, that distro modern enough — have "ip"-utility and python.
 +
 
 +
<source lang="ruby">
 +
def set_network(conf)
 +
    default_host_gw = `ip route show`[/default.*/][/\d+\.\d+\.\d+\.\d+/]
 +
 
 +
    start_port = (Digest::SHA256.hexdigest conf.vm.hostname)[1..4].to_i(16)+1000
 +
    subip = (Digest::SHA256.hexdigest conf.vm.hostname)[0..1].to_i(16) % 250 + 2
 +
    conf.vm.network "private_network", ip: "192.168.4.#{subip}"
 +
 
 +
    preferred_interfaces = ['eth0.*', 'eth\d.*', 'enp0s.*', 'enp\ds.*']
 +
    host_interfaces = %x( VBoxManage list bridgedifs | grep ^Name ).gsub(/Name:\s+/, '').split("\n")
 +
    network_interface_to_use = preferred_interfaces.map{ |pi| host_interfaces.find { |vm| vm =~ /#{Regexp.new(pi)}/ } }.compact[0]
 +
    conf.vm.network "public_network", bridge: network_interface_to_use #, adapter: "1"
 +
 
 +
    python_script_4_routing = <<-PYTHON
 +
#!/usr/bin/env python
 +
import commands
 +
 
 +
output = commands.getstatusoutput('ip route')
 +
maybedefault = output[1].splitlines()[2]
 +
terms = maybedefault.split()
 +
first = terms[0]
 +
last = terms[-1]
 +
 
 +
output = commands.getoutput('ip -o link show')
 +
lastinterface = output.splitlines()[-1].split(':')[1].strip()
 +
if first != 'default' or last != lastinterface:
 +
      print "Fixing routing from %s to %s " % (last, lastinterface)
 +
      scmd = 'ip route add default via #{default_host_gw} dev %s' % lastinterface
 +
      commands.getstatusoutput('ip route del default')
 +
      commands.getstatusoutput(scmd)
 +
 
 +
PYTHON
 +
 
 +
    conf.vm.provision "shell", run: "always", inline: python_script_4_routing
 +
end
 +
 
 +
def set_std_provision(conf)
 +
    if Vagrant.has_plugin?("vagrant-hosts")
 +
      conf.vm.provision :hosts, :sync_hosts => true   
 +
    end 
 +
    set_network(conf)     
 +
end
 +
 
 +
# Add lines above ↑↑↑ to beginning of Vagrantfile
 +
 
 +
# Below ↓↓↓ sample of some V-box initialization.
 +
 
 +
DOMAIN = "local.com"
 +
UBUNTU_BOX = "bento/ubuntu-17.04"
 +
 
 +
# Compact vbox description:
 +
 
 +
Vagrant.configure(2) do |config|
 +
  config.vm.provider "virtualbox" do |conf|
 +
    conf.memory = "256"
 +
    conf.cpus = 2
 +
  end
 +
 
 +
  config.vm.box = UBUNTU_BOX
 +
 
 +
  config.vm.define "somevm" do |conf|
 +
    vmname = "somevm"
 +
    conf.vm.hostname = vmname  + "." + DOMAIN
 +
    set_std_provision(conf)
 +
  end
 +
end
 +
</source>
  
  

Revision as of 16:11, 20 September 2017

When we are playing with vagrant boxes on VirtualBox, we should use three network adapters for each box. What, why, what problems and how to deal with it? Lets discuss.

NAT

  • NAT-adapter, should be everyone by default.

Vagrant box with NAT

  • can see external internet
  • isolated from any other vagrant boxes
  • can be accessed outside only by portforwarding throught host box. That is "control" SSH port of guest mapped to some (2222 or other unpriviliged) port of host, 80-guest to 1080-host, etc. So website inside guest box can be accessed by http://localhost:1080.

You can ask vagrant about these ports:

>vagrant port  tms-ubuntu                                                                                                          
The forwarded ports for the machine are listed below. Please note that                                                                                                       
these values may differ from values configured in the Vagrantfile if the                                                                                                     
provider supports automatic port collision detection and resolution.                                                                                                         
 
    22 (guest) => 2206 (host)                                    

Vagrant manages SSH-ports, trying to avoid collisions, so you can always get inside guest box by «vagrant ssh» without any knowledge about port forwarding. Vagrant also used NAT-adapter and this SSH-port for provisioning-by-ssh (shell, ansible, …). But you should manage all other ports.

Pro
  • NAT-adapter always available[1];
  • Simple and enough for simple web project with one external web-port.
Contra


Private network

Internal virtual network for bunch of guest boxes inside host.

Pro
  • Here we can get several interacted boxes ("db"-role, "webapp"-role, "mock-testing"-role…) — they can see each other and external internet.
Contra
  • We have to manage networking for guest boxes:
    • IP-addressing — to give guest boxes non-intersecting IP-addresses
    • DNS — give a names to box and make that all guest boxes know each other.
  • Guest boxes can be accessed from host, but invisible outside — impossible to get access[2] to these boxes from LAN — for demo, testing, bug investigation, etc.

Public network

Actually, network bridge with one of host network adapters[3].

Pro
  • IP-address get get from DHCP-service on local, corporate or home network.
  • Can expose guest box on local network — demo, bug-hunting, even some useful public service.
Contra
  • For intercommunication of vboxes private network more useful than public. Because we can manage names/IP-addresses and vbox IPs are independent from unstable DHCP. So public network should be addition, not substitution of private one.
  • For bridging with Vagrant we have to know external network adapter name. Yes, it possible to specify Vagrant several network adapters, but impossible to use regexps or wildcards. So impossible to enumerate all this stuff[4] like "eth0", "eth41", «enp0s31f6». Same problem with initializing packer-boxes.
  • By default, vagrant-born boxes, even with public network adapter not seen from other network serments. That is if vbox have IP=10.200.9.70, it will be seen from 10.200.9.nnn, but unreacheable from 10.200.8.*. WTF??!!

Solution

What adapter is best, and how to make all work with couple lines of code without hardcoding IPs and such stuff? I did not see such solution before.

How I make it: we should take all of them and fix their problems.

NAT
Let it be. But only for «vagrant ssh» and «vagrant provision».
Private
We have to dispatch IP-addresses and make something like DNS.

First, lets, install vagrant plugin (from user, not root):

 vagrant plugin install vagrant-hosts

This plugin will authomatically update /etc/hosts for all all boxes from same Vagrantfile.

But how we can get unique IPs for each host without some DHCP server or monkey hardcoding? Let's use CryptoPower — using hash from hostnames, in potential 254 slots we can pack few dozens hash-generated IPs without collisions.

    start_port = (Digest::SHA256.hexdigest conf.vm.hostname)[1..4].to_i(16)+1000
    subip = (Digest::SHA256.hexdigest conf.vm.hostname)[0..1].to_i(16) % 250 + 2
    conf.vm.network "private_network", ip: "192.168.4.#{subip}"

How to reach v-boxes from host by name? Unfortunatelly, my solution now not fully authomated: We have to get list of mapping «name←→IP»:

>vagrant hosts list                                                                                                                
192.168.4.132 tms-ubuntu.local.com tms-ubuntu                                                                                                                                
192.168.4.110 tms-db-centos-1.local.com tms-db-centos-1                                                                                                                      
192.168.4.212 tms-centos.local.com tms-centos                                                                                                                                
192.168.4.104 tms-ubuntu-new.local.com tms-ubuntu-new                                                                                                                        
192.168.4.62 tms-ubuntu-latest.local.com tms-ubuntu-latest                                                                                                                   
192.168.4.216 tms-monitoring.local.com tms-monitoring                                                                                                                        
192.168.4.97 tms-db-centos-3.local.com tms-db-centos-3                                                                                                                       
192.168.4.166 rhel4packages.local.com rhel4packages                                                                                                                          
192.168.4.21 tms-db-centos-2.local.com tms-db-centos-2                                                                                                                       
192.168.4.226 tms-ubuntu-bridgebag.local.com tms-ubuntu-bridgebag        

Then, arrr, we should paste it to /etc/hosts, from root

sudo mcedit /etc/hosts

Very pity, that this can not be done without root rights, so this cannot be automated, without violation of princip "vagrant needs only user rights". Unfortunately, Linux has no "user level hosts files". Yes, many people wants this, and even exists some interesting forks of libnss, but it is not standard it is unlikely that something will change in the near future.

But if you know right way — please, let me know.

Public network

First problem — smartbridging. Lets talk with Virtualbox about avalable bridged interfaces and select adapter from prioritized list of regexp:

    preferred_interfaces = ['eth0.*', 'enp0s.*']
    host_interfaces = %x( VBoxManage list bridgedifs | grep ^Name ).gsub(/Name:\s+/, '').split("\n")
    network_interface_to_use = preferred_interfaces.map{ |pi| host_interfaces.find { |vm| vm =~ /#{Regexp.new(pi)}/ } }.compact[0]
    conf.vm.network "public_network", bridge: network_interface_to_use 

Why is the problem from accessing from other network segments:

  • Vagrant hardcoded, that NAT-adapter always first (or no NAT, we we force another adapter be first). All bugs about it — wontfix.
    • On start/reload/etc first (usually NAT) adapter maked default on routing. So if packets goes from another segment, they actually reach a v-box, but all responses routed to NAT → /dev/null. [1]

We should fix it without some additional scripts/settings inside VM. Without hardcoding IPs or vendorlocking to some distros.

My solution — lets Vagrant fix all that it breaks. Lets assume, that last interface will be public, and make small script on python (no bash-awk unreadable magic) inside ruby-based Vagrant file. Lets assume, that distro modern enough — have "ip"-utility and python.

def set_network(conf)
    default_host_gw = `ip route show`[/default.*/][/\d+\.\d+\.\d+\.\d+/]
 
    start_port = (Digest::SHA256.hexdigest conf.vm.hostname)[1..4].to_i(16)+1000
    subip = (Digest::SHA256.hexdigest conf.vm.hostname)[0..1].to_i(16) % 250 + 2
    conf.vm.network "private_network", ip: "192.168.4.#{subip}"
 
    preferred_interfaces = ['eth0.*', 'eth\d.*', 'enp0s.*', 'enp\ds.*']
    host_interfaces = %x( VBoxManage list bridgedifs | grep ^Name ).gsub(/Name:\s+/, '').split("\n")
    network_interface_to_use = preferred_interfaces.map{ |pi| host_interfaces.find { |vm| vm =~ /#{Regexp.new(pi)}/ } }.compact[0]
    conf.vm.network "public_network", bridge: network_interface_to_use #, adapter: "1"
 
    python_script_4_routing = <<-PYTHON
#!/usr/bin/env python
import commands
 
output = commands.getstatusoutput('ip route')
maybedefault = output[1].splitlines()[2]
terms = maybedefault.split()
first = terms[0]
last = terms[-1]
 
output = commands.getoutput('ip -o link show')
lastinterface = output.splitlines()[-1].split(':')[1].strip()
if first != 'default' or last != lastinterface:
      print "Fixing routing from %s to %s " % (last, lastinterface)
      scmd = 'ip route add default via #{default_host_gw} dev %s' % lastinterface
      commands.getstatusoutput('ip route del default')
      commands.getstatusoutput(scmd)
 
PYTHON
 
    conf.vm.provision "shell", run: "always", inline: python_script_4_routing
end
 
def set_std_provision(conf)
    if Vagrant.has_plugin?("vagrant-hosts")
      conf.vm.provision :hosts, :sync_hosts => true    
    end  
    set_network(conf)      
end
 
# Add lines above ↑↑↑ to beginning of Vagrantfile
 
# Below ↓↓↓ sample of some V-box initialization.
 
DOMAIN = "local.com"
UBUNTU_BOX = "bento/ubuntu-17.04"
 
# Compact vbox description:
 
Vagrant.configure(2) do |config|
  config.vm.provider "virtualbox" do |conf|
    conf.memory = "256" 
    conf.cpus = 2
  end
 
  config.vm.box = UBUNTU_BOX 
 
  config.vm.define "somevm" do |conf|
    vmname = "somevm"
    conf.vm.hostname = vmname  + "." + DOMAIN
    set_std_provision(conf)
  end
end


References

  • Can be disabled if we specify "adapter=1" in specification of any other guest network adapter. But why?
  • Without ssh-tunnels-portforwarding etc
  • Usually with ethernet-adapter, because most desktop has no WiFi, and most WiFi-adapter cannot be bridged.
  • Diffirent rules of different linux distros