Easy Networking with Vagrant and VirtualBox

Revision as of 16:26, 20 September 2017

When we are playing with vagrant boxes on VirtualBox, we should use three network adapters for each box. What, why, what problems and how to deal with it? Lets discuss.

NAT

• NAT-adapter, should be used by default.

Vagrant box with NAT

• can see external internet
• isolated from any other vagrant boxes
• can be accessed outside only by portforwarding throught host box. That is "control" SSH port of guest mapped to some (2222 or other unpriviliged) port of host, 80-guest to 1080-host, etc. So website inside guest box can be accessed by http://localhost:1080.

You can ask vagrant about these ports:

>vagrant port  project-ubuntu                                                                                                          
The forwarded ports for the machine are listed below. Please note that                                                                                                       
these values may differ from values configured in the Vagrantfile if the                                                                                                     
provider supports automatic port collision detection and resolution.                                                                                                         
 
    22 (guest) => 2206 (host)                                    

Vagrant manages SSH-ports, trying to avoid collisions, so you can always get inside guest box by «vagrant ssh» without any knowledge about port forwarding. Vagrant also used NAT-adapter and this SSH-port for provisioning-by-ssh (shell, ansible, …). But you should manage all other ports.

Pro

• NAT-adapter always available^[1];
• Simple and enough for simple web project with one external web-port.

Contra

• If we have several working ports on guest, we will be tired what and where → T.e http://localhost:1488, http://localhost:31415, http://localhost:4242 instead of good old «http://experimental.local.com», «http://stable.local.com», «http://myhomepetprojectidoonwork.local.com»
• If we have to get several interacted boxes ("db"-role, "webapp"-role, "mock-testing"-role…) — no way.

Private network

Internal virtual network for bunch of guest boxes inside host.

Pro

• Here we can get several interacted boxes ("db"-role, "webapp"-role, "mock-testing"-role…) — they can see each other and external internet.

Contra

• We have to manage networking for guest boxes:
  • IP-addressing — to give guest boxes non-intersecting IP-addresses
  • DNS — give a names to box and make that all guest boxes know each other.
• Guest boxes can be accessed from host, but invisible outside — impossible to get access^[2] to these boxes from LAN — for demo, testing, bug investigation, etc.

Public network

Actually, network bridge with one of host network adapters^[3].

Pro

• IP-address get get from DHCP-service on local, corporate or home network.
• Can expose guest box on local network — demo, bug-hunting, even some useful public service.

Contra

• For intercommunication of vboxes private network more useful than public. Because we can manage names/IP-addresses and vbox IPs are independent from unstable DHCP. So public network should be addition, not substitution of private one.
• For bridging with Vagrant we have to know external network adapter name. Yes, it possible to specify Vagrant several network adapters, but impossible to use regexps or wildcards. So impossible to enumerate all this stuff^[4] like "eth0", "eth41", «enp0s31f6». Same problem with initializing packer-boxes.
• By default, vagrant-born boxes, even with public network adapter not seen from other network serments. That is if vbox have IP=10.200.9.70, it will be seen from 10.200.9.nnn, but unreacheable from 10.200.8.*. WTF??!!

Solution

What adapter is best, and how to make all work with couple lines of code without hardcoding IPs and such stuff? I did not see such solution before.

How I make it: we should take all of them and fix their problems.

NAT

Let it be. But only for «vagrant ssh» and «vagrant provision».

Private

We have to dispatch IP-addresses and make something like DNS.

First, lets, install vagrant plugin (from user, not root):

 vagrant plugin install vagrant-hosts

This plugin will authomatically update /etc/hosts for all all boxes from same Vagrantfile.

But how we can get unique IPs for each host without some DHCP server or monkey hardcoding? Let's use CryptoPower — using hash from hostnames, in potential 254 slots we can pack few dozens hash-generated IPs without collisions.

    start_port = (Digest::SHA256.hexdigest conf.vm.hostname)[1..4].to_i(16)+1000
    subip = (Digest::SHA256.hexdigest conf.vm.hostname)[0..1].to_i(16) % 250 + 2
    conf.vm.network "private_network", ip: "192.168.4.#{subip}"

How to reach v-boxes from host by name? Unfortunatelly, my solution now not fully authomated: We have to get list of mapping «name←→IP»:

>vagrant hosts list                                                                                                                
192.168.4.132 project-ubuntu.local.com project-ubuntu                                                                                                                                
192.168.4.110 project-db-centos-1.local.com project-db-centos-1                                                                                                                      
192.168.4.212 project-centos.local.com project-centos                                                                                                                                
192.168.4.104 project-ubuntu-new.local.com project-ubuntu-new                                                                                                                        
192.168.4.62 project-ubuntu-latest.local.com project-ubuntu-latest                                                                                                                   
192.168.4.216 project-monitoring.local.com project-monitoring                                                                                                                        
192.168.4.97 project-db-centos-3.local.com project-db-centos-3                                                                                                                       
192.168.4.166 rhel4packages.local.com rhel4packages                                                                                                                          
192.168.4.21 project-db-centos-2.local.com project-db-centos-2                                                                                                                       
192.168.4.226 project-ubuntu-bridgebag.local.com project-ubuntu-bridgebag        

Then, arrr, we should paste it to /etc/hosts, from root

sudo mcedit /etc/hosts

Very pity, that this can not be done without root rights, so this cannot be automated, without violation of princip "vagrant needs only user rights". Unfortunately, Linux has no "user level hosts files". Yes, many people wants this, and even exists some interesting forks of libnss, but it is not standard it is unlikely that something will change in the near future.

But if you know right way — please, let me know.

Public network

First problem — smartbridging. Lets talk with Virtualbox about avalable bridged interfaces and select adapter from prioritized list of regexp:

    preferred_interfaces = ['eth0.*', 'eth\d.*', 'enp0s.*', 'enp\ds.*']
    host_interfaces = %x( VBoxManage list bridgedifs | grep ^Name ).gsub(/Name:\s+/, '').split("\n")
    network_interface_to_use = preferred_interfaces.map{ |pi| host_interfaces.find { |vm| vm =~ /#{Regexp.new(pi)}/ } }.compact[0]
    conf.vm.network "public_network", bridge: network_interface_to_use 

Why is the problem from accessing from other network segments:

• Vagrant hardcoded, that NAT-adapter should always be first (or no NAT, we we force another adapter be first). All bugs about it — wontfix.
  • On start/reload/etc first (usually NAT) adapter maked default on routing. So if packets goes from another segment, they actually reach a v-box, but all responses routed to NAT → /dev/null. [1]

We should fix it without some additional scripts/settings inside VM. Without hardcoding IPs or vendorlocking to some distros.

My solution — lets Vagrant fix all that it breaks. Lets assume, that last interface will be public, and make small script on python (no bash-awk unreadable magic) inside ruby-based Vagrant file. Lets assume, that distro modern enough — have "ip"-utility and python.

def set_network(conf)
    default_host_gw = `ip route show`[/default.*/][/\d+\.\d+\.\d+\.\d+/]
 
    start_port = (Digest::SHA256.hexdigest conf.vm.hostname)[1..4].to_i(16)+1000
    subip = (Digest::SHA256.hexdigest conf.vm.hostname)[0..1].to_i(16) % 250 + 2
    conf.vm.network "private_network", ip: "192.168.4.#{subip}"
 
    preferred_interfaces = ['eth0.*', 'eth\d.*', 'enp0s.*', 'enp\ds.*']
    host_interfaces = %x( VBoxManage list bridgedifs | grep ^Name ).gsub(/Name:\s+/, '').split("\n")
    network_interface_to_use = preferred_interfaces.map{ |pi| host_interfaces.find { |vm| vm =~ /#{Regexp.new(pi)}/ } }.compact[0]
    conf.vm.network "public_network", bridge: network_interface_to_use #, adapter: "1"
 
    python_script_4_routing = <<-PYTHON
#!/usr/bin/env python
import commands
 
output = commands.getstatusoutput('ip route')
maybedefault = output[1].splitlines()[2]
terms = maybedefault.split()
first = terms[0]
last = terms[-1]
 
output = commands.getoutput('ip -o link show')
lastinterface = output.splitlines()[-1].split(':')[1].strip()
if first != 'default' or last != lastinterface:
      print "Fixing routing from %s to %s " % (last, lastinterface)
      scmd = 'ip route add default via #{default_host_gw} dev %s' % lastinterface
      commands.getstatusoutput('ip route del default')
      commands.getstatusoutput(scmd)
 
PYTHON
 
    conf.vm.provision "shell", run: "always", inline: python_script_4_routing
end
 
def set_std_provision(conf)
    if Vagrant.has_plugin?("vagrant-hosts")
      conf.vm.provision :hosts, :sync_hosts => true    
    end  
    set_network(conf)      
end
 
# Add lines above ↑↑↑ to beginning of Vagrantfile
 
# Below ↓↓↓ sample of some V-box initialization.
 
DOMAIN = "local.com"
UBUNTU_BOX = "bento/ubuntu-17.04"
 
# Compact vbox description:
 
Vagrant.configure(2) do |config|
  config.vm.provider "virtualbox" do |conf|
    conf.memory = "256" 
    conf.cpus = 2
  end
 
  config.vm.box = UBUNTU_BOX 
 
  config.vm.define "somevm" do |conf|
    vmname = "somevm"
    conf.vm.hostname = vmname  + "." + DOMAIN
    set_std_provision(conf)
  end
end

References

• Official Virtualbox documentation on networking
• Networking on Vagrant