Easy Networking with Vagrant and VirtualBox
When we are playing with vagrant boxes on VirtualBox, we should use three network adapters for each box. What, why, what problems and how to deal with it? Lets discuss.
NAT
- NAT-adapter, should be used by.
Vagrant box with NAT
- can see external internet
- isolated from any other vagrant boxes
- can be accessed outside only by portforwarding throught host box. That is "control" SSH port of guest mapped to some (2222 or other unpriviliged) port of host, 80-guest to 1080-host, etc. So website inside guest box can be accessed by http://localhost:1080.
You can ask vagrant about these ports:
>vagrant port tms-ubuntu The forwarded ports for the machine are listed below. Please note that these values may differ from values configured in the Vagrantfile if the provider supports automatic port collision detection and resolution. 22 (guest) => 2206 (host)
Vagrant manages SSH-ports, trying to avoid collisions, so you can always get inside guest box by «vagrant ssh» without any knowledge about port forwarding. Vagrant also used NAT-adapter and this SSH-port for provisioning-by-ssh (shell, ansible, …). But you should manage all other ports.
- Pro
- NAT-adapter always available[1];
- Simple and enough for simple web project with one external web-port.
- Contra
- If we have several working ports on guest, we will be tired what and where → T.e http://localhost:1488, http://localhost:31415, http://localhost:4242 instead of good old «http://experimental.local.com», «http://stable.local.com», «http://myhomepetprojectidoonwork.local.com»
- If we have to get several interacted boxes ("db"-role, "webapp"-role, "mock-testing"-role…) — no way.
Private network
Internal virtual network for bunch of guest boxes inside host.
- Pro
- Here we can get several interacted boxes ("db"-role, "webapp"-role, "mock-testing"-role…) — they can see each other and external internet.
- Contra
- We have to manage networking for guest boxes:
- IP-addressing — to give guest boxes non-intersecting IP-addresses
- DNS — give a names to box and make that all guest boxes know each other.
- Guest boxes can be accessed from host, but invisible outside — impossible to get access[2] to these boxes from LAN — for demo, testing, bug investigation, etc.
Public network
Actually, network bridge with one of host network adapters[3].
- Pro
- IP-address get get from DHCP-service on local, corporate or home network.
- Can expose guest box on local network — demo, bug-hunting, even some useful public service.
- Contra
- For intercommunication of vboxes private network more useful than public. Because we can manage names/IP-addresses and vbox IPs are independent from unstable DHCP. So public network should be addition, not substitution of private one.
- For bridging with Vagrant we have to know external network adapter name. Yes, it possible to specify Vagrant several network adapters, but impossible to use regexps or wildcards. So impossible to enumerate all this stuff[4] like "eth0", "eth41", «enp0s31f6». Same problem with initializing packer-boxes.
- By default, vagrant-born boxes, even with public network adapter not seen from other network serments. That is if vbox have IP=10.200.9.70, it will be seen from 10.200.9.nnn, but unreacheable from 10.200.8.*. WTF??!!
Solution
What adapter is best, and how to make all work with couple lines of code without hardcoding IPs and such stuff? I did not see such solution before.
How I make it: we should take all of them and fix their problems.
- NAT
- Let it be. But only for «vagrant ssh» and «vagrant provision».
- Private
- We have to dispatch IP-addresses and make something like DNS.
First, lets, install vagrant plugin (from user, not root):
vagrant plugin install vagrant-hosts
This plugin will authomatically update /etc/hosts for all all boxes from same Vagrantfile.
But how we can get unique IPs for each host without some DHCP server or monkey hardcoding? Let's use CryptoPower — using hash from hostnames, in potential 254 slots we can pack few dozens hash-generated IPs without collisions.
start_port = (Digest::SHA256.hexdigest conf.vm.hostname)[1..4].to_i(16)+1000 subip = (Digest::SHA256.hexdigest conf.vm.hostname)[0..1].to_i(16) % 250 + 2 conf.vm.network "private_network", ip: "192.168.4.#{subip}"
How to reach v-boxes from host by name? Unfortunatelly, my solution now not fully authomated: We have to get list of mapping «name←→IP»:
>vagrant hosts list 192.168.4.132 tms-ubuntu.local.com tms-ubuntu 192.168.4.110 tms-db-centos-1.local.com tms-db-centos-1 192.168.4.212 tms-centos.local.com tms-centos 192.168.4.104 tms-ubuntu-new.local.com tms-ubuntu-new 192.168.4.62 tms-ubuntu-latest.local.com tms-ubuntu-latest 192.168.4.216 tms-monitoring.local.com tms-monitoring 192.168.4.97 tms-db-centos-3.local.com tms-db-centos-3 192.168.4.166 rhel4packages.local.com rhel4packages 192.168.4.21 tms-db-centos-2.local.com tms-db-centos-2 192.168.4.226 tms-ubuntu-bridgebag.local.com tms-ubuntu-bridgebag
Then, arrr, we should paste it to /etc/hosts, from root
sudo mcedit /etc/hosts
Very pity, that this can not be done without root rights, so this cannot be automated, without violation of princip "vagrant needs only user rights". Unfortunately, Linux has no "user level hosts files". Yes, many people wants this, and even exists some interesting forks of libnss, but it is not standard it is unlikely that something will change in the near future.
But if you know right way — please, let me know.
- Public network
First problem — smartbridging. Lets talk with Virtualbox about avalable bridged interfaces and select adapter from prioritized list of regexp:
preferred_interfaces = ['eth0.*', 'eth\d.*', 'enp0s.*', 'enp\ds.*'] host_interfaces = %x( VBoxManage list bridgedifs | grep ^Name ).gsub(/Name:\s+/, '').split("\n") network_interface_to_use = preferred_interfaces.map{ |pi| host_interfaces.find { |vm| vm =~ /#{Regexp.new(pi)}/ } }.compact[0] conf.vm.network "public_network", bridge: network_interface_to_use
Why is the problem from accessing from other network segments:
- Vagrant hardcoded, that NAT-adapter always first (or no NAT, we we force another adapter be first). All bugs about it — wontfix.
- On start/reload/etc first (usually NAT) adapter maked default on routing. So if packets goes from another segment, they actually reach a v-box, but all responses routed to NAT → /dev/null. [1]
We should fix it without some additional scripts/settings inside VM. Without hardcoding IPs or vendorlocking to some distros.
My solution — lets Vagrant fix all that it breaks. Lets assume, that last interface will be public, and make small script on python (no bash-awk unreadable magic) inside ruby-based Vagrant file. Lets assume, that distro modern enough — have "ip"-utility and python.
def set_network(conf) default_host_gw = `ip route show`[/default.*/][/\d+\.\d+\.\d+\.\d+/] start_port = (Digest::SHA256.hexdigest conf.vm.hostname)[1..4].to_i(16)+1000 subip = (Digest::SHA256.hexdigest conf.vm.hostname)[0..1].to_i(16) % 250 + 2 conf.vm.network "private_network", ip: "192.168.4.#{subip}" preferred_interfaces = ['eth0.*', 'eth\d.*', 'enp0s.*', 'enp\ds.*'] host_interfaces = %x( VBoxManage list bridgedifs | grep ^Name ).gsub(/Name:\s+/, '').split("\n") network_interface_to_use = preferred_interfaces.map{ |pi| host_interfaces.find { |vm| vm =~ /#{Regexp.new(pi)}/ } }.compact[0] conf.vm.network "public_network", bridge: network_interface_to_use #, adapter: "1" python_script_4_routing = <<-PYTHON #!/usr/bin/env python import commands output = commands.getstatusoutput('ip route') maybedefault = output[1].splitlines()[2] terms = maybedefault.split() first = terms[0] last = terms[-1] output = commands.getoutput('ip -o link show') lastinterface = output.splitlines()[-1].split(':')[1].strip() if first != 'default' or last != lastinterface: print "Fixing routing from %s to %s " % (last, lastinterface) scmd = 'ip route add default via #{default_host_gw} dev %s' % lastinterface commands.getstatusoutput('ip route del default') commands.getstatusoutput(scmd) PYTHON conf.vm.provision "shell", run: "always", inline: python_script_4_routing end def set_std_provision(conf) if Vagrant.has_plugin?("vagrant-hosts") conf.vm.provision :hosts, :sync_hosts => true end set_network(conf) end # Add lines above ↑↑↑ to beginning of Vagrantfile # Below ↓↓↓ sample of some V-box initialization. DOMAIN = "local.com" UBUNTU_BOX = "bento/ubuntu-17.04" # Compact vbox description: Vagrant.configure(2) do |config| config.vm.provider "virtualbox" do |conf| conf.memory = "256" conf.cpus = 2 end config.vm.box = UBUNTU_BOX config.vm.define "somevm" do |conf| vmname = "somevm" conf.vm.hostname = vmname + "." + DOMAIN set_std_provision(conf) end end
[ List view ]Comments
Please login to comment.