Simulate Prolog backtracking using Scala Stream

    21 Oct 2016

    It’s a kind of interesting problem. For a bunch of algorithms or data structures, it is fairly easy and very straight forward using Prolog to solve. And with backtracking, it is just a matter of ; to get next solution and findall to get everything back, or even findnsols to get at most N solutions.

    When it comes to Scala (or basically any other non-declarative language), things are a bit of complicated. The fundamental reason is one has to explicitly think about how to compute all solutions.

    As an example, let’s take a look at one of the p-99 problems described here, 6.02.

    One of the possible Prolog solutions could be:

    path(G, A, B, P) :-
        path(G, A, B, [A], P0),
        reverse(P0, P).
    path(_, A, A, P, P) :- !.
    path(G, A, B, P0, P) :-
        neighbour(G, A, N),
        \+ memberchk(N, P0),
        path(G, N, B, [N|P0], P).
    
    neighbour(graph(_, E), A, N) :-
        member(e(A, N), E);
        member(e(N, A), E).
    

    The above snippet focuses on how to compute one and only one solution and the backtracking point is generated by member and ;.

    One can then ask Prolog to compute next, next next solution, …

    What would be a Scala simulation of this kind of lazy computation? Naturally Stream.

    def neighbours[T](graph: Graph[T], node: T) = {
      if (!graph.nodes.contains(node)) Nil
      else
        graph.edges.collect {
          case (x, neighbour) if (x == node) => neighbour
          case (neighbour, x) if (x == node) => neighbour
        }
    }
    
    def path[T](graph: Graph[T], from: T, to: T) = {
      def path0[S](graph: Graph[S], from: S, to: S, 
                   visited: List[S]): Stream[List[S]] = {
        if (from == to) Stream(List(to))
        else
          neighbours(graph, from).toStream.filterNot(visited.contains(_)).flatMap { x =>
            path0(graph, x, to, from :: visited).map(from :: _)
        }
      }
    
      path0(graph, from, to, Nil)
    }
    

    Of course Stream doesn’t ease at all the complexity because one still needs to think about all solutions explicitly as shown by flatMap. But, it does the lazy computation, because after computing all neighbours of a node, a Stream is generated and whatever after that is computed lazily.

    As all the other Scala collections, Stream has a lot of functions one can then use to retrieve solutions.

    As I went though p-99 using Scala, this pattern appears over and over again, so it is worth to document.

    Read More

    PXE on Synology

    14 Sep 2016

    As my home network grows more and more complex, I need to find something to make it even more complex. Yes, why not setting up a PXE server? I actually have and may have more Linux boxes to install.

    While there are many articles talking about setting up PXE on Synolgoy NAS, none of them can fit my requirement, that is using the most complicated way.

    Well I’m lying. I actually want the most simple and flexible approach, which is using Cobbler.

    The Synolgoy NAS I have runs on x86, which means it has the capability to run Docker properly, and the good news is there are a few existing Cobbler Docker images on Docker hub. But unfortunatley none of them is properly made.

    Then this article is quite helpful except that it doesn’t work, at least not very much. The problem is tftp server simply cannot work without specifying --net host when creating the container.

    Without --net host, Docker will use iptables and docker-proxy to do NAT to containers, and the way tftp server works is similar to the active mode ftp server, which means it initiates connection from an ephemeral port toward the client and of course this won’t work under NAT.

    OK. I start from that article to make my own customization, so here it is: docker-cobbler.

    The Dockerfile is pretty straight forward, installing cobbler, cobbler-web, pykickstart (for cobbler validateks), exposing related port. Since systemd refuses to work on my NAS (lower version of Docker, maybe?), I need to start httpd and cobbler manually.

    So, now the problem is how about tftp server. Fortunately I can run it on my NAS directly by following the official document.

    I create a shared folder as TFTP root folder, but it’s not necessary and it’s up to you. But that folder should be mapped to a cobbler container volume in order for cobbler to access, for example -v /volume1/tftpboot:/var/lib/tftpboot.

    ./build.sh will create the image, and ./start.sh will start a container using the image.

    But before that, let’s talk about networking. Did I mention I have a pfSense firewall at home?

    pfSense manages DHCP for my home network, so first all I need to configure next server and Default BIOS file name. next server points to my NAS where tftp is running.

    I assign a virtual IP address to pfSense and port forward that to my NAS, so that I can use that IP address to access exported ports from the container.

    Note that there is a “black hole” that doesn’t exist in my home network to handle all the other ports, otherwise accessing the virutal IP will bring you to pfSense. You can of course create firewall rules to further block everything else. As a side note, pfSense does NAT first and then firewall. In the above screenshots, 192.168.17.4 leads us to cobbler.

    Since I don’t use cobbler to manage anything, in settings file I disable everything. It is very important to set server: 192.168.17.4 because this is how cobbler can be accessed by client, and this IP is used to generate files under pxelinux.cfg. Super super important!

    That is pretty much everything, hopefully.

    Read More

    pfSense as a proper router in OpenStack

    24 Nov 2015

    Well, this is a little bit cheating because I started with an already made pfSense image by my colleague. Nevertheless, it’s still a long story.

    Fix Disk Error

    When making the image, UUID wasn’t used in fstab, so when booting the instance it will complain about disk not found. This is easy to solve. Type ? and it will show you all available devices. Then type in whatever is corrrect and in my case is ufs:vtbd0s1a. After successfully booted, modify /etc/fstab accordingly for both / and swap.

    Enable the GUI

    Next step is to temporarily assign a floating IP address to pfSense and disable firewall by pfctl -d from Shell, then you will be able to access the GUI, but that’s not end of the story because you will probably see this error:

    An HTTP_REFERER was detected other than what is defined in System -> Advanced (https://213.159.184.84/).
    You can disable this check if needed in System -> Advanced -> Admin.
    

    After a bit of googling, here is the solution, in general you need to add two lines to /conf/config.xml:

    <webgui>
      <nodnsrebindcheck/>
      <nohttpreferercheck/>
    </webgui>
    

    After that, delete /tmp/config.cache and restart webConfigurator. Try again and voila, you are in!

    Continue to Tailor pfSense

    Setting up WAN, LAN, VPN, etc. All the usual suspects. In my case I have WAN and two LANs named LAN and OPT1. After everything has been set up, you probably want to change /conf/config.xml back and restart pfSense. Note that almost for every change you made, you probably have to re-execute pfctl -d to disable firewall because pfSense is indeed very secure.

    If you find yourself want to access GUI via WAN again, redo the modification and try enableallowallwan from pfSense Developer Shell, and execute pfctl -d from Shell. But at this point you should really have disabled WAN access because it’s not secure. To be honest I’m still not 100% sure how it works, so random clicks (meaning trying the above steps combinatorially) may do the job.

    Let’s start hacking

    My requirement is to use pfSense to route packets between two subnets and between subnets and another network because I need full firewall control. That means I’m not going to use neutron router as gateway.

    
    +----+            +----+               +----+       +----+
    |    |            |    |               |    |       |    |
    |    |            |    |               |    |       |    |
    |    |            |    |               |    |       |    |
    |    |            |    |               |    |       |    |
    |    |            |    |               |    |       |    |
    | E  |            | I  |               |    |       |    |
    | X  |            | N  |               |    |       |    |
    | T  |            | T  |  +---------+  | L  |       | O  |
    | E  |            | E  |  |         +--+ A  |       | P  |
    | R  |            | R  +--+ pfSense |  | N  |       | T  |
    | N  |            | N  |  |         +--+----+-------+ 1  |
    | A  |            | A  |  +---------+  |    |       |    |
    | L  |            | L  |               |    |       |    |
    |    | +--------+ |    |               |    |       |    |
    | N  | |        | | N  |               |    |       |    |
    | W  +-+ Router +-+ W  |               |    |       |    |
    |    | |        | |    |               |    |       |    |
    |    | +--------+ |    |               |    |       |    |
    |    |            |    |               |    |       |    |
    |    |            |    |               |    |       |    |
    +----+            +----+               +----+       +----+
                   10.0.0.0/24       192.168.101.0/24  192.168.102.0/24
    
    

    So in my case pfSense will be the gateway for LAN and OPT1. To enable this, modify both subnets in OpenStack by disabling gateway and put static host routes for example 0.0.0.0/0,192.168.101.2 where 192.168.101.2 is the LAN IP address of pfSense. This will force any virtual machines attached to LAN to use pfSense as default gateway. Note that you cannot simply put 192.168.101.2 as the gateway IP because this IP is from allocation pool and neutron doesn’t allow that.

    Another important thing here is both LAN and OPT1 should be set using static IP address instead of DHCP, because otherwise they will also pick up the host routes and a loop will be created that no virtual machine can access external network (somehow external IP address will be treated as a local one and ARP is fired to get MAC address which is obviously wrong!).

    Next is to enable routing between two subnets using pfSense. This requires a little bit hack according to this. Basically executing the following commands for both ports of pfSense.

    neutron port-update <LAN_pfsense_uuid> --allowed-address-pairs
    type=dict list=true mac_address=<MAC_LAN_pfsense>,ip_address=0.0.0.0/0
    
    neutron port-update <OPT1_pfsense_uuid> --allowed-address-pairs
    type=dict list=true mac_address=<MAC_OPT1_pfsense>,ip_address=0.0.0.0/0
    

    So far so good, but when booting up a virtual machine, you will have trouble retrieving metadata from neutron. Neutron starts a Python process for each router serving metadata for virtual machines attached to it.

    We will need to create two routers and one of them attached to LAN and other to OPT1. We don’t want them to really route anything so we will not attach any of them to external network in this case.

    To elaborate more of this:

    Link-local IP address 169.254.169.254 is used by cloud-init to retrieve metadata. To understand how it works you can check it here. Basically a prerouting entry is defined under the router’s namespace and that will direct the request towards 169.254.169.254 to the Python process which is listening on some port.

    Then we have to go back to subnet configuration to add more host routes, for example 169.254.0.0/16,192.168.101.1 where 192.168.101.1 is the IP address of the router.

    Now at this point, everything should be working and it’s time to add fine grained firewall rules and that part I will not cover.

    To Summarize

    Neutron/OpenStack is not supposed to be used like what’s described here so the whole thing is quite a hack:

    • First of all we need to have a proper pfSense image for OpenStack;
    • then fix disk error followed by some “standard” hacks to bootstrap pfSense;
    • after that enable routing through pfSense by disabling gateway and using customized host routes
    • finally fix metadata service by creating dangling router and adding additionl customized host routes to route link-local request to neutron router

    Read More

    Grunt Specify Browser

    08 Oct 2014

    It’s weird that not many people searching for this question: How can I choose other browser than the default one to open the served page?

    I found only one here.

    Although this has been addressed by grunt-contrib-connect, but obviously there is a mistake in README that they put it to wrong place: instead of under connect.options, it’s actually under connect.livereload.options.

    connect: {
          ...
      options: {
        port: 9005,
        // Change this to '0.0.0.0' to access the server from outside.
        hostname: 'localhost',
        livereload: 35729
      },
      livereload: {
        options: {
          open: {
            appName: 'Google Chrome'
          },
        }
      ...
      }
    }
    

    Read More

    fencevbox

    16 Feb 2014

    As a continuation of previous article introducing Cobbler inside VirtualBox, this one will talk about power management.

    First of all, as a principle of no reinventing of the wheel, here is the fence agent for VirtualBox. But there are a few bugs which are slightly annoying, so I clone it and make it on Github.

    I suppose you already have Perl execution environment, then just follow the steps below:

    1. Download either from the original repository or from my cloned one;
    2. Copy fence_vbox to /usr/local/sbin;
    3. cobbler sync or even service cobbler restart and service apache2 restart;
    4. Create a file named fence_vbox.template under /etc/cobbler/power/ with content: vmname=$power_id vboxhost=$power_address login=$power_user action=$power_mode
    5. In Systems configuration of Cobbler Web UI, fill in following info: Power Management IP address 10.0.2.2 here shows that NAT is used and this IP points to your host;
    6. As this Perl script uses simple ssh login to interact with VBoxManage, it will prompt for password, so you need to generate public key for root user inside the virtual machine and put it to the user’s home of the host machine, which is $HOME/.ssh/authorized_keys, and make sure to do an ssh login at least once to accept your host machine as a “known host”;
    7. Then run cobbler system powerstatus --name=<name of the system>, you should be able to check the status.

    BTW, I do think Cobbler does not respect FenceAgentAPI, because powerstatus checking should result return value as 2 as specified by the API spec, but Cobbler keeps checking again and again and finally gets timeout, and if I put the return value as 0, it is then OK.

    Read More