Blog Remote solver execution with PIFOP - IFORS 2023 talk PIFOP is now an AMPL Technology Partner Optimization Server on University Premises Can you solve these 5 simple network problems? 3+1 Operations Research practical assignment ideas

Optimization Server on University* Premises

*Also valid for research labs, companies and other organizations

By Davi Doro — January 18th, 2023

image/svg+xml


For more than two years, the university shown above has been hosting an Optimization Server — or just Opt-server — in one of its high-end computers, allowing more than 200 students, teachers and researchers to run mathematical programs remotely in the university's computing facilities from inside and outside the campus using only a web browser.

The purpose of this post is to explain how other universities and institutions can do the same, and what benefits are there in doing so.

What does an Opt-server do?

Simply put, a PIFOP Opt-server is a program that executes commands entered by users in PIFOP IDE terminals. It is a process spawner that executes users' programs in the host machine — e.g. AMPL, GAMS and Python programs — and redirects the output back to the user. It is one of the vertices of the PIFOP triangle Browser IDE — Central server — Opt-server :

Opt-server diagram

For more technical details, see Introduction to Optimization Server Hosting.

Why host an Opt-server in an University?

Here are four reasons why:

  1. Install solvers in one machine and use them from anywhere: This has at least two benefits. First, it spares students and faculty from the trouble of having to setup solvers and interpreter themselves in their own machines. And second, students and faculty don't need to be in the campus to run optimization models in its computer labs.
  2. Easy and safe remote execution: Say goodbye to SSH-based remote execution solutions, which not only are hard to use but often give to the user more control over the machine than they should have.
  3. Save money by making the best use of existing computing resources in your premises: Many universities already have number-crunching machines perfectly capable of running multiple optimization models in parallel. This is especially true for models developed by students, who usually work with small to medium sized problems. This means that the university doesn't need to spend extra money on cloud infrastructure in order to enable remote code execution: it can provide this convenience to its affiliates with the resources that they already have!
  4. A single solution for every solver. Many proprietary solver vendors offer their own cloud solutions for their users, allowing them to run the solver remotely via a web platform of some kind. However, that solution will only work for that specific solver, whereas the PIFOP Opt-server is compatible with most of them — including open source ones — and does not bind you to any one in particular.

How to setup an Opt-server in an University?

In this tutorial, we are going to explain how to setup an Opt-server in a college campus in such a way that it can be used by its students and faculty to run three of the most used programs among mathematical programmers: AMPL, GAMS and Python. The AMPL and GAMS bundles usually come with a selection of proprietary and open source solvers, which you will also be able to use from PIFOP.

1. Create an Enterprise account for the University

The way that you get PIFOP for multiple students and faculty is by creating an account for the university — which is just like any other account — and upgrading it to the highly discounted Enterprise plan with subscription keys for each student and faculty member.

Before upgrading to the Enterprise plan, we recomend that you use the 7 day trial that you get for subscribing to the Advanced plan to setup and troubleshoot your Opt-server up to the step 5 of this guide. Once everything is working, then you can upgrade your account to the Enterprise plan with subscription keys. This guide is for hosting an Opt-server on the University's premises, so you can ignore the Provisioned Opt-server section at the Enterprise checkout, which is for launching an Opt-server hosted and managed by us.

If you would actually prefer a Provisioned Opt-server managed by us, take a look at this page.

2. Download and run PIFOP Opt-server on the host machine

Follow the first 5 steps of the Hosting an Opt-server guide.

3. Download and install the solvers and interpreters that you want the Opt-server to support

Here we are going to download and install the the academic versions of AMPL and GAMS — both of which come with academic licenses of commercial and free solvers — and Python 3.9.

AMPL for Courses

  1. Request the free AMPL for Courses license. Fill the form and download the license. Other kinds of licenses should work as well.
  2. Extract the downloaded .tgz file into $HOME/programs — or into another directory of your choice: 
    $ mkdir $HOME/programs
$ tar -xf ampl*.tgz -C $HOME/programs
  3. Test the installation. The following command should display AMPL's version and your license: 
    $ cd $HOME/programs/ampl*
$ ./ampl -v

GAMS Community License

  1. Request the free GAMS Community License. You will receive a license file by email. Other kinds of licenses should work as well.
  2. Download GAMS for Linux.
  3. Execute the downloaded .exe from $HOME/programs — GAMS will extracted itself inside the directory from which the installer is called: 
    $ mv ./linux_x64_64_sfx.exe $HOME/programs/
$ cd $HOME/programs
$ chmod +x ./linux_x64_64_sfx.exe
$ ./linux_x64_64_sfx.exe
  4. Save the license that you have received in your email in a file named gamslice.txt inside the extracted GAMS directory. 
    $ cd $HOME/programs/gams*
$ echo 'GAMS_Demo_license_for_xxxxxxxxxxxx_____________xxxxxxx|xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx______________________________
xxxxxxxxxx_**GAMS_Demo_license_restricted_to_non-commercial_use**
xxxxxxxxxx_______________________________________________________
xxxxxxxx_______________________________________C_DEMO____________
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_________________________________
' > ./gamslice.txt
  5. Run the gamsinst. Follow the steps to configure GAMS default solvers. 
    $ ./gamsinst
  6. Test the installation. Running ./gams should display "The installed license is valid" just before its expiration date.

Python 3 (and PIP)

Instead of using the default Python installation of your system, we highly recommend that you manually install a Python version of your choice, as explained below.
  1. Install Python 3.9 — or another version of your choice. 
    $ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt-get install python3.9
$ sudo apt-get install python3.9-distutils
  2. Manually install PIP via the get-pip.py script. 
    $ wget https://bootstrap.pypa.io/get-pip.py
$ python3.9 get-pip.py
$ python3.9 -m pip install --upgrade setuptools
  3. Optional, but recommended: create Python $HOME supporting directories to avoid potential problems for not having them in-place beforehand when trying to install Python packages from PIFOP. 
    $ mkdir -p -- $HOME/.local/lib/python3.9/site-packages
$ mkdir -p -- $HOME/.local/bin
$ mkdir -p -- $HOME/.cache/pip
  4. Test the installation. 
    $ python3.9 --version

4. Configure the Opt-server

Now we need to tell the Opt-server that we've installed on Step 2 where we have installed AMPL, GAMS and Python and configure the commands that PIFOP users will enter to execute them. We do this by modifying the config.json file inside the Opt-server directory.

Use the file below as a template configuration file for your Opt-server. We have highlighted the parts that you need to modify, which include:

  • The server-name — we've used University in this example
  • $HOME explicit path — we've used /home/joe in this example
  • The directory into which you have extracted AMPL and GAMS — we've used /home/joe/programs in this example
  • The version numbers of GAMS and Python — we've used 40.1 and 3.9 in this example

For an explanation of each configuration option, see List of Configuration Options.

The file below is only a template. Modify the highlighted entries according to your system. 
{
    "server-name" : "University",
    "credentials-file" : true,
    
    "sandboxes" : [
        {
            "name" : "general-purpose",
            
            "mount-readable" : [
                "/lib",
                "/lib64",
                "/dev/random",
                "/dev/urandom",
                "/usr/lib/x86_64-linux-gnu",
                
                "/home/joe/programs",
                
                "/usr/lib/python3.9",
                "/usr/bin/python3.9",
                "/usr/bin/python3.9:/usr/bin/python3",
                "/home/joe/.local/bin",
                "/home/joe/.local/lib/python3.9/site-packages"
            ],
            
            "mount-writable" : [
                "/dev/null"
            ],
            
            "environment-variables" : [
                "PATH=/home/joe/programs/ampl.linux-intel64",
                "LD_LIBRARY_PATH=/home/joe/programs/ampl.linux-intel64/amplapi/lib",
                
                "PATH=/home/joe/programs/gams40.1_linux_x64_64_sfx",
                "PYTHONPATH=/home/joe/programs/gams40.1_linux_x64_64_sfx/apifiles/Python/api_38",
                "PYTHONPATH=/home/joe/programs/gams40.1_linux_x64_64_sfx/apifiles/Python/gams",
                "LD_LIBRARY_PATH=/home/joe/programs/gams40.1_linux_x64_64_sfx",
                "LD_LIBRARY_PATH=/home/joe/programs/gams40.1_linux_x64_64_sfx/apifiles/C++/lib",
                
                "PATH=/usr/bin",
                "PYTHONPATH=/usr/lib/python3.9",
                "PYTHONPATH=/home/joe/.local/lib/python3.9/site-packages",
                "PYTHONPATH=/home/joe/.local/bin"
            ]
        }
    ],
    
    "commands" : [
        {
            "name" : "ampl",
            "executable" : "/home/joe/programs/ampl.linux-intel64/ampl",
            "sandbox" : "general-purpose"
        },
        {
            "name" : "gams",
            "executable" : "/home/joe/programs/gams40.1_linux_x64_64_sfx/gams",
            "sandbox" : "general-purpose"
        },
        {
            "name" : "python3",
            "executable" : "/usr/bin/python3.9",
            "file-extensions" : [ "py" ],
            "sandbox" : "general-purpose"
        },
        {
            "name" : "pip3",
            "executable" : "/usr/bin/python3.9",
            "prepend-options" : "-m pip",
            "auto-mount" : "disabled",
            "enable-network" : true,
            "inherit-user-id" : true,
            "admin-only" : true,
            "mount-readable" : [
                "/lib",
                "/lib64",
                "/etc/resolv.conf",
                "/usr/bin/python3.9",
                "/usr/lib/python3.9"
            ],
            "mount-writable" : [
                "/home/joe/.cache/pip",
                "/home/joe/.local/bin",
                "/home/joe/.local/share",
                "/home/joe/.local/lib/python3.9/site-packages",
                "/dev/null"
            ],
            "environment-variables" : [
                "PYTHONPATH=/usr/lib/python3.9",
                "PYTHONPATH=/home/joe/.local/lib/python3.9/site-packages",
                "PYTHONPATH=/home/joe/.local/bin",
                "PATH=/home/joe/.local/bin"
            ]
        }
    ]
}
Interrupt and restart the Opt-server after changing its config.json

5. Test your Opt-server in PIFOP

  1. Open a PIFOP project and select the Opt-server. If you don't see the option to select your Opt-server, it is probably because you forgot to start it: 
    $ ./opt-server --start
  2. Enter the command help in the terminal. If you don't see one of the commands that you have configured — ampl, gams, python3 and pip3 — make sure that they are properly configured in the Opt-server's config.json (step 4).
  3. Try running each of the commands that you have configured. You can try to execute the AMPL/GAMS examples in this list. If you see error messages containing NSJAIL that means that the sandbox is not properly configured to run the program it is trying to run. Make sure that you have no missing or mispelled paths in the sandbox configuration (step 4).
  4. If everything is working, you are ready to upgrade your account to the Enterprise plan and allow students and faculty to use the Opt-server — see the next step.

6. Allow students and faculty to use the Opt-server

  1. Go to the subscription keys management page and hand-out the keys that you've got to your students and faculty using their PIFOP user names or emails.
  2. Make sure the Opt-server is running. 
    $ ./opt-server --start
  3. Go to the Opt-server management page and select the Opt-server.
  4. Create a New User Group. Name it Enterprise and fill-in the other fields of the form.
  5. Once the Enterprise user group is created, every member of your Enterprise will be able to select the Opt-server and use it from PIFOP. Every user that you give a subscription key to is considered to be a member of your Enterprise.

7. Recommended: Keep the Opt-server running 24/7

Use screen to keep the Opt-server running even after you logout from the machine

You can accomplish the same with tmux.
  1. Install screen if needed: 
    $ sudo apt-get install screen
  2. Create a new screen session named "opt-server": 
    $ screen -S opt-server
  3. Navigate to the Opt-server directory and run it using the the run-forever script: 
    $ cd /path/to/opt-server
$ ./run-forever ./opt-server --start
    The run-forever script is a small utility script that is shipped in the Opt-server bundle. All that it does is to run whatever command line that you pass to it in a loop so that, in the event a program failure, the command is executed again after 2 seconds. This is its core logic: 
    while true
do
    $@
    sleep 2
done
    To interrupt the above command, press Ctrl+c twice, the first to interrupt the Opt-server, and the second to interrupt the script.
  4. To Dettach from the screen session, press Ctrl+a, release, and then press d. To Reattach to the screen session, enter the command 
    $ screen -r opt-server
  5. If you want to learn more screen commands, checkout this screen cheatcheat by Carlos Tosta.

Add a Cron job to start the Opt-server on boot

Cron is a linux program that schedules commands to be executed on specific dates or when special events happen — such as a reboot. See below how to automate the screen session creation shown in the previous section upon reboot.

  1. Start editing your cron file. 
    crontab -e
  2. Append the following line to the file and save it — modify paths if needed: 
    @reboot cd ~/opt-server ; screen -S opt-server -d -m bash -c './run-forever ./opt-server --start ; bash'

Done!

I hope that this guide can help you hosting an Opt-server in your own premises. If you have any questions, don't hesitate to contact us!

Until next time!

— Davi

© PIFOP Solutions
Terms and Privacy Help Pricing About Blog Contact us Server Status Twitter LinkedIn