Add code interpreter into your LLM apps with llm-sandbox

LLM Sandbox is a lightweight, portable environment built to run code generated by Large Language Models (LLMs) in a secure, isolated setting using Docker containers. With its intuitive interface, you can easily set up, manage, and execute code within a controlled environment, simplifying the process of testing and running LLM-generated scripts.

Key Features

  • Effortless Setup: Create sandbox environments quickly with minimal configuration.
  • Complete Isolation: Run code in isolated Docker containers to safeguard your host system.
  • Versatile: Supports multiple programming languages, including Python, Java, JavaScript, C++, Go, and Ruby.
  • Highly Portable: Use predefined Docker images or custom Dockerfiles for ultimate flexibility.
  • Scalable: Seamlessly integrate with Kubernetes and remote Docker hosts for larger deployments.

Using llm-sandbox with Cloudfleet

Set up CLoudfleet CLI

Use the Cloudfleet console to get instructions for the CLI installation and setup (Clusters -> Connect). Your commands will look similar to the following:

cloudfleet auth add-profile user default 12312312-abcd-1234-5678-90abcdef1234

cloudfleet clusters kubeconfig abcdefgh-1234-5678-90ab-cdef01234567

kubectl config set-context abcdefgh-1234-5678-90ab-cdef01234567/default

Install llm-sandbox 

pip install llm-sandbox

Create script

Create a new file named sandbox_demo.py and add the following code:

from llm_sandbox import SandboxSession

with SandboxSession(
    lang="python", keep_template=True, verbose=True, use_kubernetes=True
) as session:
    output = session.run("print('Hello, World!')")
    print(output.text)

    output = session.run(
        "import numpy as np\nprint(np.random.rand())", libraries=["numpy"]
    )
    print(output.text)

    session.execute_command("pip install pandas")
    output = session.run("import pandas as pd\nprint(pd.__version__)")
    print(output.text)

Execute the script

Run the script using the following command: python3 sandbox_demo.py. The output will be similar to the following:

Using local Kubernetes context since client is not provided..
Copying /tmp/code.py to sandbox-python-2afe03b697ed4432af7209edeea84b87:/tmp/code.py..
Executing command: mkdir -p /tmp
Output: code.py

Copied /tmp/code.py to sandbox-python-2afe03b697ed4432af7209edeea84b87:/tmp/code.py in 3.52 seconds
Executing command: python /tmp/code.py
Output: Hello, World!
Hello, World!

Executing command: pip install numpy
Output: Collecting numpy
  Downloading numpy-2.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.5/19.5 MB 39.4 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-2.0.2

[notice] A new release of pip is available: 23.0.1 -> 24.2
[notice] To update, run: pip install --upgrade pip
Copying /tmp/code.py to sandbox-python-2afe03b697ed4432af7209edeea84b87:/tmp/code.py..
Executing command: mkdir -p /tmp
Output: code.py

Copied /tmp/code.py to sandbox-python-2afe03b697ed4432af7209edeea84b87:/tmp/code.py in 3.36 seconds
Executing command: python /tmp/code.py
Output: 0.24481829174507796
0.24481829174507796

Executing command: pip install pandas
Output: Collecting pandas
  Downloading pandas-2.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.1/13.1 MB 65.0 MB/s eta 0:00:00
Collecting tzdata>=2022.7
  Downloading tzdata-2024.1-py2.py3-none-any.whl (345 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 345.4/345.4 kB 40.1 MB/s eta 0:00:00
Collecting python-dateutil>=2.8.2
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.9/229.9 kB 36.1 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.22.4 in /usr/local/lib/python3.9/site-packages (from pandas) (2.0.2)
Collecting pytz>=2020.1
  Downloading pytz-2024.2-py2.py3-none-any.whl (508 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 508.0/508.0 kB 47.7 MB/s eta 0:00:00
Collecting six>=1.5
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: pytz, tzdata, six, python-dateutil, pandas
Successfully installed pandas-2.2.3 python-dateutil-2.9.0.post0 pytz-2024.2 six-1.16.0 tzdata-2024.1

[notice] A new release of pip is available: 23.0.1 -> 24.2
[notice] To update, run: pip install --upgrade pip
Copying /tmp/code.py to sandbox-python-2afe03b697ed4432af7209edeea84b87:/tmp/code.py..
Executing command: mkdir -p /tmp
Output: code.py

Copied /tmp/code.py to sandbox-python-2afe03b697ed4432af7209edeea84b87:/tmp/code.py in 3.20 seconds
Executing command: python /tmp/code.py
Output: 2.2.3
2.2.3

Monitor the execution

Cloudfleet will automatically create a new Kubernetes pod for each script execution. You can monitor the pods, nodes, and node claims using kubectl. The output will be similar to the following:

$  kubectl get nodes -o wide
NAME                  STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
apt-bull-3752625927   Ready    <none>   1m    v1.29.3   100.71.182.46   52.24.15.49   Ubuntu 22.04.4 LTS   6.5.0-1024-aws   containerd://1.7.19

$  kubectl get nodeclaims -o wide
NAME            TYPE                         CAPACITY   ZONE   NODE                  READY   AGE   ID                                           NODEPOOL   NODECLASS
managed-x79s7   AWS-USW2-AZ2-T3-MICRO-SPOT   spot       west   apt-bull-3752625927   True    1m    AWS://managed/usw2-az2/apt-bull-3752625927   managed    managed

$  kubectl get pods -o wide
NAME                                              READY   STATUS        RESTARTS   AGE   IP             NODE                  NOMINATED NODE   READINESS GATES
sandbox-python-2afe03b697ed4432af7209edeea84b87   1/1     Running       0          59s   10.244.1.56    apt-bull-3752625927   <none>           <none>
On this page