Design
Prerequisite
- Linux Kernel Version >= 3.10
- Cgroup file system mounted at /sys/fs/cgroup. Usually done by systemd
Architecture
Execution Workflow
Return Status
- Accepted: Program exited with status code 0 within time & memory limits
- Memory Limit Exceeded: Program uses more memory than memory limits
- Time Limit Exceeded:
- Program uses more CPU time than cpuLimit
- Or, program uses more clock time than clockLimit
- Output Limit Exceeded:
- Program output more than pipeCollector limits
- Or, program output more than output-limit
- File Error:
- CopyIn file is not existed
- Or, CopyIn file too large for container file system
- Or, CopyOut file is not existed after program exited
- Non Zero Exit Status: Program exited with non 0 status code within time & memory limits
- Signalled: Program exited with signal (e.g. SIGSEGV)
- Dangerous Syscall: Program killed by seccomp filter
- Internal Error:
- Program is not exist
- Or, container create not successful (e.g. not privileged docker)
- Or, other errors
Packages
- envexec: run single / group of programs in parallel within restricted environment and resource constraints
- env: reference implementation environments to inject into envexec
Windows Support
- Build
go-judgeby:go build ./cmd/go-judge/ - Build
go_judge.dll: (need to installgccas well)go build -buildmode=c-shared -o go_judge.so ./cmd/go-judge-ffi/ - Run:
./go-judge
Windows Security
- Resources are limited by JobObject
- Privilege are limited by Restricted Low Mandatory Level Token
- Low Mandatory Level directory is created for read / write
MacOS Support
- Build
go-judgeby:go build ./cmd/go-judge/ - Build
go_judge.dylib: (need to installXCode)go build -buildmode=c-shared -o go_judge.dylib ./cmd/go-judge-ffi/ - Run:
./go-judge
MacOS Security
sandbox-initprofile deny network access and file read / write and read / write to/Usersdirectory
Notice
WARNING
Window and macOS support are experimental and should not be used in production environments
cgroup usage
For cgroup v1, the go-judge need root privilege to create cgroup. Either creates sub-directory /sys/fs/cgroup/cpuacct/gojudge, /sys/fs/cgroup/memory/gojudge, /sys/fs/cgroup/pids/gojudge and make execution user readable or use sudo to run it.
For cgroup v2, systemd dbus will be used to create a transient scope for cgroup integration.
If no permission to create cgroup, the cgroup related limit will not be effective.
cgroup v2 support
The cgroup v2 is supported by go-judge now when running as root since more Linux distribution are enabling cgroup v2 by default (e.g. Ubuntu 21.10+, Fedora 31+). However, for kernel < 5.19, due to missing memory.max_usage_in_bytes in memory controller, the memory usage is now accounted by maxrss returned by wait4 syscall. Thus, the memory usage appears higher than those who uses cgroup v1. For kernel >= 5.19, memory.peak is being used.
When running in containers, the go-judge will migrate all processed into /api hierarchy to enable nesting support.
When running in Linux distributions powered by systemd, the go-judge will contact systemd via dbus to create a transient scope as cgroup root.
When running with kernel >= 5.7, the go-judge will try faster clone3(CLONE_INTO_CGROUP) method.
Memory Usage
The controller will consume 20M memory and each container will consume 20M + size of tmpfs 2 * 128M. For each request, it consumes as much as user program limit + extra limit (16k) + total copy out max. Notice that the cached file stores in the shared memory (/dev/shm) of the host, so please ensure enough size allocated.
For example, when concurrency = 4, the container itself can consume as much as 60 + (20+32) * 4M = 268M + 4 * total copy out + total max memory of requests.
Due to limitation of GO runtime, the memory will not return to OS automatically, which could lead to OOM killer. The background worker was introduced to checks heap usage and invokes GC when necessary.
-force-gc-targetdefault20m, the minimal size to trigger GC-force-gc-intervaldefault5s, the interval to check memory usage