Preamble

I just realized: the title can be shortened to CCCC lol. Well anyway, the whole world is going apeshit about ARM, but who takes care of poor MIPS? Maybe I was looking in the wrong places, but I couldn’t really find a comprehensive guide about this topic, only sporadic Stack Overflow questions, and generic, or even vague answer. I wanted MIPS cross-compilation because I have a Ubiquiti ER-X, and that happens to be running on MIPS, and I happen to be a friggin’ nerd who wants to run custom stuff on it.

To clarify, this guide is about little-endian MIPS32, also known as mipsel. I’ll use these terms interchangeably from now.

Let’s cut to the chase!

Docker

Install Docker, because this whole thing’s gonna get messy, so you wanna contain that trash in your containers. (You get it? They’re called containers for a reason!) Also, if you screw something up, it’s so much easier to start over. Once done with the Docker setup, create a mips directory, then start your container:

docker run -it --rm --mount type=bind,source="$(pwd)"/mips,target=/mips ubuntu:18.04

Why Ubuntu 18.04? Because it’s kind of middle of the road, usually not too old and not too new. Of course, there are exceptions to that, then you can just switch to 16.04 or 20.04, that’s the whole beauty of Docker.

As a first step, you want to obtain an initial cache of repo data, and update any outdated packages:

apt -y update && apt -y dist-upgrade

Toolchain

First some crucial tools:

apt -y install wget git sudo unzip mc file

Then the C and C++ compilers for MIPS, and QEMU to make it possible to run MIPS binaries on your x86_64 PC:

apt -y install gcc-mipsel-linux-gnu g++-mipsel-linux-gnu qemu qemu-user

Hello World

Time to compile your first MIPS binary. Create hello.c:

#include <stdio.h>

int main()
{
    printf("Yo dawg!\n");
    return 0;
}

Now turn this source code into an executable, or binary with the compiler:

mipsel-linux-gnu-gcc hello.c -o hello

Now try executing it:

$ ./hello 
bash: ./hello: No such file or directory

But the file is there, why does Bash say it doesn’t exist? Because Bash is stupid af. The real reason is that because there’s an architecture mismatch. Your machine is (most likely) an x86_64 one:

$ uname -p 
x86_64

How about that hello program though?

$ file hello
hello: ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1 (SYSV), dynamically linked, interpreter /lib/ld.so.1, for GNU/Linux 3.2.0, BuildID[sha1]=bf7879e1e3c4071b9f350f562a8757a49d3f738f, not stripped

Oops, that’s slightly different, isn’t it? So what can you do? Use QEMU:

$ qemu-mipsel -L /usr/mipsel-linux-gnu hello  
Yo dawg!

Awesome. Now you can compile and run MIPS executables. You can try it out, copy this hello file to your EdgeRouter and see if it runs. It should.

GNU Make

Most established open source software uses GNU Make for compilation. Install it first:

apt -y install make binutils

C

Let’s try compiling zlib, since it has virtually no dependencies.

# start in the home dir
cd
# download zlib
wget https://zlib.net/zlib-1.2.11.tar.gz
# extract it
tar xf zlib-1.2.11.tar.gz
# enter the source tree
cd zlib-1.2.11
# prepare the software for compilation, and point it to your MIPS compiler and linker, use the /usr/local prefix so that it won't mix with the system files
CC=mipsel-linux-gnu-gcc CC_LD=mipsel-linux-gnu-ld ./configure --prefix=/usr/local
# perform the actual compilation, run as many threads as your CPU has
make -j $(nproc)
# install the compiled binaries
make -j $(nproc) install

C++

For this example, we use Jansson, which is another very portable and simple to build library. Notice the new CXX variable being used, which is used to tell what C++ compiler to use.

$ cd
$ wget https://digip.org/jansson/releases/jansson-2.13.1.tar.gz
$ tar xf jansson-2.13.1.tar.gz
$ cd jansson-2.13.1
$ CC=mipsel-linux-gnu-gcc CXX=mipsel-linux-gnu-g++ CC_LD=mipsel-linux-gnu-ld ./configure --prefix=/usr/local
checking whether we are cross compiling... configure: error: in `/root/jansson-2.13.1':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details

Well then. As you can see, every software is a little different. So let’s do as it says:

CC=mipsel-linux-gnu-gcc CXX=mipsel-linux-gnu-g++ CC_LD=mipsel-linux-gnu-ld ./configure --prefix=/usr/local --host=mipsel-linux-gnu
make -j $(nproc)
make -j $(nproc) install

Easy as pie.

Dependencies

So what if you want to build software that depends on other software? Let’s try out OpenSSL, it’s such an ubiquitous library, and it also happens to optionally depend on zlib, so it’s a match made in heaven.

cd
wget https://www.openssl.org/source/openssl-1.1.1i.tar.gz
tar xf openssl-1.1.1i.tar.gz
cd openssl-1.1.1i

Now, OpenSSL has its own custom Perl Configure script (notice the different capitalization). There’s a short help printed via ./Configure --help, but it’s not too talkative, so if you want to dig deeper, read the INSTALL file. TLDR you need the linux-mips32 flag for mipsel compilation, and the zlib flag to tell OpenSSL to build with zlib support:

$ ./Configure --prefix=/usr/local linux-mips32 zlib
$ make -j$(nproc)
/bin/sh: 1: gcc: not found
Makefile:700: recipe for target 'apps/app_rand.o' failed
make[1]: *** [apps/app_rand.o] Error 127
make[1]: Leaving directory '/root/openssl-1.1.1i'
Makefile:172: recipe for target 'all' failed
make: *** [all] Error 2

Oops. We don’t want to use gcc, that’s the native compiler, we want mipsel-linux-gnu-gcc. For zlib, we used the CC and CC_LD variables to achieve that, but OpenSSL’s Configure has a flag for that, --cross-compile-prefix:

$ ./Configure linux-mips32 --prefix=/usr/local zlib --cross-compile-prefix=mipsel-linux-gnu-
$ make -j$(nproc)
crypto/comp/c_zlib.c:35:11: fatal error: zlib.h: No such file or directory
 # include <zlib.h>
           ^~~~~~~~
compilation terminated.

Still not quite there. Now it uses the proper compiler, but can’t find the zlib parts we just installed. So what can you do? After consulting the INSTALL file, here’s the proper Configure line:

$ ./Configure linux-mips32 --prefix=/usr/local zlib --cross-compile-prefix=mipsel-linux-gnu- --with-zlib-include=/usr/local/include --with-zlib-lib=/usr/local/lib
$ make -j$(nproc)
$ make -j$(nproc) install

OpenSSL is cool, because unlike zlib, it has actual executables that you can run.

$ qemu-mipsel -L /usr/mipsel-linux-gnu /usr/local/bin/openssl version
/usr/local/bin/openssl: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory

Except when you can’t. Well yeah, that library is under /usr/local/lib:

$ qemu-mipsel -L /usr/local /usr/local/bin/openssl version
/lib/ld.so.1: No such file or directory

Ah yeah, the MIPS ld.so is still under /usr/mipsel-linux-gnu. So how do we resolve this?

$ LD_LIBRARY_PATH=/usr/local/lib qemu-mipsel -L /usr/mipsel-linux-gnu /usr/local/bin/openssl version  
OpenSSL 1.1.1i  8 Dec 2020

Now we’re talking!

Meson & Ninja

Think of Meson as a different configure script and of Ninja as a different make. Install them:

apt -y install ninja-build meson

Now grab Jose, which is a utility that luckily happens to depend on the libs we just built, Obtain it:

cd
git clone https://github.com/latchset/jose.git
mkdir jose/build
cd jose/build

Now try to do something:

$ meson .. --prefix=/usr/local
The Meson build system
Version: 0.45.1
Source dir: /root/jose
Build dir: /root/jose/build
Build type: native build
Project name: jose

meson.build:1:0: ERROR: Unknown compiler(s): ['cc', 'gcc', 'clang']
The follow exceptions were encountered:
Running "cc --version" gave "[Errno 2] No such file or directory: 'cc': 'cc'"
Running "gcc --version" gave "[Errno 2] No such file or directory: 'gcc': 'gcc'"
Running "clang --version" gave "[Errno 2] No such file or directory: 'clang': 'clang'"

Well then. It says native build, that can’t be good. Apparently you need a so-called “cross file” for a Meson cross-compile. Something like this should do, save it as /usr/local/etc/mipsel.cross:

[binaries]
c = '/usr/bin/mipsel-linux-gnu-gcc'
cpp = '/usr/bin/mipsel-linux-gnu-g++'
ar = '/usr/bin/mipsel-linux-gnu-ar'
strip = '/usr/bin/mipsel-linux-gnu-strip'
pkgconfig = 'pkg-config'
exe_wrapper = '/usr/local/bin/qemu-mipsel-wrapper'

[properties]
has_function_printf = true
needs_exe_wrapper = true
c_arch = 'mipsel'
cc_arch = 'mipsel'
bits = 32

[build_machine]
system = 'linux'
cpu_family = 'x86_64'
endian = 'little'
cpu = 'x86_64'

[host_machine]
system = 'linux'
cpu_family = 'x86_64'
endian = 'little'
cpu = 'x86_64'

[target_machine]
system = 'linux'
cpu_family = 'mips'
endian = 'little'
cpu = 'mips'

What’s that /usr/local/bin/qemu-mipsel-wrapper though? That’s our old friend, qemu-mipsel -L /usr/mipsel-linux-gnu, but it seems Meson is so bleeding-edge that it only accepts executable paths, but not arguments to them. So create this /usr/local/bin/qemu-mipsel-wrapper file like so:

#!/bin/bash
qemu-mipsel -L /usr/mipsel-linux-gnu "$@"

Also make sure it’s executable:

chmod +x /usr/local/bin/qemu-mipsel-wrapper

After all that trouble, we’re finally back to our Meson command:

$ meson .. --prefix=/usr/local --cross-file /usr/local/etc/mipsel.cross
The Meson build system
Version: 0.45.1
Source dir: /root/jose
Build dir: /root/jose/build
Build type: cross build
Project name: jose

meson.build:1:0: ERROR: Unknown compiler(s): ['cc', 'gcc', 'clang']
The follow exceptions were encountered:
Running "cc --version" gave "[Errno 2] No such file or directory: 'cc': 'cc'"
Running "gcc --version" gave "[Errno 2] No such file or directory: 'gcc': 'gcc'"
Running "clang --version" gave "[Errno 2] No such file or directory: 'clang': 'clang'"

Why the hell is that? We told Meson to use the mipsel tools, why is it looking for native GCC? I tell you why: because it’s dumb af. So lacking better options, install native GCC to calm Meson down:

apt -y install gcc

Try again:

$ meson .. --cross-file /usr/local/etc/mipsel.cross --prefix=/usr/local
The Meson build system
Version: 0.45.1
Source dir: /root/jose
Build dir: /root/jose/build
Build type: cross build
Project name: jose
Native C compiler: cc (gcc 7.5.0 "cc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0")
Cross C compiler: /usr/bin/mipsel-linux-gnu-gcc (gcc 7.5.0)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Target machine cpu family: mips
Target machine cpu: mips
Build machine cpu family: x86_64
Build machine cpu: x86_64

meson.build:29:0: ERROR: Pkg-config not found.

Gee, one step at a time I guess. Install pkg-config:

apt -y install pkg-config

So it’ll work now, right? Right?

$ meson .. --prefix=/usr/local --cross-file /usr/local/etc/mipsel.cross
The Meson build system
Version: 0.45.1
Source dir: /root/jose
Build dir: /root/jose/build
Build type: cross build
Project name: jose
Native C compiler: cc (gcc 7.5.0 "cc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0")
Cross C compiler: /usr/bin/mipsel-linux-gnu-gcc (gcc 7.5.0)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Target machine cpu family: mips
Target machine cpu: mips
Build machine cpu family: x86_64
Build machine cpu: x86_64
Cross dependency zlib found: YES 1.2.11
Dependency threads found: YES
Cross dependency jansson found: YES 2.13.1
Cross dependency libcrypto found: YES 1.1.1i
Configuring jose.h using configuration
Checking if "-Wl,--version-script=..." links: YES
Program ./jose-alg found: YES (/root/jose/tests/./jose-alg)
Program ./jose-fmt found: YES (/root/jose/tests/./jose-fmt)
Program ./jose-b64-enc found: YES (/root/jose/tests/./jose-b64-enc)
Program ./jose-b64-dec found: YES (/root/jose/tests/./jose-b64-dec)
Program ./jose-jwk-eql found: YES (/root/jose/tests/./jose-jwk-eql)
Program ./jose-jwk-exc found: YES (/root/jose/tests/./jose-jwk-exc)
Program ./jose-jwk-gen found: YES (/root/jose/tests/./jose-jwk-gen)
Program ./jose-jwk-pub found: YES (/root/jose/tests/./jose-jwk-pub)
Program ./jose-jwk-use found: YES (/root/jose/tests/./jose-jwk-use)
Program ./jose-jwk-thp found: YES (/root/jose/tests/./jose-jwk-thp)
Program ./jose-jws-fmt found: YES (/root/jose/tests/./jose-jws-fmt)
Program ./jose-jws-ver found: YES (/root/jose/tests/./jose-jws-ver)
Program ./jose-jws-sig found: YES (/root/jose/tests/./jose-jws-sig)
Program ./jose-jwe-fmt found: YES (/root/jose/tests/./jose-jwe-fmt)
Program ./jose-jwe-dec found: YES (/root/jose/tests/./jose-jwe-dec)
Program ./jose-jwe-enc found: YES (/root/jose/tests/./jose-jwe-enc)
Traceback (most recent call last):
  File "/usr/share/meson/mesonbuild/mesonmain.py", line 361, in run
    app.generate()
  File "/usr/share/meson/mesonbuild/mesonmain.py", line 150, in generate
    self._generate(env)
  File "/usr/share/meson/mesonbuild/mesonmain.py", line 197, in _generate
    intr.run()
  File "/usr/share/meson/mesonbuild/interpreter.py", line 2991, in run
    super().run()
  File "/usr/share/meson/mesonbuild/interpreterbase.py", line 173, in run
    self.evaluate_codeblock(self.ast, start=1)
  File "/usr/share/meson/mesonbuild/interpreterbase.py", line 195, in evaluate_codeblock
    raise e
  File "/usr/share/meson/mesonbuild/interpreterbase.py", line 189, in evaluate_codeblock
    self.evaluate_statement(cur)
  File "/usr/share/meson/mesonbuild/interpreterbase.py", line 204, in evaluate_statement
    return self.method_call(cur)
  File "/usr/share/meson/mesonbuild/interpreterbase.py", line 490, in method_call
    return obj.method_call(method_name, self.flatten(args), kwargs)
  File "/usr/share/meson/mesonbuild/interpreter.py", line 1154, in method_call
    value = fn(state, args, kwargs)
  File "/usr/share/meson/mesonbuild/interpreterbase.py", line 79, in wrapped
    return f(s, node_or_state, args, kwargs)
  File "/usr/share/meson/mesonbuild/modules/pkgconfig.py", line 303, in generate
    version, pcfile, conflicts, variables)
  File "/usr/share/meson/mesonbuild/modules/pkgconfig.py", line 176, in generate_pkgconfig_file
    ofile.write('Name: %s\n' % name)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 9: ordinal not in range(128)

That’s just Python for ya. After hours of messing around, it turned out to be a locale problem. The default locale in Docker is POSIX, and Meson just loves to shit its pants on POSIX. So let’s do something about it:

apt -y install locales
locale-gen en_US.UTF-8
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

Now it better be working!

meson .. --prefix=/usr/local --cross-file /usr/local/etc/mipsel.cross
ninja -j$(nproc)
ninja -j$(nproc) install

Holy damn, it actually does. But why is libjose.so under the /usr/local/lib/x86_64-linux-gnu directory? It’s not an x86_64 lib. Fortunately you can work around that:

meson .. --prefix=/usr/local --cross-file /usr/local/etc/mipsel.cross --libdir=lib

Custom Directories

In these examples we used the /usr/local prefix, which is a standard Unix path for user-installed software. If you use something different, your build tools will probably not find them out of the box. There are several ways to fix this, one example was the Configure flags we used for OpenSSL to correctly locate zlib.

In the case of Meson, it usually uses pkg-config, which is essentially just a bunch of .pc files under various lib/pkgconfig folders. You can point Meson to such folders with the PKG_CONFIG_PATH variable, e.g.:

PKG_CONFIG_PATH=/opt/lib/pkgconfig meson .. --cross-file /usr/local/etc/mipsel.cross

Welp, that was a mouthful. Have fun with your MIPS builds!