-sh: uum: not found

While shaving today’s yak I stumbled across the strangest error message I’ve seen a while. It began innocently enough:

tc@CZC60880J6:~$ uum
-sh: uum: not found

But read on to follow my descent into confusion and ELF related rage.

uum is a binary that exits successfully. The implementation is extremely boring and not at all relevant to the story (honest). The only thing you need to know is I’m building it on Debian and running it on tinycorelinux.

So:

tc@CZC60880J6:~$ uum
-sh: uum: not found

Naturally I assume I have missed uum off PATH somehow:

tc@CZC60880J6:~$ which uum
/usr/local/bin/uum

Seems like it’s there but just to be safe lets remove all doubt:

tc@CZC60880J6:~$ /usr/local/bin/uum
-sh: /usr/local/bin/uum: not found

Now the whole point of today’s yak is to figure out building tinycorelinux extensions (“.tcz”). Quoting the book tcz extensions are “a loop-mounted squashfs 4.x archive, with specified parameters, usually symlinked into the main file system”. ‘Specified parameters’ is covering a lot of ground here. Amongst other things tinycorelinux is pretty finicky about the exact uid, guid, and permissions of the files in the extension.

So the next port of call is double checking symlinks and permissions:

tc@CZC60880J6:~$ ls -al /usr/local/bin/uum
lrwxrwxrwx    1 root     root            33 Apr 22 14:03 /usr/local/bin/uum -> /tmp/tcloop/uum/usr/local/bin/uum
tc@CZC60880J6:~$ ls -al /tmp/tcloop/uum/usr/local/bin/uum
-rwxr-xr-x    1 root     root       5380112 Apr 22 11:58 /tmp/tcloop/uum/usr/local/bin/uum

…which seems fine. Lets compare to dropbear which is definitely working:

tc@CZC60880J6:~$ ls -al /usr/local/bin/ssh
lrwxrwxrwx    1 root     root            38 Apr 22 14:03 /usr/local/bin/ssh -> /tmp/tcloop/dropbear/usr/local/bin/ssh
tc@CZC60880J6:~$ ls -al /tmp/tcloop/dropbear/usr/local/bin/ssh
lrwxrwxrwx    1 root     root            28 Nov 11  2020 /tmp/tcloop/dropbear/usr/local/bin/ssh -> /usr/local/bin/dropbearmulti
tc@CZC60880J6:~$ ls -al /usr/local/bin/dropbearmulti
lrwxrwxrwx    1 root     root            48 Apr 22 14:03 /usr/local/bin/dropbearmulti -> /tmp/tcloop/dropbear/usr/local/bin/dropbearmulti
tc@CZC60880J6:~$ ls -al /tmp/tcloop/dropbear/usr/local/bin/dropbearmulti
-rwxr-xr-x    1 root     root        237280 Nov 11  2020 /tmp/tcloop/dropbear/usr/local/bin/dropbearmulti

dropbear is doing a busybox style monobinary so it is a tangled hive of symlinks and villainy – but the villainy/permissions match those of uum.

Lets go right to the source:

tc@CZC60880J6:~$ /tmp/tcloop/uum/usr/local/bin/uum
-sh: /tmp/tcloop/uum/usr/local/bin/uum: not found

I’m starting to feel like I’m losing my grip on the situation. The file is definitely there:

tc@CZC60880J6:~$ cat /tmp/tcloop/uum/usr/local/bin/uum | wc
    30591    149115   5380112

It’s a normal ELF file:

tc@CZC60880J6:~$ cat /tmp/tcloop/uum/usr/local/bin/uum | xxd | head
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0300 3e00 0100 0000 00d8 0700 0000 0000  ..>.............
00000020: 4000 0000 0000 0000 100e 5200 0000 0000  @.........R.....
00000030: 0000 0000 4000 3800 0b00 4000 2800 2600  [email protected]...@.(.&.
00000040: 0600 0000 0400 0000 4000 0000 0000 0000  ........@.......
00000050: 4000 0000 0000 0000 4000 0000 0000 0000  @.......@.......
00000060: 6802 0000 0000 0000 6802 0000 0000 0000  h.......h.......
00000070: 0800 0000 0000 0000 0300 0000 0400 0000  ................
00000080: a802 0000 0000 0000 a802 0000 0000 0000  ................
00000090: a802 0000 0000 0000 1c00 0000 0000 0000  ................

But as soon as I try to run it turns invisible:

tc@CZC60880J6:~$ /tmp/tcloop/uum/usr/local/bin/uum
-sh: /tmp/tcloop/uum/usr/local/bin/uum: not found

In fact I get the same error message with a binary really doesn’t exist.

tc@CZC60880J6:~$  /tmp/tcloop/uum/usr/local/bin/foobarbaz
-sh: /tmp/tcloop/uum/usr/local/bin/foobarbaz: not found

Okay well maybe there is something about squashfs I don’t understand.

tc@CZC60880J6:~$ cp /tmp/tcloop/uum/usr/local/bin/uum uum
tc@CZC60880J6:~$ ./uum
-sh: ./uum: not found

If it’s not squashfs and it’s not permissions and it’s not PATH and it’s not a broken symlink then it has to be the shell somehow? Clearly busyboxs ash implementation is doing something very odd.

tc@CZC60880J6:~$ python3
Python 3.9.18 (main, Jan 27 2024, 10:38:50)
[GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> with open("uum", "rb") as f:
...     print(len(f.read()))
...
5380112
>>> import subprocess
>>> subprocess.run("uum")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.9/subprocess.py", line 1837, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'uum'
>>>

What is happening. The straws are officially clutched. At this point I’m entertaining ever more absurd conspiracy theories about what is happening. Maybe uum is invoking a shell somehow and that shell is failing? I could set a breakpoint in _start!

tc@CZC60880J6:~$ gdb uum
GNU gdb (GDB) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from uum...
(gdb) set startup-with-shell off
(gdb) start
warning: could not convert 'main' from the host encoding (ANSI_X3.4-1968) to UTF-32.
This normally should not happen, please file a bug report.
Temporary breakpoint 1 at 0x7d916: file gen/src/examples/uum_main.cc, line 5.
Starting program: /home/tc/uum
warning: Cannot exec /home/tc/uum
warning: Error: No such file or directory
During startup program exited with code 127.
(gdb)

I did not even get to setting a breakpoint.

Okay – time to stop mucking around and bring out the big guns and see what is happening:

tc@CZC60880J6:~$ strace uum
execve("/usr/local/bin/uum", ["uum"], 0x7ffc9f1886c0 /* 21 vars */) = -1 ENOENT (No such file or directory)
strace: exec: No such file or directory
+++ exited with 1 +++

I could cry.

I will spare you several hours of searching, examining the sourcecode of busybox, and a not insignificant period where I realised I had started to type umm instead of uum and not noticed because – again – the error message is the exactly the same. However know that eventually I was reduced to running readelf and diffing uum vs. dropbearmulti.

Amongst the output I spot a funny looking line:

-      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
+      [Requesting program interpreter: /lib/ld-linux-x86-64.so.2]

Wait a second.

tc@CZC60880J6:~$ ls  /
bin/     home/    linuxrc  proc/    sbin/    usr/
dev/     init     mnt/     root/    sys/     var/
etc/     lib/     opt/     run/     tmp/

There is no /lib64! With shaking hands:

tc@CZC60880J6:~$ sudo ln -s lib /lib64
tc@CZC60880J6:~$ ./uum
tc@CZC60880J6:~$ echo $?
0

The absence of output has never looked so sweet.

Addendum

tc@CZC60880J6:~$ ./uum
bash: ./uum: cannot execute: required file not found
tc@CZC60880J6:~$ ./foobarbaz
bash: ./foobarbaz: No such file or directory

So trying a different shell would have helped.

Home