← Linux Sans Flashy Stuff To Mitigate Distraction

Linux FFMPEG Interactive Cut Section Of Audio Video File →

Linux Consolidated Advance Materials

March 20, 2024 Leave a comment

TL;DR It is a longish post, if you run out of patience, that is certainly not my fault. 🙂

Well, all of these, what I am going to dish out in this post is coming out of my long accumulated note files. I thought these might be useful to some people, who are looking for materials to study and apply. This is in no particular order, I just scanned through and found a few I can share at this moment. So, pay attention to your benefit. 🙂

Make file variables shorthand

Why? Delving with Linux sooner or later pushes you into a situation, where you are supposed to write some make file for your good. And here are some file-specific variable shorthand that might come in handy.

$@: the target filename.
$*: the target filename without the file extension.
$<: the first prerequisite filename.
$^: the filenames of all the prerequisites, separated by spaces, discard duplicates. 
$+: similar to $^, but includes duplicates. 
$?: the names of all prerequisites that are newer than the target, separated by spaces. 
$@: mute the command itself
$(info the message need to be printed)

Library linking breakage and fixing

This is a very common thing if you haven’t lived long with Linux. Why? It entails lots of reasons. So, knowing the reason and importantly fixing it might help in a good way.

You see the points extracted out of this: https://rosshemsley.co.uk/posts/linking

Points:

Dynamic or shared libraries are loaded up by your program at runtime. They contain lookup tables that map symbols to shared executable code. If you give someone a binary that links a dynamic library that they don’t already have, the OS will complain about missing libraries when they try to run it.

Dynamic or “shared” libraries have names that start with lib and finish with .so. Unless you’re on a Mac, where they end with .dylib.1

Dynamic libraries themselves can link to other dynamic libraries. These are known as transitive dependencies. All dependencies will need to be found to successfully run your binary.

If you want to move a binary from one machine (where it was compiled) to another, you’ll almost certainly find that at least some of the shared libraries needed by your binary are no longer found. This is usually the first sign of trouble…

Linux knows how to find libraries because it has a list of known locations for shared libraries in /etc/ld.so.conf. Each time you run ldconfig, the OS updates its cache of known libraries by going through directories in this file and reading the libraries it finds. OS X works differently… see ld and friends.

Use ldd (linux) or otool -L (OS X) to query your binary for the missing libraries. Beware that it is not safe to do this on a binary you suspect may be malicious 😞.

You can safely copy dynamic libraries from one machine to another. As long as the environments are similar enough…2 . In a perfect world (on linux), you could just copy the library you want to use into /usr/local/lib (the recommended place for unstable libraries) and then run ldconfig to make your OS reload its library cache.

Of course, on OS X things work totally differently. Dynamic libraries have an install name that contains the absolute path. This path is baked into your binary at compile time. You can use install_name_tool to change it. Good luck!

On linux, Adding libraries to /usr/local/lib makes them visible to everything, so you may want to copy your library somewhere else so that only your binary knows how to find it. One way to do this is using rpath…

You can set the rpath attribute of your binary to contain a directory hint for your OS to look in for libraries. This hint can be relative to your binary. This is especially useful if you always ship libraries in a relative directory to your binary. You can use @origin as a placeholder for the path of the binary itself, so a rpath of @origin/lib causes the OS to always look in <path to your binary>/lib for shared libraries at runtime. This can be used on both OS X and linux, and is one of the most useful tools for actually getting things working in practice.

If your OS isn’t finding a dynamic library that you know exists, you can try helping your OS by setting the environment variable LD_LIBRARY_PATH to the directory containing it – your OS will look there first before default system paths. Beware, this is considered bad practice, but it might unblock you in a pinch. OS X has DYLD_LIBRARY_PATH, which is similar, and also DYLD_FALLBACK_LIBRARY_PATH, which is similar, but different (sorry).

Dynamic libraries also have a thing called a soname, which is the name of the library, plus version information. You have seen this if you’ve seen libfoo.so.3.1 or similar. This allows us to use different versions of the same library on the same OS, and to make non backwards-compatible changes to libraries. The soname is also baked into the library itself.

Often, your OS will have multiple symlinks to a single library in the same directory, just with different paths containing version information, e.g. libfoo.so.3, libfoo.so.3.1. This is to allow programs to find compatible libraries with slightly different versions. Everything starts to get rather messy here… if you really need to get into the weeds, this article will help. You probably only need to understand this if you are distributing libraries to users and need to support compatibility across versions.

Of course, even if your binary only depends on a single symbol in a dynamic library, it must still link to that library. Now consider that the dependency itself may also link other unused transitive dependencies. Accidentally “catching a dependency” can cause your list of shared library dependencies to grow out of control, so that your simple hello world binary ends up depending on hundreds of megabytes of totally unused shared libraries 😞.

One solution to avoiding “dependency explosions” is to statically link symbols directly into your binary, so let’s start to look at static linking!

Static libraries (.a files) contain a symbol lookup table, similarly to dynamic libraries. However, they are much more dumb and also a total PITA to use correctly.

If you compile your binary and link in only static dependencies, you will end up with a static binary. This binary will not need to load any dependencies at runtime and is thus much easier to share with others!

People On The Internet will recommend that you do not not distribute static binaries, because it makes it hard to patch security flaws. With dynamic libraries, you just have to patch a single library e.g. libssl.so, instead of re-compiling everything on your machine that may have linked the broken library without your knowledge (i.e. everything).

People who build production systems at companies recommend static libraries because it’s way the hell easier to just deploy a single binary with zero dependencies that can basically run anywhere. No one cares about how big binaries are these days anyway.

Still, more people on the internet remind you that only one copy of a dynamic library is loaded into memory by the OS even when it is used by multiple processes, saving on memory pressure.

The static library people remind you that modern computers have plenty of memory and library size is hardly the thing killing us right now.

The OS X people point out that OS X strongly discourages the use of statically linked binaries.

Static libraries can’t declare any kind of library dependencies. This means it is your responsibility to ensure all symbols are baked correctly into your binary at link time – otherwise, your linker will fail. This can make linking static libraries painfully error-prone.

If you get symbol not found errors but literally swear that you linked every damn thing, you probably linked a static library and forgot a transitive dependency that is needed by it. This pretty much sucks as it’s basically impossible to figure out where that library comes from. Try having a guess by looking at the error messages. Or something?

Oh, and you must ensure that you link your static libraries in the correct order, otherwise you can still get symbol not found errors.

If you are starting to think it might be hard to keep track of static libraries, you are following along correctly. There are tools that can help you here, such as pkgconfig, CMake, autotools… or bazel. It’s quite easy to get going and achieve deterministic platform-independent static builds with no dynamic dependencies… Said no one ever 😓.

One classic way to screw up, is to compile a static library without using the -fPIC flag (for “position independent code”). If you do not do this, you will be able to use the static library in a binary, but you will not be able to link it into a dynamic library. This is especially frustrating if you were provided with a static library that was compiled without this flag and you can’t easily recompile it.

Beware that -fpic is not the same as -fPIC. Apparently, -fPIC always works but may result in a few nanoseconds of slowdown, or something. Probably you should use -fPIC and try not to think about it too much.

Your compiler toolchain (e.g. CMake) usually has a one-liner way to link a bunch of static libraries into a single dynamic library with no dependencies of its own. However, should you want to link a bunch of static libraries into another static library… well I’ve never successfully found a reliable way to do this 😞. Why do this you may ask? Mostly for cffi – when I want to build a single static library from C++ and then link it into e.g. a go binary.

Beware that your compiler/linker is not smart! Just because the header files declare a function and your linker manages to find symbols for it in your library, doesn’t mean that the function is remotely the same. You will discover this when you get undefined behavior at runtime.

Oh, and if the library you are linking was compiled with a #define switch set, but when you include the library’s headers, you do not set the define to the same value, welcome again to runtime undefined behavior land! This is the same problem as the one above, where the symbols end up being incompatible.

If you are trying to ship C++, another thing that can bite you is that the C++ standard library uses dynamic linking. This means that even the most basic hello world program cannot be distributed to others unless they have a compatible version of libstdc++. Very often you’ll end up compiling with a shiny new version of this library, only to find that your target is using an older, incompatible version.

One way to get around libstdc++ problems is to statically link it into your binary. However, if you create a static library that statically links libstdc++, and your library uses C++ types in its public interface… welcome again to undefined behavior land ☠️.

Another piece of classic advice is to statically link everything in your binary apart from core system libraries, such as glibc – which is basically a thin wrapper around syscalls. A practical goal I usually aim for is to statically link everything apart from libc and (preferably an older version of) libstdc++. This seems to be the safest approach.

Ultimately, my rule of thumb for building distributed systems is to statically link everything apart from libc and (an older version of) libstdc++. You can then put this library/binary into a Debian package, or an extremely lightweight Docker container that will run virtually anywhere. Setting up the static linking is a pain, but IMO worth the effort – the main benefits of dynamic libraries generally do not apply anymore when you are putting the binary in a container anyway.

Finally, for ultimate peace of mind, use a language that has a less insane build toolchain than C++. For example, Go builds everything statically by default and can link in both dynamic or static libraries if needed, using cargo. Rust also seems to work this way. Static binaries have started becoming fashionable

LD_LIBRARY_PATH Trouble and Solutions

It is very often caused by the misaligned or misplaced library in the system.

Pointer : https://www.hpc.dtu.dk/?page_id=1180

This little note is about one of the most “misused” environment variables on Unix systems: LD_LIBRARY_PATH . If used right, it can be very useful, but very often – not to say, most of the time – people apply it in the wrong way, and that is where they are calling for trouble. So, what does it do?

LD_LIBRARY_PATH tells the dynamic link loader (ld. so – this little program that starts all your applications) where to search for the dynamic shared libraries an application was linked against. Multiple directories can be listed, separated by a colon (:), and this list is then searched before the compiled-in search path(s), and the standard locations (typically /lib, /usr/lib, …).

This can be used for

testing new versions of a shared library against an already compiled application re-locating shared libraries, e.g. to preserve old versions creating a self-contained, relocatable(!) environment for larger applications, such that they do not depend on (changing) system libraries – many software vendors use that approach.

Sounds very useful, where is the problem?

Yes, it is useful – if you apply it in the way it was invented for, like the three cases above. However, very often it is used as a crutch to fix a problem that could have been avoided by other means (see below). It is even getting worse, if this crutch is applied globally into a user’s (or the system’s!) environment: applications compiled with those settings get dependent on this crutch – and if it is eventually taken away, they start to stumble (i.e. fail to run).

There are other implications as well:

Security: Remember that the directories specified in LD_LIBRARY_PATH get searched before(!) the standard locations? In that way, a nasty person could get your application to load a version of a shared library that contains malicious code! That’s one reason why setuid/setgid executables neglect that variable! Performance: The link loader has to search all the directories specified until it finds the directory where the shared library resides – for ALL shared libraries the application is linked against! This means a lot of system calls to open(), that will fail with “ENOENT (No such file or directory)”! If the path contains many directories, the number of failed calls will increase linearly, and you can tell that from the start-up time of the application. If some (or all) of the directories are in an NFS environment, the start-up time of your applications can really get long – and it can slow down the whole system! Inconsistency: This is the most common problem. LD_LIBRARY_PATH forces an application to load a shared library it wasn’t linked against, and that is quite likely not compatible with the original version. This can either be very obvious, i.e. the application crashes, or it can lead to wrong results if the picked-up library does not quite do what the original version would have done. Especially the latter is sometimes hard to debug.

How can I check which dynamic libraries are loaded?

There is the ldd command, that shows you which libraries are needed by a dynamically linked executable, e.g.

$ ldd /usr/bin/file
        linux-vdso.so.1 =>  (0x00007fff9646c000)
        libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00000030f9a00000)
        libz.so.1 => /lib64/libz.so.1 (0x00000030f8e00000)
        libc.so.6 => /lib64/libc.so.6 (0x00000030f8200000)
        /lib64/ld-linux-x86-64.so.2 (0x00000030f7a00000)

This is a ‘static’ view since ldd doesn’t resolve dependencies and libraries that will get loaded at runtime, e.g. by a library that depends on others. To get an overview of libraries loaded at runtime, you can use the pldd command:

$ ldd /bin/bash
        linux-vdso.so.1 =>  (0x00007ffff63ff000)
        libtinfo.so.5 => /lib64/libtinfo.so.5 (0x0000003108a00000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00000030f8600000)
        libc.so.6 => /lib64/libc.so.6 (0x00000030f8200000)
        /lib64/ld-linux-x86-64.so.2 (0x00000030f7a00000)

$ pldd 24362
24362:  -bash
/lib64/ld-2.12.so
/lib64/libc-2.12.so
/lib64/libdl-2.12.so
/lib64/libtinfo.so.5.7
/usr/lib64/gconv/ISO8859-1.so
/lib64/libnss_files-2.12.so

As you can see, there are two more .so-files loaded at runtime, that weren’t on the ‘static’ list.

Note: pldd is originally a Solaris command, that usually is not available on Linux. However, there is a Perl-script available (and installed on our machines) that extracts this information from the /proc/<PID>/maps file. How to avoid those problems with LD_LIBRARY_PATH?

A very simplistic answer would be: “just don’t use LD_LIBRARY_PATH!” The more realistic answer is, “the less you use it, the better off you will be”.

Below is a list of ways how to avoid LD_LIBRARY_PATH, inspired by reference [1] below. The best solution is on the top, going down to the last resort.

If you compile your application(s) yourself, you can solve the problem by specifying the correct location of the shared libraries and telling the linker to add those to the runpath of your executable, specifying the path in the ‘-rpath’ linker option:

cc -o myprog obj1.o ... objn.o -Wl,-rpath=/path/to/lib \
   -L/path/to/lib -lmylib

The linker also reads the LD_RUN_PATH environment variable, if set, and thus you can specify more than one path in an easy way, without having to use the above linker option:

export LD_RUN_PATH=/path/to/lib1:/path/to/lib2:/path/to/lib3
cc -o myprog obj1.o ... objn.o -L/path/to/lib1 -lmylib1 \
   -L/path/to/lib2 -lmylib2 ...

In both cases, you can check with ldd, that your executable will find the right libraries at start-up (see above). If there is a ‘not found’ message in the ldd output, you have done something wrong and should review your Makefile and/or your LD_RUN_PATH settings. There are tools around, to fix/change the runpath in a binary executable, e.g. chrpath under Linux. The problem with this method is, that the space in the executable that contains this information (i.e. the string defining the path) cannot be extended, i.e. you cannot add additional information – only overwrite an existing path. Furthermore, if no runpath exists in the executable, there is no way to change it. Read the man page for chrpath for more information. If you can’t fix the executable, create a wrapper script that calls the executable with the right LD_LIBRARY_PATH setting. In that way, the setting gets exposed to this application, only – and the applications that get started by that. The latter can lead to the inconsistency problem above, though.

#!/bin/sh

LD_LIBRARY_PATH=/path/to/lib1:/path/to/lib2:/path/to/lib3

export LD_LIBRARY_PATH

exec /path/to/bin/myprog $@

Testing a LD_LIBRARY_PATH from the command line:

$ env LD_LIBRARY_PATH=/path/to/lib1:/path/to/lib2:/path/to/lib3 ./myprog

This sets LD_LIBRARY_PATH for this command only. Do NOT do:

$ export LD_LIBRARY_PATH=/path/to/lib1:/path/to/lib2:/path/to/lib3

$ ./myprog

since this will pollute the shell environment for all consecutive commands! Never put LD_LIBRARY_PATH in your login profiles! In that way you will expose all the applications you start to this – probably problematic – path!

Unfortunately, some ISVs ship software, that puts global LD_LIBRARY_PATH settings into the system profiles during the installation, or they ask the user to add those settings to their profiles. Just say no! Try you solve the problem by other means, e.g. by creating a wrapper script, or tell the vendor to fix this problem.

Glibc installation dependency

This is a crucial piece. Because a lot depends on this software.

Bash: sh
Binutils: ar, as, ld, ranlib, readelf
Diffutils: cmp
Fileutils: chmod, cp, install, ln, mknod, mv, mkdir, rm, touch
Gcc: cc, cc1, collect2, cpp, gcc
Grep: egrep, grep
Gzip: gzip
Make: make
Gawk: gawk
Sed: sed
Sh-utils: date, expr, hostname, pwd, uname
Texinfo: install-info, makeinfo
Textutils: cat, cut, sort, tr