dash maximum variable length under systemd

manpreet Best Answer 2 years ago

I have a shell script that uses a single variable as an associative array (one KEY=VALUE per line).

Throughout the execution of the script, the variable is manipulated to add, remove or modify entries:

Append:

VARIABLE="$(printf "%s\n%s" "$VARIABLE" "KEY=VALUE")"

Modify:

VARIABLE="$(printf "%s\n" "$VARIABLE" | sed -E "s,^(KEY=).*$,\1VALUE,")"

Remove:

VARIABLE="$(printf "%s\n" "$VARIABLE" | grep -E -v "^KEY=.*$")"

When executed in a terminal (or as a service on an old machine under sysv through an init script), the script runs fine, but when run as a service under systemd, after a while, the script starts to spit out error messages in the log :

sh: printf: I/O error

After a lot of trial and error, I couldn't determine exactly what command(s) in the script produced those errors, but I noticed that they start to appear when the length of the variable reaches 8000 bytes (I guess 8192, but I couldn't pinpoint it exactly since whole lines are appended).

I'm pretty sure the variable length is the problem because I implemented a routine that trims the oldest entries of the array whenever the variable length approaches 8192 bytes, and now the script does run under systemd for a long time without errors; but that's of course not ideal, as there is some information lost.

I searched the web for information about maximum variable length in shell scripts, but didn't find anything useful :

• dash manual page doesn't say anything about maximum variable length.

• GNU sed documentation says:

For those who want to write portable sed scripts, be aware that some implementations have been known to limit line lengths (for the pattern and hold spaces) to be no more than 4000 bytes. The POSIX standard specifies that conforming sed implementations shall support at least 8192 byte line lengths. GNU sed has no built-in limit on line length; as long as it can malloc() more (virtual) memory, you can feed or construct lines as long as you like.

...but this applies to line length, not to the whole text length (the individual lines don't exceed 80 characters)

Anyway, since the errors appear only when the script is run through systemd, I tried to increase LimitMSGQUEUE and/or LimitSTACK in the unit file, to no avail (this was a blind guess, as I don't understand exactly the concepts of message queues or process stack, but the numbers shown by systemctl show looked like 8 KB or so). All other limits regarding memory (LimitRSS, LimitAS, LimitMEMLOCK) seem high enough (way past 8192 bytes), so I don't know what to do next.

What can I do to get this script running under systemd without errors when the variable length exceeds 8 KB ?