Saturday, September 30, 2023

Jacksum: How to skip files during hash calculation or the verification process

You could skip files during the hash calcualation process so that you do not create hash values at all for particular files and/or you could skip files during the verification process.

Skip files during hash calculation

To keep the example short, let's say we don't want to hash program libraries. Those files usually end with .dll on Windows, and with .so on GNU/Linux.

1. Create a list of files

Let's say we want to list all files in the current working directory (.) and below.

On GNU/Linux and macOS:

> find . -type f > files.list

On Windows:

> dir /A-D /B /S . > files.list

Using Jacksum:

> jacksum --style files-only . > files.list

2. Modify the file list as required

The -v option for grep and findstr lists files only that do not match the criteria given. You can use regular expressions. The $ at the end marks the end of a string.

On GNU/Linux or macOS:

> grep -v ".so$" files.list > files-filtered.list

On Windows:

> findstr /V ".dll$" files.list > files-filtered.list

3. Hash the file list

We can hash that list using the Jacksum option -L (or --file-list):

> jacksum -a sha256 -L files-filtered.list > hashes.sha256


Skip files during the verification process

If you want to ignore particular paths from a hash file you can do it in a similar way.

1. Modify the hash file as required

On GNU/Linux or macOS:

> grep -v ".so$" hashes.sha256 > hashes-filtered.sha256

On Windows:

> findstr /V ".dll$" hashes.sha256 > hashes-filtered.sha256

2. Start the integrity verification process

> jacksum -a sha256 -c hashes-filtered.sha256 .

Note: the dot at the end of the command says: verify not only the integrity of the paths stored in the hashes.sha256, but also read the current working directory (.) and all files below (-r max is set implicitly). With that you will also find new files that have been added after the file called hashes.sha256 was produced. Without the dot you will verify the files only that have a trace in hashes-filtered.sha256.



Saturday, September 23, 2023

How to create unique and secure passwords for websites with a master password and Jacksum


Jacksum 3.7.0 introduced a new feature: read text from the console without echoing, hash that string using your preferred hash-function and encode the hash-value using your preferred encoding. In other words, you can treat Jacksum as a password generator that generates uniq and strong passwords.

You only have to remember ONE secure password. Yet, you can still use different, strong passwords for all your accounts.

No password manager is required, because nothing is stored on disk, the master password will be in your brain only. Even the generated passwords are not stored somewhere, they will be regenerated on demand.

In the following examples I use "do-not-use-this-password" as the master password for demonstration purposes only. Note that this password is not secure (since it is public here and easy to remember) and shouldn't be used. Actually you should select a secure password. To learn how to create secure passwords go to the recommendation of the BSI.

Let's see how we can generate uniq strong passwords using Jacksum ...


Simple approach with sha256 and base64-nopadding encoding (a-z, A-Z, 0-9, two special chars: +/ ):

> jacksum -a sha256 -8 -q password -E base64-nopadding
Password: facebook.com do-not-use-this-password

IMY5mnvgt44sWLNZZxOusnwepHP0mAJjMH4Q0rE2AF8

 

Better approach with HMAC:sha256 and base64-nopadding-encoding (a-z, A-Z, 0-9, two special chars: +/ ):

Hashing using a a crytpographic hash algorithm is a one-way action, and in theory it is impossible to calculate the master password if someone knows the hashed password. However, there is still the risk of precalculated rainbow tables that increses the speed of an attack in contrast to a brute-force attack. Therefore you can make it even more secure if you use an HMAC. The advantage with the HMAC is that you can individualize the actual hashing with another secret which makes precalculated rainbow tables useless for attack purposes which increases the security again. Since we need to keep the HMAC key and the master password as secret, we can simply use that token for both. Otherwise we would have to remember two passwords.

After entering the command, you will be asked twice. The 1st password prompt is the key for the HMAC, the 2nd password prompt is your actual message that you want to hash. You can use the same master password at both prompts.

> jacksum -a hmac:sha256 -8 -k password -q password -E base64-nopadding
Password:
do-not-use-this-password
Password: facebook.com do-not-use-this-password


5kAL95XIPnB/yDFZcMgWDOo72kFnvDeMvnzgju+8xuM

 

Strong approach with HMAC:sha3-512:240 and base64-encoding (a-z, A-Z, 0-9, two special chars: +/ ):

In order to avoid padding for the base64 encoding and not to use base64-nopadding, the number of encoded bytes (the hash) must be a multiple of 3. So we can truncate the HMAC by specifying HMAC:sha256:240 which will result in a 30 byte hash. Since it increases the security further if we don't store the entire hash to the website, but only a fraction of it - with the knowledge of only a fraction of the hash it is impossible to precalculate a 1:1 relationship between the hash and the password, because more than 50% of the hash is not being transferred, we could also use the more modern SHA3-512 and truncate it after 240 bits. This will result in a 40 character password with at least 3 different character categories (upper- and lowercase characters, numbers, and sometimes even a special character).

> jacksum -a hmac:sha3-512:240 -8 -k password -q password -E base64
Password:
do-not-use-this-password
Password: facebook.com do-not-use-this-password


ms2WOeYyYShCs9txS6jc5UtagHTviGJtIrZYRT+e

 

The examples above show how to read a password or passphrase from the console, not including any line-termination characters. You can use that mode to generate both unique and strong passwords for websites. The returned hash values will be stronger than anything a normal human brain could remember, and nothing needs to be stored on disk. If the password is compromised, the master password will still remain secret. Oh, please do not use the password from the example above, because now it is known to all people who read this article.

Some notes and recommendations:

  • Combine a master password with website-specific information, such as the domain name, to get unique passwords. In the example above I have used facebook.com
  • For the master password you should use a strong password that is at least 8 characters long; the longer, the better, and you should be able to remember it easily. Again, please do not use the password from the example above.
  • You should use a non-broken, strong cryptographic hash algorithm for the task. In the example I have used "ascon-hash" which is supported since version 3.7.0 of Jacksum. For more information about the Ascon-Hash, please type `jacksum -h ascon-hash`. Of course, if you like it stronger, you could also use sha3-512 for example.
  • Set the character set explicitly if you use multiple different operating systems or environments that do not use UTF-8 for the console by default, and make sure to remember the character set as well. In the example above I have used option -8 which sets UTF-8 for both stdout and stderr. See also `jacksum -h -8`.
  • You can use all of the available characters for the password.
  • You can copy and paste the password to the prompt, but it is better if you remember it and type it, as this trains your brain to remember the password, and your brain will be the only place where the master password resides.
  • Dependent on the allowed characters of the website login you can select one of the encodings that Jacksum supports. See also `jacksum -h -E`. In the example above I have used z85 which perfectly works for most websites. Alternatively I recommend base64-nopadding or base32hex-nopadding.
  • To increase the security further you can use HMAC. Note that not all hash algorithms work as a HMAC.

Due to security reasons the following limits apply:

  • You won't see the password that you enter; in other words, echoing is disabled to prevent shoulder-surfing attacks.
  • Only the hash is printed by default; it is not possible to print the password in clear text, even if particular format options such as -F or --style are set. Use option "-q readline" if you prefer echoing.
  • Operating system piping is not possible, because we want to make sure that the password is coming from a keyboard and not transfer passwords in clear text between processes via piping.
  • Operating system redirection is not possible, because a console is required to enter the password. Use the options -o/-O to save the hash to a file if you do no wish to see the hash value in the console.
  • To minimize the lifetime of sensitive data in memory, the password is cleared from memory after processing. Java's String interning is not used for the password.

Stay safe!

Regards,
Johann

 


Wednesday, August 30, 2023

Fixed the 10µF/25V Tantalkondensator on C15 on the mainboard of the Commodore C64 floppy drive VC 1541, restored an old prg of mine and refreshed my memory

Vor kurzem habe ich mir diesen tollen Adapter gekauft, der es mir erlauben sollte ein C64-Diskettenlaufwerk über USB an meinen PC anzubinden und so alte C64-Disketten auslesen zu können.


Mein Commodore Diskettenlaufwerk VC 1541 lag über 30 Jahre lang gut verstaut vor Staub und Wasser in einer Aufbewahrungsbox und ich ging davon aus, dass ich es genauso wieder schnell in Betrieb nehmen könnte, wie ich es damals hineintat. Pustekuchen. Leider qualmte das Diskettenlaufwerk unmittelbar nach dem Einschalten aus den Lüftungsschlitzen wie eine antike Dampflok und auch olfaktorisch merkte ich schnell dass was nicht stimmte. Mit anderen Worten: es stank - und zwar gewaltig - nach verbranntem Elektronikbauteil! Schnell abgeschaltet, abgesteckt, abkühlen lassen, dann das Laufwerk auf den Bauch gedreht. Vier Schräubchen sind schnell gelöst und ich habe den Gehäusedeckel abgemacht. Sofort fällt mein Blick auf einen kleinen, verkohlten Stumpf, vor kurzem noch ein Kondensator - möglicherweise, denn die Werte kann ich nicht mehr erkennen. Auf C15 steckt er jedenfalls. Schnell gegoogelt. Aha, ein Tantal-Kondensator, 10µF/25V. Gibts für 0,35 €, Versandkosten 5,95 €. Na toll. Egal. Bestellt. Gleich mehrere - man weiss ja nie.

Vorher:

 




Ein paar Tage später: Lötkolben heiss gemacht und weg mit dem alten, neuen rein.

Nachher:


Test mit dem Adapter verlief auch ohne Probleme. Perfekt. Dann lesen wir doch mal das 5 1/4 Zoll-Diskettchen hier aus und schauen, was ich damals Nutzloses Nützliches programmiert habe ab dem Jahre 1984 und den folgenden meiner Kindheit. Aha, "potenzen.prg", "gleichungen.prg", "weihnachten.prg" usw. - klingt alles vielversprechend, auch wenn ich mich zugegeben nur noch sehr vage erinnere. VICE runtergeladen, installiert, erstes Programm geladen ...


 Und RUN it baby ...

Ach ich sage euch, es ist wunderbar, wenn man sehr alte Erinnerungen durch selbst reaktivierte alte Hardware auffrischen kann :-) Ach ja. Seufz. Nun gut. Deckel drauf, Schrauben rein und wieder ab damit in den Keller. Sehr gute Bewertung für den ebay-Händler für den tollen Adapter!

Mission completed.

Sunday, June 19, 2022

CVE-2021-44832: Find vulnerable .jar files using Jacksum 3.4.0 or later

In December 2021, a zero-day remote code execution vulnerability (CVE-2021-442281, Base Score 10.0) was found in Apache Log4j, a widely used Java logging library. The vulnerability was publicly disclosed via GitHub on Dec 9, 2021. The vulnerability allows attackers to take full control of systems without authentication. The vulnerability is also known as "log4shell".

Many recommendations out there suggest the user to find the filenames by typing the GNU/Linux command

find / -iname "*log4j*"

or to find live processes with log4j in their names

ps aux | grep –i 'log4j'
lsof | grep –i "log4j"

The problem with those approaches is that you won't find the vulnerable jar files if the file names have been renamed. You know, sometimes some vendors simply rename libs to whatever reason original jar files. Also in Java's classpath the name of the .jar file is irrelavant, from a security perspective it is the vulnerable classes in the jar file that really matter.

So a better approach is not to search only for their names, search also for their hashes!

This receipt can be used at any time a new critical vulnerability is found - even if it is not in log4j. Nonetheless I will use the log4j example to demonstrate how it works.

1. Identify the files/libs that are vulnerable


CVE-2021-44832 tells us that the issue has been fixed in Log4j 2.17.1 (for Java 8), 2.12.4 (for Java 7) and 2.3.2 (for Java 6). See also https://logging.apache.org/log4j/2.x/security.html#CVE-2021-44832

In other words all other older releases are vulnerable.

2. Download all vulnerable libs

If you don't find older libs on the apache page, you can go to mavencentral to download all vulnerable libs. Go to https://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-core
You should download all vulnerable libs and not just those that are affected by CVE-2021-44832.

3. Create hashes from the libs

Since we want to minimize false positives and minimize hash collisions, we use a non-broken, crytographic well known modern hash algorithm: SHA-3-256
which is the default in Jacksum 3.4.0. Go to the directory where all the libs have been stored and run

jacksum -a sha3-256 --header --no-path -O log4j.hashes .

Note: you don't need to specify -a, because the default is already sha3-256, but it is not guaranteed that future Jacksum releases have the same default algorithm, so it is recommended to specify the algorithm even you don't need to. You can set --header to print invocation args so you will know by which Jacksum version, which algorithm, and when the hashes were generated. Set the option called --no-path, because we don't need the path, we are just interested in the file name. The file will be written to log4j.hashes, and it will be overwritten if it exists.

4. Optional: test with the current working directory

jacksum -w log4j.hashes .

5. Optional: edit log4j.hashes and add CVS info to it

For example you could change the line from

d5a4aa7b06fd43b142caae4381e94ddbff840886470135b0f4314181e2b9bb27 log4j-core-2.17.0.jar

to

d5a4aa7b06fd43b142caae4381e94ddbff840886470135b0f4314181e2b9bb27 original: log4j-core-2.17.0.jar, vulnerability: CVE-2021-44832

and repeat that with additional comments.

6. Go and let find all files that match any of the hashes in log4j.hashes

You can use Jacksum 3.4.0 or later to do that job:

jacksum -w log4j.hashes /

Hint: with this approach you can not only find vulnerable libraries, you can also find copyright protected material, pornography, software in particular versions - anything that you know the digital hashes of.

Bonus: search your system for any Jacksum libraries:

jacksum -w jacksum.hashes /

Downloads

You find Jacksum at https://jacksum.net, and https://github.com/jonelo/jacksum. If you don't like the CLI, you can also use the File Browser Integration which comes with a GUI called HashGarten which is available for Microsoft Windows, macOS, and GNU/Linux.


Sunday, June 5, 2022

Announcement: HashGarten - a GUI for Jacksum

Announcement

I am pleased to announce HashGarten - a GUI for Jacksum. 


What the heck ...

Well, for those who don't know what I am talking about ...

  • A GUI is a graphical user interface
  • A hash is the result of a hash function. A hash function maps a bit string of arbitrary length to a bit string of fixed length. In other words, a hash is like a fingerprint for data, and you can use hashes to verify integrity of files for instance.
  • Jacksum is the free, friendly, and open source hash tool from the neighborhood, Jacksum supports more than 470 hash functions and many features, see also https://jacksum.net
  • "Garten" is the German word for "garden" as you guys from the US probably know - it is the same word as in Kindergarten.
  • HashGarten is the GUI for Jacksum. It allows you to access algorithms and features from Jacksum in a comfortable, graphical way.

Screenshot

HashGarten can run standalone and it can be accessed by the SendTo feature of your file brower, it looks like this (light and dark themes are supported) ...


 

Download

You can download Jacksum and HashGarten as part of the Jacksum File Browser Integration at https://jacksum.net


Credits

The announcement image above was made by using the great graphics from icons8.com and it's Mega Creator.

Both the light and dark look and feel of HashGarten's GUI is powered by the fantastic looking FlatLaF for Java Swing desktop apps, see also https://www.formdev.com/flatlaf/

Wednesday, May 11, 2022

Eclipse Temurin Download Buttons are now available!

Eclipse Temurin Download Buttons are now available at adoptium.net/de/temurin/button to promote Eclipse Temurin.

Eclipse Temurin is the name of the OpenJDK distribution from Adoptium. Those free prebuild OpenJDK binaries are great for both developing and running (not only) FOSS that is written in Java.

The button request was initiated by issue https://github.com/adoptium/website-v2/issues/395 ;-)


Oh, and just in case you have set Security Headers at your Web Server, don't forget to adjust your Content Security Policy so that the images from adoptium.net can be loaded properly. See also https://scotthelme.co.uk/content-security-policy-an-introduction/

Sunday, September 12, 2021

Find algorithms to a hash value using Jacksum

 If you need to find the algorithm to a CRC, checksum or hash value you can use Jacksum.

For those who don't know: Jacksum is a free and cross platform data integrity software tool. For more information go to https://jacksum.net

Let's keep things simple and let's pretend that you know your algorithm returns a message digest of 16 bits, and the the message digest is d893 in hex, lowercase. Input was 050000 in hex, lowercase. You can call jacksum with the following options:

  • Option -a unknown:16 means you don't know the algorithm, but at least you know it returns 16 bits
  • Option -E hex means you want a hexadecimal encoding for the message digest
  • Option -q hex:050000 means you want to calcualate the message digest from the hex input 050000 quickly
  • Option -e d893 means an expected hash value of d893, expressed as hex

jacksum -a unknown:16 -q hex:050000 -E hex -e d893 

produces the following output:

Trying 13 algorithms with a width of 16 bits that are supported by Jacksum 3.0.0 ...

Trying 30 CRC algorithms with a width of 16 bits by testing against well known CRCs ...
crc:16,1021,FFFF,false,false,FFFF
    --> CRC-16/GENIBUS

Trying all CRC algorithms with a width of 16 bits by brute force (be patient!) ...
crc:16,1021,FFFF,false,false,FFFF
crc:16,37D2,FFFF,true,false,FFFF
crc:16,3E2D,0000,true,false,FFFF
crc:16,4175,FFFF,true,false,FFFF
crc:16,4A5B,FFFF,true,true,0000
crc:16,5A41,FFFF,true,false,FFFF
crc:16,5C63,FFFF,true,true,0000
crc:16,6287,FFFF,true,true,0000
crc:16,649C,0000,false,true,FFFF
crc:16,6D55,FFFF,true,true,0000
crc:16,75AC,FFFF,true,false,FFFF
crc:16,7D64,FFFF,false,false,FFFF
crc:16,81A6,FFFF,true,false,FFFF
crc:16,B9F9,FFFF,true,true,0000
crc:16,C3D6,FFFF,false,false,FFFF
crc:16,D436,0000,true,false,FFFF
crc:16,D6D2,0000,false,true,FFFF
crc:16,DA9C,FFFF,true,false,FFFF
crc:16,E03E,FFFF,false,false,FFFF
crc:16,F701,FFFF,true,false,FFFF


Jacksum: algorithms tested: 1048620
Jacksum: algorithms found: 21

Jacksum: elapsed time: 6 s, 460 ms

Means Jacksum has tested more than one million algorithms in about 7 seconds and it found 21 matching algorithms. Each of those returns the same CRC value. Test with more input/output sequences and/or longer input sequences in order to find the right algorithm. The most likely algorithm is printed with a name if it is a well known CRC. In this example it has been identified as the CRC-16/GENIBUS.

Once you have identified the correct algorithm, you can calculate your own input data using the CRC definitions that have been found:

jacksum -a crc:16,1021,FFFF,false,false,FFFF -E hex -q hex:050000
d893 3

The output "d893 3" means that 3 bytes have been read (050000) in order to produce the 16 bit (2 byte) hexadecimal value d893 using the algorithm as defined by -a.

Mission completed.

Cheers,
Johann


Be very careful with echo on Windows

Recently one of my users of Jacksum reported a strange behavior if echo is used on Microsoft Windows.

For those who don't know: Jacksum is a data integrity tool which can (not only) calculate and verify hash values. For more information go to https://jacksum.net

So the report (which was sent privately to me) was this:

jacksum -q "txt:Hello World!"
d0e47486bbf4c16acac26f8b653592973c1362909f90262877089f9c8a4536af

that is correct, but if Jacksum reads from stdin and echo on Windows is used, it returns a completely different hash:

echo "Hello World!" | jacksum -
61ab266cecda9b9885aedbbb5b3d9914738cec74930301ec7b68312b6b436b8b <stdin>

Is Jacksum right?

Yes, Jacksum is right. The problem is with the echo command on Windows!

I started Jacksum with "--verbose summary" which gives us additional information:

echo "Hello World!" | jacksum --verbose summary -
61ab266cecda9b9885aedbbb5b3d9914738cec74930301ec7b68312b6b436b8b <stdin>

Jacksum: files read successfully: 1
Jacksum: files read with errors: 0
Jacksum: total bytes read: 17
Jacksum: total bytes read (human readable): 17 bytes

Jacksum: elapsed time: 139 ms

As you can see, Jacksum have read 17 bytes from stdin (the 3rd line of the summary). "Hello World!" (without quotes) are just 12 characters, what are the other 5 characters?

I pasted the output of echo to a file ...

echo "Hello World!" > hello.txt

and loaded the file in a hex editor (https://hexed.it will work perfectly well for that task)

As you can see, the Windows echo

- does not only transfers the Hello World! to the pipe (Hello World! => 12 bytes), but also ...
- it appends the Windows carriage return, line feed chars (0x0D, 0x0A => 2 bytes) which is expected on Windows
- it doesn't strip the quotes (2x" => 2 bytes) - which is probably not known to everybody, and
- ATTENTION!: it doesn't strip the blank that is between the 2nd quote (") and the redirection sign (>) => 1 byte

That means a 

echo "Hello World!"        > hello2.txt

would even add more blanks to the output.



And even if we would do this:

echo Hello World!> hello3.txt

the CR LF will always be there, because there is no -n option (which GNU/Linux and macOS have for example).

So the workaround on Microsoft Windows 10 we found is this:

echo | set /p="Hello World!" | jacksum -
d0e47486bbf4c16acac26f8b653592973c1362909f90262877089f9c8a4536af <stdin>

which produces the expected hash.

Mission completed.

BTW, on GNU/Linux and macOS you can still use echo as expected, but you need to use single quotes rather than double quotes, because the ! has a special meaning on many *nix-shells:

$ echo -n 'Hello World!' | jacksum -
d0e47486bbf4c16acac26f8b653592973c1362909f90262877089f9c8a4536af <stdin>

Cheers,
Johann

Monday, September 6, 2021

Jacksum 3 is on the web!

I am pleased to announce that Jacksum 3 has been released!

Jacksum 3 is a major release. I call it the "good things will take time" release ;-)

See also https://jacksum.net

Detailed release notes at https://jacksum.net/en/release-notes.html

Here is an eagle view comparison chart with Jacksum 1.7.0. There was never an official Jacksum 2 release and I lifted the version number immediately to 3 due to the feature richness of Jacksum 3.

 


Jacksum 1.7.0  
Jacksum 3.0.0

Algorithm support
Supported Algorithms (including different lengths) 58 470
Default algorithm SHA-1 SHA-3-256
Full SHA-3 family support no yes
Full SHA-2 family support no*1 yes
Full BLAKE-family support no almost*2
Full FNV-familiy support*3
no yes
Customizable CRCs
yes yes
Modern non-US modern national standards*4 no yes
Older non-US national standards*5 yes yes
All algos from round three of the NIST SHA-3 competition*6  
no yes
proposals from the NIST crypto workshops*7  
no yes
eXtendable Output Functions (XOF) as cryptographic hash functions*8  
no yes

Multi core/processor support
multiple algorithms calculation
yes*9 yes
multiple algorithms calculation simultanously no yes
parallel hash calculation for files no yes
multi processor support for checking files no yes

Other highlights
Finding an algorithm/CRC by knowing both data and hash
no yes
Select algorithms that match a bit width or a search string
no yes
producing the output in a format you want   
yes yes
checking an output produced by a foreign software using parser definitions   
no yes
License
GPLv2+ GPLv3+

 

*1 SHA-512/224, SHA-512/256 were missing, because those algos were introduced in March 2012.

*2 BLAKE-[224,384,256,512], BLAKE2s-[8..256/8], BLAKE2b-[8..512/8], and BLAKE3 are supported; BLAKE2sp and BLAKE2bp are not yet supported

*3 including FNV-0, FNV-1, and FNV-1a for all bit lengths [32,64,128,256,512,1024]

*4 Streebog-[256,512] (Russia, GOST R 34.11-2012); SM3 (China); Kupyna[256,384,512] (Ukraine, DSTU 7564:2014); LSH-256-[224,256], LSH-512-[224,256,384,512] (South Korea, KS X 3262)

*5 GOST, GOST Crypto-Pro (Russia, GOST R 34.11-94); HAS-160 (KISA, South Korea) 

 *6 all five candidates from round 3 the NIST SHA-3 competition were BLAKE, Groestl, JH, Keccak, and  Skein

 *7 before the SHA-3 competition they were proposals from the NIST crypto workshops called FORK-256, DHA-256, and VSH-1024

*8 SHAKE128, SHAKE256, KangarooTwelve, MarsupilamiFourteen

*9 while a file is read only once if multiple algorithms have been selected, the actual hash calculation occurred sequentially and not in parallel



 

Saturday, September 19, 2020

An AppImage for the NumericalChameleon, problems and solutions

Preface

What is an AppImage?

AppImage is a format for distributing portable software on Linux without needing superuser permissions to install the application. See also https://appimage.org/

What is the NumericalChameleon?

The NumericalChameleon is a free, open source unit converter running on the desktop on Windows, macOS, and Linux. It supports more than 6000 units in more than 90 categories. It is entirely written in Java. See also http://numericalchameleon.net

The Goal

On GNU/Linux the NumericalChameleon already supports several ways for a deployment: .deb, .rpm, self extracting file and a bzip2 compressed tarball. I wanted to support .AppImage as well, but I run into problems. That article should help other developers to bypass those issues that I have been facing with.

Problems and Solutions

While I was working on the .AppImage for the NumericalChameleon two major problems arise. To say it in advance, without code modification the AppImage did not work out of the box. Fortunately the code changes can be minimized.

Problem 1: Restart of the application didn't work

There are situations where the user has to restart the app. For example, if the user selects a different language for the GUI or if the user selects an option in order to let render the frame decoration by Java rather then by the operating system. The NumericalChameleon also offers the user to force a restart explicitly by selecting the menu item called "Restart" from the program menu. Example:

If a restart is being triggered, the current implementation determines the Java class that is being called when the user double clicks on the (portable or non-portable) jarball. It passes that class (along with some program args) to the startJarApplication method. The following is a short excerpt from the source of the NumericalChameleon 3.0.0:

                Class clazz = isPortable() ? net.numericalchameleon.launchers.MainGUIPortable.class : net.numericalchameleon.launchers.MainGUI.class;
                ProcessHelper.startJarApplication(JVMoptions, clazz, args);

See also https://github.com/jonelo/n16n-desktop

The startJarApplication method basically finds both the java executable binary and the .jar file that were used to start the app. It builds a new process using ProcessBuilder and starts it so that a new process with a new PID gets created. The parent process simply exists and the new process will survive.

That approach worked perfectly fine on all supported operating systems Windows, Linux, and macOS. However it failed on Linux with the .AppImage, because it is the AppImage itself that starts its contained application. The AppImage does that by mounting its payload in read only mode and calling an AppRun script that launches the JVM which launches the .jar file. So knowning the path to the java executable isn't helpful in this case, because it will change each time the AppImage starts.

Solution to Problem 1: Check whether we are running from an .AppImage?

The solution to the problem 1 is to determine whether the app is running from an .AppImage. And if it does, we have to restart the AppImage binary rather than the .jar file. To to that we can check, whether we are on Linux, because .AppImages are Linux only and we can check whether the system proberty called java.home starts with "/tmp/.mount", because that is the mount point that AppImage uses to mount its payload. If we are runnig from an AppImage, we also need to know the path to the .AppImage file itself. This can be done by passing the $APPIMAGE environement variable as a system property to the application launcher called AppRun:

# -- snip --
# start the application in the expected working folder ../openjdk/bin/java -Dapplication.appimage="$APPIMAGE" -jar nc.jar "$@"

 And in Java land, we need to add some code to achieve that:


        if (System.getProperty("os.name").equalsIgnoreCase("Linux")
         && System.getProperty("java.home").startsWith("/tmp/.mount_")) {
        ...
        String appimage = System.getProperty("application.appimage");

Since we now know the entire path to the AppImage, it is easy to restart the AppImage using a standard ProcessBuilder.


Problem 2: Files are not writable

The NumericalChameleon comes with a lot of files, many of them can be modified by the user. For example, if the user would like to load historic or current exchange rates, the user can do that by selecting the appropriate function in the app. However, since the AppImage mounts everything in ready-only mode, persistance is disabled and the app data cannot be updated by the user. The user has to wait for an updated AppImage. Well, since I am not going to upload an updated AppImage every day, those volatile data should be continued to be updated by the user on demand. The NC stores all changable data in the folder called data.

Solution to Problem 2: let's use a symlink to /tmp

Let's see whether we can find a solution for problem 2 that doesn't require any code chages in Java land. Since all files in the AppImage are being mounted ready-only and cannot be changed, the app image needs to hardcode a path that is available on all Linux platforms and it also must be accessible by the user. The home folder is not an option here, because the user's home is unknown at the time when the AppImage is being built. The solution is to use the /tmp path, because /tmp has the sticky bit set by default and it is writable by the user. So we bake a symlink called data that points to /tmp/.NumericalChameleon/data into the AppImage. The preciding dot is to mark that folder hidden on Linux. Here is an excerpt from my build script:

# -- snip --
# prepare a pointer to /tmp which is both known and user writable
mv "${n16nDir}/data" "${n16nDir}/data.ori"
cd "${n16nDir}"
ln -s /tmp/.NumericalChameleon/data data
cd -
"./${appImageTool}" --no-appstream "${appdir}" "${outdir}/${outfile}"

If the application starts, we simply need to check whether there is an up to date data folder and if it isn't, we simply copy the actual data folder from the mounted .AppImage mount point to /tmp/.NumericalChameleon/data:

# -- snip --
if [[ $refresh -eq 1 ]]; then
    mkdir -p /tmp/.NumericalChameleon
    printf "copying data files to %s\n" "$dataFolder"
    cp -r ../data.ori/. "$dataFolder"
    printf "%s\n" "$currentVersion" > "$versionFile"
fi

# start the application in the expected working folder

../openjdk/bin/java -Dapplication.appimage="$APPIMAGE" -jar nc.jar "$@"


And now we also can update the exchange rates on the fly:

./NumericalChameleon-x86_64.AppImage --filter ebc.europa.eu --continue

Mission completed.

The next release of the NumericalChameleon will also support the AppImage :-)