Johann N. Löfflmann's Weblog

Sunday, September 12, 2021

Find algorithms to a hash value using Jacksum

If you need to find the algorithm to a CRC, checksum or hash value you can use Jacksum.

For those who don't know: Jacksum is a free and cross platform data integrity software tool. For more information go to https://jacksum.net

Let's keep things simple and let's pretend that you know your algorithm returns a message digest of 16 bits, and the the message digest is d893 in hex, lowercase. Input was 050000 in hex, lowercase. You can call jacksum with the following options:

Option -a unknown:16 means you don't know the algorithm, but at least you know it returns 16 bits
Option -E hex means you want a hexadecimal encoding for the message digest
Option -q hex:050000 means you want to calcualate the message digest from the hex input 050000 quickly
Option -e d893 means an expected hash value of d893, expressed as hex

jacksum -a unknown:16 -q hex:050000 -E hex -e d893

produces the following output:

Trying 13 algorithms with a width of 16 bits that are supported by Jacksum 3.0.0 ...

Trying 30 CRC algorithms with a width of 16 bits by testing against well known CRCs ...
crc:16,1021,FFFF,false,false,FFFF
--> CRC-16/GENIBUS

Trying all CRC algorithms with a width of 16 bits by brute force (be patient!) ...
crc:16,1021,FFFF,false,false,FFFF
crc:16,37D2,FFFF,true,false,FFFF
crc:16,3E2D,0000,true,false,FFFF
crc:16,4175,FFFF,true,false,FFFF
crc:16,4A5B,FFFF,true,true,0000
crc:16,5A41,FFFF,true,false,FFFF
crc:16,5C63,FFFF,true,true,0000
crc:16,6287,FFFF,true,true,0000
crc:16,649C,0000,false,true,FFFF
crc:16,6D55,FFFF,true,true,0000
crc:16,75AC,FFFF,true,false,FFFF
crc:16,7D64,FFFF,false,false,FFFF
crc:16,81A6,FFFF,true,false,FFFF
crc:16,B9F9,FFFF,true,true,0000
crc:16,C3D6,FFFF,false,false,FFFF
crc:16,D436,0000,true,false,FFFF
crc:16,D6D2,0000,false,true,FFFF
crc:16,DA9C,FFFF,true,false,FFFF
crc:16,E03E,FFFF,false,false,FFFF
crc:16,F701,FFFF,true,false,FFFF

Jacksum: algorithms tested: 1048620
Jacksum: algorithms found: 21

Jacksum: elapsed time: 6 s, 460 ms

Means Jacksum has tested more than one million algorithms in about 7 seconds and it found 21 matching algorithms. Each of those returns the same CRC value. Test with more input/output sequences and/or longer input sequences in order to find the right algorithm. The most likely algorithm is printed with a name if it is a well known CRC. In this example it has been identified as the CRC-16/GENIBUS.

Once you have identified the correct algorithm, you can calculate your own input data using the CRC definitions that have been found:

jacksum -a crc:16,1021,FFFF,false,false,FFFF -E hex -q hex:050000
d893 3

The output "d893 3" means that 3 bytes have been read (050000) in order to produce the 16 bit (2 byte) hexadecimal value d893 using the algorithm as defined by -a.

Mission completed.

Cheers,
Johann

Be very careful with echo on Windows

Recently one of my users of Jacksum reported a strange behavior if echo is used on Microsoft Windows.

For those who don't know: Jacksum is a data integrity tool which can (not only) calculate and verify hash values. For more information go to https://jacksum.net

So the report (which was sent privately to me) was this:

jacksum -q "txt:Hello World!"
d0e47486bbf4c16acac26f8b653592973c1362909f90262877089f9c8a4536af

that is correct, but if Jacksum reads from stdin and echo on Windows is used, it returns a completely different hash:

echo "Hello World!" | jacksum -
61ab266cecda9b9885aedbbb5b3d9914738cec74930301ec7b68312b6b436b8b <stdin>

Is Jacksum right?

Yes, Jacksum is right. The problem is with the echo command on Windows!

I started Jacksum with "--verbose summary" which gives us additional information:

echo "Hello World!" | jacksum --verbose summary -
61ab266cecda9b9885aedbbb5b3d9914738cec74930301ec7b68312b6b436b8b <stdin>

Jacksum: files read successfully: 1
Jacksum: files read with errors: 0
Jacksum: total bytes read: 17
Jacksum: total bytes read (human readable): 17 bytes

Jacksum: elapsed time: 139 ms

As you can see, Jacksum have read 17 bytes from stdin (the 3rd line of the summary). "Hello World!" (without quotes) are just 12 characters, what are the other 5 characters?

I pasted the output of echo to a file ...

echo "Hello World!" > hello.txt

and loaded the file in a hex editor (https://hexed.it will work perfectly well for that task)

As you can see, the Windows echo

- does not only transfers the Hello World! to the pipe (Hello World! => 12 bytes), but also ...
- it appends the Windows carriage return, line feed chars (0x0D, 0x0A => 2 bytes) which is expected on Windows
- it doesn't strip the quotes (2x" => 2 bytes) - which is probably not known to everybody, and
- ATTENTION!: it doesn't strip the blank that is between the 2nd quote (") and the redirection sign (>) => 1 byte

That means a

echo "Hello World!" > hello2.txt

would even add more blanks to the output.

And even if we would do this:

echo Hello World!> hello3.txt

the CR LF will always be there, because there is no -n option (which GNU/Linux and macOS have for example).

So the workaround on Microsoft Windows 10 we found is this:

echo | set /p="Hello World!" | jacksum -
d0e47486bbf4c16acac26f8b653592973c1362909f90262877089f9c8a4536af <stdin>

which produces the expected hash.

Mission completed.

BTW, on GNU/Linux and macOS you can still use echo as expected, but you need to use single quotes rather than double quotes, because the ! has a special meaning on many *nix-shells:

$ echo -n 'Hello World!' | jacksum -
d0e47486bbf4c16acac26f8b653592973c1362909f90262877089f9c8a4536af <stdin>

Cheers,
Johann

Monday, September 6, 2021

Jacksum 3 is on the web!

I am pleased to announce that Jacksum 3 has been released!

Jacksum 3 is a major release. I call it the "good things will take time" release ;-)

	Jacksum 1.7.0	Jacksum 3.0.0
Algorithm support
Supported Algorithms (including different lengths)	58	470
Default algorithm	SHA-1	SHA-3-256
Full SHA-3 family support	no	yes
Full SHA-2 family support	no*1	yes
Full BLAKE-family support	no	almost*2
Full FNV-familiy support*3	no	yes
Customizable CRCs	yes	yes
Modern non-US modern national standards*4	no	yes
Older non-US national standards*5	yes	yes
All algos from round three of the NIST SHA-3 competition*6	no	yes
proposals from the NIST crypto workshops*7	no	yes
eXtendable Output Functions (XOF) as cryptographic hash functions*8	no	yes
Multi core/processor support
multiple algorithms calculation	yes*9	yes
multiple algorithms calculation simultanously	no	yes
parallel hash calculation for files	no	yes
multi processor support for checking files	no	yes
Other highlights
Finding an algorithm/CRC by knowing both data and hash	no	yes
Select algorithms that match a bit width or a search string	no	yes
producing the output in a format you want	yes	yes
checking an output produced by a foreign software using parser definitions	no	yes
License	GPLv2+	GPLv3+

Saturday, September 19, 2020

An AppImage for the NumericalChameleon, problems and solutions

Preface

What is an AppImage?

AppImage is a format for distributing portable software on Linux without needing superuser permissions to install the application. See also https://appimage.org/

What is the NumericalChameleon?

The NumericalChameleon is a free, open source unit converter running on the desktop on Windows, macOS, and Linux. It supports more than 6000 units in more than 90 categories. It is entirely written in Java. See also http://numericalchameleon.net

The Goal

On GNU/Linux the NumericalChameleon already supports several ways for a deployment: .deb, .rpm, self extracting file and a bzip2 compressed tarball. I wanted to support .AppImage as well, but I run into problems. That article should help other developers to bypass those issues that I have been facing with.

Problems and Solutions

While I was working on the .AppImage for the NumericalChameleon two major problems arise. To say it in advance, without code modification the AppImage did not work out of the box. Fortunately the code changes can be minimized.

Problem 1: Restart of the application didn't work

There are situations where the user has to restart the app. For example, if the user selects a different language for the GUI or if the user selects an option in order to let render the frame decoration by Java rather then by the operating system. The NumericalChameleon also offers the user to force a restart explicitly by selecting the menu item called "Restart" from the program menu. Example:

If a restart is being triggered, the current implementation determines the Java class that is being called when the user double clicks on the (portable or non-portable) jarball. It passes that class (along with some program args) to the startJarApplication method. The following is a short excerpt from the source of the NumericalChameleon 3.0.0:

Class clazz = isPortable() ? net.numericalchameleon.launchers.MainGUIPortable.class : net.numericalchameleon.launchers.MainGUI.class;
ProcessHelper.startJarApplication(JVMoptions, clazz, args);

The startJarApplication method basically finds both the java executable binary and the .jar file that were used to start the app. It builds a new process using ProcessBuilder and starts it so that a new process with a new PID gets created. The parent process simply exists and the new process will survive.

That approach worked perfectly fine on all supported operating systems Windows, Linux, and macOS. However it failed on Linux with the .AppImage, because it is the AppImage itself that starts its contained application. The AppImage does that by mounting its payload in read only mode and calling an AppRun script that launches the JVM which launches the .jar file. So knowning the path to the java executable isn't helpful in this case, because it will change each time the AppImage starts.

Solution to Problem 1: Check whether we are running from an .AppImage?

The solution to the problem 1 is to determine whether the app is running from an .AppImage. And if it does, we have to restart the AppImage binary rather than the .jar file. To to that we can check, whether we are on Linux, because .AppImages are Linux only and we can check whether the system proberty called java.home starts with "/tmp/.mount", because that is the mount point that AppImage uses to mount its payload. If we are runnig from an AppImage, we also need to know the path to the .AppImage file itself. This can be done by passing the $APPIMAGE environement variable as a system property to the application launcher called AppRun:

# -- snip --
# start the application in the expected working folder
../openjdk/bin/java -Dapplication.appimage="$APPIMAGE" -jar nc.jar "$@"

And in Java land, we need to add some code to achieve that:

        if (System.getProperty("os.name").equalsIgnoreCase("Linux")
         && System.getProperty("java.home").startsWith("/tmp/.mount_")) {
...
      String appimage = System.getProperty("application.appimage");

Since we now know the entire path to the AppImage, it is easy to restart the AppImage using a standard ProcessBuilder.

Problem 2: Files are not writable

The NumericalChameleon comes with a lot of files, many of them can be modified by the user. For example, if the user would like to load historic or current exchange rates, the user can do that by selecting the appropriate function in the app. However, since the AppImage mounts everything in ready-only mode, persistance is disabled and the app data cannot be updated by the user. The user has to wait for an updated AppImage. Well, since I am not going to upload an updated AppImage every day, those volatile data should be continued to be updated by the user on demand. The NC stores all changable data in the folder called data.

Solution to Problem 2: let's use a symlink to /tmp

Let's see whether we can find a solution for problem 2 that doesn't require any code chages in Java land. Since all files in the AppImage are being mounted ready-only and cannot be changed, the app image needs to hardcode a path that is available on all Linux platforms and it also must be accessible by the user. The home folder is not an option here, because the user's home is unknown at the time when the AppImage is being built. The solution is to use the /tmp path, because /tmp has the sticky bit set by default and it is writable by the user. So we bake a symlink called data that points to /tmp/.NumericalChameleon/data into the AppImage. The preciding dot is to mark that folder hidden on Linux. Here is an excerpt from my build script:

# -- snip --
# prepare a pointer to /tmp which is both known and user writable
mv "${n16nDir}/data" "${n16nDir}/data.ori"
cd "${n16nDir}"
ln -s /tmp/.NumericalChameleon/data data
cd -
"./${appImageTool}" --no-appstream "${appdir}" "${outdir}/${outfile}"

If the application starts, we simply need to check whether there is an up to date data folder and if it isn't, we simply copy the actual data folder from the mounted .AppImage mount point to /tmp/.NumericalChameleon/data:

# -- snip --
if [[ $refresh -eq 1 ]]; then
    mkdir -p /tmp/.NumericalChameleon
    printf "copying data files to %s\n" "$dataFolder"
    cp -r ../data.ori/. "$dataFolder"
    printf "%s\n" "$currentVersion" > "$versionFile"
fi

# start the application in the expected working folder
../openjdk/bin/java -Dapplication.appimage="$APPIMAGE" -jar nc.jar "$@"

And now we also can update the exchange rates on the fly:

./NumericalChameleon-x86_64.AppImage --filter ebc.europa.eu --continue

Mission completed.

The next release of the NumericalChameleon will also support the AppImage :-)