Jacksum: How to skip files during hash calculation or the verification process

You could skip files during the hash calcualation process so that you do not create hash values at all for particular files and/or you could skip files during the verification process.

Skip files during hash calculation

To keep the example short, let's say we don't want to hash program libraries. Those files usually end with .dll on Windows, and with .so on GNU/Linux.

1. Create a list of files

Let's say we want to list all files in the current working directory (.) and below.

On GNU/Linux and macOS:

> find . -type f > files.list

On Windows:

> dir /A-D /B /S . > files.list

Using Jacksum:

> jacksum --style files-only . > files.list

2. Modify the file list as required

The -v option for grep and findstr lists files only that do not match the criteria given. You can use regular expressions. The $ at the end marks the end of a string.

On GNU/Linux or macOS:

> grep -v ".so$" files.list > files-filtered.list

On Windows:

> findstr /V ".dll$" files.list > files-filtered.list

3. Hash the file list

We can hash that list using the Jacksum option -L (or --file-list):

> jacksum -a sha256 -L files-filtered.list > hashes.sha256

Skip files during the verification process

If you want to ignore particular paths from a hash file you can do it in a similar way.

1. Modify the hash file as required

On GNU/Linux or macOS:

> grep -v ".so$" hashes.sha256 > hashes-filtered.sha256

On Windows:

> findstr /V ".dll$" hashes.sha256 > hashes-filtered.sha256

2. Start the integrity verification process

> jacksum -a sha256 -c hashes-filtered.sha256 .

Note: the dot at the end of the command says: verify not only the integrity of the paths stored in the hashes.sha256, but also read the current working directory (.) and all files below (-r max is set implicitly). With that you will also find new files that have been added after the file called hashes.sha256 was produced. Without the dot you will verify the files only that have a trace in hashes-filtered.sha256.

How to create unique and secure passwords for websites with a master password and Jacksum

Jacksum 3.7.0 introduced a new feature: read text from the console without echoing, hash that string using your preferred hash-function and encode the hash-value using your preferred encoding. In other words, you can treat Jacksum as a password generator that generates uniq and strong passwords.

You only have to remember ONE secure password. Yet, you can still use different, strong passwords for all your accounts.

No password manager is required, because nothing is stored on disk, the master password will be in your brain only. Even the generated passwords are not stored somewhere, they will be regenerated on demand.

In the following examples I use "do-not-use-this-password" as the master password for demonstration purposes only. Note that this password is not secure (since it is public here and easy to remember) and shouldn't be used. Actually you should select a secure password. To learn how to create secure passwords go to the recommendation of the BSI.

Let's see how we can generate uniq strong passwords using Jacksum ...

Simple approach with sha256 and base64-nopadding encoding (a-z, A-Z, 0-9, two special chars: +/ ):

> jacksum -a sha256 -8 -q password -E base64-nopadding
Password: facebook.com do-not-use-this-password

IMY5mnvgt44sWLNZZxOusnwepHP0mAJjMH4Q0rE2AF8

 

Better approach with HMAC:sha256 and base64-nopadding-encoding (a-z, A-Z, 0-9, two special chars: +/ ):

Hashing using a a crytpographic hash algorithm is a one-way action, and in theory it is impossible to calculate the master password if someone knows the hashed password. However, there is still the risk of precalculated rainbow tables that increses the speed of an attack in contrast to a brute-force attack. Therefore you can make it even more secure if you use an HMAC. The advantage with the HMAC is that you can individualize the actual hashing with another secret which makes precalculated rainbow tables useless for attack purposes which increases the security again. Since we need to keep the HMAC key and the master password as secret, we can simply use that token for both. Otherwise we would have to remember two passwords.

After entering the command, you will be asked twice. The 1st password prompt is the key for the HMAC, the 2nd password prompt is your actual message that you want to hash. You can use the same master password at both prompts.

> jacksum -a hmac:sha256 -8 -k password -q password -E base64-nopadding
Password: do-not-use-this-password
Password: facebook.com do-not-use-this-password

5kAL95XIPnB/yDFZcMgWDOo72kFnvDeMvnzgju+8xuM

 

Strong approach with HMAC:sha3-512:240 and base64-encoding (a-z, A-Z, 0-9, two special chars: +/ ):

In order to avoid padding for the base64 encoding and not to use base64-nopadding, the number of encoded bytes (the hash) must be a multiple of 3. So we can truncate the HMAC by specifying HMAC:sha256:240 which will result in a 30 byte hash. Since it increases the security further if we don't store the entire hash to the website, but only a fraction of it - with the knowledge of only a fraction of the hash it is impossible to precalculate a 1:1 relationship between the hash and the password, because more than 50% of the hash is not being transferred, we could also use the more modern SHA3-512 and truncate it after 240 bits. This will result in a 40 character password with at least 3 different character categories (upper- and lowercase characters, numbers, and sometimes even a special character).

> jacksum -a hmac:sha3-512:240 -8 -k password -q password -E base64
Password: do-not-use-this-password
Password: facebook.com do-not-use-this-password

ms2WOeYyYShCs9txS6jc5UtagHTviGJtIrZYRT+e

The examples above show how to read a password or passphrase from the console, not including any line-termination characters. You can use that mode to generate both unique and strong passwords for websites. The returned hash values will be stronger than anything a normal human brain could remember, and nothing needs to be stored on disk. If the password is compromised, the master password will still remain secret. Oh, please do not use the password from the example above, because now it is known to all people who read this article.

Some notes and recommendations:

• Combine a master password with website-specific information, such as the domain name, to get unique passwords. In the example above I have used facebook.com
  
• For the master password you should use a strong password that is at least 8 characters long; the longer, the better, and you should be able to remember it easily. Again, please do not use the password from the example above.
  
• You should use a non-broken, strong cryptographic hash algorithm for the task. For the examples above I have used sha256, and sha3-512.
• Set the character set explicitly if you use multiple different operating systems or environments that do not use UTF-8 for the console by default, and make sure to remember the character set as well. In the example above I have used option -8 which sets UTF-8 for both stdout and stderr. See also `jacksum -h -8`.
  
• You can use all of the available characters for the password.
• You can copy and paste the password to the prompt, but it is better if you remember it and type it, as this trains your brain to remember the password, and your brain will be the only place where the master password resides.
• Dependent on the allowed characters of the website login you can select one of the encodings that Jacksum supports. See also `jacksum -h -E`. In the example above I have used base64 resp. base64-nopadding which perfectly works for the password complexity requirements of most websites. Alternatively I can also recommend base32hex-nopadding or z85.
  
• To increase the security further you can use a HMAC. Note that not all hash algorithms work as a HMAC. Or, use a truncated HMAC which is even more secure, because we do not reveal the entire hash value to the website's password store.
  

Due to security reasons the following limits apply:

• You won't see the password that you enter; in other words, echoing is disabled to prevent shoulder-surfing attacks.
• Only the hash is printed by default; it is not possible to print the password in clear text, even if particular format options such as -F or --style are set. Use option "-q readline" if you prefer echoing.
• Operating system piping is not possible, because we want to make sure that the password is coming from a keyboard and not transfer passwords in clear text between processes via piping.
• Operating system redirection is not possible, because a console is required to enter the password. Use the options -o/-O to save the hash to a file if you do no wish to see the hash value in the console.
• To minimize the lifetime of sensitive data in memory, the password is cleared from memory after processing. Java's String interning is not used for the password.

Stay safe!

Regards,
Johann

 

CVE-2021-44832: Find vulnerable .jar files using Jacksum 3.4.0 or later

In December 2021, a zero-day remote code execution vulnerability (CVE-2021-442281, Base Score 10.0) was found in Apache Log4j, a widely used Java logging library. The vulnerability was publicly disclosed via GitHub on Dec 9, 2021. The vulnerability allows attackers to take full control of systems without authentication. The vulnerability is also known as "log4shell".

Many recommendations out there suggest the user to find the filenames by typing the GNU/Linux command

find / -iname "*log4j*"

or to find live processes with log4j in their names

ps aux | grep –i 'log4j'
lsof | grep –i "log4j"

The problem with those approaches is that you won't find the vulnerable jar files if the file names have been renamed. You know, sometimes some vendors simply rename libs to whatever reason original jar files. Also in Java's classpath the name of the .jar file is irrelavant, from a security perspective it is the vulnerable classes in the jar file that really matter.

So a better approach is not to search only for their names, search also for their hashes!

This receipt can be used at any time a new critical vulnerability is found - even if it is not in log4j. Nonetheless I will use the log4j example to demonstrate how it works.

1. Identify the files/libs that are vulnerable

CVE-2021-44832 tells us that the issue has been fixed in Log4j 2.17.1 (for Java 8), 2.12.4 (for Java 7) and 2.3.2 (for Java 6). See also https://logging.apache.org/log4j/2.x/security.html#CVE-2021-44832

In other words all other older releases are vulnerable.

2. Download all vulnerable libs

If you don't find older libs on the apache page, you can go to mavencentral to download all vulnerable libs. Go to https://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-core
You should download all vulnerable libs and not just those that are affected by CVE-2021-44832.

3. Create hashes from the libs

Since we want to minimize false positives and minimize hash collisions, we use a non-broken, crytographic well known modern hash algorithm: SHA-3-256
which is the default in Jacksum 3.4.0. Go to the directory where all the libs have been stored and run

jacksum -a sha3-256 --header --no-path -O log4j.hashes .

Note: you don't need to specify -a, because the default is already sha3-256, but it is not guaranteed that future Jacksum releases have the same default algorithm, so it is recommended to specify the algorithm even you don't need to. You can set --header to print invocation args so you will know by which Jacksum version, which algorithm, and when the hashes were generated. Set the option called --no-path, because we don't need the path, we are just interested in the file name. The file will be written to log4j.hashes, and it will be overwritten if it exists.

4. Optional: test with the current working directory

jacksum -w log4j.hashes .

5. Optional: edit log4j.hashes and add CVS info to it

For example you could change the line from

d5a4aa7b06fd43b142caae4381e94ddbff840886470135b0f4314181e2b9bb27 log4j-core-2.17.0.jar

to

d5a4aa7b06fd43b142caae4381e94ddbff840886470135b0f4314181e2b9bb27 original: log4j-core-2.17.0.jar, vulnerability: CVE-2021-44832

and repeat that with additional comments.

6. Go and let find all files that match any of the hashes in log4j.hashes

You can use Jacksum 3.4.0 or later to do that job:

jacksum -w log4j.hashes /

Hint: with this approach you can not only find vulnerable libraries, you can also find copyright protected material, pornography, software in particular versions - anything that you know the digital hashes of.

Bonus: search your system for any Jacksum libraries:

jacksum -w jacksum.hashes /

Downloads

You find Jacksum at https://jacksum.net, and https://github.com/jonelo/jacksum. If you don't like the CLI, you can also use the File Browser Integration which comes with a GUI called HashGarten which is available for Microsoft Windows, macOS, and GNU/Linux.

Announcement: HashGarten - a GUI for Jacksum

Announcement

I am pleased to announce HashGarten - a GUI for Jacksum. 

What the heck ...

Well, for those who don't know what I am talking about ...

• A GUI is a graphical user interface
• A hash is the result of a hash function. A hash function maps a bit string of arbitrary length to a bit string of fixed length. In other words, a hash is like a fingerprint for data, and you can use hashes to verify integrity of files for instance.
  
• Jacksum is the free, friendly, and open source hash tool from the neighborhood, Jacksum supports more than 470 hash functions and many features, see also https://jacksum.net
• "Garten" is the German word for "garden" as you guys from the US probably know - it is the same word as in Kindergarten.
  
• HashGarten is the GUI for Jacksum. It allows you to access algorithms and features from Jacksum in a comfortable, graphical way.

Screenshot

HashGarten can run standalone and it can be accessed by the SendTo feature of your file brower, it looks like this (light and dark themes are supported) ...

 

Download

You can download Jacksum and HashGarten as part of the Jacksum File Browser Integration at https://jacksum.net

Credits

The announcement image above was made by using the great graphics from icons8.com and it's Mega Creator.

Both the light and dark look and feel of HashGarten's GUI is powered by the fantastic looking FlatLaF for Java Swing desktop apps, see also https://www.formdev.com/flatlaf/

Find algorithms to a hash value using Jacksum

 If you need to find the algorithm to a CRC, checksum or hash value you can use Jacksum.

For those who don't know: Jacksum is a free and cross platform data integrity software tool. For more information go to https://jacksum.net

Let's keep things simple and let's pretend that you know your algorithm returns a message digest of 16 bits, and the the message digest is d893 in hex, lowercase. Input was 050000 in hex, lowercase. You can call jacksum with the following options:

• Option -a unknown:16 means you don't know the algorithm, but at least you know it returns 16 bits
• Option -E hex means you want a hexadecimal encoding for the message digest
• Option -q hex:050000 means you want to calcualate the message digest from the hex input 050000 quickly
• Option -e d893 means an expected hash value of d893, expressed as hex
  

jacksum -a unknown:16 -q hex:050000 -E hex -e d893 

produces the following output:

Trying 13 algorithms with a width of 16 bits that are supported by Jacksum 3.0.0 ...

Trying 30 CRC algorithms with a width of 16 bits by testing against well known CRCs ...
crc:16,1021,FFFF,false,false,FFFF
    --> CRC-16/GENIBUS

Trying all CRC algorithms with a width of 16 bits by brute force (be patient!) ...
crc:16,1021,FFFF,false,false,FFFF
crc:16,37D2,FFFF,true,false,FFFF
crc:16,3E2D,0000,true,false,FFFF
crc:16,4175,FFFF,true,false,FFFF
crc:16,4A5B,FFFF,true,true,0000
crc:16,5A41,FFFF,true,false,FFFF
crc:16,5C63,FFFF,true,true,0000
crc:16,6287,FFFF,true,true,0000
crc:16,649C,0000,false,true,FFFF
crc:16,6D55,FFFF,true,true,0000
crc:16,75AC,FFFF,true,false,FFFF
crc:16,7D64,FFFF,false,false,FFFF
crc:16,81A6,FFFF,true,false,FFFF
crc:16,B9F9,FFFF,true,true,0000
crc:16,C3D6,FFFF,false,false,FFFF
crc:16,D436,0000,true,false,FFFF
crc:16,D6D2,0000,false,true,FFFF
crc:16,DA9C,FFFF,true,false,FFFF
crc:16,E03E,FFFF,false,false,FFFF
crc:16,F701,FFFF,true,false,FFFF


Jacksum: algorithms tested: 1048620
Jacksum: algorithms found: 21

Jacksum: elapsed time: 6 s, 460 ms

Means Jacksum has tested more than one million algorithms in about 7 seconds and it found 21 matching algorithms. Each of those returns the same CRC value. Test with more input/output sequences and/or longer input sequences in order to find the right algorithm. The most likely algorithm is printed with a name if it is a well known CRC. In this example it has been identified as the CRC-16/GENIBUS.

Once you have identified the correct algorithm, you can calculate your own input data using the CRC definitions that have been found:

jacksum -a crc:16,1021,FFFF,false,false,FFFF -E hex -q hex:050000
d893 3

The output "d893 3" means that 3 bytes have been read (050000) in order to produce the 16 bit (2 byte) hexadecimal value d893 using the algorithm as defined by -a.

Mission completed.

Cheers,
Johann

Be very careful with echo on Windows

Recently one of my users of Jacksum reported a strange behavior if echo is used on Microsoft Windows.

For those who don't know: Jacksum is a data integrity tool which can (not only) calculate and verify hash values. For more information go to https://jacksum.net

So the report (which was sent privately to me) was this:

jacksum -q "txt:Hello World!"
d0e47486bbf4c16acac26f8b653592973c1362909f90262877089f9c8a4536af

that is correct, but if Jacksum reads from stdin and echo on Windows is used, it returns a completely different hash:

echo "Hello World!" | jacksum -
61ab266cecda9b9885aedbbb5b3d9914738cec74930301ec7b68312b6b436b8b <stdin>

Is Jacksum right?

Yes, Jacksum is right. The problem is with the echo command on Windows!

I started Jacksum with "--verbose summary" which gives us additional information:

echo "Hello World!" | jacksum --verbose summary -
61ab266cecda9b9885aedbbb5b3d9914738cec74930301ec7b68312b6b436b8b <stdin>

Jacksum: files read successfully: 1
Jacksum: files read with errors: 0
Jacksum: total bytes read: 17
Jacksum: total bytes read (human readable): 17 bytes

Jacksum: elapsed time: 139 ms

As you can see, Jacksum have read 17 bytes from stdin (the 3rd line of the summary). "Hello World!" (without quotes) are just 12 characters, what are the other 5 characters?

I pasted the output of echo to a file ...

echo "Hello World!" > hello.txt

and loaded the file in a hex editor (https://hexed.it will work perfectly well for that task)

As you can see, the Windows echo

- does not only transfers the Hello World! to the pipe (Hello World! => 12 bytes), but also ...
- it appends the Windows carriage return, line feed chars (0x0D, 0x0A => 2 bytes) which is expected on Windows
- it doesn't strip the quotes (2x" => 2 bytes) - which is probably not known to everybody, and
- ATTENTION!: it doesn't strip the blank that is between the 2nd quote (") and the redirection sign (>) => 1 byte

That means a 

echo "Hello World!"        > hello2.txt

would even add more blanks to the output.

And even if we would do this:

echo Hello World!> hello3.txt

the CR LF will always be there, because there is no -n option (which GNU/Linux and macOS have for example).

So the workaround on Microsoft Windows 10 we found is this:

echo | set /p="Hello World!" | jacksum -
d0e47486bbf4c16acac26f8b653592973c1362909f90262877089f9c8a4536af <stdin>

which produces the expected hash.

Mission completed.

BTW, on GNU/Linux and macOS you can still use echo as expected, but you need to use single quotes rather than double quotes, because the ! has a special meaning on many *nix-shells:

$ echo -n 'Hello World!' | jacksum -
d0e47486bbf4c16acac26f8b653592973c1362909f90262877089f9c8a4536af <stdin>

Cheers,
Johann

Jacksum 3 is on the web!

I am pleased to announce that Jacksum 3 has been released!

Jacksum 3 is a major release. I call it the "good things will take time" release ;-)

See also https://jacksum.net

Detailed release notes at https://jacksum.net/en/release-notes.html

Here is an eagle view comparison chart with Jacksum 1.7.0. There was never an official Jacksum 2 release and I lifted the version number immediately to 3 due to the feature richness of Jacksum 3.

 

	Jacksum 1.7.0	Jacksum 3.0.0
Algorithm support
Supported Algorithms (including different lengths)	58	470
Default algorithm	SHA-1	SHA-3-256
Full SHA-3 family support	no	yes
Full SHA-2 family support	no*1	yes
Full BLAKE-family support	no	almost*2
Full FNV-familiy support*3	no	yes
Customizable CRCs	yes	yes
Modern non-US modern national standards*4	no	yes
Older non-US national standards*5	yes	yes
All algos from round three of the NIST SHA-3 competition*6	no	yes
proposals from the NIST crypto workshops*7	no	yes
eXtendable Output Functions (XOF) as cryptographic hash functions*8	no	yes
Multi core/processor support
multiple algorithms calculation	yes*9	yes
multiple algorithms calculation simultanously	no	yes
parallel hash calculation for files	no	yes
multi processor support for checking files	no	yes
Other highlights
Finding an algorithm/CRC by knowing both data and hash	no	yes
Select algorithms that match a bit width or a search string	no	yes
producing the output in a format you want	yes	yes
checking an output produced by a foreign software using parser definitions	no	yes
License	GPLv2+	GPLv3+

 

*1 SHA-512/224, SHA-512/256 were missing, because those algos were introduced in March 2012.

*2 BLAKE-[224,384,256,512], BLAKE2s-[8..256/8], BLAKE2b-[8..512/8], and BLAKE3 are supported; BLAKE2sp and BLAKE2bp are not yet supported

*3 including FNV-0, FNV-1, and FNV-1a for all bit lengths [32,64,128,256,512,1024]

*4 Streebog-[256,512] (Russia, GOST R 34.11-2012); SM3 (China); Kupyna[256,384,512] (Ukraine, DSTU 7564:2014); LSH-256-[224,256], LSH-512-[224,256,384,512] (South Korea, KS X 3262)

*5 GOST, GOST Crypto-Pro (Russia, GOST R 34.11-94); HAS-160 (KISA, South Korea) 

 *6 all five candidates from round 3 the NIST SHA-3 competition were BLAKE, Groestl, JH, Keccak, and  Skein

 *7 before the SHA-3 competition they were proposals from the NIST crypto workshops called FORK-256, DHA-256, and VSH-1024

*8 SHAKE128, SHAKE256, KangarooTwelve, MarsupilamiFourteen

*9 while a file is read only once if multiple algorithms have been selected, the actual hash calculation occurred sequentially and not in parallel

 

macOS: To use the java command-line tool you need to install a JDK - are you kidding me?

Recently I detected an error message on macOS saying that it seems to be required to install a JDK in order to just use the java command-line tool. A JRE is not enough? Really? Are you kidding me?

The screenshot below shows the error message in German: "Um das java-Befehlszeilenprogramm nutzen zu können, musst du ein Java-Entwicklerpaket installieren." In English it means: "To use the java command-line tool you need to install a JDK".

Since I am a developer, I always installed the JDK on my Mac and I detected that phenomenon very late. Actually the web is full of those traces - however, without a suitable solution in my opinion. Well, I simply don't want to tell my users to install a JDK if a simple JRE is enough. Any existing JRE on the system should do the job in my humble opinion.

Here we go, here is my little bash launcher that tries its best to launch java even if you have installed a JRE only on your Mac:

#!/bin/bash

if [[ ! -z $JAVA_HOME ]]; then

    JEXEC=$JAVA_HOME/bin/java

else

    LIBEXEC=$(/usr/libexec/java_home 2> /dev/null | head -1)

    # is there a JDK?

    if [[ ! -z $LIBEXEC ]]; then

        JEXEC="$LIBEXEC/bin/java"

    else

       # is there a JRE?

       JRE=/Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/java

       if [[ -f "$JRE" ]]; then

           JEXEC="$JRE"

       else

           JEXEC=java

       fi

    fi

fi

"$JEXEC" "$@"

The script checks for the JAVA_HOME environment variable and if that is not set, it checks for any registered JDKs by calling /usr/libexec/java_home and if that didn't return anything, it simply uses the JRE that could be available at a well known path on macOS (tested with both Java 8u131 and Java 9-ea) and if that fails as well, it uses java and if even that fails it means you really don't have any Java installed and you should get the error message above again.

I use the launcher above already as part of the Jacksum macOS Finder integration. See also
http://jacksum.net/de/tutorials/integration_jacksum_osx_finder.html

Feel free to use the launcher script for your Java app as well if it meets your needs.

A logo for Jacksum

Jacksum is a free cross platform checksum utility. It exists since July 2002. Time to create a logo for it.

Considerations:

Jacksum is entirely written in Java, it runs on Apple's OS X, Microsoft Windows, GNU/Linux, and any other operating system that has a Java Runtime Environment. So Jacksum is really cross platform, without the need for the user to recompile it. The purpose of Jacksum is to compute and verify checksums, mainly to check whether a data transfer was successful.

Actually "Jacksum" is a synthetic word made of JAva and ChecKSUM and I wanted to create a logo that reflects both the J as in Java (cross platform feature) and the check as in checksum. Checkboxes usually have a rectangle shape and the circle should reflect the comprehensiveness of the sum of all Jacksum's features. Black as in Jack and green as in successful check.

Furthermore the logo should be recognized even it has been resized to 16x16 pixels. That is important, because Jacksum also supports the File Browser Integration. So it was clear to make a very simple logo, but not simpler.

The logo:

Here it is:

And the icon resized to 16x16 pixels, integrated in the Windows Explorer (menu is localized in German):

And on the website called jacksum.net it will look like this:

Stay tuned, there will be more announcements with respect to Jacksum in the near future.