Sunday, May 28, 2017

Solaris' pargs, penv, pfiles, pmap, pstack, and pwdx on macOS

Three weeks ago I posted Solaris' pargs, penv, pfiles, pmap, and pstack on GNU/Linux and since I am also a Mac user, I thought it could be a good idea to have those commands also on macOS. Here we go ...

function pargs()  { L=$(ps ww $1 | tail -1); echo ${L:27}; }
function penv()   { L=$(pargs $1); C=${#L}; L=$(ps wwe $1 | tail -1); L=${L:27}; echo ${L:$C} | tr ' ' '\n'; }
function pfiles() { lsof -p $1; }
function pmap()   { vmmap $1; }
function pstack() { echo "thread backtrace all" | lldb -p $1; }
function pwdx()   { L=$(lsof -a  -d cwd -p $1 | tail -1); echo /${L#*/}; }

Since there is no access to a proc file system on macOS (at least not by default), both pargs and penv call the ps command and pmap calls vmmap. Furthermore pwdx calls lsof with the current working directory descriptor request in order to get the required info. Since "ps wwe" returns not only the environment variables for the given process but also the program arguments on macOS, we need to strip the program arguments from the output. This has been done by calling pargs, determining the length of that output and cutting that length from the string again before we pass it to the tr command that gives us an environment variable for each line. For blog purposes I have shortened the variable names, L stands for line and C for count.

References:
http://wiki.bash-hackers.org/syntax/pe
http://yongsun.me/2009/01/tips-the-equivalents-of-ldd1-and-pmap1-on-mac-os-x/

Sunday, May 7, 2017

Solaris' pargs, penv, pfiles, pmap, and pstack on GNU/Linux

I really like the p-commands on Sun/Oracle Solaris and I miss those on GNU/Linux. Therefore I have gathered/created rudimental equivalent one-liners that work on a PID.

I have added the following functions to my ~/.bash_aliases file that is being sourced by my ~/.bashrc. pargs, penv, and pmap are gathering the proc file system, while pfiles and pstack are calling lsof resp. gdb.

function pargs() { cat /proc/$1/cmdline | tr '\0' ' ' | sed 's/ $/\n/g'; }
function penv() { cat /proc/$1/environ | tr '\0' '\n'; }
function pfiles() { lsof -p $1; }

function pmap() { cat /proc/$1/maps; }
function pstack() { sudo gdb --pid=$1 --batch -ex "thread apply all bt"; }


Note that due to a kernel hardening (ptrace protection) on Ubuntu 10.10 and later you need to call gdb under the control of sudo, or alternatively modify the ptrace_scope property. See also https://wiki.ubuntu.com/SecurityTeam/Roadmap/KernelHardening#ptrace_Protection and https://askubuntu.com/questions/41629/after-upgrade-gdb-wont-attach-to-process


Friday, February 10, 2017

How the NumericalChameleon Installer obtains the latest JRE for Windows

Today I will explain how the NumericalChameleon installer obtains the latest Java Runtime Environment (JRE) for Microsoft Windows.

The NumericalChameleon (http://numericalchameleon.net) is written in Java and it relies on a JRE that is installed on your system. The NumericalChameleon installer checks at first whether a JRE is installed. If there is one, everything is fine and the installer will continue with a normal installation. If there is none, the installer downloads the latest JRE offline installer from Oracle and launches it. The offline installer installs the JRE on your system and once it is installed, it will give back the control to the NumericalChameleon installer that continues with the installation until the NumericalChameleon software package is installed as well.

Below you find a screenshot of the NumericalChameleon installer, localized in German, running on Windows 10 x64 while downloading the latest JRE offline installer:



Since the actual locations of the JRE offline exe installers are different for each Java version, and those locations are both unpredictable and volatile, it is important for the NumericalChameleon installer to rely on well known static URIs, because the installer binary cannot be changed/patched anymore once it is deployed on the web.

Those well known URIs are redirects in the .htaccess file on my Apache server actually, and the redirects are being updated every day. I create those redirects by parsing the website that has the locations of the Windows JRE offline installers.

$ cat bin/update_htaccess
#!/bin/bash
HTACCESS="$HOME/numericalchameleon/.htaccess"
STATIC="$HOME/numericalchameleon/.htaccess.static"

cat $STATIC > $HTACCESS

# the locations of the JRE offline installers
URL="https://www.java.com/en/download/manual.jsp"
CONTENT=$(curl -s "$URL")

IDENTIFIER="Windows Offline</a>"
if [[ "$CONTENT" =~ BundleId=([^\"]+)\"\>[[:space:]]*"$IDENTIFIER" ]]; then
    printf "Redirect /get_java_win32bit http://javadl.oracle.com/webapps/download/AutoDL?BundleId=%s\n" ${BASH_REMATCH[1]} >> $HTACCESS
fi

IDENTIFIER="Windows Offline (64-bit)</a>"
if [[ "$CONTENT" =~ BundleId=([^\"]+)\"\>[[:space:]]*"$IDENTIFIER" ]]; then
    printf "Redirect /get_java_win64bit http://javadl.oracle.com/webapps/download/AutoDL?BundleId=%s\n" ${BASH_REMATCH[1]} >> $HTACCESS
fi

The .htaccess.static file just contains entries that never change. The update_htaccess script runs by cron periodically on the server in order to have entries that are up to date.

$ crontab -l | tail -1

9 3 * * * cd bin; ./update_htaccess

After the cron was running, updated redirects can be found in the .htaccess file. In the example below the redirects are from Java 8u121:

$ tail -2 .htaccess

Redirect /get_java_win32bit http://javadl.oracle.com/webapps/download/AutoDL?BundleId=218831_e9e7ea248e2c4826b92b3f075a80e441
Redirect /get_java_win64bit http://javadl.oracle.com/webapps/download/AutoDL?BundleId=218833_e9e7ea248e2c4826b92b3f075a80e441

The installer can now rely on the fixed addresses below, dependent whether a 32 bit or a 64 bit system has been found:
Now you know how the NumericalChameleon installer gets the latest JRE on Windows.

Hint: if the approach above should ever fail and the redirects are not being created, the current installer will fail with a 404. In that case you can simply install the JRE manually before you start the NumericalChameleon installer. In that case the installer won't even go to the internet, because the condition is met already that a JRE has to be installed.

Monday, June 6, 2016

The NumericalChameleon 2.0.0 is on the web

I am pleased to announce that the NumericalChameleon 2.0.0 is on the web:

http://NumericalChameleon.net


The NumericalChameleon is free, open source, cross platform and comprehensive software in order to convert units with a precision of up to 1000 decimal places. It supports more than 5200 units in 93 categories, including not only all important physical units, but also useful units in non-standard categories like exchange rates, time zones, spoken numbers (literally and by audio), roman numerals, geographic coordinates, radixes, fractions, checksums, bits&bytes, screen resolutions, colorcodes, unicodes, international dial codes, calendar and holiday calculations and many more.


Release notes for the version 2.0.0 are at

http://numericalchameleon.net/en/release-notes.html

Kind Regards,
Johann




Sunday, February 14, 2016

A logo for Jacksum

Jacksum is a free cross platform checksum utility. It exists since July 2002. Time to create a logo for it.

Considerations:

Jacksum is entirely written in Java, it runs on Apple's OS X, Microsoft Windows, GNU/Linux, and any other operating system that has a Java Runtime Environment. So Jacksum is really cross platform, without the need for the user to recompile it. The purpose of Jacksum is to compute and verify checksums, mainly to check whether a data transfer was successful.

Actually "Jacksum" is a synthetic word made of JAva and ChecKSUM and I wanted to create a logo that reflects both the J as in Java (cross platform feature) and the check as in checksum. Checkboxes usually have a rectangle shape and the circle should reflect the comprehensiveness of the sum of all Jacksum's features. Black as in Jack and green as in successful check.

Furthermore the logo should be recognized even it has been resized to 16x16 pixels. That is important, because Jacksum also supports the File Browser Integration. So it was clear to make a very simple logo, but not simpler.

The logo:

Here it is:


And the icon resized to 16x16 pixels, integrated in the Windows Explorer (menu is localized in German):




And on the website called jacksum.net it will look like this:

Stay tuned, there will be more announcements with respect to Jacksum in the near future.

Saturday, June 27, 2015

How to protect your privacy in the text editor called Atom 1.0

Yesterday Atom 1.0 was released and the way to protect your privacy has slightly changed compared to earlier versions. So this is actually an update to my blog that I wrote nine months ago. See also http://loefflmann.blogspot.de/2014/09/how-to-protect-your-privacy-in-text.html

Screenshots have been taken on a Mac, but instructions apply to all platforms that are supported by Atom 1.0 (OS X 10.8 or later, Windows 7 & 8, RedHat Linux, and Ubuntu Linux).


Tuesday, March 3, 2015

Finding the OS X version and build information in an Install*OS X*.app

The problem

If you have downloaded [Mac] OS X from the the Apple App Store, the installer is stored to the folder called /Applications and the application name is equal for all update releases of a particular OS X version. For example, the Yosemite installer can be found in "/Applications/Install OS X Yosemite.app" and if you open it, it provides information that it is going to install OS X 10.10. Unfortunately it doesn't tell you whether it is going to install OS X 10.10, OS X 10.10.1 or OS X 10.10.2. In other words, the GUI is not suitable to determine the exact OS X version that the installer is loaded with. Both the update number and the build number are not visible. Screenshot below shows the installer of 10.10.2:



You can find the OS X version and build information of your running OS X, see also https://support.apple.com/en-us/HT201260 - IMHO it should be possible to get those details before an actual installation as well.

So the question is how to find the complete OS X product version and build version in an "Install*OS X*.app package?

Eleven code lines to success

The command line tool called sw_vers can print out both the product version and the build version of an installed product. Example from Mavericks:

$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.9.5
BuildVersion:   13F34

The idea is to get access to the sw_vers respectively the version information that is stored in the installer image.

At first we need to mount the InstallESD.img that is stored in /Applications/Install*OS X* Installer.app/Contents/SharedSupport. Note that in the example below, the environment variable APPNAME has to be set to the name of the installer app, in case of Yosemite it is "Install OS X Yosemite.app". Also set DEBUG to /dev/stdout in order to see the output of the mount actions. The $$ will be replaced by the pid of the shell which will be our unique number for the session in order to avoid collisions with other potential volumes.

hdiutil attach "/Applications/$APPNAME/Contents/SharedSupport/InstallESD.dmg" -noverify -nobrowse -mountpoint /Volumes/InstallESD.$$ > $DEBUG

Once that is done, we can mount the BaseSystem.dmg

hdiutil attach "/Volumes/InstallESD.$$/BaseSystem.dmg" -noverify -nobrowse -mountpoint /Volumes/BaseSystem.$$ > $DEBUG

Now we have access to sw_vers. However, calling the /Volume/BaseSystem.<pid>/usr/bin/sw_vers won't give us the results we expect, though. Actually it still gathers version information from the host system and not from the mounted volume. That is because the version number is read from the absolute path called /System/Library/CoreServices/SystemVersion.plist. Below is an excerpt from the SystemVersion.plist on the host system running Mavericks.

<key>ProductBuildVersion</key>
<string>13F34</string>
<key>ProductCopyright</key>
<string>1983-2014 Apple Inc.</string>
<key>ProductName</key>
<string>Mac OS X</string>
<key>ProductUserVisibleVersion</key>
<string>10.9.5</string>
<key>ProductVersion</key>
<string>10.9.5</string>

If we use a changed root environment in order to let read sw_vers the correct file sounds like an easy solution, but tests have shown that it didn't work with the latest Yosemite installer. Well, why not simply mimic the sw_vers functionality by reading the xml file called /Volumes/BaseSystem.$$/System/Library/CoreServices/SystemVersion.plist using bash's regular expression build-in features? That approach is even faster than initiating a changed root environment, root permissions are not required and it is more comfortable than to print out the entire xml:

XMLCONTENT=$(<"/Volumes/BaseSystem.$$/System/Library/CoreServices/SystemVersion.plist")
if [[ "$XMLCONTENT" =~ \<key\>ProductVersion\</key\>[[:space:]]*\<string\>([0-9\.]+)\</string\> ]]; then
    printf "ProductVersion: %s\n" ${BASH_REMATCH[1]}
fi

XMLCONTENT stores the content of the plist file and the first expression in the brackets resp. ${BASH_REMATCH[1] stores the complete product version of OS X.

In order to extract the build version information, we can enter:

if [[ "$XMLCONTENT" =~ \<key\>ProductBuildVersion\</key\>[[:space:]]*\<string\>([0-9A-Z]+)\</string\> ]]; then
    printf "BuildVersion:   %s\n" ${BASH_REMATCH[1]}
fi

If the desired information has been gathered, we just need to unmount the two volumes again in reverse order for cleanup purposes:

hdiutil detach "/Volumes/BaseSystem.$$" > $DEBUG
hdiutil detach "/Volumes/InstallESD.$$" > $DEBUG

The solution

Put the eleven code lines above to a small script called osxapp_vers, set execute permissions to it, set APPNAME to an appropriate value and set DEBUG to /dev/null ...

chmod +x ./osxapp_vers
export APPNAME="Install OS X Yosemite.app"
export DEBUG="/dev/null"

... and you can determine the Installer OS X app's product version and build version just by calling:

./osxapp_vers
ProductVersion: 10.10.2
BuildVersion:   14C109

Note: I have tested the solution above on all my [Mac] OS X downloads from the Apple App Store and I can confirm that this article applies at least to Mac OS X 10.7.5 (Lion) until OS X 10.10.2 (Yosemite).

Mission completed ;-)

Update on  March 4, 2015

The build version is not a hex value and therefore the regular expression has to be ([0-9A-Z]) instead of ([0-9A-F]). I have fixed the bug in the code above.


Monday, September 15, 2014

How to protect your privacy in the text editor called Atom

The problem

Atom is a very nice text editor for OS X, but it comes preinstalled with a package called Metrics. By default that package reports usage information to Google Analytics including a unique identifier that is generated by computing the SHA-1 hash of the machine's MAC address. I am sure there are people who don't like that. For example one could get the Mac's mac address by using rainbow tables.
Well, the information that is being reported to Google Analytics is well documented on the Atom's FAQ and github, but the instructions how to disable it are pretty spartanic IMHO: "If you do not want this information reported, disable this package from the Metrics section of the Settings view".


The solution

This graphical note should help you to disable the Metrics package in Atom in order to prevent sending usage information to Google Analytics.





Friday, January 31, 2014

Your TV does not recognize your external USB hard disk drive anymore and how to fix it

The problem

Recently my Samsung TV didn't recognize my external USB hard disk drive anymore. That was a really bad feeling since I recorded some interesting videos on it actually. The TV suggested to format the disk when I was trying to start the recording. Since I have set the user language of my TV to German, I got the error message called "Angeschl. USB evtl. nicht für Aufnahmefunktion geeignet! Mögchten Sie das Gerät formatieren?"


Well, formatting the disk was an option at all, because I didn't want to loose those videos.

The solution

I knew that the disk was formatted by the TV with a non-standard file system, and I was wondering whether I could fix the issue by using Linux. I did and I have created the following instructions, because the issue happened to me the 2nd time already and if you read this, you have a similar problem with your TV/hard disk drive right now, probably.  Here we go, this is my recipe ...

1. Start Linux

If you don't have Linux installed on your computer, download a live CD/DVD from the web, burn it and boot from it. Alternatively setup Linux on an USB pendrive and boot from there. In my case, I was using Ubuntu 12.04 since that is one of my preferred Linux systems, but any modern GNU/Linux system should do the trick as well. You can download Ubuntu from ubuntu.com. Nonetheless, please take into account that the following instructions have been tested on Ubuntu 12.04 only and dependent on your actual problem/environment, the instructions below have to be modified probably in order to meet your actual needs.

2. Open a terminal

Shortcut on Ubuntu: Ctrl+Alt+T

3. Disable the automounter

In worst case, the automounter could try to mount a corrupt file system and it could crash the Linux kernel. Therefore at first please disable the automounter by running gsettings under the normal user:

$ gsettings set org.gnome.desktop.media-handling automount 'false'
$ gsettings set org.gnome.desktop.media-handling automount-open 'false'
 


4. Attach the USB disk to your computer

If your computer crashes, you didn't follow my instructions, go to step 2 ;-)

 

5. Identify both the device and the filesystem

I use gparted since it is the most comfortable graphical way to get an overview on the disks that have been attached. gparted is part of the Live CD, but it is not installed by default. If you don't have it, install it by typing

$ sudo apt-get install gparted

Start gparted by typing

$ sudo gparted

In my case, the external disk is /dev/sdb and it has one partition called /dev/sdb1. It has been formatted by the televison set with the xfs filesystem:




Close gparted again.

6. Install the xfs packages if not done yet

$ sudo apt-get install xfsprogs
$ sudo apt-get install xfsdump


7. Perform a xfs_check

$ sudo xfs_check /dev/sdb1
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_check.  If you are unable to mount the filesystem, then use
the xfs_repair -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.


Since we cannot mount the file system, we have to execute the xfs_repair program.

8. Execute xfs_repair

$ sudo xfs_repair -L /dev/sdb1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
destroyed because the -L option was used.
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 3
        - agno = 2
        - agno = 1
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done


9. Perform an xfs_check again

$ sudo xfs_check /dev/sdb1
$

No errors anymore :-)

10. Enable the automounter again

$ gsettings set org.gnome.desktop.media-handling automount 'true'
$ gsettings set org.gnome.desktop.media-handling automount-open 'true'


11. Disconnect the disk by unplugging the USB-disk from the computer

The file system wasn't mounted yet, so it is ok to disconnect the disk by unplugging the cable.

12. Connect the disk by plugging the USB-disk to the computer

The automounter has been enabled again, you should see the disk icon on your desktop.

13. Check whether files are there

$ ls /media/*-*/CONTENTS/*.srf | wc -l
41


Bingo. In my case the filenames of the videos are stored with suffix .srf, that could be different in your case. The wc command counts the lines that the ls command produces, in other words there are 41 videos on that partition.

14. Unmount the disk again

Since it has been mounted, unmount the disk from your computer by selecting the "savely remove" on the popup menu from the disk icon.

15. Disconnect the disk by unplugging the USB-disk from the computer

Make sure that you have unmounted the disk (see step 13) before unplugging the disk.

16. Connect the disk again to your TV

If all went well, your TV should recognize the USB hard disk drive again and you can continue watching your recorded videos :-)