Skip to Content.
Sympa Menu

sm-discuss - Re: [SM-Discuss] [RFC] Faster checksums with b2sum

sm-discuss AT lists.ibiblio.org

Subject: Public SourceMage Discussion List

List archive

Chronological Thread  
  • From: Pavel Vinogradov <vin.public AT gmail.com>
  • To: sm-discuss AT lists.ibiblio.org
  • Subject: Re: [SM-Discuss] [RFC] Faster checksums with b2sum
  • Date: Sat, 15 Jul 2017 20:34:53 -0400

My pending coreutils patch to sorcery (w/o b2sum though) is attached below. Hope that would help.

On Sat, Jul 15, 2017 at 03:56:04PM -0300, Ismael Luceno wrote:
b2sum is a tool that computes 512-bit BLAKE2b digests of files, a much faster and equally secure alternative to sha512sum.

It's included in coreutils (although for some reason it wasn't installed for me). And I've put together an alternative implementation [0], based on the BLAKE2 reference implementation, with a few modifications; it's small, by far the smallest *sum implementation, being just about 15 KB. It doesn't have the checking mode provided by the other tools though; but we don't need that.

I'm going to implement support for it in sorcery, and I would like to make it the default (depending on either implementation).

Here's a bechmark running on tmpfs, on a Core i7-2670QM:

$ stat texlive-20170524-texmf.tar.xz | grep Size Size: 2398259780 Blocks: 4684104 IO Block: 4096 regular file

$ >/dev/null time sha512sum texlive-20170524-texmf.tar.xz 0m40.07s real 0m38.08s user 0m01.98s system

$ >/dev/null time b2sum texlive-20170524-texmf.tar.xz 0m21.83s real 0m20.26s user 0m01.56s system

$ >/dev/null time sha1sum texlive-20170524-texmf.tar.xz 0m27.83s real 0m26.25s user 0m01.54s system

$ >/dev/null time md5sum texlive-20170524-texmf.tar.xz 0m18.89s real 0m17.32s user 0m01.50s system


[0] http://git.iodev.co.uk/ismael/smgl/b2sum.git/ _______________________________________________ SM-Discuss mailing list SM-Discuss AT lists.ibiblio.org https://lists.ibiblio.org/mailman/listinfo/sm-discuss


-- Sincerely, Pavel Vinogradov
From 4e1c229aa9ed4b7fc457f8ac56a36a4334cb010a Mon Sep 17 00:00:00 2001
From: Pavel Vinogradov <public AT sourcemage.org>
Date: Thu, 2 Mar 2017 22:25:06 -0500
Subject: [PATCH] switched to coreutils' *sum in sorcery, libgpg and libunpack

---
ChangeLog | 6 ++++++
etc/sorcery/config | 1 +
usr/sbin/sorcery | 14 +++-----------
var/lib/sorcery/modules/libgpg | 14 +-------------
var/lib/sorcery/modules/libunpack | 21 ++++++++-------------
5 files changed, 19 insertions(+), 37 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index bf6b7c80..60869055 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,9 @@
+2017-03-02 Pavel Vinogradov <public AT sourcemage.org>
+ * etc/sorcery/config: added AVAILABLE_HASHES
+ * sorcery: adjusted to use AVAILABLE_HASHES in vrf_select_hashes()
+ * libgpg, libunpack: switched to coreutils' *sum for computing
+ hashsums, removed gpg_get_hashes()
+
2016-12-17 Jaka Kranjc <lynxlynxlynx AT sourcemage.org>
* libmisc: use mime type detection in show_file
sanitize constructed variable name in debug
diff --git a/etc/sorcery/config b/etc/sorcery/config
index c5557502..e121eefc 100755
--- a/etc/sorcery/config
+++ b/etc/sorcery/config
@@ -107,6 +107,7 @@
SM_CONFIG_OPTION_CACHE=${SM_CONFIG_OPTION_CACHE:-/etc/sorcery/local/config_optio
GPG_VERIFY_SORCERY=${GPG_VERIFY_SORCERY:-no}
GPG_VERIFY_GRIMOIRE=${GPG_VERIFY_GRIMOIRE:-no}

+ AVAILABLE_HASHES=${AVAILABLE_HASHES:-"md5 sha1 ripemd160 sha256 sha384
sha512 sha224"}
VERIFY_SPELL_LEVELS=${VERIFY_SPELL_LEVELS:-"WORKS_FOR_ME UPSTREAM_HASH
UPSTREAM_KEY ESTABLISHED_UPSTREAM_KEY VERIFIED_UPSTREAM_HASH
VERIFIED_UPSTREAM_KEY ID_CHECK_UPSTREAM_KEY"}
DEFAULT_SPELL_VRF_LEVEL=${DEFAULT_SPELL_VRF_LEVEL:-"WORKS_FOR_ME"}
VRF_ALLOWED_LEVELS=${VRF_ALLOWED_LEVELS:-""}
diff --git a/usr/sbin/sorcery b/usr/sbin/sorcery
index 3a5a1688..c9192fbd 100755
--- a/usr/sbin/sorcery
+++ b/usr/sbin/sorcery
@@ -945,18 +945,11 @@ function gpg_download_menu() {
function vrf_select_hashes() {
local DIALOG_TITLE="Select the source hashes you would like to use"
local DIALOG_HELP="Select the source hashes you would like to consider
valid hashes"
- local DEFAULT_LIST=$(gpg_get_hashes)
local rc=0
local CHECKS=""
local check=""
local hash=""
local DIALOG_LIST=""
- if [ -z "$DEFAULT_LIST" ] ; then
- eval $DIALOG '--title "ERROR" \
- --msgbox \
- "ERROR: gpg isn't installed or isn't in your PATH. Please
cast gnupg." 20 20' &&
- return 1
- else
# move old values (named with GPG) to new ones
# this can be removed after 11/26/05 (afk -- 10/26/05)
if [[ -n "$GPG_ALLOWED_HASHES" ]] ; then
@@ -973,12 +966,12 @@ function vrf_select_hashes() {
DIALOG_LIST=""
if [[ -z "$VRF_ALLOWED_HASHES" ]]
then
- for hash in $DEFAULT_LIST
+ for hash in ${AVAILABLE_HASHES}
do
DIALOG_LIST="${DIALOG_LIST}${hash} '${hash} was selected' on "
done
else
- for hash in $DEFAULT_LIST
+ for hash in ${AVAILABLE_HASHES}
do
if list_find "$VRF_ALLOWED_HASHES" "${hash}:on"
then
@@ -1009,7 +1002,7 @@ function vrf_select_hashes() {
else
VRF_ALLOW_NEW_HASHES=off
fi
- for hash in ${DEFAULT_LIST}
+ for hash in ${AVAILABLE_HASHES}
do
if list_find "${CHECKS}" $hash
then
@@ -1021,7 +1014,6 @@ function vrf_select_hashes() {
modify_local_config VRF_ALLOWED_HASHES "${VRF_ALLOWED_HASHES}"
modify_local_config VRF_ALLOW_NEW_HASHES "${VRF_ALLOW_NEW_HASHES}"
fi
- fi
return 0
}

diff --git a/var/lib/sorcery/modules/libgpg b/var/lib/sorcery/modules/libgpg
index 53e7732f..2801a527 100755
--- a/var/lib/sorcery/modules/libgpg
+++ b/var/lib/sorcery/modules/libgpg
@@ -341,19 +341,7 @@ function gpg_user_query() {
function gpg_hashsum() {
local algorithm=$1
local file=$2
- LC_ALL=C gpg --print-md "$algorithm" "$file"| tr -d '\n ' |cut -f2 -d:|
- tr 'A-F' 'a-f'|tr -d '\n'
- echo " $file"
-}
-
-#---------------------------------------------------------------------
-## @stdout All the hash algorithms supported by gpg, algorithms printed in
-## @stdout lower case.
-##
-## This assumes the caller has already verified that gpg is installed.
-#---------------------------------------------------------------------
-function gpg_get_hashes() {
- LC_ALL=C gpg --version 2> /dev/null | awk -F: '/^Hash/ { gsub(","," ",$2);
print tolower($2); }'
+ eval "${algorithm}sum ${file}"
}

#---------------------------------------------------------------------
diff --git a/var/lib/sorcery/modules/libunpack
b/var/lib/sorcery/modules/libunpack
index 7e19f15a..eadfcce8 100755
--- a/var/lib/sorcery/modules/libunpack
+++ b/var/lib/sorcery/modules/libunpack
@@ -451,19 +451,14 @@ function unpack_hash() {
local HASH
if [ "$MD5SUM_DL" != "off" ]; then

- if [[ "$ALGORITHM" == md5 ]] || [[ "$ALGORITHM" == sha1 ]] ; then
- unpack_spell_required coreutils "${ALGORITHM}sum" "$ALGORITHM" ||
- return "$?"
- HASH="$(${ALGORITHM}sum "$FILENAME" | cut -d' ' -f1)"
- else
- unpack_spell_required gnupg gpg "$ALGORITHM" || return "$?"
- if list_find "$(gpg_get_hashes)" $ALGORITHM; then
- HASH="$(gpg_hashsum "${ALGORITHM}" "$FILENAME" | cut -d' ' -f1)"
- else
- message "${PROBLEM_COLOR}Algorithm $ALGORITHM is not"\
- "known!${DEFAULT_COLOR}"
- return 200
- fi
+ unpack_spell_required coreutils "${ALGORITHM}sum" "$ALGORITHM" || return
"$?"
+ HASH="$(${ALGORITHM}sum "$FILENAME" | cut -d' ' -f1)"
+ if list_find "${AVAILABLE_HASHES}" "${ALGORITHM}"; then
+ HASH="$(gpg_hashsum "${ALGORITHM}" "$FILENAME" | cut -d' ' -f1)"
+ else
+ message "${PROBLEM_COLOR}Algorithm $ALGORITHM is not"\
+ "known!${DEFAULT_COLOR}"
+ return 200
fi
local rc=$?

--
2.12.0

Attachment: signature.asc
Description: PGP signature




Archive powered by MHonArc 2.6.24.

Top of Page