[john-users] RFC: Hashkiller Rosetta Stone

[Royce Williams]

I'm working on a Hashkiller Rosetta stone (list of upload formats supported
by Hashkiller, and how to use those modes in hashcat, john, and MDXfind.

A draft is here:

https://gist.github.com/roycewilliams/28a9e940e7cd37268ceeac4962bda757

Any help/tips appreciated. I don't know the underlying algorithm of many
product-specific formats, so I'm almost certainly missing some obvious ones.

My future ambition is to expand this concept to be a Rosetta Stone for the
superset of all formats supported by any known product. Small steps first.
:)

Royce

[Royce Williams]

I'm working on a Hashkiller Rosetta Stone (a list of upload formats
supported by Hashkiller, and how to use those modes in hashcat, john, and
MDXfind.)

A draft is here:

https://gist.github.com/roycewilliams/28a9e940e7cd37268ceeac4962bda757

Any help/tips appreciated. I don't know the underlying algorithm of many
product-specific formats, so I'm almost certainly missing some obvious ones.

My future ambition is to expand this concept to be a Rosetta Stone for the
superset of all formats supported by any known product. Small steps first.
:)

Royce / tychotithonus

[Solar Designer]

Post by Royce Williams
I'm working on a Hashkiller Rosetta Stone (a list of upload formats
supported by Hashkiller, and how to use those modes in hashcat, john, and
MDXfind.)
https://gist.github.com/roycewilliams/28a9e940e7cd37268ceeac4962bda757
Any help/tips appreciated. I don't know the underlying algorithm of many
product-specific formats, so I'm almost certainly missing some obvious ones.

It'd help if you list example hashes, preferably all for a fixed
password like "password", so that you don't need to list the
corresponding different plaintext passwords as well.

JtR supports command-line dynamic formats now (and has been for a couple
of years, due to work by Jim), so most if not all of your "unsupported"s
are actually supported at least in this way.

Those command-line dynamics typically allow for a higher password
length, too. For example, when experimenting with Update 2 from
https://haveibeenpwned.com/Passwords I found that "--external=Repeats"
cracks plenty of passwords of lengths up to 109 (and I've just tested
that it cracks 110 too, but not 111 - as expected) with
"--format=dynamic='sha1($p)'", whereas "--format=raw-sha1" only goes up
to length 55 (also as expected).

Post by Royce Williams
My future ambition is to expand this concept to be a Rosetta Stone for the
superset of all formats supported by any known product. Small steps first.
:)

Cool. With JtR's command-line dynamics, its list of supported formats
is sort of "infinite", though. ;-) But I guess you'll list only those
actually seen in use somewhere.

BTW, I found that the command-line dynamics are much easier to use than
having to remember the old numeric dynamics.

Alexander

[Frank Dittrich]

Post by Solar Designer

Post by Royce Williams
I'm working on a Hashkiller Rosetta Stone (a list of upload formats
supported by Hashkiller, and how to use those modes in hashcat, john, and
MDXfind.)
https://gist.github.com/roycewilliams/28a9e940e7cd37268ceeac4962bda757
Any help/tips appreciated. I don't know the underlying algorithm of many
product-specific formats, so I'm almost certainly missing some obvious ones.

It'd help if you list example hashes, preferably all for a fixed
password like "password", so that you don't need to list the
corresponding different plaintext passwords as well.

It would also help if you would use another syntax for those "generic"
algorithms.

Currenntly, it is unclear whether MD5-MD5SALTMD5PASS means
md5(md5(salt).md5(pass)) or not.

Post by Solar Designer
JtR supports command-line dynamic formats now (and has been for a couple
of years, due to work by Jim), so most if not all of your "unsupported"s
are actually supported at least in this way.

Yes, you can combine several basic hash functions, use password and/or
sewveral salts, even constant expressions, etc.
Check doc/DYNAMIC*,especially doc/DYNAMIC_COMPILER_FORMATS.md.

Post by Solar Designer
Those command-line dynamics typically allow for a higher password
length, too. For example, when experimenting with Update 2 from
https://haveibeenpwned.com/Passwords I found that "--external=Repeats"
cracks plenty of passwords of lengths up to 109 (and I've just tested
that it cracks 110 too, but not 111 - as expected) with
"--format=dynamic='sha1($p)'", whereas "--format=raw-sha1" only goes up
to length 55 (also as expected).

But the dynamic_* formats are usually somewhat faster than the
dynamic='<expression>'.

Post by Solar Designer

Post by Royce Williams
My future ambition is to expand this concept to be a Rosetta Stone for the
superset of all formats supported by any known product. Small steps first.
:)

Cool. With JtR's command-line dynamics, its list of supported formats
is sort of "infinite", though. ;-)

You even get basic bash completion support for the dynamic-formats:

$ ./john --format=dynamic=[tab][tab]
lists these possible completions:

$ ./john --format=dynamic=
gost\( haval192_5\( md2\( sha1_64\( skein224\(
haval128_3\( haval224_3\( md4\( sha224\( skein256\(
haval128_4\( haval224_4\( md5\( sha256\( skein384\(
haval128_5\( haval224_5\( panama\( sha384\( skein512\(
haval160_3\( haval256_3\( ripemd128\( sha3_224\( tiger\(
haval160_4\( haval256_4\( ripemd160\( sha3_256\( whirlpool\(
haval160_5\( haval256_5\( ripemd256\( sha3_384\(
haval192_3\( keccak_256\( ripemd320\( sha3_512\(
haval192_4\( keccak_512\( sha1\( sha512\(


The \ is needed unless you quote the expression. I'll see whether I
manage to support bash completion even for --format='dynamic= or
--format=dynamic'= etc. Then I can drop the backslashes.


If you quote the expression, it could look like this:

$ ../run/john -test -form:dynamic='sha1(md5(md4($p.$s).$s2))'
Benchmarking: dynamic=sha1(md5(md4($p.$s).$s2)) [128/128 AVX 4x1]... DONE
Many salts: 5243K c/s real, 5243K c/s virtual
Only one salt: 4813K c/s real, 4813K c/s virtual

($s is the first salt, $s2 a second salt in that example)

Frank

[Royce Williams]

Post by Frank Dittrich

Post by Solar Designer

Post by Royce Williams
I'm working on a Hashkiller Rosetta Stone (a list of upload formats
supported by Hashkiller, and how to use those modes in hashcat, john, and
MDXfind.)
https://gist.github.com/roycewilliams/28a9e940e7cd37268ceeac4962bda757
Any help/tips appreciated. I don't know the underlying algorithm of many
product-specific formats, so I'm almost certainly missing some obvious ones.

It'd help if you list example hashes, preferably all for a fixed
password like "password", so that you don't need to list the
corresponding different plaintext passwords as well.

It would also help if you would use another syntax for those "generic"
algorithms.
Currenntly, it is unclear whether MD5-MD5SALTMD5PASS means
md5(md5(salt).md5(pass)) or not.

This isn't my syntax. Those are the labels used by MDXfind itself. I am new
to MDXfind, so I'm still learning about how to read these; they do indeed
require interpretation that is opaque to the uninitiated. That's part of
what I was hoping to accomplish with the table - caching what I learn for
the benefit of others.

Post by Frank Dittrich
JtR supports command-line dynamic formats now (and has been for a couple

Post by Solar Designer
of years, due to work by Jim), so most if not all of your "unsupported"s
are actually supported at least in this way.

Yes, you can combine several basic hash functions, use password and/or
sewveral salts, even constant expressions, etc.
Check doc/DYNAMIC*,especially doc/DYNAMIC_COMPILER_FORMATS.md.

Thanks for this - will do.

Post by Frank Dittrich
Those command-line dynamics typically allow for a higher password

Post by Solar Designer
length, too. For example, when experimenting with Update 2 from
https://haveibeenpwned.com/Passwords I found that "--external=Repeats"
cracks plenty of passwords of lengths up to 109 (and I've just tested
that it cracks 110 too, but not 111 - as expected) with
"--format=dynamic='sha1($p)'", whereas "--format=raw-sha1" only goes up
to length 55 (also as expected).

But the dynamic_* formats are usually somewhat faster than the
dynamic='<expression>'.

That makes sense.

Post by Frank Dittrich
My future ambition is to expand this concept to be a Rosetta Stone for the

Post by Solar Designer

Post by Royce Williams
superset of all formats supported by any known product. Small steps first.
:)

Cool. With JtR's command-line dynamics, its list of supported formats

Post by Solar Designer
is sort of "infinite", though. ;-)

$ ./john --format=dynamic=[tab][tab]
$ ./john --format=dynamic=
gost\( haval192_5\( md2\( sha1_64\( skein224\(
haval128_3\( haval224_3\( md4\( sha224\( skein256\(
haval128_4\( haval224_4\( md5\( sha256\( skein384\(
haval128_5\( haval224_5\( panama\( sha384\( skein512\(
haval160_3\( haval256_3\( ripemd128\( sha3_224\( tiger\(
haval160_4\( haval256_4\( ripemd160\( sha3_256\( whirlpool\(
haval160_5\( haval256_5\( ripemd256\( sha3_384\(
haval192_3\( keccak_256\( ripemd320\( sha3_512\(
haval192_4\( keccak_512\( sha1\( sha512\(
The \ is needed unless you quote the expression. I'll see whether I manage
to support bash completion even for --format='dynamic= or
--format=dynamic'= etc. Then I can drop the backslashes.
$ ../run/john -test -form:dynamic='sha1(md5(md4($p.$s).$s2))'
Benchmarking: dynamic=sha1(md5(md4($p.$s).$s2)) [128/128 AVX 4x1]... DONE
Many salts: 5243K c/s real, 5243K c/s virtual
Only one salt: 4813K c/s real, 4813K c/s virtual
($s is the first salt, $s2 a second salt in that example)

Very informative - thank you, Frank!

Royce

[Royce Williams]

(I apologize for the duplicate post; I'll treat this one as the "correct"
one)

Post by Solar Designer

Post by Royce Williams
I'm working on a Hashkiller Rosetta Stone (a list of upload formats
supported by Hashkiller, and how to use those modes in hashcat, john, and
MDXfind.)
https://gist.github.com/roycewilliams/28a9e940e7cd37268ceeac4962bda757
Any help/tips appreciated. I don't know the underlying algorithm of many
product-specific formats, so I'm almost certainly missing some obvious

ones.
It'd help if you list example hashes, preferably all for a fixed
password like "password", so that you don't need to list the
corresponding different plaintext passwords as well.

That's a really good idea. I will do this.

Post by Solar Designer
JtR supports command-line dynamic formats now (and has been for a couple
of years, due to work by Jim), so most if not all of your "unsupported"s
are actually supported at least in this way.

Cool. I will investigate this.

Post by Solar Designer
Those command-line dynamics typically allow for a higher password
length, too. For example, when experimenting with Update 2 from
https://haveibeenpwned.com/Passwords I found that "--external=Repeats"
cracks plenty of passwords of lengths up to 109 (and I've just tested
that it cracks 110 too, but not 111 - as expected) with
"--format=dynamic='sha1($p)'", whereas "--format=raw-sha1" only goes up
to length 55 (also as expected).

Indeed. For completeness, I'll try to make such differences explicit
somehow in the table.

Post by Solar Designer

Post by Royce Williams
My future ambition is to expand this concept to be a Rosetta Stone for

the

Post by Royce Williams
superset of all formats supported by any known product. Small steps

first.

Post by Royce Williams
:)

Cool. With JtR's command-line dynamics, its list of supported formats
is sort of "infinite", though. ;-) But I guess you'll list only those
actually seen in use somewhere.

Heh, fair enough.

Post by Solar Designer
BTW, I found that the command-line dynamics are much easier to use than
having to remember the old numeric dynamics.

That makes sense.

[magnum]

Post by Royce Williams
I'm working on a Hashkiller Rosetta stone (list of upload formats supported
by Hashkiller, and how to use those modes in hashcat, john, and MDXfind.
https://gist.github.com/roycewilliams/28a9e940e7cd37268ceeac4962bda757
Any help/tips appreciated. I don't know the underlying algorithm of many
product-specific formats, so I'm almost certainly missing some obvious ones.
My future ambition is to expand this concept to be a Rosetta Stone for the
superset of all formats supported by any known product. Small steps first.
:)

Most formats consisting of a combination of simple primitives, like
'md5(md5($pass).md5($salt))' are possible in JtR on CPU using the
"dynamic compiler" format. That very example would be written like this:

$ ../run/john -test -form:dynamic='md5(md5($pass).md5($salt))'
Benchmarking: dynamic=md5(md5($p).md5($s)) [128/128 AVX 4x3]... DONE
Many salts: 12448K c/s real, 12573K c/s virtual
Only one salt: 5458K c/s real, 5404K c/s virtual

It can do many crazy combinations that is (probably) not used anywhere:

$ ../run/john -test -form:dynamic='sha1(md5(md4($pass).$salt))'
Benchmarking: dynamic=sha1(md5(md4($p).$s)) [128/128 AVX 4x1]... DONE
Many salts: 5816K c/s real, 5816K c/s virtual
Only one salt: 5340K c/s real, 5340K c/s virtual

Sometimes it's not very fast, but it's always there - very handy. Note
that any time there is a dedicated format for your need, you can bet
that one is faster, and sometimes a LOT faster, than this "dynamic
compiler" stuff.


On another note, I'd add input syntax (or sample hashes) to the Rosetta
Stone if I were you. Sometimes they differ, especially in how/where you
specify the salt. Hashcat uses hash:salt while JtR never EVER has a ':'
within a ciphertext (it's impossible, by design, and that makes the pot
file unambigous).
Sometimes the various crackers also differ in whether they want the salt
as plaintext or hexified, and (if plaintext) they might have different
ways to hex-escape stuff (eg. hashcat $HEX[cafe] vs. JtR $HEX$cafe)


Cheers,
magnum

[Royce Williams]

(Apologies for the duplicate post; I've arbitrarily picked the other one:

http://www.openwall.com/lists/john-users/2017/09/29/6

... as the "real" one, and will tie this thread off)

Post by magnum

Post by Royce Williams
I'm working on a Hashkiller Rosetta stone (list of upload formats supported
by Hashkiller, and how to use those modes in hashcat, john, and MDXfind.
https://gist.github.com/roycewilliams/28a9e940e7cd37268ceeac4962bda757
Any help/tips appreciated. I don't know the underlying algorithm of many
product-specific formats, so I'm almost certainly missing some obvious ones.
My future ambition is to expand this concept to be a Rosetta Stone for the
superset of all formats supported by any known product. Small steps first.
:)

Most formats consisting of a combination of simple primitives, like
'md5(md5($pass).md5($salt))' are possible in JtR on CPU using the "dynamic
$ ../run/john -test -form:dynamic='md5(md5($pass).md5($salt))'
Benchmarking: dynamic=md5(md5($p).md5($s)) [128/128 AVX 4x3]... DONE
Many salts: 12448K c/s real, 12573K c/s virtual
Only one salt: 5458K c/s real, 5404K c/s virtual
$ ../run/john -test -form:dynamic='sha1(md5(md4($pass).$salt))'
Benchmarking: dynamic=sha1(md5(md4($p).$s)) [128/128 AVX 4x1]... DONE
Many salts: 5816K c/s real, 5816K c/s virtual
Only one salt: 5340K c/s real, 5340K c/s virtual
Sometimes it's not very fast, but it's always there - very handy. Note
that any time there is a dedicated format for your need, you can bet that
one is faster, and sometimes a LOT faster, than this "dynamic compiler"
stuff.

Indeed. As noted in the other thread, I will try to make that clear in the
table somehow when I move it from a gist to something more durable.

Post by magnum
On another note, I'd add input syntax (or sample hashes) to the Rosetta
Stone if I were you. Sometimes they differ, especially in how/where you
specify the salt. Hashcat uses hash:salt while JtR never EVER has a ':'
within a ciphertext (it's impossible, by design, and that makes the pot
file unambigous).
Sometimes the various crackers also differ in whether they want the salt
as plaintext or hexified, and (if plaintext) they might have different ways
to hex-escape stuff (eg. hashcat $HEX[cafe] vs. JtR $HEX$cafe)

Yes, syntax would be key for a Rosetta Stone! I do intend to make sure to
include it. Making it directly verifiable by example is the best way to
ensure that people can be confident in the accuracy.

Thanks!

Royce