[john-users] Getting full performance out of multiple GPU's

[Adam Lininger]

I've got an 8-gpu cracking rig that I've inherited from someone who had it
setup with hashcat. Apparently it worked well there.

As I prefer john, I've been trying to get it set up. I have it up and
running, but the Utilization output from --list=opencl-devices hovers
around 0-1%. Is there a way to improve this?

I'm running John the Ripper 1.8.0.12-jumbo-1-bleeding-6144b8f (freshly
re-pulled and re-built today).

Thanks,
Adam

[Solar Designer]

Hi Adam,

Post by Adam Lininger
I've got an 8-gpu cracking rig that I've inherited from someone who had it
setup with hashcat. Apparently it worked well there.
As I prefer john, I've been trying to get it set up. I have it up and
running, but the Utilization output from --list=opencl-devices hovers
around 0-1%. Is there a way to improve this?
I'm running John the Ripper 1.8.0.12-jumbo-1-bleeding-6144b8f (freshly
re-pulled and re-built today).

To use a GPU at all, you need to be invoking one of JtR "formats" with
"-opencl" in its name. You do this with an option like e.g.
"--format=sha512crypt-opencl", but corresponding to whatever hash type
you're actually cracking.

And if it's one of so-called "fast hashes" - like raw MD5, and thus
"--format=raw-md5-opencl" - then you also need to use "--mask" (and
specify a mask suitable for your attack) in order to have candidate
passwords at least partially generated on GPU. You may use this mask
mode along with another cracking mode, in which case you'd refer to that
other mode's generated partial candidate password as "?w" in the mask.

Then, as magnum has just pointed out in another thread, to use more than
one GPU you need to list all GPU device numbers and also specify at
least an equivalent number of "forks", like this: "--dev=0,1,2,3,4,5,6,7
--fork=8". (We got to enhance JtR to print a warning when the specified
number of OpenCL devices is greater than the number of forks.)

Please post your current JtR command line and explain its resulting
behavior if you require further help, or just to share your experience
with the community.

Alexander

[Adam Lininger]

Alexander,

Thank you for the help. I believe I am properly using a GPU and the actual
hashing is being done there. The nvidia-smi command seems to agree. Based
on your email, I suspect the problem is that the korelogic rule generation
is still happening on the CPU rather than the GPU. Is there an easy way to
correct that? I've copied my command below.

john --wordlist=rockyou.dict --format=descrypt-opencl
--devices=1,2,3,4,5,6,7,8 --fork=8 --rules=korelogic htpasswd

Thanks,
Adam

Post by Solar Designer
Hi Adam,

Post by Adam Lininger
I've got an 8-gpu cracking rig that I've inherited from someone who had

it

Post by Adam Lininger
setup with hashcat. Apparently it worked well there.
As I prefer john, I've been trying to get it set up. I have it up and
running, but the Utilization output from --list=opencl-devices hovers
around 0-1%. Is there a way to improve this?
I'm running John the Ripper 1.8.0.12-jumbo-1-bleeding-6144b8f (freshly
re-pulled and re-built today).

To use a GPU at all, you need to be invoking one of JtR "formats" with
"-opencl" in its name. You do this with an option like e.g.
"--format=sha512crypt-opencl", but corresponding to whatever hash type
you're actually cracking.
And if it's one of so-called "fast hashes" - like raw MD5, and thus
"--format=raw-md5-opencl" - then you also need to use "--mask" (and
specify a mask suitable for your attack) in order to have candidate
passwords at least partially generated on GPU. You may use this mask
mode along with another cracking mode, in which case you'd refer to that
other mode's generated partial candidate password as "?w" in the mask.
Then, as magnum has just pointed out in another thread, to use more than
one GPU you need to list all GPU device numbers and also specify at
least an equivalent number of "forks", like this: "--dev=0,1,2,3,4,5,6,7
--fork=8". (We got to enhance JtR to print a warning when the specified
number of OpenCL devices is greater than the number of forks.)
Please post your current JtR command line and explain its resulting
behavior if you require further help, or just to share your experience
with the community.
Alexander

[Solar Designer]

Post by Adam Lininger
Thank you for the help. I believe I am properly using a GPU and the actual
hashing is being done there. The nvidia-smi command seems to agree. Based
on your email, I suspect the problem is that the korelogic rule generation
is still happening on the CPU rather than the GPU. Is there an easy way to
correct that? I've copied my command below.
john --wordlist=rockyou.dict --format=descrypt-opencl
--devices=1,2,3,4,5,6,7,8 --fork=8 --rules=korelogic htpasswd

Yes, it's one of those cases where you'll have to use --mask, and this
prevents you from using solely the KoreLogic rules - at best, you may
add a mask on top of them, but a mask would push many candidates beyond
the length of 8, which is the maximum for descrypt, making this approach
not entirely reasonable.

You may have better luck running the rules against your descrypt hashes
on CPU, and using your GPUs for something like:

john --incremental --mask='?w?a?a?a?a' --format=descrypt-opencl --devices=1,2,3,4,5,6,7,8 --fork=8 htpasswd

This will output something like:

Loaded 1 password hash (descrypt-opencl, traditional crypt(3) [DES OpenCL])
Warning: MaxLen = 13 is too large for the current hash type, reduced to 4

The warning is OK - it means that JtR correctly reduced incremental
mode's maximum length to 4 due to descrypt's maximum of 8 and mask of 4.

You could also want to use the --min-length and --max-length options to
split this into several invocations of JtR varying by length (perhaps
from 5 to 8), so that you keep track of what lengths have already been
exhausted vs. not yet. You'll probably complete lengths up to 7
inclusive within a day, and then focus on length 8 for some days more
(if for one hash; otherwise multiply by the number of different salts).

Alexander

[Adam Lininger]

Thanks Alexander. That gives me a much better understanding of how JtR does
things internally also. I believe I will have much better luck with this
guidance.

Adam

Post by Solar Designer

Post by Adam Lininger
Thank you for the help. I believe I am properly using a GPU and the

actual

Post by Adam Lininger
hashing is being done there. The nvidia-smi command seems to agree. Based
on your email, I suspect the problem is that the korelogic rule

generation

Post by Adam Lininger
is still happening on the CPU rather than the GPU. Is there an easy way

to

Post by Adam Lininger
correct that? I've copied my command below.
john --wordlist=rockyou.dict --format=descrypt-opencl
--devices=1,2,3,4,5,6,7,8 --fork=8 --rules=korelogic htpasswd

Yes, it's one of those cases where you'll have to use --mask, and this
prevents you from using solely the KoreLogic rules - at best, you may
add a mask on top of them, but a mask would push many candidates beyond
the length of 8, which is the maximum for descrypt, making this approach
not entirely reasonable.
You may have better luck running the rules against your descrypt hashes
john --incremental --mask='?w?a?a?a?a' --format=descrypt-opencl
--devices=1,2,3,4,5,6,7,8 --fork=8 htpasswd
Loaded 1 password hash (descrypt-opencl, traditional crypt(3) [DES OpenCL])
Warning: MaxLen = 13 is too large for the current hash type, reduced to 4
The warning is OK - it means that JtR correctly reduced incremental
mode's maximum length to 4 due to descrypt's maximum of 8 and mask of 4.
You could also want to use the --min-length and --max-length options to
split this into several invocations of JtR varying by length (perhaps
from 5 to 8), so that you keep track of what lengths have already been
exhausted vs. not yet. You'll probably complete lengths up to 7
inclusive within a day, and then focus on length 8 for some days more
(if for one hash; otherwise multiply by the number of different salts).
Alexander

[Piotr Dobrogost]

Post by Adam Lininger
Alexander,
Thank you for the help. I believe I am properly using a GPU and the actual
hashing is being done there. The nvidia-smi command seems to agree. Based

What gpu and memory utilization does nvidia-smi show in your case?

Below are some exemplary values of utilization when running
--format=gpg-opencl on 8 gpus:

utilization.gpu [%], utilization.memory [%]
13 %, 6 %
11 %, 4 %
13 %, 6 %
10 %, 4 %
75 %, 48 %
7 %, 3 %
13 %, 6 %
7 %, 3 %
utilization.gpu [%], utilization.memory [%]
7 %, 3 %
13 %, 6 %
7 %, 3 %
7 %, 3 %
0 %, 0 %
7 %, 3 %
7 %, 3 %
7 %, 3 %
utilization.gpu [%], utilization.memory [%]
7 %, 3 %
13 %, 6 %
13 %, 6 %
10 %, 4 %
0 %, 0 %
13 %, 6 %
13 %, 6 %
13 %, 6 %
utilization.gpu [%], utilization.memory [%]
13 %, 6 %
13 %, 6 %
7 %, 3 %
12 %, 5 %
47 %, 39 %
11 %, 6 %
13 %, 6 %
7 %, 3 %

These are rather low figures
 Any idea why are they so low?
For comparison mining with https://github.com/ethereum-mining/ethminer/
results in both gpu and memory continuously utilized in 100%.

Regards,
Piotr

[Solar Designer]

Post by Piotr Dobrogost
Below are some exemplary values of utilization when running

GPU time utilization should be near 100%. GPU memory utilization may
vary depending on actual needs - there's no universally correct/expected
level of GPU memory utilization.

Post by Piotr Dobrogost
utilization.gpu [%], utilization.memory [%]
13 %, 6 %
11 %, 4 %
13 %, 6 %
10 %, 4 %
75 %, 48 %
7 %, 3 %
13 %, 6 %
7 %, 3 %

The GPU time utilization figures here are indeed very wrong, but it's
unrealistic to troubleshoot this without you posting the exact JtR
command line you used and the output from JtR (just exclude the cracked
passwords, if any).

Alexander

[Piotr Dobrogost]

Post by Solar Designer

Post by Piotr Dobrogost
Below are some exemplary values of utilization when running

GPU time utilization should be near 100%.

That's what anyone would like and that's what I was expecting :)

Post by Solar Designer
GPU memory utilization may
vary depending on actual needs - there's no universally correct/expected
level of GPU memory utilization.

True but still you expected near 100% and I think it's reasonable
expectation.

Post by Solar Designer
The GPU time utilization figures here are indeed very wrong, but it's
unrealistic to troubleshoot this without you posting the exact JtR
command line you used and the output from JtR (just exclude the cracked
passwords, if any).

Command is
$ john --format=gpg-opencl --fork=8 --dev=2,3,4,5,6,7,8,9 test.txt.gpg
where test.txt.gpg is result of
$ gpg --symmetric --passphrase 'ala ma kota' test.txt

john.log:

1 0:00:00:00 Starting a new session
1 0:00:00:00 Loaded a total of 1 password hash
1 0:00:00:00 Cost 1 (s2k-count) is 65536 for all loaded hashes
1 0:00:00:00 Cost 2 (hash algorithm [2:SHA1 8:SHA256 10:SHA512]) is 2
for all loaded hashes
1 0:00:00:00 Cost 3 (cipher algorithm [1:IDEA 2:3DES 3:CAST5
4:Blowfish 7:AES128 8:AES192 9:AES256 10:Twofish 11:Camellia128
12:Camellia192 13:Camellia256]) is 7 for all loaded hashes
1 0:00:00:00 - Node numbers 1-8 of 8 (fork)
1 0:00:00:00 Device 2: GeForce GTX 1070
1 0:00:00:00 Command line:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/john
--format=gpg-opencl --fork=8 --devices=2,3,4,5,6,7,8,9 hash
1 0:00:00:00 - UTF-8 input encoding enabled
1 0:00:00:00 - Passwords will be stored UTF-8 encoded in .pot file
1 0:00:00:00 Sorting salts, for performance
1 0:00:00:00 Device 2: GeForce GTX 1070
1 0:00:00:00 - OpenCL (auto-tuned) LWS: 128, GWS: 15360 (120 blocks)
1 0:00:00:00 - Hash type: gpg-opencl, OpenPGP / GnuPG Secret Key
(min-len 0, max-len 41 [worst case UTF-8] to 125 [ASCII])
1 0:00:00:00 - Algorithm: SHA1/SHA2 OpenCL
1 0:00:00:00 - Candidate passwords will be buffered and tried in chunks of 15360
1 0:00:00:00 - Will reject candidates longer than 125 bytes
1 0:00:00:00 Proceeding with "single crack" mode
1 0:00:00:00 - SingleWordsPairMax used is 6
1 0:00:00:00 - SingleRetestGuessed = true
1 0:00:00:00 - 1129 preprocessed word mangling rules
1 0:00:00:00 - Allocated 1 buffer of 128 candidate passwords
1 0:00:00:00 - Rule #1: ':' accepted as ''
1 0:00:00:00 - No information to base candidate passwords on
1 0:00:00:00 - Processing the remaining buffered candidate passwords, if any
1 0:00:00:00 Proceeding with wordlist mode
1 0:00:00:00 - Rules: Wordlist
1 0:00:00:00 - Wordlist file:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/password.lst
1 0:00:00:00 - memory mapping wordlist (26325 bytes)
1 0:00:00:00 - 57 preprocessed word mangling rules
1 0:00:00:00 - Will distribute rules across nodes, then switch to
distributing words
1 0:00:00:00 - Rule #1: ':' accepted as ''
1 0:00:00:00 - Rule #9: '>2 !?A l ^1' accepted as '>2!?Al^1'
1 0:00:00:00 - Rule #17: '<* >2 !?A l $4' accepted as '<*>2!?Al$4'
1 0:00:00:00 - Rule #25: '/?w @?w >3 l' accepted as '/?w@?w>3l'
1 0:00:00:00 - Rule #33: '-c <* >2 !?A c $2' accepted as '<*>2!?Ac$2'
1 0:00:00:00 - Rule #41: '-c <* >2 !?A c $8' accepted as '<*>2!?Ac$8'
1 0:00:00:00 - Rule #49: '>2 !?A l ^6' accepted as '>2!?Al^6'
1 0:00:00:00 - Switching to distributing words
1 0:00:00:00 - Rule #57: '-s-c x** M l Q' rejected
1 0:00:00:00 Proceeding with "incremental" mode: ASCII
1 0:00:00:00 - Lengths 0 to 13, up to 95 different characters
8 0:00:00:00 Device 9: GeForce GTX 1070
8 0:00:00:00 Command line:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/john
--format=gpg-opencl --fork=8 --devices=2,3,4,5,6,7,8,9 hash
8 0:00:00:00 - UTF-8 input encoding enabled
8 0:00:00:00 - Passwords will be stored UTF-8 encoded in .pot file
8 0:00:00:00 Sorting salts, for performance
8 0:00:00:00 Device 9: GeForce GTX 1070
8 0:00:00:00 - OpenCL (auto-tuned) LWS: 128, GWS: 15360 (120 blocks)
8 0:00:00:00 - Will reject candidates longer than 125 bytes
8 0:00:00:00 Proceeding with "single crack" mode
8 0:00:00:00 - SingleWordsPairMax used is 6
8 0:00:00:00 - SingleRetestGuessed = true
8 0:00:00:00 - 1129 preprocessed word mangling rules
8 0:00:00:00 - Allocated 1 buffer of 128 candidate passwords
8 0:00:00:00 - Rule #8: '-<6 -c >6 '6 /?u l' accepted as '>6'6/?ul'
8 0:00:00:00 - No information to base further candidate passwords on
8 0:00:00:00 - Processing the remaining buffered candidate passwords, if any
8 0:00:00:00 Proceeding with wordlist mode
8 0:00:00:00 - Rules: Wordlist
8 0:00:00:00 - Wordlist file:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/password.lst
8 0:00:00:00 - memory mapping wordlist (26325 bytes)
8 0:00:00:00 - 57 preprocessed word mangling rules
5 0:00:00:00 Device 6: GeForce GTX 1070
5 0:00:00:00 Command line:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/john
--format=gpg-opencl --fork=8 --devices=2,3,4,5,6,7,8,9 hash
5 0:00:00:00 - UTF-8 input encoding enabled
5 0:00:00:00 - Passwords will be stored UTF-8 encoded in .pot file
5 0:00:00:00 Sorting salts, for performance
5 0:00:00:00 Device 6: GeForce GTX 1070
5 0:00:00:00 - OpenCL (auto-tuned) LWS: 128, GWS: 15360 (120 blocks)
5 0:00:00:00 - Will reject candidates longer than 125 bytes
5 0:00:00:00 Proceeding with "single crack" mode
5 0:00:00:00 - SingleWordsPairMax used is 6
5 0:00:00:00 - SingleRetestGuessed = true
5 0:00:00:00 - 1129 preprocessed word mangling rules
5 0:00:00:00 - Allocated 1 buffer of 128 candidate passwords
5 0:00:00:00 - Rule #5: '-s-c x** /?u l' rejected
5 0:00:00:00 - Rule #13: '/?d @?d >4 M c Q' accepted as '/?d@?d>4McQ'
5 0:00:00:00 - No information to base further candidate passwords on
5 0:00:00:00 - Processing the remaining buffered candidate passwords, if any
5 0:00:00:00 Proceeding with wordlist mode
5 0:00:00:00 - Rules: Wordlist
5 0:00:00:00 - Wordlist file:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/password.lst
5 0:00:00:00 - memory mapping wordlist (26325 bytes)
5 0:00:00:00 - 57 preprocessed word mangling rules
8 0:00:00:00 - Will distribute rules across nodes, then switch to
distributing words
8 0:00:00:00 - Rule #8: '>3 !?A l M r Q' accepted as '>3!?AlMrQ'
8 0:00:00:00 - Rule #16: '<* >2 !?A l $5' accepted as '<*>2!?Al$5'
8 0:00:00:00 - Rule #24: '/?v @?v >3 l' accepted as '/?v@?v>3l'
8 0:00:00:00 - Rule #32: '>2 !?A l ^4' accepted as '>2!?Al^4'
8 0:00:00:00 - Rule #40: '-c <* >2 !?A c $6' accepted as '<*>2!?Ac$6'
8 0:00:00:00 - Rule #48: '>2 !?A l ^5' accepted as '>2!?Al^5'
8 0:00:00:00 - Rule #56: '-s x**' rejected
8 0:00:00:00 - Switching to distributing words
8 0:00:00:00 - Rule #57: '-s-c x** M l Q' rejected
8 0:00:00:00 Proceeding with "incremental" mode: ASCII
8 0:00:00:00 - Lengths 0 to 13, up to 95 different characters
5 0:00:00:00 - Will distribute rules across nodes, then switch to
distributing words
5 0:00:00:00 - Rule #5: '<* >2 !?A l $1' accepted as '<*>2!?Al$1'
5 0:00:00:00 - Rule #13: '<* >2 !?A l $3' accepted as '<*>2!?Al$3'
5 0:00:00:00 - Rule #21: '<* >2 !?A l $.' accepted as '<*>2!?Al$.'
5 0:00:00:00 - Rule #29: '<7 >1 !?A l d M 'l f Q' accepted as '<7>1!?AldM'lfQ'
5 0:00:00:00 - Rule #37: '-c <* >2 !?A c $5' accepted as '<*>2!?Ac$5'
5 0:00:00:00 - Rule #45: '>2 !?A l ^3' accepted as '>2!?Al^3'
5 0:00:00:00 - Rule #53: '-: <* >2 !?A l M I Q' accepted as '<*>2!?AlMIQ'
5 0:00:00:00 - Switching to distributing words
5 0:00:00:00 - Rule #57: '-s-c x** M l Q' rejected
5 0:00:00:00 Proceeding with "incremental" mode: ASCII
5 0:00:00:00 - Lengths 0 to 13, up to 95 different characters
3 0:00:00:00 Device 4: GeForce GTX 1070
3 0:00:00:00 Command line:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/john
--format=gpg-opencl --fork=8 --devices=2,3,4,5,6,7,8,9 hash
3 0:00:00:00 - UTF-8 input encoding enabled
3 0:00:00:00 - Passwords will be stored UTF-8 encoded in .pot file
3 0:00:00:00 Sorting salts, for performance
3 0:00:00:00 Device 4: GeForce GTX 1070
3 0:00:00:00 - OpenCL (auto-tuned) LWS: 128, GWS: 15360 (120 blocks)
3 0:00:00:00 - Will reject candidates longer than 125 bytes
3 0:00:00:00 Proceeding with "single crack" mode
3 0:00:00:00 - SingleWordsPairMax used is 6
3 0:00:00:00 - SingleRetestGuessed = true
3 0:00:00:00 - 1129 preprocessed word mangling rules
3 0:00:00:00 - Allocated 1 buffer of 128 candidate passwords
3 0:00:00:00 - Rule #3: '-c (?a c Q' accepted as '(?acQ'
3 0:00:00:00 - No information to base further candidate passwords on
3 0:00:00:00 - Processing the remaining buffered candidate passwords, if any
3 0:00:00:00 Proceeding with wordlist mode
3 0:00:00:00 - Rules: Wordlist
3 0:00:00:00 - Wordlist file:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/password.lst
3 0:00:00:00 - memory mapping wordlist (26325 bytes)
3 0:00:00:00 - 57 preprocessed word mangling rules
3 0:00:00:00 - Will distribute rules across nodes, then switch to
distributing words
3 0:00:00:00 - Rule #3: '-c (?a >2 !?X c Q' accepted as '(?a>2!?XcQ'
3 0:00:00:00 - Rule #11: '<* >2 !?A l $2' accepted as '<*>2!?Al$2'
3 0:00:00:00 - Rule #19: '<* >2 !?A l $6' accepted as '<*>2!?Al$6'
3 0:00:00:00 - Rule #27: '-c <+ >2 !?A c r' accepted as '<+>2!?Acr'
3 0:00:00:00 - Rule #35: '-c <* >2 !?A c $3' accepted as '<*>2!?Ac$3'
3 0:00:00:00 - Rule #43: '-c <* >2 !?A c $?' accepted as '<*>2!?Ac$?'
3 0:00:00:00 - Rule #51: '-c <* >2 !?A c p' accepted as '<*>2!?Acp'
3 0:00:00:00 - Switching to distributing words
3 0:00:00:00 - Rule #57: '-s-c x** M l Q' rejected
3 0:00:00:00 Proceeding with "incremental" mode: ASCII
3 0:00:00:00 - Lengths 0 to 13, up to 95 different characters
2 0:00:00:00 Device 3: GeForce GTX 1070
2 0:00:00:00 Command line:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/john
--format=gpg-opencl --fork=8 --devices=2,3,4,5,6,7,8,9 hash
2 0:00:00:00 - UTF-8 input encoding enabled
2 0:00:00:00 - Passwords will be stored UTF-8 encoded in .pot file
2 0:00:00:00 Sorting salts, for performance
2 0:00:00:00 Device 3: GeForce GTX 1070
2 0:00:00:00 - OpenCL (auto-tuned) LWS: 128, GWS: 15360 (120 blocks)
2 0:00:00:00 - Will reject candidates longer than 125 bytes
2 0:00:00:00 Proceeding with "single crack" mode
2 0:00:00:00 - SingleWordsPairMax used is 6
2 0:00:00:00 - SingleRetestGuessed = true
2 0:00:00:00 - 1129 preprocessed word mangling rules
2 0:00:00:00 - Allocated 1 buffer of 128 candidate passwords
2 0:00:00:00 - Rule #2: '-s x**' rejected
2 0:00:00:00 - Rule #10: '/?d @?d >4' accepted as '/?d@?d>4'
2 0:00:00:00 - No information to base further candidate passwords on
2 0:00:00:00 - Processing the remaining buffered candidate passwords, if any
2 0:00:00:00 Proceeding with wordlist mode
2 0:00:00:00 - Rules: Wordlist
2 0:00:00:00 - Wordlist file:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/password.lst
2 0:00:00:00 - memory mapping wordlist (26325 bytes)
2 0:00:00:00 - 57 preprocessed word mangling rules
7 0:00:00:00 Device 8: GeForce GTX 1070
7 0:00:00:00 Command line:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/john
--format=gpg-opencl --fork=8 --devices=2,3,4,5,6,7,8,9 hash
7 0:00:00:00 - UTF-8 input encoding enabled
7 0:00:00:00 - Passwords will be stored UTF-8 encoded in .pot file
7 0:00:00:00 Sorting salts, for performance
7 0:00:00:00 Device 8: GeForce GTX 1070
7 0:00:00:00 - OpenCL (auto-tuned) LWS: 128, GWS: 15360 (120 blocks)
7 0:00:00:00 - Will reject candidates longer than 125 bytes
7 0:00:00:00 Proceeding with "single crack" mode
7 0:00:00:00 - SingleWordsPairMax used is 6
7 0:00:00:00 - SingleRetestGuessed = true
7 0:00:00:00 - 1129 preprocessed word mangling rules
7 0:00:00:00 - Allocated 1 buffer of 128 candidate passwords
7 0:00:00:00 - Rule #7: '-<7 >7 '7 l' accepted as '>7'7l'
7 0:00:00:00 - No information to base further candidate passwords on
7 0:00:00:00 - Processing the remaining buffered candidate passwords, if any
7 0:00:00:00 Proceeding with wordlist mode
7 0:00:00:00 - Rules: Wordlist
7 0:00:00:00 - Wordlist file:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/password.lst
7 0:00:00:00 - memory mapping wordlist (26325 bytes)
7 0:00:00:00 - 57 preprocessed word mangling rules
2 0:00:00:00 - Will distribute rules across nodes, then switch to
distributing words
2 0:00:00:00 - Rule #2: '-c >3 !?X l Q' accepted as '>3!?XlQ'
2 0:00:00:00 - Rule #10: '-c >2 !?X u Q M c Q u' accepted as '>2!?XuQMcQu'
2 0:00:00:00 - Rule #18: '<* >2 !?A l $8' accepted as '<*>2!?Al$8'
2 0:00:00:00 - Rule #26: '-c <7 >1 !?A c d' accepted as '<7>1!?Acd'
2 0:00:00:00 - Rule #34: '-c <* >2 !?A c $!' accepted as '<*>2!?Ac$!'
2 0:00:00:00 - Rule #42: '-c <* >2 !?A c $.' accepted as '<*>2!?Ac$.'
2 0:00:00:00 - Rule #50: '>2 !?A l ^8' accepted as '>2!?Al^8'
2 0:00:00:00 - Switching to distributing words
2 0:00:00:00 - Rule #57: '-s-c x** M l Q' rejected
2 0:00:00:00 Proceeding with "incremental" mode: ASCII
2 0:00:00:00 - Lengths 0 to 13, up to 95 different characters
7 0:00:00:00 - Will distribute rules across nodes, then switch to
distributing words
7 0:00:00:00 - Rule #7: '<7 >1 !?A l d' accepted as '<7>1!?Ald'
7 0:00:00:00 - Rule #15: '<* >2 !?A l $9' accepted as '<*>2!?Al$9'
7 0:00:00:00 - Rule #23: '/?p @?p >3 l' accepted as '/?p@?p>3l'
7 0:00:00:00 - Rule #31: '>2 !?A l ^2' accepted as '>2!?Al^2'
7 0:00:00:00 - Rule #39: '-c <* >2 !?A c $4' accepted as '<*>2!?Ac$4'
7 0:00:00:00 - Rule #47: '>2 !?A l ^9' accepted as '>2!?Al^9'
7 0:00:00:00 - Rule #55: '-c <* >2 !?A c M I Q' accepted as '<*>2!?AcMIQ'
7 0:00:00:00 - Switching to distributing words
7 0:00:00:00 - Rule #57: '-s-c x** M l Q' rejected
7 0:00:00:00 Proceeding with "incremental" mode: ASCII
7 0:00:00:00 - Lengths 0 to 13, up to 95 different characters
6 0:00:00:00 Device 7: GeForce GTX 1070
6 0:00:00:00 Command line:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/john
--format=gpg-opencl --fork=8 --devices=2,3,4,5,6,7,8,9 hash
6 0:00:00:00 - UTF-8 input encoding enabled
6 0:00:00:00 - Passwords will be stored UTF-8 encoded in .pot file
6 0:00:00:00 Sorting salts, for performance
6 0:00:00:00 Device 7: GeForce GTX 1070
6 0:00:00:00 - OpenCL (auto-tuned) LWS: 32, GWS: 15360 (480 blocks)
6 0:00:00:00 - Will reject candidates longer than 125 bytes
6 0:00:00:00 Proceeding with "single crack" mode
6 0:00:00:00 - SingleWordsPairMax used is 6
6 0:00:00:00 - SingleRetestGuessed = true
6 0:00:00:00 - 1129 preprocessed word mangling rules
6 0:00:00:00 - Allocated 1 buffer of 32 candidate passwords
6 0:00:00:00 - Rule #6: '-<6 >6 '6' accepted as '>6'6'
6 0:00:00:00 - No information to base further candidate passwords on
6 0:00:00:00 - Processing the remaining buffered candidate passwords, if any
6 0:00:00:00 Proceeding with wordlist mode
6 0:00:00:00 - Rules: Wordlist
6 0:00:00:00 - Wordlist file:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/password.lst
6 0:00:00:00 - memory mapping wordlist (26325 bytes)
6 0:00:00:00 - 57 preprocessed word mangling rules
6 0:00:00:00 - Will distribute rules across nodes, then switch to
distributing words
6 0:00:00:00 - Rule #6: '-c <* >2 !?A c $1' accepted as '<*>2!?Ac$1'
6 0:00:00:00 - Rule #14: '<* >2 !?A l $7' accepted as '<*>2!?Al$7'
6 0:00:00:00 - Rule #22: '<* >2 !?A l $?' accepted as '<*>2!?Al$?'
6 0:00:00:00 - Rule #30: '-c <+ >2 !?A l M r Q c r' accepted as '<+>2!?AlMrQcr'
6 0:00:00:00 - Rule #38: '-c <* >2 !?A c $7' accepted as '<*>2!?Ac$7'
6 0:00:00:00 - Rule #46: '>2 !?A l ^7' accepted as '>2!?Al^7'
6 0:00:00:00 - Rule #54: '-c <* >2 !?A c M P Q' accepted as '<*>2!?AcMPQ'
6 0:00:00:00 - Switching to distributing words
6 0:00:00:00 - Rule #57: '-s-c x** M l Q' rejected
6 0:00:00:00 Proceeding with "incremental" mode: ASCII
6 0:00:00:00 - Lengths 0 to 13, up to 95 different characters
4 0:00:00:00 Device 5: GeForce GTX 1070
4 0:00:00:00 Command line:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/john
--format=gpg-opencl --fork=8 --devices=2,3,4,5,6,7,8,9 hash
4 0:00:00:00 - UTF-8 input encoding enabled
4 0:00:00:00 - Passwords will be stored UTF-8 encoded in .pot file
4 0:00:00:00 Sorting salts, for performance
4 0:00:00:00 Device 5: GeForce GTX 1070
4 0:00:00:00 - OpenCL (auto-tuned) LWS: 128, GWS: 15360 (120 blocks)
4 0:00:00:00 - Will reject candidates longer than 125 bytes
4 0:00:00:00 Proceeding with "single crack" mode
4 0:00:00:00 - SingleWordsPairMax used is 6
4 0:00:00:00 - SingleRetestGuessed = true
4 0:00:00:00 - 1129 preprocessed word mangling rules
4 0:00:00:00 - Allocated 1 buffer of 128 candidate passwords
4 0:00:00:00 - Rule #4: '-c l Q' accepted as 'lQ'
4 0:00:00:00 - No information to base further candidate passwords on
4 0:00:00:00 - Processing the remaining buffered candidate passwords, if any
4 0:00:00:00 Proceeding with wordlist mode
4 0:00:00:00 - Rules: Wordlist
4 0:00:00:00 - Wordlist file:
/home/miner/johntheripper/JohnTheRipper-bleeding-jumbo/run/password.lst
4 0:00:00:00 - memory mapping wordlist (26325 bytes)
4 0:00:00:00 - 57 preprocessed word mangling rules
4 0:00:00:00 - Will distribute rules across nodes, then switch to
distributing words
4 0:00:00:00 - Rule #4: '<* >2 !?A l p' accepted as '<*>2!?Alp'
4 0:00:00:00 - Rule #12: '<* >2 !?A l $!' accepted as '<*>2!?Al$!'
4 0:00:00:00 - Rule #20: '<* >2 !?A l $0' accepted as '<*>2!?Al$0'
4 0:00:00:00 - Rule #28: '-c >2 !?A l M r Q c' accepted as '>2!?AlMrQc'
4 0:00:00:00 - Rule #36: '-c <* >2 !?A c $9' accepted as '<*>2!?Ac$9'
4 0:00:00:00 - Rule #44: '-c <* >2 !?A c $0' accepted as '<*>2!?Ac$0'
4 0:00:00:00 - Rule #52: '-: <* >2 !?A l M P Q' accepted as '<*>2!?AlMPQ'
4 0:00:00:00 - Switching to distributing words
4 0:00:00:00 - Rule #57: '-s-c x** M l Q' rejected
4 0:00:00:00 Proceeding with "incremental" mode: ASCII
4 0:00:00:00 - Lengths 0 to 13, up to 95 different characters
1 0:00:00:00 - Switching to length 6
1 0:00:00:00 - Expanding tables for length 6 to character count 1
1 0:00:00:00 - Trying length 6, fixed @1, character count 1
1 0:00:00:00 - Switching to length 5
1 0:00:00:00 - Expanding tables for length 5 to character count 2
1 0:00:00:00 - Trying length 5, fixed @4, character count 2
1 0:00:00:00 - Expanding tables for length 5 to character count 3
1 0:00:00:00 - Trying length 5, fixed @2, character count 3




3 0:00:01:32 - Expanding tables for length 2 to character count 48
3 0:00:01:32 - Trying length 2, fixed @2, character count 48
3 0:00:01:32 - Switching to length 6
3 0:00:01:32 - Expanding tables for length 6 to character count 24
3 0:00:01:32 - Trying length 6, fixed @4, character count 24
3 0:00:01:42 Session aborted


Regards,
Piotr

[magnum]

Post by Piotr Dobrogost

Post by Solar Designer
GPU memory utilization may
vary depending on actual needs - there's no universally correct/expected
level of GPU memory utilization.

True but still you expected near 100% and I think it's reasonable
expectation.

No, it's not. For illustrative purposes, let's say you have a dream GPU
with 128 GB memory and you need to crack a single hash that need about 1
billionth of that. How/why would we "utilize" the rest of your memory,
and in what way do you picture it would be a good thing!?

Your GPU utilization though, should ideally be pegged very close to 100%
and that's the only important parameter here.

magnum

[Solar Designer]

Post by Piotr Dobrogost

Post by Solar Designer
GPU memory utilization may
vary depending on actual needs - there's no universally correct/expected
level of GPU memory utilization.

True but still you expected near 100% and I think it's reasonable
expectation.

No, I didn't expect that and it's not a reasonable expectation. magnum
has just illustrated that with an example.

Post by Piotr Dobrogost
Command is
$ john --format=gpg-opencl --fork=8 --dev=2,3,4,5,6,7,8,9 test.txt.gpg

Oh, OK. So, as seen per the log file you sent you have a young session
that is still frequently switching between incremental mode's lengths,
trying to test candidate passwords in a more optimal order. To achieve
a steady speed in terms of candidate passwords tested per second, and
full GPU utilization, you'll need to either let this session run for
much longer or instead run attacks that are a better fit for in those
terms right away - such as locking incremental mode to a specific length
(with "--min-length" and "--max-length") or running a large wordlist.
Please try that and let us know of the results.

Post by Piotr Dobrogost
where test.txt.gpg is result of
$ gpg --symmetric --passphrase 'ala ma kota' test.txt

This passphrase is unlikely to be cracked against that file type soon,
unless you focus the attack.
Actually, I meant output seen on the terminal - not the log file - but
this works, too. It's missing the reported speeds, though.

Alexander