[john-users] p3.16xlarge AWS instance with 8 x Tesla V100 cards.

[Luis Rocha]

Hi,

For the enthusiasts, I thought it would be interesting to share the
performance of JtR mask mode on raw-sha1 running on a p3.16xlarge AWS
instance which contains 8 x Tesla V100 cards.

$ nvidia-smi
Sat Oct 28 18:39:05 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.59 Driver Version: 384.59
|
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.
ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute
M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... Off | 00000000:00:17.0 Off |
0 |
| N/A 43C P0 36W / 300W | 0MiB / 16152MiB | 0%
Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... Off | 00000000:00:18.0 Off |
0 |
| N/A 40C P0 35W / 300W | 0MiB / 16152MiB | 0%
Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla V100-SXM2... Off | 00000000:00:19.0 Off |
0 |
| N/A 40C P0 35W / 300W | 0MiB / 16152MiB | 0%
Default |
+-------------------------------+----------------------+----------------------+
| 3 Tesla V100-SXM2... Off | 00000000:00:1A.0 Off |
0 |
| N/A 42C P0 36W / 300W | 0MiB / 16152MiB | 0%
Default |
+-------------------------------+----------------------+----------------------+
| 4 Tesla V100-SXM2... Off | 00000000:00:1B.0 Off |
0 |
| N/A 46C P0 38W / 300W | 0MiB / 16152MiB | 0%
Default |
+-------------------------------+----------------------+----------------------+
| 5 Tesla V100-SXM2... Off | 00000000:00:1C.0 Off |
0 |
| N/A 42C P0 36W / 300W | 0MiB / 16152MiB | 0%
Default |
+-------------------------------+----------------------+----------------------+
| 6 Tesla V100-SXM2... Off | 00000000:00:1D.0 Off |
0 |
| N/A 41C P0 36W / 300W | 0MiB / 16152MiB | 0%
Default |
+-------------------------------+----------------------+----------------------+
| 7 Tesla V100-SXM2... Off | 00000000:00:1E.0 Off |
0 |
| N/A 42C P0 36W / 300W | 0MiB / 16152MiB | 3%
Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU
Memory |
| GPU PID Type Process name Usage
|
|=============================================================================|
| No running processes found
|
+-----------------------------------------------------------------------------+




$ ./john --format=Raw-SHA1-opencl --pot=big.pot hashes --nolog --verb
=1 --save-memory=1 --fork=8 --mask=?a?a?a?a?a?a?a?a
Using default input encoding: UTF-8
Loaded 61829207 password hashes with no different salts (Raw-SHA1-opencl
[SHA1 OpenCL])
Node numbers 1-8 of 8 (fork)

2 20g 0:00:14:51 0.22% (ETA: 2017-11-02 13:21) 0.02244g/s 2025Mp/s 2025Mc/s
125206TC/s ar%ecP]t..a-J~yP]t
6 16g 0:00:14:50 0.22% (ETA: 2017-11-02 11:54) 0.01795g/s 2051Mp/s 2051Mc/s
126818TC/s a@$=x2O-..a4CWR2O-
7 165g 0:00:14:50 0.22% (ETA: 2017-11-02 11:56) 0.1852g/s 2051Mp/s 2051Mc/s
126791TC/s a5%15uk,..a\J:6uk,
3 120g 0:00:14:51 0.22% (ETA: 2017-11-02 11:03) 0.1346g/s 2067Mp/s 2067Mc/s
127781TC/s ao(0?,&6..aGU|;,&6
8 518g 0:00:14:50 0.22% (ETA: 2017-11-02 12:39) 0.5816g/s 2038Mp/s 2038Mc/s
125988TC/s a(5gPwc%..aI`8*wc%
5 207g 0:00:14:50 0.22% (ETA: 2017-11-02 12:36) 0.2324g/s 2039Mp/s 2039Mc/s
126044TC/s a-rZn{B...ad?P2{B.
1 38693g 0:00:14:50 0.22% (ETA: 2017-11-02 12:55) 43.45g/s 2033Mp/s
2033Mc/s 125675TC/s a&~K"***@C^T1a
4 42g 0:00:14:50 0.22% (ETA: 2017-11-02 11:56) 0.04715g/s 2051Mp/s 2051Mc/s
126791TC/s a;***@U; x..aL|H-; x


Best,
Luis

[Jeroen]

Thanks for sharing.
Quite an expensive setup for long runs I guess?

Cheers,

Jeroen

-----Original Message-----
Sent: zaterdag 28 oktober 2017 22:00
Subject: [john-users] p3.16xlarge AWS instance with 8 x Tesla V100 cards.
Hi,
For the enthusiasts, I thought it would be interesting to share the
performance of JtR mask mode on raw-sha1 running on a p3.16xlarge AWS instance
which contains 8 x Tesla V100 cards.
$ nvidia-smi
Sat Oct 28 18:39:05 2017
+-----------------------------------------------------------------------------
+
| NVIDIA-SMI 384.59 Driver Version: 384.59
|
|-------------------------------+----------------------+----------------------
+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.
ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute
M. |
|===============================+======================+======================
|
| 0 Tesla V100-SXM2... Off | 00000000:00:17.0 Off |
0 |
| N/A 43C P0 36W / 300W | 0MiB / 16152MiB | 0%
Default |
+-------------------------------+----------------------+----------------------
+
| 1 Tesla V100-SXM2... Off | 00000000:00:18.0 Off |
0 |
| N/A 40C P0 35W / 300W | 0MiB / 16152MiB | 0%
Default |
+-------------------------------+----------------------+----------------------
+
| 2 Tesla V100-SXM2... Off | 00000000:00:19.0 Off |
0 |
| N/A 40C P0 35W / 300W | 0MiB / 16152MiB | 0%
Default |
+-------------------------------+----------------------+----------------------
+
| 3 Tesla V100-SXM2... Off | 00000000:00:1A.0 Off |
0 |
| N/A 42C P0 36W / 300W | 0MiB / 16152MiB | 0%
Default |
+-------------------------------+----------------------+----------------------
+
| 4 Tesla V100-SXM2... Off | 00000000:00:1B.0 Off |
0 |
| N/A 46C P0 38W / 300W | 0MiB / 16152MiB | 0%
Default |
+-------------------------------+----------------------+----------------------
+
| 5 Tesla V100-SXM2... Off | 00000000:00:1C.0 Off |
0 |
| N/A 42C P0 36W / 300W | 0MiB / 16152MiB | 0%
Default |
+-------------------------------+----------------------+----------------------
+
| 6 Tesla V100-SXM2... Off | 00000000:00:1D.0 Off |
0 |
| N/A 41C P0 36W / 300W | 0MiB / 16152MiB | 0%
Default |
+-------------------------------+----------------------+----------------------
+
| 7 Tesla V100-SXM2... Off | 00000000:00:1E.0 Off |
0 |
| N/A 42C P0 36W / 300W | 0MiB / 16152MiB | 3%
Default |
+-------------------------------+----------------------+----------------------
+
+-----------------------------------------------------------------------------
+
| Processes: GPU
Memory |
| GPU PID Type Process name Usage
|
|=======================================================================
|======|
| No running processes found
|
+-----------------------------------------------------------------------------
+
$ ./john --format=Raw-SHA1-opencl --pot=big.pot hashes --nolog --verb
=1 --save-memory=1 --fork=8 --mask=?a?a?a?a?a?a?a?a Using default input
encoding: UTF-8 Loaded 61829207 password hashes with no different salts (Raw-
SHA1-opencl
[SHA1 OpenCL])
Node numbers 1-8 of 8 (fork)
2 20g 0:00:14:51 0.22% (ETA: 2017-11-02 13:21) 0.02244g/s 2025Mp/s 2025Mc/s
125206TC/s ar%ecP]t..a-J~yP]t
6 16g 0:00:14:50 0.22% (ETA: 2017-11-02 11:54) 0.01795g/s 2051Mp/s 2051Mc/s
7 165g 0:00:14:50 0.22% (ETA: 2017-11-02 11:56) 0.1852g/s 2051Mp/s 2051Mc/s
126791TC/s a5%15uk,..a\J:6uk,
3 120g 0:00:14:51 0.22% (ETA: 2017-11-02 11:03) 0.1346g/s 2067Mp/s 2067Mc/s
127781TC/s ao(0?,&6..aGU|;,&6
8 518g 0:00:14:50 0.22% (ETA: 2017-11-02 12:39) 0.5816g/s 2038Mp/s 2038Mc/s
125988TC/s a(5gPwc%..aI`8*wc%
5 207g 0:00:14:50 0.22% (ETA: 2017-11-02 12:36) 0.2324g/s 2039Mp/s 2039Mc/s
126044TC/s a-rZn{B...ad?P2{B.
1 38693g 0:00:14:50 0.22% (ETA: 2017-11-02 12:55) 43.45g/s 2033Mp/s 2033Mc/s
4 42g 0:00:14:50 0.22% (ETA: 2017-11-02 11:56) 0.04715g/s 2051Mp/s 2051Mc/s
Best,
Luis

[Luis Rocha]

Post by Jeroen
Thanks for sharing.
Quite an expensive setup for long runs I guess?

The instance is rated at $24.48/hour but this one is a spot instance with
85% discount.

[Luis Rocha]

btw, during a second run I got the following output:

Node numbers 1-8 of 8 (fork)
john: malloc.c:2374: sysmalloc: Assertion `(old_top == (((mbinptr) (((char
*) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk,
fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned
long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2
*(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) &&
((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)'
failed.
Proceeding with mask:?a?a?a?a?a?a?a?a