[john-users] Compiling John as a shared object

[harrim4n]

Hi all,

I'm trying to compile John as a shared object to use in python bindings
for a project I'm working on. I'm using the latest version from
https://github.com/magnumripper/JohnTheRipper/tree/bleeding-jumbo.

However I'm running into some issues doing so, namely the x86-64.S file.
As this file is assembly, I can't simply recompile it with -fPIC.

This causes the following error when trying to create a "libjohn.so" file:

/usr/bin/ld: x86-64.o: relocation R_X86_64_PC32 against symbol
`nt_buffer8x' can not be used when making a shared object; recompile
with -fPIC

However, to me, the assembly already looks to be position independent.
This is also indicated by the comment on line 1769:

/* ...converted to use %rip-relative addressing, still public domain */

Does anyone have pointers as to how I could either modify the assembly
to be PI or any other way I could create python bindings for John?


I am aware that I could rebuild the python interpreter and use static
linking, however I would like to avoid this option because of obvious
maintainability issues. I also tried simply using a statically built lib
in the setup.py file, but then I get

ImportError: /usr/lib/python2.7/site-packages/pyJohn.so: undefined
symbol: fmt_nsec3

Thanks!

harrim4n

[Solar Designer]

Post by harrim4n
I'm trying to compile John as a shared object to use in python bindings
for a project I'm working on. I'm using the latest version from
https://github.com/magnumripper/JohnTheRipper/tree/bleeding-jumbo.

Great.

Post by harrim4n
However I'm running into some issues doing so, namely the x86-64.S file.
As this file is assembly, I can't simply recompile it with -fPIC.
/usr/bin/ld: x86-64.o: relocation R_X86_64_PC32 against symbol
`nt_buffer8x' can not be used when making a shared object; recompile
with -fPIC
However, to me, the assembly already looks to be position independent.

Unfortunately, it is not. For PIC code, the RIP-relative addressing
should be used to access the GOT entry, through which the actual
"non-local" variables (not in the same translation unit) are accessed.
Accessing them directly via RIP-relative addressing is not enough.

Post by harrim4n
Does anyone have pointers as to how I could either modify the assembly
to be PI

Here's the kind of changes needed:

$ cat x.c
extern int x;
int f(void) { return x; }
$ gcc -S -O2 x.c -o x1.s
$ gcc -S -O2 x.c -o x2.s -fPIC
$ diff -U0 x1.s x2.s
[...]
- movl x(%rip), %eax
+ movq ***@GOTPCREL(%rip), %rax
+ movl (%rax), %eax

Post by harrim4n
or any other way I could create python bindings for John?

You could drop the assembly files altogether. We have equivalent (and
sometimes superior) C code for everything contained in there, except for
the runtime CPU detection. You just need to set the corresponding *_ASM
defines in arch.h (a file that's generated when you ./configure, or is
symlinked to e.g. x86-64.h) to 0 and "#undef NT_X86_64" to tell the rest
of JtR not to expect assembly code for any of the crypto.

Yet another option might be to look at and reuse some experience from
this project:

https://github.com/L0phtCrack/jtrdll

This is how JtR is integrated with LC7, so it focuses on Windows builds
and is not usable for you directly.

Post by harrim4n
I am aware that I could rebuild the python interpreter and use static
linking, however I would like to avoid this option because of obvious
maintainability issues. I also tried simply using a statically built lib
in the setup.py file, but then I get
ImportError: /usr/lib/python2.7/site-packages/pyJohn.so: undefined
symbol: fmt_nsec3

I suppose you merely include samples of these errors, and you actually
got a lot more of similar errors? It's hard to debug this via a mailing
list. I guess you'd need to investigate it on your own.

Alexander

[harrim4n]

Post by Solar Designer

Post by harrim4n
I'm trying to compile John as a shared object to use in python bindings
for a project I'm working on. I'm using the latest version from
https://github.com/magnumripper/JohnTheRipper/tree/bleeding-jumbo.

Great.

Post by harrim4n
However I'm running into some issues doing so, namely the x86-64.S file.
As this file is assembly, I can't simply recompile it with -fPIC.
/usr/bin/ld: x86-64.o: relocation R_X86_64_PC32 against symbol
`nt_buffer8x' can not be used when making a shared object; recompile
with -fPIC
However, to me, the assembly already looks to be position independent.

Unfortunately, it is not. For PIC code, the RIP-relative addressing
should be used to access the GOT entry, through which the actual
"non-local" variables (not in the same translation unit) are accessed.
Accessing them directly via RIP-relative addressing is not enough.

Post by harrim4n
Does anyone have pointers as to how I could either modify the assembly
to be PI

$ cat x.c
extern int x;
int f(void) { return x; }
$ gcc -S -O2 x.c -o x1.s
$ gcc -S -O2 x.c -o x2.s -fPIC
$ diff -U0 x1.s x2.s
[...]
- movl x(%rip), %eax
+ movl (%rax), %eax

Ah, I see.

Post by Solar Designer

Post by harrim4n
or any other way I could create python bindings for John?

You could drop the assembly files altogether. We have equivalent (and
sometimes superior) C code for everything contained in there, except for
the runtime CPU detection. You just need to set the corresponding *_ASM
defines in arch.h (a file that's generated when you ./configure, or is
symlinked to e.g. x86-64.h) to 0 and "#undef NT_X86_64" to tell the rest
of JtR not to expect assembly code for any of the crypto.

This seems like the best option to me. I'll give this a shot when I get
time and report back if I get any errors.

Post by Solar Designer
Yet another option might be to look at and reuse some experience from
https://github.com/L0phtCrack/jtrdll
This is how JtR is integrated with LC7, so it focuses on Windows builds
and is not usable for you directly.

Post by harrim4n
I am aware that I could rebuild the python interpreter and use static
linking, however I would like to avoid this option because of obvious
maintainability issues. I also tried simply using a statically built lib
in the setup.py file, but then I get
ImportError: /usr/lib/python2.7/site-packages/pyJohn.so: undefined
symbol: fmt_nsec3

I suppose you merely include samples of these errors, and you actually
got a lot more of similar errors? It's hard to debug this via a mailing
list. I guess you'd need to investigate it on your own.
Alexander

Actually, this was the only error I got on import. But I suspect that
the python interpreter aborts on the first error it encounters, so if
this error was "fixed" a new one would pop up.

Thanks for your help!

harrim4n