Paradoxon: Silent Crash On Python's Ctypes.cdll When Importing, But Not When Running Directly - How Is This Possible?

October 11, 2024 Post a Comment

So, being a Linux guy I stumbled into something pretty puzzling on Windows that I just can't explain. I have a project structure analougus to this example: D:\PROJECT | | tolkien

Solution 1:

From the moment I saw this question, I wanted to say it's Undefined Behavior (UB). Python comes with its C runtime (UCRTLib), while the Cygwin.dll comes with its own. Mixing compilers and C runtimes in a process, is generally a recipe for disaster. I found an official statement [Cygwin]: 6.15. Can I link with both MSVCRT*.DLL and cygwin1.dll? (emphasis is mine):

No, you must use one or the other, they are mutually exclusive.

Check [SO]: How to circumvent Windows Universal CRT headers dependency on vcruntime.h (@CristiFati's answer) for more details on MSVCRT*.DLL

Now, the beauty of UB is that it describes a seemingly random behavior.

I've prepared a comprehensive example (slightly modifying your code).

isengrad.c:

#if defined(_WIN32)#  define ISENGRAD_EXPORT_API __declspec(dllexport)#else#  define ISENGRAD_EXPORT_API#endif


ISENGRAD_EXPORT_API intisengrad(int hobbit) {
    return hobbit / 2;
}

script0.py:

#!/usr/bin/env python3import sys
import ctypes


dll_name = "./lib/isengrad_{0:s}_{1:03d}.dll".format(sys.argv[1][:3] if sys.argv else sys.platform[:3].lower(), ctypes.sizeof(ctypes.c_void_p) * 8)
print("Attempting to load: {0:s}".format(dll_name))
isengrad_dll = ctypes.CDLL(dll_name)
print("DLL Loaded")


defmain():
    isengrad_func = isengrad_dll.isengrad
    isengrad_func.argtypes = [ctypes.c_int]
    isengrad_func.restype = ctypes.c_int

    res = isengrad_func(46)
    print("{0:s} returned {1:}".format(isengrad_func.__name__, res))


if __name__ == "__main__":
    print("Python {0:s} {1:d}bit on {2:s}\n".format(" ".join(item.strip() for item in sys.version.split("\n")), 64if sys.maxsize > 0x100000000else32, sys.platform))
    main()
    print("\nDone.")

script1.py:

#!/usr/bin/env python3import sys
import script0


defmain():
    passif __name__ == "__main__":
    print("Python {0:s} {1:d}bit on {2:s}\n".format(" ".join(item.strip() for item in sys.version.split("\n")), 64if sys.maxsize > 0x100000000else32, sys.platform))
    main()
    print("\nDone.")

Outputs:

I'll be using 3 windows:
- cmd - Win (32bit and 64bit)
- Cygwin's Mintty:
  - 64bit
  - 32bit
Note that even if I'll paste each's contents in one chunk (to avoid scattering them), I switched between them when running the commands

Cygwin32bit:

[cfati@cfati-5510-0:/cygdrive/e/Work/Dev/StackOverflow/q056855348]> ~/sopr.sh
*** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ***

[032bit prompt]> gcc -shared -fPIC -o lib/isengrad_cyg_032.dll isengrad.c
[032bit prompt]>  ls lib/*.dll
lib/isengrad_cyg_032.dll  lib/isengrad_cyg_064.dll  lib/isengrad_win_032.dll  lib/isengrad_win_064.dll
[032bit prompt]>
[032bit prompt]> python3 script0.py cyg
Attempting to load: ./lib/isengrad_cyg_032.dll
DLL Loaded
Python 3.6.4 (default, Jan  7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin

isengrad returned 23

Done.
[032bit prompt]>
[032bit prompt]> python3 script1.py cyg
Attempting to load: ./lib/isengrad_cyg_032.dll
DLL Loaded
Python 3.6.4 (default, Jan  7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin


Done.
[032bit prompt]>
[032bit prompt]> python3 script0.py win
Attempting to load: ./lib/isengrad_win_032.dll
DLL Loaded
Python 3.6.4 (default, Jan  7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin

isengrad returned 23

Done.
[032bit prompt]>
[032bit prompt]> python3 script1.py win
Attempting to load: ./lib/isengrad_win_032.dll
DLL Loaded
Python 3.6.4 (default, Jan  7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin


Done.

Cygwin64bit:

[cfati@cfati-5510-0:/cygdrive/e/Work/Dev/StackOverflow/q056855348]> ~/sopr.sh
*** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ***

[064bit prompt]>  gcc -shared -fPIC -o lib/isengrad_cyg_064.dll isengrad.c
[064bit prompt]> ls lib/*.dll
lib/isengrad_cyg_032.dll  lib/isengrad_cyg_064.dll  lib/isengrad_win_032.dll  lib/isengrad_win_064.dll
[064bit prompt]>
[064bit prompt]> python3 script0.py cyg
Attempting to load: ./lib/isengrad_cyg_064.dll
DLL Loaded
Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin

isengrad returned 23

Done.
[064bit prompt]>
[064bit prompt]> python3 script1.py cyg
Attempting to load: ./lib/isengrad_cyg_064.dll
DLL Loaded
Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin


Done.
[064bit prompt]>
[064bit prompt]> python3 script0.py win
Attempting to load: ./lib/isengrad_win_064.dll
DLL Loaded
Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin

isengrad returned 23

Done.
[064bit prompt]>
[064bit prompt]> python3 script1.py win
Attempting to load: ./lib/isengrad_win_064.dll
DLL Loaded
Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin


Done.

cmd:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q056855348]> sopr.bat
*** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ***

[prompt]> dir /b lib

[prompt]> "c:\Install\x86\Microsoft\Visual Studio Community\2017\VC\Auxiliary\Build\vcvarsall.bat" x64
**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.14
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

[prompt]> cl /nologo /DDLL isengrad.c  /link /NOLOGO /DLL /OUT:lib\isengrad_win_064.dll
isengrad.c
   Creating library lib\isengrad_win_064.lib and object lib\isengrad_win_064.exp

[prompt]>
[prompt]> "c:\Install\x86\Microsoft\Visual Studio Community\2017\VC\Auxiliary\Build\vcvarsall.bat" x86
**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.14
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x86'

[prompt]> cl /nologo /DDLL isengrad.c  /link /NOLOGO /DLL /OUT:lib\isengrad_win_032.dll
isengrad.c
   Creating library lib\isengrad_win_032.lib and object lib\isengrad_win_032.exp

[prompt]> dir /b lib\*.dll
isengrad_cyg_032.dll
isengrad_cyg_064.dll
isengrad_win_032.dll
isengrad_win_064.dll

[prompt]> set _PATH=%PATH%

[prompt]> :: Python 32bit
[prompt]> set PATH=%_PATH%;e:\Install\x86\Cygwin\Cygwin\Version\bin

[prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" script0.py win
Attempting to load: ./lib/isengrad_win_032.dll
DLL Loaded
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32

isengrad returned 23

Done.

[prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" script1.py win
Attempting to load: ./lib/isengrad_win_032.dll
DLL Loaded
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32


Done.

[prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" script0.py cyg
Attempting to load: ./lib/isengrad_cyg_032.dll
DLL Loaded
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32

isengrad returned 23

Done.

[prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" script1.py cyg
Attempting to load: ./lib/isengrad_cyg_032.dll
DLL Loaded
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32


Done.

[prompt]> :: Python 64bit
[prompt]> set PATH=%_PATH%;c:\Install\x64\Cygwin\Cygwin\AllVers\bin

[prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" script0.py win
Attempting to load: ./lib/isengrad_win_064.dll
DLL Loaded
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32

isengrad returned 23

Done.

[prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" script1.py win
Attempting to load: ./lib/isengrad_win_064.dll
DLL Loaded
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32


Done.

[prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" script0.py cyg
Attempting to load: ./lib/isengrad_cyg_064.dll
DLL Loaded
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32

isengrad returned 23

Done.

[prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" script1.py cyg
Attempting to load: ./lib/isengrad_cyg_064.dll

[prompt]>
[prompt]> echo %errorlevel%
-1073741819

As seen, cross compiler .exe - .dll worked in 7 (out of 8) cases (crashed on 64bitWinPython with script1.py), while the same compiler worked in all 8 of them.

So, I'd advise that when playing with such environments, try to keep the compilers used to build various parts consistent (or compatible at least).

Update #0

I just thought of a reason why things could go wrong on 64bit: sizeof(long) generally differs (sizes below are in bytes):

4 on Win
8 on Cygwin (on Nix, in general)

Same thing for sizeof(long double) (which is 2 * sizeof(long)).

So, if the Cygwin.dll exposes some long value greater than 2 ** 64 (1 << 64), it will be truncated in the Win process, and in this case a crash might occur. Theoretically, this situation should affect the reverse scenario as well, but it doesn't.

There are other factors that might lead to this behavior, like default memory alignment, and so on.

tmahurin

Paradoxon: Silent Crash On Python's Ctypes.cdll When Importing, But Not When Running Directly - How Is This Possible?

Solution 1:

Update #0

Post a Comment for "Paradoxon: Silent Crash On Python's Ctypes.cdll When Importing, But Not When Running Directly - How Is This Possible?"

Widget HTML #3