[SOLVED] Crashing when calling QTcpSocket::setSocketDescriptor()

Issue

my project using QTcpSocket and the function setSocketDescriptor(). The code is very normal

QTcpSocket *socket = new QTcpSocket();
socket->setSocketDescriptor(this->m_socketDescriptor);

This coding worked fine most of the time until I ran a performance testing on Windows Server 2016, the crash occurred. I debugging with the crash dump, here is the log

0000004f`ad1ff4e0 : ucrtbase!abort+0x4e
00000000`6ed19790 : Qt5Core!qt_logging_to_console+0x15a
000001b7`79015508 : Qt5Core!QMessageLogger::fatal+0x6d
0000004f`ad1ff0f0 : Qt5Core!QEventDispatcherWin32::installMessageHook+0xc0
00000000`00000000 : Qt5Core!QEventDispatcherWin32::createInternalHwnd+0xf3
000001b7`785b0000 : Qt5Core!QEventDispatcherWin32::registerSocketNotifier+0x13e
000001b7`7ad57580 : Qt5Core!QSocketNotifier::QSocketNotifier+0xf9
00000000`00000001 : Qt5Network!QLocalSocket::socketDescriptor+0x4cf7
00000000`00000000 : Qt5Network!QAbstractSocket::setSocketDescriptor+0x256

In the stderr log, I see those logs

CreateWindow() for QEventDispatcherWin32 internal window failed (Not enough storage is available to process this command.)
Qt: INTERNAL ERROR: failed to install GetMessage hook: 8, Not enough storage is available to process this command.

Here is the function, where the code was stopped on the Qt codebase

void QEventDispatcherWin32::installMessageHook()
{
    Q_D(QEventDispatcherWin32);

    if (d->getMessageHook)
        return;

    // setup GetMessage hook needed to drive our posted events
    d->getMessageHook = SetWindowsHookEx(WH_GETMESSAGE, (HOOKPROC) qt_GetMessageHook, NULL, GetCurrentThreadId());
    if (Q_UNLIKELY(!d->getMessageHook)) {
        int errorCode = GetLastError();
        qFatal("Qt: INTERNAL ERROR: failed to install GetMessage hook: %d, %s",
               errorCode, qPrintable(qt_error_string(errorCode)));
    }
}

I did research and the error Not enough storage is available to process this command. maybe the OS (Windows) does not have enough resources to process this function (SetWindowsHookEx) and failed to create a hook, and then Qt fire a fatal signal, finally my app is killed.
I tested this on Windows Server 2019, the app is working fine, no crashes appear.
I just want to know more about the meaning of the error message (stderr) cause I don’t really know what is "Not enough storage"? I think it is maybe the limit or bug of the Windows Server 2016? If yes, is there any way to overcome this issue on Windows Server 2016?

Solution

After researching for a few days I finally can configure the Windows Server 2016 setting (registry) to prevent the crash.
So basically it is a limitation of the OS itself, it is called desktop heap limitation.
https://docs.microsoft.com/en-us/troubleshoot/windows-server/performance/desktop-heap-limitation-out-of-memory

(The funny thing is the error message is Not enough storage is available to process this command but the real problem came to desktop heap limitation. )

So for the solution, flowing the steps in this link: https://docs.microsoft.com/en-us/troubleshoot/system-center/orchestrator/increase-maximum-number-concurrent-policy-instances

I increased the 3rd parameter of SharedSection to 2048 and it fix the issue.

Summary steps:

Desktop Heap for the non-interactive desktops is identified by the third parameter of the SharedSection= segment of the following registry value:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows

The default data for this registry value will look something like the following:

%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,3072,512 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off MaxRequestThreads=16

The value to be entered into the Third Parameter of the SharedSection= segment should be based on the calculation of:

(number of desired concurrent policies) * 10 = (third parameter value)

Example: If it’s desired to have 200 concurrent policy instances, then 200 * 10 = 2000, rounding up to a nice memory number gives you 2048as the third parameter resulting in the following update to be made to the registry value:

SharedSection=1024,3072,2048

Answered By – Le Ngoc Thuong

Answer Checked By – Senaida (BugsFixing Volunteer)