Clientedge Windhawk mod (issues)

anixx , there are many issues with your code, and some questions I have too.

// @compilerOptions -lcomdlg32 -ldwmapi
This tells the compiler to link comdlg32.lib and dwmapi.lib, which correspond to comdlg32.dll and dwmapi.dll respectively. None of the functions used here so you can (and should) unlink these libraries by removing that whole line.

#include <dwmapi.h>
Again, dwmapi is not needed.

Why are you hooking ShowWindow in particular? Most mods which change window styles will hook CreateWindowExW instead, since it is generally more reliable. A good example of a somewhat simple mod which hooks CreateWindowExW is my Themed Regedit ListView mod.

Also, while WINBOOL and BOOL are the same type, BOOL is much more commonly used. I could barely find anything about WINBOOL online.

HWND cn = NULL;
HWND p = hWnd;

Don't really see the point of setting p equal to hWnd. I can see you use it later in those FindWindowEx calls, but it's not really necessary when you can just substitute it for hWnd.

//Mathematica

if (lstrcmpW(bufr, L"NotebookFrame") == 0)
{
   p = FindWindowEx(p, NULL, L"NotebookContent", NULL);
   if(p != NULL)
   {
       cn = p;
   }

}

// Notepad

if (lstrcmpW(bufr, L"Notepad") == 0)
{
   p = FindWindowEx(p, NULL, L"Edit", NULL);
   if(p != NULL)
   {
       cn = p;
   }

}

// Console

if (lstrcmpW(bufr, L"ConsoleWindowClass") == 0)
{
   cn = p;
}
Many comments here. For one, you should probably be using wcscmp, which is a standard C library function, instead of lstrcmpW.

Second, with Mathematica, you are most likely looking for the child of a window that does not exist yet.

Third, with console windows: Console windows are spawned by csrss.exe which Windhawk does not inject to, and therefore cannot hook its CreateProcess function. Basically, Windhawk will always be late to inject to console windows, unless you launch conhost.exe, which doesn't even work with legacy console anyway.

Fourth, you can condense p != NULL to simply p.

if(cn != NULL)
{
   SetWindowLongPtrW(cn, GWL_EXSTYLE, GetWindowLongPtrW(cn, GWL_EXSTYLE) | 0x200); //Sets the style to include WS_EX_CLIENTEDGE    
   RECT rect;
   GetWindowRect(cn, &rect);
   SetWindowPos(cn, NULL, NULL, NULL, rect.right - rect.left + 1, rect.bottom - rect.top, SWP_NOMOVE | SWP_NOZORDER | SWP_DEFERERASE);
   SetWindowPos(cn, NULL, NULL, NULL, rect.right - rect.left, rect.bottom - rect.top, SWP_NOMOVE | SWP_NOZORDER | SWP_DEFERERASE);
}
I think what you are doing here is trying to trigger a repaint of the window. This is a pretty awful way of doing this, and you should instead rely on functions actually relating to painting.

Example:

if (cn)
{
   RECT rcClient;
   GetClientRect(cn, &rcClient);
   InvalidateRect(cn, &rcClient, TRUE);
   UpdateWindow(cn);
}
Here, we're getting the rect of the client area, invalidating it so that the system knows it should repaint it on the next WM_PAINT message, and then we call UpdateWindow which actually sends a WM_PAINT message to the window.