new fails, no exception, no handler

[TheSheep]

Code:

int MyNewHandler( size_t size )
{
puts("[MyNewHandler] Allocation error!");

return 0; //abort
}

void TryMemAlloc()
{
unsigned char *c;

puts("[TryMemAlloc] Started......");

_set_new_handler(MyNewHandler);

try
	{
	c=new unsigned char;
	}
catch(...)
	{
	puts("[TryMemAlloc] Exception thrown!");
	}
if(c==0)
	puts("[TryMemAlloc] char not allocated!");
else
	printf("[TryMemAlloc] char allocated at %p.\n",c);

puts("[TryMemAlloc] Done");
}

Output:

[TryMemAlloc] Started......
[TryMemAlloc] char not allocated!
[TryMemAlloc] Done

As you can see, new fails but gives no indication that it's out of memory by throwing an exception (which AFAIK it shouldn't do when I set up my own handler, OK) or calling MyNewHandler. And I don't have any reason to think it would run out of memory either, especially when allocating 1 byte!!
Obviously I'm not doing all this just to allocate a byte - the same problem occurs when trying to dynamically allocate a class I've made.

Compiled using MS VC++ 5.0, non-MFC Win32 DLL, multithreaded (2 threads).


Edit - I appreciate it's preferable to post full compilable code but it's not really possible. I have a large DLL with a few thousand lines and 18 different CPP modules. This is a late stage in the second thread, after much network I/O and HTTP downloading (using a separate, MFC DLL).
For that reason I'd appreciate any suggestions, no matter how general, and I'll try them out.

[Valmont]

What makes you think the allocation failed to start wiith?

[Locutus]

If I compile your code into a small program like this:

int MyNewHandler(size_t)
{
	puts("[MyNewHandler] Allocation error!");
	return 0;
}

int main()
{
	unsigned char *c;

	puts("[TryMemAlloc] Started......");

	_set_new_handler(MyNewHandler);

	try
	{
		c=new unsigned char[0x7fffffff];
	}
	catch(...)
	{
		puts("[TryMemAlloc] Exception thrown!");
	}
	if(c==0)
		puts("[TryMemAlloc] char not allocated!");
	else
		printf("[TryMemAlloc] char allocated at %p.\n",c);

	puts("[TryMemAlloc] Done");

	return EXIT_SUCCESS;
}

I get this as output (MSVC6):

[TryMemAlloc] Started......
[MyNewHandler] Allocation error!
[TryMemAlloc] char not allocated!
[TryMemAlloc] Done

Originally Posted by TheSheep

Edit - I appreciate it's preferable to post full compilable code but it's not really possible. I have a large DLL with a few thousand lines and 18 different CPP modules. This is a late stage in the second thread, after much network I/O and HTTP downloading (using a separate, MFC DLL).
For that reason I'd appreciate any suggestions, no matter how general, and I'll try them out.

At first sight, I don't see anything obvious in your code that would prevent the handler from getting called. It is fairly rare for new to fail due to "memory exhaustion" these days, unless you've got for example a huge memory leak somewhere. It might be a good idea to have a look at how much memory your program allocates, for example by using the Windows task manager. (I assume you're running on Windows since you're using MSVC). The fact that the program is multithreaded doesn't make things easier either . In general, if you want us to be usefull, your best bet is probably to try and create the smallest possible program that fails in this way. In fact, it's quite possible that doing so leads to you finding the problem yourself .

Note that this isn't necessarily standard C++ either, since _set_new_handler is an MS specific "extension". (Arguably, doing *anything* semi-non-trivial with MSVC probably isn't standard )

[TheSheep]

@Valmont: new returned NULL.
@Locutus: like you suggested, I pulled out huge chunks of my program until it worked.

Part of my program is the Unreal API, a set of headers and libraries for making mods for Unreal-engine games. I stumbled across this:
inline void* operator new( unsigned int Size )
A little tweaking, and the output is now this:

[TryMemAlloc] Started......
"inline void* operator new (unsigned int size)" has been called.
[TryMemAlloc] char not allocated!
[TryMemAlloc] Done.

So I think I've found my problem.

Thanks for all the advice. The learning curve continues...

[Valmont]

Since this is the standard C/C++ forum, the code would be different. See comments as well for a few tips:

#include <iostream>
#include <new> //set_new_handler()
#include <cstdlib> //abort()

using namespace std;

//The standard denotes set_new_handler to be defined as:
// typedef void (*ptrF)()
void MyNewHandler()
{
  cerr<<"[MyNewHandler] Allocation error!";
  
  //std::abort()... use this as a very last resort. 
  //Solution: use the standard provided exception handling mechanism for 
  //allocation failures by calling its appropriate exception.
  abort();
}

int main()
{
  unsigned char *c;
  
  //Now we install our own handler. 
  //NOTE: if this handler is called, operator new() will NEVER retrun NULL.
  set_new_handler(MyNewHandler);

  try
  {
    c=new unsigned char[0x7fffffff];
  }
  catch(...)
  {
    //This will never be shown!
    cout <<"[TryMemAlloc] Exception thrown!";
  }
  
  cout<<" char allocated at address: " << c << endl;
  delete[] c;

  return 0;
}

The code posted by the OP is not part of the ISO standard.

[Valmont]

@Valmont: new returned NULL.

Perhaps (not on mine though!) . Best thing is to upgrade to a new compiler. See the sticky for two IDE's with new compilers in it.

[Valmont]

Before the days of exception handling, a failure of operator new set the pointer to NULL. The pointer needed to be tested explicitly for NULL to check for failures. These days that is not needed anymore because we have a more neat way of exception handling. But setting the pointer to NULL will not occur anymore.
See how easy it can be:

#include <iostream>
#include <new> 
using namespace std;

int main()
{
  //Always healthy to initialize pointers with 0.
  unsigned char *c = 0;
  
  
  try
  {
    c=new unsigned char[0x7fffffff];
  }
  catch(const bad_alloc& xa)
  {
    cerr << "ERROR: bad allocation caught." << endl;
    //New compilers will call operator delete() if the constructor throws
    //an exception. So no worries about deletion here.
  }
  
  //Avoid sementation fault. Now it shows why initializing with 0 is handy.
  if(c!=0) //If allocated successfully...
  {
    delete[] c; //...delete pointer.
  }

  return 0;
}