Re: user defined malloc

["Ashif S. Harji" <asharji at plg2 dot math dot uwaterloo dot ca>]

> > My question is do I have to also override the _malloc_r routine, and
> > if so, could someone provide some information or a link on what is
> > required?  It appears to be a reentrant version of malloc.
> 
> Yup, newlib is heavy into reentrancy.  However, _malloc_r isn't
> exported by cygwin1.dll and cygwin doesn't expect your program to
> replace it.  I'm not sure how much of a "bug" this is, but I'm open
> for suggestions.  Might want to ask on the newlib list also, I'm sure
> they have really good reasons to call the reentrant malloc.
> 


Note, if you wish to skip the preamble, the patches are included at the bottom
of the message.

As per the suggestion, I posted a message in the newlib mailing list.  I
received the following replies:

http://sources.redhat.com/ml/newlib/2000/msg00143.html
http://sources.redhat.com/ml/newlib/2000/msg00145.html

Basically, it appears that because of reentrancy any reentrant library routines
must call _malloc_r instead of malloc.  Therefore, only overriding malloc can
result in two separate allocation algorithms trying to manage the same memory.
In fact, it does not matter if the same algorithm is used to manage the memory
because the instances of the algorithms operate independently.  Thus, the _r
functions such as _malloc_r need to be overriden.  Unfortunately, this is not
currently possible.  Note, it is probably sufficient to simply override the _r
functions as malloc, for example, simply ends up calling _malloc_r anyways.

Ideally, it would great if newlib included hooks similar to glibc that allowed
the user to override the default memory allocation routines such as malloc, but
this is currently not the case.  Overriding _malloc_r is difficult because the
cygwin1.dll appears to be statically linked.  Therefore, all calls to _malloc_r
are statically resolved and so all calls from inside the dll would ignore a
user provided _malloc_r.  The dll gets around this problem with malloc by
storing the address of the user defined malloc (if one is provided) in the
startup code of the program.  As well, the dll provides its *own* version of
malloc which either calls the user defined malloc or _malloc_r directly.  This
bypasses the version of malloc included in newlib because only one malloc can
be included in the dll.

Mimicking these semantics to override _malloc_r is complicated because only one
version of _malloc_r can be included in the dll.  If the dll provides its own
version of _malloc_r, then, similar to malloc routine described above, this
function can call the user defined _malloc_r if it exists (by using the startup
code trick to store the address).  However, in the default circumstance there
is a problem as the actual _malloc_r defined in newlib is not available as a
version is already defined by the dll.  Note, malloc gets around this problem
by calling _malloc_r in the default case and not the malloc provided by newlib.

One possible solution is to try to change newlib so that _malloc_r calls
another routine, for example, doMalloc which does the actual allocation.  Then
the version defined of _malloc_r in the dll can either call the user defined
version of _malloc_r if it exists or call doMalloc directly.  However, it is
not clear that this solution provides any advantages to newlib.  The patch
given below tries to achieve these semantics without requiring any changes to
newlib.  It basically uses the --wrap parameter to the linker to rename calls
to _malloc_r inside the dll.  This change allows the dll to provide its own
_malloc_r (called __wrap__malloc_r) and to include the _malloc_r inside of
newlib (accessed via __real__malloc_r).  So, any calls to _malloc_r in the dll
get resolved to __wrap__malloc_r which can then call the user defined _malloc_r
or the newlib version directly through __real__malloc_r.

This patch is based on the cygwin 1.1.2 source.  Let me know if I need to port
it to the latest development version.  Note, the check

  if ( user_data->_malloc_r != NULL ) {

inside __wrap__malloc_r allows programs not compiled with these changes in
the dll to continue to run.

************************* Patch Makefile.in ***********************


--- back/Makefile.in	Tue Jun  6 22:12:52 2000
+++ Makefile.in	Sat Jul 22 01:39:16 2000
@@ -180,7 +180,8 @@ new-$(LIB_NAME): $(DEF_FILE) $(LIBCOS)
 # Rule to build cygwin.dll
 
 new-$(DLL_NAME): $(DLL_OFILES) $(DEF_FILE) $(DLL_IMPORTS) $(LIBC) $(LIBM) Makefile winver_stamp
-	$(LD) -shared -o $@ -e $(DLL_ENTRY) $(DEF_FILE) $(DLL_OFILES) version.o \
+	$(LD) --wrap _malloc_r --wrap _free_r --wrap _calloc_r --wrap _realloc_r \
+	-shared -o $@ -e $(DLL_ENTRY) $(DEF_FILE) $(DLL_OFILES) version.o \
 	winver.o $(DLL_IMPORTS) $(LIBM) $(LIBGCC) $(MALLOC_OBJ) $(LIBC) $(LIBGCC)
 
 dll_ofiles: $(DLL_OFILES)


************************* Patch libccrt0.cc ***********************

--- back/libccrt0.cc	Sun May 14 12:20:26 2000
+++ libccrt0.cc	Sat Jul 22 01:35:42 2000
@@ -64,6 +64,11 @@ cygwin_crt0_common (MainFunc f)
   this_proc.realloc = &realloc;
   this_proc.calloc = &calloc;
 
+  this_proc._malloc_r = &_malloc_r;
+  this_proc._free_r = &_free_r;
+  this_proc._realloc_r = &_realloc_r;
+  this_proc._calloc_r = &_calloc_r;
+
   /* Setup the module handle so fork can get the path name. */
   this_proc.hmodule = GetModuleHandle (0);

************************* Patch malloc.cc ***********************

--- back/malloc.cc	Sun May 21 21:18:16 2000
+++ malloc.cc	Sat Jul 22 01:35:10 2000
@@ -121,6 +121,73 @@ calloc (size_t nmemb, size_t size)
   res = user_data->calloc (nmemb, size);
   return res;
 }
+
+extern "C" 
+void *
+__real__malloc_r (_reent * reent, size_t size);
+
+extern "C"
+void
+__real__free_r (_reent * reent, void *p);
+
+extern "C"
+void *
+__real__realloc_r (_reent * reent, void *p, size_t size);
+
+extern "C"
+void *
+__real__calloc_r (_reent * reent, size_t nmemb, size_t size);
+
+extern "C"
+void *
+__wrap__malloc_r (_reent * reent, size_t size)
+{
+  void *res;
+  if ( user_data->_malloc_r != NULL ) {
+      res = user_data->_malloc_r (reent, size);
+  } else {
+      res = __real__malloc_r (reent, size);
+  }
+  return res;
+}
+
+extern "C"
+void
+__wrap__free_r (_reent * reent, void *p)
+{
+  if ( user_data->_free_r != NULL ) {
+      user_data->_free_r (reent, p);
+  } else {
+      __real__free_r (reent, p);
+  }
+}
+
+extern "C"
+void *
+__wrap__realloc_r (_reent * reent, void *p, size_t size)
+{
+  void *res;
+  if ( user_data->_realloc_r != NULL ) {
+      res = user_data->_realloc_r (reent, p, size);
+  } else {
+      res = __real__realloc_r (reent, p, size);
+  }
+  return res;
+}
+
+extern "C"
+void *
+__wrap__calloc_r (_reent * reent, size_t nmemb, size_t size)
+{
+  void *res;
+  if ( user_data->_calloc_r != NULL ) {
+      res = user_data->_calloc_r (reent, nmemb, size);
+  } else {
+      res = __real__calloc_r (reent, nmemb, size);
+  }
+  return res;
+}
+
 #endif
 
 /* These routines are used by the application if it
@@ -173,6 +240,57 @@ export_calloc (size_t nmemb, size_t size
     res = _calloc_r (_impure_ptr, nmemb, size);
   else
     res = user_data->calloc (nmemb, size);
+  malloc_printf ("(%d, %d) = %x, called by %x", nmemb, size, res, ((int *)&nmemb)[-1]);
+  return res;
+}
+
+extern "C"
+void
+export__free_r (_reent * reent, void *p)
+{
+  malloc_printf ("(%p), called by %x", p, ((int *)&p)[-1]);
+  if (use_internal_malloc)
+    __real__free_r (reent, p);
+  else
+    user_data->_free_r (reent, p);
+}
+
+extern "C"
+void *
+export__malloc_r (_reent * reent, int size)
+{
+  void *res;
+  export_malloc_called = 1;
+  if (use_internal_malloc)
+    res = __real__malloc_r (reent, size);
+  else
+    res = user_data->_malloc_r (reent, size);
+  malloc_printf ("(%d) = %x, called by %x", size, res, ((int *)&size)[-1]);
+  return res;
+}
+
+extern "C"
+void *
+export__realloc_r (_reent * reent, void *p, int size)
+{
+  void *res;
+  if (use_internal_malloc)
+    res = __real__realloc_r (reent, p, size);
+  else
+    res = user_data->_realloc_r (reent, p, size);
+  malloc_printf ("(%x, %d) = %x, called by %x", p, size, res, ((int *)&p)[-1]);
+  return res;
+}
+
+extern "C"
+void *
+export__calloc_r (_reent * reent, size_t nmemb, size_t size)
+{
+  void *res;
+  if (use_internal_malloc)
+    res = __real__calloc_r (_impure_ptr, nmemb, size);
+  else
+    res = user_data->_calloc_r (reent, nmemb, size);
   malloc_printf ("(%d, %d) = %x, called by %x", nmemb, size, res, ((int *)&nmemb)[-1]);
   return res;
 }

************************* Patch winsup.h ***********************

--- back/winsup.h	Sun May 21 21:18:16 2000
+++ winsup.h	Sat Jul  8 14:47:48 2000
@@ -142,8 +142,16 @@ class per_process
   void *bss_end;
 
   void *(*calloc)(size_t, size_t);
+
+  /* Used to point to the memory machine we should use.  Usually these
+     point back into the dll, but they can be overridden by the user. */
+  void *(*_malloc_r)( _reent *, size_t);
+  void (*_free_r)( _reent *, void *);
+  void *(*_realloc_r)( _reent *, void *, size_t);
+  void *(*_calloc_r)( _reent *, size_t, size_t);
+
   /* For future expansion of values set by the app. */
-  void *public_reserved[4];
+  //void *public_reserved[4];
 
   /* The rest are *internal* to cygwin.dll.
      Those that are here because we want the child to inherit the value from


************************* Patch cygwin.din ***********************

--- back/cygwin.din	Sun May 14 12:20:24 2000
+++ cygwin.din	Sat Jul 22 01:25:54 2000
@@ -75,6 +75,8 @@ cabsf
 _cabsf = cabsf
 calloc = export_calloc
 _calloc = export_calloc
+_calloc_r = export__calloc_r
+__calloc_r = export__calloc_r
 cbrt
 _cbrt = cbrt
 cbrtf
@@ -269,6 +271,8 @@ fread
 _fread = fread
 free = export_free
 _free = export_free
+_free_r = export__free_r
+__free_r = export__free_r
 freopen
 _freopen = freopen
 frexp
@@ -471,6 +475,8 @@ lstat
 _lstat = lstat
 malloc = export_malloc
 _malloc = export_malloc
+_malloc_r = export__malloc_r
+_malloc = export__malloc_r
 matherr
 _matherr = matherr
 mblen
@@ -558,6 +564,8 @@ readv
 _readv = readv
 realloc = export_realloc
 _realloc = export_realloc
+_realloc_r = export__realloc_r
+__realloc_r = export__realloc_r
 regcomp
 _regcomp = regcomp
 regexec



*******************************************************


Ashif Harji.


--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com