Organization and symbol lookup for shared libraries

Wed Nov 10 15:30:00 GMT 2004

Andrew Haley wrote:
>Bryce McKinlay writes:
> > Andrew Haley wrote:
> > 
> > >James Damour writes:
> > > > Bryce McKinlay writes:
> > > > > Perhaps this step (running gcj-dbtool) could be avoided altogether: if 
> > > > > the cache is found to be out-of-date at runtime, because someone added 
> > > > > or changed a shared library without re-running dbtool, then we'll need 
> > > > > to open those modified shared libraries at runtime anyway. Since we have 
> > > > > to open them, the runtime could simply update the cache file on-the-fly?
> > > > 
> > > > The problem with that is permissions.
> >
> > What should happen in this situation is for libgcj to fall back to 
> > loading the shared libraries without the cache, and print a warning that 
> > the cache file needs to be updated by running gcj-dbtool.
>>libgcj would need to take a list of DSOs from the cache and open every
>directory that contained one in order to check the date stamp of a
>DSO. But the whole idea of the cache is to load precompiled classes
>without any additional overhead for scanning directories, etc. We
>presumably don't want to be in a situation where precompiled Java apps
>have a startup time that is proportional to the number of libraries
>that are installed.
>>
The overhead of checking date stamps on even 100 DSOs is probably 
smaller than actually loading even one of them, so I don't think thats a 
significant issue. Besides, a) if we are to load any DSOs from that 
directory, we'll (or the kernel, at least) will need to read the 
directory later anyway; and b) it might not be necessary to actually 
read all the possible directorys, only the ones that classes are 
actually loaded from, or to search for new libraries in the event that a 
class is requested that does not exist in the cache.
Its also important to remember that all of this only happens if an 
attempt is made to resolve a class that wasn't explicitly linked. "hello 
world", nor any app that only uses libgcj or other libraries that are 
explicitly linked with "-l", will not incur any of this overhead.
I strongly believe that user convenience must be the overriding goal 
here. Even if that were to mean performance is a little slower, I think 
the trade-off is worthwhile. If we aren't able to detect and warn about 
out-of-date caches, there is just too much room for confusion and 
weirdness if the user forgets to update a cache. Plus, its a big 
disadvantage compared to traditional VMs if manually updating the cache 
is required - users don't need to run special commands to deploy a .jar, 
they don't want to deal with extra steps for an .so.
> > One thing we do need to consider carefully, though, is security - we 
> > don't want a suid root gcj application blindly opening shared libraries 
> > from user-writable directories!
> > 
> > >Also, there's a potential issue with shared access. I'm not
> > >sure it's wothwhile.
> > > 
> > >
> > I don't see what the problem is with sharing. Surely, provided the
> > cache file is updated atomically, it will never be seen in an
> > inconsistent state?
>>The devil is in the details. Once you'd started to update the cache
>you'd want to finish before anything else starts. There may be
>several instances of gcj running, so somehow you'd have to lock them
>out while doing the update. MAP_PRIVATE wouldn't help, because writes
>to the region don't affect the underlying file. So, you'd need to
>create a new cache file, copy everything from the old cache into it,
>and rename() the new file to the cache. And you would want to be sure
>you were the only process doing that.
>>
Right, if an update is needed, we'd create a new cache file and rename() 
it once its complete. It doesn't matter if two processes end up doing 
the work simultaneously - rename() is atomic.
>There may well be something useful we can do in this area, but I think
>we ought to get some experience first.
>>
Agreed...
Bryce