Skip to Content.
Sympa Menu

sm-discuss - [SM-Discuss] large file support on SMGL, or just generally in libraries

sm-discuss AT lists.ibiblio.org

Subject: Public SourceMage Discussion List

List archive

Chronological Thread  
  • From: Thomas Orgis <thomas-forum AT orgis.org>
  • To: sm-discuss AT lists.ibiblio.org
  • Subject: [SM-Discuss] large file support on SMGL, or just generally in libraries
  • Date: Sat, 6 Mar 2010 14:31:06 +0100

Hi, folks.

As some of you may know, I am responsible for a library that happens to deal
with files & offsets: libmpg123 (part of the mpg123 spell). At that time
being rather unaware of the whole extend of the related mess, I initially
chose to use off_t in my interfaces, functions like

off_t mpg123_tell(mpg123_handle *mh);
off_t mpg123_seek(mpg123_handle *mh, off_t sampleoff, int whence);

with rather obvious meaning... now that decision totally blows when
considering the broken design of large file support on Linux/Solaris systems:
You can make off_t 64 bit at build time, replacing the I/O calls your program
uses, but the default is 32 bit. But then again, if you build something using
autoconf, the default will likely be 64 bit off_t because there's a macro
checking that option.

This results in a happy mix of code that uses off_t with 32 bit and with 64
bit; some code at the same time, which not good, generally ... see
http://ac-archive.sourceforge.net/largefile/ for an account on these issues.
Read the site in whole and be scared.

This site is rather old, but issues still remain present. As an example, I
ran the test script from that site on a 32 bit chroot (barely enough stuff in
there to get wine running... for starcraft;-):

bash$ perl off_t_problem.pl --smart --silent /opt/smgl32/lib/*.so
/opt/smgl32/usr/lib/*.so
ignored: (0xffffe000)
objdump: /opt/smgl32/usr/lib/libpthread.so: File format not recognized
objdump: /opt/smgl32/usr/lib/libc.so: File format not recognized
Badlink: /opt/smgl32/lib/libhistory.so -32->>-64-
/opt/smgl32/lib/libncurses.so.5
Badlink: /opt/smgl32/lib/libreadline.so -32->>-64-
/opt/smgl32/lib/libncurses.so.5
Badlink: /opt/smgl32/usr/lib/libGLU.so -32->>-64-
/opt/smgl32/usr/lib/libstdc++.so.6
Badlink: /opt/smgl32/usr/lib/libXaw.so -32->>-64-
/opt/smgl32/lib/libuuid.so.1
Badlink: /opt/smgl32/usr/lib/libXaw6.so -32->>-64-
/opt/smgl32/lib/libuuid.so.1
Badlink: /opt/smgl32/usr/lib/libXaw7.so -32->>-64-
/opt/smgl32/lib/libuuid.so.1
Badlink: /opt/smgl32/usr/lib/libXmu.so -32->>-64-
/opt/smgl32/lib/libuuid.so.1
Badlink: /opt/smgl32/usr/lib/libXt.so -32->>-64-
/opt/smgl32/lib/libuuid.so.1
Badlink: /opt/smgl32/usr/lib/libext2fs.so 3264>>-32-
/opt/smgl32/lib/libcom_err.so.2
Badlink: /opt/smgl32/usr/lib/libfontforge.so 3264>>-32-
/opt/smgl32/usr/lib/libgutils.so.1
Badlink: /opt/smgl32/usr/lib/libgdraw.so -32->>-64-
/opt/smgl32/usr/lib/libpython2.6.so.1.0
Badlink: /opt/smgl32/usr/lib/libgdraw.so -32->>-64-
/opt/smgl32/lib/libuuid.so.1
Badlink: /opt/smgl32/usr/lib/libgif.so -32->>-64-
/opt/smgl32/lib/libuuid.so.1
Badlink: /opt/smgl32/usr/lib/libgutils.so -32->>-64-
/opt/smgl32/usr/lib/libpython2.6.so.1.0
Badlink: /opt/smgl32/usr/lib/libxml2.so -64->>-32-
/opt/smgl32/lib/libz.so.1
summary: found 15 badlinks to be checked closer (0 are clean)
summary: check symbols with --nonclean or even --noncleanall

There are some libraries in that list that are not actually rare in use... if
the particular library really wrecks havoc with the mismatch is an open
question, but what one clearly sees is that a considerable amount of
libraries on a 2009/2010 Source Mage system is built without large file
support.
Running the test script on my 64 bit (meaning: x86-64) main system shows many
possible mismatches, too, but they are not relevant in practice since open64
is really identical to open, as there is no 32 bit file offset variant.

Well, there still seems to be major lack of consensus about applications
being aware of large files, and that sucks big time. Realistically, only the
change to x86-64 systems might drive the point home at last... Now, I wonder,
if we at SMGL should be paying attention and make sure that all our apps/libs
are built with large file support. Much code that is large file aware is
already built with 64 bit offsets as autoconf macros deal with that.
But there is the prominent example of zlib, code which can support large
files, but needs specific action in the build script to enable it. Should we
go with upstream default there or rather configure the build for large files?
What would remove the problematic usage by libxml2.so from the above list,
but might add other entries for client code that relies on zlib using 32 bit
offsets (note: I am only talking about libraries here, but the real damage
occurs in client apps that can have their own direct bad linking).

I am asking SMGL folks if we as a distribution want to care about that
mess... try to make our builds more consistent. Or, did we already have a
policy on that?

Coming back to my specific concern, libmpg123, I am finally asking on what
you expect a library to do on this issue. The situation with libmpg123 is
even that:
Large file support has been added late in version 1.5 . If support is enabled
(on 32 bit Linux), concerned function symbols are renamed to include a suffix
_64 to prevent bad usage by client code not being aware of 64 bit offsets.
Now this created a nasty incompatibility with old binaries: An update of
libmpg123 will not offer mpg123_seek() anymore, just mpg123_seek_64(), unless
the packager explicitly disables large file support via configure.
This would have been a practical problem for SMGL if there would be more
stuff using old libmpg123, but the issue as such remains: The ABI is
dependent on configure switches. I now see that this is a bad thing. Really
bad.

A "proper" but very work-intensive solution would be to make the library
multi-mode... offering 32 bit and 64 bit offset code automacically, just like
glibc does. But, can Source Mage expect that all libraries go through that
chore and don't mess it up at that, too? Reality shows that we cannot even
depend on libraries being aware of large file support at all.
So, what would you, as Source Mage gurus

a) expect from the author of the library (me;-) to do to help the messy
situation
b) try to do to fix up the situation for Source Mage, generally
c) ...

?

There is a chance that one of you lot spent some time thinking about large
file support on Linux and comes up with the ultimate solution, right?


Alrighty then,

Thomas.

Attachment: signature.asc
Description: PGP signature




Archive powered by MHonArc 2.6.24.

Top of Page