[opensuse-packaging] libffmpeg-api
Project home:jnweiger has a libffmpeg-api that is supposed to allow linking against ffmpeg without having a real copy of it, so avoiding patent problems. But it doesn't works and doesn't seems mantained anymore. The package right now transform the Packman package to obtain a new package with all the headers and bogus libraries (from gcc -o empty.so --shared empty.c). So packages depending of it compile but fail at link. So, what else it needs? I don't know too much about the linker and ELF internals, I could use some help. An "objcopy -R .text <lib>" to libavcodec, libavdevice, etc. should remove all the codec code from them and from that point should not be problems in including it in the OBS, I'm wrong? Perhaps only libavcodec/format need it? I don't know if libavdevice has any codec/patented code. At this point it still doesn't works. First, a "nm -D" shows that doing this some symbols change from 'D' (initialized data section) to 'B' (BSS). Ok, makes sense... but that would mean a problem? I can't think of any. Something more confusing is a symbol (mm_flags from libavcodec) changing from 'B' to 'N' (debugging symbol)... I don't really understand why, but neither I know the effects. And then there is the problem of undefined symbols. libav* also depends of other libraries with codec code (libavcodec -> libx264). Best solution? a) 'objcopy -R .text' all the deps of ffmpeg. Simple, but "lot" of work. b) If I'm not wrong at link time the linker doesn't really needs them, or at least I don't know why it needs them. The final binary will be the same, true? So, there is some way to stop the linker from asking for it? Something like "--allow-shlib-undefined" that could be put in the .pc files? c) Something completly different. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Well, it seems it "works". At least I was able to compile aegisub
against the false libffmpeg-api and then run it with the real ffmpeg
from Packman.
The libffmpeg-api is created manually. I don't know perl to improve
the devel2api script.
2009/3/11 Cristian Morales Vega
At this point it still doesn't works. First, a "nm -D" shows that doing this some symbols change from 'D' (initialized data section) to 'B' (BSS). Ok, makes sense... but that would mean a problem? I can't think of any. Something more confusing is a symbol (mm_flags from libavcodec) changing from 'B' to 'N' (debugging symbol)... I don't really understand why, but neither I know the effects.
I would still like to know from someone with more knowledge of linker internal working. But seems to not matter? In the binary linked against ffmpeg the symbols are just marked like undefined whatever they are D or B in ffmpeg?
And then there is the problem of undefined symbols. libav* also depends of other libraries with codec code (libavcodec -> libx264). Best solution? a) 'objcopy -R .text' all the deps of ffmpeg. Simple, but "lot" of work. b) If I'm not wrong at link time the linker doesn't really needs them, or at least I don't know why it needs them. The final binary will be the same, true? So, there is some way to stop the linker from asking for it? Something like "--allow-shlib-undefined" that could be put in the .pc files? c) Something completly different.
There were not so much deps, so I just included "emptied" copies of the deps: 'a'. It would be easier if ffmpeg undefined symbols could just be marked like defined, but everything still is just 1MiB. The only problem I had with aegisub is that the configure script compiles and runs a test program to verify ffmpeg is correctly installed... since the ffmpeg copy isn't real the test segfaults. I "just" needed to patch configure, but I would be better if these checks could be disabled when using libffmpeg-api. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
2009/3/11 Cristian Morales Vega
I would still like to know from someone with more knowledge of linker internal working. But seems to not matter? In the binary linked against ffmpeg the symbols are just marked like undefined whatever they are D or B in ffmpeg?
After reading a little, I understand thing a little better. If I'm not wrong, a false library needs: - Since the linker search symbols by name the .dynsym and .dynstr sections are needed. But, does the value (st_value) needs to be the same? It's my understanding that different compilations will have different values, so this is only checked at load time and for libffmpeg-api purpose the value can be anything? I don't think st_size is needed. st_info must be correct, at least the binding info not sure about the type. st_other also must be correct since specifies the symbol visibility. st_shndx, the section index... since st_value isn't section relative I don't think this value matters. - .hash and .gnu.hash. Not sure if they are used only at load time or also at link time. If they are used .gnu.hash should be enough. - .gnu.version and .gnu.version_r sections... I suppose they are needed to avoid problems with versioned symbols. - Perhaps .note.gnu.build-id, .comment and/or .comment.SUSE.OPTs are used to do some kind of compatibility check? - I have no idea what the .jcr section contains. - The .dynamic section would be good to have to know where undefined symbols from the library should be search on. - So .rela.dyn, .rela.plt, .init, .plt, .text, .fini, .rodata, .eh_frame_hdr, .eh_frame, .ctors, .dtors, .data.rel.ro, .got, .got.plt, .data and .bss are just needed at load and runtime and aren't needed at all for libffmpeg-api. - The ELF header must be there and be correct. - The section header probably must be there. But doesn't needs to be equal to the original. It doesn't even needs to contains all the sections referenced by symbols st_shndx since st_shndx isn't important. Still, it would be "pretty" if "st_shndx"s would point to the correct section. - The program header isn't needed but specs say must be there for the file to be valid. But after all this... I'm asking myself why the library is needed to start with. For the final executable linked against ffmpeg libraries: - A DT_NEEDED will be added for every -lxxx in LDFLAGS with no check at all (not true if --as-needed is used). - Every symbol not defined in the executable, but in the library will be marked undefined in the executable .dynsym section. No additional info is stored in the executable. So, when the linker asks me for the libraries... it jut does checks? It could create the same executable without these libraries (at least when not linking statically). That would be good to know, if that's true I don't need to worry about the compatibility with Packman packages... if I'm able to create an executable I will be sure it will work (if the headers are the same). What I would do is "dd count=1 ibs=<X> obs=1 seek=<Y> conv=notrunc if=/dev/zero of=<library>" every library from Packman packages. 'X' would be the section size and 'Y' the offset obtained from "readelf -S" output. That would be done for every section that wants to be deleted. I would do the dd thing over a "strip -R <section>" since this way I don't touch the section header and so st_shndx would continue being correct. This would be good enough. Then what would make it perfect would be looking for functions through the symbol table, and from the symbol value add an "exit(0)" instruction to every function... so when a configure script wants to check ffmpeg is working it will not segfault. I would be unable to do the latter. I'm still a newbie with all this linking/ELF thing. Someone corrects me? -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hi, On Fri, 13 Mar 2009, Cristian Morales Vega wrote:
2009/3/11 Cristian Morales Vega
: I would still like to know from someone with more knowledge of linker internal working. But seems to not matter? In the binary linked against ffmpeg the symbols are just marked like undefined whatever they are D or B in ffmpeg?
After reading a little, I understand thing a little better.
I haven't investigated what the package you talk about actually does to create the fake library, but from what you say in this thread it seems that it constructs the fake library by taking the binary full binary, i.e. the ET_DYN .so file, and removing some sections. Is that correct, or is it only how you try to solve the problem? In any case, this would be _extremely_ fragile, and generally a terrible idea. The only way to sensibly generate such fake library is by parsing the original exported symbols from the .so file, generating a C source file from that info, implementing empty functions and defining data symbols (of the right size), of the right visibility and version info (the latter can be done by inline asm or by a linker version script). But just for entertainment purposes I answer the following questions:
If I'm not wrong, a false library needs: - Since the linker search symbols by name the .dynsym and .dynstr sections are needed. But, does the value (st_value) needs to be the same?
Yes, in case of exported constant symbols. It's not important for functions.
I don't think st_size is needed.
It is necessary for data objects (because it results in COPY relocations in the executable that have to have the correct size).
st_info must be correct, at least the binding info not sure about the type.
The type must be correct, ergo the whole st_info field.
st_other also must be correct since specifies the symbol visibility. st_shndx, the section index... since st_value isn't section relative I don't think this value matters.
It needs to point to a sensible section, though. It doesn't have to be the same as in the original library, but it has to be correct enough to make the whole fake library a "correct" ELF file. E.g. it's custom (not strictly necessary, but nice) that STT_FUNC symbols (that are defined) have a st_shndx of a code section.
- .hash and .gnu.hash. Not sure if they are used only at load time or also at link time. If they are used .gnu.hash should be enough.
Both would be better. They are used also at link time (for exactly the same purpose as at load time, speeding up symbol lookups).
- .gnu.version and .gnu.version_r sections... I suppose they are needed to avoid problems with versioned symbols.
Yes, both necessary.
- Perhaps .note.gnu.build-id, .comment and/or .comment.SUSE.OPTs are used to do some kind of compatibility check?
They have to be in installed libraries (by SuSE policy), but aren't needed at link time.
- I have no idea what the .jcr section contains.
java class registration, usually just an empty section (containing only a null terminator), only needed at run time.
- The .dynamic section would be good to have to know where undefined symbols from the library should be search on.
It also contains the shared lib name, pointers to various tables and assorted vital info about the shared lib itself. It's required at load and link time.
- So .rela.dyn, .rela.plt, .init, .plt, .text, .fini, .rodata, .eh_frame_hdr, .eh_frame, .ctors, .dtors, .data.rel.ro, .got, .got.plt, .data and .bss are just needed at load and runtime and aren't needed at all for libffmpeg-api.
Correct.
- The ELF header must be there and be correct.
Yes.
- The section header probably must be there. But doesn't needs to be equal to the original. It doesn't even needs to contains all the sections referenced by symbols st_shndx since st_shndx isn't important. Still, it would be "pretty" if "st_shndx"s would point to the correct section.
Right.
- The program header isn't needed but specs say must be there for the file to be valid.
Right again (I'm also rather sure that the linker simply would refuse to link against a library not containing the program headers).
But after all this... I'm asking myself why the library is needed to start with. For the final executable linked against ffmpeg libraries: - A DT_NEEDED will be added for every -lxxx in LDFLAGS with no check at all (not true if --as-needed is used). - Every symbol not defined in the executable, but in the library will be marked undefined in the executable .dynsym section. No additional info is stored in the executable.
Except for COPY relocs and version information. I.e. you really do want to have a real library to link against (not to mention that the link editor bitches if it doesn't find a libbla.so for a -lbla argument).
What I would do is "dd count=1 ibs=<X> obs=1 seek=<Y> conv=notrunc if=/dev/zero of=<library>" every library from Packman packages. 'X' would be the section size and 'Y' the offset obtained from "readelf -S" output. That would be done for every section that wants to be deleted. I would do the dd thing over a "strip -R <section>" since this way I don't touch the section header and so st_shndx would continue being correct.
Uhh. This all sounds horrible. You rather want to go with the very first hint from my mail. Generate a C source from the symbols and compile that into a library. Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
2009/3/16 Michael Matz
Hi,
On Fri, 13 Mar 2009, Cristian Morales Vega wrote:
2009/3/11 Cristian Morales Vega
: I would still like to know from someone with more knowledge of linker internal working. But seems to not matter? In the binary linked against ffmpeg the symbols are just marked like undefined whatever they are D or B in ffmpeg?
After reading a little, I understand thing a little better.
I haven't investigated what the package you talk about actually does to create the fake library, but from what you say in this thread it seems that it constructs the fake library by taking the binary full binary, i.e. the ET_DYN .so file, and removing some sections. Is that correct, or is it only how you try to solve the problem?
The original libffmpeg-api packages just creates a library from an empty file doing "gcc -o empty.so --shared empty.c"... it doesn't works. The package was never completed to a point where it would work. The section removing idea was my attempt to make it work.
In any case, this would be _extremely_ fragile, and generally a terrible idea. The only way to sensibly generate such fake library is by parsing the original exported symbols from the .so file, generating a C source file from that info, implementing empty functions and defining data symbols (of the right size), of the right visibility and version info (the latter can be done by inline asm or by a linker version script).
Something for this summer. Too much work for me right now, I don't even know asm or relocation types (I just looked to know what a COPY reloc was). Still, a good way to learn all the internals. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Michael Matz wrote:
[...] In any case, this would be _extremely_ fragile, and generally a terrible idea. The only way to sensibly generate such fake library is by parsing the original exported symbols from the .so file, generating a C source file from that info, implementing empty functions and defining data symbols (of the right size), of the right visibility and version info (the latter can be done by inline asm or by a linker version script).
This script basically uses that technique to create filter objects: http://xqf.svn.sourceforge.net/viewvc/xqf/trunk/xqf/src/gensyms.pl?revision=... It only handles functions. No data. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
2009/3/17 Ludwig Nussel: > Michael Matz wrote: >> [...] >> In any case, this would be _extremely_ fragile, and generally a terrible >> idea. The only way to sensibly generate such fake library is by parsing >> the original exported symbols from the .so file, generating a C source >> file from that info, implementing empty functions and defining data >> symbols (of the right size), of the right visibility and version info (the >> latter can be done by inline asm or by a linker version script). > > This script basically uses that technique to create filter objects: > http://xqf.svn.sourceforge.net/viewvc/xqf/trunk/xqf/src/gensyms.pl?revision=HEAD > It only handles functions. No data. >From the "I will need to learn perl" guy: http://rafb.net/p/5yePWB20.html That should add objects, but: - I don't know perl - There can be weak symbols that aren't functions? Anyway, I didn't though about weak symbols. - I don't know perl - secton number -> sections name is hardcoded. Something to improve. - I don't know perl - Symbol visibility is ignored, but libavcodec seems to use default visibility for everything. - I don't know perl - I don't know how that "__asm__(\".symver $vsym2,$sym$at$ver\");" thing works... it's the same for data than for functions? - I don't know perl - I'm not very sure about why the "$seen" variable is there. Could be two symbols for the same function?? Anyway, I suppose an OBJECT symbol can have the same name than a FUNC symbol, so even if a symbol has already been $seen perhaps shouldn't be ignored? - I don't know perl -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
2009/3/17 Cristian Morales Vega
2009/3/17 Ludwig Nussel
: Michael Matz wrote:
[...] In any case, this would be _extremely_ fragile, and generally a terrible idea. The only way to sensibly generate such fake library is by parsing the original exported symbols from the .so file, generating a C source file from that info, implementing empty functions and defining data symbols (of the right size), of the right visibility and version info (the latter can be done by inline asm or by a linker version script).
This script basically uses that technique to create filter objects: http://xqf.svn.sourceforge.net/viewvc/xqf/trunk/xqf/src/gensyms.pl?revision=... It only handles functions. No data.
From the "I will need to learn perl" guy: http://rafb.net/p/5yePWB20.html That should add objects, but: - I don't know perl - There can be weak symbols that aren't functions? Anyway, I didn't though about weak symbols. - I don't know perl - secton number -> sections name is hardcoded. Something to improve. - I don't know perl - Symbol visibility is ignored, but libavcodec seems to use default visibility for everything. - I don't know perl - I don't know how that "__asm__(\".symver $vsym2,$sym$at$ver\");" thing works... it's the same for data than for functions? - I don't know perl - I'm not very sure about why the "$seen" variable is there. Could be two symbols for the same function?? Anyway, I suppose an OBJECT symbol can have the same name than a FUNC symbol, so even if a symbol has already been $seen perhaps shouldn't be ignored? - I don't know perl
After modifying it so it doesn't hardcodes section numbers, checks the symbol visibility, etc. I have modified home:RedDwarf:fakePackman to use it. And, without too much tests, it seems to work ok. And now I don't need to patch aegisub configure script. The two scripts are available at https://build.opensuse.org/package/show?package=ffmpeg&project=home:RedDwarf:fakePackman. Any comment is welcome. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hi, On Tue, 17 Mar 2009, Ludwig Nussel wrote:
In any case, this would be _extremely_ fragile, and generally a terrible idea. The only way to sensibly generate such fake library is by parsing the original exported symbols from the .so file, generating a C source file from that info, implementing empty functions and defining data symbols (of the right size), of the right visibility and version info (the latter can be done by inline asm or by a linker version script).
This script basically uses that technique to create filter objects: http://xqf.svn.sourceforge.net/viewvc/xqf/trunk/xqf/src/gensyms.pl?revision=... It only handles functions. No data.
Cool. Great minds think alike (and even greater ones actually then do something) :) Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Mar 13, 09 20:48:36 +0100, Cristian Morales Vega wrote:
But after all this... I'm asking myself why the library is needed to start with. For the final executable linked against ffmpeg libraries: - A DT_NEEDED will be added for every -lxxx in LDFLAGS with no check at all (not true if --as-needed is used).
I am not familiar with DT_NEEDED and --as-needed
So, when the linker asks me for the libraries... it jut does checks?
Yes, that is my understanding. At build time, the library is examined, to verify that all symbols are present and of correct type and version. For dynamic linking, nothing else is done. For static linking, code is copied from the library into the executable.
It could create the same executable without these libraries (at least when not linking statically). That would be good to know, if that's true I don't need to worry about the compatibility with Packman packages... if I'm able to create an executable I will be sure it will work (if the headers are the same).
The only reason for having libffmpeg-api is to allow these checks to succeed without having the actual code around. cheers, JW- -- o \ Juergen Weigert paint it green! __/ _=======.=======_ <V> | jw@suse.de back to ascii! __/ _---|____________\/ \ | 0911 74053-508 __/ (____/ /\ (/) | _____________________________/ _/ \_ vim:set sw=2 wm=8 SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Juergen Weigert escribió:
I am not familiar with DT_NEEDED and --as-needed
See http://www.gentoo.org/proj/en/qa/asneeded.xml hopefully someday becoming the default... -- "If this is the best God can do, I am not impressed" -George Carlin (1937-2008) Cristian Rodríguez R. Software Developer Platform/OpenSUSE - Core Services SUSE LINUX Products GmbH Research & Development http://www.opensuse.org/
2009/3/16 Cristian Rodríguez
Juergen Weigert escribió:
I am not familiar with DT_NEEDED and --as-needed
See http://www.gentoo.org/proj/en/qa/asneeded.xml hopefully someday becoming the default...
While looking at all this I found that there is a new check done in post-2.19 binutils that should avoid a lot of problems: http://sourceware.org/cgi-bin/cvsweb.cgi/src/ld/NEWS?rev=1.100&content-type=text/x-cvsweb-markup&cvsroot=src * --as-needed now links in a dynamic library if it satisfies undefined symbols in regular objects, or in other dynamic libraries. In the latter case the library is not linked if it is found in a DT_NEEDED entry of one of the libraries already linked. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Cristian Morales Vega escribió:
2009/3/16 Cristian Rodríguez
: Juergen Weigert escribió:
I am not familiar with DT_NEEDED and --as-needed See http://www.gentoo.org/proj/en/qa/asneeded.xml hopefully someday becoming the default...
While looking at all this I found that there is a new check done post-2.19 binutils that should avoid a lot of problems:
yes, this change is known, commented in bug #362947, the only way to really test it is doing a full rebuild with gcc calling ld with --as-needed by default, if something fails, it should be either fixed or linked with --no-as-needed in a case by case basis (IMHO) -- "If this is the best God can do, I am not impressed" -George Carlin (1937-2008) Cristian Rodríguez R. Software Developer Platform/OpenSUSE - Core Services SUSE LINUX Products GmbH Research & Development http://www.opensuse.org/
Hi, On Mon, 16 Mar 2009, Juergen Weigert wrote:
I am not familiar with DT_NEEDED and --as-needed
So, when the linker asks me for the libraries... it jut does checks?
Yes, that is my understanding. At build time, the library is examined, to verify that all symbols are present and of correct type and version. For dynamic linking, nothing else is done.
This is incorrect, as explained in my last mail. There are at least copy relocations (which copy the symbol size from the lib) and symbol versions (which copy the versions of referred to symbols to the executable). So it's not only checking but really data copying taking place. Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
participants (5)
-
Cristian Morales Vega
-
Cristian Rodríguez
-
Juergen Weigert
-
Ludwig Nussel
-
Michael Matz