On Feb 19, 2008 8:01 AM, Petr Cerny wrote:

Greg Freemyer wrote:

...

All,

I have a huge text file (1.7 million lines) full of unicode and ascii text (half and half).

note: disk space for copies is not a problem, if I need to manipulate this file

Also, I have a 30 line file full of ascii text.

I need to search the large file for any occurrences of the keywords in the 30 line file.

Ignoring the unicode issue, I could use grep (of fgrep, egrep) with appropriate args.

I have no idea how to handle the unicode issue.

ASCII is a subset of UTF-8, so if your Unicode coding is UTF-8 there should be no problem. In other case I wonder how did you manage to create file with interspersed single and (fixed size) multibyte encodings... I guess you would have to split the file somehow.

Best regards Petr

You can see I'm a bit confused. I have not worked much with Unicode, and not at all on Linux. To me the Unicode file looks like normal ascii with a null between every byte. I think thats normal from my limited exposure to Unicode. Status: I'm not at the office, but they recreated the 1.7 million line file last night as pure Unicode. I can't test it for a couple hours, but is the claim: I can use grep to search a Unicode file for a ascii text pattern? Yesterday I looked for a grep switch that would tell it the file being searched was Unicode but I didn't see one. I guess it could be automatic? I did not actually attempt it. If that is not feasible, a tool to convert that Unicode file into a ascii file would be great. All the text I am interested in should convert to ascii in the obvious way. No need for any mappings. Basically, just need to get rid of the nulls. Greg -- Greg Freemyer Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer First 99 Days Litigation White Paper - http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Tuesday 19 February 2008 20:27, Petr Cerny wrote:

iconv is your friend. Basically you would need to do something like: 'iconv -f some_unicode_encoding -t ascii < unicode_file > ascii_file' This will also choke on any non-ascii characters (unless you make it translate unknown characters with  "//TRANSLIT" appended to output encoding).

For lots of files, I'd rather use "recode" with can operate on multiple files in-place without the need for a temp file for each one: recode unicode..utf8 *.txt CU -- Stefan Hundhammer Penguin by conviction. YaST2 Development SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) Nürnberg, Germany -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org