RE: [SLE] editor for extremely large test file
On Wednesday 11 May 2005 00:06, James D. Parra wrote:
Hello,
Anyone know of an editor that can open a 39 GB text file? I need to locate and copy out a large block of text from this file, about 8 million rows, at roughly 200 MB of data.
sed
Thanks Anders. However, I don't know where the text I need is located in the file. I need an editor that can actually open this file so I can look for the text I want. Thanks again, ~James
On Tue, 2005-05-10 at 15:30 -0700, James D. Parra wrote:
On Wednesday 11 May 2005 00:06, James D. Parra wrote:
Hello,
Anyone know of an editor that can open a 39 GB text file? I need to locate and copy out a large block of text from this file, about 8 million rows, at roughly 200 MB of data.
sed
Thanks Anders. However, I don't know where the text I need is located in the file. I need an editor that can actually open this file so I can look for the text I want.
You can also use less to look at the file and it will show line numbers. Again because of the size it will take a very long time to open. If you know some of the text you can search with / followed with the search string. You can also use grep -n to search for any lines that have text you are looking for, this will show the line number of the line in the file. -- Ken Schneider UNIX since 1989, linux since 1994, SuSE since 1998 "The day Microsoft makes something that doesn't suck is probably the day they start making vacuum cleaners." -Ernst Jan Plugge
Hi, On Tuesday 10 May 2005 16:04, Ken Schneider wrote:
...
You can also use less to look at the file and it will show line numbers. Again because of the size it will take a very long time to open. If you know some of the text you can search with / followed with the search string.
Less wants to pre-compute line numbers, so it will pre-scan the entire file at start-up. If you are going to use Vim to extract portions of very large files, you'll do well to use the "-r" option or to use the synonymous alias "view". If Vim won't be editing the file, it does not need to do as much pre-processing and doesn't need to build all the structures needed to edit the file in-place. Nonetheless, I doubt it's going to be able to open a 39 GB file.
You can also use grep -n to search for any lines that have text you are looking for, this will show the line number of the line in the file.
-- Ken Schneider
Randall Schulz
James D. Parra wrote:
Thanks Anders. However, I don't know where the text I need is located in the file. I need an editor that can actually open this file so I can look for I am confused. When I was a kid my dad used to say "if you don't know what you are looking for, how will you know when you find it?" It seems you must know something about what you are looking for so why can't the Anders recommended SED find that text? Putting the size issue aside for a moment, why can't any editor find your text? Even the simplest Notepad in Windows can search.
Now I have a question about editors and big files. Does anyone remember the PDP-8? Twenty years ago when I started work here, there were a lot of PDP-8 word processors. They had a big 4K of RAM. When they opened a file, they didn't load the whole thing, just the first piece. As the user moved through the file, the computer would load the next piece. Does SED or any other editor have this abililty to load only a portion at a time? I could be wrong but it seems to me that this might be the only way to deal with such a large file. My favorite editor is Nedit from http://www.nedit.org/. I don't know how it is with the big files but in general I have found it to be a good program that seems to run efficiently. Damon Register
On Wednesday 11 May 2005 20:46, Damon Register wrote:
James D. Parra wrote:
Thanks Anders. However, I don't know where the text I need is located in the file. I need an editor that can actually open this file so I can look for
I am confused. When I was a kid my dad used to say "if you don't know what you are looking for, how will you know when you find it?" It seems you must know something about what you are looking for so why can't the Anders recommended SED find that text? Putting the size issue aside for a moment, why can't any editor find your text? Even the simplest Notepad in Windows can search.
Now I have a question about editors and big files. Does anyone remember the PDP-8? Twenty years ago when I started work here, there were a lot of PDP-8 word processors. They had a big 4K of RAM. When they opened a file, they didn't load the whole thing, just the first piece. As the user moved through the file, the computer would load the next piece. Does SED or any other editor have this abililty to load only a portion at a time? I could be wrong but it seems to me that this might be the only way to deal with such a large file. Actually, doesn't emacs do that? My favorite editor is Nedit from http://www.nedit.org/. I don't know how it is with the big files but in general I have found it to be a good program that seems to run efficiently.
Damon Register
On 5/11/05, Colin Carter
On Wednesday 11 May 2005 20:46, Damon Register wrote:
James D. Parra wrote:
Thanks Anders. However, I don't know where the text I need is located in the file. I need an editor that can actually open this file so I can look for
I am confused. When I was a kid my dad used to say "if you don't know what you are looking for, how will you know when you find it?" It seems you must know something about what you are looking for so why can't the Anders recommended SED find that text? Putting the size issue aside for a moment, why can't any editor find your text? Even the simplest Notepad in Windows can search.
Now I have a question about editors and big files. Does anyone remember the PDP-8? Twenty years ago when I started work here, there were a lot of PDP-8 word processors. They had a big 4K of RAM. When they opened a file, they didn't load the whole thing, just the first piece. As the user moved through the file, the computer would load the next piece. Does SED or any other editor have this abililty to load only a portion at a time? I could be wrong but it seems to me that this might be the only way to deal with such a large file. Actually, doesn't emacs do that?
IIRC, sed stands for "stream editor", It is non-interactive. It is typically used from shell scripts, or by gurus that have memorized its rather complex command-line argument structure. You can even put sed in a pipeline: cat bigfile | sed AndersArgs > resulting_sub_section So yes it works on files a little bit at a time. That is why Anders recommended it I'm sure. Greg -- Greg Freemyer The Norcross Group Forensics for the 21st Century
participants (6)
-
Colin Carter -
Damon Register -
Greg Freemyer -
James D. Parra -
Ken Schneider -
Randall R Schulz