[opensuse] 13.1 Bash HISTCONTROL=erasedups -- not working? confirm?
All, A value of HISTCONTROL=erasedups causes all previous lines matching the current line to be removed from the history list before that line is saved. (man bash) This isn't working on 13.1 for me. Can anyone else confirm that duplicates are NOT removed in 13.1 when this value is set? Searching, it looks like other distros have patched bash to insure this works when multiple terminals are open. (e.g.: https://bugs.launchpad.net/ubuntu/+source/bash/+bug/189881) If this is confirmed, could something similar be done here? The reason, after moving my history file from 11.4 to 13.1 and running for close to a year, I have close to 14,000 duplicate lines -- despite having 'erasedups' set: current lines: 20000 - uniq lines: 6099 = 13901 duplicates To fix the problem temporarily, I wrote the current file out, then sorted uniquely, then deleted the current history, read the sorted unique file in as current, and wrote the current back to HISTFILE: $ history -w | sort | uniq > uniqfile $ history -c $ history -r uniqfile $ history -w $ rm uniqfile That seems to have fixed the issue in the current session, and ~/.bash_history is current correct, but when closing the remaining sessions, I suspect the duplicates will once again be written to ~/.bash_history, even though 'erasedups' is set. If someone can confirm 13.1 is ignoring 'erasedups', I'll file the bug. -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sat, 28 Mar 2015 21:50, David C. Rankin wrote:
All,
A value of HISTCONTROL=erasedups causes all previous lines matching the current line to be removed from the history list before that line is saved. (man bash)
This isn't working on 13.1 for me. Can anyone else confirm that duplicates are NOT removed in 13.1 when this value is set? Searching, it looks like other distros have patched bash to insure this works when multiple terminals are open. (e.g.: https://bugs.launchpad.net/ubuntu/+source/bash/+bug/189881)
If this is confirmed, could something similar be done here? The reason, after moving my history file from 11.4 to 13.1 and running for close to a year, I have close to 14,000 duplicate lines -- despite having 'erasedups' set:
current lines: 20000 - uniq lines: 6099 = 13901 duplicates
To fix the problem temporarily, I wrote the current file out, then sorted uniquely, then deleted the current history, read the sorted unique file in as current, and wrote the current back to HISTFILE:
$ history -w | sort | uniq > uniqfile $ history -c $ history -r uniqfile $ history -w $ rm uniqfile
That seems to have fixed the issue in the current session, and ~/.bash_history is current correct, but when closing the remaining sessions, I suspect the duplicates will once again be written to ~/.bash_history, even though 'erasedups' is set.
If someone can confirm 13.1 is ignoring 'erasedups', I'll file the bug.
Confirmed for openSUSE 12.3 and for 13.1, and it can be confirmed with and without 'timestamps' for the history ($HISTTIMEFORMAT). - Yamaban. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 03/28/2015 04:03 PM, Yamaban wrote:
If someone can confirm 13.1 is ignoring 'erasedups', I'll file the bug.
Confirmed for openSUSE 12.3 and for 13.1, and it can be confirmed with and without 'timestamps' for the history ($HISTTIMEFORMAT).
Thank you, Bug filed: https://bugzilla.opensuse.org/show_bug.cgi?id=924818 -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hello, On Sat, 28 Mar 2015, David C. Rankin wrote:
On 03/28/2015 04:03 PM, Yamaban wrote:
If someone can confirm 13.1 is ignoring 'erasedups', I'll file the bug.
Confirmed for openSUSE 12.3 and for 13.1, and it can be confirmed with and without 'timestamps' for the history ($HISTTIMEFORMAT).
Thank you,
Bug filed:
That is all weird, and, mind you, I use the same bash as you guys, even though I'm on 12.1 ;) $ rpm -q bash bash-4.3-275.1.x86_64 $ rpm -q --changelog bash | head -2 * Wed Mar 04 2015 jdelvare@suse.de - bash-4.3-loadables.dif: One more warning fixed, in So, I guess it's in libhistory / libreadline, but I think I use the same as well: $ rpm -qf /lib64/libhistory.so.6.3 libreadline6-6.3-275.1.x86_64 $ rpm -q --changelog libreadline6 | head -2 * Wed Mar 04 2015 jdelvare@suse.de - bash-4.3-loadables.dif: One more warning fixed, in I just tried a few "history -w", "ls foo", "history -w" from different xterms, and it all comes out as just one single entry each (one "ls" and one "history -w"). $ env | grep HIST HISTSIZE=50000 HISTFILESIZE=500000 HISTIGNORE=*PROMPT_COMMAND=* HISTCONTROL=ignoreboth:erasedups ==== man bash ==== HISTCONTROL A colon-separated list of values controlling how commands are saved on the history list. If the list of values includes ignorespace, lines which begin with a space character are not saved in the history list. A value of ignoredups causes lines matching the previous history entry to not be saved. A value of ignoreboth is shorthand for ignorespace and ignoredups. A value of erasedups causes all previous lines matching the current line to be removed from the history list before that line is saved. ==== I suspect you might be missing the 'ignoredups' value to HISTCONTROL ;) $ wc -l ~/.bash_history 49410 /home/dh/.bash_history Oh, my, gotta save that again to ".history.big" ... starting xterms etc. has become a tad slowish too (esp. if you start 10 at once as I do :) $ cat ~/.bash_history >> ~/.bash_history.big $ mv -i ~/.bash_history ~/.bash_history.t $ tail -20000 ~/.bash_history.t > ~/.bash_history $ rm ~/.bash_history.t $ wc -l ~/.bash_history.big 587130 ~/.bash_history.big HTH, -dnh -- By golly, I'm beginning to think Linux really *is* the best thing since sliced bread. -- Vance Petree, Virginia Power -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, 30 Mar 2015 18:52, David Haller wrote:
On Sat, 28 Mar 2015, David C. Rankin wrote:
On 03/28/2015 04:03 PM, Yamaban wrote:
If someone can confirm 13.1 is ignoring 'erasedups', I'll file the bug.
Confirmed for openSUSE 12.3 and for 13.1, and it can be confirmed with and without 'timestamps' for the history ($HISTTIMEFORMAT).
Thank you,
Bug filed:
That is all weird, and, mind you, I use the same bash as you guys, even though I'm on 12.1 ;)
$ rpm -q bash bash-4.3-275.1.x86_64 $ rpm -q --changelog bash | head -2 * Wed Mar 04 2015 jdelvare@suse.de - bash-4.3-loadables.dif: One more warning fixed, in
So, I guess it's in libhistory / libreadline, but I think I use the same as well:
$ rpm -qf /lib64/libhistory.so.6.3 libreadline6-6.3-275.1.x86_64 $ rpm -q --changelog libreadline6 | head -2 * Wed Mar 04 2015 jdelvare@suse.de - bash-4.3-loadables.dif: One more warning fixed, in
I just tried a few "history -w", "ls foo", "history -w" from different xterms, and it all comes out as just one single entry each (one "ls" and one "history -w").
$ env | grep HIST HISTSIZE=50000 HISTFILESIZE=500000 HISTIGNORE=*PROMPT_COMMAND=* HISTCONTROL=ignoreboth:erasedups
==== man bash ==== HISTCONTROL A colon-separated list of values controlling how commands are saved on the history list. If the list of values includes ignorespace, lines which begin with a space character are not saved in the history list. A value of ignoredups causes lines matching the previous history entry to not be saved. A value of ignoreboth is shorthand for ignorespace and ignoredups. A value of erasedups causes all previous lines matching the current line to be removed from the history list before that line is saved. ====
I suspect you might be missing the 'ignoredups' value to HISTCONTROL ;)
$ wc -l ~/.bash_history 49410 /home/dh/.bash_history
Oh, my, gotta save that again to ".history.big" ... starting xterms etc. has become a tad slowish too (esp. if you start 10 at once as I do :)
$ cat ~/.bash_history >> ~/.bash_history.big $ mv -i ~/.bash_history ~/.bash_history.t $ tail -20000 ~/.bash_history.t > ~/.bash_history $ rm ~/.bash_history.t $ wc -l ~/.bash_history.big 587130 ~/.bash_history.big
HTH, -dnh
I'm using urxvt(d), small, fast, mostly compatible with xterm. Normal workflow uses ca 12 terms. Duplicate commands on different terms are a usual happening. In my .xinitrc I'm using a now 20 year old scriptlet, that copies the ~/.bash_history to a temp file, and appends it to a bigger saved history file (similar to dnh), then the temp file gets pumped into a perlscript that: 1: reads files of the .bash_history (with timestamps) format, 2: builds a hash with the command as key, and the timestamp as value, 3: for a existing key the value gets the bigger timestamp, 4: then a array is formed, for each key/value pair: '#${timestamp}\n${command}\n', 5: this array gets sorted by a self defined sorting sub (timestamp, then command) 6: and the whole, now uniq history gets written into ~/.bash_history This works for me, as I reboot the machine about once a month, and during that time I use suspend-to-disk for the nightly pause. A added 2-4 seconds on X11 login after reboot is acceptable for me. HISTSIZE=50000 HISTFILESIZE=500000 HISTCONTROL=ignorespace:erasedups HISTTIMEFORMAT="%F %T " - Yamaban. PS: if someone wants such a perlscript, please say so, and, yes, I know, nowadays it is possible to do such things in bash. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 03/30/2015 01:02 PM, Yamaban wrote:
I'm using urxvt(d), small, fast, mostly compatible with xterm. Normal workflow uses ca 12 terms. Duplicate commands on different terms are a usual happening.
In my .xinitrc I'm using a now 20 year old scriptlet, that copies the ~/.bash_history to a temp file, and appends it to a bigger saved history file (similar to dnh), then the temp file gets pumped into a perlscript that: 1: reads files of the .bash_history (with timestamps) format, 2: builds a hash with the command as key, and the timestamp as value, 3: for a existing key the value gets the bigger timestamp, 4: then a array is formed, for each key/value pair: '#${timestamp}\n${command}\n', 5: this array gets sorted by a self defined sorting sub (timestamp, then command) 6: and the whole, now uniq history gets written into ~/.bash_history
This works for me, as I reboot the machine about once a month, and during that time I use suspend-to-disk for the nightly pause.
A added 2-4 seconds on X11 login after reboot is acceptable for me.
HISTSIZE=50000 HISTFILESIZE=500000 HISTCONTROL=ignorespace:erasedups HISTTIMEFORMAT="%F %T "
Yamaban, Can you add your info to the bug. Werner is looking at it, but is suggesting a bug upstream. I agree, but I would like to capture your information in the report before sending it up. https://bugzilla.suse.com/show_bug.cgi?id=924818 -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 03/30/2015 11:52 AM, David Haller wrote:
That is all weird, and, mind you, I use the same bash as you guys, even though I'm on 12.1 ;)
$ rpm -q bash bash-4.3-275.1.x86_64 $ rpm -q --changelog bash | head -2 * Wed Mar 04 2015jdelvare@suse.de - bash-4.3-loadables.dif: One more warning fixed, in
So, I guess it's in libhistory / libreadline, but I think I use the same as well:
$ rpm -qf /lib64/libhistory.so.6.3 libreadline6-6.3-275.1.x86_64 $ rpm -q --changelog libreadline6 | head -2 * Wed Mar 04 2015jdelvare@suse.de - bash-4.3-loadables.dif: One more warning fixed, in
I just tried a few "history -w", "ls foo", "history -w" from different xterms, and it all comes out as just one single entry each (one "ls" and one "history -w").
dnh There is a bug somewhere, even though you are on 12.1, you seem to be ahead of 13.1: $ rpm -q bash bash-4.2-68.12.1.x86_64 $ rpm -q --changelog bash | head -2 * Mon Oct 06 2014 werner@suse.de - Replace patches bash-4.2-heredoc-eof-delim.patch and $ rpm -qf /lib64/libhistory.so.6.2 libreadline6-6.2-68.12.1.x86_64 Global: $ cat /etc/bash.bashrc.local PATH=$PATH:/usr/sbin HISTCONTROL=ignoreboth:erasedups User: ~/.bashrc: export HISTFILESIZE=20000 export HISTSIZE=20000 export HISTCONTROL=ignoreboth:erasedups ## below: ignore space : ignore previous : ignore 1 : ignore 2 char cmds ## any pattern must match entire line (no presumed *) export HISTIGNORE=' *:&:?:??' This is running 13.1 in konsole in kde3. It will be interesting to see where/what this is narrowed down too. What other tests does the master have -- that I can snatch from your hand to help narrow this down? -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 04/01/2015 10:40 PM, David C. Rankin wrote:
$ cat /etc/bash.bashrc.local PATH=$PATH:/usr/sbin HISTCONTROL=ignoreboth:erasedups
User: ~/.bashrc:
export HISTFILESIZE=20000 export HISTSIZE=20000 export HISTCONTROL=ignoreboth:erasedups ## below: ignore space : ignore previous : ignore 1 : ignore 2 char cmds ## any pattern must match entire line (no presumed *) export HISTIGNORE=' *:&:?:??'
Confirmed in env: $ env | grep HIST HISTSIZE=20000 HISTFILESIZE=20000 HISTIGNORE= *:&:?:?? HISTCONTROL=ignoreboth:erasedups -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hello, On Wed, 01 Apr 2015, David C. Rankin wrote:
On 03/30/2015 11:52 AM, David Haller wrote:
That is all weird, and, mind you, I use the same bash as you guys, even though I'm on 12.1 ;)
$ rpm -q bash bash-4.3-275.1.x86_64 $ rpm -q --changelog bash | head -2 * Wed Mar 04 2015jdelvare@suse.de - bash-4.3-loadables.dif: One more warning fixed, in [..] There is a bug somewhere, even though you are on 12.1, you seem to be ahead of 13.1:
;) I suggest you get the current 4.3 from the 'shells' repo.
$ rpm -q bash bash-4.2-68.12.1.x86_64 $ rpm -q --changelog bash | head -2 * Mon Oct 06 2014 werner@suse.de - Replace patches bash-4.2-heredoc-eof-delim.patch and [..] What other tests does the master have -- that I can snatch from your hand to help narrow this down?
I just remembered: how are your shopt settings? $ shopt|grep hist cmdhist on histappend on histreedit off histverify off lithist off Anyway, I just use this to uniqify my big historyfile: ==== #!/bin/bash HFILE="$1" TMPF="$(mktemp "${HFILE}.$$.XXXXXXXXXX")" if time tac "$HFILE" | awk '!x[$0]++ { print; }' > "${TMPF}"; then tac "${TMPF}" > "${HFILE}" && rm "${TMPF}" fi ==== HTH, -dnh -- The nice thing about Windows is - It does not just crash, it displays a dialog box and lets you press 'OK' first. (Arno Schaefer's .sig) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 04/02/2015 11:59 AM, David Haller wrote:
I just remembered: how are your shopt settings?
$ shopt|grep hist cmdhist on histappend on histreedit off histverify off lithist off
Anyway, I just use this to uniqify my big historyfile:
==== #!/bin/bash HFILE="$1" TMPF="$(mktemp "${HFILE}.$$.XXXXXXXXXX")" if time tac "$HFILE" | awk '!x[$0]++ { print; }' > "${TMPF}"; then tac "${TMPF}" > "${HFILE}" && rm "${TMPF}" fi ====
HTH, -dnh
Always does. $ shopt|grep hist cmdhist on histappend on histreedit off histverify off lithist off I'll grab the current from the shells repo. I hope that doesn't break something else :-) -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hello, On Thu, 02 Apr 2015, David C. Rankin wrote:
On 04/02/2015 11:59 AM, David Haller wrote:
I just remembered: how are your shopt settings?
$ shopt|grep hist [..] Anyway, I just use this to uniqify my big historyfile:
==== #!/bin/bash HFILE="$1" TMPF="$(mktemp "${HFILE}.$$.XXXXXXXXXX")" if time tac "$HFILE" | awk '!x[$0]++ { print; }' > "${TMPF}"; then tac "${TMPF}" > "${HFILE}" && rm "${TMPF}" fi ====
Always does.
$ shopt|grep hist cmdhist on histappend on histreedit off histverify off lithist off [ == same as me]
I'll grab the current from the shells repo. I hope that doesn't break something else :-)
I've been using that bash-version since that (Env-)Variable bug thing (6 months ago, revisions tells me) and all is well as it used be to minus a few bugs ;) I build it in my home-repo though (for 12.1-13.1, just a link to Base:System, just as the one in "shells", without any changes), if you already have my repo in your list, grab it from there. -dnh -- I'm too busy to insult you, but your humiliation is important. Please hold. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 04/02/2015 11:59 AM, David Haller wrote:
There is a bug somewhere, even though you are on 12.1, you seem to be
ahead of 13.1: ;) I suggest you get the current 4.3 from the 'shells' repo.
Install: $ rpm -qa --queryformat '%{installtime} %{installtime:date} \ %{name} %{VERSION}-%{RELEASE}\n' | sort -n | cut -c 12- | tail -n10 <snip> Thu 02 Apr 2015 05:42:59 PM CDT libreadline6 6.3-275.1 Thu 02 Apr 2015 05:43:01 PM CDT readline-devel 6.3-275.1 Thu 02 Apr 2015 05:43:02 PM CDT bash 4.3-275.1 Thu 02 Apr 2015 05:43:06 PM CDT zsh 5.0.7-130.1 Thu 02 Apr 2015 05:43:07 PM CDT readline-doc 6.3-275.1 Thu 02 Apr 2015 05:43:08 PM CDT ksh 93v-251.2 Thu 02 Apr 2015 05:43:10 PM CDT bash-doc 4.3-275.1 Seems to be working now! Thanks dnh! -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hello, On Thu, 02 Apr 2015, David C. Rankin wrote:
On 04/02/2015 11:59 AM, David Haller wrote:
There is a bug somewhere, even though you are on 12.1, you seem to be
ahead of 13.1: ;) I suggest you get the current 4.3 from the 'shells' repo.
Install:
$ rpm -qa --queryformat '%{installtime} %{installtime:date} \ %{name} %{VERSION}-%{RELEASE}\n' | sort -n | cut -c 12- | tail -n10 <snip> Thu 02 Apr 2015 05:43:02 PM CDT bash 4.3-275.1
BTW: better just use | cut -f2- | instead of that breakable 'cut -c'. Remember, when unix-timestamps were just 9 digits long? You'd have used 'cut -c 11-' and on Sun 09.09.2001 03:46:40 CEST (date +%s == 1000000000) your script would have failed. Ok, the next digit is a while away (10000000000 == Sat 20.11.2286 18:46:40 CET), so you should be safe there (but what about a package installed before 09.09.2001?[1]) Stuff like this, is what _always_ gets people into trouble (Y2K anyone?). Generally: avoid 'cut -c' if you can get away with it in almost any way! Use awk instead if 'cut -fN' does not work, or even perl / python / ruby. But do not use count-of-character based stuff if you can avoid it in any way (or the spec states _explicitly_ that the field is N-{chars,bytes,digits} (what chars? ASCII? UTF-8?) long. And even then: distrust the specs! BTDT. [..]
Seems to be working now! Thanks dnh!
That's nice to read :) I have found some patches/changes regarding the libhistory (part of libreadline) since bash-4.2/accompanying libreadline, but nothing obvious. In such cases, I tend to try a newer (but tested) version, of which this is a fine example, as the bash-4.3 / libreadline / libhistory has been long tested. And if you find someone using that new stuff under the same or older oS (like me that 4.3 under 12.1), just the better, eh :) BTW: one indicator of a "well behaving" package is, that no tweaks are neccessary to build it for old(er) distro-versions. With bash, a simple link to the Base:System/bash suffices to be able to build it cleanly, so there should be no trouble[2]. When the packager has to change stuff to get the current package to build on e.g. 12.1, there might be problems (depending on the type of changes needed, they may be trivial and to be ignored (e.g. upstream just did not test against libfoo-(N-1) and thus requires libfoo-N in configure or such), but they might of the be "flaky" but "works-for-me" kind. HTH, -dnh [1] until "recently", I had such packages installed ;) [2] and in the >= 6 months I use it, I noticed none -- Beware of bugs in the above code; I have only proved it correct, not tried it. - Donald Knuth -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
David Haller wrote:
==== #!/bin/bash HFILE="$1" TMPF="$(mktemp "${HFILE}.$$.XXXXXXXXXX")" if time tac "$HFILE" | awk '!x[$0]++ { print; }' > "${TMPF}"; then tac "${TMPF}" > "${HFILE}" && rm "${TMPF}" fi ====
I'm not 100% sure, but the above looks like it might not filter out your dups as it doesn't sort them. I.e. if you run unsorted input into 'uniq', you won't get 'uniq' output. Also not sure why the time command is there? Are you trying to time the tac command? I ran your script on a concatenation of my 'tty-hist files' and first few lines of output from your script were: First several lines of your output: #1382638179 ps -ef|grep fetch #1382791092 #1382791960 #1382792415 #1382793525 -------------- Feeding same input into my script showed top entries: #1328153686 ls repo-oss #1328153702 ls repo-oss/suse/x86_64/ #1328153720 rpm -qpi repo-oss/suse/x86_64/xfce4-panel-plugin-xkb-0.5.3.3-7.1.x86_64.rpm #1328153783 rm -fr repo-{update,oss,source} #1328153788 --- Two observations: 1) why would your script keep 'uniq' blank lines? I.e. the consecutive timestamps w/no command & 2, the output from your script shows its 1st command starting 60 million seconds later than my script (which I think is related to it not being sorted). Besides blank lines, I try to filter out bogus or trival commands, and unlike bash -- if my script finds a dup -- it updates the 'seconds' field for that command so the time field shows the last time I used that command. That way my more frequently used commands end up near the end of the list... Your script is *way* shorter than mine though: (not including library calls! *sigh*)...
wc bin/hist_dedup 137 487 3707 bin/hist_dedup
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hello, On Fri, 03 Apr 2015, Linda Walsh wrote:
David Haller wrote:
==== #!/bin/bash HFILE="$1" TMPF="$(mktemp "${HFILE}.$$.XXXXXXXXXX")" if time tac "$HFILE" | awk '!x[$0]++ { print; }' > "${TMPF}"; then tac "${TMPF}" > "${HFILE}" && rm "${TMPF}" fi ==== I'm not 100% sure, but the above looks like it might not filter out your dups as it doesn't sort them.
It reads the file from the tail and dedups by using a hash, storing the first occurence (i.e. the latest!) and ignores duplicate same lines later (i.e. earlier). That '!x[$0]++' could be written as: current_line = $0; if( ! seen_this_line[current_line] ) { seen_this_line[current_line] = 1; print current_line; } No need to sort anything ;) Think of this history: cd foo ls cd ls tac reverses that, so awk reads: ls, cd, ls, cd foo first line read: x["ls"] is not there, so false, so x["ls"] is assigned and gets x["ls"] == 1, a true value. Next ! x["cd"] is again true as x["cd"] is non-existent i.e. false, but gets assigned 1 too. Next up "ls" again. Now, x["ls"] is 1, i.e. true, it gets incremented to 2 (you could use that for some stats at the end ;) and the next line is read, as ! x["ls"] is false. Next up: ! x["cd foo"] is true, as we've not seen that before. At the end, the whole shebang is reversed again, and "tada", we get cd foo cd ls
I.e. if you run unsorted input into 'uniq', you won't get 'uniq' output. Also not sure why the time command is there? Are you trying to time the tac command?
No, the whole thing, just out of curiosity. On my ~225k lines it took about 4.5s.
I ran your script on a concatenation of my 'tty-hist files' and first few lines of output from your script were: [..] #1328153686 ls repo-oss #1328153702 ls repo-oss/suse/x86_64/ #1328153720 rpm -qpi repo-oss/suse/x86_64/xfce4-panel-plugin-xkb-0.5.3.3-7.1.x86_64.rpm
Oh, you use timestamps. It won't work then. If you grab a test file and filter out the timestamps, e.g.: grep '^#' historyfile > tmp.hist and run my script on that tmp.hist, you'll see it works ;)
1) why would your script keep 'uniq' blank lines? I.e. the consecutive timestamps w/no command & 2, the output from your script shows its 1st command starting
They're not blank.
Besides blank lines, I try to filter out bogus or trival commands, and unlike bash -- if my script finds a dup -- it updates the 'seconds' field for that command so the time field shows the last time I used that command. That way my more frequently used commands end up near the end of the list...
Your script is *way* shorter than mine though: (not including library calls! *sigh*)...
wc bin/hist_dedup 137 487 3707 bin/hist_dedup
Have a try with this: ==== #!/bin/bash HFILE="$1" TMPF="$(mktemp "${HFILE}.$$.XXXXXXXX")" mkuniq() { gawk ' /^#[0-9]+/ { next; } # skip timestamps /^(ls|cd)$/ { # add more stuff to "filter" inside the (), note that I've # anchored these. Add more after the $, e.g.: # cd)$|^foo bar. Or make a whole block like this. next; } ! cmds[$0]++ { print; # print current cmd in $0 getline; # get timestamp (after cmd, file is reversed) print; # print timestamp (after cmd, will be reversed) }' } if tac "${HFILE}" | mkuniq > "$TMPF"; then tac "${TMPF}" > "${HFILE}" && rm "${TMPF}" fi ==== You should crosscheck results with your script etc. Any questions? HTH, -dnh -- Just go ahead and write your own multitasking multiuser os! Worked for me all the times. -- Linus Torvalds -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
David Haller wrote:
current_line = $0; if( ! seen_this_line[current_line] ) { seen_this_line[current_line] = 1; print current_line; }
No need to sort anything ;)
Think of this history:
.... I think we have a different idea of history. When I took history, it wasn't about cataloging a random sequence of events, but having some idea of when they occurred. I.e. If I know I've used a certain command for years, then figured out an improvement 3 weeks ago, I want to look for the commands from 3 weeks ago, on to see the corrected version. The behavior your script shows may be what *you* want: a grab-bag of previously used commands in a random order, but for my usage, I NEED commands grouped and sorted by when they occurred. I will often build up scripts through successive edits and then alternate specific parameters. If my history isn't sorted, I won't see the context of the command (CWD, variable values, defined functions and aliases). Without those, the usefulness of recalling a command with none of its "setup" or "environment" command is negligible. As an example -- wanted to install a package. If I search on the package I might find: spm -Uhv /factory/computertemp-0.9.6.1-24.1.x86_64.rpm But that won't help since *after* that command * see successive building up of requirements in the history. After 7 retries, I I ended up with: spm --replacepkgs -Uhv /factory/computertemp-0.9.6.1-24.1.x86_64.rpm /factory/gnome-python-desktop-2.32.0-19.1.x86_64.rpm python-gnomedesktop-2.32.0-19.1.x86_64.rpm python-gnomeprint-2.32.0-19.1.x86_64.rpm python-gtop-2.32.0-19.1.x86_64.rpm python-rsvg-2.32.0-19.1.x86_64.rpm python-totem-2.32.0-19.1.x86_64.rpm python-wnck-2.32.0-19.1.x86_64.rpm libgnome-desktop-3-7-3.8.2-1.1.x86_64.rpm libgnome-desktop-2-17-2.32.1-20.1.x86_64.rpm libgnomeprintui-2.18.6-14.1.x86_64.rpm If I am searching backwards -- because the lines are sorted w/date & time, The final version is the 1 that will come up first.
I.e. if you run unsorted input into 'uniq', you won't get 'uniq' output. Also not sure why the time command is there? Are you trying to time the tac command?
No, the whole thing, just out of curiosity. On my ~225k lines it took about 4.5s.
ah.
I ran your script on a concatenation of my 'tty-hist files' and first few lines of output from your script were: [..] #1328153686 ls repo-oss #1328153702 ls repo-oss/suse/x86_64/ #1328153720 rpm -qpi repo-oss/suse/x86_64/xfce4-panel-plugin-xkb-0.5.3.3-7.1.x86_64.rpm
Oh, you use timestamps. It won't work then. If you grab a test file and filter out the timestamps, e.g.:
---- A history file in a random order isn't a history file. It's a grab-bag.
grep '^#' historyfile > tmp.hist
and run my script on that tmp.hist, you'll see it works ;)
--- I'd lose potentially all context.
1) why would your script keep 'uniq' blank lines? I.e. the consecutive timestamps w/no command & 2, the output from your script shows its 1st command starting
They're not blank.
head -13 cat.hist #1382638179
---- Except for the 1st , the rest are blank lines -- a timestamp, no content followed by a 'LF': ps -ef|grep fetch #1382791092 #1382791960 #1382792415 #1382793525 #1382815570 #1382816176 #1382815576 #1382816222 #1382870743 #1382870876 #1382815581
From the 2nd timestamp through the last, they are blank (only an "\n" on the timestamp line):
head -13 cat.hist|hexdump -C 00000000 23 31 33 38 32 36 33 38 31 37 39 0a 70 73 20 20 |#1382638179.ps | 00000010 2d 65 66 7c 67 72 65 70 20 66 65 74 63 68 0a 23 |-ef|grep fetch.#| 00000020 31 33 38 32 37 39 31 30 39 32 0a 23 31 33 38 32 |1382791092.#1382| 00000030 37 39 31 39 36 30 0a 23 31 33 38 32 37 39 32 34 |791960.#13827924| 00000040 31 35 0a 23 31 33 38 32 37 39 33 35 32 35 0a 23 |15.#1382793525.#| 00000050 31 33 38 32 38 31 35 35 37 30 0a 23 31 33 38 32 |1382815570.#1382| 00000060 38 31 36 31 37 36 0a 23 31 33 38 32 38 31 35 35 |816176.#13828155| 00000070 37 36 0a 23 31 33 38 32 38 31 36 32 32 32 0a 23 |76.#1382816222.#| 00000080 31 33 38 32 38 37 30 37 34 33 0a 23 31 33 38 32 |1382870743.#1382| 00000090 38 37 30 38 37 36 0a 23 31 33 38 32 38 31 35 35 |870876.#13828155| 000000a0 38 31 0a |81.|
From the 2nd timestamp through the last, they are blank.
You should crosscheck results with your script etc. Any questions?
---- I _know_ it won't produce the results that _I_ want for history: I want a 'history', not a list of events w/no context. What is history with no reference to when or in what context it occurred? FWIW -- I may have 10 concurrent history files building up via 10 pty-windows. Sorting and merging I can see results of inter-related commands in separate windows. Sorting using 'tac' wouldn't work for retrieving the most recent usage. Example -- in my .hist$HOSTNAME_pts dir I have 'N' separate files: pty's: ~/.histIshtar_pts> ll -rt [0-9] 1[0-9] -rw-rw-r--+ 1 890 Jan 1 2014 17 -rw-rw-r--+ 1 3007 Aug 24 2014 16 ... -rw-rw-r--+ 1 2652854 Apr 4 17:24 1 -rw-rw-r--+ 1 640283 Apr 5 06:59 3 -rw-rw-r--+ 1 999883 Apr 5 07:02 0 As you run your merge on each pty you'll end up with one big file that has 'N' "ranges" of history containing, overlapping time-ranges -- with commands in different tty sessions: File 1, above ranges from:
head -1 1|grep -P '#\d+$' #1328153686 ~/.histIshtar_pts> unix_epoch2date 1328153686 Wed Feb 1 19:34:46 PST 2012 <<--start
~/.histIshtar_pts> tail -2 1|grep -P '#\d+$' #1428143038 ~/.histIshtar_pts> unix_epoch2date 1428143038 Sat Apr 4 03:23:58 PDT 2015 <<--end --- File 3 (defining a func)
sub drange { echo "Start: $(unix_epoch2date $(head -1 $1|grep -P '#\d+$'|tr -d '#'))" echo "End: $(unix_epoch2date $(tail -2 $1|grep -P '#\d+$'|tr -d '#'))" } drange 3 Start: Thu Oct 24 03:59:17 PDT 2013 End: Sat Apr 4 18:59:17 PDT 2015 ... etc.
If you drop out the timestamp, then once you've concatenated the results you'll have a mish-mash of recent->early commands and if you return the most recent and don't have a it sorted, you won't really get the most recent usage of that command. However, if all you want is a semi-random-grab-bag of previous commands, you don't need the timestamp -- but is it really a "HISTORY" file then? But if it works for your purposes... that's all that's needed! ;-) (And BTW, I, really, have done little testing on the correctness of my script, but it seemed to do about 90% of what I wanted, so I didn't spend more time on it...) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
David C. Rankin wrote:
If someone can confirm 13.1 is ignoring 'erasedups', I'll file the bug.
I can confirm the opposite(?). Try this:
echo blah echo blah echo foo echo blah history | tail -5 8363 0402@003632: history 8364 0402@003903: history|tail 8365 0402@003939: echo foo 8366 0402@003942: echo blah 8367 0402@003950: history |tail -5
Note, only 1 'echo blah' in the hist file. I too, keep long histories, but don't read the full hist in all the time...;-)
wc master_history 117447 252753 2663179 master_history
FYI - hist vals:
env|grep HIST HISTSIZE=1073741824 HISTFILESIZE=1073741824 HISTCONTROL=erasedups HISTTIMEFORMAT=%m%d@%H%M%S: HISTFILE=/home/law/.histIshtar_pts/6
Note: HISTFILE=~/.hist${HOSTNAME}_$TTY_NAME with a check done to make sure the '_pts' dir is present: if [ ! -d ${_local_dir:=`dirname $HISTFILE`} ] ; then.. rpms: libreadline6-6.2-68.1.5.x86_64 bash-4.2-68.1.5.x86_64 Every once in a while I merge them with a perl script into a 'master_history' file that eliminates all but the newest dup and "tries" to weed out improbable commands (things that can occur due to histfile overwrites or pasting text into a prompt). -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (4)
-
David C. Rankin -
David Haller -
Linda Walsh -
Yamaban