[opensuse] what's the best way to make charts of spreadsheet data?
I just made a chart of some data in LibreOffice Calc and was disappointed both by the process (very slow) and by the results (apparent lack of ability to modify the view). I've used spreadsheets sporadically over the years to make calculations but wouldn't call myself an expert and I can't remember the last time I tried to make a chart. So I know nothing. I'm wondering if I've missed something in LO that would help me have a better experience with this chart, or if there are better tools to use to draw charts from spreadsheet data. I asked on the LO list but only got one reply that only addressed one small issue, so I suppose there is no solution in LO. I thought I'd ask here to see what other software there is that can draw line charts, ideally from .ods data, and ideally with a GUI to set up the chart. But I can export the data in some other format and I can write config files to drive the layout if necessary. I'm running Version 6.1.3.2 of LibreOffice (actually the openSUSE build for Leap 15.0). I'm running a machine with an i5-4460 CPU @ 3.20GHz and 8 GB main memory. It's normally fast enough for whatever I want to do. My spreadsheet data is in columns. Hourly samples of temperature and humidity. Column A is dates and column B is the times of day. I've also created column C that adds A+B to get a single value. Then ten columns of humidity data and eleven columns of temperature data. There are currently just over 5000 rows of samples. Making the chart was extremely slow. Every time I changed the Name of a line of the chart I had to wait whilst it cycled through several steps to make the change. Initially when I removed some empty columns from the data range, it took an age to do so. etc. Now I've created the chart and got some reasonable lines across it, but what I hope are the dates and times along the X axis are just a mess and I've no idea how to make them display usefully. I haven't found any way to select individual points on the chart to display the detailed value of that point or all the values at that time (a cursor). Nor do I see a way to temporarily hide individual data lines to more clearly see other particular lines, for example. If there's any way to do these things with an LO chart, and to speed it up, I'd be grateful for any pointers. Or if there are better options out there that can read .ods or .csv and display charts, that would be very interesting. Google just wants to tell me about a lot of diagramming tools. TIA, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 04/01/2019 14.08, Dave Howorth wrote:
I just made a chart of some data in LibreOffice Calc and was disappointed both by the process (very slow) and by the results (apparent lack of ability to modify the view).
I've used spreadsheets sporadically over the years to make calculations but wouldn't call myself an expert and I can't remember the last time I tried to make a chart. So I know nothing.
I'm wondering if I've missed something in LO that would help me have a better experience with this chart, or if there are better tools to use to draw charts from spreadsheet data.
I use LO when I need a chart, but long ago I used other tools. Maybe I wrote my own about 1992. There is a gnu chart or graph tool somewhere, I don't remember the exact name. gnuplot Graphing scientific data has a problem compared to "business" use, that we want both X and Y data proportional on both axis, and this is not the default in LO. You could upload a sample file and we can try to have a go. Or mail it to me, as you prefer.
My spreadsheet data is in columns. Hourly samples of temperature and humidity. Column A is dates and column B is the times of day. I've also created column C that adds A+B to get a single value. Then ten columns of humidity data and eleven columns of temperature data. There are currently just over 5000 rows of samples.
That's a complicated layout. I think you should have one column of timestamps and another of temperatures. One column for each sensor, humidity, more temps, whatever.
Making the chart was extremely slow. Every time I changed the Name of a line of the chart I had to wait whilst it cycled through several steps to make the change. Initially when I removed some empty columns from the data range, it took an age to do so. etc.
I suppose the demands of the software are high now, yes. -- Cheers / Saludos, Carlos E. R. (from openSUSE 15.0 (Legolas))
Op vrijdag 4 januari 2019 14:08:01 CET schreef Dave Howorth:
I just made a chart of some data in LibreOffice Calc and was disappointed both by the process (very slow) and by the results (apparent lack of ability to modify the view).
I've used spreadsheets sporadically over the years to make calculations but wouldn't call myself an expert and I can't remember the last time I tried to make a chart. So I know nothing.
I'm wondering if I've missed something in LO that would help me have a better experience with this chart, or if there are better tools to use to draw charts from spreadsheet data.
I asked on the LO list but only got one reply that only addressed one small issue, so I suppose there is no solution in LO. I thought I'd ask here to see what other software there is that can draw line charts, ideally from .ods data, and ideally with a GUI to set up the chart. But I can export the data in some other format and I can write config files to drive the layout if necessary.
I'm running Version 6.1.3.2 of LibreOffice (actually the openSUSE build for Leap 15.0). I'm running a machine with an i5-4460 CPU @ 3.20GHz and 8 GB main memory. It's normally fast enough for whatever I want to do.
My spreadsheet data is in columns. Hourly samples of temperature and humidity. Column A is dates and column B is the times of day. I've also created column C that adds A+B to get a single value. Then ten columns of humidity data and eleven columns of temperature data. There are currently just over 5000 rows of samples.
Does col. C contain a formula or actual values in a date time format? If a formula, that will slow down things IME. I've also found in the past that using a single cell for date+time works better for not only calculations but also in creating graphs.
Making the chart was extremely slow. Every time I changed the Name of a line of the chart I had to wait whilst it cycled through several steps to make the change. Initially when I removed some empty columns from the data range, it took an age to do so. etc.
Now I've created the chart and got some reasonable lines across it, but what I hope are the dates and times along the X axis are just a mess and I've no idea how to make them display usefully.
I haven't found any way to select individual points on the chart to display the detailed value of that point or all the values at that time (a cursor). Nor do I see a way to temporarily hide individual data lines to more clearly see other particular lines, for example.
If there's any way to do these things with an LO chart, and to speed it up, I'd be grateful for any pointers. Or if there are better options out there that can read .ods or .csv and display charts, that would be very interesting. Google just wants to tell me about a lot of diagramming tools.
TIA, Dave
-- Gertjan Lettink a.k.a. Knurpht openSUSE Board Member openSUSE Forums Team -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Dave Howorth wrote:
I just made a chart of some data in LibreOffice Calc and was disappointed both by the process (very slow) and by the results (apparent lack of ability to modify the view).
Slow? How much data are we talking about?
I've used spreadsheets sporadically over the years to make calculations but wouldn't call myself an expert and I can't remember the last time I tried to make a chart. So I know nothing.
I'm wondering if I've missed something in LO that would help me have a better experience with this chart, or if there are better tools to use to draw charts from spreadsheet data.
I use the chart facilities of LO quite a lot, I have not found anything that match. I quite like the charting ability of rrdtool though. I have occasionally gone to look for tools I could use in batch-mode, e.g. from a Makefile. Haven't found much sofar.
My spreadsheet data is in columns. Hourly samples of temperature and humidity. Column A is dates and column B is the times of day. I've also created column C that adds A+B to get a single value. Then ten columns of humidity data and eleven columns of temperature data. There are currently just over 5000 rows of samples.
Sounds like rrdtool would be an alternative worth investigating.
Making the chart was extremely slow. Every time I changed the Name of a line of the chart I had to wait whilst it cycled through several steps to make the change. Initially when I removed some empty columns from the data range, it took an age to do so. etc.
I don't recall having tried with that amount of data - seems like a lot to want to put into one chart?
I haven't found any way to select individual points on the chart to display the detailed value of that point or all the values at that time (a cursor).
I think you get that if you hover over a data point. -- Per Jessen, Zürich (0.4°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Op vrijdag 4 januari 2019 18:43:34 CET schreef Per Jessen:
Dave Howorth wrote:
I just made a chart of some data in LibreOffice Calc and was disappointed both by the process (very slow) and by the results (apparent lack of ability to modify the view).
Slow? How much data are we talking about?
I've used spreadsheets sporadically over the years to make calculations but wouldn't call myself an expert and I can't remember the last time I tried to make a chart. So I know nothing.
I'm wondering if I've missed something in LO that would help me have a better experience with this chart, or if there are better tools to use to draw charts from spreadsheet data.
I use the chart facilities of LO quite a lot, I have not found anything that match. I quite like the charting ability of rrdtool though. I have occasionally gone to look for tools I could use in batch-mode, e.g. from a Makefile. Haven't found much sofar.
My spreadsheet data is in columns. Hourly samples of temperature and humidity. Column A is dates and column B is the times of day. I've also created column C that adds A+B to get a single value. Then ten columns of humidity data and eleven columns of temperature data. There are currently just over 5000 rows of samples.
Sounds like rrdtool would be an alternative worth investigating.
Making the chart was extremely slow. Every time I changed the Name of a line of the chart I had to wait whilst it cycled through several steps to make the change. Initially when I removed some empty columns from the data range, it took an age to do so. etc.
I don't recall having tried with that amount of data - seems like a lot to want to put into one chart?
I haven't found any way to select individual points on the chart to display the detailed value of that point or all the values at that time (a cursor).
I think you get that if you hover over a data point. I've made them much bigger than Dave states, i.e. > 2,000,000 cells. A rough workaround I used was create a new sheet, then copy paste(values+format) and generate the charts on that one. That worked quite snappy, also when running a macro that f.e. added a random number between 1 and 5 to a columns content.
-- Gertjan Lettink a.k.a. Knurpht openSUSE Board Member openSUSE Forums Team -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
I've made them much bigger than Dave states, i.e. > 2,000,000 cells.
How do you think about to share any more information for run time characteristics around similar data processing? Would you like to discuss precise execution durations for the generation of charts for the purpose of time series analysis? Will it make sense to look at another flame graph from a profiling session for a software like “LibreOffice Calc” (together with the diagram components)? http://www.brendangregg.com/flamegraphs.html Regards, Markus -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 04/01/2019 15:08, Dave Howorth wrote:
I just made a chart of some data in LibreOffice Calc
- just a remark : Metastock Stock Charts can be used to chart other things. Metastock has utilities to import stuff : - search : Metastock utility import data from Excel <https://www.yandex.com/search/?text=Metastock%20utility%20import%20data%20from%20Excel&lr=10418> ...... regards -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/04/2019 07:08 AM, Dave Howorth wrote:
My spreadsheet data is in columns. Hourly samples of temperature and humidity. Column A is dates and column B is the times of day. I've also created column C that adds A+B to get a single value. Then ten columns of humidity data and eleven columns of temperature data. There are currently just over 5000 rows of samples.
Making the chart was extremely slow. Every time I changed the Name of a line of the chart I had to wait whilst it cycled through several steps to make the change. Initially when I removed some empty columns from the data range, it took an age to do so. etc.
Dave, LO calc charting is on par with just about anything else. Performance is on par with Excel. The biggest problems I've found with both is they seem to be written by someone who came from the windows server coding world where it's good UI design to have to click through at least 5 nested dialogs accessed by clicking in non-intuitive place and activating secondary features by either a shift-click, or standing on one foot holding your left ear while whistling Dixie. If you post your data somewhere (or a subset), I'm happy to pull it in and play with it to see if anything shows up from a performance hit standpoint. For technical charting (and batch processing), there is always Gnuplot. It is a bit quirky with syntax, and good luck finding a concise howto that covers everything (apparently the demo should be enough for anyone -- not). But where it shines is batch processing and handling large data sets. You can easily use the language of your choice to process your data and then spit out as many differing charts as needed, directly to .pdf or .png, etc.. if you like. Let us know if you want a couple of compare and contrast experiences pulling your data in. I've always found just muddling through with the (LO Help in the past), now the woefully incomplete, but improving 'wiki' usually is enough to get me through the task. Check into the difference between using a calculated field and actual values as Gertjan suggested. I can see where that may cause chart generation fit continually checking whether a recalc is needed for every combined date and time. -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, 4 Jan 2019 13:08:01 +0000 Dave Howorth <dave@howorth.org.uk> wrote:
I just made a chart of some data in LibreOffice Calc and was disappointed both by the process (very slow) and by the results (apparent lack of ability to modify the view).
I've used spreadsheets sporadically over the years to make calculations but wouldn't call myself an expert and I can't remember the last time I tried to make a chart. So I know nothing.
I'm wondering if I've missed something in LO that would help me have a better experience with this chart, or if there are better tools to use to draw charts from spreadsheet data.
I asked on the LO list but only got one reply that only addressed one small issue, so I suppose there is no solution in LO. I thought I'd ask here to see what other software there is that can draw line charts, ideally from .ods data, and ideally with a GUI to set up the chart. But I can export the data in some other format and I can write config files to drive the layout if necessary.
I'm running Version 6.1.3.2 of LibreOffice (actually the openSUSE build for Leap 15.0). I'm running a machine with an i5-4460 CPU @ 3.20GHz and 8 GB main memory. It's normally fast enough for whatever I want to do.
My spreadsheet data is in columns. Hourly samples of temperature and humidity. Column A is dates and column B is the times of day. I've also created column C that adds A+B to get a single value. Then ten columns of humidity data and eleven columns of temperature data. There are currently just over 5000 rows of samples.
Making the chart was extremely slow. Every time I changed the Name of a line of the chart I had to wait whilst it cycled through several steps to make the change. Initially when I removed some empty columns from the data range, it took an age to do so. etc.
Now I've created the chart and got some reasonable lines across it, but what I hope are the dates and times along the X axis are just a mess and I've no idea how to make them display usefully.
I haven't found any way to select individual points on the chart to display the detailed value of that point or all the values at that time (a cursor). Nor do I see a way to temporarily hide individual data lines to more clearly see other particular lines, for example.
If there's any way to do these things with an LO chart, and to speed it up, I'd be grateful for any pointers. Or if there are better options out there that can read .ods or .csv and display charts, that would be very interesting. Google just wants to tell me about a lot of diagramming tools.
TIA, Dave
Thanks one and all. I got a little more help on the libreoffice list that got it to work as well as I think its going to. My current plan is to import the data into my emoncms system and use its graphing tool, which is fairly flexible. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
I just made a chart of some data in LibreOffice Calc and was disappointed both by the process (very slow) and by the results (apparent lack of ability to modify the view).
I am looking also for possibilities to improve the software behaviour further.
I'm running Version 6.1.3.2 of LibreOffice (actually the openSUSE build for Leap 15.0). I'm running a machine with an i5-4460 CPU @ 3.20GHz and 8 GB main memory.
I assume that your system configuration was not too busy with other stuff besides the desired diagram generation.
My spreadsheet data is in columns. Hourly samples of temperature and humidity.
Did you import these values as facts from a sensor data text file? Would you like to perform any advanced time series analysis? https://en.wikipedia.org/wiki/Time_series
Column A is dates and column B is the times of day. I've also created column C that adds A+B to get a single value.
How do you think about to combine these fields in a data preparation step so that extra formulas could be avoided?
Then ten columns of humidity data and eleven columns of temperature data. There are currently just over 5000 rows of samples.
The data amount is also interesting. Would you like to filter such records any more for your needs?
Making the chart was extremely slow. Every time I changed the Name of a line of the chart I had to wait whilst it cycled through several steps to make the change.
Will it help to store computation results into corresponding aggregate tables? https://en.wikipedia.org/wiki/Aggregate_(data_warehouse)
Initially when I removed some empty columns from the data range, it took an age to do so. etc.
How do you think about to move data cleaning operations to an other step in your work flow? https://en.wikipedia.org/wiki/Data_cleaning Does the selected chart type influence your run time experience in significant ways? https://help.libreoffice.org/6.1/en-GB/text/schart/01/wiz_chart_type.html
Now I've created the chart and got some reasonable lines across it, but what I hope are the dates and times along the X axis are just a mess and I've no idea how to make them display usefully.
Which display style would you find useful for data from the time dimension?
I haven't found any way to select individual points on the chart to display the detailed value of that point or all the values at that time (a cursor).
Is there any fine-tuning needed in the graphical user interface?
Nor do I see a way to temporarily hide individual data lines to more clearly see other particular lines, for example.
Would you need outlines here? https://help.libreoffice.org/6.1/en-GB/text/scalc/01/12080000.html
If there's any way to do these things with an LO chart, and to speed it up, I'd be grateful for any pointers.
Do you find the talk “Improving Calc parallel calculations” by Luboš Luňák interesting? https://libocon.org/assets/Conference/Tirana/llunak-calc.pdf https://www.youtube.com/watch?v=7QpTsUGUOkM Would you like to trigger any more evolution for chart software? https://bugs.documentfoundation.org/show_bug.cgi?id=122011
Or if there are better options out there that can read .ods or .csv and display charts, that would be very interesting. Google just wants to tell me about a lot of diagramming tools.
How helpful do you find the alternatives? https://en.wikipedia.org/wiki/List_of_charting_software https://en.wikipedia.org/wiki/List_of_statistical_packages Regards, Markus -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sun, 6 Jan 2019 19:09:59 +0100 Markus Elfring <Markus.Elfring@web.de> wrote:
I just made a chart of some data in LibreOffice Calc and was disappointed both by the process (very slow) and by the results (apparent lack of ability to modify the view).
I am looking also for possibilities to improve the software behaviour further.
I'm running Version 6.1.3.2 of LibreOffice (actually the openSUSE build for Leap 15.0). I'm running a machine with an i5-4460 CPU @ 3.20GHz and 8 GB main memory.
I assume that your system configuration was not too busy with other stuff besides the desired diagram generation.
Correct.
My spreadsheet data is in columns. Hourly samples of temperature and humidity.
Did you import these values as facts from a sensor data text file?
Yes, a CSV file.
Would you like to perform any advanced time series analysis? https://en.wikipedia.org/wiki/Time_series
I want to look at the data to see if anything stands out and keep it for future analysis of my house's performance. Whether any of that counts as 'advanced' I do not know.
Column A is dates and column B is the times of day. I've also created column C that adds A+B to get a single value.
How do you think about to combine these fields in a data preparation step so that extra formulas could be avoided?
I think I shouldn't need to worry about stuff like this. In fact the data in the CSV file was a DATETIME, but I couldn't figure out whether it was possible to import that to LO in a sensible way, so I added space to the separators and split it into two fields so the DATE format could cope.
Then ten columns of humidity data and eleven columns of temperature data. There are currently just over 5000 rows of samples.
The data amount is also interesting.
In what way?
Would you like to filter such records any more for your needs?
I don't know.
Making the chart was extremely slow. Every time I changed the Name of a line of the chart I had to wait whilst it cycled through several steps to make the change.
Will it help to store computation results into corresponding aggregate tables? https://en.wikipedia.org/wiki/Aggregate_(data_warehouse)
Err, I shouldn't need to think about any of this. The software tool should handle it competently.
Initially when I removed some empty columns from the data range, it took an age to do so. etc.
How do you think about to move data cleaning operations to an other step in your work flow? https://en.wikipedia.org/wiki/Data_cleaning
I'm not sure what you're suggesting. You mean the LO CSV import should be able to discard some columns?
Does the selected chart type influence your run time experience in significant ways? https://help.libreoffice.org/6.1/en-GB/text/schart/01/wiz_chart_type.html
Yes. A line chart is the wrong type. LO requires me to select XY. The tool should either give me better help or automatically select the chart type better based on the data.
Now I've created the chart and got some reasonable lines across it, but what I hope are the dates and times along the X axis are just a mess and I've no idea how to make them display usefully.
Which display style would you find useful for data from the time dimension?
LO XY works.
I haven't found any way to select individual points on the chart to display the detailed value of that point or all the values at that time (a cursor).
Is there any fine-tuning needed in the graphical user interface?
Several people have mentioned tooltips should appear, but they don't for me for some reason. And I feel sure there are lots of things that could be improved in the UI.
Nor do I see a way to temporarily hide individual data lines to more clearly see other particular lines, for example.
Would you need outlines here? https://help.libreoffice.org/6.1/en-GB/text/scalc/01/12080000.html
I don't think so, I just want to be able to temprarily hide some of the lines representing individual data sets.
If there's any way to do these things with an LO chart, and to speed it up, I'd be grateful for any pointers.
Do you find the talk “Improving Calc parallel calculations” by Luboš Luňák interesting? https://libocon.org/assets/Conference/Tirana/llunak-calc.pdf https://www.youtube.com/watch?v=7QpTsUGUOkM
Would you like to trigger any more evolution for chart software? https://bugs.documentfoundation.org/show_bug.cgi?id=122011
Or if there are better options out there that can read .ods or .csv and display charts, that would be very interesting. Google just wants to tell me about a lot of diagramming tools.
How helpful do you find the alternatives? https://en.wikipedia.org/wiki/List_of_charting_software https://en.wikipedia.org/wiki/List_of_statistical_packages
The lists aren't terribly helpful. Lists of time-series software are closer, but still no banana. Pretty much all assume that the graph is a final output of some carefully crafted process, whereas I see it as an initial attempt to view some data that I should be able to change on the fly (zoom time, hide datasets, integrate/differentiate etc). I had decided instead to import the CSV data into my emoncms system, which has a graph module that is much more functional and performant. But I've now learned of InfluxDB and Graphana, which may be interesting. I haven't had time to investigate them yet. Cheers, Dave
Regards, Markus
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Yes, a CSV file.
Would you like to share any more information about the system infrastructure which provided the sensor data? Are any special data processing systems involved there?
Would you like to perform any advanced time series analysis? https://en.wikipedia.org/wiki/Time_series
I want to look at the data to see if anything stands out and keep it for future analysis of my house's performance.
Thanks for such background information. Are you looking for trend displays?
Whether any of that counts as 'advanced' I do not know.
Would you like to get informed about extraordinary value distributions?
How do you think about to combine these fields in a data preparation step so that extra formulas could be avoided?
I think I shouldn't need to worry about stuff like this.
If you care for efficient (and considerably fast) data processing, corresponding development considerations are an usual consequence, aren't they?
In fact the data in the CSV file was a DATETIME,
Does this data format get automatically recognised by the used software?
but I couldn't figure out whether it was possible to import that to LO in a sensible way,
Did you get additional experiences on this detail?
so I added space to the separators and split it into two fields so the DATE format could cope.
Would you like to see an other adjustment approach at this place?
The data amount is also interesting.
In what way?
You observed surprisingly long data processing durations for diagram generation here, didn't you? The data size growth will trigger to reconsider analysis approaches, won't it?
Would you like to filter such records any more for your needs?
I don't know.
* Do you need to drill down into a time range? * Which conditions will become relevant for occasional data exclusion?
Will it help to store computation results into corresponding aggregate tables? https://en.wikipedia.org/wiki/Aggregate_(data_warehouse)
Err, I shouldn't need to think about any of this.
Other users came along similar software situations already. Dedicated data structures will provide special benefits so that available application knowledge is reused in easier ways.
The software tool should handle it competently.
Specific tools might support a more convenient way.
I'm not sure what you're suggesting. You mean the LO CSV import should be able to discard some columns?
Which tool would be more appropriate for data cleaning operations? How do you think about to split adjustments in the work flow?
https://help.libreoffice.org/6.1/en-GB/text/schart/01/wiz_chart_type.html
A line chart is the wrong type. LO requires me to select XY. The tool should either give me better help or automatically select the chart type better based on the data.
Would this situation motivate you to improve the program documentation?
Would you need outlines here? https://help.libreoffice.org/6.1/en-GB/text/scalc/01/12080000.html
I don't think so, I just want to be able to temprarily hide some of the lines representing individual data sets.
Would you like to explain corresponding constraints?
How helpful do you find the alternatives? https://en.wikipedia.org/wiki/List_of_charting_software https://en.wikipedia.org/wiki/List_of_statistical_packages
The lists aren't terribly helpful. Lists of time-series software are closer, but still no banana. Pretty much all assume that the graph is a final output of some carefully crafted process, whereas I see it as an initial attempt to view some data that I should be able to change on the fly (zoom time, hide datasets, integrate/differentiate etc).
Are you looking for more data mining and monitoring capabilities?
I had decided instead to import the CSV data into my emoncms system, which has a graph module that is much more functional and performant. But I've now learned of InfluxDB and Graphana, which may be interesting.
Is there any healthy competition evolving because of these tools? Regards, Markus -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, 7 Jan 2019 19:11:26 +0100 Markus Elfring <Markus.Elfring@web.de> wrote:
Yes, a CSV file.
Would you like to share any more information about the system infrastructure which provided the sensor data?
Are any special data processing systems involved there?
I'm not sure what the relevance is? It's a special purpose humidity logger.
Would you like to perform any advanced time series analysis? https://en.wikipedia.org/wiki/Time_series
I want to look at the data to see if anything stands out and keep it for future analysis of my house's performance.
Thanks for such background information.
Are you looking for trend displays?
I'm interested in trends and/or moving averages, yes. Having them displayed explicitly is useful but not necessary.
Whether any of that counts as 'advanced' I do not know.
Would you like to get informed about extraordinary value distributions?
?
How do you think about to combine these fields in a data preparation step so that extra formulas could be avoided?
I think I shouldn't need to worry about stuff like this.
If you care for efficient (and considerably fast) data processing, corresponding development considerations are an usual consequence, aren't they?
I'm not considering development, I'm trying to use already developed systems. Those systems should have considered efficient performance for sure, but it seems that LO didn't bother.
In fact the data in the CSV file was a DATETIME,
Does this data format get automatically recognised by the used software?
LO doesn't appear to recognize it, no, as I said (see below).
but I couldn't figure out whether it was possible to import that to LO in a sensible way,
Did you get additional experiences on this detail?
I have no idea what you mean.
so I added space to the separators and split it into two fields so the DATE format could cope.
Would you like to see an other adjustment approach at this place?
Well, either LO could combine values in separate date and time columns to produce a datetime on which to base the X axis of the graph, or it could parse the input in the first place.
The data amount is also interesting.
In what way?
You observed surprisingly long data processing durations for diagram generation here, didn't you?
The data size growth will trigger to reconsider analysis approaches, won't it?
I don't know what you mean. LO is very slow, so LO should consider improving its performance, IMHO. I know of other software that is faster, so I may be forced to build an interface and use that instead.
Would you like to filter such records any more for your needs?
I don't know.
* Do you need to drill down into a time range?
Yes
* Which conditions will become relevant for occasional data exclusion?
Dunno.
Will it help to store computation results into corresponding aggregate tables? https://en.wikipedia.org/wiki/Aggregate_(data_warehouse)
Err, I shouldn't need to think about any of this.
Other users came along similar software situations already. Dedicated data structures will provide special benefits so that available application knowledge is reused in easier ways.
The software tool should handle it competently.
Specific tools might support a more convenient way.
I'm not sure what you're suggesting. You mean the LO CSV import should be able to discard some columns?
Which tool would be more appropriate for data cleaning operations?
How do you think about to split adjustments in the work flow?
https://help.libreoffice.org/6.1/en-GB/text/schart/01/wiz_chart_type.html
A line chart is the wrong type. LO requires me to select XY. The tool should either give me better help or automatically select the chart type better based on the data.
Would this situation motivate you to improve the program documentation?
No, it will motivate me to choose some other program instead.
Would you need outlines here? https://help.libreoffice.org/6.1/en-GB/text/scalc/01/12080000.html
I don't think so, I just want to be able to temprarily hide some of the lines representing individual data sets.
Would you like to explain corresponding constraints?
Again, I don't understand what you mean.
How helpful do you find the alternatives? https://en.wikipedia.org/wiki/List_of_charting_software https://en.wikipedia.org/wiki/List_of_statistical_packages
The lists aren't terribly helpful. Lists of time-series software are closer, but still no banana. Pretty much all assume that the graph is a final output of some carefully crafted process, whereas I see it as an initial attempt to view some data that I should be able to change on the fly (zoom time, hide datasets, integrate/differentiate etc).
Are you looking for more data mining and monitoring capabilities?
I had decided instead to import the CSV data into my emoncms system, which has a graph module that is much more functional and performant. But I've now learned of InfluxDB and Graphana, which may be interesting.
Is there any healthy competition evolving because of these tools?
I don't know yet. I think what's happened so far is I've narrowed the sort of problems that I will consider LO as a possible solution for. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 07/01/2019 20.59, Dave Howorth wrote:
On Mon, 7 Jan 2019 19:11:26 +0100 Markus Elfring <> wrote:
Yes, a CSV file.
...
In fact the data in the CSV file was a DATETIME,
Does this data format get automatically recognised by the used software?
LO doesn't appear to recognize it, no, as I said (see below).
Possibly the locale format for your data is not the same as LO expects. Or maybe you have to tick a box in LO import to "handle special numbers" (such as dates). -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
It's a special purpose humidity logger.
Have you got any influence on the data format for specific fields by this software?
I'm not considering development, I'm trying to use already developed systems.
You are also in the process of selecting appropriate tools for some tasks.
Those systems should have considered efficient performance for sure,
This aspect was checked to some degree.
but it seems that LO didn't bother.
I imagine that there are also special software design constraints involved. A program like “LibreOffice Calc” can mostly be used as a convenient graphical user interface including the presentation of charts. These diagrams need to transform the provided input data into structures for more efficient processing. Would you like to change any software limitations in this area?
The data size growth will trigger to reconsider analysis approaches, won't it?
I don't know what you mean. LO is very slow, so LO should consider improving its performance, IMHO.
I guess that we observe scalability difficulties.
I know of other software that is faster,
Such a fact is also good to know, isn't it? Will any other chart modules become quicker, too?
so I may be forced to build an interface and use that instead.
Do you interpret this as another interesting challenge?
I don't think so, I just want to be able to temprarily hide some of the lines representing individual data sets.
Would you like to explain corresponding constraints?
Again, I don't understand what you mean.
How do you think about to clarify the challenges around the mentioned data selections a bit more? Regards, Markus -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, 8 Jan 2019 16:18:46 +0100 Markus Elfring <Markus.Elfring@web.de> wrote:
It's a special purpose humidity logger.
Have you got any influence on the data format for specific fields by this software?
No
I'm not considering development, I'm trying to use already developed systems.
You are also in the process of selecting appropriate tools for some tasks.
I wish you'd leave more context when replying. I hate having to refer back to other messages. For that matter, I'm not clear what the point of this discussion is. Are you trying to help me, or am I trying to help you?
Those systems should have considered efficient performance for sure,
This aspect was checked to some degree.
but it seems that LO didn't bother.
I imagine that there are also special software design constraints involved.
A program like “LibreOffice Calc” can mostly be used as a convenient graphical user interface including the presentation of charts. These diagrams need to transform the provided input data into structures for more efficient processing. Would you like to change any software limitations in this area?
I don't think it does provide a convenient interface for charting, especially apparently for time-series, but also in terms of abilities to change the view: expand time, simplify the display etc.
The data size growth will trigger to reconsider analysis approaches, won't it?
I don't know what you mean. LO is very slow, so LO should consider improving its performance, IMHO.
I guess that we observe scalability difficulties.
I know of other software that is faster,
Such a fact is also good to know, isn't it?
Will any other chart modules become quicker, too?
I have no idea.
so I may be forced to build an interface and use that instead.
Do you interpret this as another interesting challenge?
I hope it will be interesting, but it's a nuisance rather than a challenge. I have other things I'd rather do than build a bit of special software interface glue.
I don't think so, I just want to be able to temprarily hide some of the lines representing individual data sets.
Would you like to explain corresponding constraints?
Again, I don't understand what you mean.
How do you think about to clarify the challenges around the mentioned data selections a bit more?
I don't see any challenges. In emoncms, for example, I just click on the legend for a line, to temporarily remove that line from the display. Another click restores it. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
For that matter, I'm not clear what the point of this discussion is.
I am trying to clarify the goals a bit more.
Are you trying to help me,
I hope so.
or am I trying to help you?
I imagine that more people can benefit from further clarification of the mentioned technical properties.
A program like “LibreOffice Calc” can mostly be used as a convenient graphical user interface including the presentation of charts. These diagrams need to transform the provided input data into structures for more efficient processing. Would you like to change any software limitations in this area?
I don't think it does provide a convenient interface for charting, especially apparently for time-series, but also in terms of abilities to change the view: expand time, simplify the display etc.
I imagine that you could become interested in data warehouse technology for such use cases.
How do you think about to clarify the challenges around the mentioned data selections a bit more?
I don't see any challenges. In emoncms, for example, I just click on the legend for a line, to temporarily remove that line from the display. Another click restores it.
Do find the supported functionality already nicer there in comparison to other evolving data analysis and monitoring software? Regards, Markus -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hi, I didn't follow the whole discussion but as far as I understood it's about statistical analysis of time series and about how to create meaningful graphical ways to display them. Have you looked into R? It has a lot of different packages that allow both. Good luck, Karl Am 06.01.19 um 19:09 schrieb Markus Elfring:
I just made a chart of some data in LibreOffice Calc and was disappointed both by the process (very slow) and by the results (apparent lack of ability to modify the view). I am looking also for possibilities to improve the software behaviour further.
I'm running Version 6.1.3.2 of LibreOffice (actually the openSUSE build for Leap 15.0). I'm running a machine with an i5-4460 CPU @ 3.20GHz and 8 GB main memory. I assume that your system configuration was not too busy with other stuff besides the desired diagram generation.
My spreadsheet data is in columns. Hourly samples of temperature and humidity. Did you import these values as facts from a sensor data text file?
Would you like to perform any advanced time series analysis? https://en.wikipedia.org/wiki/Time_series
Column A is dates and column B is the times of day. I've also created column C that adds A+B to get a single value. How do you think about to combine these fields in a data preparation step so that extra formulas could be avoided?
Then ten columns of humidity data and eleven columns of temperature data. There are currently just over 5000 rows of samples. The data amount is also interesting.
Would you like to filter such records any more for your needs?
Making the chart was extremely slow. Every time I changed the Name of a line of the chart I had to wait whilst it cycled through several steps to make the change. Will it help to store computation results into corresponding aggregate tables? https://en.wikipedia.org/wiki/Aggregate_(data_warehouse)
Initially when I removed some empty columns from the data range, it took an age to do so. etc. How do you think about to move data cleaning operations to an other step in your work flow? https://en.wikipedia.org/wiki/Data_cleaning
Does the selected chart type influence your run time experience in significant ways? https://help.libreoffice.org/6.1/en-GB/text/schart/01/wiz_chart_type.html
Now I've created the chart and got some reasonable lines across it, but what I hope are the dates and times along the X axis are just a mess and I've no idea how to make them display usefully. Which display style would you find useful for data from the time dimension?
I haven't found any way to select individual points on the chart to display the detailed value of that point or all the values at that time (a cursor). Is there any fine-tuning needed in the graphical user interface?
Nor do I see a way to temporarily hide individual data lines to more clearly see other particular lines, for example. Would you need outlines here? https://help.libreoffice.org/6.1/en-GB/text/scalc/01/12080000.html
If there's any way to do these things with an LO chart, and to speed it up, I'd be grateful for any pointers. Do you find the talk “Improving Calc parallel calculations” by Luboš Luňák interesting? https://libocon.org/assets/Conference/Tirana/llunak-calc.pdf https://www.youtube.com/watch?v=7QpTsUGUOkM
Would you like to trigger any more evolution for chart software? https://bugs.documentfoundation.org/show_bug.cgi?id=122011
Or if there are better options out there that can read .ods or .csv and display charts, that would be very interesting. Google just wants to tell me about a lot of diagramming tools. How helpful do you find the alternatives? https://en.wikipedia.org/wiki/List_of_charting_software https://en.wikipedia.org/wiki/List_of_statistical_packages
Regards, Markus
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (8)
-
Carlos E. R. -
Dave Howorth -
David C. Rankin -
ellanios82 -
Karl Sinn -
Knurpht-openSUSE -
Markus Elfring -
Per Jessen