Re: [SLE] Developing a Real Time Data System
John wrote:
Just html?
not only html, i have worked under other languages too
Real time under Windows and linux is not generally possible, but can be obtained in linux with special provisions in a heavily modified kernel, or in either with a separate master real-time program that provides real time functions with linux as a subordinate module.
Even with the specialized additions to make linux or Windows real-time, you won't get real-time performance unless you know how to use it -- and that's not trivial, or even easy!
Although real-time modified Windows and linux are possible, and becoming more popular, it's almost always easier to use a real-time operating system or executive (depending upon what exactly you want to do), and you'll always come out at the end with a more robust system, at least from a real-time point of view. Other, non-real time considerations sometimes make it better to use the desktop and server programs (Win and Linux), but this is separate from real-time considerations.
Could explaine about this desktops and server programs more detailed..??
With all this said, many people often miscall "interactive", "event-driven", and "quick-response" programming real-time, so we'd have to know what you're trying to do.
have you worked under labview?? (sorry this is not free software, but i must mention it), i would like to create an application which allows me to get data from a Data Acquisition Card. Actually I am thinking of working with a National Instruments Acquisition data card, this card is used for applications on which is required to measure any process, from voltage to temperature to an electric engine speed. This card just get an analog input that can vary from 10 volts to 10 volts (no matter the process to be tested it is required to transform it into a ±10 v signal and connect it to the card) then internally the analog signal is transformed to a digital signal and sent to the computer using a PCI, USB or PCMCIA bus. For the USB (low cost option) bus the specifications as follows: Bus Interface: USB 2.0 full-speed (12 Mb/s) Analog Input Sample rate: 10 kS/s This is just a basic description of course such card can handle up to 16 analog inputs
With all this said, many people often miscall "interactive", "event-driven", and "quick-response" programming real-time, so we'd have to know what you're trying to do.
And, of course, procedural programming, real-time or not, is completely different from text markup programming, so you have a lot to learn before you go very far, no matter what you're really doing :-).
I know this requires lot of reading and learning, that´s why I am asking for help at this forum : ) any manual, tutorial or link to introduce me to the real data programming I would appreciate your suggesttions
If your html programming included javascript and/or php, you have a vague idea what the difference is.
Thinking of python to develop the GUI interface, there is a powerful library called Matplotlib which allows graphics interface familiar to matlab and code seems simple, on the other hand could i handle the data by coding C ?? Thanks for your help in advance Regards.- Juan
By the way, the data must be stored into a data base: mysql or postgresql. All these designed on open source environment. Thnks for your help again regards
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Wednesday 2005-11-23 at 15:48 -0000, JUAN ERNESTO FLORES BELTRAN wrote:
the data must be stored into a data base: mysql or postgresql. All these designed on open source environment.
You still haven't said what sampling speed you need, but unless it is slow, you will need to save somewhere else first (as fast as posible), and then move to the database as a post process. - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFDhSjztTMYHG2NR9URAnxVAJ9Ux+C4aFp/iEggONtqnjXfnu/OMwCfVmt6 idsIp+jnnw3ECglV5OwfAl8= =X4Fe -----END PGP SIGNATURE-----
carlos wrote:
You still haven't said what sampling speed you need, but unless it is slow, you will need to save somewhere else first (as fast as posible), and then move to the database as a post process.
Sampling speed will be as slow as possible because of this is a project for the university...not a product for anyone so speed is subject to the designer programming skills and available resources: adquisition data card specifications, programming language, etc... just to get an idea...what limit is considered "too fast" to save somewhere else before saving into the database?? and what technical reasons would force me to do this in case sampling speed increases where neccesary?? is this subject to any mysql or postgresql managing/storaging data limitations?? Any link, tutorial or manual to get inmersed into real time data handling?? would be helpfull!! Regards.- Juan.
On Thu, 2005-11-24 at 13:18 +0000, JUAN ERNESTO FLORES BELTRAN wrote:
carlos wrote:
You still haven't said what sampling speed you need, but unless it is slow, you will need to save somewhere else first (as fast as posible), and then move to the database as a post process.
Sampling speed will be as slow as possible because of this is a project for the university...not a product for anyone so speed is subject to the designer programming skills and available resources: adquisition data card specifications, programming language, etc...
just to get an idea...what limit is considered "too fast" to save somewhere else before saving into the database?? and what technical reasons would
Depends on the acquisition card. Some intelligently buffer data so an OS can take it when it can. Some only return a current value. So, the requirements of the OS are determined first by the card and secondly by how well the device driver is written. Are you wanting 200 values a second? 32000? 1000000?
force me to do this in case sampling speed increases where neccesary?? is this subject to any mysql or postgresql managing/storaging data limitations??
I doubt you would make the database direct. With some planning, you could, for example, make an acquisition program that writes to a flat disk file (for speed), and another program that reads this file and puts it in a database. They could run at the same time. -- Roger Oberholtzer OPQ Systems AB
John wrote:
If you're using an NI card outside the Labview environment, then yes, you're on your own, and >will need a lot more information about your data acquisition project, the card's interface specs, and >your computing environment.
That is the case, I will try to develop the system on my own
If you're going to roll your own system software, you'd do better to get another data acquisition >card, which will be much less expensive and usually easier to interface. Much of NI's stuff is >premium quality and performance, but others have premium stuff, too, with a less premium price
is a good option to get a cheaper card, not sure yet about which one to buy...
This is not to say that you can't build a good data acquisition system with suse. It will perform >much better and more reliably than Windows, but you'll need much more support than you can >get here. Indeed, you'll need more than you'll likely get on c.r or c.a.e, if you don't have a good
knowledge base to build upon. If this is a student or hobby project, I'd say go ahead full steam -- >if you stick it out, you'll build the necessary knowledge base to understand what people are saying >when they try to help. If someone is depending upon this project for something important, find someone knowledgeable you can talk to frequently for help. If it's needed soon, get professional >help :-).
This is a university project, and as you recommend I will try to get the required knowledge in both: real time system (QNX, OS-9, LYNXOS) and "quick-response" system (desktop). I think I could design the quick response system before starting a truly Real Time design. I have to get the knowledge no matter what system I decide to design first. Thanks for your help Regards.- juan
Thu, 24 Nov 2005, by roger@opq.se:
On Thu, 2005-11-24 at 13:18 +0000, JUAN ERNESTO FLORES BELTRAN wrote:
carlos wrote:
You still haven't said what sampling speed you need, but unless it is slow, you will need to save somewhere else first (as fast as posible), and then move to the database as a post process.
Sampling speed will be as slow as possible because of this is a project for the university...not a product for anyone so speed is subject to the designer programming skills and available resources: adquisition data card specifications, programming language, etc...
just to get an idea...what limit is considered "too fast" to save somewhere else before saving into the database?? and what technical reasons would
Depends on the acquisition card. Some intelligently buffer data so an OS can take it when it can. Some only return a current value. So, the requirements of the OS are determined first by the card and secondly by how well the device driver is written.
For best speed you'd probably want to use [a card with] dual-port memory and DMA, so that both the acquisition and processing can occure at the same time. Polling or interrupts is usually way too slow. Like you said, one thread would then directly transfer the data from DP mem to the harddisk (in binary format), while another would read from the disk and send the processed data to your database. Theo -- Theo v. Werkhoven Registered Linux user# 99872 http://counter.li.org ICBM 52 13 26N , 4 29 47E. + ICQ: 277217131 SUSE 9.2 + Jabber: muadib@jabber.xs4all.nl Kernel 2.6.8 + See headers for PGP/GPG info. Claimer: any email I receive will become my property. Disclaimers do not apply.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Saturday 2005-11-26 at 13:03 +0100, Theo v. Werkhoven wrote:
For best speed you'd probably want to use [a card with] dual-port memory and DMA, so that both the acquisition and processing can occure at the same time. Polling or interrupts is usually way too slow.
It depends on the needs. If you are going to simply sample at 1 second intervals, then a card doing 20Ks/s is very fast. If you need to create the equivalent of a 20Mhz oscilloscope in a PC, then anything below 100Ms/s is way too slow. The first thing is to fix the requirements, then search for cards and design. ¡Standard engineering! ;-) - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFDiF23tTMYHG2NR9URAmkZAJ0X1pwiKbMzADnS8MsFZ6ORdsYh6wCfQLjN ascJt1XY0T+n9wk5gtUFwNA= =pin+ -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Thursday 2005-11-24 at 13:18 -0000, JUAN ERNESTO FLORES BELTRAN wrote:
You still haven't said what sampling speed you need, but unless it is slow, you will need to save somewhere else first (as fast as posible), and then move to the database as a post process.
Sampling speed will be as slow as possible because of this is a project for the university...not a product for anyone so speed is subject to the designer programming skills and available resources: adquisition data card specifications, programming language, etc...
It is rather the other way round: you know what speed you require, then define what hardware/software requirements you need to meet it. Supposing you are to measure some sort of experimental data, like engine speed/accel: then somewhere about ten samples per second is enough. If you are measuring oven temperatures, then one per second or per minute. If atmospheric weather, one per quarter/hour, say. If you want to measure an audio amplifier response, I'd be happy with a million samples per second. It depends on what you need! You will find that sampling and storing data for a dozen channels at 10..100 K-s/s, non stop, quite taxing. That is withing the capabilities of a standard daq card. And related to that is the precision needed... 12bit is typical in industry, 16 bits is more expensive, 8 cheaper (and for home use, really). 12 bits is roughly equivalent to what a 3 1/2 digit multimeter gives.
just to get an idea...what limit is considered "too fast" to save somewhere else before saving into the database??
¡Test it! It will not be real time, in any case. Real time means that the response time is guaranteed to be below certain value, by the way, that's the most important characteristic. You can not use a database like that and give that guarantee. You can, however, simulate a test run, send data to a database at a fixed continuous rate for some time, and measure unexpected delays. You will have to define how exactly are you going to save the data, what format: one sample per register or a thousand, or the whole experiment in one register.
and what technical reasons would force me to do this in case sampling speed increases where neccesary?? is this subject to any mysql or postgresql managing/storaging data limitations??
I'm no expert in databases! X-) But they are certainly not designed for a continuous data flow storage, non delay, non stop.
Any link, tutorial or manual to get inmersed into real time data handling?? would be helpfull!!
I don't have any... I had to learn it on my own. What I did, roughly said, was to program the ISA card (that's what we used) to measured the channels and call an IRQ, at a fixed rate. An IRQ handler routine would save the data (a word) in memory or save to a file, raw. This could be an independent resident program (that was in Dos). Another routine, in the foreground, would read the data, filter it, process, display, perhaps even react to it, whatever. The card manufacturer supplied several code samples, from simple polling routines to irq or dma read. Later they supplied a driver that did all the dirty work, I only needed to call up the setup functions. I usually worked with Advantech cards, rather common here and not expensive (except for a hobbyist), meaning about 10 times less than National's. There are many more manufacturers, you will have to search around, depending on your requirements and purse. National people were usually helpful, they could give you sound advice using their software and hardware depending on your needs: not cheap, of course. Don't forget when looking around what kind of software and sourcecode samples they include, it makes a very big difference. And that that code was tested in Linux! ;-) I'm not sure how I'd do it in Linux, but I'd like to have a device driver that would copy data to a file or stream at a fixed rate. Probably it exists already. - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFDhl1ktTMYHG2NR9URAhQpAKCT8/t+vKiTt/nNpRReKo3N3/YRHACfRR8j nvk51LHpbH2FHO/4CKfvEu4= =MdL7 -----END PGP SIGNATURE-----
JUAN ERNESTO FLORES BELTRAN wrote:
John wrote:
Just html?
not only html, i have worked under other languages too
OK, this sounds less discouraging...
... Other, non-real time considerations sometimes make it better to use the desktop and server programs (Win and Linux), but this is separate from real-time considerations.
Could explaine about this desktops and server programs more detailed..??
The typical business computer complex has a set of computers called "file servers", whose function is to keep files and databases, and supply them to other computers for use by users or applications. Individuals have on their desks computers they use for their own computing purposes, called "desktop computers". These are all connected by a local area network (lan), which carries the information from one computer to another. The desktops typically use information from the servers, so it's all in one place and can be maintained and backed up easily. Scientific and academic organizations frequently need very high computing power, so have a set of supercomuters frequently called "compute servers". Scientists and students use their desktop computers to gain access to these compute servers. These organizations will also have file servers in their network. File servers are usually based on linux or special server editions of Windows. Desktops are usually based on Windows. Linux and Macintosh have a small but growing presence in desktop computer operating systems.
With all this said, many people often miscall "interactive", "event-driven", and "quick-response" programming real-time, so we'd have to know what you're trying to do.
For genuine real-time applications, there are several specially designed operating systems that provide the guaranteed response to events that Windows and linux can't ordinarily provide. Examples would be QNX, OS-9, and LYNXOS. There are also bare-bones programs called "executives" that provide real-time functions, but no file systems or, usually, networking. Examples would be VXWorks, RTEMS, or UC/OS II.
have you worked under labview?? (sorry this is not free software, but i must mention it), i would like to create an application which allows me to get data from a Data Acquisition Card. Actually I am thinking of working with a National Instruments Acquisition data card, this card is used for applications on which is required to measure any process, from voltage to temperature to an electric engine speed.
Don't worry about empty-headed ideologs. Are you talking about using Labview, or using the National Instruments card under other programs? If you're using Labview and a National Instruments board, I'm very surprised that you want to do any programming at all, since NI normally provides their hardware fully interfaced to their software (that's not always true for brand-new stuff, as I found out in my last data acquisition project, but they respond pretty well with bug fixes). If you're using an NI card outside the Labview environment, then yes, you're on your own, and will need a lot more information about your data acquisition project, the card's interface specs, and your computing environment.
This card just get an analog input that can vary from –10 volts to –10 volts (no matter the process to be tested it is required to transform it into a ±10 v signal and connect it to the card) then internally the analog signal is transformed to a digital signal and sent to the computer using a PCI, USB or PCMCIA bus.
For the USB (low cost option) bus the specifications as follows:
Bus Interface: USB 2.0 full-speed (12 Mb/s) Analog Input Sample rate: 10 kS/s
This is just a basic description of course…such card can handle up to 16 analog inputs
And this sounds as though you're just starting to design your system. If you're going to use Labview, you need to talk to a Labview applications engineer for help in setting up your system; then simply use Labview, realizing that although it performs well, it is not real-time (that is, there are no guaranteed response times) unles you spend a good deal of money for the special Labview real-time system. If you're going to roll your own system software, you'd do better to get another data acquisition card, which will be much less expensive and usually easier to interface. Much of NI's stuff is premium quality and performance, but others have premium stuff, too, with a less premium price :-).
With all this said, many people often miscall "interactive", "event-driven", and "quick-response" programming real-time, so we'd have to know what you're trying to do.
And, of course, procedural programming, real-time or not, is completely different from text markup programming, so you have a lot to learn before you go very far, no matter what you're really doing :-).
I know this requires lot of reading and learning, that´s why I am asking for help at this forum : ) any manual, tutorial or link to introduce me to the real data programming…I would appreciate your suggesttions
comp.realtime comp.arch.embedded Application notes from NI Application notes from QNX http://hardware.ittoolbox.com/topics/t.asp?t=371&p=371&h1=371 All have appropriate information available, and comp.realtime at least has frequently posted faqs. Either of these is more appropriate than SLE, which is concerned with desktop and server environments. This is not to say that you can't build a good data acquisition system with suse. It will perform much better and more reliably than Windows, but you'll need much more support than you can get here. Indeed, you'll need more than you'll likely get on c.r or c.a.e, if you don't have a good knowledge base to build upon. If this is a student or hobby project, I'd say go ahead full steam -- if you stick it out, you'll build the necessary knowledge base to understand what people are saying when they try to help. If someone is depending upon this project for something important, find someone knowledgeable you can talk to frequently for help. If it's needed soon, get professional help :-).
If your html programming included javascript and/or php, you have a vague idea what the difference is.
Thinking of python to develop the GUI interface, there is a powerful library called Matplotlib which allows graphics interface familiar to matlab and code seems simple, on the other hand could i handle the data by coding C ??
The data acquisition part is harder, but much less complicated, than data storage and display. But you can use any of the above, and many other languages besides, to do the job. If you have to. If you really have to do your own card driver, you'll need C and/or assembly language. If on the other hand, you're looking to get a data acquisition/control job done, why not just use Labview or one of its lookalikes? They've already done the dirty detail work, and you just need to build your application. John Perry
participants (5)
-
Carlos E. R. -
John Perry -
JUAN ERNESTO FLORES BELTRAN -
Roger Oberholtzer -
Theo v. Werkhoven