We use data everyday and every place. It's almost impossible to exist in a modern society without generating a tremendous amount of data. So what do we use this data for? That's a huge question, but the simplified version of that would be to optimize our process and procedures. We do this to save time, money, lives, or energy. It has even become a saying that the currency of today is information. How we gain information is by converting raw data into something that is useful. Aka information. How to acquire data is done in multiple ways. It can be gathered by people but that tends to be flawed and time consuming. One of the better ways of gathering data is to have small micro-controllers do this task for us.
The concept is simple. Take a energy efficient device, put some expansions on it and have it gather data all day long. For example, you want to measure sun exposure during a typical day. Or, how about you want to measure the air quality in your restaurant throughout the day and relate it to customer satisfaction. These are all valuable uses of data and how to convert them to information. What is important for us today is how this code works and how could we use this code to generically do our own data logging.
Equipment used:
- mikroMedia for XMEGA
- Battery Boost Shield
- SHT1X Sensor
The equipment isn't what is important, but the code is. The questions we need to answer are:
- What information and how to mark the collection event
- Where do we store the data
- What format do we store it
- How often do we collect the data
- For how long do we need the data collected
How to Mark the Collection Event
On the XMEGA this is childs play. It just so happens that this little 8 bit monster has its own RTC. This answers the question of how to mark the event. The clearest event is one that is marked with a reference to day and time. In the example, initializing the RTC is done with:
// Set Real Time Clock: 1000ms, external clk, use interrupt RTC16_Init(1000, _RTC_EXTERNAL_CLOCK, 1); RTC16_Start(); // Enables RTC operation
The source for the magic behind the RTC is found in the __Lib_Rtc.c file included with the project.
How we get the time is by calling RTC16_Get_Time(); , but the example makes things very easy by passing the temp and humidity to a function called New_measurement(Ta_Res, RH_Res); which calls get time and places the new measurement onto the storage medium.
What you would probably want to do is keep the functions as generic as possible so that the application would be capable of supporting any type of sensor reading.
Where to Store it?
The most logical choice is one that is cheap, vast, and can be easily read or written to. On the mikroMedia the choice is pretty clear. Although flash memory does exist on board, to meet all of the criteria the clear winner is MicroSD. With FAT formatted memory the reading of the data becomes easy and convenient. With the ease of using the FAT32 library, data logging can be done in files. Not only can they be put in files but if there is a huge amounts of data, you can store them in separate files named for their date.
The example was written without use of FAT32. Although it still works fine as is, FAT32 comes with some advantages. How about larger files sizes? The biggest advantage is the ability to use much larger SD cards without the need to re-partition and format.
Another consideration is when you have wireless or wired network connectivity. When this is available, writing to flash might be inconvenient. This all depends on your environment.
For the FAT32 library - You will find it on Libstock.com
Although the New_measurement(Ta_Res, RH_Res); takes care of the writing to the medium, your choice might be to separate the writing of the data until another time or when a buffer is filled. Inside the _MMC.c file included with the project is where all the MMC media action happens.
void MMC_WriteJK( char *datetime, char *temperature, char *humidity ){
// Construct variables needed for data logging
char txt[50];
char tmptxt[6];
char comma[3] = " ,";
char crlf[3] = "rn";
// Construct the string to be written to SDCard
txt[0] = 0;
strcat(txt, datetime);
strcat(txt, comma);
strcat(txt, temperature);
strcat(txt, comma);
strcat(txt, humidity);
strcat(txt, crlf);
if (MMC_Check_For_SDcard() == 0) { // Check for SDCard presents
SR_flg++;
if (SR_flg == 2){
Mmc_Fat_Assign(filename,0xA0); // Assign existing file
Mmc_Fat_Append(); // Add new data at the end of the file
Mmc_Fat_Write(txt,strlen(txt));// Write the string to flash
What Format do we Store it?
In the example each date / time, temperature and humidity is stored on a single, new line in the file. Each field is separated by a comma. In most cases this is a valid approach to data logging. Having the data in a CSV or Coma Separated Value file is quite convenient due to ease when importing into Excel, Libre Calc, or even visualization tools. Another option that is beyond this review but one I enjoy is the ability to data log directly to an SQL database. A discussion for another time.
How Often do we Collect the Data?
This is highly dependent on the question of how much data will I need to make this into useful information? Do you need 1 measurement per hour or 100,000 per second? In the example they are data logging every second. How I can tell is the by the following line in project.c :
if (old_seconds != t.sec){ // Occurs every new second
old_seconds = t.sec;
// data logging magic here
Every second that changes the measurements are taken and written to the medium. For temp and humidity this might be a little too often. For instance, if I were going to place this unit on a motor to measure its vibration and torsion I would want this measurement to be at least 100 x per second if I were looking for low resonance vibrations. You know, they kind that break motors. If I were looking for more of the higher frequency vibrations that loosen nuts and bolts on a motor I might be wanting 20,000 x per second. It's all up to the question you are trying to answer.
Another consideration is power. The more often you measure, the more often you'll be replacing batteries. Power is always a consideration with running on 2 AA batteries.
For How Long do we Need the Data Collected?
Another subjective question but one that can be answered in another question. How many events do I need to sufficiently prove the occurrence of a condition or event? The more rare the occurrence the longer the measurement time. If you are looking for that event that happens once a week, your data logging will need to be long enough to capture several of those events in order to prove a re-occurring trend. 1 event usually isn't enough data to prove real information.
Lastly, consider the storage requirement when data collection periods are high and data collection frequency is high as well.
In the example:
- collection is 1/sec
- 3 pieces of data are collected with a total of 32 bytes per log
- Data time "28/03/2016 14:15:23," is 20 bytes
- Temperature "28.2," is 5 bytes
- Humidity "45.5," is 5 bytes
- "/r/n" is 2 bytes
Let's consider a 24 hour period, 24 hours x 60 min x 60 sec = 86,400 samples per day x 32 bytes per sample = 2.76 Mb
But... if you were going to collect 10,000 of these samples per second.
86,400 x 10,000 = 864,000,000 x 32 bytes = 27,648,000,000 or 27.6 Gb oh my.
Additional Features
The additional information that you get with the Make project is the fact that visual feedback of the data is displayed on a TFT. This can be advantageous if you are using your data in a more immediate way. Think oscilloscope or the word meter. Even short periods of data logging can be useful.
Summary
With the world overflowing in data, let's add to the ocean by collecting the data that is useful for your own purposes. A microcontroller is the easiest and most cost efficient way of collecting, storing, and retrieving data from. Picking a strategy for data logging is like picking the right tool for any job. Power, sensors, and storage are the primary questions that will determine the appropriate equipment.