FAT32_EOF Question

General discussion on mikroPascal PRO for PIC32.
Post Reply
Author
Message
Rotary_Ed
Posts: 756
Joined: 26 Dec 2004 23:10
Location: Matthews, NC, USA
Contact:

FAT32_EOF Question

#1 Post by Rotary_Ed » 23 Nov 2014 20:23

I have two questions regarding the FAT32_EOF function with the MK PIC32 Pascal compiler.

The first is shown in the following code which reads data from the boards FRAM external non-volatile memory into the Write_Buffer and then onto a Flash Drive.

I initially had the read/write procedure ( 2** in code) stop when the incrementing chunks ( index_Fram := Index_Fram + sizeof(Write_Buffer);) of data exceed the last valid data write given by variable Mem_Next. This mostly works. However, they could be times that mem_next is not the largest address written to as there is an option where the data written to the FRAM may overwrite old data such that the new data is overwriting in the address of older data. So in that case Mem_next may be below the address write (after the wrap around) of older data which would mean this limit would not read the older (not yet written over data).

Therefore, I thought that if I put an EOF marker at the largest used memory address and read until that marker that I would get all of the data. However, when I compile 1**. in the code, it compiles without error, but unrelated code (such as the display code) no longer works. If looks like when the procedure hits the EOF marker it should return a 1 and drop out of the loop. While it may do that, without the display working I can not tell.

If I switch back to 2**. then it all (including the display) works correctly - so I figure it must be some basic misunderstanding on my part about what the FAT32_EOF function does.

This leads to an associated question. Do I even need to insert an EOF marker at the highest/largest data recorded point in the FRAM memory and if so -

what is the EOF marker for FAT32.

The best I could find from outside sources is that it is normally 0x0fffffff - for a 32 bit EOF marker. But, it was also noted that not all FAT routines necessarily comply with that format. So does anyone know what it is for the MK PIC32 Pascal compiler??

Any suggestions welcome

Thanks folks

Code: Select all

   ........................
   ..........
             fHandle := FAT32_Open(fileName,FILE_APPEND_F32});  // 15 Oct 2014
               Writebuffer := @Write_Buffer;   //Points to data in buffer from FRAM external non-volatile memory
               if (fHandle >= 0) then        // was the file opened successfully?
                  begin   // write bytes in Writebuffer to Flash Drive
                     //Loop loading data (EEPROM_To_Write_Buffer) from FRAM into Write_Buffer array  in
                     // Fram_chunk bytes size and then append ( FAT32_Open(fileName,FILE_APPEND_F32))
                     // to Flash Drive (FAT32_Write) File until the index pointer to FRAM equal
                     // or exceeds Mem_Next (last data address written)

                     repeat   //Transfer data from FRAM to Flash Drive
                     
                     EEPROM_TO_WRITE_BUFFER;   // 29 Sept 2014 Get Fram_chunks bytes of FRAM data
                     FAT32_Write(fhandle,WriteBuffer,Fram_Chunk);  // 29 Sept 2014 30 Aug 2014
      
                   
                    1**.  until (FAT32_Eof(fhandle) >= 1) ;  //Transfer data until end of FRAM File - PROBLEM Here
                    2**.   until index_FRAM >= (Mem_next);  // write until all of FRAM holding data is written - WORKS
                     
                     FAT32_Close(fhandle);
                     screen_erase;
                     OLED_OUT(20,2, 4,line_color,'FILE WRITTEN'); // if successful
                     Delay_ms(1000);
                     screen_Erase;

                  end
Rotary_Ed
Matthews, NC USA
Rv-6A N494BW Rotary Powered

Dany
Posts: 3854
Joined: 18 Jun 2008 11:43
Location: Nieuwpoort, Belgium
Contact:

Re: FAT32_EOF Question

#2 Post by Dany » 23 Nov 2014 20:49

Hi,
I think one can not use Fat32_Eof during an appending operation:

Extract of help:

function FAT32_Eof (fHandle : short) : int8;
Returns
1 - the cursor is at the end of file.
0 - the cursor is not at the end of file.
-1 - there was an error during function call.

function FAT32_Open (fn : ^char; mode : uint8) : int8;
...
In append mode, file cursor is moved at the end of the file.

Fat32_Eof is only meant to be used in read mode... I think.
I also think there is no "EOF marker" in a file. "Eof" simply means that the cursor is pointing beyond the the last byte in the file (all content is read...). Since Fat32 knows the size of the file it knows if one will try to read beyond it...
Kind regards, Dany.
Forget your perfect offering. There is a crack in everything, that's how the light gets in... (L. Cohen)
Remember when we were young? We shone like the sun. (David Gilmour)

Rotary_Ed
Posts: 756
Joined: 26 Dec 2004 23:10
Location: Matthews, NC, USA
Contact:

Re: FAT32_EOF Question

#3 Post by Rotary_Ed » 23 Nov 2014 23:33

Hi Dany,

I had been using Write rather than Append - but that was when the data file was small enough - just could not create a holding buffer 256KB large. So made the write_buffer 512 bytes and move it up the FRAM address chain reading the data in chunks of 512 bytes and then writing to Flash Drive. So had to resort to "Append" to keep the 512byte chunks in one Flash Dr file.

Well, your explanation would certainly explain why I can't get it to work. I had read up and thought I understood the eof marker for FAT32 was 0x0FFFFFFF. But, if it is only used for Reading a file then that would certainly prevent me from using it as I planned. I had thought I could insert the 0x0FFFFFFF at the end of the data and then just read the FRAM EOF marker to the Flashdrive and that would stop the transfer.

However, I don't think the FAT32_Write routine already knows the total size of the data in FRAM to be written to Flash drive. All it really knows is the size of my Write_buffer (512 bytes), but it will repeatedly extract and write 512 byte data chunks until it somehow knows it has them all.

How do it know??

I had thought to put the end of file marker in the FRAM data which when written to the flash drive record would then terminate the transfer. The total FRAM memory capacity is 128KB - but, the size of the flash drive record correlates to the actually size of the FRAM data stored -- so the FAT32_Write routine somehow know the where the data in the FRAM ends and does not just read to the end (capacity). Right now it is apparently because I tell it with Mem_next (next available address to write to). But, when the FRAM address pointer wraps back around, then Mem_next is now less than extend of the data it is trying to overwrite meaning if I only copy up to the current (wrapped around) Mem_next address, it seems I would lose the old data beyond that point.

Looks like I need to re-think the approach. Thanks as always, Dany, for your insight and suggestions.
Rotary_Ed
Matthews, NC USA
Rv-6A N494BW Rotary Powered

Rotary_Ed
Posts: 756
Joined: 26 Dec 2004 23:10
Location: Matthews, NC, USA
Contact:

Re: FAT32_EOF Question

#4 Post by Rotary_Ed » 24 Nov 2014 16:55

Hi Dany,

Doing a bit more research on EOF, I found this:

...now there comes the final part, it (FAT system) writes desired EOF value, but into FAT table, not into the "data part" of the HDD. So when the file is read next time, it knows this is the end, don´t look any further.
It appears that the last cluster associate with the file record has the "EOF" marker, and is located in the FAT table and that is put there by the FAT system. So me "manually" adding an EOF marker in my data does nothing and it is basically treated as more data.

So back to the drawing board.

It appears that I must find another way to deal with the overwrite and associated addresses.
It looks like I need a small routine to keep track of the maximum value of Mem_next so that when Mem_next rolls over and back to the lowest address of my FRAM and becomes less than the maximum of Mem_next that my system will check Max_mem_next and extract data from lowest address to Mem_next or Max_Mem_Next (if larger than mem_next indicating an overwrite condition).

Thanks again, Dany
Rotary_Ed
Matthews, NC USA
Rv-6A N494BW Rotary Powered

Dany
Posts: 3854
Joined: 18 Jun 2008 11:43
Location: Nieuwpoort, Belgium
Contact:

Re: FAT32_EOF Question

#5 Post by Dany » 24 Nov 2014 19:50

It appears that the last cluster associate with the file record has the "EOF" marker, and is located in the FAT table and that is put there by the FAT system. So me "manually" adding an EOF marker in my data does nothing and it is basically treated as more data.
Yes, the fat is a linked list, each entry containing the number of the next cluster of the file. Of course the last cluster has no "link" to the next one, hence the special "no next cluster" value in the fat.
The fat is completely maintained by the Fat32 filesystem, the user has (should have) no control over it.
It appears that I must find another way to deal with the overwrite and associated addresses.
It looks like I need a small routine to keep track of the maximum value of Mem_next so that when Mem_next rolls over and back to the lowest address of my FRAM and becomes less than the maximum of Mem_next that my system will check Max_mem_next and extract data from lowest address to Mem_next or Max_Mem_Next (if larger than mem_next indicating an overwrite condition).
I must say I do not understand this... :oops:
Kind regards, Dany.
Forget your perfect offering. There is a crack in everything, that's how the light gets in... (L. Cohen)
Remember when we were young? We shone like the sun. (David Gilmour)

Rotary_Ed
Posts: 756
Joined: 26 Dec 2004 23:10
Location: Matthews, NC, USA
Contact:

Re: FAT32_EOF Question

#6 Post by Rotary_Ed » 24 Nov 2014 22:28

My apologies, Dany for my poor explanation about what I intend to try.

Let me try again. The FRAM external memory chip I am using has approx. 128KB of storage capacity. The data storage routine has two modes: Overwrite and Non-overwrite. In the normal Overwrite mode, the system will store data to the FRAM chip until it reaches its maximum address at a nominal 128KB position. Then the address pointer Mem_next will wrap back around to the lowest address and the new data will start to overwrite the old (previously written) data from the lowest address toward the highest address. The current position being written with data is always pointed to by Mem_Next.

The non-overwrite mode is fairly trival in that when the address pointer reaches the chips capacity - the data write ceases. But, on to the Overwrite mode.

So lets say the FRAM has been written to max capacity once in Overwrite Mode and the pointer mem_next has started to write over the old data at the lowest address. Now I have data from lowest to max capacity with the lower part being steadily overwritten by new data. When I go to down-load the FRAM data to the Flash Dr, I do not wish to transfer only the new data (with the current byte address pointed to by Mem_next), but to also load the older data above/great than that address. But, if I let the FAT32_Write routine be terminated at Mem_Next then I only get the newest overwriting data and none of the old data above that address.

So to get the old data, I create a new variable Max_Data_mem, when the first data write reaches the max capacity (or upper limit that I permit the chip to be written to) - this variable is set Max_Data_Mem := Mem_next. Mem_next then wrap around to the lowest address and starts to over write the old data. But now I have the FAT32_Write procedure set to write until it reaches Max_Data_mem address. So it will now transfer all of the new data below the current Mem_next address and all of the old data above that address up to Max_Data_Mem which now contains the maximum address of the first data write (which is above Mem_next). I think that would work.

I will try it and see if I understand what I mean to do even if I do not explain it very well.

Best Regards

Ed
Rotary_Ed
Matthews, NC USA
Rv-6A N494BW Rotary Powered

Dany
Posts: 3854
Joined: 18 Jun 2008 11:43
Location: Nieuwpoort, Belgium
Contact:

Re: FAT32_EOF Question

#7 Post by Dany » 25 Nov 2014 15:37

Rotary_Ed wrote:So to get the old data, I create a new variable Max_Data_mem, when the first data write reaches the max capacity (or upper limit that I permit the chip to be written to) - this variable is set Max_Data_Mem := Mem_next. Mem_next then wrap around to the lowest address and starts to over write the old data. But now I have the FAT32_Write procedure set to write until it reaches Max_Data_mem address. So it will now transfer all of the new data below the current Mem_next address and all of the old data above that address up to Max_Data_Mem which now contains the maximum address of the first data write (which is above Mem_next). I think that would work.
Yes. It is the same mechanism that is used if circular (fifo) buffers. They also have an "overflow" of the read/write pointers from maximum to zero.

A question: the file you write the memory contents to will always grow in size, is this what you actually want?

In fact there is no relation/identification possibility of the data in memory and the data on sd card regarding its position, since data is always written to the end of the file, regardless its "source" position in memory.

With what purpose do you save the data to sd-card, a log file?
Kind regards, Dany.
Forget your perfect offering. There is a crack in everything, that's how the light gets in... (L. Cohen)
Remember when we were young? We shone like the sun. (David Gilmour)

Malcolm_M
Posts: 184
Joined: 12 May 2007 13:01
Location: Bristol, UK

Re: FAT32_EOF Question

#8 Post by Malcolm_M » 25 Nov 2014 16:08

Hi Rotary_Ed,

From your posts I see that you have working event logging software but it may be that there are some useful ideas below.

I have used the following Event Logging system in a number of projects and it is designed to be simple and robust but is not memory efficient (memory is reasonable inexpensive).

Code: Select all

Memory Dump:

ENTER for next 512bytes / ESC to exit

    0000 6F30303030303030370F200F120B0E43 o00000007 15:32:15 18/11/14
    0010 6F30303030303030380F200F120B0E43 o00000008 15:32:15 18/11/14
    0020 6F30303030303030390F200F120B0E43 o00000009 15:32:15 18/11/14
    0030 6F30303030303030410F200F120B0E43 o0000000A 15:32:15 18/11/14
    0040 6F30303030303030410F200F120B0EFF o0000000A 15:32:15 18/11/14
    0050 693030313030383045121223120B0E43 i0010080E 18:18:35 18/11/14
    0060 613030303030303030121231120B0E43 a00000000 18:18:49 18/11/14  <-- Last / newest event
    0070 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF _________ --:--:-- --/--/--  <-- End of event log marker, next event will be written here
    0080 6930303030303030380C112A1C0A0E43 i00000008 12:17:42 28/10/14  <-- First / oldest event
    0090 6930303030303030390C112C1C0A0E43 i00000009 12:17:44 28/10/14
    00A0 6930303030303030410C112C1C0A0E43 i0000000A 12:17:44 28/10/14
    00B0 6930303130303030360C112D1C0A0E43 i00100006 12:17:45 28/10/14
    00C0 6930303130303030370C112D1C0A0E43 i00100007 12:17:45 28/10/14
    00D0 6930303130303030380C112D1C0A0E43 i00100008 12:17:45 28/10/14
    00E0 6930303130303030390C112D1C0A0E43 i00100009 12:17:45 28/10/14
    00F0 6930303130303030410C112D1C0A0E43 i0010000A 12:17:45 28/10/14
    0100 6930303130303030330C112E1C0A0E43 i00100003 12:17:46 28/10/14
    0110 6930303130303030340C112E1C0A0E43 i00100004 12:17:46 28/10/14
    0120 6930303130303030350C112E1C0A0E43 i00100005 12:17:46 28/10/14
    0130 6930303130303030310C11311C0A0E43 i00100001 12:17:49 28/10/14
    0140 6F30303030303030390C16151C0A0E43 o00000009 12:22:21 28/10/14
    0150 6F30303030303030410C16161C0A0E43 o0000000A 12:22:22 28/10/14
    0160 6F30303130303030360C16161C0A0E43 o00100006 12:22:22 28/10/14
    0170 6F30303130303030370C16161C0A0E43 o00100007 12:22:22 28/10/14
    0180 6F30303130303030380C16161C0A0E43 o00100008 12:22:22 28/10/14
    0190 6F30303130303030390C16161C0A0E43 o00100009 12:22:22 28/10/14
    01A0 6F30303130303030330C16161C0A0E43 o00100003 12:22:22 28/10/14
    01B0 6F30303130303030340C16161C0A0E43 o00100004 12:22:22 28/10/14
    01C0 6F30303130303030350C16161C0A0E43 o00100005 12:22:22 28/10/14
    01D0 6F30303130303030310C16161C0A0E43 o00100001 12:22:22 28/10/14
    01E0 6F30303130303030320C16161C0A0E43 o00100002 12:22:22 28/10/14
    01F0 6F30303030303030310C16171C0A0E43 o00000001 12:22:23 28/10/14
1) Events are of fixed length.
2) The first byte of each event is the event type e.g. low oil pressure, high temperature etc. with $FF reserved for empty event/ End of event log marker.
3) The end of event log marker is written in the same memory as the events so seperate start and end of event log markers are not required.
4) After reset, a procedure searches for the last event by testing the first byte (every 16th byte in my case) of each possible event until $FF is found.

Code: Select all

//
// Finds last Logged Event by checking the 1st byte of each 16 byte event and points to next 
// Event address to be written
//
// 'End_of_Log' is the last address of EEProm/ Fram
//
procedure Find_Last_Log_Event;

begin
   SPI_EE_End_Log_Add := $0000;
   byte_0 := 00;

   while byte_0 <> $FF do
      begin
         SPI_EE_Add := SPI_EE_End_Log_Add;
         Read_SPI_FRam; // Returns contents of FRam address 'SPI_EE_Add' in 'byte_0'
         if SPI_EE_End_Log_Add >= End_of_Log then break;
         SPI_EE_End_Log_Add := SPI_EE_End_Log_Add + $10; // Event 16 bytes long
      end;
   
// Undo last 'SPI_EE_End_Log_Add' INC
   SPI_EE_End_Log_Add := SPI_EE_End_Log_Add - $10;

end;
5) To erase the log $FF is written to the first byte of each possible event.

Code: Select all

//
// Writes $FF to 1st byte of each 16 byte event
//
procedure Log_Set_FF;

begin
   SPI_EE_Erase_Add := $0000;
   SPI_EE_Data := $FF;
   
   while TRUE do
      begin
         
         SPI_EE_Add := SPI_EE_Erase_Add;

         Write_SPI_Fram; // Writes 'SPI_EE_Data' to 'SPI_EE_Add'

         SPI_EE_Erase_Add := SPI_EE_Erase_Add + $10; // Event 16 bytes long

         if SPI_EE_Erase_Add > End_of_Log then break;
      end;
end;
6) The date is included in each event which avoids the complication of writing a date change event.

7) Different pointers are used for different operations on the event log: SPI_EE_End_Log_Add, SPI_EE_Erase_Add, SPI_EE_View_Add etc.

8) To make the end of event log marker easy to spot in a memory dump the whole of the event is filled with $FF even though only the first byte being $FF is a requirement.

Finally FIFO/ circular buffers/ Overwrite Mode are tricky at best :D

Rotary_Ed
Posts: 756
Joined: 26 Dec 2004 23:10
Location: Matthews, NC, USA
Contact:

Re: FAT32_EOF Question

#9 Post by Rotary_Ed » 25 Nov 2014 17:15

Hi Malcolm

Thanks for sharing your code and thoughts - can't have too many good ideas to choose from!
Finally FIFO/ circular buffers/ Overwrite Mode are tricky at best :D
I certainly agree with your assessment! :)

I need to examine your data structure and see where/how I can apply it.

Thanks again
Rotary_Ed
Matthews, NC USA
Rv-6A N494BW Rotary Powered

Rotary_Ed
Posts: 756
Joined: 26 Dec 2004 23:10
Location: Matthews, NC, USA
Contact:

Re: FAT32_EOF Question

#10 Post by Rotary_Ed » 25 Nov 2014 17:33

Hi Dany,

Good catch! Yes, with using FAT32_Write(Append) the file would continuously grow.

However, what you did not see is my code that changes the file name every time the flash drive is inserted. There is a one-up number (XXX) added to its title (LOGXXX.CVS). So the first time you download, the file name is LOG1.CVS and the 250th time the file name is LOG250.CVS - so the original file does not grow continuously. Then after 250 it rolls back to 1 - or you can manually reset it at any time. The data stored in the FRAM external memory can also be erased by pushing a menu screen button.

I keep user preferences and other program data/constants/default values in the first 1024 bytes and the data logging area is all memory above that point. So I only erase the area above 1024 address to erase the stored data.

Really appreciate your keen eye catching such things.
Rotary_Ed
Matthews, NC USA
Rv-6A N494BW Rotary Powered

Rotary_Ed
Posts: 756
Joined: 26 Dec 2004 23:10
Location: Matthews, NC, USA
Contact:

Re: Data Logged was FAT32_EOF Question

#11 Post by Rotary_Ed » 26 Nov 2014 01:00

Ok,

This shows the spreadsheet into which I have transferred the data logged to the FRAM memory chip and downloaded to a Flash Drive. This data is with the Max_mem set to only 5K bytes (so I would not have to wait over an hour for the data to wrap around) whereas the chip can actually store around 128KB of data.

The first section is the configuration data for an electronic ignition system for an experimental aircraft (there is a Right (R) and Left(L) ignition). This is the complete configuration that is current set in the two ignitions. That data is recorded once per flight, before engine start. Then after engine start there is the periodically reported data by each ignition (refer to as streaming data as it streams out of each ignition without having to query the ignition). This data is recorded at a once per second nominal rate the Time variable is seconds after engine start. This set of data shows that it "wrapped around" approx. 77 seconds after engine start. With the full chip capacity used for recording this figure would be more like 1 hour and 20 minutes. If it had not yet wrapped around you could tell because the time would be showing "0,1,2,3,4,5...." in the first few rows right after engine starts.

Each segment of data has its own column header file (text) with an indication of what data is located in that column. For example Pmag (the electronic ignition) will have an "L" or "R" indicating which ignition's data is logged on that line, other columns give the reported "RPM", "AdvC (ignition timing in deg), "Volts", Coil1 (a value indicating the health of the coil, wire and spark plug pair - therefore two values for two pairs for four plugs). and so forth.

Now show in this data is a feature that monitors the timing mark for both ignitions and if they should diverge (I call it the Timing Divergence Angle or TDA for short) by more 4.5 deg, a TDA mark is placed in the Right ignitions identifier column and the value of the TDA is continuously recorded as long as it stays above 4.5 deg. This divergence could be caused by several factors - such as gear wear, failed component in the system, etc.

My customers will have access to this data to examine should any ignition anomaly occur. The user can also forward the file to me for more detail analysis - the objective is to maintain knowledge of the status and health of the ignitions - somewhat important in an aircraft as you might imagine :D

About the only real difficulty at this time is the apparent shortage :( of the large capacity FRAM chips I am using.
Wrap around Data
Wrap around Data
Data Logged.jpg (137.44 KiB) Viewed 7688 times
Rotary_Ed
Matthews, NC USA
Rv-6A N494BW Rotary Powered

Rotary_Ed
Posts: 756
Joined: 26 Dec 2004 23:10
Location: Matthews, NC, USA
Contact:

Re: FAT32_EOF Question

#12 Post by Rotary_Ed » 26 Nov 2014 14:45

Sorry, had a mistype in previous message.

The sentence
Now show in this data is a feature that monitors the timing mark for both ignitions and if they should diverge (I call it the Timing Divergence Angle or TDA for short) by more 4.5 deg, a TDA mark is placed in the Right ignitions identifier column and the value of the TDA is continuously recorded as long as it stays above 4.5 deg. This divergence could be caused by several factors - such as gear wear, failed component in the system, etc.
should read " NOT SHOWN in this data .... " rather than "Now Show".

This image shows the TDA value being written in the R Pmags identification column. The Pilot is also provided with a bar graph and numerical value display of the TDA. Should the TDA value exceed 6 deg then a attention getting warning message is flashed on the display. Anything 6 deg and beyond is considered unacceptable and a landing is advised and investigation to determine the cause should be made by the pilot. In one case, the increasing TDA value was due to the wearing of gears that drive the Pmag - apparently the gear's metal was not hardened properly.
TDA.jpg
TDA.jpg (93.99 KiB) Viewed 7657 times
Rotary_Ed
Matthews, NC USA
Rv-6A N494BW Rotary Powered

Post Reply

Return to “mikroPascal PRO for PIC32 General”