strok and compiler efficiency

[paulfjujo]

hello


strok function need a char pointer as second argument ..

Code: Select all

char * strtok(char * s1, char * s2);

what is strange : this function also accept a char or a byte constant or a variable !
I tested with ";" as field separator into a string buffer.
and i can use

Code: Select all

p1=strok(buffer,";");  // a string as separator
or 
p1=strok(buffer,';');   // a ascii char as separator
or
p1=strok(buffer,59);  // a byte constant  as separator
or  
cc=';';
p1=strok(buffer,cc);  // a variable as separator

it is due to the high efficiency of the compiler ?

link of source mikroC

[aCkO]

what is strange : this function also accept a char or a byte constant or a variable !

But that doesn't mean you will get correct results. The second argument will be implicitly converted to pointer to char, i.e. the integer variable you specified will be converted to memory address. You should get a warning about that in compiler output (messages window). Obviously, this produces bad results.

If you need a function that will extract tokens from a string based on single character delimiter, it is much more efficient to write one yourself (maybe 10-15 lines of code without additional function calls). You will probably get 4x smaller (and faster) solution than strtok. That's because strtok internally calls strspn, strcspn and strchr.

Also, strtok is useless for parsing protocols (e.g. GPS NMEA sentences) because it discards empty tokens. For example:

Code: Select all

char txt[] = ";;1;22;333;;4444";
char result[10];
char * p;

p = strtok(txt, ";");
while (p != 0) {
   strcpy(result, p);
   p = strtok(0, ";");
}

The results will be:

Code: Select all

"1"
"22"
"333"
"4444"

Normally, you would expect the following output:

Code: Select all

""
""
"1"
"22"
"333"
""
"4444"

Regards

[paulfjujo]

Code: Select all

But that doesn't mean you will get correct results

Yes, it does..

thanks for your advise, concerning the behavior of strok
compare to a classic research.. .

[aCkO]

Code: Select all

But that doesn't mean you will get correct results

Yes, it does..

No, it doesn't. Trust me.

Function signature determines the conversion type. If you specify ';' or 59 as delimiter, that will be converted to pointer to char (i.e. memory address) because that is what function expects:

ptr conversion.png (35.85 KiB) Viewed 2522 times

That means strtok will look for a string located at address 59 (0x3B). Of course, that string is usually completely random as you can see from the screenshot below:

strtok.png (150.55 KiB) Viewed 2522 times

If you are still not convinced, then let's place some string at location 59 (0x3B) using absolute directive:

Code: Select all

char delim[] = "2" absolute 59;
  
void main() {
  char result[10];
  char txt[] = "1;2;3";
  char *p;

  if (delim);   // dummy call to prevent linker from eliminating delim

  p = strtok (txt, ';');
  while (p != 0)
  {
    strcpy(result, p);
    p = strtok (0, ';');
  }
}

Output:

Code: Select all

"1;"
";3"

As you can see, the string was split using "2" as delimiter, not ';'.

I hope you are convinced now

Regards

[paulfjujo]

hello aCko ;


yes , you convinced me !

i had some doubts about that, it was the reason of my post.
So i cam back on my previous test and found my mistake..

i didn't used the last compiled Hex file for on my mikroBootloader
so it was allways the versus with ";" wich was running ..

I tested again, and in fact only a string pointer can be used as "separateur"
so i tested with
p1=strok(buffer,";");
and also by defining a string variable
char Separateur[2]; // minimum size
strcpy(Separateur,";");
p1=strok(buffer,Separateur);

Result Ok, even when adding your absolute variable at adresse 59.

Thank's for your support.

Best Reagrds

paul

Code: Select all

#ifdef Test1
  // waiting HH;MM;SS from the keyboard
  //  avec strok
      i=0;
      strcpy(Separateur,";");   // <--OK
      p1=strtok(buffer1,Separateur ); // Ok if string variable
     // p1=strtok(buffer1,";");  // 1er appel pour detection point virgule
      UART1_Write_CText("\r\n usage variable Separateur =\";\" \r\nparam #");
      UART1_Write(i+49);      // 48 +1 , car on compte plutot à partir de 1
      UART1_Write(TAB);
      UART1_Write_Text(buffer1);CRLF1();
      Time[i]=atoi(buffer1);
      Delay_ms(1000);
      i++;
          do
          {   //pour les appels suivants, pointeur null en reference
             //p1=strtok(0, ";");    // detection des separateurs (point virgule) suivants
             p1=strtok(0,Separateur);
             if (p1!=0)
              {
              UART1_Write_CText("param #");
              UART1_Write(i+49);
              UART1_Write(TAB);
              UART1_Write_Text(p1);   CRLF1();
              Time[i]=atoi(p1);
              Delay_ms(1000);
              i++;
              }
           } while (p1!=0) ;
           CRLF1();
           Delay_ms(1000);
        // verification stockage valeur numeriques
        for (i=0 ;i<3;i++)
        {
            ByteToStr(Time[i],CRam1);
            strConstRamCpy(txt,"Time[ ] = ");
            *(txt+5)=i+48;   // indice en ascii
            strcat(txt,CRam1); // rajoute la valeur convertie en ascii prealablement
            UART1_Write_Text(txt);
            CRLF1();
          }
   #endif

on terminal display

Reception de :
10;23;36

usage variable Separateur =";"
param #1 10
param #2 23
param #3 36

Time[0] = 10
Time[1] = 23
Time[2] = 36

Somme Time = 37416
End of test