CODESYS - the IEC 61131-3 automation software

Welcome to the official CODESYS Forum by 3S-Smart Software Solutions GmbH | A member of the CODESYS Group
Deutsche Version English version russian version 
It is currently Sun Feb 16, 2020 11:24 pm

All times are UTC+01:00




Post new topic  Reply to topic  [ 8 posts ] 
Author Message
 Post subject: CSV Writer Speed
PostPosted: Fri Jan 31, 2020 8:16 pm 
Offline

Joined: Thu Nov 10, 2016 4:18 am
Posts: 5
Hello - There are a large number of data points which need to be added to a CSV file. There is an indexing routine which cycles through all of the data points in the array and uses the CSV.AddString FB to add the strings to the buffer. When the buffer write is complete the items are saved to file using the CSV.WriteFile FB. Everything works as expected. The problem is that the process is extremely slow. The system takes 5 minutes to complete 5000 data points. The customer is requesting 45,000 data points.

I have taken the following steps to increased speed:
1. Change the write to a separate task with a cyclic time of 2ms
2. Changed the logic to only perform the WriteFile after adding 100 data points (cannot do all of the data points at once due to buffer size)
3. Tried to perform an CSV.AddString during every cycle without paying attention to the done from the FB. This did not report the data correctly.

Attached is the code I am using to process the data. Any advice on a different, faster approach would be much appreciated. Thank you for your time.
Code:
FUNCTION_BLOCK TestDataProcessorLREAL


VAR_INPUT
   bExecute: BOOL;
   InputArray             :    ARRAY[0..50000] OF LREAL;
   uiSampleNumber         :   UINT;
END_VAR
VAR_IN_OUT
   cCSVWriter      : CSV.CSVWriter;
END_VAR
VAR_OUTPUT
   bDone: BOOL;
END_VAR
VAR
   i: UINT;
   sTempString          : STRING;
   csvAddTempString       : CSV.AddSTRING;
   csvWriteFile         : CSV.WriteFile;
   bcsvAddStringExecute   :   BOOL;
   bcsvWriteFileExecute   : BOOL;
END_VAR


IF bExecute AND NOT bDone THEN
   bcsvAddStringExecute := TRUE;
   
   //Convert the input data to string to be stored by the CSV writer
   sTempString  := LREAL_TO_STRING(InputArray[i]);

   //Check to see if the previous string Add is complete, and there is not a file write requested and then reset the add string bit and index ahead 1
   IF csvAddTempString.xDone AND NOT bcsvWriteFileExecute THEN
      bcsvAddStringExecute :=FALSE;
      i:=i+1;
   END_IF

   //Only call the Write file every 100 calls of the add string FB, this was done to increase speed
   IF (i MOD 100) = 0 AND i >0 AND csvAddTempString.xDone THEN
      bcsvWriteFileExecute :=TRUE;
   END_IF
   
//When the write file is complete, turn off the Execute bit
   IF csvWriteFile.xDone THEN
      bcsvWriteFileExecute := FALSE;
   END_IF
   

//These calls put the data points into the buffer and then write it to the file
CSVAddTempSTRING(sInput:=sTempString,rCSVWriter:=cCSVWriter,xExecute:=bcsvAddStringExecute);
csvWriteFile(xExecute:=csvAddTempString.xDone,rCSVWriter:=cCSVWriter);


END_IF


//Reset the index and stop execution if
IF NOT bExecute THEN
   bcsvAddStringExecute :=FALSE;
   i := 0;
   bDone := FALSE;
END_IF


//Set the DONE bit when the end of array index is reached
IF i >= uiSampleNumber AND bExecute THEN
   bDone := TRUE;
END_IF


Top
   
 Post subject: Re: CSV Writer Speed
PostPosted: Tue Feb 04, 2020 11:09 am 
Offline

Joined: Wed Jan 08, 2020 1:50 pm
Posts: 18
You are inserting one point per task cycle, so if you have a 5 ms cycle time, it will take 250 seconds (4 minutes and 10 seconds) to insert 50k points.
How about a FOR loop calling CSVAddTempSTRING 100 times in a single cycle task? I would bet you will speed up operations by a similar factor (down to about 3 seconds).
Furthermore, IIRC VAR_INPUT mandates a copy of array is made upon each invocation. Try passing the array as VAR_IN_OUT CONSTANT to avoid copies.


Top
   
 Post subject: Re: CSV Writer Speed
PostPosted: Wed Feb 05, 2020 12:06 am 
Offline

Joined: Thu Nov 10, 2016 4:18 am
Posts: 5
Gronchi - Thank you for the response. Unfortunately it does not work to call the AddString FB (instance name is CSVAddTempString) repeatedly. I tried this method in the beginning. You have to wait for the Done bit of the FB before updating the input and calling it again. Otherwise it does not finish writing the data to the buffer and a lot of data goes missing.

You are correct that there is a better way to pass the data around: a pass by reference would probably be best. However, there is no memory shortage so I haven't tackled this yet.


Top
   
 Post subject: Re: CSV Writer Speed
PostPosted: Wed Feb 05, 2020 6:12 am 
Offline
Site Admin

Joined: Mon Sep 05, 2005 9:42 am
Posts: 3623
Hi,
on which plc is this?

BR
Edwin


Top
   
 Post subject: Re: CSV Writer Speed
PostPosted: Wed Feb 05, 2020 9:32 am 
Offline
Frequent User
Frequent User

Joined: Fri Feb 23, 2018 3:41 pm
Posts: 180
I don't get this :
Code:
bcsvWriteFileExecute :=TRUE;

Code:
bcsvWriteFileExecute :=FALSE;

Code:
csvWriteFile(xExecute:=csvAddTempString.xDone,rCSVWriter:=cCSVWriter);


Feels like it is still writing every cycle (beging that CSVAddTempSTRING is a synchronous fb).


Top
   
 Post subject: Re: CSV Writer Speed
PostPosted: Wed Feb 05, 2020 6:26 pm 
Offline

Joined: Thu Nov 10, 2016 4:18 am
Posts: 5
dFx wrote:
I don't get this :
Code:
bcsvWriteFileExecute :=TRUE;

Code:
bcsvWriteFileExecute :=FALSE;

Code:
csvWriteFile(xExecute:=csvAddTempString.xDone,rCSVWriter:=cCSVWriter);


Feels like it is still writing every cycle (beging that CSVAddTempSTRING is a synchronous fb).


That was another attempt to speed it up, although probably not to much effect. In the beginning I was trying to stay away from a state machine as that would inherently add one more scan to each change of state. However this became hard to manage, so i re-wrote the code in a CASE statement. It is much easier to follow now.... with one major change, I am concatenating a series of the data points into a single string and manually adding the commas. In this way, the WriteFile still has each data point separated by a comma but significantly reduces the number of AddString calls that need to be made. The limit on this is the CONCAT function can only take 255 characters at one time. It is much faster now, but still not fast enough. What I probably need to do is allocate an area of memory which is as large as the buffer for the CSV writer and fill that completely, then write to file. This should be fast enough.

Code:
IF  NOT bExecute  THEN
   step_number := 0;
   bDone := FALSE;
END_IF


   CASE step_number OF
      0:
         i:=0;
         IF bExecute THEN
            step_number := 10;
         END_IF
      10:
         i_step := i + 19;
         sTempString := '';
         FOR i := i TO i_step DO
            sTempString  := Concat(STempString, Concat(LREAL_TO_STRING(InputArray[i]), ','));
         END_FOR
         sTempString  := Concat(STempString, LREAL_TO_STRING(InputArray[i]));
         step_number := 20;
      20:
         bcsvAddStringExecute := TRUE;
         IF csvAddTempString.xDone THEN
            bcsvAddStringExecute := FALSE;
            step_number := 30;
         END_IF
      30:
         IF (i MOD 100) = 0 THEN
            bcsvWriteFileExecute :=TRUE;
            step_number := 40;
         ELSE
            step_number := 10;
         END_IF
      40:
         IF csvWriteFile.xDone THEN
            bcsvWriteFileExecute :=FALSE;
            IF i >= uiSampleNumber THEN
               bDone := TRUE;
               ELSE
               step_number := 10;   
            END_IF
         END_IF
      
         
      
   END_CASE
   
   
   

   

//These calls put the data points into the buffer and then write it to the file
CSVAddTempSTRING(sInput:=sTempString,rCSVWriter:=cCSVWriter,xExecute:=bcsvAddStringExecute);
csvWriteFile(xExecute:=bcsvWriteFileExecute,rCSVWriter:=cCSVWriter);


Top
   
 Post subject: Re: CSV Writer Speed
PostPosted: Wed Feb 05, 2020 6:56 pm 
Offline

Joined: Thu Nov 10, 2016 4:18 am
Posts: 5
Edwin Schwellinger wrote:
Hi,
on which plc is this?

BR
Edwin



This is on a Parker Automation Controller.


Top
   
 Post subject: Re: CSV Writer Speed
PostPosted: Tue Feb 11, 2020 8:57 am 
Offline
Frequent User
Frequent User

Joined: Fri Feb 23, 2018 3:41 pm
Posts: 180
Strings are no more than byte arrays ending with a 16#0 value. To reduce your treatment, just manage by yourself the length and offset for new string write inside a byte buffer. OSCAT lib may help for this.

Also you are still writing the csv format (with comas), you could aswell handle the cariage return and line feed.

Then, last but not least, you could try opening the file once, and then only writing the new data to it before closing it once done. This could be way more efficient thant opening and closing the file at each write.


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 8 posts ] 

All times are UTC+01:00


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Limited