Tokenize()

Description

Tokenize() is used to break a single string into a list of strings.

Call Signature

mstring Tokenize ( string input_string [, string delimiters] )

Return Value

mstring

An mstring containing the tokens of the string.

Arguments

string input_string:

The input is a string expression.

[ string delimiters ]:

A list of characters to use as token delimiters. If not specified, any white space will be considered a delimiter. This parameter is optional - see Remarks.

Remarks

The string is split up based on white spaced separated tokens, with the white space being removed. White space includes spaces, tabs, and newline characters. Empty tokens are never produced. Leading and trailing white space are ignored. Multiple white space characters are treated as one.

In addition, the Tokenize() function treats portions of the input string contained in double quotes specially. The double quotes are removed, but the entire string remains as a single token regardless of white space.

 `this and that'               --> "this", "and", "that"
 `this  and that" extra"word'  --> "this", "and", "that extraword"
 `1+2 * 3'                     --> "1+2", "*", "3"

Optionally, a user specified list of delimiters can be passed. When this is done, the special treatment of quoted strings is disabled.

Example

Read input from an ASCII file with multiple space delimited data items per line and write out the data with only one data item per line.

 local int     fd_in, fd_out, i
 local string  line
 local mstring tokens

 fd_in = TextOpen( "input.txt", "r" )
 fd_out = TextOpen( "output.txt", "w" )

 line = TextRead( fd_in )
 while( line != "<EOF>" )
   tokens = Tokenize(line)
   for( i = 1 to f$len(tokens)
     call TextWrite( fd_out, tokens[i] )
   endfor 
  
   line = TextRead( fd_in )
 endwhile  

 call TextClose( fd_out )
 call TextClose( fd_in )

Split the components of a directory path apart at each slash. Note that a no token is generated for the contents of the string before the first slash, as leading (and trailing) delimiters are ignored.

 local string  path
 local mstring tokens
 local int     i

 path = GetPCIHOME()+"/demo/irvine.pix"

 tokens = Tokenize(path,"/")
 print "Filename = ", tokens[f$len(tokens)]
 
 for i = f$len(tokens) - 1 to 1 by -1
   print "in directory: ", tokens[i]
 endfor

See also

F$LEN()