Tokenize a string in C#

This article illustrates the different techniques to tokenize a string in C#.

The standard solution to break a delimited string into substrings is using the String.Split() method. It returns a string array containing the substrings that are delimited by the specified separator. For example,

using System;

public class Example

{

public static void Main()

{

String s = "Split by space";

string[] tokens = s.Split(' ');

Console.WriteLine(String.Join(", ", tokens)); // Split, by, space

}

Download Run Code

There are overloads of the Split() method that allows you to pass StringSplitOptions. For example, the following code excludes empty substrings from the resultant string array using StringSplitOptions.RemoveEmptyEntries for the second parameter.

using System;

public class Example

{

public static void Main()

{

String s = "Split by space";

string[] tokens = s.Split(' ', StringSplitOptions.RemoveEmptyEntries);

Console.WriteLine(String.Join(", ", tokens)); // Split, by, space

}

Download Run Code

The separator delimits the substrings in the string. It can be a single character, a string, a character array, or a string array. For example, the following sample uses a character array to specify the delimited strings.

using System;

public class Example

{

public static void Main()

{

String s = "C#: Split by,char.array";

char[] delims = new char[] { ' ', ',', '.', ':' };

string[] tokens = s.Split(delims, StringSplitOptions.RemoveEmptyEntries);

Console.WriteLine(String.Join(", ", tokens)); // C#, Split, by, char, array

}

Download Run Code

That’s all about tokenizing a string in C#.

Share this: