<?php
$content = "This is a test string, which is used for
demonstrating the tokenization using PHP. PHP is a very (strong) scripting-language";
$words = array_unique(str_word_count(preg_replace('/-/', ' ', $content), 1));
print "<pre>";
print_r($words);
print "</pre>";
?>
<pre>Array
(
[0] => This
[1] => is
[2] => a
[3] => test
[4] => string
[5] => which
[7] => used
[8] => for
[9] => demonstrating
[10] => the
[11] => tokenization
[12] => using
[13] => PHP
[17] => very
[18] => strong
[19] => scripting
[20] => language
)
</pre>
Output for 8.3.5
Warning: PHP Startup: Unable to load dynamic library 'sodium.so' (tried: /usr/lib/php/8.3.5/modules/sodium.so (libsodium.so.23: cannot open shared object file: No such file or directory), /usr/lib/php/8.3.5/modules/sodium.so.so (/usr/lib/php/8.3.5/modules/sodium.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
<pre>Array
(
[0] => This
[1] => is
[2] => a
[3] => test
[4] => string
[5] => which
[7] => used
[8] => for
[9] => demonstrating
[10] => the
[11] => tokenization
[12] => using
[13] => PHP
[17] => very
[18] => strong
[19] => scripting
[20] => language
)
</pre>