UseMod Wiki: WikiPatches/UTF8

Note: This is included in the Usemod 1.0 release see UseModWiki/NewFeatures,

I think this patch's name is a bit misleading. --- If people really interested in starting a new Usemod wiki that uses UTF-8 from the beginning, this patch is not needed at all. In fact, the trick implemented in patches does not solve most of the problem if the wiki that plans on using UTF8 to support Chinese, for example

For a better solution in the above case, people may want to read SupportForUtf8.

I have added most of the patch below to the current development script (0.97-prerelease). The changes I made were:

Not changing the $HttpCharset (since UTF8 can cause problems with some browsers, it should be explicitly enabled by the site admin)
Changing the $FS-escaping code to read like $text =~ s/$FS(\d)/$1/g; so that it acts like the old code, but only when the $FS is followed by a digit.

--CliffordAdams

I disagree with "\d" idea because it increases a chance that UTF-8 characters are destroyed. Is there any case that $FS is followed by other than 1, 2, and 3? And I agree UTF-8 is disabled by default.

Sounds like I'd better get development script before posting a patch. Do you have any plan to make it public (I mean code, not site)? I made a patch below because I need to apply it to my wiki site. --TakuyaMurata

Here is a tiny patch to enable UseMod supports UTF-8 [1] charset. Depite it doesn't completely supports UTF-8, I put this patch for the sake of those who need to use UTF-8 right away like me. See discussion at SupportForUtf8

Note:

Nicely, you don't need to change the database format at all.
Most of words are supported. Most are all of words except ones containing 0xb311, 0xb312 and 0xb313.

--- wiki_92.pl	2002-12-22 12:01:49.000000000 -0600
+++ wiki_utf8.cgi	2002-12-23 10:44:38.000000000 -0600
@@ -82,7 +82,7 @@
 $EditNote    = "";              # HTML notice above buttons on edit page
 $MaxPost     = 1024 * 210;      # Maximum 210K posts (about 200K for pages)
 $NewText     = "";              # New page text ("" for default message)
-$HttpCharset = "";              # Charset for pages, like "iso-8859-2"
+$HttpCharset = "UTF-8";         # Charset for pages, like "iso-8859-2"
 $UserGotoBar = "";              # HTML added to end of goto bar

 # Major options:
@@ -986,6 +986,7 @@
     return $q->header(-cookie=>$cookie);
   }
   if ($HttpCharset ne '') {
+    $q->charset($HttpCharset);
     return $q->header(-type=>"text/html; charset=$HttpCharset");
   }
   return $q->header();
@@ -1157,7 +1157,7 @@
   %SaveNumUrl = ();
   $SaveUrlIndex = 0;
   $SaveNumUrlIndex = 0;
-  $pageText =~ s/$FS//g;              # Remove separators (paranoia)
+  $pageText =~ s/$FS[123]//g;              # Remove separators (paranoia)
   if ($RawHtml) {
     $pageText =~ s/<html>((.|\n)*?)<\/html>/&StoreRaw($1)/ige;
   }
@@ -1678,7 +1678,7 @@
   %SaveNumUrl = ();
   $SaveUrlIndex = 0;
   $SaveNumUrlIndex = 0;
-  $diff =~ s/$FS//g;
+  $diff =~ s/$FS[123]//g;
   $diff =  &CommonMarkup($diff, 0, 1);      # No images, all patterns
   $diff =~ s/$FS(\d+)$FS/$SaveUrl{$1}/ge;   # Restore saved text
   $diff =~ s/$FS(\d+)$FS/$SaveUrl{$1}/ge;   # Restore nested saved text
@@ -3214,8 +3214,8 @@
     &ReportError(Ts('[[%s]] cannot be defined.', $id));
     return;
   }
-  $string =~ s/$FS//g;
-  $summary =~ s/$FS//g;
+  $string =~ s/$FS[123]//g;
+  $summary =~ s/$FS[123]//g;
   $summary =~ s/[\r\n]//g;
   # Add a newline to the end of the string (if it doesn't have one)
   $string .= "\n"  if (!($string =~ /\n$/));
@@ -3820,7 +3820,7 @@
   # Much of this is taken from the common markup
   %SaveUrl = ();
   $SaveUrlIndex = 0;
-  $text =~ s/$FS//g;              # Remove separators (paranoia)
+  $text =~ s/$FS[123]//g;              # Remove separators (paranoia)
   if ($RawHtml) {
     $text =~ s/(<html>((.|\n)*?)<\/html>)/&StoreRaw($1)/ige;
   }

(redirected from WikiPatches/UtfEight)

WikiPatches/UTF8