Конвертирование из UTF-16 / UCS-2 в UTF-8 - Работа с текстом - Интересные решения для программистов - Примеры кода

Работа с текстом → Конвертирование из UTF-16 / UCS-2 в UTF-8 → PHP

Преобразует строку из кодировки UTF-16 / UCS-2 в UTF-8, без использования iconv

/**
	 * Convert UTF-16 / UCS-2 encoding string to UTF-8.
	 * Surrogates UTF-16 are supported!
	 * 
	 * In Russian:
	 * Преобразует строку из кодировки UTF-16 / UCS-2 в UTF-8.
	 * Суррогаты UTF-16 поддерживаются!
	 *
	 * @param    string        $s
	 * @param    string        $type      'BE' -- big endian byte order
	 *                                    'LE' -- little endian byte order
	 * @param    bool          $to_array  returns array chars instead whole string?
	 * @return   string|array|bool        UTF-8 string, array chars or FALSE if error occurred
	 */
	private static function _convert_from_utf16($s, $type = 'BE', $to_array = false)
	{
		static $types = array(
			'BE' => 'n',  #unsigned short (always 16 bit, big endian byte order)
			'LE' => 'v',  #unsigned short (always 16 bit, little endian byte order)
		);
		if (! array_key_exists($type, $types))
		{
			trigger_error('Unexpected value in 2-nd parameter, "' . $type . '" given!', E_USER_WARNING);
			return false;
		}
		#the fastest way:
		if (function_exists('iconv') || function_exists('mb_convert_encoding'))
		{
			if (function_exists('iconv'))                   $s = iconv('UTF-16' . $type, 'UTF-8', $s);
			elseif (function_exists('mb_convert_encoding')) $s = mb_convert_encoding($s, 'UTF-8', 'UTF-16' . $type);
			if (! $to_array) return $s;
			return self::str_split($s);
		}

/*
        http://en.wikipedia.org/wiki/UTF-16

The improvement that UTF-16 made over UCS-2 is its ability to encode
        characters in planes 1-16, not just those in plane 0 (BMP).

UTF-16 represents non-BMP characters (those from U+10000 through U+10FFFF)
        using a pair of 16-bit words, known as a surrogate pair.
        First 1000016 is subtracted from the code point to give a 20-bit value.
        This is then split into two separate 10-bit values each of which is represented
        as a surrogate with the most significant half placed in the first surrogate.
        To allow safe use of simple word-oriented string processing, separate ranges
        of values are used for the two surrogates: 0xD800-0xDBFF for the first, most
        significant surrogate and 0xDC00-0xDFFF for the second, least significant surrogate.

For example, the character at code point U+10000 becomes the code unit sequence 0xD800 0xDC00,
        and the character at U+10FFFD, the upper limit of Unicode, becomes the sequence 0xDBFF 0xDFFD.
        Unicode and ISO/IEC 10646 do not, and will never, assign characters to any of the code points
        in the U+D800-U+DFFF range, so an individual code value from a surrogate pair does not ever
        represent a character.

http://www.russellcottrell.com/greek/utilities/SurrogatePairCalculator.htm
        http://www.russellcottrell.com/greek/utilities/UnicodeRanges.htm

Conversion of a Unicode scalar value S to a surrogate pair <H, L>:
          H = Math.floor((S - 0x10000) / 0x400) + 0xD800;
          L = ((S - 0x10000) % 0x400) + 0xDC00;
        The conversion of a surrogate pair <H, L> to a scalar value:
          N = ((H - 0xD800) * 0x400) + (L - 0xDC00) + 0x10000;
		*/
		$a = array();
		$hi = false;
		foreach (unpack($types[$type] . '*', $s) as $codepoint)
		{
			#surrogate process
			if ($hi !== false)
			{
				$lo = $codepoint;
				if ($lo < 0xDC00 || $lo > 0xDFFF) $a[] = "\xEF\xBF\xBD"; #U+FFFD REPLACEMENT CHARACTER (for broken char)
				else
				{
					$codepoint = (($hi - 0xD800) * 0x400) + ($lo - 0xDC00) + 0x10000;
					$a[] = self::chr($codepoint);
				}
				$hi = false;
			}
			elseif ($codepoint < 0xD800 || $codepoint > 0xDBFF) $a[] = self::chr($codepoint); #not surrogate
			else $hi = $codepoint; #surrogate was found
		}
		return $to_array ? $a : implode('', $a);
	}

charset, convert, php, ucs2, utf16, utf8

KOS_MOS

Юрий Насретдинов

07.06.2008

Рэй [Добавлено 16.08.2011 12:35:25]

//elseif ($c < 0x800) $a[] = chr(0xC0 ($c >> 6)) . chr(0x80 ($c & 0x3F));
//else $a[] = chr(0xE0 ($c >> 12)) . chr(0x80 (0x3F & ($c >> 6))) . chr(0x80 //($c & 0x3F));

Что это за фигня такая а? Не пашет вообще? Какие там должны быть операторы: Между hex числами?

Павел [Добавлено 16.08.2011 15:47:49]

Рэй, исправил

Меню

Подписка на RSS:

Подпишись на RSS-ленту

Поиск

Облако тегов

	Работа с текстом → Конвертирование из UTF-16 / UCS-2 в UTF-8 → PHP Преобразует строку из кодировки UTF-16 / UCS-2 в UTF-8, без использования iconv /** * Convert UTF-16 / UCS-2 encoding string to UTF-8. * Surrogates UTF-16 are supported! * * In Russian: * Преобразует строку из кодировки UTF-16 / UCS-2 в UTF-8. * Суррогаты UTF-16 поддерживаются! * * @param string $s * @param string $type 'BE' -- big endian byte order * 'LE' -- little endian byte order * @param bool $to_array returns array chars instead whole string? * @return string\|array\|bool UTF-8 string, array chars or FALSE if error occurred / private static function _convert_from_utf16($s, $type = 'BE', $to_array = false) { static $types = array( 'BE' => 'n', #unsigned short (always 16 bit, big endian byte order) 'LE' => 'v', #unsigned short (always 16 bit, little endian byte order) ); if (! array_key_exists($type, $types)) { trigger_error('Unexpected value in 2-nd parameter, "' . $type . '" given!', E_USER_WARNING); return false; } #the fastest way: if (function_exists('iconv') \|\| function_exists('mb_convert_encoding')) { if (function_exists('iconv')) $s = iconv('UTF-16' . $type, 'UTF-8', $s); elseif (function_exists('mb_convert_encoding')) $s = mb_convert_encoding($s, 'UTF-8', 'UTF-16' . $type); if (! $to_array) return $s; return self::str_split($s); } / http://en.wikipedia.org/wiki/UTF-16 The improvement that UTF-16 made over UCS-2 is its ability to encode characters in planes 1-16, not just those in plane 0 (BMP). UTF-16 represents non-BMP characters (those from U+10000 through U+10FFFF) using a pair of 16-bit words, known as a surrogate pair. First 1000016 is subtracted from the code point to give a 20-bit value. This is then split into two separate 10-bit values each of which is represented as a surrogate with the most significant half placed in the first surrogate. To allow safe use of simple word-oriented string processing, separate ranges of values are used for the two surrogates: 0xD800-0xDBFF for the first, most significant surrogate and 0xDC00-0xDFFF for the second, least significant surrogate. For example, the character at code point U+10000 becomes the code unit sequence 0xD800 0xDC00, and the character at U+10FFFD, the upper limit of Unicode, becomes the sequence 0xDBFF 0xDFFD. Unicode and ISO/IEC 10646 do not, and will never, assign characters to any of the code points in the U+D800-U+DFFF range, so an individual code value from a surrogate pair does not ever represent a character. http://www.russellcottrell.com/greek/utilities/SurrogatePairCalculator.htm http://www.russellcottrell.com/greek/utilities/UnicodeRanges.htm Conversion of a Unicode scalar value S to a surrogate pair <H, L>: H = Math.floor((S - 0x10000) / 0x400) + 0xD800; L = ((S - 0x10000) % 0x400) + 0xDC00; The conversion of a surrogate pair <H, L> to a scalar value: N = ((H - 0xD800) * 0x400) + (L - 0xDC00) + 0x10000; / $a = array(); $hi = false; foreach (unpack($types[$type] . '', $s) as $codepoint) { #surrogate process if ($hi !== false) { $lo = $codepoint; if ($lo < 0xDC00 \|\| $lo > 0xDFFF) $a[] = "\xEF\xBF\xBD"; #U+FFFD REPLACEMENT CHARACTER (for broken char) else { $codepoint = (($hi - 0xD800) * 0x400) + ($lo - 0xDC00) + 0x10000; $a[] = self::chr($codepoint); } $hi = false; } elseif ($codepoint < 0xD800 \|\| $codepoint > 0xDBFF) $a[] = self::chr($codepoint); #not surrogate else $hi = $codepoint; #surrogate was found } return $to_array ? $a : implode('', $a); } charset, convert, php, ucs2, utf16, utf8 KOS_MOS Юрий Насретдинов 07.06.2008 Рэй [Добавлено 16.08.2011 12:35:25] //elseif ($c < 0x800) $a[] = chr(0xC0 ($c >> 6)) . chr(0x80 ($c & 0x3F)); //else $a[] = chr(0xE0 ($c >> 12)) . chr(0x80 (0x3F & ($c >> 6))) . chr(0x80 //($c & 0x3F)); Что это за фигня такая а? Не пашет вообще? Какие там должны быть операторы: Между hex числами? Павел [Добавлено 16.08.2011 15:47:49] Рэй, исправил Ваше имя Ваш E-mail (если хотите отслеживать комментарии) Дoкажи, что ты не робот Текст комментария	Меню CodeIgniter Алгоритмы Базы данных Дата и Время Интернет Интерфейс Мультимедиа ОС и Железо Работа с текстом Разное Технологии Файлы Подписка на RSS: Подпишись на RSS-ленту Поиск Облако тегов .net array asp bash browser charset check class clean convert count create css curl database date delete directory download element email equivalent extension file find form format function get height html http ie image javascript jquery load mysql number page perl php random scriplet send shell size sql string text time url utf8 variable vb visual basic width window xml xslt