模組:Unicode data

下列說明文檔位於Module:Unicode data/doc。^[編輯]

本模組用於存取 Unicode 字元資料庫（16.0 版），該資料庫源自 Unicode 字元資料庫（UCD）中的檔案。

函數

以下所有函數只能在模組中使用。對於可在模板中呼叫的函數，請參見 Module:Unicode data/templates。

lookup_name: 查詢碼位的名稱。由 Module:character info、Module:character list、Module:also 和 Module:mul-domino def 使用。
lookup_image: 查詢碼位的圖像。由 Module:character info 和 Module:character list 使用。
enum_blocks: 在 Module:character list 的 for in 迴圈中用於遍歷區塊範圍陣列。
lookup_plane: 返回碼位所屬的平面名稱。目前未使用。
get_block_range: 給定區塊名稱，返回該區塊包含的碼位範圍。由 Module:character list 使用。
is_valid_pagename: 給定字串，根據該字串是否可用作頁面標題返回 true 或 false。
add_dotted_circle: 給定字串，在字串中每個組合字元前放置一個虛線圓圈。由 Module:grc-utilities 使用。
lookup_script: 給定碼位，使用 Module:Unicode data/scripts 返回 Unicode 腳本代碼。
get_entry_title: 返回字元條目的標題，對於不能出現在頁面名稱中的字元，標題是不支援的頁面名稱的子頁面。有關更多資訊，請參閱 MediaWiki 網站上的 Manual:頁面標題。由 Module:character info 和 Module:character list 使用。

以下函數與碼位的 General Category 相關。它們接收一個碼位並返回兩個值：一個布林值（表示該碼位是否滿足條件）和一個描述通用類別的字串（如果字元屬於分隔符（Z）或其他（C）通用類別），或字串 "assigned"。它們使用 Module:Unicode data/control。

is_assigned: 檢查碼位的通用類別是否不是 Cn（其他，未分配）。由 Module:character list 使用。
is_printable: 檢查碼位是否可列印：其通用類別不是「其他」（C）、「分隔符，行」（Zl）,或「分隔符，段落」（Zp）。由 Module:character info 和 Module:character list 使用。
is_whitespace: 檢查碼位的通用類別是否為 Zs（「分隔符，空格」）。由 Module:character list 和 Module:links/print data 使用。

資料模組

本模組函數使用的資料位於子頁面上。

Module:Unicode data/aliases：字元的正式名稱別名（來自 NameAliases.txt）
Module:Unicode data/blocks：Unicode 區塊列表（來自 Blocks.txt）；目前由 Module:Unicode data/sandbox 使用
Module:Unicode data/category：將字元映射到其通用類別的資料（來自 DerivedGeneralCategory.txt）
Module:Unicode data/control：用於識別屬於分隔符和其他通用類別的字元的資料（來自 DerivedGeneralCategory.txt）
Module:Unicode data/combining：將字元映射到其組合類的資料（來自 DerivedCombiningClass.txt）
Module:Unicode data/Hangul：用於生成諺文音節名稱的資料（來自 Jamo.txt）
Module:Unicode data/scripts：將字元映射到 Unicode 腳本屬性的資料（來自 Scripts.txt）

名稱和圖像資料模組（Module:Unicode data/names/xxx 和 Module:Unicode data/images/xxx）是從 UnicodeData.txt 編譯而來。每個模組最多包含從 U+xxx000 到 U+xxxFFF 的碼位。然而，以下範圍內的名稱由此基礎模組自動生成，不需要單獨的名稱模組：

BMP: 0000–001F（C0 控制字元）, 007F–009F（DEL 和 C1 控制字元）, 3400–4DBF（中日韓）,

4E00–9FFF（中日韓）, AC00–D7A3（諺文）, D800–DFFF（代理）, E000–F8FF（私用）, F900–FAFF（中日韓）

SMP: 17000–18AFF（西夏文）, 18B00–18CD5（契丹文）, 18D00–18D08（西夏文）, 1B170–1B2FB（女書）

平面 2–3（中日韓）

平面 14: E0100–E01EF（變體選擇符）

平面 15–16（私用）

Unicode name 和 image 模塊，
前三碼組成以十六進位代碼點為基礎
	0	1	2	3	4	5	6	7	8	9	A	B	C	D	E	F
00x	names images	names images	names images	names images	names images	images	images	images	images	images	names images	images	images	names images		names images
01x	names images	names images	names images	names images	names images		names images	images	names images		names images	names images	names images	names images	names images	names images
02x	images	images	images	images	images	images	images	images	images	images	images	images	images	images	images	names images
03x	images	images	images
0Ex	names images

更新

每當發布新的 Unicode 版本時，需要更新幾個子模組，有時還需要更新主模組中的名稱規則。名稱、區塊、控制字元和組合字元類別模組是由 User:Kephir/Unicode 中給出的腳本構建的；腳本和通用類別模組是由 Wikipedia 上的模組生成的。大多數名稱規則在 Unicode 核心規範的第 4 章第 4.8 節（名稱）中給出，其他規則是手動添加的，每個 Unicode 版本可能需要擴展這些規則適用的字元範圍。

Module:Unicode data/Hangul 和諺文音節名稱規則永遠不需要更新。

更新這些模組後，還需要更新 Module:character list 中的版本號。

版權

Unicode 資料庫由 Unicode Inc. 根據以下條款發布：

Copyright © 1991-2022 Unicode, Inc. All rights reserved. Distributed under the Terms of Use in https://www.unicode.org/copyright.html.
Permission is hereby granted, free of charge, to any person obtaining a copy of the Unicode data files and any associated documentation (the "Data Files") or Unicode software and any associated documentation (the "Software") to deal in the Data Files or Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, and/or sell copies of the Data Files or Software, and to permit persons to whom the Data Files or Software are furnished to do so, provided that either (a) this copyright and permission notice appear with all copies of the Data Files or Software, or (b) this copyright and permission notice appear in associated Documentation.
THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE DATA FILES OR SOFTWARE.
Except as contained in this notice, the name of a copyright holder shall not be used in advertising or otherwise to promote the sale, use or other dealings in these Data Files or Software without prior written authorization of the copyright holder.

local m_str_utils = require("Module:string utilities")

local cp = m_str_utils.codepoint
local gcodepoint = m_str_utils.gcodepoint
local gsub = string.gsub
local u = m_str_utils.char

local export = {}
local udata = mw.loadData("Module:Unicode data/data")

local floor = math.floor

local function errorf(first_arg, ...)
	if type(first_arg) == "number" then
		return error(string.format(...), first_arg + 1)
	else
		return error(string.format(first_arg, ...), 2)
	end
end

local function binary_range_search(codepoint, ranges)
	local low, mid, high
	low, high = 1, ranges.length or require "Module:table".length(ranges)
	while low <= high do
		mid = floor((low + high) / 2)
		local range = ranges[mid]
		if codepoint < range[1] then
			high = mid - 1
		elseif codepoint <= range[2] then
			return range, mid
		else
			low = mid + 1
		end
	end
	return nil, mid
end
export.binary_range_search = binary_range_search

local function linear_range_search(codepoint, ranges)
	for i, range in ipairs(ranges) do
		if codepoint < range[1] then
			break
		elseif codepoint <= range[2] then
			return range
		end
	end
end

-- Load a module by indexing "loader" with the name of the module minus the
-- "Module:Unicode data/" part. For instance, loader.blocks returns
-- [[Module:Unicode data/blocks]]. If a module cannot be loaded, false will be
-- returned.
local loader = setmetatable({}, {
	__index = function (self, key)
		local success, data = pcall(mw.loadData, "Module:Unicode data/" .. key)
		if not success then
			data = false
		end
		self[key] = data
		return data
	end
})

-- For the algorithm used to generate Hangul Syllable names,
-- see "Hangul Syllable Name Generation" in section 3.12 of the
-- Unicode Specification:
-- https://www.unicode.org/versions/latest/ch03.pdf
-- For most of the name rules given here, see the subsection
-- "Unicode Name Property" in section 4.8 (Name) and the table 4-8
-- (Name Derivation Rule Prefix Strings):
-- https://www.unicode.org/versions/latest/ch04.pdf
local name_hooks = {
	{   0x0000,   0x001F, "<控制字符-%04X>" }, -- C0 控制字符
	{   0x007F,   0x009F, "<控制字符-%04X>" }, -- DEL 和 C1 控制字符
	{   0x3400,   0x4DBF, "中日韓統一表意文字-%04X" }, -- 中日韓統一表意文字A
	{   0x4E00,   0x9FFF, "中日韓統一表意文字-%04X" }, -- 中日韓統一表意文字
	{   0xAC00,   0xD7A3, function (codepoint) -- 諺文音節
		local Hangul_data = loader.Hangul
		local syllable_index = codepoint - 0xAC00

		return ("諺文音節%s%s%s"):format(
			Hangul_data.leads[floor(syllable_index / Hangul_data.final_count)],
			Hangul_data.vowels[floor((syllable_index % Hangul_data.final_count)
				/ Hangul_data.trail_count)],
			Hangul_data.trails[syllable_index % Hangul_data.trail_count]
		)
	end },
	-- High Surrogates, High Private Use Surrogates, Low Surrogates
	{   0xD800,   0xDFFF, "<代用區-%04X>" },
	{   0xE000,   0xF8FF, "<私人使用區-%04X>" }, -- 私人使用區
	-- 中日韓相容表意文字
	{   0xF900,   0xFA6D, "中日韓相容表意文字-%04X" },
	{   0xFA70,   0xFAD9, "中日韓相容表意文字-%04X" },
	{   0xFDD0,   0xFDEF, "<noncharacter-%04X>" },
	{   0xFE00,   0xFE0F, function (codepoint) -- 變體選擇符補充
		return ("變體選擇符-%d"):format(codepoint - 0xFE00 + 1)
	end},
    {  0x13460,  0x143FA, "埃及象形文字-%04X" }, -- 埃及聖書體擴展-A
	{  0x17000,  0x187F7, "西夏文表意文字-%04X" }, -- 西夏文
	{  0x18800,  0x18AFF, function (codepoint)
		return ("西夏文部件-%03d"):format(codepoint - 0x187FF)
	end },
	{  0x18B00,  0x18CD5, "契丹小字-%04X" }, -- 契丹小字
	{  0x18CFF,  0x18CFF, "契丹小字-%04X" },
	{  0x18D00,  0x18D08, "西夏文表意文字-%04X" }, -- 西夏文補充
	{  0x1B170,  0x1B2FB, "女書文字-%04X" }, -- 女書
	{  0x20000,  0x2A6DF, "中日韓統一表意文字-%04X" }, -- 中日韓統一表意文字B
	{  0x2A700,  0x2B739, "中日韓統一表意文字-%04X" }, -- 中日韓統一表意文字C
	{  0x2B740,  0x2B81D, "中日韓統一表意文字-%04X" }, -- 中日韓統一表意文字D
	{  0x2B820,  0x2CEA1, "中日韓統一表意文字-%04X" }, -- 中日韓統一表意文字E
	{  0x2CEB0,  0x2EBE0, "中日韓統一表意文字-%04X" }, -- 中日韓統一表意文字F
	{  0x2EBF0,  0x2EE5D, "中日韓統一表意文字-%04X" }, -- 中日韓統一表意文字I
	-- 中日韓相容表意文字補充區 (表意文字補充平面)
	{  0x2F800,  0x2FA1D, "CJK COMPATIBILITY IDEOGRAPH-%04X" },
	{  0x30000,  0x3134A, "中日韓統一表意文字-%04X" }, -- 中日韓統一表意文字G
	{  0x31350,  0x323AF, "中日韓統一表意文字-%04X" }, -- 中日韓統一表意文字H
	{  0xE0100,  0xE01EF, function (codepoint) -- 選擇器變化補充
		return ("變體選擇符-%d"):format(codepoint - 0xE0100 + 17)
	end},
	{  0xF0000,  0xFFFFD, "<私人使用區-%04X>" }, -- 第15平面私人使用區
	{ 0x100000, 0x10FFFD, "<私人使用區-%04X>" }, -- 第16平面私人使用區
}
name_hooks.length = #name_hooks
--[[ Add another - in this line to test the code point ordering in name_hooks.
local i = 1
local function print_it(a, b, c)
	if type(c) == "string" then
		mw.log(c:format(a), c:format(b))
	else
		mw.log(c(a), c(b))
	end
end
while true do
	local first, second = name_hooks[i], name_hooks[i + 1]
	if not (first and second) then break end
	local message
	if not (first[1] < first[2] and first[2] < second[1] and second[1] < second[2]) then
		message = "Bad name label ordering at index " .. i .. ":"
	elseif second[1] == first[2] + 1 and second[3] == first[3] then
		message = "Name hooks can be merged at index " .. i .. ":"
	end
	if message then
		mw.log(message)
		print_it(unpack(first))
		print_it(unpack(second))
	end
	i = i + 1
end
--]]

local name_range_cache

local function generate_name(data, codepoint)
	if type(data) == "string" then
		return data:format(codepoint)
	else
		return data(codepoint)
	end
end

--[[
-- Checks that the code point is a number and in range.
-- Does not check whether code point is an integer.
-- Not used
local function check_codepoint(funcName, argIdx, val)
	require 'libraryUtil'.checkType(funcName, argIdx, val, 'number')
	if codepoint < 0 or 0x10FFFF < codepoint then
		errorf("代碼點 %04X 超出範圍", codepoint)
	end
end
--]]

-- https://www.unicode.org/versions/latest/ch04.pdf, section 4.8
function export.lookup_name(codepoint)
	-- U+FDD0-U+FDEF and all code points ending in FFFE or FFFF are Unassigned
	-- (Cn) and specifically noncharacters:
	-- https://www.unicode.org/faq/private_use.html#nonchar4
	if codepoint >= 0xFFFE and floor(codepoint % 0x10000) >= 0xFFFE then
		return ("<noncharacter-%04X>"):format(codepoint)
	end

	if name_range_cache -- Check if previously used "name hook" applies to this code point.
			and codepoint >= name_range_cache[1]
			and codepoint <= name_range_cache[2] then
		return generate_name(name_range_cache[3], codepoint)
	end
	
	local range = binary_range_search(codepoint, name_hooks)
	if range then
		name_range_cache = range
		return generate_name(range[3], codepoint)
	end

	local data = loader[('names/%03X'):format(codepoint / 0x1000)]
	
	if data and data[codepoint] then
		return data[codepoint]
	
	-- Unassigned (Cn) consists of noncharacters and reserved characters.
	-- The character has been established not to be a noncharacter,
	-- and if it were assigned, its name would already been retrieved,
	-- so it must be reserved.
	else
		return ("<reserved-%04X>"):format(codepoint)
	end
end

function export.lookup_image(codepoint)
	local data = loader[('images/%03X'):format(codepoint / 0x1000)]
	
	if data then
		return data[codepoint]
	end
end

function export.lookup_image_emoji(codepoint)
	local data = loader[('emoji_images/%03X'):format(codepoint / 0x1000)]
	
	if data then
		return data[codepoint]
	end
end

-- Load [[Module:Unicode data/blocks]] if needed and assign it to this variable.
local blocks

local function block_iter(blocks, i)
	i = i + 1
	local data = blocks[i]
	if data then
		 -- Unpack doesn't work on tables loaded with mw.loadData.
		return i, data[3], data[1], data[2]
	end
end

-- An ipairs-type iterator generator for the list of blocks.
function export.enum_blocks()
	local blocks = loader.blocks
	return block_iter, blocks, 0
end

function export.get_block_range(name)
	for i, block in ipairs(loader.blocks) do
		if block[3] == name then
			return block[1], block[2]
		end
	end
end

function export.lookup_plane(codepoint)
	local i = floor(codepoint / 0x10000)
	return udata.planes[i] or ("第%u平面"):format(i)
end

function export.lookup_block(codepoint)
	local blocks = loader.blocks
	local range = binary_range_search(codepoint, blocks)
	if range then
		return range[3]
	else
		return "找無區段"
	end
end

function export.get_block_info(name)
	for i, block in ipairs(loader.blocks) do
		if block[3] == name then
			return block
		end
	end
end

function export.is_valid_pagename(pagename)
	local has_nonws = false

	for codepoint in gcodepoint(pagename) do
		if (codepoint == 0x0023) -- #
		or (codepoint == 0x005B) -- [
		or (codepoint == 0x005D) -- ]
		or (codepoint == 0x007B) -- {
		or (codepoint == 0x007C) -- |
		or (codepoint == 0x007D) -- }
		or (codepoint == 0x180E) -- MONGOLIAN VOWEL SEPARATOR
		or ((codepoint >= 0x2000) and (codepoint <= 0x200A)) -- spaces in General Punctuation block
		or (codepoint == 0xFFFD) -- REPLACEMENT CHARACTER
		then
			return false
		end

		local printable, result = export.is_printable(codepoint)
		if not printable then
			return false
		end

		if result ~= "space-separator" then
			has_nonws = true
		end
	end

	return has_nonws
end

local function manual_unpack(what, from)
	if what[from + 1] == nil then
		return what[from]
	end
	
	local result = {}
	from = from or 1
	for i, item in ipairs(what) do
		if i >= from then
			table.insert(result, item)
		end
	end
	return unpack(result)
end

local function compare_ranges(range1, range2)
	return range1[1] < range2[1]
end

-- Creates a function to look up data in a module that contains "singles" (a
-- code point-to-data map) and "ranges" (an array containing arrays that contain
-- the low and high code points of a range and the data associated with that
-- range).
-- "loader" loads and returns the "singles" and "ranges" tables.
-- "match_func" is passed the code point and either the data or the "dots", and
-- generates the final result of the function.
-- The varargs ("dots") describes the default data to be returned if there wasn't
-- a match.
-- In case the function is used more than once, "cache" saves ranges that have
-- already been found to match, or a range whose data is the default if there
-- was no match.
local function codepoint_lookup(data_module_subpage, match_func, ...)
	local dots = { ... }
	local cache = {}
	local singles, ranges

	return function (codepoint)
		if not singles then
			local data_module = loader[data_module_subpage]
			singles, ranges = data_module.singles, data_module.ranges
		end

		if singles[codepoint] then
			return match_func(codepoint, singles[codepoint])
		end

		local range = binary_range_search(codepoint, cache)
		if range then
			return match_func(codepoint, manual_unpack(range, 3))
		end
		
		local range, index = binary_range_search(codepoint, ranges)
		if range then
			table.insert(cache, range)
			table.sort(cache, compare_ranges)
			return match_func(codepoint, manual_unpack(range, 3))
		end
		
		if ranges[index] then
			local dots_range
			if codepoint > ranges[index][2] then
				dots_range = {
					ranges[index][2] + 1,
					ranges[index + 1] and ranges[index + 1][1] - 1 or 0x10FFFF,
					unpack(dots)
				}
			else -- codepoint < range[index][1]
				dots_range = {
					ranges[index - 1] and ranges[index - 1][2] + 1 or 0,
					ranges[index][1] - 1,
					unpack(dots)
				}
			end
			table.insert(cache, dots_range)
			table.sort(cache, compare_ranges)
		end
		
		return match_func(codepoint, unpack(dots))
	end
end

-- Return a character's combining class value from [[Module:Unicode data/combining classes]],
-- or otherwise 0, which is treated as the default value.
do
	local combining
	
	function export.combining_class(ch)
		combining = combining or mw.loadData("Module:Unicode data/combining classes")
		return combining[type(ch) == "number" and u(ch) or ch] or 0
	end
end

-- FIXME: Some combining characters have a combining class of 0, so this needs rethinking.
function export.is_combining(ch)
	return export.combining_class(ch) ~= 0
end

do
	local function dotted_circle(ch)
		if export.combining_class(ch) ~= 0 then
			return "◌" .. ch
		end
	end
	
	function export.add_dotted_circle(str)
		return (gsub(str, ".[\128-\191]*", dotted_circle))
	end
end

local lookup_control = codepoint_lookup(
	"control",
	function (codepoint, ccc)
		return ccc or "assigned"
	end,
	"assigned")
export.lookup_control = lookup_control

function export.is_assigned(codepoint)
	return lookup_control(codepoint) ~= "unassigned"
end

function export.is_printable(codepoint)
	local result = lookup_control(codepoint)
	return (result == "assigned") or (result == "space-separator"), result
end

function export.is_whitespace(codepoint)
	local result = lookup_control(codepoint)
	return (result == "space-separator"), result
end

export.lookup_category = codepoint_lookup(
	"category",
	function (codepoint, category)
		return category
	end,
	"Cn"
)

function export.get_category_long_name(category)
	return loader["category"].long_names[category]
end

export.lookup_script = codepoint_lookup(
	"scripts",
	function (codepoint, script)
		return script
	end,
	"Zzzz"
)

function export.get_script_alias(script)
	return loader["scripts"].aliases[script]
end

function export.get_entry_title(codepoint)
	if udata.unsupported_title[codepoint] then
		return udata.unsupported_title[codepoint]
	end
	if lookup_control(codepoint) ~= "assigned" then
		return nil
	end
	return u(codepoint)
end

return export