Release v1.10.1

* Add zero padding to the color code

* Change color ending into total formatting reset

.github/ISSUE_TEMPLATE/Bug_report.md

@@ -0,0 +1,26 @@
+---
+name: Bug report
+about: Create a report to help us improve. (Check the FAQ on the wiki first)
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots/debug logs**
+If applicable, add screenshots to help explain your problem.
+Use logs from running `matterbridge -debug` if possible.
+
+**Environment (please complete the following information):**
+ - OS: [e.g. linux]
+ - Matterbridge version: output of  `matterbridge -version`
+ - If self compiled: output of `git rev-parse HEAD`
+
+**Additional context**
+Please add your configuration file  (be sure to exclude or anonymize private data (tokens/passwords))

.github/ISSUE_TEMPLATE/Feature_request.md

@@ -0,0 +1,17 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.travis.yml

@@ -43,6 +43,8 @@ script:
 
 deploy:
   provider: bintray
+  edge:
+    branch: v1.8.47
   file: ci/deploy.json
   user: 42wim
   key:

README.md

@@ -1,16 +1,19 @@
 # matterbridge
 Click on one of the badges below to join the chat   
 
-[![Gitter](https://img.shields.io/gitter/room/nwjs/nw.js.svg?colorB=42f4242)](https://gitter.im/42wim/matterbridge) [![Join the IRC chat at https://webchat.freenode.net/?channels=matterbridgechat](https://img.shields.io/badge/IRC-matterbridgechat-green.svg?colorB=42f4242)](https://webchat.freenode.net/?channels=matterbridgechat) [![Discord](https://img.shields.io/badge/discord-matterbridge-green.svg?colorB=42f4242)](https://discord.gg/AkKPtrQ) [![Matrix](https://img.shields.io/badge/matrix-matterbridge-green.svg?colorB=42f4242)](https://riot.im/app/#/room/#matterbridge:matrix.org) [![Slack](https://img.shields.io/badge/slack-matterbridgechat-green.svg?colorB=42f4242)](https://join.slack.com/matterbridgechat/shared_invite/MjEwODMxNjU1NDMwLTE0OTk2MTU3NTMtMzZkZmRiNDZhOA) [![Mattermost](https://img.shields.io/badge/mattermost-matterbridge-green.svg?colorB=42f4242)](https://framateam.org/signup_user_complete/?id=tfqm33ggop8x3qgu4boeieta6e) [![Xmpp](https://img.shields.io/badge/xmpp-matterbridge@conference.jabber.de-green.svg?colorB=42f4242)](https://inverse.chat) [![Twitch](https://img.shields.io/badge/twitch-matterbridge-green.svg?colorB=42f4242)](https://www.twitch.tv/matterbridge)
+[![Gitter](https://img.shields.io/gitter/room/nwjs/nw.js.svg?colorB=42f4242)](https://gitter.im/42wim/matterbridge) [![Join the IRC chat at https://webchat.freenode.net/?channels=matterbridgechat](https://img.shields.io/badge/IRC-matterbridgechat-green.svg?colorB=42f4242)](https://webchat.freenode.net/?channels=matterbridgechat) [![Discord](https://img.shields.io/badge/discord-matterbridge-green.svg?colorB=42f4242)](https://discord.gg/AkKPtrQ) [![Matrix](https://img.shields.io/badge/matrix-matterbridge-green.svg?colorB=42f4242)](https://riot.im/app/#/room/#matterbridge:matrix.org) [![Slack](https://img.shields.io/badge/slack-matterbridgechat-green.svg?colorB=42f4242)](https://join.slack.com/matterbridgechat/shared_invite/MjEwODMxNjU1NDMwLTE0OTk2MTU3NTMtMzZkZmRiNDZhOA) [![Mattermost](https://img.shields.io/badge/mattermost-matterbridge-green.svg?colorB=42f4242)](https://framateam.org/signup_user_complete/?id=tfqm33ggop8x3qgu4boeieta6e) [![Xmpp](https://img.shields.io/badge/xmpp-matterbridge@conference.jabber.de-green.svg?colorB=42f4242)](https://inverse.chat) [![Twitch](https://img.shields.io/badge/twitch-matterbridge-green.svg?colorB=42f4242)](https://www.twitch.tv/matterbridge) [![Zulip](https://img.shields.io/badge/zulip-matterbridge-green.svg?colorB=42f4242)](https://matterbridge.zulipchat.com/register/)
 
 [![Download stable](https://img.shields.io/github/release/42wim/matterbridge.svg?label=download%20stable)](https://github.com/42wim/matterbridge/releases/latest) [![Download dev](https://img.shields.io/bintray/v/42wim/nightly/Matterbridge.svg?label=download%20dev&colorB=007ec6)](https://bintray.com/42wim/nightly/Matterbridge/_latestVersion)
 
-![matterbridge.gif](https://s15.postimg.org/qpjhp6y3f/matterbridge.gif)
+![matterbridge.gif](https://github.com/42wim/matterbridge/blob/master/img/matterbridge.gif)
 
-Simple bridge between Mattermost, IRC, XMPP, Gitter, Slack, Discord, Telegram, Rocket.Chat, Hipchat(via xmpp), Matrix, Steam and ssh-chat
-Has a REST API.  
+Simple bridge between IRC, XMPP, Gitter, Mattermost, Slack, Discord, Telegram, Rocket.Chat, Hipchat(via xmpp), Matrix, Steam, ssh-chat and Zulip
+Has a REST API.   
 Minecraft server chat support via [MatterLink](https://github.com/elytra/MatterLink)
 
+**Mattermost isn't required to run matterbridge. It bridges between any supported protocol.**   
+(The name matterbridge is a remnant when it was only bridging mattermost)
+
 # Table of Contents
  * [Features](https://github.com/42wim/matterbridge/wiki/Features)
  * [Requirements](#requirements)
@@ -59,13 +62,14 @@ Accounts to one of the supported bridges
 * [Steam](https://store.steampowered.com/)
 * [Twitch](https://twitch.tv)
 * [Ssh-chat](https://github.com/shazow/ssh-chat)
+* [Zulip](https://zulipchat.com)
 
 # Screenshots
 See https://github.com/42wim/matterbridge/wiki
 
 # Installing
 ## Binaries
-* Latest stable release [v1.9.0](https://github.com/42wim/matterbridge/releases/latest)
+* Latest stable release [v1.10.1](https://github.com/42wim/matterbridge/releases/latest)
 * Development releases (follows master) can be downloaded [here](https://dl.bintray.com/42wim/nightly/)  
 
 ## Building
@@ -181,11 +185,14 @@ Want to tip ?
 * btc: 1N7cKHj5SfqBHBzDJ6kad4BzeqUBBS2zhs
 
 # Thanks
+[![Digitalocean](https://snag.gy/3LVifX.jpg)](https://www.digitalocean.com/) for sponsoring demo/testing droplets.
+
 Matterbridge wouldn't exist without these libraries:
 * discord - https://github.com/bwmarrin/discordgo
 * echo - https://github.com/labstack/echo
 * gitter - https://github.com/sromku/go-gitter
 * gops - https://github.com/google/gops
+* gozulipbot - https://github.com/ifo/gozulipbot
 * irc - https://github.com/lrstanley/girc
 * mattermost - https://github.com/mattermost/platform
 * matrix - https://github.com/matrix-org/gomatrix
@@ -193,3 +200,4 @@ Matterbridge wouldn't exist without these libraries:
 * steam - https://github.com/Philipp15b/go-steam
 * telegram - https://github.com/go-telegram-bot-api/telegram-bot-api
 * xmpp - https://github.com/mattn/go-xmpp
+* zulip - https://github.com/ifo/gozulipbot

bridge/config/config.go

@@ -2,8 +2,10 @@ package config
 
 import (
 	"bytes"
+	"github.com/fsnotify/fsnotify"
 	log "github.com/sirupsen/logrus"
 	"github.com/spf13/viper"
+	prefixed "github.com/x-cray/logrus-prefixed-formatter"
 	"os"
 	"strings"
 	"sync"
@@ -60,6 +62,7 @@ type Protocol struct {
 	BindAddress            string // mattermost, slack // DEPRECATED
 	Buffer                 int    // api
 	Charset                string // irc
+	ColorNicks             bool   // only irc for now
 	Debug                  bool   // general
 	DebugLevel             int    // only for irc now
 	EditSuffix             string // mattermost, slack, discord, telegram, gitter
@@ -92,6 +95,8 @@ type Protocol struct {
 	Password               string     // IRC,mattermost,XMPP,matrix
 	PrefixMessagesWithNick bool       // mattemost, slack
 	Protocol               string     // all protocols
+	QuoteDisable           bool       // telegram
+	QuoteFormat            string     // telegram
 	RejoinDelay            int        // IRC
 	ReplaceMessages        [][]string // all protocols
 	ReplaceNicks           [][]string // all protocols
@@ -104,6 +109,7 @@ type Protocol struct {
 	StripNick              bool       // all protocols
 	Team                   string     // mattermost
 	Token                  string     // gitter, slack, discord, api
+	Topic                  string     // zulip
 	URL                    string     // mattermost, slack // DEPRECATED
 	UseAPI                 bool       // mattermost, slack
 	UseSASL                bool       // IRC
@@ -156,6 +162,7 @@ type ConfigValues struct {
 	Telegram           map[string]Protocol
 	Rocketchat         map[string]Protocol
 	Sshchat            map[string]Protocol
+	Zulip              map[string]Protocol
 	General            Protocol
 	Gateway            []Gateway
 	SameChannelGateway []SameChannelGateway
@@ -168,9 +175,13 @@ type Config struct {
 }
 
 func NewConfig(cfgfile string) *Config {
+	log.SetFormatter(&prefixed.TextFormatter{PrefixPadding: 13, DisableColors: true, FullTimestamp: false})
+	flog := log.WithFields(log.Fields{"prefix": "config"})
 	var cfg ConfigValues
 	viper.SetConfigType("toml")
+	viper.SetConfigFile(cfgfile)
 	viper.SetEnvPrefix("matterbridge")
+	viper.AddConfigPath(".")
 	viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
 	viper.AutomaticEnv()
 	f, err := os.Open(cfgfile)
@@ -190,6 +201,11 @@ func NewConfig(cfgfile string) *Config {
 	if cfg.General.MediaDownloadSize == 0 {
 		cfg.General.MediaDownloadSize = 1000000
 	}
+	viper.WatchConfig()
+	viper.OnConfigChange(func(e fsnotify.Event) {
+		flog.Println("Config file changed:", e.Name)
+	})
+
 	mycfg.ConfigValues = &cfg
 	return mycfg
 }
@@ -242,11 +258,18 @@ func (c *Config) GetStringSlice(key string) []string {
 func (c *Config) GetStringSlice2D(key string) [][]string {
 	c.RLock()
 	defer c.RUnlock()
-	if res, ok := c.v.Get(key).([][]string); ok {
-		return res
+	result := [][]string{}
+	if res, ok := c.v.Get(key).([]interface{}); ok {
+		for _, entry := range res {
+			result2 := []string{}
+			for _, entry2 := range entry.([]interface{}) {
+				result2 = append(result2, entry2.(string))
+			}
+			result = append(result, result2)
+		}
+		return result
 	}
-	// log.Debugf("getting StringSlice2D %s = %#v", key, c.v.Get(key))
-	return [][]string{}
+	return result
 }
 
 func GetIconURL(msg *Message, iconURL string) string {

bridge/discord/discord.go

@@ -80,6 +80,9 @@ func (b *Bdiscord) Connect() error {
 			}
 		}
 	}
+	for _, channel := range b.Channels {
+		b.Log.Debugf("found channel %#v", channel)
+	}
 	return nil
 }
 
@@ -127,6 +130,12 @@ func (b *Bdiscord) Send(msg config.Message) (string, error) {
 			return "", nil
 		}
 		b.Log.Debugf("Broadcasting using Webhook")
+		for _, f := range msg.Extra["file"] {
+			fi := f.(config.FileInfo)
+			if fi.URL != "" {
+				msg.Text += fi.URL + " "
+			}
+		}
 		err := b.c.WebhookExecute(
 			wID,
 			wToken,

bridge/gitter/gitter.go

@@ -168,6 +168,9 @@ func (b *Bgitter) handleUploadFile(msg *config.Message, roomID string) (string,
 		}
 		if fi.URL != "" {
 			msg.Text = fi.URL
+			if fi.Comment != "" {
+				msg.Text = fi.Comment + ": " + fi.URL
+			}
 		}
 		_, err := b.c.SendMessage(roomID, msg.Username+msg.Text)
 		if err != nil {

bridge/helper/helper.go

@@ -7,6 +7,7 @@ import (
 	log "github.com/sirupsen/logrus"
 	"io"
 	"net/http"
+	"strings"
 	"time"
 )
 
@@ -88,3 +89,14 @@ func HandleDownloadData(flog *log.Entry, msg *config.Message, name, comment, url
 	}
 	msg.Extra["file"] = append(msg.Extra["file"], config.FileInfo{Name: name, Data: data, URL: url, Comment: comment, Avatar: avatar})
 }
+
+func RemoveEmptyNewLines(msg string) string {
+	lines := ""
+	for _, line := range strings.Split(msg, "\n") {
+		if line != "" {
+			lines += line + "\n"
+		}
+	}
+	lines = strings.TrimRight(lines, "\n")
+	return lines
+}

bridge/irc/irc.go

@@ -7,10 +7,12 @@ import (
 	"github.com/42wim/matterbridge/bridge"
 	"github.com/42wim/matterbridge/bridge/config"
 	"github.com/42wim/matterbridge/bridge/helper"
+	"github.com/dfordsoft/golib/ic"
 	"github.com/lrstanley/girc"
 	"github.com/paulrosania/go-charset/charset"
 	_ "github.com/paulrosania/go-charset/data"
 	"github.com/saintfish/chardet"
+	"hash/crc32"
 	"io"
 	"io/ioutil"
 	"net"
@@ -177,15 +179,20 @@ func (b *Birc) Send(msg config.Message) (string, error) {
 
 	// convert to specified charset
 	if b.GetString("Charset") != "" {
-		buf := new(bytes.Buffer)
-		w, err := charset.NewWriter(b.GetString("Charset"), buf)
-		if err != nil {
-			b.Log.Errorf("charset from utf-8 conversion failed: %s", err)
-			return "", err
+		switch b.GetString("Charset") {
+		case "gbk", "gb18030", "gb2312", "big5", "euc-kr", "euc-jp", "shift-jis", "iso-2022-jp":
+			msg.Text = ic.ConvertString("utf-8", b.GetString("Charset"), msg.Text)
+		default:
+			buf := new(bytes.Buffer)
+			w, err := charset.NewWriter(b.GetString("Charset"), buf)
+			if err != nil {
+				b.Log.Errorf("charset from utf-8 conversion failed: %s", err)
+				return "", err
+			}
+			fmt.Fprint(w, msg.Text)
+			w.Close()
+			msg.Text = buf.String()
 		}
-		fmt.Fprintf(w, msg.Text)
-		w.Close()
-		msg.Text = buf.String()
 	}
 
 	// Handle files
@@ -201,6 +208,9 @@ func (b *Birc) Send(msg config.Message) (string, error) {
 				}
 				if fi.URL != "" {
 					msg.Text = fi.URL
+					if fi.Comment != "" {
+						msg.Text = fi.Comment + ": " + fi.URL
+					}
 				}
 				b.Local <- config.Message{Text: msg.Text, Username: msg.Username, Channel: msg.Channel, Event: msg.Event}
 			}
@@ -237,11 +247,16 @@ func (b *Birc) doSend() {
 	throttle := time.NewTicker(rate)
 	for msg := range b.Local {
 		<-throttle.C
+		username := msg.Username
+		if b.GetBool("Colornicks") {
+			checksum := crc32.ChecksumIEEE([]byte(msg.Username))
+			username = fmt.Sprintf("\x03%02d%s\x0F", checksum%0x10, msg.Username)
+		}
 		if msg.Event == config.EVENT_USER_ACTION {
-			b.i.Cmd.Action(msg.Channel, msg.Username+msg.Text)
+			b.i.Cmd.Action(msg.Channel, username+msg.Text)
 		} else {
 			b.Log.Debugf("Sending to channel %s", msg.Channel)
-			b.i.Cmd.Message(msg.Channel, msg.Username+msg.Text)
+			b.i.Cmd.Message(msg.Channel, username+msg.Text)
 		}
 	}
 }
@@ -401,13 +416,18 @@ func (b *Birc) handlePrivMsg(client *girc.Client, event girc.Event) {
 			mycharset = "ISO-8859-1"
 		}
 	}
-	r, err = charset.NewReader(mycharset, strings.NewReader(rmsg.Text))
-	if err != nil {
-		b.Log.Errorf("charset to utf-8 conversion failed: %s", err)
-		return
+	switch mycharset {
+	case "gbk", "gb18030", "gb2312", "big5", "euc-kr", "euc-jp", "shift-jis", "iso-2022-jp":
+		rmsg.Text = ic.ConvertString("utf-8", b.GetString("Charset"), rmsg.Text)
+	default:
+		r, err = charset.NewReader(mycharset, strings.NewReader(rmsg.Text))
+		if err != nil {
+			b.Log.Errorf("charset to utf-8 conversion failed: %s", err)
+			return
+		}
+		output, _ := ioutil.ReadAll(r)
+		rmsg.Text = string(output)
 	}
-	output, _ := ioutil.ReadAll(r)
-	rmsg.Text = string(output)
 
 	b.Log.Debugf("<= Sending message from %s on %s to gateway", event.Params[0], b.Account)
 	b.Remote <- rmsg

bridge/mattermost/mattermost.go

@@ -8,12 +8,14 @@ import (
 	"github.com/42wim/matterbridge/bridge/helper"
 	"github.com/42wim/matterbridge/matterclient"
 	"github.com/42wim/matterbridge/matterhook"
+	"github.com/rs/xid"
 	"strings"
 )
 
 type Bmattermost struct {
 	mh     *matterhook.Client
 	mc     *matterclient.MMClient
+	uuid   string
 	TeamID string
 	*bridge.Config
 	avatarMap map[string]string
@@ -21,6 +23,7 @@ type Bmattermost struct {
 
 func New(cfg *bridge.Config) bridge.Bridger {
 	b := &Bmattermost{Config: cfg, avatarMap: make(map[string]string)}
+	b.uuid = xid.New().String()
 	return b
 }
 
@@ -365,8 +368,9 @@ func (b *Bmattermost) sendWebhook(msg config.Message) (string, error) {
 	if msg.Extra != nil {
 		// this sends a message only if we received a config.EVENT_FILE_FAILURE_SIZE
 		for _, rmsg := range helper.HandleExtra(&msg, b.General) {
-			matterMessage := matterhook.OMessage{IconURL: b.GetString("IconURL"), Channel: rmsg.Channel, UserName: rmsg.Username, Text: rmsg.Text, Props: make(map[string]interface{})}
-			matterMessage.Props["matterbridge"] = true
+			iconURL := config.GetIconURL(&rmsg, b.GetString("iconurl"))
+			matterMessage := matterhook.OMessage{IconURL: iconURL, Channel: rmsg.Channel, UserName: rmsg.Username, Text: rmsg.Text, Props: make(map[string]interface{})}
+			matterMessage.Props["matterbridge_"+b.uuid] = true
 			b.mh.Send(matterMessage)
 		}
 
@@ -381,11 +385,12 @@ func (b *Bmattermost) sendWebhook(msg config.Message) (string, error) {
 		}
 	}
 
-	matterMessage := matterhook.OMessage{IconURL: b.GetString("IconURL"), Channel: msg.Channel, UserName: msg.Username, Text: msg.Text, Props: make(map[string]interface{})}
+	iconURL := config.GetIconURL(&msg, b.GetString("iconurl"))
+	matterMessage := matterhook.OMessage{IconURL: iconURL, Channel: msg.Channel, UserName: msg.Username, Text: msg.Text, Props: make(map[string]interface{})}
 	if msg.Avatar != "" {
 		matterMessage.IconURL = msg.Avatar
 	}
-	matterMessage.Props["matterbridge"] = true
+	matterMessage.Props["matterbridge_"+b.uuid] = true
 	err := b.mh.Send(matterMessage)
 	if err != nil {
 		b.Log.Info(err)
@@ -415,7 +420,7 @@ func (b *Bmattermost) skipMessage(message *matterclient.Message) bool {
 
 	// Ignore messages sent from matterbridge
 	if message.Post.Props != nil {
-		if _, ok := message.Post.Props["matterbridge"].(bool); ok {
+		if _, ok := message.Post.Props["matterbridge_"+b.uuid].(bool); ok {
 			b.Log.Debugf("sent by matterbridge, ignoring")
 			return true
 		}

bridge/rocketchat/rocketchat.go

@@ -53,7 +53,8 @@ func (b *Brocketchat) Send(msg config.Message) (string, error) {
 	b.Log.Debugf("=> Receiving %#v", msg)
 	if msg.Extra != nil {
 		for _, rmsg := range helper.HandleExtra(&msg, b.General) {
-			matterMessage := matterhook.OMessage{IconURL: b.GetString("IconURL"), Channel: rmsg.Channel, UserName: rmsg.Username, Text: rmsg.Text}
+			iconURL := config.GetIconURL(&rmsg, b.GetString("iconurl"))
+			matterMessage := matterhook.OMessage{IconURL: iconURL, Channel: rmsg.Channel, UserName: rmsg.Username, Text: rmsg.Text}
 			b.mh.Send(matterMessage)
 		}
 		if len(msg.Extra["file"]) > 0 {
@@ -66,7 +67,8 @@ func (b *Brocketchat) Send(msg config.Message) (string, error) {
 		}
 	}
 
-	matterMessage := matterhook.OMessage{IconURL: b.GetString("IconURL")}
+	iconURL := config.GetIconURL(&msg, b.GetString("iconurl"))
+	matterMessage := matterhook.OMessage{IconURL: iconURL}
 	matterMessage.Channel = msg.Channel
 	matterMessage.UserName = msg.Username
 	matterMessage.Type = ""

bridge/slack/slack.go

@@ -4,16 +4,18 @@ import (
 	"bytes"
 	"errors"
 	"fmt"
-	"github.com/42wim/matterbridge/bridge"
-	"github.com/42wim/matterbridge/bridge/config"
-	"github.com/42wim/matterbridge/bridge/helper"
-	"github.com/42wim/matterbridge/matterhook"
-	"github.com/nlopes/slack"
 	"html"
 	"regexp"
 	"strings"
 	"sync"
 	"time"
+
+	"github.com/42wim/matterbridge/bridge"
+	"github.com/42wim/matterbridge/bridge/config"
+	"github.com/42wim/matterbridge/bridge/helper"
+	"github.com/42wim/matterbridge/matterhook"
+	"github.com/nlopes/slack"
+	"github.com/rs/xid"
 )
 
 type Bslack struct {
@@ -24,6 +26,7 @@ type Bslack struct {
 	Usergroups []slack.UserGroup
 	si         *slack.Info
 	channels   []slack.Channel
+	uuid       string
 	*bridge.Config
 	sync.RWMutex
 }
@@ -31,7 +34,7 @@ type Bslack struct {
 const messageDeleted = "message_deleted"
 
 func New(cfg *bridge.Config) bridge.Bridger {
-	return &Bslack{Config: cfg}
+	return &Bslack{Config: cfg, uuid: xid.New().String()}
 }
 
 func (b *Bslack) Command(cmd string) string {
@@ -176,7 +179,7 @@ func (b *Bslack) Send(msg config.Message) (string, error) {
 		np.IconURL = msg.Avatar
 	}
 	// add a callback ID so we can see we created it
-	np.Attachments = append(np.Attachments, slack.Attachment{CallbackID: "matterbridge"})
+	np.Attachments = append(np.Attachments, slack.Attachment{CallbackID: "matterbridge_" + b.uuid})
 	// add file attachments
 	np.Attachments = append(np.Attachments, b.createAttach(msg.Extra)...)
 	// add slack attachments (from another slack bridge)
@@ -267,7 +270,7 @@ func (b *Bslack) handleSlack() {
 		message.Text = html.UnescapeString(message.Text)
 
 		// Add the avatar
-		message.Avatar = b.getAvatar(message.Username)
+		message.Avatar = b.getAvatar(strings.ToLower(message.Username))
 
 		b.Log.Debugf("<= Message is %#v", message)
 		b.Remote <- *message
@@ -387,7 +390,11 @@ func (b *Bslack) replaceVariable(text string) string {
 func (b *Bslack) replaceURL(text string) string {
 	results := regexp.MustCompile(`<(.*?)(\|.*?)?>`).FindAllStringSubmatch(text, -1)
 	for _, r := range results {
-		text = strings.Replace(text, r[0], r[1], -1)
+		if len(strings.TrimSpace(r[2])) == 1 { // A display text separator was found, but the text was blank
+			text = strings.Replace(text, r[0], "", -1)
+		} else {
+			text = strings.Replace(text, r[0], r[1], -1)
+		}
 	}
 	return text
 }
@@ -480,7 +487,7 @@ func (b *Bslack) handleMessageEvent(ev *slack.MessageEvent) (*config.Message, er
 	rmsg := config.Message{Text: ev.Text, Channel: channel.Name, Account: b.Account, ID: "slack " + ev.Timestamp, Extra: make(map[string][]interface{})}
 
 	// find the user id and name
-	if ev.BotID == "" && ev.SubType != messageDeleted && ev.SubType != "file_comment" {
+	if ev.User != "" && ev.SubType != messageDeleted && ev.SubType != "file_comment" {
 		user, err := b.rtm.GetUserInfo(ev.User)
 		if err != nil {
 			return nil, err
@@ -504,7 +511,7 @@ func (b *Bslack) handleMessageEvent(ev *slack.MessageEvent) (*config.Message, er
 	}
 
 	// when using webhookURL we can't check if it's our webhook or not for now
-	if ev.BotID != "" && b.GetString("WebhookURL") == "" {
+	if rmsg.Username == "" && ev.BotID != "" && b.GetString("WebhookURL") == "" {
 		bot, err := b.rtm.GetBotInfo(ev.BotID)
 		if err != nil {
 			return nil, err
@@ -516,6 +523,19 @@ func (b *Bslack) handleMessageEvent(ev *slack.MessageEvent) (*config.Message, er
 			}
 			rmsg.UserID = bot.ID
 		}
+
+		// fixes issues with matterircd users
+		if bot.Name == "Slack API Tester" {
+			user, err := b.rtm.GetUserInfo(ev.User)
+			if err != nil {
+				return nil, err
+			}
+			rmsg.UserID = user.ID
+			rmsg.Username = user.Name
+			if user.Profile.DisplayName != "" {
+				rmsg.Username = user.Profile.DisplayName
+			}
+		}
 	}
 
 	// file comments are set by the system (because there is no username given)
@@ -553,6 +573,10 @@ func (b *Bslack) handleMessageEvent(ev *slack.MessageEvent) (*config.Message, er
 
 	// Only deleted messages can have a empty username and text
 	if (rmsg.Text == "" || rmsg.Username == "") && ev.SubType != messageDeleted {
+		// this is probably a webhook we couldn't resolve
+		if ev.BotID != "" {
+			return nil, fmt.Errorf("probably an incoming webhook we couldn't resolve (maybe ourselves)")
+		}
 		return nil, fmt.Errorf("empty message and not a deleted message")
 	}
 
@@ -586,7 +610,8 @@ func (b *Bslack) sendWebhook(msg config.Message) (string, error) {
 	if msg.Extra != nil {
 		// this sends a message only if we received a config.EVENT_FILE_FAILURE_SIZE
 		for _, rmsg := range helper.HandleExtra(&msg, b.General) {
-			matterMessage := matterhook.OMessage{IconURL: b.GetString("IconURL"), Channel: msg.Channel, UserName: rmsg.Username, Text: rmsg.Text}
+			iconURL := config.GetIconURL(&rmsg, b.GetString("iconurl"))
+			matterMessage := matterhook.OMessage{IconURL: iconURL, Channel: msg.Channel, UserName: rmsg.Username, Text: rmsg.Text}
 			b.mh.Send(matterMessage)
 		}
 
@@ -609,7 +634,8 @@ func (b *Bslack) sendWebhook(msg config.Message) (string, error) {
 		}
 	}
 
-	matterMessage := matterhook.OMessage{IconURL: b.GetString("IconURL"), Attachments: attachs, Channel: msg.Channel, UserName: msg.Username, Text: msg.Text}
+	iconURL := config.GetIconURL(&msg, b.GetString("iconurl"))
+	matterMessage := matterhook.OMessage{IconURL: iconURL, Attachments: attachs, Channel: msg.Channel, UserName: msg.Username, Text: msg.Text}
 	if msg.Avatar != "" {
 		matterMessage.IconURL = msg.Avatar
 	}
@@ -639,7 +665,7 @@ func (b *Bslack) skipMessageEvent(ev *slack.MessageEvent) bool {
 
 	// skip messages we made ourselves
 	if len(ev.Attachments) > 0 {
-		if ev.Attachments[0].CallbackID == "matterbridge" {
+		if ev.Attachments[0].CallbackID == "matterbridge_"+b.uuid {
 			return true
 		}
 	}

bridge/sshchat/sshchat.go

@@ -68,6 +68,9 @@ func (b *Bsshchat) Send(msg config.Message) (string, error) {
 				}
 				if fi.URL != "" {
 					msg.Text = fi.URL
+					if fi.Comment != "" {
+						msg.Text = fi.Comment + ": " + fi.URL
+					}
 				}
 				b.w.Write([]byte(msg.Username + msg.Text))
 			}

bridge/steam/steam.go

@@ -2,8 +2,10 @@ package bsteam
 
 import (
 	"fmt"
+
 	"github.com/42wim/matterbridge/bridge"
 	"github.com/42wim/matterbridge/bridge/config"
+	"github.com/42wim/matterbridge/bridge/helper"
 	"github.com/Philipp15b/go-steam"
 	"github.com/Philipp15b/go-steam/protocol/steamlang"
 	"github.com/Philipp15b/go-steam/steamid"
@@ -66,6 +68,30 @@ func (b *Bsteam) Send(msg config.Message) (string, error) {
 	if err != nil {
 		return "", err
 	}
+
+	// Handle files
+	if msg.Extra != nil {
+		for _, rmsg := range helper.HandleExtra(&msg, b.General) {
+			b.c.Social.SendMessage(id, steamlang.EChatEntryType_ChatMsg, rmsg.Username+rmsg.Text)
+		}
+		if len(msg.Extra["file"]) > 0 {
+			for _, f := range msg.Extra["file"] {
+				fi := f.(config.FileInfo)
+				if fi.Comment != "" {
+					msg.Text += fi.Comment + ": "
+				}
+				if fi.URL != "" {
+					msg.Text = fi.URL
+					if fi.Comment != "" {
+						msg.Text = fi.Comment + ": " + fi.URL
+					}
+				}
+				b.c.Social.SendMessage(id, steamlang.EChatEntryType_ChatMsg, msg.Username+msg.Text)
+			}
+			return "", nil
+		}
+	}
+
 	b.c.Social.SendMessage(id, steamlang.EChatEntryType_ChatMsg, msg.Username+msg.Text)
 	return "", nil
 }

bridge/telegram/telegram.go

@@ -1,7 +1,6 @@
 package btelegram
 
 import (
-	"html"
 	"regexp"
 	"strconv"
 	"strings"
@@ -122,7 +121,7 @@ func (b *Btelegram) handleRecv(updates <-chan tgbotapi.Update) {
 	for update := range updates {
 		b.Log.Debugf("== Receiving event: %#v", update.Message)
 
-		if update.Message == nil && update.ChannelPost == nil {
+		if update.Message == nil && update.ChannelPost == nil && update.EditedMessage == nil && update.EditedChannelPost == nil {
 			b.Log.Error("Getting nil messages, this shouldn't happen.")
 			continue
 		}
@@ -134,6 +133,7 @@ func (b *Btelegram) handleRecv(updates <-chan tgbotapi.Update) {
 		// handle channels
 		if update.ChannelPost != nil {
 			message = update.ChannelPost
+			rmsg.Text = message.Text
 		}
 
 		// edited channel message
@@ -145,6 +145,7 @@ func (b *Btelegram) handleRecv(updates <-chan tgbotapi.Update) {
 		// handle groups
 		if update.Message != nil {
 			message = update.Message
+			rmsg.Text = message.Text
 		}
 
 		// edited group message
@@ -155,11 +156,11 @@ func (b *Btelegram) handleRecv(updates <-chan tgbotapi.Update) {
 
 		// set the ID's from the channel or group message
 		rmsg.ID = strconv.Itoa(message.MessageID)
-		rmsg.UserID = strconv.Itoa(message.From.ID)
 		rmsg.Channel = strconv.FormatInt(message.Chat.ID, 10)
 
 		// handle username
 		if message.From != nil {
+			rmsg.UserID = strconv.Itoa(message.From.ID)
 			if b.GetBool("UseFirstName") {
 				rmsg.Username = message.From.FirstName
 			}
@@ -169,7 +170,6 @@ func (b *Btelegram) handleRecv(updates <-chan tgbotapi.Update) {
 					rmsg.Username = message.From.FirstName
 				}
 			}
-			rmsg.Text += message.Text
 			// only download avatars if we have a place to upload them (configured mediaserver)
 			if b.General.MediaServerUpload != "" {
 				b.handleDownloadAvatar(message.From.ID, rmsg.Channel)
@@ -222,11 +222,17 @@ func (b *Btelegram) handleRecv(updates <-chan tgbotapi.Update) {
 			if usernameReply == "" {
 				usernameReply = "unknown"
 			}
-			rmsg.Text = rmsg.Text + " (re @" + usernameReply + ":" + message.ReplyToMessage.Text + ")"
+			if !b.GetBool("QuoteDisable") {
+				rmsg.Text = b.handleQuote(rmsg.Text, usernameReply, message.ReplyToMessage.Text)
+			}
 		}
 
 		if rmsg.Text != "" || len(rmsg.Extra) > 0 {
-			rmsg.Avatar = helper.GetAvatar(b.avatarMap, strconv.Itoa(message.From.ID), b.General)
+			rmsg.Text = helper.RemoveEmptyNewLines(rmsg.Text)
+			// channels don't have (always?) user information. see #410
+			if message.From != nil {
+				rmsg.Avatar = helper.GetAvatar(b.avatarMap, strconv.Itoa(message.From.ID), b.General)
+			}
 
 			b.Log.Debugf("<= Sending message from %s on %s to gateway", rmsg.Username, b.Account)
 			b.Log.Debugf("<= Message is %#v", rmsg)
@@ -385,7 +391,6 @@ func (b *Btelegram) sendMessage(chatid int64, username, text string) (string, er
 	m.Text = username + text
 	if b.GetString("MessageFormat") == "HTML" {
 		b.Log.Debug("Using mode HTML")
-		username = html.EscapeString(username)
 		m.Text = username + text
 		m.ParseMode = tgbotapi.ModeHTML
 	}
@@ -410,3 +415,14 @@ func (b *Btelegram) cacheAvatar(msg *config.Message) (string, error) {
 	}
 	return "", nil
 }
+
+func (b *Btelegram) handleQuote(message, quoteNick, quoteMessage string) string {
+	format := b.GetString("quoteformat")
+	if format == "" {
+		format = "{MESSAGE} (re @{QUOTENICK}: {QUOTEMESSAGE})"
+	}
+	format = strings.Replace(format, "{MESSAGE}", message, -1)
+	format = strings.Replace(format, "{QUOTENICK}", quoteNick, -1)
+	format = strings.Replace(format, "{QUOTEMESSAGE}", quoteMessage, -1)
+	return format
+}

bridge/xmpp/xmpp.go

@@ -180,18 +180,26 @@ func (b *Bxmpp) replaceAction(text string) (string, bool) {
 
 // handleUploadFile handles native upload of files
 func (b *Bxmpp) handleUploadFile(msg *config.Message) (string, error) {
+	var urldesc = ""
 	for _, f := range msg.Extra["file"] {
 		fi := f.(config.FileInfo)
 		if fi.Comment != "" {
 			msg.Text += fi.Comment + ": "
 		}
 		if fi.URL != "" {
-			msg.Text += fi.URL
+			msg.Text = fi.URL
+			if fi.Comment != "" {
+				msg.Text = fi.Comment + ": " + fi.URL
+				urldesc = fi.Comment
+			}
 		}
 		_, err := b.xc.Send(xmpp.Chat{Type: "groupchat", Remote: msg.Channel + "@" + b.GetString("Muc"), Text: msg.Username + msg.Text})
 		if err != nil {
 			return "", err
 		}
+		if fi.URL != "" {
+			b.xc.SendOOB(xmpp.Chat{Type: "groupchat", Remote: msg.Channel + "@" + b.GetString("Muc"), Ooburl: fi.URL, Oobdesc: urldesc})
+		}
 	}
 	return "", nil
 }

bridge/zulip/zulip.go

@@ -0,0 +1,170 @@
+package bzulip
+
+import (
+	"encoding/json"
+	"io/ioutil"
+	"strconv"
+	"time"
+
+	"github.com/42wim/matterbridge/bridge"
+	"github.com/42wim/matterbridge/bridge/config"
+	"github.com/42wim/matterbridge/bridge/helper"
+	gzb "github.com/matterbridge/gozulipbot"
+)
+
+type Bzulip struct {
+	q       *gzb.Queue
+	bot     *gzb.Bot
+	streams map[int]string
+	*bridge.Config
+}
+
+func New(cfg *bridge.Config) bridge.Bridger {
+	return &Bzulip{Config: cfg, streams: make(map[int]string)}
+}
+
+func (b *Bzulip) Connect() error {
+	bot := gzb.Bot{APIKey: b.GetString("token"), APIURL: b.GetString("server") + "/api/v1/", Email: b.GetString("login")}
+	bot.Init()
+	q, err := bot.RegisterAll()
+	b.q = q
+	b.bot = &bot
+	if err != nil {
+		b.Log.Errorf("Connect() %#v", err)
+		return err
+	}
+	// init stream
+	b.getChannel(0)
+	b.Log.Info("Connection succeeded")
+	go b.handleQueue()
+	return nil
+}
+
+func (b *Bzulip) Disconnect() error {
+	return nil
+}
+
+func (b *Bzulip) JoinChannel(channel config.ChannelInfo) error {
+	return nil
+}
+
+func (b *Bzulip) Send(msg config.Message) (string, error) {
+	b.Log.Debugf("=> Receiving %#v", msg)
+
+	// Delete message
+	if msg.Event == config.EVENT_MSG_DELETE {
+		if msg.ID == "" {
+			return "", nil
+		}
+		_, err := b.bot.UpdateMessage(msg.ID, "")
+		return "", err
+	}
+
+	// Upload a file if it exists
+	if msg.Extra != nil {
+		for _, rmsg := range helper.HandleExtra(&msg, b.General) {
+			b.sendMessage(rmsg)
+		}
+		if len(msg.Extra["file"]) > 0 {
+			return b.handleUploadFile(&msg)
+		}
+	}
+
+	// edit the message if we have a msg ID
+	if msg.ID != "" {
+		_, err := b.bot.UpdateMessage(msg.ID, msg.Username+msg.Text)
+		return "", err
+	}
+
+	// Post normal message
+	return b.sendMessage(msg)
+}
+
+func (b *Bzulip) getChannel(id int) string {
+	if name, ok := b.streams[id]; ok {
+		return name
+	}
+	streams, err := b.bot.GetRawStreams()
+	if err != nil {
+		b.Log.Errorf("getChannel: %#v", err)
+		return ""
+	}
+	for _, stream := range streams.Streams {
+		b.streams[stream.StreamID] = stream.Name
+	}
+	if name, ok := b.streams[id]; ok {
+		return name
+	}
+	return ""
+}
+
+func (b *Bzulip) handleQueue() error {
+	for {
+		messages, _ := b.q.GetEvents()
+		for _, m := range messages {
+			b.Log.Debugf("== Receiving %#v", m)
+			// ignore our own messages
+			if m.SenderEmail == b.GetString("login") {
+				continue
+			}
+			rmsg := config.Message{Username: m.SenderFullName, Text: m.Content, Channel: b.getChannel(m.StreamID), Account: b.Account, UserID: strconv.Itoa(m.SenderID), Avatar: m.AvatarURL}
+			b.Log.Debugf("<= Sending message from %s on %s to gateway", rmsg.Username, b.Account)
+			b.Log.Debugf("<= Message is %#v", rmsg)
+			b.Remote <- rmsg
+			b.q.LastEventID = m.ID
+		}
+		time.Sleep(time.Second * 3)
+	}
+}
+
+func (b *Bzulip) sendMessage(msg config.Message) (string, error) {
+	topic := "matterbridge"
+	if b.GetString("topic") != "" {
+		topic = b.GetString("topic")
+	}
+	m := gzb.Message{
+		Stream:  msg.Channel,
+		Topic:   topic,
+		Content: msg.Username + msg.Text,
+	}
+	resp, err := b.bot.Message(m)
+	if err != nil {
+		return "", err
+	}
+	if resp != nil {
+		defer resp.Body.Close()
+		res, err := ioutil.ReadAll(resp.Body)
+		if err != nil {
+			return "", err
+		}
+		var jr struct {
+			ID int `json:"id"`
+		}
+		err = json.Unmarshal(res, &jr)
+		if err != nil {
+			return "", err
+		}
+		return strconv.Itoa(jr.ID), nil
+	}
+	return "", nil
+}
+
+func (b *Bzulip) handleUploadFile(msg *config.Message) (string, error) {
+	for _, f := range msg.Extra["file"] {
+		fi := f.(config.FileInfo)
+		if fi.Comment != "" {
+			msg.Text += fi.Comment + ": "
+		}
+		if fi.URL != "" {
+			msg.Text = fi.URL
+			if fi.Comment != "" {
+				msg.Text = fi.Comment + ": " + fi.URL
+			}
+		}
+		_, err := b.sendMessage(*msg)
+		if err != nil {
+			return "", err
+		}
+	}
+	return "", nil
+}

changelog.md

@@ -1,3 +1,46 @@
+# v1.10.1
+## Bugfix
+* general: updated irc/xmpp/telegram libraries
+* mattermost/slack/rocketchat: Fix iconurl regression. Closes #430
+* mattermost/slack: Use uuid instead of userid. Fixes #429
+* slack: Avatar spoofing from Slack to Discord with uppercase in nick doesn't work (#433)
+* irc: Fix format string bug (irc) (#428)
+* irc: Colorize username sent to IRC using its crc32 IEEE checksum (#423). See `ColorNicks` in matterbridge.toml.sample
+* irc: Add support for CJK to/from utf-8 (irc). #400
+* telegram: Add QuoteFormat option (telegram). Closes #41. See `QuoteFormat` in matterbridge.toml.sample
+* xmpp: Send attached files to XMPP in different message with OOB data and without body (#421)
+
+# v1.10.0
+## New features
+* general: Add support for reloading all settings automatically after changing config except connection and gateway configuration. Closes #373
+* zulip: New protocol support added (https://zulipchat.com)
+
+## Enhancements
+* general: Handle file comment better
+* steam: Handle file uploads to mediaserver (steam)
+* slack: Properly set Slack user who initiated slash command (#394)
+
+## Bugfix
+* general: Use only alphanumeric for file uploads to mediaserver. Closes #416
+* general: Fix crash on invalid filenames
+* general: Fix regression in ReplaceMessages and ReplaceNicks. Closes #407
+* telegram: Fix possible nil when using channels (telegram). #410
+* telegram: Fix panic (telegram). Closes #410
+* telegram: Handle channel posts correctly
+* mattermost: Update GetFileLinks to API_V4
+
+# v1.9.1
+## New features
+* telegram: Add QuoteDisable option (telegram). Closes #399. See QuoteDisable in matterbridge.toml.sample
+## Enhancements
+* discord: Send mediaserver link to Discord in Webhook mode (discord) (#405)
+* mattermost: Print list of valid team names when team not found (#390)
+* slack: Strip markdown URLs with blank text (slack) (#392)
+## Bugfix
+* slack/mattermost: Make our callbackid more unique. Fixes issue with running multiple matterbridge on the same channel (slack,mattermost)
+* telegram: fix newlines in multiline messages #399
+* telegram: Revert #378
+
 # v1.9.0 (the refactor release)
 ## New features
 * general: better debug messages

gateway/gateway.go

@@ -17,12 +17,14 @@ import (
 	"github.com/42wim/matterbridge/bridge/steam"
 	"github.com/42wim/matterbridge/bridge/telegram"
 	"github.com/42wim/matterbridge/bridge/xmpp"
+	"github.com/42wim/matterbridge/bridge/zulip"
 	log "github.com/sirupsen/logrus"
 	//	"github.com/davecgh/go-spew/spew"
 	"crypto/sha1"
 	"github.com/hashicorp/golang-lru"
 	"github.com/peterhellberg/emojilib"
 	"net/http"
+	"path/filepath"
 	"regexp"
 	"strings"
 	"time"
@@ -61,6 +63,7 @@ var bridgeMap = map[string]bridge.Factory{
 	"steam":      bsteam.New,
 	"telegram":   btelegram.New,
 	"xmpp":       bxmpp.New,
+	"zulip":      bzulip.New,
 }
 
 func init() {
@@ -409,6 +412,7 @@ func (gw *Gateway) modifyMessage(msg *config.Message) {
 }
 
 func (gw *Gateway) handleFiles(msg *config.Message) {
+	reg := regexp.MustCompile("[^a-zA-Z0-9]+")
 	// if we don't have a attachfield or we don't have a mediaserver configured return
 	if msg.Extra == nil || gw.Config.General.MediaServerUpload == "" {
 		return
@@ -421,15 +425,23 @@ func (gw *Gateway) handleFiles(msg *config.Message) {
 		}
 		for i, f := range msg.Extra["file"] {
 			fi := f.(config.FileInfo)
+			ext := filepath.Ext(fi.Name)
+			fi.Name = fi.Name[0 : len(fi.Name)-len(ext)]
+			fi.Name = reg.ReplaceAllString(fi.Name, "_")
+			fi.Name = fi.Name + ext
 			sha1sum := fmt.Sprintf("%x", sha1.Sum(*fi.Data))
 			reader := bytes.NewReader(*fi.Data)
 			url := gw.Config.General.MediaServerUpload + "/" + sha1sum + "/" + fi.Name
 			durl := gw.Config.General.MediaServerDownload + "/" + sha1sum + "/" + fi.Name
 			extra := msg.Extra["file"][i].(config.FileInfo)
 			extra.URL = durl
-			req, _ := http.NewRequest("PUT", url, reader)
+			req, err := http.NewRequest("PUT", url, reader)
+			if err != nil {
+				flog.Errorf("mediaserver upload failed: %#v", err)
+				continue
+			}
 			req.Header.Set("Content-Type", "binary/octet-stream")
-			_, err := client.Do(req)
+			_, err = client.Do(req)
 			if err != nil {
 				flog.Errorf("mediaserver upload failed: %#v", err)
 				continue

matterbridge.go

@@ -13,7 +13,7 @@ import (
 )
 
 var (
-	version = "1.9.0"
+	version = "1.10.1"
 	githash string
 )
 

matterbridge.toml.sample

@@ -64,6 +64,9 @@ NickServPassword="secret"
 #OPTIONAL only used for quakenet auth
 NickServUsername="username"
 
+## RELOADABLE SETTINGS
+## Settings below can be reloaded by editing the file
+
 #Flood control
 #Delay in milliseconds between each message send to the IRC server
 #OPTIONAL (default 1300)
@@ -89,6 +92,10 @@ MessageSplit=false
 #OPTIONAL (default 0)
 RejoinDelay=0
 
+#ColorNicks will show each nickname in a different color.
+#Only works in IRC right now.
+ColorNicks=false
+
 #Nicks you want to ignore. 
 #Messages from those users will not be sent to other bridges.
 #OPTIONAL
@@ -184,6 +191,9 @@ Nick="xmppbot"
 #OPTIONAL (default false)
 SkipTLSVerify=true
 
+## RELOADABLE SETTINGS
+## Settings below can be reloaded by editing the file
+
 #Nicks you want to ignore. 
 #Messages from those users will not be sent to other bridges.
 #OPTIONAL
@@ -265,6 +275,9 @@ Muc="conf.hipchat.com"
 #REQUIRED
 Nick="yourlogin"
 
+## RELOADABLE SETTINGS
+## Settings below can be reloaded by editing the file
+
 #Nicks you want to ignore. 
 #Messages from those users will not be sent to other bridges.
 #OPTIONAL
@@ -382,6 +395,9 @@ IconURL="http://youricon.png"
 #OPTIONAL (default false)
 SkipTLSVerify=true
 
+## RELOADABLE SETTINGS
+## Settings below can be reloaded by editing the file
+
 #how to format the list of IRC nicks when displayed in mattermost. 
 #Possible options are "table" and "plain"
 #OPTIONAL (default plain)
@@ -482,6 +498,9 @@ ShowTopicChange=false
 #REQUIRED
 Token="Yourtokenhere"
 
+## RELOADABLE SETTINGS
+## Settings below can be reloaded by editing the file
+
 #Nicks you want to ignore. 
 #Messages from those users will not be sent to other bridges.
 #OPTIONAL
@@ -577,6 +596,9 @@ WebhookBindAddress="0.0.0.0:9999"
 #OPTIONAL
 IconURL="https://robohash.org/{NICK}.png?size=48x48"
 
+## RELOADABLE SETTINGS
+## Settings below can be reloaded by editing the file
+
 #how to format the list of IRC nicks when displayed in slack
 #Possible options are "table" and "plain"
 #OPTIONAL (default plain)
@@ -680,6 +702,9 @@ Token="Yourtokenhere"
 #REQUIRED
 Server="yourservername"
 
+## RELOADABLE SETTINGS
+## Settings below can be reloaded by editing the file
+
 #Shows title, description and URL of embedded messages (sent by other bots)
 #OPTIONAL (default false)
 ShowEmbeds=false
@@ -770,6 +795,9 @@ ShowTopicChange=false
 #REQUIRED
 Token="Yourtokenhere"
 
+## RELOADABLE SETTINGS
+## Settings below can be reloaded by editing the file
+
 #OPTIONAL (default empty)
 #Only supported format is "HTML", messages will be sent in html parsemode.
 #See https://core.telegram.org/bots/api#html-style
@@ -787,6 +815,14 @@ UseFirstName=false
 #OPTIONAL (default false)
 UseInsecureURL=false
 
+#Disable quoted/reply messages
+#OPTIONAL (default false)
+QuoteDisable=false
+
+#Format quoted/reply messages
+#OPTIONAL (default "{MESSAGE} (re @{QUOTENICK}: {QUOTEMESSAGE})")
+QuoteFormat="{MESSAGE} (re @{QUOTENICK}: {QUOTEMESSAGE})"
+
 #Disable sending of edits to other bridges
 #OPTIONAL (default false)
 EditDisable=false
@@ -832,6 +868,11 @@ Label=""
 #The string "{BRIDGE}" (case sensitive) will be replaced by the sending bridge
 #The string "{LABEL}" (case sensitive) will be replaced by label= field of the sending bridge
 #The string "{PROTOCOL}" (case sensitive) will be replaced by the protocol used by the bridge
+#
+#WARNING: if you have set MessageFormat="HTML" be sure that this format matches the guidelines
+#on https://core.telegram.org/bots/api#html-style otherwise the message will not go through to
+#telegram! eg <{NICK}> should be &lt;{NICK}&gt;
+#
 #OPTIONAL (default empty)
 RemoteNickFormat="[{PROTOCOL}] <{NICK}> "
 
@@ -883,6 +924,9 @@ NoTLS=false
 #OPTIONAL (default false)
 SkipTLSVerify=true
 
+## RELOADABLE SETTINGS
+## Settings below can be reloaded by editing the file
+
 #Whether to prefix messages from other bridges to rocketchat with the sender's nick. 
 #Useful if username overrides for incoming webhooks isn't enabled on the 
 #rocketchat server. If you set PrefixMessagesWithNick to true, each message 
@@ -970,6 +1014,9 @@ Password="yourpass"
 #OPTIONAL (default false)
 NoHomeServerSuffix=false
 
+## RELOADABLE SETTINGS
+## Settings below can be reloaded by editing the file
+
 #Whether to prefix messages from other bridges to matrix with the sender's nick. 
 #Useful if username overrides for incoming webhooks isn't enabled on the 
 #matrix server. If you set PrefixMessagesWithNick to true, each message 
@@ -1051,6 +1098,9 @@ Password="yourpass"
 #OPTIONAL 
 Authcode="ABCE12"
 
+## RELOADABLE SETTINGS
+## Settings below can be reloaded by editing the file
+
 #Whether to prefix messages from other bridges to matrix with the sender's nick. 
 #Useful if username overrides for incoming webhooks isn't enabled on the 
 #matrix server. If you set PrefixMessagesWithNick to true, each message 
@@ -1113,6 +1163,90 @@ StripNick=false
 #OPTIONAL (default false)
 ShowTopicChange=false
 
+###################################################################
+#zulip section
+###################################################################
+[zulip]
+#You can configure multiple servers "[zulip.name]" or "[zulip.name2]"
+#In this example we use [zulip.streamchat]
+#REQUIRED
+
+[zulip.streamchat]
+#Token to connect with zulip API (called bot API key in Settings - Your bots)
+#REQUIRED
+Token="Yourtokenhere"
+
+#Username of the bot, normally called yourbot-bot@yourserver.zulipchat.com 
+#See username in Settings - Your bots 
+#REQUIRED
+Login="yourbot-bot@yourserver.zulipchat.com"
+
+#Servername of your zulip instance
+#REQUIRED 
+Server="https://yourserver.zulipchat.com"
+
+#Topic of the messages matterbridge will use
+#OPTIONAL (default "matterbridge")
+Topic="matterbridge"
+
+## RELOADABLE SETTINGS
+## Settings below can be reloaded by editing the file
+
+#Nicks you want to ignore. 
+#Messages from those users will not be sent to other bridges.
+#OPTIONAL
+IgnoreNicks="spammer1 spammer2"
+
+#Messages you want to ignore. 
+#Messages matching these regexp will be ignored and not sent to other bridges
+#See https://regex-golang.appspot.com/assets/html/index.html for more regex info
+#OPTIONAL (example below ignores messages starting with ~~ or messages containing badword
+IgnoreMessages="^~~ badword"
+
+#messages you want to replace.
+#it replaces outgoing messages from the bridge.
+#so you need to place it by the sending bridge definition.
+#regular expressions supported
+#some examples:
+#this replaces cat => dog and sleep => awake
+#replacemessages=[ ["cat","dog"], ["sleep","awake"] ]
+#this replaces every number with number.  123 => numbernumbernumber
+#replacemessages=[ ["[0-9]","number"] ]
+#optional (default empty)
+ReplaceMessages=[ ["cat","dog"] ]
+
+#nicks you want to replace.
+#see replacemessages for syntaxa
+#optional (default empty)
+ReplaceNicks=[ ["user--","user"] ]
+
+#extra label that can be used in the RemoteNickFormat
+#optional (default empty)
+Label=""
+
+#RemoteNickFormat defines how remote users appear on this bridge 
+#The string "{NICK}" (case sensitive) will be replaced by the actual nick / username.
+#The string "{BRIDGE}" (case sensitive) will be replaced by the sending bridge
+#The string "{LABEL}" (case sensitive) will be replaced by label= field of the sending bridge
+#The string "{PROTOCOL}" (case sensitive) will be replaced by the protocol used by the bridge
+#OPTIONAL (default empty)
+RemoteNickFormat="[{PROTOCOL}] <{NICK}> "
+
+#Enable to show users joins/parts from other bridges 
+#Currently works for messages from the following bridges: irc, mattermost, slack
+#OPTIONAL (default false)
+ShowJoinPart=false
+
+#StripNick only allows alphanumerical nicks. See https://github.com/42wim/matterbridge/issues/285
+#It will strip other characters from the nick
+#OPTIONAL (default false)
+StripNick=false
+
+#Enable to show topic changes from other bridges 
+#Only works hiding/show topic changes from slack bridge for now
+#OPTIONAL (default false)
+ShowTopicChange=false
+
 ###################################################################
 #API
 ###################################################################
@@ -1153,6 +1287,10 @@ RemoteNickFormat="{NICK}"
 ###################################################################
 # Settings here are defaults that each protocol can override
 [general]
+
+## RELOADABLE SETTINGS
+## Settings below can be reloaded by editing the file
+
 #RemoteNickFormat defines how remote users appear on this bridge 
 #The string "{NICK}" (case sensitive) will be replaced by the actual nick / username.
 #The string "{BRIDGE}" (case sensitive) will be replaced by the sending bridge
@@ -1237,6 +1375,7 @@ enable=true
     #           - encrypted rooms are not supported in matrix
     #steam      - chatid (a large number). 
     #             The number in the URL when you click "enter chat room" in the browser
+    #zulip      - stream (without the #)
     #                  
     #REQUIRED
     channel="#testing"

matterclient/matterclient.go

@@ -190,7 +190,11 @@ func (m *MMClient) Login() error {
 	}
 
 	if m.Team == nil {
-		return errors.New("team not found")
+		validTeamNames := make([]string, len(m.OtherTeams))
+		for i, t := range m.OtherTeams {
+			validTeamNames[i] = t.Team.Name
+		}
+		return fmt.Errorf("Team '%s' not found in %v", m.Credentials.Team, validTeamNames)
 	}
 
 	m.wsConnect()
@@ -570,7 +574,7 @@ func (m *MMClient) GetFileLinks(filenames []string) []string {
 		res, resp := m.Client.GetFileLink(f)
 		if resp.Error != nil {
 			// public links is probably disabled, create the link ourselves
-			output = append(output, uriScheme+m.Credentials.Server+model.API_URL_SUFFIX_V3+"/files/"+f+"/get")
+			output = append(output, uriScheme+m.Credentials.Server+model.API_URL_SUFFIX_V4+"/files/"+f)
 			continue
 		}
 		output = append(output, res)
@@ -768,6 +772,14 @@ func (m *MMClient) GetStatus(userId string) string {
 	return "offline"
 }
 
+func (m *MMClient) UpdateStatus(userId string, status string) error {
+	_, resp := m.Client.UpdateUserStatus(userId, &model.Status{Status: status})
+	if resp.Error != nil {
+		return resp.Error
+	}
+	return nil
+}
+
 func (m *MMClient) GetStatuses() map[string]string {
 	var ids []string
 	statuses := make(map[string]string)

vendor/github.com/dfordsoft/golib/ic/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 DForD Software
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

vendor/github.com/dfordsoft/golib/ic/convutf8.go

@@ -0,0 +1,72 @@
+// Package ic convert text between CJK and UTF-8 in pure Go way
+package ic
+
+import (
+	"bytes"
+	"errors"
+	"io/ioutil"
+	"strings"
+
+	"golang.org/x/text/encoding"
+	"golang.org/x/text/encoding/japanese"
+	"golang.org/x/text/encoding/korean"
+	"golang.org/x/text/encoding/simplifiedchinese"
+	"golang.org/x/text/encoding/traditionalchinese"
+	"golang.org/x/text/transform"
+)
+
+var (
+	transformers = map[string]encoding.Encoding{
+		"gbk":         simplifiedchinese.GBK,
+		"cp936":       simplifiedchinese.GBK,
+		"windows-936": simplifiedchinese.GBK,
+		"gb18030":     simplifiedchinese.GB18030,
+		"gb2312":      simplifiedchinese.HZGB2312,
+		"big5":        traditionalchinese.Big5,
+		"big-5":       traditionalchinese.Big5,
+		"cp950":       traditionalchinese.Big5,
+		"euc-kr":      korean.EUCKR,
+		"euckr":       korean.EUCKR,
+		"cp949":       korean.EUCKR,
+		"euc-jp":      japanese.EUCJP,
+		"eucjp":       japanese.EUCJP,
+		"shift-jis":   japanese.ShiftJIS,
+		"iso-2022-jp": japanese.ISO2022JP,
+		"cp932":       japanese.ISO2022JP,
+		"windows-31j": japanese.ISO2022JP,
+	}
+)
+
+// ToUTF8 convert from CJK encoding to UTF-8
+func ToUTF8(from string, s []byte) ([]byte, error) {
+	var reader *transform.Reader
+
+	transformer, ok := transformers[strings.ToLower(from)]
+	if !ok {
+		return s, errors.New("Unsupported encoding " + from)
+	}
+	reader = transform.NewReader(bytes.NewReader(s), transformer.NewDecoder())
+
+	d, e := ioutil.ReadAll(reader)
+	if e != nil {
+		return nil, e
+	}
+	return d, nil
+}
+
+// FromUTF8 convert from UTF-8 encoding to CJK encoding
+func FromUTF8(to string, s []byte) ([]byte, error) {
+	var reader *transform.Reader
+
+	transformer, ok := transformers[strings.ToLower(to)]
+	if !ok {
+		return s, errors.New("Unsupported encoding " + to)
+	}
+	reader = transform.NewReader(bytes.NewReader(s), transformer.NewEncoder())
+
+	d, e := ioutil.ReadAll(reader)
+	if e != nil {
+		return nil, e
+	}
+	return d, nil
+}

vendor/github.com/dfordsoft/golib/ic/ic.go

@@ -0,0 +1,31 @@
+package ic
+
+import "log"
+
+// Convert convert bytes from CJK or UTF-8 to UTF-8 or CJK
+func Convert(from string, to string, src []byte) []byte {
+	if to == "utf-8" {
+		out, e := ToUTF8(from, src)
+		if e == nil {
+			return out
+		}
+		log.Printf("converting from %s to UTF-8 failed: %v", from, e)
+		return src
+	}
+
+	if from == "utf-8" {
+		out, e := FromUTF8(to, src)
+		if e == nil {
+			return out
+		}
+		log.Printf("converting from UTF-8 to %s failed: %v", to, e)
+		return src
+	}
+	log.Println("only converting between CJK encodings and UTF-8 is supported")
+	return src
+}
+
+// ConvertString convert string from CJK or UTF-8 to UTF-8 or CJK
+func ConvertString(from string, to string, src string) string {
+	return string(Convert(from, to, []byte(src)))
+}

vendor/github.com/go-telegram-bot-api/telegram-bot-api/bot.go

@@ -7,6 +7,7 @@ import (
 	"encoding/json"
 	"errors"
 	"fmt"
+	"io"
 	"io/ioutil"
 	"log"
 	"net/http"
@@ -67,33 +68,51 @@ func (bot *BotAPI) MakeRequest(endpoint string, params url.Values) (APIResponse,
 	}
 	defer resp.Body.Close()
 
-	if resp.StatusCode == http.StatusForbidden {
-		return APIResponse{}, errors.New(ErrAPIForbidden)
-	}
-
-	if resp.StatusCode != http.StatusOK {
-		return APIResponse{}, errors.New(http.StatusText(resp.StatusCode))
-	}
-
-	bytes, err := ioutil.ReadAll(resp.Body)
+	var apiResp APIResponse
+	bytes, err := bot.decodeAPIResponse(resp.Body, &apiResp)
 	if err != nil {
-		return APIResponse{}, err
+		return apiResp, err
 	}
 
 	if bot.Debug {
-		log.Println(endpoint, string(bytes))
+		log.Printf("%s resp: %s", endpoint, bytes)
 	}
 
-	var apiResp APIResponse
-	json.Unmarshal(bytes, &apiResp)
-
 	if !apiResp.Ok {
-		return apiResp, errors.New(apiResp.Description)
+		parameters := ResponseParameters{}
+		if apiResp.Parameters != nil {
+			parameters = *apiResp.Parameters
+		}
+		return apiResp, Error{apiResp.Description, parameters}
 	}
 
 	return apiResp, nil
 }
 
+// decodeAPIResponse decode response and return slice of bytes if debug enabled.
+// If debug disabled, just decode http.Response.Body stream to APIResponse struct
+// for efficient memory usage
+func (bot *BotAPI) decodeAPIResponse(responseBody io.Reader, resp *APIResponse) (_ []byte, err error) {
+	if !bot.Debug {
+		dec := json.NewDecoder(responseBody)
+		err = dec.Decode(resp)
+		return
+	}
+
+	// if debug, read reponse body
+	data, err := ioutil.ReadAll(responseBody)
+	if err != nil {
+		return
+	}
+
+	err = json.Unmarshal(data, resp)
+	if err != nil {
+		return
+	}
+
+	return data, nil
+}
+
 // makeMessageRequest makes a request to a method that returns a Message.
 func (bot *BotAPI) makeMessageRequest(endpoint string, params url.Values) (Message, error) {
 	resp, err := bot.MakeRequest(endpoint, params)
@@ -712,24 +731,28 @@ func (bot *BotAPI) RestrictChatMember(config RestrictChatMemberConfig) (APIRespo
 	}
 	v.Add("user_id", strconv.Itoa(config.UserID))
 
-	if &config.CanSendMessages != nil {
+	if config.CanSendMessages != nil {
 		v.Add("can_send_messages", strconv.FormatBool(*config.CanSendMessages))
 	}
-	if &config.CanSendMediaMessages != nil {
+	if config.CanSendMediaMessages != nil {
 		v.Add("can_send_media_messages", strconv.FormatBool(*config.CanSendMediaMessages))
 	}
-	if &config.CanSendOtherMessages != nil {
+	if config.CanSendOtherMessages != nil {
 		v.Add("can_send_other_messages", strconv.FormatBool(*config.CanSendOtherMessages))
 	}
-	if &config.CanAddWebPagePreviews != nil {
+	if config.CanAddWebPagePreviews != nil {
 		v.Add("can_add_web_page_previews", strconv.FormatBool(*config.CanAddWebPagePreviews))
 	}
+	if config.UntilDate != 0 {
+		v.Add("until_date", strconv.FormatInt(config.UntilDate, 10))
+	}
 
 	bot.debugLog("restrictChatMember", v, nil)
 
 	return bot.MakeRequest("restrictChatMember", v)
 }
 
+// PromoteChatMember add admin rights to user
 func (bot *BotAPI) PromoteChatMember(config PromoteChatMemberConfig) (APIResponse, error) {
 	v := url.Values{}
 
@@ -742,28 +765,28 @@ func (bot *BotAPI) PromoteChatMember(config PromoteChatMemberConfig) (APIRespons
 	}
 	v.Add("user_id", strconv.Itoa(config.UserID))
 
-	if &config.CanChangeInfo != nil {
+	if config.CanChangeInfo != nil {
 		v.Add("can_change_info", strconv.FormatBool(*config.CanChangeInfo))
 	}
-	if &config.CanPostMessages != nil {
+	if config.CanPostMessages != nil {
 		v.Add("can_post_messages", strconv.FormatBool(*config.CanPostMessages))
 	}
-	if &config.CanEditMessages != nil {
+	if config.CanEditMessages != nil {
 		v.Add("can_edit_messages", strconv.FormatBool(*config.CanEditMessages))
 	}
-	if &config.CanDeleteMessages != nil {
+	if config.CanDeleteMessages != nil {
 		v.Add("can_delete_messages", strconv.FormatBool(*config.CanDeleteMessages))
 	}
-	if &config.CanInviteUsers != nil {
+	if config.CanInviteUsers != nil {
 		v.Add("can_invite_users", strconv.FormatBool(*config.CanInviteUsers))
 	}
-	if &config.CanRestrictMembers != nil {
+	if config.CanRestrictMembers != nil {
 		v.Add("can_restrict_members", strconv.FormatBool(*config.CanRestrictMembers))
 	}
-	if &config.CanPinMessages != nil {
+	if config.CanPinMessages != nil {
 		v.Add("can_pin_messages", strconv.FormatBool(*config.CanPinMessages))
 	}
-	if &config.CanPromoteMembers != nil {
+	if config.CanPromoteMembers != nil {
 		v.Add("can_promote_members", strconv.FormatBool(*config.CanPromoteMembers))
 	}
 
@@ -846,6 +869,9 @@ func (bot *BotAPI) GetInviteLink(config ChatConfig) (string, error) {
 	}
 
 	resp, err := bot.MakeRequest("exportChatInviteLink", v)
+	if err != nil {
+		return "", err
+	}
 
 	var inviteLink string
 	err = json.Unmarshal(resp.Result, &inviteLink)
@@ -853,7 +879,7 @@ func (bot *BotAPI) GetInviteLink(config ChatConfig) (string, error) {
 	return inviteLink, err
 }
 
-// Pin message in supergroup
+// PinChatMessage pin message in supergroup
 func (bot *BotAPI) PinChatMessage(config PinChatMessageConfig) (APIResponse, error) {
 	v, err := config.values()
 	if err != nil {
@@ -865,7 +891,7 @@ func (bot *BotAPI) PinChatMessage(config PinChatMessageConfig) (APIResponse, err
 	return bot.MakeRequest(config.method(), v)
 }
 
-// Unpin message in supergroup
+// UnpinChatMessage unpin message in supergroup
 func (bot *BotAPI) UnpinChatMessage(config UnpinChatMessageConfig) (APIResponse, error) {
 	v, err := config.values()
 	if err != nil {
@@ -875,4 +901,52 @@ func (bot *BotAPI) UnpinChatMessage(config UnpinChatMessageConfig) (APIResponse,
 	bot.debugLog(config.method(), v, nil)
 
 	return bot.MakeRequest(config.method(), v)
-}
+}
+
+// SetChatTitle change title of chat.
+func (bot *BotAPI) SetChatTitle(config SetChatTitleConfig) (APIResponse, error) {
+	v, err := config.values()
+	if err != nil {
+		return APIResponse{}, err
+	}
+
+	bot.debugLog(config.method(), v, nil)
+
+	return bot.MakeRequest(config.method(), v)
+}
+
+// SetChatDescription change description of chat.
+func (bot *BotAPI) SetChatDescription(config SetChatDescriptionConfig) (APIResponse, error) {
+	v, err := config.values()
+	if err != nil {
+		return APIResponse{}, err
+	}
+
+	bot.debugLog(config.method(), v, nil)
+
+	return bot.MakeRequest(config.method(), v)
+}
+
+// SetChatPhoto change photo of chat.
+func (bot *BotAPI) SetChatPhoto(config SetChatPhotoConfig) (APIResponse, error) {
+	params, err := config.params()
+	if err != nil {
+		return APIResponse{}, err
+	}
+
+	file := config.getFile()
+
+	return bot.UploadFile(config.method(), params, config.name(), file)
+}
+
+// DeleteChatPhoto delete photo of chat.
+func (bot *BotAPI) DeleteChatPhoto(config DeleteChatPhotoConfig) (APIResponse, error) {
+	v, err := config.values()
+	if err != nil {
+		return APIResponse{}, err
+	}
+
+	bot.debugLog(config.method(), v, nil)
+
+	return bot.MakeRequest(config.method(), v)
+}

vendor/github.com/go-telegram-bot-api/telegram-bot-api/configs.go

@@ -676,7 +676,7 @@ type SetGameScoreConfig struct {
 	Score              int
 	Force              bool
 	DisableEditMessage bool
-	ChatID             int
+	ChatID             int64
 	ChannelUsername    string
 	MessageID          int
 	InlineMessageID    string
@@ -689,7 +689,7 @@ func (config SetGameScoreConfig) values() (url.Values, error) {
 	v.Add("score", strconv.Itoa(config.Score))
 	if config.InlineMessageID == "" {
 		if config.ChannelUsername == "" {
-			v.Add("chat_id", strconv.Itoa(config.ChatID))
+			v.Add("chat_id", strconv.FormatInt(config.ChatID, 10))
 		} else {
 			v.Add("chat_id", config.ChannelUsername)
 		}
@@ -1038,8 +1038,8 @@ func (config DeleteMessageConfig) values() (url.Values, error) {
 
 // PinChatMessageConfig contains information of a message in a chat to pin.
 type PinChatMessageConfig struct {
-	ChatID int64
-	MessageID int
+	ChatID              int64
+	MessageID           int
 	DisableNotification bool
 }
 
@@ -1072,4 +1072,74 @@ func (config UnpinChatMessageConfig) values() (url.Values, error) {
 	v.Add("chat_id", strconv.FormatInt(config.ChatID, 10))
 
 	return v, nil
-}
+}
+
+// SetChatTitleConfig contains information for change chat title.
+type SetChatTitleConfig struct {
+	ChatID int64
+	Title  string
+}
+
+func (config SetChatTitleConfig) method() string {
+	return "setChatTitle"
+}
+
+func (config SetChatTitleConfig) values() (url.Values, error) {
+	v := url.Values{}
+
+	v.Add("chat_id", strconv.FormatInt(config.ChatID, 10))
+	v.Add("title", config.Title)
+
+	return v, nil
+}
+
+// SetChatDescriptionConfig contains information for change chat description.
+type SetChatDescriptionConfig struct {
+	ChatID      int64
+	Description string
+}
+
+func (config SetChatDescriptionConfig) method() string {
+	return "setChatDescription"
+}
+
+func (config SetChatDescriptionConfig) values() (url.Values, error) {
+	v := url.Values{}
+
+	v.Add("chat_id", strconv.FormatInt(config.ChatID, 10))
+	v.Add("description", config.Description)
+
+	return v, nil
+}
+
+// SetChatPhotoConfig contains information for change chat photo
+type SetChatPhotoConfig struct {
+	BaseFile
+}
+
+// name returns the field name for the Photo.
+func (config SetChatPhotoConfig) name() string {
+	return "photo"
+}
+
+// method returns Telegram API method name for sending Photo.
+func (config SetChatPhotoConfig) method() string {
+	return "setChatPhoto"
+}
+
+// DeleteChatPhotoConfig contains information for delete chat photo.
+type DeleteChatPhotoConfig struct {
+	ChatID int64
+}
+
+func (config DeleteChatPhotoConfig) method() string {
+	return "deleteChatPhoto"
+}
+
+func (config DeleteChatPhotoConfig) values() (url.Values, error) {
+	v := url.Values{}
+
+	v.Add("chat_id", strconv.FormatInt(config.ChatID, 10))
+
+	return v, nil
+}

vendor/github.com/go-telegram-bot-api/telegram-bot-api/helpers.go

@@ -19,6 +19,13 @@ func NewMessage(chatID int64, text string) MessageConfig {
 	}
 }
 
+func NewDeleteMessage(chatID int64, messageID int) DeleteMessageConfig {
+	return DeleteMessageConfig{
+		ChatID:    chatID,
+		MessageID: messageID,
+	}
+}
+
 // NewMessageToChannel creates a new Message that is sent to a channel
 // by username.
 //
@@ -641,7 +648,7 @@ func NewCallbackWithAlert(id, text string) CallbackConfig {
 	}
 }
 
-// NewInvoice created a new Invoice request to the user.
+// NewInvoice creates a new Invoice request to the user.
 func NewInvoice(chatID int64, title, description, payload, providerToken, startParameter, currency string, prices *[]LabeledPrice) InvoiceConfig {
 	return InvoiceConfig{
 		BaseChat:       BaseChat{ChatID: chatID},
@@ -653,3 +660,34 @@ func NewInvoice(chatID int64, title, description, payload, providerToken, startP
 		Currency:       currency,
 		Prices:         prices}
 }
+
+// NewSetChatPhotoUpload creates a new chat photo uploader.
+//
+// chatID is where to send it, file is a string path to the file,
+// FileReader, or FileBytes.
+//
+// Note that you must send animated GIFs as a document.
+func NewSetChatPhotoUpload(chatID int64, file interface{}) SetChatPhotoConfig {
+	return SetChatPhotoConfig{
+		BaseFile: BaseFile{
+			BaseChat:    BaseChat{ChatID: chatID},
+			File:        file,
+			UseExisting: false,
+		},
+	}
+}
+
+// NewSetChatPhotoShare shares an existing photo.
+// You may use this to reshare an existing photo without reuploading it.
+//
+// chatID is where to send it, fileID is the ID of the file
+// already uploaded.
+func NewSetChatPhotoShare(chatID int64, fileID string) SetChatPhotoConfig {
+	return SetChatPhotoConfig{
+		BaseFile: BaseFile{
+			BaseChat:    BaseChat{ChatID: chatID},
+			FileID:      fileID,
+			UseExisting: true,
+		},
+	}
+}

vendor/github.com/go-telegram-bot-api/telegram-bot-api/types.go

@@ -56,6 +56,7 @@ type User struct {
 	LastName     string `json:"last_name"`     // optional
 	UserName     string `json:"username"`      // optional
 	LanguageCode string `json:"language_code"` // optional
+	IsBot        bool   `json:"is_bot"`        // optional
 }
 
 // String displays a simple text version of a user.
@@ -173,21 +174,23 @@ func (m *Message) Time() time.Time {
 	return time.Unix(int64(m.Date), 0)
 }
 
-// IsCommand returns true if message starts with '/'.
+// IsCommand returns true if message starts with a "bot_command" entity.
 func (m *Message) IsCommand() bool {
-	return m.Text != "" && m.Text[0] == '/'
+	if m.Entities == nil || len(*m.Entities) == 0 {
+		return false
+	}
+
+	entity := (*m.Entities)[0]
+	return entity.Offset == 0 && entity.Type == "bot_command"
 }
 
 // Command checks if the message was a command and if it was, returns the
 // command. If the Message was not a command, it returns an empty string.
 //
-// If the command contains the at bot syntax, it removes the bot name.
+// If the command contains the at name syntax, it is removed. Use
+// CommandWithAt() if you do not want that.
 func (m *Message) Command() string {
-	if !m.IsCommand() {
-		return ""
-	}
-
-	command := strings.SplitN(m.Text, " ", 2)[0][1:]
+	command := m.CommandWithAt()
 
 	if i := strings.Index(command, "@"); i != -1 {
 		command = command[:i]
@@ -196,20 +199,42 @@ func (m *Message) Command() string {
 	return command
 }
 
+// CommandWithAt checks if the message was a command and if it was, returns the
+// command. If the Message was not a command, it returns an empty string.
+//
+// If the command contains the at name syntax, it is not removed. Use Command()
+// if you want that.
+func (m *Message) CommandWithAt() string {
+	if !m.IsCommand() {
+		return ""
+	}
+
+	// IsCommand() checks that the message begins with a bot_command entity
+	entity := (*m.Entities)[0]
+	return m.Text[1:entity.Length]
+}
+
 // CommandArguments checks if the message was a command and if it was,
 // returns all text after the command name. If the Message was not a
 // command, it returns an empty string.
+//
+// Note: The first character after the command name is omitted:
+// - "/foo bar baz" yields "bar baz", not " bar baz"
+// - "/foo-bar baz" yields "bar baz", too
+// Even though the latter is not a command conforming to the spec, the API
+// marks "/foo" as command entity.
 func (m *Message) CommandArguments() string {
 	if !m.IsCommand() {
 		return ""
 	}
 
-	split := strings.SplitN(m.Text, " ", 2)
-	if len(split) != 2 {
-		return ""
+	// IsCommand() checks that the message begins with a bot_command entity
+	entity := (*m.Entities)[0]
+	if len(m.Text) == entity.Length {
+		return "" // The command makes up the whole message
 	}
 
-	return split[1]
+	return m.Text[entity.Length+1:]
 }
 
 // MessageEntity contains information about data in a Message.
@@ -265,6 +290,7 @@ type Sticker struct {
 	Thumbnail *PhotoSize `json:"thumb"`     // optional
 	Emoji     string     `json:"emoji"`     // optional
 	FileSize  int        `json:"file_size"` // optional
+	SetName   string     `json:"set_name"`  // optional
 }
 
 // Video contains information about a video.
@@ -385,7 +411,7 @@ type InlineKeyboardButton struct {
 	SwitchInlineQuery            *string       `json:"switch_inline_query,omitempty"`              // optional
 	SwitchInlineQueryCurrentChat *string       `json:"switch_inline_query_current_chat,omitempty"` // optional
 	CallbackGame                 *CallbackGame `json:"callback_game,omitempty"`                    // optional
-	Pay                          bool          `json:"pay,omitempty"`			       // optional
+	Pay                          bool          `json:"pay,omitempty"`                              // optional
 }
 
 // CallbackQuery is data sent when a keyboard button with callback data
@@ -632,7 +658,7 @@ type InlineQueryResultGame struct {
 	Type          string                `json:"type"`
 	ID            string                `json:"id"`
 	GameShortName string                `json:"game_short_name"`
-	ReplyMarkup   *InlineKeyboardMarkup `json:"reply_markup"`
+	ReplyMarkup   *InlineKeyboardMarkup `json:"reply_markup,omitempty"`
 }
 
 // ChosenInlineResult is an inline query result chosen by a User
@@ -746,3 +772,13 @@ type PreCheckoutQuery struct {
 	ShippingOptionID string     `json:"shipping_option_id,omitempty"`
 	OrderInfo        *OrderInfo `json:"order_info,omitempty"`
 }
+
+// Error is an error containing extra information returned by the Telegram API.
+type Error struct {
+	Message string
+	ResponseParameters
+}
+
+func (e Error) Error() string {
+	return e.Message
+}

vendor/github.com/lrstanley/girc/builtin.go

@@ -113,7 +113,9 @@ func handlePING(c *Client, e Event) {
 }
 
 func handlePONG(c *Client, e Event) {
+	c.conn.mu.Lock()
 	c.conn.lastPong = time.Now()
+	c.conn.mu.Unlock()
 }
 
 // handleJOIN ensures that the state has updated users and channels.

vendor/github.com/lrstanley/girc/cap.go

@@ -5,14 +5,11 @@
 package girc
 
 import (
-	"bytes"
-	"encoding/base64"
-	"fmt"
-	"io"
-	"sort"
 	"strings"
 )
 
+// Something not in the list? Depending on the type of capability, you can
+// enable it using Config.SupportedCaps.
 var possibleCap = map[string][]string{
 	"account-notify":    nil,
 	"account-tag":       nil,
@@ -22,11 +19,25 @@ var possibleCap = map[string][]string{
 	"chghost":           nil,
 	"extended-join":     nil,
 	"invite-notify":     nil,
-	"message-tags":      nil,
 	"multi-prefix":      nil,
+	"server-time":       nil,
 	"userhost-in-names": nil,
+
+	"draft/message-tags-0.2": nil,
+	"draft/msgid":            nil,
+
+	// "echo-message" is supported, but it's not enabled by default. This is
+	// to prevent unwanted confusion and utilize less traffic if it's not needed.
+	// echo messages aren't sent to girc.PRIVMSG and girc.NOTICE handlers,
+	// rather they are only sent to girc.ALL_EVENTS handlers (this is to prevent
+	// each handler to have to check these types of things for each message).
+	// You can compare events using Event.Equals() to see if they are the same.
 }
 
+// https://ircv3.net/specs/extensions/server-time-3.2.html
+// <value> ::= YYYY-MM-DDThh:mm:ss.sssZ
+const capServerTimeFormat = "2006-01-02T15:04:05.999Z"
+
 func (c *Client) listCAP() {
 	if !c.Config.disableTracking {
 		c.write(&Event{Command: CAP, Params: []string{CAP_LS, "302"}})
@@ -74,8 +85,8 @@ func parseCap(raw string) map[string][]string {
 }
 
 // handleCAP attempts to find out what IRCv3 capabilities the server supports.
-// This will lock further registration until we have acknowledged the
-// capabilities.
+// This will lock further registration until we have acknowledged (or denied)
+// the capabilities.
 func handleCAP(c *Client, e Event) {
 	if len(e.Params) >= 2 && (e.Params[1] == CAP_NEW || e.Params[1] == CAP_DEL) {
 		c.listCAP()
@@ -172,133 +183,6 @@ func handleCAP(c *Client, e Event) {
 	}
 }
 
-// SASLMech is an representation of what a SASL mechanism should support.
-// See SASLExternal and SASLPlain for implementations of this.
-type SASLMech interface {
-	// Method returns the uppercase version of the SASL mechanism name.
-	Method() string
-	// Encode returns the response that the SASL mechanism wants to use. If
-	// the returned string is empty (e.g. the mechanism gives up), the handler
-	// will attempt to panic, as expectation is that if SASL authentication
-	// fails, the client will disconnect.
-	Encode(params []string) (output string)
-}
-
-// SASLExternal implements the "EXTERNAL" SASL type.
-type SASLExternal struct {
-	// Identity is an optional field which allows the client to specify
-	// pre-authentication identification. This means that EXTERNAL will
-	// supply this in the initial response. This usually isn't needed (e.g.
-	// CertFP).
-	Identity string `json:"identity"`
-}
-
-// Method identifies what type of SASL this implements.
-func (sasl *SASLExternal) Method() string {
-	return "EXTERNAL"
-}
-
-// Encode for external SALS authentication should really only return a "+",
-// unless the user has specified pre-authentication or identification data.
-// See https://tools.ietf.org/html/rfc4422#appendix-A for more info.
-func (sasl *SASLExternal) Encode(params []string) string {
-	if len(params) != 1 || params[0] != "+" {
-		return ""
-	}
-
-	if sasl.Identity != "" {
-		return sasl.Identity
-	}
-
-	return "+"
-}
-
-// SASLPlain contains the user and password needed for PLAIN SASL authentication.
-type SASLPlain struct {
-	User string `json:"user"` // User is the username for SASL.
-	Pass string `json:"pass"` // Pass is the password for SASL.
-}
-
-// Method identifies what type of SASL this implements.
-func (sasl *SASLPlain) Method() string {
-	return "PLAIN"
-}
-
-// Encode encodes the plain user+password into a SASL PLAIN implementation.
-// See https://tools.ietf.org/rfc/rfc4422.txt for more info.
-func (sasl *SASLPlain) Encode(params []string) string {
-	if len(params) != 1 || params[0] != "+" {
-		return ""
-	}
-
-	in := []byte(sasl.User)
-
-	in = append(in, 0x0)
-	in = append(in, []byte(sasl.User)...)
-	in = append(in, 0x0)
-	in = append(in, []byte(sasl.Pass)...)
-
-	return base64.StdEncoding.EncodeToString(in)
-}
-
-const saslChunkSize = 400
-
-func handleSASL(c *Client, e Event) {
-	if e.Command == RPL_SASLSUCCESS || e.Command == ERR_SASLALREADY {
-		// Let the server know that we're done.
-		c.write(&Event{Command: CAP, Params: []string{CAP_END}})
-		return
-	}
-
-	// Assume they want us to handle sending auth.
-	auth := c.Config.SASL.Encode(e.Params)
-
-	if auth == "" {
-		// Assume the SASL authentication method doesn't want to respond for
-		// some reason. The SASL spec and IRCv3 spec do not define a clear
-		// way to abort a SASL exchange, other than to disconnect, or proceed
-		// with CAP END.
-		c.rx <- &Event{Command: ERROR, Trailing: fmt.Sprintf(
-			"closing connection: invalid %s SASL configuration provided: %s",
-			c.Config.SASL.Method(), e.Trailing,
-		)}
-		return
-	}
-
-	// Send in "saslChunkSize"-length byte chunks. If the last chuck is
-	// exactly "saslChunkSize" bytes, send a "AUTHENTICATE +" 0-byte
-	// acknowledgement response to let the server know that we're done.
-	for {
-		if len(auth) > saslChunkSize {
-			c.write(&Event{Command: AUTHENTICATE, Params: []string{auth[0 : saslChunkSize-1]}, Sensitive: true})
-			auth = auth[saslChunkSize:]
-			continue
-		}
-
-		if len(auth) <= saslChunkSize {
-			c.write(&Event{Command: AUTHENTICATE, Params: []string{auth}, Sensitive: true})
-
-			if len(auth) == 400 {
-				c.write(&Event{Command: AUTHENTICATE, Params: []string{"+"}})
-			}
-			break
-		}
-	}
-	return
-}
-
-func handleSASLError(c *Client, e Event) {
-	if c.Config.SASL == nil {
-		c.write(&Event{Command: CAP, Params: []string{CAP_END}})
-		return
-	}
-
-	// Authentication failed. The SASL spec and IRCv3 spec do not define a
-	// clear way to abort a SASL exchange, other than to disconnect, or
-	// proceed with CAP END.
-	c.rx <- &Event{Command: ERROR, Trailing: "closing connection: " + e.Trailing}
-}
-
 // handleCHGHOST handles incoming IRCv3 hostname change events. CHGHOST is
 // what occurs (when enabled) when a servers services change the hostname of
 // a user. Traditionally, this was simply resolved with a quick QUIT and JOIN,
@@ -352,288 +236,3 @@ func handleACCOUNT(c *Client, e Event) {
 	c.state.Unlock()
 	c.state.notify(c, UPDATE_STATE)
 }
-
-// handleTags handles any messages that have tags that will affect state. (e.g.
-// 'account' tags.)
-func handleTags(c *Client, e Event) {
-	if len(e.Tags) == 0 {
-		return
-	}
-
-	account, ok := e.Tags.Get("account")
-	if !ok {
-		return
-	}
-
-	c.state.Lock()
-	user := c.state.lookupUser(e.Source.Name)
-	if user != nil {
-		user.Extras.Account = account
-	}
-	c.state.Unlock()
-	c.state.notify(c, UPDATE_STATE)
-}
-
-const (
-	prefixTag      byte = '@'
-	prefixTagValue byte = '='
-	prefixUserTag  byte = '+'
-	tagSeparator   byte = ';'
-	maxTagLength   int  = 511 // 510 + @ and " " (space), though space usually not included.
-)
-
-// Tags represents the key-value pairs in IRCv3 message tags. The map contains
-// the encoded message-tag values. If the tag is present, it may still be
-// empty. See Tags.Get() and Tags.Set() for use with getting/setting
-// information within the tags.
-//
-// Note that retrieving and setting tags are not concurrent safe. If this is
-// necessary, you will need to implement it yourself.
-type Tags map[string]string
-
-// ParseTags parses out the key-value map of tags. raw should only be the tag
-// data, not a full message. For example:
-//   @aaa=bbb;ccc;example.com/ddd=eee
-// NOT:
-//   @aaa=bbb;ccc;example.com/ddd=eee :nick!ident@host.com PRIVMSG me :Hello
-func ParseTags(raw string) (t Tags) {
-	t = make(Tags)
-
-	if len(raw) > 0 && raw[0] == prefixTag {
-		raw = raw[1:]
-	}
-
-	parts := strings.Split(raw, string(tagSeparator))
-	var hasValue int
-
-	for i := 0; i < len(parts); i++ {
-		hasValue = strings.IndexByte(parts[i], prefixTagValue)
-
-		// The tag doesn't contain a value or has a splitter with no value.
-		if hasValue < 1 || len(parts[i]) < hasValue+1 {
-			if !validTag(parts[i]) {
-				continue
-			}
-
-			t[parts[i]] = ""
-			continue
-		}
-
-		// Check if tag key or decoded value are invalid.
-		if !validTag(parts[i][:hasValue]) || !validTagValue(tagDecoder.Replace(parts[i][hasValue+1:])) {
-			continue
-		}
-
-		t[parts[i][:hasValue]] = parts[i][hasValue+1:]
-	}
-
-	return t
-}
-
-// Len determines the length of the bytes representation of this tag map. This
-// does not include the trailing space required when creating an event, but
-// does include the tag prefix ("@").
-func (t Tags) Len() (length int) {
-	if t == nil {
-		return 0
-	}
-
-	return len(t.Bytes())
-}
-
-// Count finds how many total tags that there are.
-func (t Tags) Count() int {
-	if t == nil {
-		return 0
-	}
-
-	return len(t)
-}
-
-// Bytes returns a []byte representation of this tag map, including the tag
-// prefix ("@"). Note that this will return the tags sorted, regardless of
-// the order of how they were originally parsed.
-func (t Tags) Bytes() []byte {
-	if t == nil {
-		return []byte{}
-	}
-
-	max := len(t)
-	if max == 0 {
-		return nil
-	}
-
-	buffer := new(bytes.Buffer)
-	buffer.WriteByte(prefixTag)
-
-	var current int
-
-	// Sort the writing of tags so we can at least guarantee that they will
-	// be in order, and testable.
-	var names []string
-	for tagName := range t {
-		names = append(names, tagName)
-	}
-	sort.Strings(names)
-
-	for i := 0; i < len(names); i++ {
-		// Trim at max allowed chars.
-		if (buffer.Len() + len(names[i]) + len(t[names[i]]) + 2) > maxTagLength {
-			return buffer.Bytes()
-		}
-
-		buffer.WriteString(names[i])
-
-		// Write the value as necessary.
-		if len(t[names[i]]) > 0 {
-			buffer.WriteByte(prefixTagValue)
-			buffer.WriteString(t[names[i]])
-		}
-
-		// add the separator ";" between tags.
-		if current < max-1 {
-			buffer.WriteByte(tagSeparator)
-		}
-
-		current++
-	}
-
-	return buffer.Bytes()
-}
-
-// String returns a string representation of this tag map.
-func (t Tags) String() string {
-	if t == nil {
-		return ""
-	}
-
-	return string(t.Bytes())
-}
-
-// writeTo writes the necessary tag bytes to an io.Writer, including a trailing
-// space-separator.
-func (t Tags) writeTo(w io.Writer) (n int, err error) {
-	b := t.Bytes()
-	if len(b) == 0 {
-		return n, err
-	}
-
-	n, err = w.Write(b)
-	if err != nil {
-		return n, err
-	}
-
-	var j int
-	j, err = w.Write([]byte{eventSpace})
-	n += j
-
-	return n, err
-}
-
-// tagDecode are encoded -> decoded pairs for replacement to decode.
-var tagDecode = []string{
-	"\\:", ";",
-	"\\s", " ",
-	"\\\\", "\\",
-	"\\r", "\r",
-	"\\n", "\n",
-}
-var tagDecoder = strings.NewReplacer(tagDecode...)
-
-// tagEncode are decoded -> encoded pairs for replacement to decode.
-var tagEncode = []string{
-	";", "\\:",
-	" ", "\\s",
-	"\\", "\\\\",
-	"\r", "\\r",
-	"\n", "\\n",
-}
-var tagEncoder = strings.NewReplacer(tagEncode...)
-
-// Get returns the unescaped value of given tag key. Note that this is not
-// concurrent safe.
-func (t Tags) Get(key string) (tag string, success bool) {
-	if t == nil {
-		return "", false
-	}
-
-	if _, ok := t[key]; ok {
-		tag = tagDecoder.Replace(t[key])
-		success = true
-	}
-
-	return tag, success
-}
-
-// Set escapes given value and saves it as the value for given key. Note that
-// this is not concurrent safe.
-func (t Tags) Set(key, value string) error {
-	if t == nil {
-		t = make(Tags)
-	}
-
-	if !validTag(key) {
-		return fmt.Errorf("tag key %q is invalid", key)
-	}
-
-	value = tagEncoder.Replace(value)
-
-	if len(value) > 0 && !validTagValue(value) {
-		return fmt.Errorf("tag value %q of key %q is invalid", value, key)
-	}
-
-	// Check to make sure it's not too long here.
-	if (t.Len() + len(key) + len(value) + 2) > maxTagLength {
-		return fmt.Errorf("unable to set tag %q [value %q]: tags too long for message", key, value)
-	}
-
-	t[key] = value
-
-	return nil
-}
-
-// Remove deletes the tag frwom the tag map.
-func (t Tags) Remove(key string) (success bool) {
-	if t == nil {
-		return false
-	}
-
-	if _, success = t[key]; success {
-		delete(t, key)
-	}
-
-	return success
-}
-
-// validTag validates an IRC tag.
-func validTag(name string) bool {
-	if len(name) < 1 {
-		return false
-	}
-
-	// Allow user tags to be passed to validTag.
-	if len(name) >= 2 && name[0] == prefixUserTag {
-		name = name[1:]
-	}
-
-	for i := 0; i < len(name); i++ {
-		// A-Z, a-z, 0-9, -/._
-		if (name[i] < 'A' || name[i] > 'Z') && (name[i] < 'a' || name[i] > 'z') && (name[i] < '-' || name[i] > '9') && name[i] != '_' {
-			return false
-		}
-	}
-
-	return true
-}
-
-// validTagValue valids a decoded IRC tag value. If the value is not decoded
-// with tagDecoder first, it may be seen as invalid.
-func validTagValue(value string) bool {
-	for i := 0; i < len(value); i++ {
-		// Don't allow any invisible chars within the tag, or semicolons.
-		if value[i] < '!' || value[i] > '~' || value[i] == ';' {
-			return false
-		}
-	}
-	return true
-}

vendor/github.com/lrstanley/girc/cap_sasl.go

@@ -0,0 +1,137 @@
+// Copyright (c) Liam Stanley <me@liamstanley.io>. All rights reserved. Use
+// of this source code is governed by the MIT license that can be found in
+// the LICENSE file.
+
+package girc
+
+import (
+	"encoding/base64"
+	"fmt"
+)
+
+// SASLMech is an representation of what a SASL mechanism should support.
+// See SASLExternal and SASLPlain for implementations of this.
+type SASLMech interface {
+	// Method returns the uppercase version of the SASL mechanism name.
+	Method() string
+	// Encode returns the response that the SASL mechanism wants to use. If
+	// the returned string is empty (e.g. the mechanism gives up), the handler
+	// will attempt to panic, as expectation is that if SASL authentication
+	// fails, the client will disconnect.
+	Encode(params []string) (output string)
+}
+
+// SASLExternal implements the "EXTERNAL" SASL type.
+type SASLExternal struct {
+	// Identity is an optional field which allows the client to specify
+	// pre-authentication identification. This means that EXTERNAL will
+	// supply this in the initial response. This usually isn't needed (e.g.
+	// CertFP).
+	Identity string `json:"identity"`
+}
+
+// Method identifies what type of SASL this implements.
+func (sasl *SASLExternal) Method() string {
+	return "EXTERNAL"
+}
+
+// Encode for external SALS authentication should really only return a "+",
+// unless the user has specified pre-authentication or identification data.
+// See https://tools.ietf.org/html/rfc4422#appendix-A for more info.
+func (sasl *SASLExternal) Encode(params []string) string {
+	if len(params) != 1 || params[0] != "+" {
+		return ""
+	}
+
+	if sasl.Identity != "" {
+		return sasl.Identity
+	}
+
+	return "+"
+}
+
+// SASLPlain contains the user and password needed for PLAIN SASL authentication.
+type SASLPlain struct {
+	User string `json:"user"` // User is the username for SASL.
+	Pass string `json:"pass"` // Pass is the password for SASL.
+}
+
+// Method identifies what type of SASL this implements.
+func (sasl *SASLPlain) Method() string {
+	return "PLAIN"
+}
+
+// Encode encodes the plain user+password into a SASL PLAIN implementation.
+// See https://tools.ietf.org/rfc/rfc4422.txt for more info.
+func (sasl *SASLPlain) Encode(params []string) string {
+	if len(params) != 1 || params[0] != "+" {
+		return ""
+	}
+
+	in := []byte(sasl.User)
+
+	in = append(in, 0x0)
+	in = append(in, []byte(sasl.User)...)
+	in = append(in, 0x0)
+	in = append(in, []byte(sasl.Pass)...)
+
+	return base64.StdEncoding.EncodeToString(in)
+}
+
+const saslChunkSize = 400
+
+func handleSASL(c *Client, e Event) {
+	if e.Command == RPL_SASLSUCCESS || e.Command == ERR_SASLALREADY {
+		// Let the server know that we're done.
+		c.write(&Event{Command: CAP, Params: []string{CAP_END}})
+		return
+	}
+
+	// Assume they want us to handle sending auth.
+	auth := c.Config.SASL.Encode(e.Params)
+
+	if auth == "" {
+		// Assume the SASL authentication method doesn't want to respond for
+		// some reason. The SASL spec and IRCv3 spec do not define a clear
+		// way to abort a SASL exchange, other than to disconnect, or proceed
+		// with CAP END.
+		c.rx <- &Event{Command: ERROR, Trailing: fmt.Sprintf(
+			"closing connection: SASL %s failed: %s",
+			c.Config.SASL.Method(), e.Trailing,
+		)}
+		return
+	}
+
+	// Send in "saslChunkSize"-length byte chunks. If the last chuck is
+	// exactly "saslChunkSize" bytes, send a "AUTHENTICATE +" 0-byte
+	// acknowledgement response to let the server know that we're done.
+	for {
+		if len(auth) > saslChunkSize {
+			c.write(&Event{Command: AUTHENTICATE, Params: []string{auth[0 : saslChunkSize-1]}, Sensitive: true})
+			auth = auth[saslChunkSize:]
+			continue
+		}
+
+		if len(auth) <= saslChunkSize {
+			c.write(&Event{Command: AUTHENTICATE, Params: []string{auth}, Sensitive: true})
+
+			if len(auth) == 400 {
+				c.write(&Event{Command: AUTHENTICATE, Params: []string{"+"}})
+			}
+			break
+		}
+	}
+	return
+}
+
+func handleSASLError(c *Client, e Event) {
+	if c.Config.SASL == nil {
+		c.write(&Event{Command: CAP, Params: []string{CAP_END}})
+		return
+	}
+
+	// Authentication failed. The SASL spec and IRCv3 spec do not define a
+	// clear way to abort a SASL exchange, other than to disconnect, or
+	// proceed with CAP END.
+	c.rx <- &Event{Command: ERROR, Trailing: "closing connection: " + e.Trailing}
+}

vendor/github.com/lrstanley/girc/cap_tags.go

@@ -0,0 +1,318 @@
+// Copyright (c) Liam Stanley <me@liamstanley.io>. All rights reserved. Use
+// of this source code is governed by the MIT license that can be found in
+// the LICENSE file.
+
+package girc
+
+import (
+	"bytes"
+	"fmt"
+	"io"
+	"sort"
+	"strings"
+)
+
+// handleTags handles any messages that have tags that will affect state. (e.g.
+// 'account' tags.)
+func handleTags(c *Client, e Event) {
+	if len(e.Tags) == 0 {
+		return
+	}
+
+	account, ok := e.Tags.Get("account")
+	if !ok {
+		return
+	}
+
+	c.state.Lock()
+	user := c.state.lookupUser(e.Source.Name)
+	if user != nil {
+		user.Extras.Account = account
+	}
+	c.state.Unlock()
+	c.state.notify(c, UPDATE_STATE)
+}
+
+const (
+	prefixTag      byte = '@'
+	prefixTagValue byte = '='
+	prefixUserTag  byte = '+'
+	tagSeparator   byte = ';'
+	maxTagLength   int  = 511 // 510 + @ and " " (space), though space usually not included.
+)
+
+// Tags represents the key-value pairs in IRCv3 message tags. The map contains
+// the encoded message-tag values. If the tag is present, it may still be
+// empty. See Tags.Get() and Tags.Set() for use with getting/setting
+// information within the tags.
+//
+// Note that retrieving and setting tags are not concurrent safe. If this is
+// necessary, you will need to implement it yourself.
+type Tags map[string]string
+
+// ParseTags parses out the key-value map of tags. raw should only be the tag
+// data, not a full message. For example:
+//   @aaa=bbb;ccc;example.com/ddd=eee
+// NOT:
+//   @aaa=bbb;ccc;example.com/ddd=eee :nick!ident@host.com PRIVMSG me :Hello
+func ParseTags(raw string) (t Tags) {
+	t = make(Tags)
+
+	if len(raw) > 0 && raw[0] == prefixTag {
+		raw = raw[1:]
+	}
+
+	parts := strings.Split(raw, string(tagSeparator))
+	var hasValue int
+
+	for i := 0; i < len(parts); i++ {
+		hasValue = strings.IndexByte(parts[i], prefixTagValue)
+
+		// The tag doesn't contain a value or has a splitter with no value.
+		if hasValue < 1 || len(parts[i]) < hasValue+1 {
+			if !validTag(parts[i]) {
+				continue
+			}
+
+			t[parts[i]] = ""
+			continue
+		}
+
+		// Check if tag key or decoded value are invalid.
+		if !validTag(parts[i][:hasValue]) || !validTagValue(tagDecoder.Replace(parts[i][hasValue+1:])) {
+			continue
+		}
+
+		t[parts[i][:hasValue]] = parts[i][hasValue+1:]
+	}
+
+	return t
+}
+
+// Len determines the length of the bytes representation of this tag map. This
+// does not include the trailing space required when creating an event, but
+// does include the tag prefix ("@").
+func (t Tags) Len() (length int) {
+	if t == nil {
+		return 0
+	}
+
+	return len(t.Bytes())
+}
+
+// Equals compares two Tags for equality. With the msgid IRCv3 spec +\
+// echo-message (amongst others), we may receive events that have msgid's,
+// whereas our local events will not have the msgid. As such, don't compare
+// all tags, only the necessary/important tags.
+func (t Tags) Equals(tt Tags) bool {
+	// The only tag which is important at this time.
+	taccount, _ := t.Get("account")
+	ttaccount, _ := tt.Get("account")
+	return taccount == ttaccount
+}
+
+// Keys returns a slice of (unsorted) tag keys.
+func (t Tags) Keys() (keys []string) {
+	keys = make([]string, 0, t.Count())
+	for key := range t {
+		keys = append(keys, key)
+	}
+	return keys
+}
+
+// Count finds how many total tags that there are.
+func (t Tags) Count() int {
+	if t == nil {
+		return 0
+	}
+
+	return len(t)
+}
+
+// Bytes returns a []byte representation of this tag map, including the tag
+// prefix ("@"). Note that this will return the tags sorted, regardless of
+// the order of how they were originally parsed.
+func (t Tags) Bytes() []byte {
+	if t == nil {
+		return []byte{}
+	}
+
+	max := len(t)
+	if max == 0 {
+		return nil
+	}
+
+	buffer := new(bytes.Buffer)
+	buffer.WriteByte(prefixTag)
+
+	var current int
+
+	// Sort the writing of tags so we can at least guarantee that they will
+	// be in order, and testable.
+	var names []string
+	for tagName := range t {
+		names = append(names, tagName)
+	}
+	sort.Strings(names)
+
+	for i := 0; i < len(names); i++ {
+		// Trim at max allowed chars.
+		if (buffer.Len() + len(names[i]) + len(t[names[i]]) + 2) > maxTagLength {
+			return buffer.Bytes()
+		}
+
+		buffer.WriteString(names[i])
+
+		// Write the value as necessary.
+		if len(t[names[i]]) > 0 {
+			buffer.WriteByte(prefixTagValue)
+			buffer.WriteString(t[names[i]])
+		}
+
+		// add the separator ";" between tags.
+		if current < max-1 {
+			buffer.WriteByte(tagSeparator)
+		}
+
+		current++
+	}
+
+	return buffer.Bytes()
+}
+
+// String returns a string representation of this tag map.
+func (t Tags) String() string {
+	if t == nil {
+		return ""
+	}
+
+	return string(t.Bytes())
+}
+
+// writeTo writes the necessary tag bytes to an io.Writer, including a trailing
+// space-separator.
+func (t Tags) writeTo(w io.Writer) (n int, err error) {
+	b := t.Bytes()
+	if len(b) == 0 {
+		return n, err
+	}
+
+	n, err = w.Write(b)
+	if err != nil {
+		return n, err
+	}
+
+	var j int
+	j, err = w.Write([]byte{eventSpace})
+	n += j
+
+	return n, err
+}
+
+// tagDecode are encoded -> decoded pairs for replacement to decode.
+var tagDecode = []string{
+	"\\:", ";",
+	"\\s", " ",
+	"\\\\", "\\",
+	"\\r", "\r",
+	"\\n", "\n",
+}
+var tagDecoder = strings.NewReplacer(tagDecode...)
+
+// tagEncode are decoded -> encoded pairs for replacement to decode.
+var tagEncode = []string{
+	";", "\\:",
+	" ", "\\s",
+	"\\", "\\\\",
+	"\r", "\\r",
+	"\n", "\\n",
+}
+var tagEncoder = strings.NewReplacer(tagEncode...)
+
+// Get returns the unescaped value of given tag key. Note that this is not
+// concurrent safe.
+func (t Tags) Get(key string) (tag string, success bool) {
+	if t == nil {
+		return "", false
+	}
+
+	if _, ok := t[key]; ok {
+		tag = tagDecoder.Replace(t[key])
+		success = true
+	}
+
+	return tag, success
+}
+
+// Set escapes given value and saves it as the value for given key. Note that
+// this is not concurrent safe.
+func (t Tags) Set(key, value string) error {
+	if t == nil {
+		t = make(Tags)
+	}
+
+	if !validTag(key) {
+		return fmt.Errorf("tag key %q is invalid", key)
+	}
+
+	value = tagEncoder.Replace(value)
+
+	if len(value) > 0 && !validTagValue(value) {
+		return fmt.Errorf("tag value %q of key %q is invalid", value, key)
+	}
+
+	// Check to make sure it's not too long here.
+	if (t.Len() + len(key) + len(value) + 2) > maxTagLength {
+		return fmt.Errorf("unable to set tag %q [value %q]: tags too long for message", key, value)
+	}
+
+	t[key] = value
+
+	return nil
+}
+
+// Remove deletes the tag frwom the tag map.
+func (t Tags) Remove(key string) (success bool) {
+	if t == nil {
+		return false
+	}
+
+	if _, success = t[key]; success {
+		delete(t, key)
+	}
+
+	return success
+}
+
+// validTag validates an IRC tag.
+func validTag(name string) bool {
+	if len(name) < 1 {
+		return false
+	}
+
+	// Allow user tags to be passed to validTag.
+	if len(name) >= 2 && name[0] == prefixUserTag {
+		name = name[1:]
+	}
+
+	for i := 0; i < len(name); i++ {
+		// A-Z, a-z, 0-9, -/._
+		if (name[i] < 'A' || name[i] > 'Z') && (name[i] < 'a' || name[i] > 'z') && (name[i] < '-' || name[i] > '9') && name[i] != '_' {
+			return false
+		}
+	}
+
+	return true
+}
+
+// validTagValue valids a decoded IRC tag value. If the value is not decoded
+// with tagDecoder first, it may be seen as invalid.
+func validTagValue(value string) bool {
+	for i := 0; i < len(value); i++ {
+		// Don't allow any invisible chars within the tag, or semicolons.
+		if value[i] < '!' || value[i] > '~' || value[i] == ';' {
+			return false
+		}
+	}
+	return true
+}

vendor/github.com/lrstanley/girc/client.go

@@ -14,6 +14,7 @@ import (
 	"log"
 	"runtime"
 	"sort"
+	"strings"
 	"sync"
 	"time"
 )
@@ -173,8 +174,8 @@ func (conf *Config) isValid() error {
 		conf.Port = 6667
 	}
 
-	if conf.Port < 21 || conf.Port > 65535 {
-		return &ErrInvalidConfig{Conf: *conf, err: errors.New("port outside valid range (21-65535)")}
+	if conf.Port < 1 || conf.Port > 65535 {
+		return &ErrInvalidConfig{Conf: *conf, err: errors.New("port outside valid range (1-65535)")}
 	}
 
 	if !IsValidNick(conf.Nick) {
@@ -432,7 +433,6 @@ func (c *Client) GetNick() string {
 	if c.state.nick == "" {
 		return c.Config.Nick
 	}
-
 	return c.state.nick
 }
 
@@ -448,140 +448,124 @@ func (c *Client) GetIdent() string {
 	if c.state.ident == "" {
 		return c.Config.User
 	}
-
 	return c.state.ident
 }
 
 // GetHost returns the current host of the active connection. Panics if
 // tracking is disabled. May be empty, as this is obtained from when we join
 // a channel, as there is no other more efficient method to return this info.
-func (c *Client) GetHost() string {
+func (c *Client) GetHost() (host string) {
 	c.panicIfNotTracking()
 
 	c.state.RLock()
-	defer c.state.RUnlock()
-
-	return c.state.host
+	host = c.state.host
+	c.state.RUnlock()
+	return host
 }
 
-// ChannelList returns the active list of channel names that the client is in.
-// Panics if tracking is disabled.
+// ChannelList returns the (sorted) active list of channel names that the client
+// is in. Panics if tracking is disabled.
 func (c *Client) ChannelList() []string {
 	c.panicIfNotTracking()
 
 	c.state.RLock()
-	channels := make([]string, len(c.state.channels))
-	var i int
+	channels := make([]string, 0, len(c.state.channels))
 	for channel := range c.state.channels {
-		channels[i] = c.state.channels[channel].Name
-		i++
+		channels = append(channels, c.state.channels[channel].Name)
 	}
 	c.state.RUnlock()
 	sort.Strings(channels)
-
 	return channels
 }
 
-// Channels returns the active channels that the client is in. Panics if
-// tracking is disabled.
+// Channels returns the (sorted) active channels that the client is in. Panics
+// if tracking is disabled.
 func (c *Client) Channels() []*Channel {
 	c.panicIfNotTracking()
 
 	c.state.RLock()
-	channels := make([]*Channel, len(c.state.channels))
-	var i int
+	channels := make([]*Channel, 0, len(c.state.channels))
 	for channel := range c.state.channels {
-		channels[i] = c.state.channels[channel].Copy()
-		i++
+		channels = append(channels, c.state.channels[channel].Copy())
 	}
 	c.state.RUnlock()
 
+	sort.Slice(channels, func(i, j int) bool {
+		return channels[i].Name < channels[j].Name
+	})
 	return channels
 }
 
-// UserList returns the active list of nicknames that the client is tracking
-// across all networks. Panics if tracking is disabled.
+// UserList returns the (sorted) active list of nicknames that the client is
+// tracking across all channels. Panics if tracking is disabled.
 func (c *Client) UserList() []string {
 	c.panicIfNotTracking()
 
 	c.state.RLock()
-	users := make([]string, len(c.state.users))
-	var i int
+	users := make([]string, 0, len(c.state.users))
 	for user := range c.state.users {
-		users[i] = c.state.users[user].Nick
-		i++
+		users = append(users, c.state.users[user].Nick)
 	}
 	c.state.RUnlock()
 	sort.Strings(users)
-
 	return users
 }
 
-// Users returns the active users that the client is tracking across all
-// networks. Panics if tracking is disabled.
+// Users returns the (sorted) active users that the client is tracking across
+// all channels. Panics if tracking is disabled.
 func (c *Client) Users() []*User {
 	c.panicIfNotTracking()
 
 	c.state.RLock()
-	users := make([]*User, len(c.state.users))
-	var i int
+	users := make([]*User, 0, len(c.state.users))
 	for user := range c.state.users {
-		users[i] = c.state.users[user].Copy()
-		i++
+		users = append(users, c.state.users[user].Copy())
 	}
 	c.state.RUnlock()
 
+	sort.Slice(users, func(i, j int) bool {
+		return users[i].Nick < users[j].Nick
+	})
 	return users
 }
 
 // LookupChannel looks up a given channel in state. If the channel doesn't
 // exist, nil is returned. Panics if tracking is disabled.
-func (c *Client) LookupChannel(name string) *Channel {
+func (c *Client) LookupChannel(name string) (channel *Channel) {
 	c.panicIfNotTracking()
 	if name == "" {
 		return nil
 	}
 
 	c.state.RLock()
-	defer c.state.RUnlock()
-
-	channel := c.state.lookupChannel(name)
-	if channel == nil {
-		return nil
-	}
-
-	return channel.Copy()
+	channel = c.state.lookupChannel(name).Copy()
+	c.state.RUnlock()
+	return channel
 }
 
 // LookupUser looks up a given user in state. If the user doesn't exist, nil
 // is returned. Panics if tracking is disabled.
-func (c *Client) LookupUser(nick string) *User {
+func (c *Client) LookupUser(nick string) (user *User) {
 	c.panicIfNotTracking()
 	if nick == "" {
 		return nil
 	}
 
 	c.state.RLock()
-	defer c.state.RUnlock()
-
-	user := c.state.lookupUser(nick)
-	if user == nil {
-		return nil
-	}
-
-	return user.Copy()
+	user = c.state.lookupUser(nick).Copy()
+	c.state.RUnlock()
+	return user
 }
 
 // IsInChannel returns true if the client is in channel. Panics if tracking
 // is disabled.
-func (c *Client) IsInChannel(channel string) bool {
+func (c *Client) IsInChannel(channel string) (in bool) {
 	c.panicIfNotTracking()
 
 	c.state.RLock()
-	_, inChannel := c.state.channels[ToRFC1459(channel)]
+	_, in = c.state.channels[ToRFC1459(channel)]
 	c.state.RUnlock()
-
-	return inChannel
+	return in
 }
 
 // GetServerOption retrieves a server capability setting that was retrieved
@@ -596,7 +580,6 @@ func (c *Client) GetServerOption(key string) (result string, ok bool) {
 	c.state.RLock()
 	result, ok = c.state.serverOptions[key]
 	c.state.RUnlock()
-
 	return result, ok
 }
 
@@ -607,7 +590,6 @@ func (c *Client) NetworkName() (name string) {
 	c.panicIfNotTracking()
 
 	name, _ = c.GetServerOption("NETWORK")
-
 	return name
 }
 
@@ -615,33 +597,31 @@ func (c *Client) NetworkName() (name string) {
 // supplied this information during connection. May be empty if the server
 // does not support RPL_MYINFO. Will panic if used when tracking has been
 // disabled.
-func (c *Client) ServerVersion() string {
+func (c *Client) ServerVersion() (version string) {
 	c.panicIfNotTracking()
 
-	version, _ := c.GetServerOption("VERSION")
-
+	version, _ = c.GetServerOption("VERSION")
 	return version
 }
 
 // ServerMOTD returns the servers message of the day, if the server has sent
 // it upon connect. Will panic if used when tracking has been disabled.
-func (c *Client) ServerMOTD() string {
+func (c *Client) ServerMOTD() (motd string) {
 	c.panicIfNotTracking()
 
 	c.state.RLock()
-	motd := c.state.motd
+	motd = c.state.motd
 	c.state.RUnlock()
-
 	return motd
 }
 
 // Latency is the latency between the server and the client. This is measured
 // by determining the difference in time between when we ping the server, and
 // when we receive a pong.
-func (c *Client) Latency() time.Duration {
+func (c *Client) Latency() (delta time.Duration) {
 	c.mu.RLock()
 	c.conn.mu.RLock()
-	delta := c.conn.lastPong.Sub(c.conn.lastPing)
+	delta = c.conn.lastPong.Sub(c.conn.lastPing)
 	c.conn.mu.RUnlock()
 	c.mu.RUnlock()
 
@@ -652,6 +632,30 @@ func (c *Client) Latency() time.Duration {
 	return delta
 }
 
+// HasCapability checks if the client connection has the given capability. If
+// you want the full list of capabilities, listen for the girc.CAP_ACK event.
+// Will panic if used when tracking has been disabled.
+func (c *Client) HasCapability(name string) (has bool) {
+	c.panicIfNotTracking()
+
+	if !c.IsConnected() {
+		return false
+	}
+
+	name = strings.ToLower(name)
+
+	c.state.RLock()
+	for i := 0; i < len(c.state.enabledCap); i++ {
+		if strings.ToLower(c.state.enabledCap[i]) == name {
+			has = true
+			break
+		}
+	}
+	c.state.RUnlock()
+
+	return has
+}
+
 // panicIfNotTracking will throw a panic when it's called, and tracking is
 // disabled. Adds useful info like what function specifically, and where it
 // was called from.

vendor/github.com/lrstanley/girc/conn.go

@@ -371,6 +371,12 @@ func (c *Client) readLoop(ctx context.Context, errs chan error, wg *sync.WaitGro
 				return
 			}
 
+			// Check if it's an echo-message.
+			if !c.Config.disableTracking {
+				event.Echo = (event.Command == PRIVMSG || event.Command == NOTICE) &&
+					event.Source != nil && event.Source.Name == c.GetNick()
+			}
+
 			c.rx <- event
 		}
 	}
@@ -500,7 +506,7 @@ type ErrTimedOut struct {
 	Delay time.Duration
 }
 
-func (ErrTimedOut) Error() string { return "timed out during ping to server" }
+func (ErrTimedOut) Error() string { return "timed out waiting for a requested PING response" }
 
 func (c *Client) pingLoop(ctx context.Context, errs chan error, wg *sync.WaitGroup) {
 	// Don't run the pingLoop if they want to disable it.

vendor/github.com/lrstanley/girc/event.go

@@ -8,6 +8,7 @@ import (
 	"bytes"
 	"fmt"
 	"strings"
+	"time"
 )
 
 const (
@@ -33,18 +34,35 @@ func cutCRFunc(r rune) bool {
 //                   CR or LF>
 //    <crlf>     :: CR LF
 type Event struct {
-	Source        *Source  `json:"source"`         // The source of the event.
-	Tags          Tags     `json:"tags"`           // IRCv3 style message tags. Only use if network supported.
-	Command       string   `json:"command"`        // the IRC command, e.g. JOIN, PRIVMSG, KILL.
-	Params        []string `json:"params"`         // parameters to the command. Commonly nickname, channel, etc.
-	Trailing      string   `json:"trailing"`       // any trailing data. e.g. with a PRIVMSG, this is the message text.
-	EmptyTrailing bool     `json:"empty_trailing"` // if true, trailing prefix (:) will be added even if Event.Trailing is empty.
-	Sensitive     bool     `json:"sensitive"`      // if the message is sensitive (e.g. and should not be logged).
+	// Source is the origin of the event.
+	Source *Source `json:"source"`
+	// Tags are the IRCv3 style message tags for the given event. Only use
+	// if network supported.
+	Tags Tags `json:"tags"`
+	// Timestamp is the time the event was received. This could optionally be
+	// used for client-stored sent messages too. If the server supports the
+	// "server-time" capability, this is synced to the UTC time that the server
+	// specifies.
+	Timestamp time.Time `json:"timestamp"`
+	// Command that represents the event, e.g. JOIN, PRIVMSG, KILL.
+	Command string `json:"command"`
+	// Params (parameters/args) to the command. Commonly nickname, channel, etc.
+	Params []string `json:"params"`
+	// Trailing text. e.g. with a PRIVMSG, this is the message text (part
+	// after the colon.)
+	Trailing string `json:"trailing"`
+	// EmptyTrailign, if true, the text prefix (:) will be added even if
+	// Event.Trailing is empty.
+	EmptyTrailing bool `json:"empty_trailing"`
+	// Sensitive should be true if the message is sensitive (e.g. and should
+	// not be logged/shown in debugging output).
+	Sensitive bool `json:"sensitive"`
+	// If the event is an echo-message response.
+	Echo bool `json:"echo"`
 }
 
-// ParseEvent takes a string and attempts to create a Event struct.
-//
-// Returns nil if the Event is invalid.
+// ParseEvent takes a string and attempts to create a Event struct. Returns
+// nil if the Event is invalid.
 func ParseEvent(raw string) (e *Event) {
 	// Ignore empty events.
 	if raw = strings.TrimFunc(raw, cutCRFunc); len(raw) < 2 {
@@ -52,7 +70,7 @@ func ParseEvent(raw string) (e *Event) {
 	}
 
 	var i, j int
-	e = &Event{}
+	e = &Event{Timestamp: time.Now()}
 
 	if raw[0] == prefixTag {
 		// Tags end with a space.
@@ -63,6 +81,13 @@ func ParseEvent(raw string) (e *Event) {
 		}
 
 		e.Tags = ParseTags(raw[1:i])
+		if rawServerTime, ok := e.Tags.Get("time"); ok {
+			// Attempt to parse server-time. If we can't parse it, we just
+			// fall back to the time we received the message (locally.)
+			if stime, err := time.Parse(capServerTimeFormat, rawServerTime); err == nil {
+				e.Timestamp = stime.Local()
+			}
+		}
 		raw = raw[i+1:]
 	}
 
@@ -151,10 +176,12 @@ func (e *Event) Copy() *Event {
 	}
 
 	newEvent := &Event{
+		Timestamp:     e.Timestamp,
 		Command:       e.Command,
 		Trailing:      e.Trailing,
 		EmptyTrailing: e.EmptyTrailing,
 		Sensitive:     e.Sensitive,
+		Echo:          e.Echo,
 	}
 
 	// Copy Source field, as it's a pointer and needs to be dereferenced.
@@ -179,6 +206,25 @@ func (e *Event) Copy() *Event {
 	return newEvent
 }
 
+// Equals compares two Events for equality.
+func (e *Event) Equals(ev *Event) bool {
+	if e.Command != ev.Command || e.Trailing != ev.Trailing || len(e.Params) != len(ev.Params) {
+		return false
+	}
+
+	for i := 0; i < len(e.Params); i++ {
+		if e.Params[i] != ev.Params[i] {
+			return false
+		}
+	}
+
+	if !e.Source.Equals(ev.Source) || !e.Tags.Equals(ev.Tags) {
+		return false
+	}
+
+	return true
+}
+
 // Len calculates the length of the string representation of event. Note that
 // this will return the true length (even if longer than what IRC supports),
 // which may be useful if you are trying to check and see if a message is
@@ -276,7 +322,7 @@ func (e *Event) String() string {
 // an event prettier, but also to filter out events that most don't visually
 // see in normal IRC clients. e.g. most clients don't show WHO queries.
 func (e *Event) Pretty() (out string, ok bool) {
-	if e.Sensitive {
+	if e.Sensitive || e.Echo {
 		return "", false
 	}
 
@@ -377,6 +423,10 @@ func (e *Event) Pretty() (out string, ok bool) {
 		return fmt.Sprintf("[*] topic for %s is: %s", e.Params[len(e.Params)-1], e.Trailing), true
 	}
 
+	if e.Command == CAP && len(e.Params) == 2 && len(e.Trailing) > 1 && e.Params[1] == CAP_ACK {
+		return "[*] enabling capabilities: " + e.Trailing, true
+	}
+
 	return "", false
 }
 
@@ -449,6 +499,20 @@ type Source struct {
 	Host string `json:"host"`
 }
 
+// Equals compares two Sources for equality.
+func (s *Source) Equals(ss *Source) bool {
+	if s == nil && ss == nil {
+		return true
+	}
+	if s != nil && ss == nil || s == nil && ss != nil {
+		return false
+	}
+	if s.Name != ss.Name || s.Ident != ss.Ident || s.Host != ss.Host {
+		return false
+	}
+	return true
+}
+
 // Copy returns a deep copy of Source.
 func (s *Source) Copy() *Source {
 	if s == nil {

vendor/github.com/lrstanley/girc/format.go

@@ -136,7 +136,7 @@ func TrimFmt(text string) string {
 	return text
 }
 
-// This is really the only fastest way of doing this (marginably better than
+// This is really the only fastest way of doing this (marginally better than
 // actually trying to parse it manually.)
 var reStripColor = regexp.MustCompile(`\x03([019]?[0-9](,[019]?[0-9])?)?`)
 
@@ -154,7 +154,7 @@ func StripRaw(text string) string {
 	return text
 }
 
-// IsValidChannel validates if channel is an RFC complaint channel or not.
+// IsValidChannel validates if channel is an RFC compliant channel or not.
 //
 // NOTE: If you are using this to validate a channel that contains a channel
 // ID, (!<channelid>NAME), this only supports the standard 5 character length.
@@ -271,7 +271,7 @@ func IsValidUser(name string) bool {
 	}
 
 	// Check to see if the first index is alphanumeric.
-	if (name[0] < 'A' || name[0] > 'J') && (name[0] < 'a' || name[0] > 'z') && (name[0] < '0' || name[0] > '9') {
+	if (name[0] < 'A' || name[0] > 'Z') && (name[0] < 'a' || name[0] > 'z') && (name[0] < '0' || name[0] > '9') {
 		return false
 	}
 

vendor/github.com/lrstanley/girc/handler.go

@@ -22,19 +22,28 @@ func (c *Client) RunHandlers(event *Event) {
 	}
 
 	// Log the event.
-	c.debug.Print("< " + StripRaw(event.String()))
+	prefix := "< "
+	if event.Echo {
+		prefix += "[echo-message] "
+	}
+	c.debug.Print(prefix + StripRaw(event.String()))
 	if c.Config.Out != nil {
 		if pretty, ok := event.Pretty(); ok {
 			fmt.Fprintln(c.Config.Out, StripRaw(pretty))
 		}
 	}
 
-	// Background handlers first.
+	// Background handlers first. If the event is an echo-message, then only
+	// send the echo version to ALL_EVENTS.
 	c.Handlers.exec(ALL_EVENTS, true, c, event.Copy())
-	c.Handlers.exec(event.Command, true, c, event.Copy())
+	if !event.Echo {
+		c.Handlers.exec(event.Command, true, c, event.Copy())
+	}
 
 	c.Handlers.exec(ALL_EVENTS, false, c, event.Copy())
-	c.Handlers.exec(event.Command, false, c, event.Copy())
+	if !event.Echo {
+		c.Handlers.exec(event.Command, false, c, event.Copy())
+	}
 
 	// Check if it's a CTCP.
 	if ctcp := decodeCTCP(event.Copy()); ctcp != nil {

vendor/github.com/lrstanley/girc/state.go

@@ -132,6 +132,10 @@ func (u User) Channels(c *Client) []*Channel {
 // Copy returns a deep copy of the user which can be modified without making
 // changes to the actual state.
 func (u *User) Copy() *User {
+	if u == nil {
+		return nil
+	}
+
 	nu := &User{}
 	*nu = *u
 
@@ -148,7 +152,7 @@ func (u *User) addChannel(name string) {
 	}
 
 	u.ChannelList = append(u.ChannelList, ToRFC1459(name))
-	sort.StringsAreSorted(u.ChannelList)
+	sort.Strings(u.ChannelList)
 
 	u.Perms.set(name, Perms{})
 }
@@ -321,6 +325,10 @@ func (ch *Channel) deleteUser(nick string) {
 
 // Copy returns a deep copy of a given channel.
 func (ch *Channel) Copy() *Channel {
+	if ch == nil {
+		return nil
+	}
+
 	nc := &Channel{}
 	*nc = *ch
 
@@ -483,6 +491,9 @@ func (s *state) renameUser(from, to string) {
 		for j := 0; j < len(s.channels[user.ChannelList[i]].UserList); j++ {
 			if s.channels[user.ChannelList[i]].UserList[j] == from {
 				s.channels[user.ChannelList[i]].UserList[j] = ToRFC1459(to)
+
+				sort.Strings(s.channels[user.ChannelList[i]].UserList)
+				break
 			}
 		}
 	}

vendor/github.com/matterbridge/go-xmpp/xmpp.go

@@ -573,6 +573,8 @@ type Chat struct {
 	Text      string
 	Subject   string
 	Thread    string
+	Ooburl    string
+	Oobdesc   string
 	Roster    Roster
 	Other     []string
 	OtherElem []XMLElement
@@ -654,16 +656,42 @@ func (c *Client) Recv() (stanza interface{}, err error) {
 func (c *Client) Send(chat Chat) (n int, err error) {
 	var subtext = ``
 	var thdtext = ``
+	var oobtext = ``
 	if chat.Subject != `` {
 		subtext = `<subject>` + xmlEscape(chat.Subject) + `</subject>`
 	}
 	if chat.Thread != `` {
 		thdtext = `<thread>` + xmlEscape(chat.Thread) + `</thread>`
 	}
-	return fmt.Fprintf(c.conn, "<message to='%s' type='%s' xml:lang='en'>"+subtext+"<body>%s</body>"+thdtext+"</message>",
+	if chat.Ooburl != `` {
+		oobtext = `<x xmlns="jabber:x:oob"><url>` + xmlEscape(chat.Ooburl) + `</url>`
+		if chat.Oobdesc != `` {
+			oobtext += `<desc>` + xmlEscape(chat.Oobdesc) + `</desc>`
+		}
+		oobtext += `</x>`
+	}
+	return fmt.Fprintf(c.conn, "<message to='%s' type='%s' xml:lang='en'>" + subtext + "<body>%s</body>" + oobtext + thdtext + "</message>",
 		xmlEscape(chat.Remote), xmlEscape(chat.Type), xmlEscape(chat.Text))
 }
 
+// SendOOB sends OOB data wrapped inside an XMPP message stanza, without actual body.
+func (c *Client) SendOOB(chat Chat) (n int, err error) {
+	var thdtext = ``
+	var oobtext = ``
+	if chat.Thread != `` {
+		thdtext = `<thread>` + xmlEscape(chat.Thread) + `</thread>`
+	}
+	if chat.Ooburl != `` {
+		oobtext = `<x xmlns="jabber:x:oob"><url>` + xmlEscape(chat.Ooburl) + `</url>`
+		if chat.Oobdesc != `` {
+			oobtext += `<desc>` + xmlEscape(chat.Oobdesc) + `</desc>`
+		}
+		oobtext += `</x>`
+	}
+	return fmt.Fprintf(c.conn, "<message to='%s' type='%s' xml:lang='en'>" + oobtext + thdtext + "</message>",
+		xmlEscape(chat.Remote), xmlEscape(chat.Type))
+}
+
 // SendOrg sends the original text without being wrapped in an XMPP message stanza.
 func (c *Client) SendOrg(org string) (n int, err error) {
 	return fmt.Fprint(c.conn, org)

vendor/github.com/matterbridge/gozulipbot/bot.go

@@ -0,0 +1,256 @@
+package gozulipbot
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"strings"
+	"time"
+)
+
+type Bot struct {
+	APIKey  string
+	APIURL  string
+	Email   string
+	Queues  []*Queue
+	Streams []string
+	Client  Doer
+	Backoff time.Duration
+	Retries int64
+}
+
+type Doer interface {
+	Do(*http.Request) (*http.Response, error)
+}
+
+// Init adds an http client to an existing bot struct.
+func (b *Bot) Init() *Bot {
+	b.Client = &http.Client{}
+	return b
+}
+
+// GetStreamList gets the raw http response when requesting all public streams.
+func (b *Bot) GetStreamList() (*http.Response, error) {
+	req, err := b.constructRequest("GET", "streams", "")
+	if err != nil {
+		return nil, err
+	}
+
+	return b.Client.Do(req)
+}
+
+type StreamJSON struct {
+	Msg     string `json:"msg"`
+	Streams []struct {
+		StreamID    int    `json:"stream_id"`
+		InviteOnly  bool   `json:"invite_only"`
+		Description string `json:"description"`
+		Name        string `json:"name"`
+	} `json:"streams"`
+	Result string `json:"result"`
+}
+
+// GetStreams returns a list of all public streams
+func (b *Bot) GetStreams() ([]string, error) {
+	resp, err := b.GetStreamList()
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	body, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	var sj StreamJSON
+	err = json.Unmarshal(body, &sj)
+	if err != nil {
+		return nil, err
+	}
+
+	var streams []string
+	for _, s := range sj.Streams {
+		streams = append(streams, s.Name)
+	}
+
+	return streams, nil
+}
+
+// GetStreams returns a list of all public streams
+func (b *Bot) GetRawStreams() (StreamJSON, error) {
+	var sj StreamJSON
+	resp, err := b.GetStreamList()
+	if err != nil {
+		return sj, err
+	}
+	defer resp.Body.Close()
+	body, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return sj, err
+	}
+
+	err = json.Unmarshal(body, &sj)
+	if err != nil {
+		return sj, err
+	}
+	return sj, nil
+}
+
+// Subscribe will set the bot to receive messages from the given streams.
+// If no streams are given, it will subscribe the bot to the streams in the bot struct.
+func (b *Bot) Subscribe(streams []string) (*http.Response, error) {
+	if streams == nil {
+		streams = b.Streams
+	}
+
+	var toSubStreams []map[string]string
+	for _, name := range streams {
+		toSubStreams = append(toSubStreams, map[string]string{"name": name})
+	}
+
+	bodyBts, err := json.Marshal(toSubStreams)
+	if err != nil {
+		return nil, err
+	}
+
+	body := "subscriptions=" + string(bodyBts)
+
+	req, err := b.constructRequest("POST", "users/me/subscriptions", body)
+	if err != nil {
+		return nil, err
+	}
+
+	return b.Client.Do(req)
+}
+
+// Unsubscribe will remove the bot from the given streams.
+// If no streams are given, nothing will happen and the function will error.
+func (b *Bot) Unsubscribe(streams []string) (*http.Response, error) {
+	if len(streams) == 0 {
+		return nil, fmt.Errorf("No streams were provided")
+	}
+
+	body := `delete=["` + strings.Join(streams, `","`) + `"]`
+
+	req, err := b.constructRequest("PATCH", "users/me/subscriptions", body)
+	if err != nil {
+		return nil, err
+	}
+
+	return b.Client.Do(req)
+}
+
+func (b *Bot) ListSubscriptions() (*http.Response, error) {
+	req, err := b.constructRequest("GET", "users/me/subscriptions", "")
+	if err != nil {
+		return nil, err
+	}
+
+	return b.Client.Do(req)
+}
+
+type EventType string
+
+const (
+	Messages      EventType = "messages"
+	Subscriptions EventType = "subscriptions"
+	RealmUser     EventType = "realm_user"
+	Pointer       EventType = "pointer"
+)
+
+type Narrow string
+
+const (
+	NarrowPrivate Narrow = `[["is", "private"]]`
+	NarrowAt      Narrow = `[["is", "mentioned"]]`
+)
+
+// RegisterEvents adds a queue to the bot. It includes the EventTypes and
+// Narrow given. If neither is given, it will default to all Messages.
+func (b *Bot) RegisterEvents(ets []EventType, n Narrow) (*Queue, error) {
+	resp, err := b.RawRegisterEvents(ets, n)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+
+	body, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	q := &Queue{Bot: b}
+	err = json.Unmarshal(body, q)
+	if err != nil {
+		return nil, err
+	}
+
+	if q.LastEventID < q.MaxMessageID {
+		q.LastEventID = q.MaxMessageID
+	}
+
+	b.Queues = append(b.Queues, q)
+
+	return q, nil
+}
+
+func (b *Bot) RegisterAll() (*Queue, error) {
+	return b.RegisterEvents(nil, "")
+}
+
+func (b *Bot) RegisterAt() (*Queue, error) {
+	return b.RegisterEvents(nil, NarrowAt)
+}
+
+func (b *Bot) RegisterPrivate() (*Queue, error) {
+	return b.RegisterEvents(nil, NarrowPrivate)
+}
+
+func (b *Bot) RegisterSubscriptions() (*Queue, error) {
+	events := []EventType{Subscriptions}
+	return b.RegisterEvents(events, "")
+}
+
+// RawRegisterEvents tells Zulip to include message events in the bots events queue.
+// Passing nil as the slice of EventType will default to receiving Messages
+func (b *Bot) RawRegisterEvents(ets []EventType, n Narrow) (*http.Response, error) {
+	// default to Messages if no EventTypes given
+	query := `event_types=["message"]`
+
+	if len(ets) != 0 {
+		query = `event_types=["`
+		for i, s := range ets {
+			query += fmt.Sprintf("%s", s)
+			if i != len(ets)-1 {
+				query += `", "`
+			}
+		}
+		query += `"]`
+	}
+
+	if n != "" {
+		query += fmt.Sprintf("&narrow=%s", n)
+	}
+	query += fmt.Sprintf("&all_public_streams=true")
+	req, err := b.constructRequest("POST", "register", query)
+	if err != nil {
+		return nil, err
+	}
+
+	return b.Client.Do(req)
+}
+
+// constructRequest makes a zulip request and ensures the proper headers are set.
+func (b *Bot) constructRequest(method, endpoint, body string) (*http.Request, error) {
+	url := b.APIURL + endpoint
+	req, err := http.NewRequest(method, url, strings.NewReader(body))
+	if err != nil {
+		return nil, err
+	}
+
+	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req.SetBasicAuth(b.Email, b.APIKey)
+
+	return req, nil
+}

vendor/github.com/matterbridge/gozulipbot/flag.go

@@ -0,0 +1,32 @@
+package gozulipbot
+
+import (
+	"flag"
+	"fmt"
+	"time"
+)
+
+func (b *Bot) GetConfigFromFlags() error {
+	var (
+		apiKey  = flag.String("apikey", "", "bot api key")
+		apiURL  = flag.String("apiurl", "", "url of zulip server")
+		email   = flag.String("email", "", "bot email address")
+		backoff = flag.Duration("backoff", 1*time.Second, "backoff base duration")
+	)
+	flag.Parse()
+
+	if *apiKey == "" {
+		return fmt.Errorf("--apikey is required")
+	}
+	if *apiURL == "" {
+		return fmt.Errorf("--apiurl is required")
+	}
+	if *email == "" {
+		return fmt.Errorf("--email is required")
+	}
+	b.APIKey = *apiKey
+	b.APIURL = *apiURL
+	b.Email = *email
+	b.Backoff = *backoff
+	return nil
+}

vendor/github.com/matterbridge/gozulipbot/message.go

@@ -0,0 +1,263 @@
+package gozulipbot
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"net/url"
+)
+
+// A Message is all of the necessary metadata to post on Zulip.
+// It can be either a public message, where Topic is set, or a private message,
+// where there is at least one element in Emails.
+//
+// If the length of Emails is not 0, functions will always assume it is a private message.
+type Message struct {
+	Stream  string
+	Topic   string
+	Emails  []string
+	Content string
+}
+
+type EventMessage struct {
+	AvatarURL        string           `json:"avatar_url"`
+	Client           string           `json:"client"`
+	Content          string           `json:"content"`
+	ContentType      string           `json:"content_type"`
+	DisplayRecipient DisplayRecipient `json:"display_recipient"`
+	GravatarHash     string           `json:"gravatar_hash"`
+	ID               int              `json:"id"`
+	RecipientID      int              `json:"recipient_id"`
+	SenderDomain     string           `json:"sender_domain"`
+	SenderEmail      string           `json:"sender_email"`
+	SenderFullName   string           `json:"sender_full_name"`
+	SenderID         int              `json:"sender_id"`
+	SenderShortName  string           `json:"sender_short_name"`
+	Subject          string           `json:"subject"`
+	SubjectLinks     []interface{}    `json:"subject_links"`
+	StreamID         int              `json:"stream_id"`
+	Timestamp        int              `json:"timestamp"`
+	Type             string           `json:"type"`
+	Queue            *Queue           `json:"-"`
+}
+
+type DisplayRecipient struct {
+	Users []User `json:"users,omitempty"`
+	Topic string `json:"topic,omitempty"`
+}
+
+type User struct {
+	Domain        string `json:"domain"`
+	Email         string `json:"email"`
+	FullName      string `json:"full_name"`
+	ID            int    `json:"id"`
+	IsMirrorDummy bool   `json:"is_mirror_dummy"`
+	ShortName     string `json:"short_name"`
+}
+
+func (d *DisplayRecipient) UnmarshalJSON(b []byte) (err error) {
+	topic, users := "", make([]User, 1)
+	if err = json.Unmarshal(b, &topic); err == nil {
+		d.Topic = topic
+		return
+	}
+	if err = json.Unmarshal(b, &users); err == nil {
+		d.Users = users
+		return
+	}
+	return
+}
+
+// Message posts a message to Zulip. If any emails have been set on the message,
+// the message will be re-routed to the PrivateMessage function.
+func (b *Bot) Message(m Message) (*http.Response, error) {
+	if m.Content == "" {
+		return nil, fmt.Errorf("content cannot be empty")
+	}
+
+	// if any emails are set, this is a private message
+	if len(m.Emails) != 0 {
+		return b.PrivateMessage(m)
+	}
+
+	// otherwise it's a stream message
+	if m.Stream == "" {
+		return nil, fmt.Errorf("stream cannot be empty")
+	}
+	if m.Topic == "" {
+		return nil, fmt.Errorf("topic cannot be empty")
+	}
+	req, err := b.constructMessageRequest(m)
+	if err != nil {
+		return nil, err
+	}
+	return b.Client.Do(req)
+}
+
+// PrivateMessage sends a message to the users in the message email slice.
+func (b *Bot) PrivateMessage(m Message) (*http.Response, error) {
+	if len(m.Emails) == 0 {
+		return nil, fmt.Errorf("there must be at least one recipient")
+	}
+	req, err := b.constructMessageRequest(m)
+	if err != nil {
+		return nil, err
+	}
+	return b.Client.Do(req)
+}
+
+// Respond sends a given message as a response to whatever context from which
+// an EventMessage was received.
+func (b *Bot) Respond(e EventMessage, response string) (*http.Response, error) {
+	if response == "" {
+		return nil, fmt.Errorf("Message response cannot be blank")
+	}
+	m := Message{
+		Stream:  e.DisplayRecipient.Topic,
+		Topic:   e.Subject,
+		Content: response,
+	}
+	if m.Topic != "" {
+		return b.Message(m)
+	}
+	// private message
+	if m.Stream == "" {
+		emails, err := b.privateResponseList(e)
+		if err != nil {
+			return nil, err
+		}
+		m.Emails = emails
+		return b.Message(m)
+	}
+	return nil, fmt.Errorf("EventMessage is not understood: %v\n", e)
+}
+
+// privateResponseList gets the list of other users in a private multiple
+// message conversation.
+func (b *Bot) privateResponseList(e EventMessage) ([]string, error) {
+	var out []string
+	for _, u := range e.DisplayRecipient.Users {
+		if u.Email != b.Email {
+			out = append(out, u.Email)
+		}
+	}
+	if len(out) == 0 {
+		return nil, fmt.Errorf("EventMessage had no Users within the DisplayRecipient")
+	}
+	return out, nil
+}
+
+// constructMessageRequest is a helper for simplifying sending a message.
+func (b *Bot) constructMessageRequest(m Message) (*http.Request, error) {
+	to := m.Stream
+	mtype := "stream"
+
+	le := len(m.Emails)
+	if le != 0 {
+		mtype = "private"
+	}
+	if le == 1 {
+		to = m.Emails[0]
+	}
+	if le > 1 {
+		to = ""
+		for i, e := range m.Emails {
+			to += e
+			if i != le-1 {
+				to += ","
+			}
+		}
+	}
+
+	values := url.Values{}
+	values.Set("type", mtype)
+	values.Set("to", to)
+	values.Set("content", m.Content)
+	if mtype == "stream" {
+		values.Set("subject", m.Topic)
+	}
+
+	return b.constructRequest("POST", "messages", values.Encode())
+}
+
+func (b *Bot) UpdateMessage(id string, content string) (*http.Response, error) {
+	//mid, _ := strconv.Atoi(id)
+	values := url.Values{}
+	values.Set("content", content)
+	req, err := b.constructRequest("PATCH", "messages/"+id, values.Encode())
+	if err != nil {
+		return nil, err
+	}
+	return b.Client.Do(req)
+}
+
+// React adds an emoji reaction to an EventMessage.
+func (b *Bot) React(e EventMessage, emoji string) (*http.Response, error) {
+	url := fmt.Sprintf("messages/%d/emoji_reactions/%s", e.ID, emoji)
+	req, err := b.constructRequest("PUT", url, "")
+	if err != nil {
+		return nil, err
+	}
+	return b.Client.Do(req)
+}
+
+// Unreact removes an emoji reaction from an EventMessage.
+func (b *Bot) Unreact(e EventMessage, emoji string) (*http.Response, error) {
+	url := fmt.Sprintf("messages/%d/emoji_reactions/%s", e.ID, emoji)
+	req, err := b.constructRequest("DELETE", url, "")
+	if err != nil {
+		return nil, err
+	}
+	return b.Client.Do(req)
+}
+
+type Emoji struct {
+	Author     string `json:"author"`
+	DisplayURL string `json:"display_url"`
+	SourceURL  string `json:"source_url"`
+}
+
+type EmojiResponse struct {
+	Emoji  map[string]*Emoji `json:"emoji"`
+	Msg    string            `json:"msg"`
+	Result string            `json:"result"`
+}
+
+// RealmEmoji gets the custom emoji information for the Zulip instance.
+func (b *Bot) RealmEmoji() (map[string]*Emoji, error) {
+	req, err := b.constructRequest("GET", "realm/emoji", "")
+	if err != nil {
+		return nil, err
+	}
+	resp, err := b.Client.Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+
+	body, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	var emjResp EmojiResponse
+	err = json.Unmarshal(body, &emjResp)
+	if err != nil {
+		return nil, err
+	}
+	return emjResp.Emoji, nil
+}
+
+// RealmEmojiSet makes a set of the names of the custom emoji in the Zulip instance.
+func (b *Bot) RealmEmojiSet() (map[string]struct{}, error) {
+	emj, err := b.RealmEmoji()
+	if err != nil {
+		return nil, nil
+	}
+	out := map[string]struct{}{}
+	for k, _ := range emj {
+		out[k] = struct{}{}
+	}
+	return out, nil
+}

vendor/github.com/matterbridge/gozulipbot/queue.go

@@ -0,0 +1,203 @@
+package gozulipbot
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"math"
+	"net/http"
+	"net/url"
+	"strconv"
+	"sync/atomic"
+	"time"
+)
+
+var (
+	HeartbeatError    = fmt.Errorf("EventMessage is a heartbeat")
+	UnauthorizedError = fmt.Errorf("Request is unauthorized")
+	BackoffError      = fmt.Errorf("Too many requests")
+	UnknownError      = fmt.Errorf("Error was unknown")
+)
+
+type Queue struct {
+	ID           string `json:"queue_id"`
+	LastEventID  int    `json:"last_event_id"`
+	MaxMessageID int    `json:"max_message_id"`
+	Bot          *Bot   `json:"-"`
+}
+
+func (q *Queue) EventsChan() (chan EventMessage, func()) {
+	end := false
+	endFunc := func() {
+		end = true
+	}
+
+	out := make(chan EventMessage)
+	go func() {
+		defer close(out)
+		for {
+			backoffTime := time.Now().Add(q.Bot.Backoff * time.Duration(math.Pow10(int(atomic.LoadInt64(&q.Bot.Retries)))))
+			minTime := time.Now().Add(q.Bot.Backoff)
+			if end {
+				return
+			}
+			ems, err := q.GetEvents()
+			switch {
+			case err == HeartbeatError:
+				time.Sleep(time.Until(minTime))
+				continue
+			case err == BackoffError:
+				time.Sleep(time.Until(backoffTime))
+				atomic.AddInt64(&q.Bot.Retries, 1)
+			case err == UnauthorizedError:
+				// TODO? have error channel when ending the continuously running process?
+				return
+			default:
+				atomic.StoreInt64(&q.Bot.Retries, 0)
+			}
+			if err != nil {
+				// TODO: handle unknown error
+				// For now, handle this like an UnauthorizedError and end the func.
+				return
+			}
+			for _, em := range ems {
+				out <- em
+			}
+			// Always make sure we wait the minimum time before asking again.
+			time.Sleep(time.Until(minTime))
+		}
+	}()
+
+	return out, endFunc
+}
+
+// EventsCallback will repeatedly call provided callback function with
+// the output of continual queue.GetEvents calls.
+// It returns a function which can be called to end the calls.
+//
+// It will end early if it receives an UnauthorizedError, or an unknown error.
+// Note, it will never return a HeartbeatError.
+func (q *Queue) EventsCallback(fn func(EventMessage, error)) func() {
+	end := false
+	endFunc := func() {
+		end = true
+	}
+	go func() {
+		for {
+			backoffTime := time.Now().Add(q.Bot.Backoff * time.Duration(math.Pow10(int(atomic.LoadInt64(&q.Bot.Retries)))))
+			minTime := time.Now().Add(q.Bot.Backoff)
+			if end {
+				return
+			}
+			ems, err := q.GetEvents()
+			switch {
+			case err == HeartbeatError:
+				time.Sleep(time.Until(minTime))
+				continue
+			case err == BackoffError:
+				time.Sleep(time.Until(backoffTime))
+				atomic.AddInt64(&q.Bot.Retries, 1)
+			case err == UnauthorizedError:
+				// TODO? have error channel when ending the continuously running process?
+				return
+			default:
+				atomic.StoreInt64(&q.Bot.Retries, 0)
+			}
+			if err != nil {
+				// TODO: handle unknown error
+				// For now, handle this like an UnauthorizedError and end the func.
+				return
+			}
+			for _, em := range ems {
+				fn(em, err)
+			}
+			// Always make sure we wait the minimum time before asking again.
+			time.Sleep(time.Until(minTime))
+		}
+	}()
+
+	return endFunc
+}
+
+// GetEvents is a blocking call that waits for and parses a list of EventMessages.
+// There will usually only be one EventMessage returned.
+// When a heartbeat is returned, GetEvents will return a HeartbeatError.
+// When an http status code above 400 is returned, one of a BackoffError,
+// UnauthorizedError, or UnknownError will be returned.
+func (q *Queue) GetEvents() ([]EventMessage, error) {
+	resp, err := q.RawGetEvents()
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+
+	switch {
+	case resp.StatusCode == 429:
+		return nil, BackoffError
+	case resp.StatusCode == 403:
+		return nil, UnauthorizedError
+	case resp.StatusCode >= 400:
+		return nil, UnknownError
+	}
+
+	body, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	msgs, err := q.ParseEventMessages(body)
+	if err != nil {
+		return nil, err
+	}
+
+	return msgs, nil
+}
+
+// RawGetEvents is a blocking call that receives a response containing a list
+// of events (a.k.a. received messages) since the last message id in the queue.
+func (q *Queue) RawGetEvents() (*http.Response, error) {
+	values := url.Values{}
+	values.Set("queue_id", q.ID)
+	values.Set("last_event_id", strconv.Itoa(q.LastEventID))
+
+	url := "events?" + values.Encode()
+
+	req, err := q.Bot.constructRequest("GET", url, "")
+	if err != nil {
+		return nil, err
+	}
+
+	return q.Bot.Client.Do(req)
+}
+
+func (q *Queue) ParseEventMessages(rawEventResponse []byte) ([]EventMessage, error) {
+	rawResponse := map[string]json.RawMessage{}
+	err := json.Unmarshal(rawEventResponse, &rawResponse)
+	if err != nil {
+		return nil, err
+	}
+
+	events := []map[string]json.RawMessage{}
+	err = json.Unmarshal(rawResponse["events"], &events)
+	if err != nil {
+		return nil, err
+	}
+
+	messages := []EventMessage{}
+	for _, event := range events {
+		// if the event is a heartbeat, return a special error
+		if string(event["type"]) == `"heartbeat"` {
+			return nil, HeartbeatError
+		}
+		var msg EventMessage
+		err = json.Unmarshal(event["message"], &msg)
+		// TODO? should this check be here
+		if err != nil {
+			return nil, err
+		}
+		msg.Queue = q
+		messages = append(messages, msg)
+	}
+
+	return messages, nil
+}

vendor/github.com/rs/xid/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2015 Olivier Poitrey <rs@dailymotion.com>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is furnished
+to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

vendor/github.com/rs/xid/hostid_darwin.go

@@ -0,0 +1,9 @@
+// +build darwin
+
+package xid
+
+import "golang.org/x/sys/unix"
+
+func readPlatformMachineID() (string, error) {
+	return unix.Sysctl("kern.uuid")
+}

vendor/github.com/rs/xid/hostid_fallback.go

@@ -0,0 +1,9 @@
+// +build !darwin,!linux,!freebsd,!windows
+
+package xid
+
+import "errors"
+
+func readPlatformMachineID() (string, error) {
+	return "", errors.New("not implemented")
+}

vendor/github.com/rs/xid/hostid_freebsd.go

@@ -0,0 +1,9 @@
+// +build freebsd
+
+package xid
+
+import "golang.org/x/sys/unix"
+
+func readPlatformMachineID() (string, error) {
+	return unix.Sysctl("kern.hostuuid")
+}

vendor/github.com/rs/xid/hostid_linux.go

@@ -0,0 +1,10 @@
+// +build linux
+
+package xid
+
+import "io/ioutil"
+
+func readPlatformMachineID() (string, error) {
+	b, err := ioutil.ReadFile("/sys/class/dmi/id/product_uuid")
+	return string(b), err
+}

vendor/github.com/rs/xid/hostid_windows.go

@@ -0,0 +1,39 @@
+// +build windows
+
+package xid
+
+import (
+	"fmt"
+	"unsafe"
+
+	"golang.org/x/sys/windows"
+)
+
+func readPlatformMachineID() (string, error) {
+	// source: https://github.com/shirou/gopsutil/blob/master/host/host_windows.go
+	var h windows.Handle
+	err := windows.RegOpenKeyEx(windows.HKEY_LOCAL_MACHINE, windows.StringToUTF16Ptr(`SOFTWARE\Microsoft\Cryptography`), 0, windows.KEY_READ|windows.KEY_WOW64_64KEY, &h)
+	if err != nil {
+		return "", err
+	}
+	defer windows.RegCloseKey(h)
+
+	const windowsRegBufLen = 74 // len(`{`) + len(`abcdefgh-1234-456789012-123345456671` * 2) + len(`}`) // 2 == bytes/UTF16
+	const uuidLen = 36
+
+	var regBuf [windowsRegBufLen]uint16
+	bufLen := uint32(windowsRegBufLen)
+	var valType uint32
+	err = windows.RegQueryValueEx(h, windows.StringToUTF16Ptr(`MachineGuid`), nil, &valType, (*byte)(unsafe.Pointer(&regBuf[0])), &bufLen)
+	if err != nil {
+		return "", err
+	}
+
+	hostID := windows.UTF16ToString(regBuf[:])
+	hostIDLen := len(hostID)
+	if hostIDLen != uuidLen {
+		return "", fmt.Errorf("HostID incorrect: %q\n", hostID)
+	}
+
+	return hostID, nil
+}

vendor/github.com/rs/xid/id.go

@@ -0,0 +1,361 @@
+// Package xid is a globally unique id generator suited for web scale
+//
+// Xid is using Mongo Object ID algorithm to generate globally unique ids:
+// https://docs.mongodb.org/manual/reference/object-id/
+//
+//   - 4-byte value representing the seconds since the Unix epoch,
+//   - 3-byte machine identifier,
+//   - 2-byte process id, and
+//   - 3-byte counter, starting with a random value.
+//
+// The binary representation of the id is compatible with Mongo 12 bytes Object IDs.
+// The string representation is using base32 hex (w/o padding) for better space efficiency
+// when stored in that form (20 bytes). The hex variant of base32 is used to retain the
+// sortable property of the id.
+//
+// Xid doesn't use base64 because case sensitivity and the 2 non alphanum chars may be an
+// issue when transported as a string between various systems. Base36 wasn't retained either
+// because 1/ it's not standard 2/ the resulting size is not predictable (not bit aligned)
+// and 3/ it would not remain sortable. To validate a base32 `xid`, expect a 20 chars long,
+// all lowercase sequence of `a` to `v` letters and `0` to `9` numbers (`[0-9a-v]{20}`).
+//
+// UUID is 16 bytes (128 bits), snowflake is 8 bytes (64 bits), xid stands in between
+// with 12 bytes with a more compact string representation ready for the web and no
+// required configuration or central generation server.
+//
+// Features:
+//
+//   - Size: 12 bytes (96 bits), smaller than UUID, larger than snowflake
+//   - Base32 hex encoded by default (16 bytes storage when transported as printable string)
+//   - Non configured, you don't need set a unique machine and/or data center id
+//   - K-ordered
+//   - Embedded time with 1 second precision
+//   - Unicity guaranteed for 16,777,216 (24 bits) unique ids per second and per host/process
+//
+// Best used with xlog's RequestIDHandler (https://godoc.org/github.com/rs/xlog#RequestIDHandler).
+//
+// References:
+//
+//   - http://www.slideshare.net/davegardnerisme/unique-id-generation-in-distributed-systems
+//   - https://en.wikipedia.org/wiki/Universally_unique_identifier
+//   - https://blog.twitter.com/2010/announcing-snowflake
+package xid
+
+import (
+	"crypto/md5"
+	"crypto/rand"
+	"database/sql/driver"
+	"encoding/binary"
+	"errors"
+	"fmt"
+	"hash/crc32"
+	"io/ioutil"
+	"os"
+	"sync/atomic"
+	"time"
+	"bytes"
+	"sort"
+)
+
+// Code inspired from mgo/bson ObjectId
+
+// ID represents a unique request id
+type ID [rawLen]byte
+
+const (
+	encodedLen = 20 // string encoded len
+	rawLen     = 12 // binary raw len
+
+	// encoding stores a custom version of the base32 encoding with lower case
+	// letters.
+	encoding = "0123456789abcdefghijklmnopqrstuv"
+)
+
+var (
+	// ErrInvalidID is returned when trying to unmarshal an invalid ID
+	ErrInvalidID = errors.New("xid: invalid ID")
+
+	// objectIDCounter is atomically incremented when generating a new ObjectId
+	// using NewObjectId() function. It's used as a counter part of an id.
+	// This id is initialized with a random value.
+	objectIDCounter = randInt()
+
+	// machineId stores machine id generated once and used in subsequent calls
+	// to NewObjectId function.
+	machineID = readMachineID()
+
+	// pid stores the current process id
+	pid = os.Getpid()
+
+	nilID ID
+
+	// dec is the decoding map for base32 encoding
+	dec [256]byte
+)
+
+func init() {
+	for i := 0; i < len(dec); i++ {
+		dec[i] = 0xFF
+	}
+	for i := 0; i < len(encoding); i++ {
+		dec[encoding[i]] = byte(i)
+	}
+
+	// If /proc/self/cpuset exists and is not /, we can assume that we are in a
+	// form of container and use the content of cpuset xor-ed with the PID in
+	// order get a reasonable machine global unique PID.
+	b, err := ioutil.ReadFile("/proc/self/cpuset")
+	if err == nil && len(b) > 1 {
+		pid ^= int(crc32.ChecksumIEEE(b))
+	}
+}
+
+// readMachineId generates machine id and puts it into the machineId global
+// variable. If this function fails to get the hostname, it will cause
+// a runtime error.
+func readMachineID() []byte {
+	id := make([]byte, 3)
+	hid, err := readPlatformMachineID()
+	if err != nil || len(hid) == 0 {
+		hid, err = os.Hostname()
+	}
+	if err == nil && len(hid) != 0 {
+		hw := md5.New()
+		hw.Write([]byte(hid))
+		copy(id, hw.Sum(nil))
+	} else {
+		// Fallback to rand number if machine id can't be gathered
+		if _, randErr := rand.Reader.Read(id); randErr != nil {
+			panic(fmt.Errorf("xid: cannot get hostname nor generate a random number: %v; %v", err, randErr))
+		}
+	}
+	return id
+}
+
+// randInt generates a random uint32
+func randInt() uint32 {
+	b := make([]byte, 3)
+	if _, err := rand.Reader.Read(b); err != nil {
+		panic(fmt.Errorf("xid: cannot generate random number: %v;", err))
+	}
+	return uint32(b[0])<<16 | uint32(b[1])<<8 | uint32(b[2])
+}
+
+// New generates a globally unique ID
+func New() ID {
+	var id ID
+	// Timestamp, 4 bytes, big endian
+	binary.BigEndian.PutUint32(id[:], uint32(time.Now().Unix()))
+	// Machine, first 3 bytes of md5(hostname)
+	id[4] = machineID[0]
+	id[5] = machineID[1]
+	id[6] = machineID[2]
+	// Pid, 2 bytes, specs don't specify endianness, but we use big endian.
+	id[7] = byte(pid >> 8)
+	id[8] = byte(pid)
+	// Increment, 3 bytes, big endian
+	i := atomic.AddUint32(&objectIDCounter, 1)
+	id[9] = byte(i >> 16)
+	id[10] = byte(i >> 8)
+	id[11] = byte(i)
+	return id
+}
+
+// FromString reads an ID from its string representation
+func FromString(id string) (ID, error) {
+	i := &ID{}
+	err := i.UnmarshalText([]byte(id))
+	return *i, err
+}
+
+// String returns a base32 hex lowercased with no padding representation of the id (char set is 0-9, a-v).
+func (id ID) String() string {
+	text := make([]byte, encodedLen)
+	encode(text, id[:])
+	return string(text)
+}
+
+// MarshalText implements encoding/text TextMarshaler interface
+func (id ID) MarshalText() ([]byte, error) {
+	text := make([]byte, encodedLen)
+	encode(text, id[:])
+	return text, nil
+}
+
+// MarshalJSON implements encoding/json Marshaler interface
+func (id ID) MarshalJSON() ([]byte, error) {
+	if id.IsNil() {
+		return []byte("null"), nil
+	}
+	text, err := id.MarshalText()
+	return []byte(`"` + string(text) + `"`), err
+}
+
+// encode by unrolling the stdlib base32 algorithm + removing all safe checks
+func encode(dst, id []byte) {
+	dst[0] = encoding[id[0]>>3]
+	dst[1] = encoding[(id[1]>>6)&0x1F|(id[0]<<2)&0x1F]
+	dst[2] = encoding[(id[1]>>1)&0x1F]
+	dst[3] = encoding[(id[2]>>4)&0x1F|(id[1]<<4)&0x1F]
+	dst[4] = encoding[id[3]>>7|(id[2]<<1)&0x1F]
+	dst[5] = encoding[(id[3]>>2)&0x1F]
+	dst[6] = encoding[id[4]>>5|(id[3]<<3)&0x1F]
+	dst[7] = encoding[id[4]&0x1F]
+	dst[8] = encoding[id[5]>>3]
+	dst[9] = encoding[(id[6]>>6)&0x1F|(id[5]<<2)&0x1F]
+	dst[10] = encoding[(id[6]>>1)&0x1F]
+	dst[11] = encoding[(id[7]>>4)&0x1F|(id[6]<<4)&0x1F]
+	dst[12] = encoding[id[8]>>7|(id[7]<<1)&0x1F]
+	dst[13] = encoding[(id[8]>>2)&0x1F]
+	dst[14] = encoding[(id[9]>>5)|(id[8]<<3)&0x1F]
+	dst[15] = encoding[id[9]&0x1F]
+	dst[16] = encoding[id[10]>>3]
+	dst[17] = encoding[(id[11]>>6)&0x1F|(id[10]<<2)&0x1F]
+	dst[18] = encoding[(id[11]>>1)&0x1F]
+	dst[19] = encoding[(id[11]<<4)&0x1F]
+}
+
+// UnmarshalText implements encoding/text TextUnmarshaler interface
+func (id *ID) UnmarshalText(text []byte) error {
+	if len(text) != encodedLen {
+		return ErrInvalidID
+	}
+	for _, c := range text {
+		if dec[c] == 0xFF {
+			return ErrInvalidID
+		}
+	}
+	decode(id, text)
+	return nil
+}
+
+// UnmarshalJSON implements encoding/json Unmarshaler interface
+func (id *ID) UnmarshalJSON(b []byte) error {
+	s := string(b)
+	if s == "null" {
+		*id = nilID
+		return nil
+	}
+	return id.UnmarshalText(b[1 : len(b)-1])
+}
+
+// decode by unrolling the stdlib base32 algorithm + removing all safe checks
+func decode(id *ID, src []byte) {
+	id[0] = dec[src[0]]<<3 | dec[src[1]]>>2
+	id[1] = dec[src[1]]<<6 | dec[src[2]]<<1 | dec[src[3]]>>4
+	id[2] = dec[src[3]]<<4 | dec[src[4]]>>1
+	id[3] = dec[src[4]]<<7 | dec[src[5]]<<2 | dec[src[6]]>>3
+	id[4] = dec[src[6]]<<5 | dec[src[7]]
+	id[5] = dec[src[8]]<<3 | dec[src[9]]>>2
+	id[6] = dec[src[9]]<<6 | dec[src[10]]<<1 | dec[src[11]]>>4
+	id[7] = dec[src[11]]<<4 | dec[src[12]]>>1
+	id[8] = dec[src[12]]<<7 | dec[src[13]]<<2 | dec[src[14]]>>3
+	id[9] = dec[src[14]]<<5 | dec[src[15]]
+	id[10] = dec[src[16]]<<3 | dec[src[17]]>>2
+	id[11] = dec[src[17]]<<6 | dec[src[18]]<<1 | dec[src[19]]>>4
+}
+
+// Time returns the timestamp part of the id.
+// It's a runtime error to call this method with an invalid id.
+func (id ID) Time() time.Time {
+	// First 4 bytes of ObjectId is 32-bit big-endian seconds from epoch.
+	secs := int64(binary.BigEndian.Uint32(id[0:4]))
+	return time.Unix(secs, 0)
+}
+
+// Machine returns the 3-byte machine id part of the id.
+// It's a runtime error to call this method with an invalid id.
+func (id ID) Machine() []byte {
+	return id[4:7]
+}
+
+// Pid returns the process id part of the id.
+// It's a runtime error to call this method with an invalid id.
+func (id ID) Pid() uint16 {
+	return binary.BigEndian.Uint16(id[7:9])
+}
+
+// Counter returns the incrementing value part of the id.
+// It's a runtime error to call this method with an invalid id.
+func (id ID) Counter() int32 {
+	b := id[9:12]
+	// Counter is stored as big-endian 3-byte value
+	return int32(uint32(b[0])<<16 | uint32(b[1])<<8 | uint32(b[2]))
+}
+
+// Value implements the driver.Valuer interface.
+func (id ID) Value() (driver.Value, error) {
+	if id.IsNil() {
+		return nil, nil
+	}
+	b, err := id.MarshalText()
+	return string(b), err
+}
+
+// Scan implements the sql.Scanner interface.
+func (id *ID) Scan(value interface{}) (err error) {
+	switch val := value.(type) {
+	case string:
+		return id.UnmarshalText([]byte(val))
+	case []byte:
+		return id.UnmarshalText(val)
+	case nil:
+		*id = nilID
+		return nil
+	default:
+		return fmt.Errorf("xid: scanning unsupported type: %T", value)
+	}
+}
+
+// IsNil Returns true if this is a "nil" ID
+func (id ID) IsNil() bool {
+	return id == nilID
+}
+
+// NilID returns a zero value for `xid.ID`.
+func NilID() ID {
+	return nilID
+}
+
+// Bytes returns the byte array representation of `ID`
+func (id ID) Bytes() []byte {
+	return id[:]
+}
+
+// FromBytes convert the byte array representation of `ID` back to `ID`
+func FromBytes(b []byte) (ID, error) {
+	var id ID
+	if len(b) != rawLen {
+		return id, ErrInvalidID
+	}
+	copy(id[:], b)
+	return id, nil
+}
+
+// Compare returns an integer comparing two IDs. It behaves just like `bytes.Compare`.
+// The result will be 0 if two IDs are identical, -1 if current id is less than the other one,
+// and 1 if current id is greater than the other.
+func (id ID) Compare(other ID) int {
+	return bytes.Compare(id[:], other[:])
+}
+
+
+type sorter []ID
+
+func (s sorter) Len() int {
+	return len(s)
+}
+
+func (s sorter) Less(i, j int) bool {
+	return s[i].Compare(s[j]) < 0
+}
+
+func (s sorter) Swap(i, j int) {
+	s[i], s[j] = s[j], s[i]
+}
+
+// Sort sorts an array of IDs inplace.
+// It works by wrapping `[]ID` and use `sort.Sort`.
+func Sort(ids []ID) {
+	sort.Sort(sorter(ids))
+}

vendor/golang.org/x/text/encoding/charmap/charmap.go

@@ -33,32 +33,32 @@ var (
 	ISO8859_8I encoding.Encoding = &iso8859_8I
 
 	iso8859_6E = internal.Encoding{
-		ISO8859_6,
-		"ISO-8859-6E",
-		identifier.ISO88596E,
+		Encoding: ISO8859_6,
+		Name:     "ISO-8859-6E",
+		MIB:      identifier.ISO88596E,
 	}
 
 	iso8859_6I = internal.Encoding{
-		ISO8859_6,
-		"ISO-8859-6I",
-		identifier.ISO88596I,
+		Encoding: ISO8859_6,
+		Name:     "ISO-8859-6I",
+		MIB:      identifier.ISO88596I,
 	}
 
 	iso8859_8E = internal.Encoding{
-		ISO8859_8,
-		"ISO-8859-8E",
-		identifier.ISO88598E,
+		Encoding: ISO8859_8,
+		Name:     "ISO-8859-8E",
+		MIB:      identifier.ISO88598E,
 	}
 
 	iso8859_8I = internal.Encoding{
-		ISO8859_8,
-		"ISO-8859-8I",
-		identifier.ISO88598I,
+		Encoding: ISO8859_8,
+		Name:     "ISO-8859-8I",
+		MIB:      identifier.ISO88598I,
 	}
 )
 
 // All is a list of all defined encodings in this package.
-var All = listAll
+var All []encoding.Encoding = listAll
 
 // TODO: implement these encodings, in order of importance.
 // ASCII, ISO8859_1:       Rather common. Close to Windows 1252.
@@ -70,8 +70,8 @@ type utf8Enc struct {
 	data [3]byte
 }
 
-// charmap describes an 8-bit character set encoding.
-type charmap struct {
+// Charmap is an 8-bit character set encoding.
+type Charmap struct {
 	// name is the encoding's name.
 	name string
 	// mib is the encoding type of this encoder.
@@ -79,7 +79,7 @@ type charmap struct {
 	// asciiSuperset states whether the encoding is a superset of ASCII.
 	asciiSuperset bool
 	// low is the lower bound of the encoded byte for a non-ASCII rune. If
-	// charmap.asciiSuperset is true then this will be 0x80, otherwise 0x00.
+	// Charmap.asciiSuperset is true then this will be 0x80, otherwise 0x00.
 	low uint8
 	// replacement is the encoded replacement character.
 	replacement byte
@@ -91,26 +91,30 @@ type charmap struct {
 	encode [256]uint32
 }
 
-func (m *charmap) NewDecoder() *encoding.Decoder {
+// NewDecoder implements the encoding.Encoding interface.
+func (m *Charmap) NewDecoder() *encoding.Decoder {
 	return &encoding.Decoder{Transformer: charmapDecoder{charmap: m}}
 }
 
-func (m *charmap) NewEncoder() *encoding.Encoder {
+// NewEncoder implements the encoding.Encoding interface.
+func (m *Charmap) NewEncoder() *encoding.Encoder {
 	return &encoding.Encoder{Transformer: charmapEncoder{charmap: m}}
 }
 
-func (m *charmap) String() string {
+// String returns the Charmap's name.
+func (m *Charmap) String() string {
 	return m.name
 }
 
-func (m *charmap) ID() (mib identifier.MIB, other string) {
+// ID implements an internal interface.
+func (m *Charmap) ID() (mib identifier.MIB, other string) {
 	return m.mib, ""
 }
 
 // charmapDecoder implements transform.Transformer by decoding to UTF-8.
 type charmapDecoder struct {
 	transform.NopResetter
-	charmap *charmap
+	charmap *Charmap
 }
 
 func (m charmapDecoder) Transform(dst, src []byte, atEOF bool) (nDst, nSrc int, err error) {
@@ -142,10 +146,22 @@ func (m charmapDecoder) Transform(dst, src []byte, atEOF bool) (nDst, nSrc int,
 	return nDst, nSrc, err
 }
 
+// DecodeByte returns the Charmap's rune decoding of the byte b.
+func (m *Charmap) DecodeByte(b byte) rune {
+	switch x := &m.decode[b]; x.len {
+	case 1:
+		return rune(x.data[0])
+	case 2:
+		return rune(x.data[0]&0x1f)<<6 | rune(x.data[1]&0x3f)
+	default:
+		return rune(x.data[0]&0x0f)<<12 | rune(x.data[1]&0x3f)<<6 | rune(x.data[2]&0x3f)
+	}
+}
+
 // charmapEncoder implements transform.Transformer by encoding from UTF-8.
 type charmapEncoder struct {
 	transform.NopResetter
-	charmap *charmap
+	charmap *Charmap
 }
 
 func (m charmapEncoder) Transform(dst, src []byte, atEOF bool) (nDst, nSrc int, err error) {
@@ -207,3 +223,27 @@ loop:
 	}
 	return nDst, nSrc, err
 }
+
+// EncodeRune returns the Charmap's byte encoding of the rune r. ok is whether
+// r is in the Charmap's repertoire. If not, b is set to the Charmap's
+// replacement byte. This is often the ASCII substitute character '\x1a'.
+func (m *Charmap) EncodeRune(r rune) (b byte, ok bool) {
+	if r < utf8.RuneSelf && m.asciiSuperset {
+		return byte(r), true
+	}
+	for low, high := int(m.low), 0x100; ; {
+		if low >= high {
+			return m.replacement, false
+		}
+		mid := (low + high) / 2
+		got := m.encode[mid]
+		gotRune := rune(got & (1<<24 - 1))
+		if gotRune < r {
+			low = mid + 1
+		} else if gotRune > r {
+			high = mid
+		} else {
+			return byte(got >> 24), true
+		}
+	}
+}

vendor/golang.org/x/text/encoding/charmap/maketables.go

@@ -494,7 +494,7 @@ func main() {
 		if e.comment != "" {
 			printf("//\n// %s\n", e.comment)
 		}
-		printf("var %s encoding.Encoding = &%s\n\nvar %s = charmap{\nname: %q,\n",
+		printf("var %s *Charmap = &%s\n\nvar %s = Charmap{\nname: %q,\n",
 			varName, lowerVarName, lowerVarName, e.name)
 		if mibs[e.mib] {
 			log.Fatalf("MIB type %q declared multiple times.", e.mib)
@@ -540,7 +540,7 @@ func main() {
 		}
 		printf("},\n}\n")
 
-		// Add an estimate of the size of a single charmap{} struct value, which
+		// Add an estimate of the size of a single Charmap{} struct value, which
 		// includes two 256 elem arrays of 4 bytes and some extra fields, which
 		// align to 3 uint64s on 64-bit architectures.
 		w.Size += 2*4*256 + 3*8

vendor/golang.org/x/text/encoding/charmap/tables.go

@@ -1,4 +1,4 @@
-// This file was generated by go generate; DO NOT EDIT
+// Code generated by running "go generate" in golang.org/x/text. DO NOT EDIT.
 
 package charmap
 
@@ -8,9 +8,9 @@ import (
 )
 
 // CodePage037 is the IBM Code Page 037 encoding.
-var CodePage037 encoding.Encoding = &codePage037
+var CodePage037 *Charmap = &codePage037
 
-var codePage037 = charmap{
+var codePage037 = Charmap{
 	name:          "IBM Code Page 037",
 	mib:           identifier.IBM037,
 	asciiSuperset: false,
@@ -183,9 +183,9 @@ var codePage037 = charmap{
 }
 
 // CodePage437 is the IBM Code Page 437 encoding.
-var CodePage437 encoding.Encoding = &codePage437
+var CodePage437 *Charmap = &codePage437
 
-var codePage437 = charmap{
+var codePage437 = Charmap{
 	name:          "IBM Code Page 437",
 	mib:           identifier.PC8CodePage437,
 	asciiSuperset: true,
@@ -358,9 +358,9 @@ var codePage437 = charmap{
 }
 
 // CodePage850 is the IBM Code Page 850 encoding.
-var CodePage850 encoding.Encoding = &codePage850
+var CodePage850 *Charmap = &codePage850
 
-var codePage850 = charmap{
+var codePage850 = Charmap{
 	name:          "IBM Code Page 850",
 	mib:           identifier.PC850Multilingual,
 	asciiSuperset: true,
@@ -533,9 +533,9 @@ var codePage850 = charmap{
 }
 
 // CodePage852 is the IBM Code Page 852 encoding.
-var CodePage852 encoding.Encoding = &codePage852
+var CodePage852 *Charmap = &codePage852
 
-var codePage852 = charmap{
+var codePage852 = Charmap{
 	name:          "IBM Code Page 852",
 	mib:           identifier.PCp852,
 	asciiSuperset: true,
@@ -708,9 +708,9 @@ var codePage852 = charmap{
 }
 
 // CodePage855 is the IBM Code Page 855 encoding.
-var CodePage855 encoding.Encoding = &codePage855
+var CodePage855 *Charmap = &codePage855
 
-var codePage855 = charmap{
+var codePage855 = Charmap{
 	name:          "IBM Code Page 855",
 	mib:           identifier.IBM855,
 	asciiSuperset: true,
@@ -883,9 +883,9 @@ var codePage855 = charmap{
 }
 
 // CodePage858 is the Windows Code Page 858 encoding.
-var CodePage858 encoding.Encoding = &codePage858
+var CodePage858 *Charmap = &codePage858
 
-var codePage858 = charmap{
+var codePage858 = Charmap{
 	name:          "Windows Code Page 858",
 	mib:           identifier.IBM00858,
 	asciiSuperset: true,
@@ -1058,9 +1058,9 @@ var codePage858 = charmap{
 }
 
 // CodePage860 is the IBM Code Page 860 encoding.
-var CodePage860 encoding.Encoding = &codePage860
+var CodePage860 *Charmap = &codePage860
 
-var codePage860 = charmap{
+var codePage860 = Charmap{
 	name:          "IBM Code Page 860",
 	mib:           identifier.IBM860,
 	asciiSuperset: true,
@@ -1233,9 +1233,9 @@ var codePage860 = charmap{
 }
 
 // CodePage862 is the IBM Code Page 862 encoding.
-var CodePage862 encoding.Encoding = &codePage862
+var CodePage862 *Charmap = &codePage862
 
-var codePage862 = charmap{
+var codePage862 = Charmap{
 	name:          "IBM Code Page 862",
 	mib:           identifier.PC862LatinHebrew,
 	asciiSuperset: true,
@@ -1408,9 +1408,9 @@ var codePage862 = charmap{
 }
 
 // CodePage863 is the IBM Code Page 863 encoding.
-var CodePage863 encoding.Encoding = &codePage863
+var CodePage863 *Charmap = &codePage863
 
-var codePage863 = charmap{
+var codePage863 = Charmap{
 	name:          "IBM Code Page 863",
 	mib:           identifier.IBM863,
 	asciiSuperset: true,
@@ -1583,9 +1583,9 @@ var codePage863 = charmap{
 }
 
 // CodePage865 is the IBM Code Page 865 encoding.
-var CodePage865 encoding.Encoding = &codePage865
+var CodePage865 *Charmap = &codePage865
 
-var codePage865 = charmap{
+var codePage865 = Charmap{
 	name:          "IBM Code Page 865",
 	mib:           identifier.IBM865,
 	asciiSuperset: true,
@@ -1758,9 +1758,9 @@ var codePage865 = charmap{
 }
 
 // CodePage866 is the IBM Code Page 866 encoding.
-var CodePage866 encoding.Encoding = &codePage866
+var CodePage866 *Charmap = &codePage866
 
-var codePage866 = charmap{
+var codePage866 = Charmap{
 	name:          "IBM Code Page 866",
 	mib:           identifier.IBM866,
 	asciiSuperset: true,
@@ -1933,9 +1933,9 @@ var codePage866 = charmap{
 }
 
 // CodePage1047 is the IBM Code Page 1047 encoding.
-var CodePage1047 encoding.Encoding = &codePage1047
+var CodePage1047 *Charmap = &codePage1047
 
-var codePage1047 = charmap{
+var codePage1047 = Charmap{
 	name:          "IBM Code Page 1047",
 	mib:           identifier.IBM1047,
 	asciiSuperset: false,
@@ -2108,9 +2108,9 @@ var codePage1047 = charmap{
 }
 
 // CodePage1140 is the IBM Code Page 1140 encoding.
-var CodePage1140 encoding.Encoding = &codePage1140
+var CodePage1140 *Charmap = &codePage1140
 
-var codePage1140 = charmap{
+var codePage1140 = Charmap{
 	name:          "IBM Code Page 1140",
 	mib:           identifier.IBM01140,
 	asciiSuperset: false,
@@ -2283,9 +2283,9 @@ var codePage1140 = charmap{
 }
 
 // ISO8859_1 is the ISO 8859-1 encoding.
-var ISO8859_1 encoding.Encoding = &iso8859_1
+var ISO8859_1 *Charmap = &iso8859_1
 
-var iso8859_1 = charmap{
+var iso8859_1 = Charmap{
 	name:          "ISO 8859-1",
 	mib:           identifier.ISOLatin1,
 	asciiSuperset: true,
@@ -2458,9 +2458,9 @@ var iso8859_1 = charmap{
 }
 
 // ISO8859_2 is the ISO 8859-2 encoding.
-var ISO8859_2 encoding.Encoding = &iso8859_2
+var ISO8859_2 *Charmap = &iso8859_2
 
-var iso8859_2 = charmap{
+var iso8859_2 = Charmap{
 	name:          "ISO 8859-2",
 	mib:           identifier.ISOLatin2,
 	asciiSuperset: true,
@@ -2633,9 +2633,9 @@ var iso8859_2 = charmap{
 }
 
 // ISO8859_3 is the ISO 8859-3 encoding.
-var ISO8859_3 encoding.Encoding = &iso8859_3
+var ISO8859_3 *Charmap = &iso8859_3
 
-var iso8859_3 = charmap{
+var iso8859_3 = Charmap{
 	name:          "ISO 8859-3",
 	mib:           identifier.ISOLatin3,
 	asciiSuperset: true,
@@ -2808,9 +2808,9 @@ var iso8859_3 = charmap{
 }
 
 // ISO8859_4 is the ISO 8859-4 encoding.
-var ISO8859_4 encoding.Encoding = &iso8859_4
+var ISO8859_4 *Charmap = &iso8859_4
 
-var iso8859_4 = charmap{
+var iso8859_4 = Charmap{
 	name:          "ISO 8859-4",
 	mib:           identifier.ISOLatin4,
 	asciiSuperset: true,
@@ -2983,9 +2983,9 @@ var iso8859_4 = charmap{
 }
 
 // ISO8859_5 is the ISO 8859-5 encoding.
-var ISO8859_5 encoding.Encoding = &iso8859_5
+var ISO8859_5 *Charmap = &iso8859_5
 
-var iso8859_5 = charmap{
+var iso8859_5 = Charmap{
 	name:          "ISO 8859-5",
 	mib:           identifier.ISOLatinCyrillic,
 	asciiSuperset: true,
@@ -3158,9 +3158,9 @@ var iso8859_5 = charmap{
 }
 
 // ISO8859_6 is the ISO 8859-6 encoding.
-var ISO8859_6 encoding.Encoding = &iso8859_6
+var ISO8859_6 *Charmap = &iso8859_6
 
-var iso8859_6 = charmap{
+var iso8859_6 = Charmap{
 	name:          "ISO 8859-6",
 	mib:           identifier.ISOLatinArabic,
 	asciiSuperset: true,
@@ -3333,9 +3333,9 @@ var iso8859_6 = charmap{
 }
 
 // ISO8859_7 is the ISO 8859-7 encoding.
-var ISO8859_7 encoding.Encoding = &iso8859_7
+var ISO8859_7 *Charmap = &iso8859_7
 
-var iso8859_7 = charmap{
+var iso8859_7 = Charmap{
 	name:          "ISO 8859-7",
 	mib:           identifier.ISOLatinGreek,
 	asciiSuperset: true,
@@ -3508,9 +3508,9 @@ var iso8859_7 = charmap{
 }
 
 // ISO8859_8 is the ISO 8859-8 encoding.
-var ISO8859_8 encoding.Encoding = &iso8859_8
+var ISO8859_8 *Charmap = &iso8859_8
 
-var iso8859_8 = charmap{
+var iso8859_8 = Charmap{
 	name:          "ISO 8859-8",
 	mib:           identifier.ISOLatinHebrew,
 	asciiSuperset: true,
@@ -3683,9 +3683,9 @@ var iso8859_8 = charmap{
 }
 
 // ISO8859_9 is the ISO 8859-9 encoding.
-var ISO8859_9 encoding.Encoding = &iso8859_9
+var ISO8859_9 *Charmap = &iso8859_9
 
-var iso8859_9 = charmap{
+var iso8859_9 = Charmap{
 	name:          "ISO 8859-9",
 	mib:           identifier.ISOLatin5,
 	asciiSuperset: true,
@@ -3858,9 +3858,9 @@ var iso8859_9 = charmap{
 }
 
 // ISO8859_10 is the ISO 8859-10 encoding.
-var ISO8859_10 encoding.Encoding = &iso8859_10
+var ISO8859_10 *Charmap = &iso8859_10
 
-var iso8859_10 = charmap{
+var iso8859_10 = Charmap{
 	name:          "ISO 8859-10",
 	mib:           identifier.ISOLatin6,
 	asciiSuperset: true,
@@ -4033,9 +4033,9 @@ var iso8859_10 = charmap{
 }
 
 // ISO8859_13 is the ISO 8859-13 encoding.
-var ISO8859_13 encoding.Encoding = &iso8859_13
+var ISO8859_13 *Charmap = &iso8859_13
 
-var iso8859_13 = charmap{
+var iso8859_13 = Charmap{
 	name:          "ISO 8859-13",
 	mib:           identifier.ISO885913,
 	asciiSuperset: true,
@@ -4208,9 +4208,9 @@ var iso8859_13 = charmap{
 }
 
 // ISO8859_14 is the ISO 8859-14 encoding.
-var ISO8859_14 encoding.Encoding = &iso8859_14
+var ISO8859_14 *Charmap = &iso8859_14
 
-var iso8859_14 = charmap{
+var iso8859_14 = Charmap{
 	name:          "ISO 8859-14",
 	mib:           identifier.ISO885914,
 	asciiSuperset: true,
@@ -4383,9 +4383,9 @@ var iso8859_14 = charmap{
 }
 
 // ISO8859_15 is the ISO 8859-15 encoding.
-var ISO8859_15 encoding.Encoding = &iso8859_15
+var ISO8859_15 *Charmap = &iso8859_15
 
-var iso8859_15 = charmap{
+var iso8859_15 = Charmap{
 	name:          "ISO 8859-15",
 	mib:           identifier.ISO885915,
 	asciiSuperset: true,
@@ -4558,9 +4558,9 @@ var iso8859_15 = charmap{
 }
 
 // ISO8859_16 is the ISO 8859-16 encoding.
-var ISO8859_16 encoding.Encoding = &iso8859_16
+var ISO8859_16 *Charmap = &iso8859_16
 
-var iso8859_16 = charmap{
+var iso8859_16 = Charmap{
 	name:          "ISO 8859-16",
 	mib:           identifier.ISO885916,
 	asciiSuperset: true,
@@ -4733,9 +4733,9 @@ var iso8859_16 = charmap{
 }
 
 // KOI8R is the KOI8-R encoding.
-var KOI8R encoding.Encoding = &koi8R
+var KOI8R *Charmap = &koi8R
 
-var koi8R = charmap{
+var koi8R = Charmap{
 	name:          "KOI8-R",
 	mib:           identifier.KOI8R,
 	asciiSuperset: true,
@@ -4908,9 +4908,9 @@ var koi8R = charmap{
 }
 
 // KOI8U is the KOI8-U encoding.
-var KOI8U encoding.Encoding = &koi8U
+var KOI8U *Charmap = &koi8U
 
-var koi8U = charmap{
+var koi8U = Charmap{
 	name:          "KOI8-U",
 	mib:           identifier.KOI8U,
 	asciiSuperset: true,
@@ -5083,9 +5083,9 @@ var koi8U = charmap{
 }
 
 // Macintosh is the Macintosh encoding.
-var Macintosh encoding.Encoding = &macintosh
+var Macintosh *Charmap = &macintosh
 
-var macintosh = charmap{
+var macintosh = Charmap{
 	name:          "Macintosh",
 	mib:           identifier.Macintosh,
 	asciiSuperset: true,
@@ -5258,9 +5258,9 @@ var macintosh = charmap{
 }
 
 // MacintoshCyrillic is the Macintosh Cyrillic encoding.
-var MacintoshCyrillic encoding.Encoding = &macintoshCyrillic
+var MacintoshCyrillic *Charmap = &macintoshCyrillic
 
-var macintoshCyrillic = charmap{
+var macintoshCyrillic = Charmap{
 	name:          "Macintosh Cyrillic",
 	mib:           identifier.MacintoshCyrillic,
 	asciiSuperset: true,
@@ -5433,9 +5433,9 @@ var macintoshCyrillic = charmap{
 }
 
 // Windows874 is the Windows 874 encoding.
-var Windows874 encoding.Encoding = &windows874
+var Windows874 *Charmap = &windows874
 
-var windows874 = charmap{
+var windows874 = Charmap{
 	name:          "Windows 874",
 	mib:           identifier.Windows874,
 	asciiSuperset: true,
@@ -5608,9 +5608,9 @@ var windows874 = charmap{
 }
 
 // Windows1250 is the Windows 1250 encoding.
-var Windows1250 encoding.Encoding = &windows1250
+var Windows1250 *Charmap = &windows1250
 
-var windows1250 = charmap{
+var windows1250 = Charmap{
 	name:          "Windows 1250",
 	mib:           identifier.Windows1250,
 	asciiSuperset: true,
@@ -5783,9 +5783,9 @@ var windows1250 = charmap{
 }
 
 // Windows1251 is the Windows 1251 encoding.
-var Windows1251 encoding.Encoding = &windows1251
+var Windows1251 *Charmap = &windows1251
 
-var windows1251 = charmap{
+var windows1251 = Charmap{
 	name:          "Windows 1251",
 	mib:           identifier.Windows1251,
 	asciiSuperset: true,
@@ -5958,9 +5958,9 @@ var windows1251 = charmap{
 }
 
 // Windows1252 is the Windows 1252 encoding.
-var Windows1252 encoding.Encoding = &windows1252
+var Windows1252 *Charmap = &windows1252
 
-var windows1252 = charmap{
+var windows1252 = Charmap{
 	name:          "Windows 1252",
 	mib:           identifier.Windows1252,
 	asciiSuperset: true,
@@ -6133,9 +6133,9 @@ var windows1252 = charmap{
 }
 
 // Windows1253 is the Windows 1253 encoding.
-var Windows1253 encoding.Encoding = &windows1253
+var Windows1253 *Charmap = &windows1253
 
-var windows1253 = charmap{
+var windows1253 = Charmap{
 	name:          "Windows 1253",
 	mib:           identifier.Windows1253,
 	asciiSuperset: true,
@@ -6308,9 +6308,9 @@ var windows1253 = charmap{
 }
 
 // Windows1254 is the Windows 1254 encoding.
-var Windows1254 encoding.Encoding = &windows1254
+var Windows1254 *Charmap = &windows1254
 
-var windows1254 = charmap{
+var windows1254 = Charmap{
 	name:          "Windows 1254",
 	mib:           identifier.Windows1254,
 	asciiSuperset: true,
@@ -6483,9 +6483,9 @@ var windows1254 = charmap{
 }
 
 // Windows1255 is the Windows 1255 encoding.
-var Windows1255 encoding.Encoding = &windows1255
+var Windows1255 *Charmap = &windows1255
 
-var windows1255 = charmap{
+var windows1255 = Charmap{
 	name:          "Windows 1255",
 	mib:           identifier.Windows1255,
 	asciiSuperset: true,
@@ -6593,7 +6593,7 @@ var windows1255 = charmap{
 		{2, [3]byte{0xd6, 0xb4, 0x00}}, {2, [3]byte{0xd6, 0xb5, 0x00}},
 		{2, [3]byte{0xd6, 0xb6, 0x00}}, {2, [3]byte{0xd6, 0xb7, 0x00}},
 		{2, [3]byte{0xd6, 0xb8, 0x00}}, {2, [3]byte{0xd6, 0xb9, 0x00}},
-		{3, [3]byte{0xef, 0xbf, 0xbd}}, {2, [3]byte{0xd6, 0xbb, 0x00}},
+		{2, [3]byte{0xd6, 0xba, 0x00}}, {2, [3]byte{0xd6, 0xbb, 0x00}},
 		{2, [3]byte{0xd6, 0xbc, 0x00}}, {2, [3]byte{0xd6, 0xbd, 0x00}},
 		{2, [3]byte{0xd6, 0xbe, 0x00}}, {2, [3]byte{0xd6, 0xbf, 0x00}},
 		{2, [3]byte{0xd7, 0x80, 0x00}}, {2, [3]byte{0xd7, 0x81, 0x00}},
@@ -6643,24 +6643,24 @@ var windows1255 = charmap{
 		0xb20000b2, 0xb30000b3, 0xb40000b4, 0xb50000b5, 0xb60000b6, 0xb70000b7, 0xb80000b8, 0xb90000b9,
 		0xbb0000bb, 0xbc0000bc, 0xbd0000bd, 0xbe0000be, 0xbf0000bf, 0xaa0000d7, 0xba0000f7, 0x83000192,
 		0x880002c6, 0x980002dc, 0xc00005b0, 0xc10005b1, 0xc20005b2, 0xc30005b3, 0xc40005b4, 0xc50005b5,
-		0xc60005b6, 0xc70005b7, 0xc80005b8, 0xc90005b9, 0xcb0005bb, 0xcc0005bc, 0xcd0005bd, 0xce0005be,
-		0xcf0005bf, 0xd00005c0, 0xd10005c1, 0xd20005c2, 0xd30005c3, 0xe00005d0, 0xe10005d1, 0xe20005d2,
-		0xe30005d3, 0xe40005d4, 0xe50005d5, 0xe60005d6, 0xe70005d7, 0xe80005d8, 0xe90005d9, 0xea0005da,
-		0xeb0005db, 0xec0005dc, 0xed0005dd, 0xee0005de, 0xef0005df, 0xf00005e0, 0xf10005e1, 0xf20005e2,
-		0xf30005e3, 0xf40005e4, 0xf50005e5, 0xf60005e6, 0xf70005e7, 0xf80005e8, 0xf90005e9, 0xfa0005ea,
-		0xd40005f0, 0xd50005f1, 0xd60005f2, 0xd70005f3, 0xd80005f4, 0xfd00200e, 0xfe00200f, 0x96002013,
-		0x97002014, 0x91002018, 0x92002019, 0x8200201a, 0x9300201c, 0x9400201d, 0x8400201e, 0x86002020,
-		0x87002021, 0x95002022, 0x85002026, 0x89002030, 0x8b002039, 0x9b00203a, 0xa40020aa, 0x800020ac,
-		0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122,
+		0xc60005b6, 0xc70005b7, 0xc80005b8, 0xc90005b9, 0xca0005ba, 0xcb0005bb, 0xcc0005bc, 0xcd0005bd,
+		0xce0005be, 0xcf0005bf, 0xd00005c0, 0xd10005c1, 0xd20005c2, 0xd30005c3, 0xe00005d0, 0xe10005d1,
+		0xe20005d2, 0xe30005d3, 0xe40005d4, 0xe50005d5, 0xe60005d6, 0xe70005d7, 0xe80005d8, 0xe90005d9,
+		0xea0005da, 0xeb0005db, 0xec0005dc, 0xed0005dd, 0xee0005de, 0xef0005df, 0xf00005e0, 0xf10005e1,
+		0xf20005e2, 0xf30005e3, 0xf40005e4, 0xf50005e5, 0xf60005e6, 0xf70005e7, 0xf80005e8, 0xf90005e9,
+		0xfa0005ea, 0xd40005f0, 0xd50005f1, 0xd60005f2, 0xd70005f3, 0xd80005f4, 0xfd00200e, 0xfe00200f,
+		0x96002013, 0x97002014, 0x91002018, 0x92002019, 0x8200201a, 0x9300201c, 0x9400201d, 0x8400201e,
+		0x86002020, 0x87002021, 0x95002022, 0x85002026, 0x89002030, 0x8b002039, 0x9b00203a, 0xa40020aa,
+		0x800020ac, 0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122,
 		0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122,
 		0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122, 0x99002122,
 	},
 }
 
 // Windows1256 is the Windows 1256 encoding.
-var Windows1256 encoding.Encoding = &windows1256
+var Windows1256 *Charmap = &windows1256
 
-var windows1256 = charmap{
+var windows1256 = Charmap{
 	name:          "Windows 1256",
 	mib:           identifier.Windows1256,
 	asciiSuperset: true,
@@ -6833,9 +6833,9 @@ var windows1256 = charmap{
 }
 
 // Windows1257 is the Windows 1257 encoding.
-var Windows1257 encoding.Encoding = &windows1257
+var Windows1257 *Charmap = &windows1257
 
-var windows1257 = charmap{
+var windows1257 = Charmap{
 	name:          "Windows 1257",
 	mib:           identifier.Windows1257,
 	asciiSuperset: true,
@@ -7008,9 +7008,9 @@ var windows1257 = charmap{
 }
 
 // Windows1258 is the Windows 1258 encoding.
-var Windows1258 encoding.Encoding = &windows1258
+var Windows1258 *Charmap = &windows1258
 
-var windows1258 = charmap{
+var windows1258 = Charmap{
 	name:          "Windows 1258",
 	mib:           identifier.Windows1258,
 	asciiSuperset: true,
@@ -7185,9 +7185,9 @@ var windows1258 = charmap{
 // XUserDefined is the X-User-Defined encoding.
 //
 // It is defined at http://encoding.spec.whatwg.org/#x-user-defined
-var XUserDefined encoding.Encoding = &xUserDefined
+var XUserDefined *Charmap = &xUserDefined
 
-var xUserDefined = charmap{
+var xUserDefined = Charmap{
 	name:          "X-User-Defined",
 	mib:           identifier.XUserDefined,
 	asciiSuperset: true,

vendor/golang.org/x/text/encoding/htmlindex/gen.go

@@ -133,7 +133,10 @@ var consts = map[string]string{
 // locales is taken from
 // https://html.spec.whatwg.org/multipage/syntax.html#encoding-sniffing-algorithm.
 var locales = []struct{ tag, name string }{
-	{"und", "windows-1252"}, // The default value.
+	// The default value. Explicitly state latin to benefit from the exact
+	// script option, while still making 1252 the default encoding for languages
+	// written in Latin script.
+	{"und_Latn", "windows-1252"},
 	{"ar", "windows-1256"},
 	{"ba", "windows-1251"},
 	{"be", "windows-1251"},

vendor/golang.org/x/text/encoding/htmlindex/htmlindex.go

@@ -50,7 +50,7 @@ func LanguageDefault(tag language.Tag) string {
 		for _, t := range strings.Split(locales, " ") {
 			tags = append(tags, language.MustParse(t))
 		}
-		matcher = language.NewMatcher(tags)
+		matcher = language.NewMatcher(tags, language.PreferSameScript(true))
 	})
 	_, i, _ := matcher.Match(tag)
 	return canonical[localeMap[i]] // Default is Windows-1252.

vendor/golang.org/x/text/encoding/htmlindex/tables.go

@@ -1,4 +1,4 @@
-// This file was generated by go generate; DO NOT EDIT
+// Code generated by running "go generate" in golang.org/x/text. DO NOT EDIT.
 
 package htmlindex
 
@@ -313,7 +313,7 @@ var nameMap = map[string]htmlEncoding{
 }
 
 var localeMap = []htmlEncoding{
-	windows1252, // und
+	windows1252, // und_Latn
 	windows1256, // ar
 	windows1251, // ba
 	windows1251, // be
@@ -349,4 +349,4 @@ var localeMap = []htmlEncoding{
 	big5,        // zh-hant
 }
 
-const locales = "und ar ba be bg cs el et fa he hr hu ja kk ko ku ky lt lv mk pl ru sah sk sl sr tg th tr tt uk vi zh-hans zh-hant"
+const locales = "und_Latn ar ba be bg cs el et fa he hr hu ja kk ko ku ky lt lv mk pl ru sah sk sl sr tg th tr tt uk vi zh-hans zh-hant"

vendor/golang.org/x/text/encoding/ianaindex/gen.go

@@ -0,0 +1,192 @@
+// Copyright 2017 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+// +build ignore
+
+package main
+
+import (
+	"encoding/xml"
+	"fmt"
+	"io"
+	"log"
+	"sort"
+	"strconv"
+	"strings"
+
+	"golang.org/x/text/encoding/internal/identifier"
+	"golang.org/x/text/internal/gen"
+)
+
+type registry struct {
+	XMLName  xml.Name `xml:"registry"`
+	Updated  string   `xml:"updated"`
+	Registry []struct {
+		ID     string `xml:"id,attr"`
+		Record []struct {
+			Name string `xml:"name"`
+			Xref []struct {
+				Type string `xml:"type,attr"`
+				Data string `xml:"data,attr"`
+			} `xml:"xref"`
+			Desc struct {
+				Data string `xml:",innerxml"`
+			} `xml:"description,"`
+			MIB   string   `xml:"value"`
+			Alias []string `xml:"alias"`
+			MIME  string   `xml:"preferred_alias"`
+		} `xml:"record"`
+	} `xml:"registry"`
+}
+
+func main() {
+	r := gen.OpenIANAFile("assignments/character-sets/character-sets.xml")
+	reg := &registry{}
+	if err := xml.NewDecoder(r).Decode(&reg); err != nil && err != io.EOF {
+		log.Fatalf("Error decoding charset registry: %v", err)
+	}
+	if len(reg.Registry) == 0 || reg.Registry[0].ID != "character-sets-1" {
+		log.Fatalf("Unexpected ID %s", reg.Registry[0].ID)
+	}
+
+	x := &indexInfo{}
+
+	for _, rec := range reg.Registry[0].Record {
+		mib := identifier.MIB(parseInt(rec.MIB))
+		x.addEntry(mib, rec.Name)
+		for _, a := range rec.Alias {
+			a = strings.Split(a, " ")[0] // strip comments.
+			x.addAlias(a, mib)
+			// MIB name aliases are prefixed with a "cs" (character set) in the
+			// registry to identify them as display names and to ensure that
+			// the name starts with a lowercase letter in case it is used as
+			// an identifier. We remove it to be left with a nice clean name.
+			if strings.HasPrefix(a, "cs") {
+				x.setName(2, a[2:])
+			}
+		}
+		if rec.MIME != "" {
+			x.addAlias(rec.MIME, mib)
+			x.setName(1, rec.MIME)
+		}
+	}
+
+	w := gen.NewCodeWriter()
+
+	fmt.Fprintln(w, `import "golang.org/x/text/encoding/internal/identifier"`)
+
+	writeIndex(w, x)
+
+	w.WriteGoFile("tables.go", "ianaindex")
+}
+
+type alias struct {
+	name string
+	mib  identifier.MIB
+}
+
+type indexInfo struct {
+	// compacted index from code to MIB
+	codeToMIB []identifier.MIB
+	alias     []alias
+	names     [][3]string
+}
+
+func (ii *indexInfo) Len() int {
+	return len(ii.codeToMIB)
+}
+
+func (ii *indexInfo) Less(a, b int) bool {
+	return ii.codeToMIB[a] < ii.codeToMIB[b]
+}
+
+func (ii *indexInfo) Swap(a, b int) {
+	ii.codeToMIB[a], ii.codeToMIB[b] = ii.codeToMIB[b], ii.codeToMIB[a]
+	// Co-sort the names.
+	ii.names[a], ii.names[b] = ii.names[b], ii.names[a]
+}
+
+func (ii *indexInfo) setName(i int, name string) {
+	ii.names[len(ii.names)-1][i] = name
+}
+
+func (ii *indexInfo) addEntry(mib identifier.MIB, name string) {
+	ii.names = append(ii.names, [3]string{name, name, name})
+	ii.addAlias(name, mib)
+	ii.codeToMIB = append(ii.codeToMIB, mib)
+}
+
+func (ii *indexInfo) addAlias(name string, mib identifier.MIB) {
+	// Don't add duplicates for the same mib. Adding duplicate aliases for
+	// different MIBs will cause the compiler to barf on an invalid map: great!.
+	for i := len(ii.alias) - 1; i >= 0 && ii.alias[i].mib == mib; i-- {
+		if ii.alias[i].name == name {
+			return
+		}
+	}
+	ii.alias = append(ii.alias, alias{name, mib})
+	lower := strings.ToLower(name)
+	if lower != name {
+		ii.addAlias(lower, mib)
+	}
+}
+
+const maxMIMENameLen = '0' - 1 // officially 40, but we leave some buffer.
+
+func writeIndex(w *gen.CodeWriter, x *indexInfo) {
+	sort.Stable(x)
+
+	// Write constants.
+	fmt.Fprintln(w, "const (")
+	for i, m := range x.codeToMIB {
+		if i == 0 {
+			fmt.Fprintf(w, "enc%d = iota\n", m)
+		} else {
+			fmt.Fprintf(w, "enc%d\n", m)
+		}
+	}
+	fmt.Fprintln(w, "numIANA")
+	fmt.Fprintln(w, ")")
+
+	w.WriteVar("ianaToMIB", x.codeToMIB)
+
+	var ianaNames, mibNames []string
+	for _, names := range x.names {
+		n := names[0]
+		if names[0] != names[1] {
+			// MIME names are mostly identical to IANA names. We share the
+			// tables by setting the first byte of the string to an index into
+			// the string itself (< maxMIMENameLen) to the IANA name. The MIME
+			// name immediately follows the index.
+			x := len(names[1]) + 1
+			if x > maxMIMENameLen {
+				log.Fatalf("MIME name length (%d) > %d", x, maxMIMENameLen)
+			}
+			n = string(x) + names[1] + names[0]
+		}
+		ianaNames = append(ianaNames, n)
+		mibNames = append(mibNames, names[2])
+	}
+
+	w.WriteVar("ianaNames", ianaNames)
+	w.WriteVar("mibNames", mibNames)
+
+	w.WriteComment(`
+	TODO: Instead of using a map, we could use binary search strings doing
+	on-the fly lower-casing per character. This allows to always avoid
+	allocation and will be considerably more compact.`)
+	fmt.Fprintln(w, "var ianaAliases = map[string]int{")
+	for _, a := range x.alias {
+		fmt.Fprintf(w, "%q: enc%d,\n", a.name, a.mib)
+	}
+	fmt.Fprintln(w, "}")
+}
+
+func parseInt(s string) int {
+	x, err := strconv.ParseInt(s, 10, 64)
+	if err != nil {
+		log.Fatalf("Could not parse integer: %v", err)
+	}
+	return int(x)
+}

vendor/golang.org/x/text/encoding/ianaindex/ianaindex.go

@@ -2,17 +2,28 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.
 
+//go:generate go run gen.go
+
 // Package ianaindex maps names to Encodings as specified by the IANA registry.
 // This includes both the MIME and IANA names.
 //
-// Status: this package is an incomplete API sketch, and isn't usable yet.
-//
 // See http://www.iana.org/assignments/character-sets/character-sets.xhtml for
 // more details.
 package ianaindex
 
 import (
+	"errors"
+	"sort"
+	"strings"
+
 	"golang.org/x/text/encoding"
+	"golang.org/x/text/encoding/charmap"
+	"golang.org/x/text/encoding/internal/identifier"
+	"golang.org/x/text/encoding/japanese"
+	"golang.org/x/text/encoding/korean"
+	"golang.org/x/text/encoding/simplifiedchinese"
+	"golang.org/x/text/encoding/traditionalchinese"
+	"golang.org/x/text/encoding/unicode"
 )
 
 // TODO: remove the "Status... incomplete" in the package doc comment.
@@ -25,28 +36,68 @@ import (
 // support MIME otherwise.
 
 var (
-	// MIME is an index to map MIME names. It does not support aliases.
-	MIME *Index
+	// MIME is an index to map MIME names.
+	MIME *Index = mime
 
 	// IANA is an index that supports all names and aliases using IANA names as
 	// the canonical identifier.
-	IANA *Index
+	IANA *Index = iana
+
+	// MIB is an index that associates the MIB display name with an Encoding.
+	MIB *Index = mib
+
+	mime = &Index{mimeName, ianaToMIB, ianaAliases, encodings[:]}
+	iana = &Index{ianaName, ianaToMIB, ianaAliases, encodings[:]}
+	mib  = &Index{mibName, ianaToMIB, ianaAliases, encodings[:]}
 )
 
 // Index maps names registered by IANA to Encodings.
+// Currently different Indexes only differ in the names they return for
+// encodings. In the future they may also differ in supported aliases.
 type Index struct {
+	names func(i int) string
+	toMIB []identifier.MIB // Sorted slice of supported MIBs
+	alias map[string]int
+	enc   []encoding.Encoding
 }
 
-// Get returns an Encoding for IANA-registered names. Matching is
+var (
+	errInvalidName = errors.New("ianaindex: invalid encoding name")
+	errUnknown     = errors.New("ianaindex: unknown Encoding")
+	errUnsupported = errors.New("ianaindex: unsupported Encoding")
+)
+
+// Encoding returns an Encoding for IANA-registered names. Matching is
 // case-insensitive.
-func (x *Index) Get(name string) (encoding.Encoding, error) {
-	panic("TODO: implement")
+func (x *Index) Encoding(name string) (encoding.Encoding, error) {
+	name = strings.TrimSpace(name)
+	// First try without lowercasing (possibly creating an allocation).
+	i, ok := x.alias[name]
+	if !ok {
+		i, ok = x.alias[strings.ToLower(name)]
+		if !ok {
+			return nil, errInvalidName
+		}
+	}
+	return x.enc[i], nil
 }
 
 // Name reports the canonical name of the given Encoding. It will return an
 // error if the e is not associated with a known encoding scheme.
 func (x *Index) Name(e encoding.Encoding) (string, error) {
-	panic("TODO: implement")
+	id, ok := e.(identifier.Interface)
+	if !ok {
+		return "", errUnknown
+	}
+	mib, _ := id.ID()
+	if mib == 0 {
+		return "", errUnknown
+	}
+	v := findMIB(x.toMIB, mib)
+	if v == -1 {
+		return "", errUnsupported
+	}
+	return x.names(v), nil
 }
 
 // TODO: the coverage of this index is rather spotty. Allowing users to set
@@ -65,3 +116,94 @@ func (x *Index) Name(e encoding.Encoding) (string, error) {
 // func (x *Index) Set(name string, e encoding.Encoding) error {
 // 	panic("TODO: implement")
 // }
+
+func findMIB(x []identifier.MIB, mib identifier.MIB) int {
+	i := sort.Search(len(x), func(i int) bool { return x[i] >= mib })
+	if i < len(x) && x[i] == mib {
+		return i
+	}
+	return -1
+}
+
+const maxMIMENameLen = '0' - 1 // officially 40, but we leave some buffer.
+
+func mimeName(x int) string {
+	n := ianaNames[x]
+	// See gen.go for a description of the encoding.
+	if n[0] <= maxMIMENameLen {
+		return n[1:n[0]]
+	}
+	return n
+}
+
+func ianaName(x int) string {
+	n := ianaNames[x]
+	// See gen.go for a description of the encoding.
+	if n[0] <= maxMIMENameLen {
+		return n[n[0]:]
+	}
+	return n
+}
+
+func mibName(x int) string {
+	return mibNames[x]
+}
+
+var encodings = [numIANA]encoding.Encoding{
+	enc106:  unicode.UTF8,
+	enc1015: unicode.UTF16(unicode.BigEndian, unicode.UseBOM),
+	enc1013: unicode.UTF16(unicode.BigEndian, unicode.IgnoreBOM),
+	enc1014: unicode.UTF16(unicode.LittleEndian, unicode.IgnoreBOM),
+	enc2028: charmap.CodePage037,
+	enc2011: charmap.CodePage437,
+	enc2009: charmap.CodePage850,
+	enc2010: charmap.CodePage852,
+	enc2046: charmap.CodePage855,
+	enc2089: charmap.CodePage858,
+	enc2048: charmap.CodePage860,
+	enc2013: charmap.CodePage862,
+	enc2050: charmap.CodePage863,
+	enc2052: charmap.CodePage865,
+	enc2086: charmap.CodePage866,
+	enc2102: charmap.CodePage1047,
+	enc2091: charmap.CodePage1140,
+	enc4:    charmap.ISO8859_1,
+	enc5:    charmap.ISO8859_2,
+	enc6:    charmap.ISO8859_3,
+	enc7:    charmap.ISO8859_4,
+	enc8:    charmap.ISO8859_5,
+	enc9:    charmap.ISO8859_6,
+	enc81:   charmap.ISO8859_6E,
+	enc82:   charmap.ISO8859_6I,
+	enc10:   charmap.ISO8859_7,
+	enc11:   charmap.ISO8859_8,
+	enc84:   charmap.ISO8859_8E,
+	enc85:   charmap.ISO8859_8I,
+	enc12:   charmap.ISO8859_9,
+	enc13:   charmap.ISO8859_10,
+	enc109:  charmap.ISO8859_13,
+	enc110:  charmap.ISO8859_14,
+	enc111:  charmap.ISO8859_15,
+	enc112:  charmap.ISO8859_16,
+	enc2084: charmap.KOI8R,
+	enc2088: charmap.KOI8U,
+	enc2027: charmap.Macintosh,
+	enc2109: charmap.Windows874,
+	enc2250: charmap.Windows1250,
+	enc2251: charmap.Windows1251,
+	enc2252: charmap.Windows1252,
+	enc2253: charmap.Windows1253,
+	enc2254: charmap.Windows1254,
+	enc2255: charmap.Windows1255,
+	enc2256: charmap.Windows1256,
+	enc2257: charmap.Windows1257,
+	enc2258: charmap.Windows1258,
+	enc18:   japanese.EUCJP,
+	enc39:   japanese.ISO2022JP,
+	enc17:   japanese.ShiftJIS,
+	enc38:   korean.EUCKR,
+	enc114:  simplifiedchinese.GB18030,
+	enc113:  simplifiedchinese.GBK,
+	enc2085: simplifiedchinese.HZGB2312,
+	enc2026: traditionalchinese.Big5,
+}

vendor/golang.org/x/text/encoding/internal/enctest/enctest.go

@@ -0,0 +1,180 @@
+// Copyright 2017 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package enctest
+
+import (
+	"bytes"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"strings"
+	"testing"
+
+	"golang.org/x/text/encoding"
+	"golang.org/x/text/encoding/internal/identifier"
+	"golang.org/x/text/transform"
+)
+
+// Encoder or Decoder
+type Transcoder interface {
+	transform.Transformer
+	Bytes([]byte) ([]byte, error)
+	String(string) (string, error)
+}
+
+func TestEncoding(t *testing.T, e encoding.Encoding, encoded, utf8, prefix, suffix string) {
+	for _, direction := range []string{"Decode", "Encode"} {
+		t.Run(fmt.Sprintf("%v/%s", e, direction), func(t *testing.T) {
+
+			var coder Transcoder
+			var want, src, wPrefix, sPrefix, wSuffix, sSuffix string
+			if direction == "Decode" {
+				coder, want, src = e.NewDecoder(), utf8, encoded
+				wPrefix, sPrefix, wSuffix, sSuffix = "", prefix, "", suffix
+			} else {
+				coder, want, src = e.NewEncoder(), encoded, utf8
+				wPrefix, sPrefix, wSuffix, sSuffix = prefix, "", suffix, ""
+			}
+
+			dst := make([]byte, len(wPrefix)+len(want)+len(wSuffix))
+			nDst, nSrc, err := coder.Transform(dst, []byte(sPrefix+src+sSuffix), true)
+			if err != nil {
+				t.Fatal(err)
+			}
+			if nDst != len(wPrefix)+len(want)+len(wSuffix) {
+				t.Fatalf("nDst got %d, want %d",
+					nDst, len(wPrefix)+len(want)+len(wSuffix))
+			}
+			if nSrc != len(sPrefix)+len(src)+len(sSuffix) {
+				t.Fatalf("nSrc got %d, want %d",
+					nSrc, len(sPrefix)+len(src)+len(sSuffix))
+			}
+			if got := string(dst); got != wPrefix+want+wSuffix {
+				t.Fatalf("\ngot  %q\nwant %q", got, wPrefix+want+wSuffix)
+			}
+
+			for _, n := range []int{0, 1, 2, 10, 123, 4567} {
+				input := sPrefix + strings.Repeat(src, n) + sSuffix
+				g, err := coder.String(input)
+				if err != nil {
+					t.Fatalf("Bytes: n=%d: %v", n, err)
+				}
+				if len(g) == 0 && len(input) == 0 {
+					// If the input is empty then the output can be empty,
+					// regardless of whatever wPrefix is.
+					continue
+				}
+				got1, want1 := string(g), wPrefix+strings.Repeat(want, n)+wSuffix
+				if got1 != want1 {
+					t.Fatalf("ReadAll: n=%d\ngot  %q\nwant %q",
+						n, trim(got1), trim(want1))
+				}
+			}
+		})
+	}
+}
+
+func TestFile(t *testing.T, e encoding.Encoding) {
+	for _, dir := range []string{"Decode", "Encode"} {
+		t.Run(fmt.Sprintf("%s/%s", e, dir), func(t *testing.T) {
+			dst, src, transformer, err := load(dir, e)
+			if err != nil {
+				t.Fatalf("load: %v", err)
+			}
+			buf, err := transformer.Bytes(src)
+			if err != nil {
+				t.Fatalf("transform: %v", err)
+			}
+			if !bytes.Equal(buf, dst) {
+				t.Error("transformed bytes did not match golden file")
+			}
+		})
+	}
+}
+
+func Benchmark(b *testing.B, enc encoding.Encoding) {
+	for _, direction := range []string{"Decode", "Encode"} {
+		b.Run(fmt.Sprintf("%s/%s", enc, direction), func(b *testing.B) {
+			_, src, transformer, err := load(direction, enc)
+			if err != nil {
+				b.Fatal(err)
+			}
+			b.SetBytes(int64(len(src)))
+			b.ResetTimer()
+			for i := 0; i < b.N; i++ {
+				r := transform.NewReader(bytes.NewReader(src), transformer)
+				io.Copy(ioutil.Discard, r)
+			}
+		})
+	}
+}
+
+// testdataFiles are files in testdata/*.txt.
+var testdataFiles = []struct {
+	mib           identifier.MIB
+	basename, ext string
+}{
+	{identifier.Windows1252, "candide", "windows-1252"},
+	{identifier.EUCPkdFmtJapanese, "rashomon", "euc-jp"},
+	{identifier.ISO2022JP, "rashomon", "iso-2022-jp"},
+	{identifier.ShiftJIS, "rashomon", "shift-jis"},
+	{identifier.EUCKR, "unsu-joh-eun-nal", "euc-kr"},
+	{identifier.GBK, "sunzi-bingfa-simplified", "gbk"},
+	{identifier.HZGB2312, "sunzi-bingfa-gb-levels-1-and-2", "hz-gb2312"},
+	{identifier.Big5, "sunzi-bingfa-traditional", "big5"},
+	{identifier.UTF16LE, "candide", "utf-16le"},
+	{identifier.UTF8, "candide", "utf-8"},
+	{identifier.UTF32BE, "candide", "utf-32be"},
+
+	// GB18030 is a superset of GBK and is nominally a Simplified Chinese
+	// encoding, but it can also represent the entire Basic Multilingual
+	// Plane, including codepoints like 'â' that aren't encodable by GBK.
+	// GB18030 on Simplified Chinese should perform similarly to GBK on
+	// Simplified Chinese. GB18030 on "candide" is more interesting.
+	{identifier.GB18030, "candide", "gb18030"},
+}
+
+func load(direction string, enc encoding.Encoding) ([]byte, []byte, Transcoder, error) {
+	basename, ext, count := "", "", 0
+	for _, tf := range testdataFiles {
+		if mib, _ := enc.(identifier.Interface).ID(); tf.mib == mib {
+			basename, ext = tf.basename, tf.ext
+			count++
+		}
+	}
+	if count != 1 {
+		if count == 0 {
+			return nil, nil, nil, fmt.Errorf("no testdataFiles for %s", enc)
+		}
+		return nil, nil, nil, fmt.Errorf("too many testdataFiles for %s", enc)
+	}
+	dstFile := fmt.Sprintf("../testdata/%s-%s.txt", basename, ext)
+	srcFile := fmt.Sprintf("../testdata/%s-utf-8.txt", basename)
+	var coder Transcoder = encoding.ReplaceUnsupported(enc.NewEncoder())
+	if direction == "Decode" {
+		dstFile, srcFile = srcFile, dstFile
+		coder = enc.NewDecoder()
+	}
+	dst, err := ioutil.ReadFile(dstFile)
+	if err != nil {
+		if dst, err = ioutil.ReadFile("../" + dstFile); err != nil {
+			return nil, nil, nil, err
+		}
+	}
+	src, err := ioutil.ReadFile(srcFile)
+	if err != nil {
+		if src, err = ioutil.ReadFile("../" + srcFile); err != nil {
+			return nil, nil, nil, err
+		}
+	}
+	return dst, src, coder, nil
+}
+
+func trim(s string) string {
+	if len(s) < 120 {
+		return s
+	}
+	return s[:50] + "..." + s[len(s)-50:]
+}

vendor/golang.org/x/text/encoding/internal/identifier/mib.go

@@ -1,4 +1,4 @@
-// This file was generated by go generate; DO NOT EDIT
+// Code generated by running "go generate" in golang.org/x/text. DO NOT EDIT.
 
 package identifier
 

vendor/golang.org/x/text/encoding/japanese/eucjp.go

@@ -5,7 +5,6 @@
 package japanese
 
 import (
-	"errors"
 	"unicode/utf8"
 
 	"golang.org/x/text/encoding"
@@ -23,10 +22,9 @@ var eucJP = internal.Encoding{
 	identifier.EUCPkdFmtJapanese,
 }
 
-var errInvalidEUCJP = errors.New("japanese: invalid EUC-JP encoding")
-
 type eucJPDecoder struct{ transform.NopResetter }
 
+// See https://encoding.spec.whatwg.org/#euc-jp-decoder.
 func (eucJPDecoder) Transform(dst, src []byte, atEOF bool) (nDst, nSrc int, err error) {
 	r, size := rune(0), 0
 loop:
@@ -37,60 +35,79 @@ loop:
 
 		case c0 == 0x8e:
 			if nSrc+1 >= len(src) {
-				err = transform.ErrShortSrc
-				break loop
+				if !atEOF {
+					err = transform.ErrShortSrc
+					break loop
+				}
+				r, size = utf8.RuneError, 1
+				break
 			}
 			c1 := src[nSrc+1]
-			if c1 < 0xa1 || 0xdf < c1 {
-				err = errInvalidEUCJP
-				break loop
+			switch {
+			case c1 < 0xa1:
+				r, size = utf8.RuneError, 1
+			case c1 > 0xdf:
+				r, size = utf8.RuneError, 2
+				if c1 == 0xff {
+					size = 1
+				}
+			default:
+				r, size = rune(c1)+(0xff61-0xa1), 2
 			}
-			r, size = rune(c1)+(0xff61-0xa1), 2
-
 		case c0 == 0x8f:
 			if nSrc+2 >= len(src) {
-				err = transform.ErrShortSrc
-				break loop
+				if !atEOF {
+					err = transform.ErrShortSrc
+					break loop
+				}
+				r, size = utf8.RuneError, 1
+				if p := nSrc + 1; p < len(src) && 0xa1 <= src[p] && src[p] < 0xfe {
+					size = 2
+				}
+				break
 			}
 			c1 := src[nSrc+1]
 			if c1 < 0xa1 || 0xfe < c1 {
-				err = errInvalidEUCJP
-				break loop
+				r, size = utf8.RuneError, 1
+				break
 			}
 			c2 := src[nSrc+2]
 			if c2 < 0xa1 || 0xfe < c2 {
-				err = errInvalidEUCJP
-				break loop
+				r, size = utf8.RuneError, 2
+				break
 			}
-			r, size = '\ufffd', 3
+			r, size = utf8.RuneError, 3
 			if i := int(c1-0xa1)*94 + int(c2-0xa1); i < len(jis0212Decode) {
 				r = rune(jis0212Decode[i])
 				if r == 0 {
-					r = '\ufffd'
+					r = utf8.RuneError
 				}
 			}
 
 		case 0xa1 <= c0 && c0 <= 0xfe:
 			if nSrc+1 >= len(src) {
-				err = transform.ErrShortSrc
-				break loop
+				if !atEOF {
+					err = transform.ErrShortSrc
+					break loop
+				}
+				r, size = utf8.RuneError, 1
+				break
 			}
 			c1 := src[nSrc+1]
 			if c1 < 0xa1 || 0xfe < c1 {
-				err = errInvalidEUCJP
-				break loop
+				r, size = utf8.RuneError, 1
+				break
 			}
-			r, size = '\ufffd', 2
+			r, size = utf8.RuneError, 2
 			if i := int(c0-0xa1)*94 + int(c1-0xa1); i < len(jis0208Decode) {
 				r = rune(jis0208Decode[i])
 				if r == 0 {
-					r = '\ufffd'
+					r = utf8.RuneError
 				}
 			}
 
 		default:
-			err = errInvalidEUCJP
-			break loop
+			r, size = utf8.RuneError, 1
 		}
 
 		if nDst+utf8.RuneLen(r) > len(dst) {
@@ -99,9 +116,6 @@ loop:
 		}
 		nDst += utf8.EncodeRune(dst[nDst:], r)
 	}
-	if atEOF && err == transform.ErrShortSrc {
-		err = errInvalidEUCJP
-	}
 	return nDst, nSrc, err
 }
 

vendor/golang.org/x/text/encoding/japanese/iso2022jp.go

@@ -5,7 +5,6 @@
 package japanese
 
 import (
-	"errors"
 	"unicode/utf8"
 
 	"golang.org/x/text/encoding"
@@ -31,8 +30,6 @@ func iso2022JPNewEncoder() transform.Transformer {
 	return new(iso2022JPEncoder)
 }
 
-var errInvalidISO2022JP = errors.New("japanese: invalid ISO-2022-JP encoding")
-
 const (
 	asciiState = iota
 	katakanaState
@@ -50,45 +47,51 @@ func (d *iso2022JPDecoder) Reset() {
 
 func (d *iso2022JPDecoder) Transform(dst, src []byte, atEOF bool) (nDst, nSrc int, err error) {
 	r, size := rune(0), 0
-loop:
 	for ; nSrc < len(src); nSrc += size {
 		c0 := src[nSrc]
 		if c0 >= utf8.RuneSelf {
-			err = errInvalidISO2022JP
-			break loop
+			r, size = '\ufffd', 1
+			goto write
 		}
 
 		if c0 == asciiEsc {
 			if nSrc+2 >= len(src) {
-				err = transform.ErrShortSrc
-				break loop
+				if !atEOF {
+					return nDst, nSrc, transform.ErrShortSrc
+				}
+				// TODO: is it correct to only skip 1??
+				r, size = '\ufffd', 1
+				goto write
 			}
 			size = 3
 			c1 := src[nSrc+1]
 			c2 := src[nSrc+2]
 			switch {
-			case c1 == '$' && (c2 == '@' || c2 == 'B'):
+			case c1 == '$' && (c2 == '@' || c2 == 'B'): // 0x24 {0x40, 0x42}
 				*d = jis0208State
 				continue
-			case c1 == '$' && c2 == '(':
+			case c1 == '$' && c2 == '(': // 0x24 0x28
 				if nSrc+3 >= len(src) {
-					err = transform.ErrShortSrc
-					break loop
+					if !atEOF {
+						return nDst, nSrc, transform.ErrShortSrc
+					}
+					r, size = '\ufffd', 1
+					goto write
 				}
 				size = 4
-				if src[nSrc]+3 == 'D' {
+				if src[nSrc+3] == 'D' {
 					*d = jis0212State
 					continue
 				}
-			case c1 == '(' && (c2 == 'B' || c2 == 'J'):
+			case c1 == '(' && (c2 == 'B' || c2 == 'J'): // 0x28 {0x42, 0x4A}
 				*d = asciiState
 				continue
-			case c1 == '(' && c2 == 'I':
+			case c1 == '(' && c2 == 'I': // 0x28 0x49
 				*d = katakanaState
 				continue
 			}
-			err = errInvalidISO2022JP
-			break loop
+			r, size = '\ufffd', 1
+			goto write
 		}
 
 		switch *d {
@@ -97,8 +100,8 @@ loop:
 
 		case katakanaState:
 			if c0 < 0x21 || 0x60 <= c0 {
-				err = errInvalidISO2022JP
-				break loop
+				r, size = '\ufffd', 1
+				goto write
 			}
 			r, size = rune(c0)+(0xff61-0x21), 1
 
@@ -106,11 +109,14 @@ loop:
 			if c0 == 0x0a {
 				*d = asciiState
 				r, size = rune(c0), 1
-				break
+				goto write
 			}
 			if nSrc+1 >= len(src) {
-				err = transform.ErrShortSrc
-				break loop
+				if !atEOF {
+					return nDst, nSrc, transform.ErrShortSrc
+				}
+				r, size = '\ufffd', 1
+				goto write
 			}
 			size = 2
 			c1 := src[nSrc+1]
@@ -121,22 +127,19 @@ loop:
 				r = rune(jis0212Decode[i])
 			} else {
 				r = '\ufffd'
-				break
+				goto write
 			}
 			if r == 0 {
 				r = '\ufffd'
 			}
 		}
 
+	write:
 		if nDst+utf8.RuneLen(r) > len(dst) {
-			err = transform.ErrShortDst
-			break loop
+			return nDst, nSrc, transform.ErrShortDst
 		}
 		nDst += utf8.EncodeRune(dst[nDst:], r)
 	}
-	if atEOF && err == transform.ErrShortSrc {
-		err = errInvalidISO2022JP
-	}
 	return nDst, nSrc, err
 }
 

vendor/golang.org/x/text/encoding/japanese/shiftjis.go

@@ -5,7 +5,6 @@
 package japanese
 
 import (
-	"errors"
 	"unicode/utf8"
 
 	"golang.org/x/text/encoding"
@@ -24,8 +23,6 @@ var shiftJIS = internal.Encoding{
 	identifier.ShiftJIS,
 }
 
-var errInvalidShiftJIS = errors.New("japanese: invalid Shift JIS encoding")
-
 type shiftJISDecoder struct{ transform.NopResetter }
 
 func (shiftJISDecoder) Transform(dst, src []byte, atEOF bool) (nDst, nSrc int, err error) {
@@ -48,28 +45,32 @@ loop:
 			c0 = 2*c0 - 0x21
 
 			if nSrc+1 >= len(src) {
-				err = transform.ErrShortSrc
-				break loop
+				if !atEOF {
+					err = transform.ErrShortSrc
+					break loop
+				}
+				r, size = '\ufffd', 1
+				goto write
 			}
 			c1 := src[nSrc+1]
 			switch {
 			case c1 < 0x40:
-				err = errInvalidShiftJIS
-				break loop
+				r, size = '\ufffd', 1 // c1 is ASCII so output on next round
+				goto write
 			case c1 < 0x7f:
 				c0--
 				c1 -= 0x40
 			case c1 == 0x7f:
-				err = errInvalidShiftJIS
-				break loop
+				r, size = '\ufffd', 1 // c1 is ASCII so output on next round
+				goto write
 			case c1 < 0x9f:
 				c0--
 				c1 -= 0x41
 			case c1 < 0xfd:
 				c1 -= 0x9f
 			default:
-				err = errInvalidShiftJIS
-				break loop
+				r, size = '\ufffd', 2
+				goto write
 			}
 			r, size = '\ufffd', 2
 			if i := int(c0)*94 + int(c1); i < len(jis0208Decode) {
@@ -79,20 +80,19 @@ loop:
 				}
 			}
 
-		default:
-			err = errInvalidShiftJIS
-			break loop
-		}
+		case c0 == 0x80:
+			r, size = 0x80, 1
 
+		default:
+			r, size = '\ufffd', 1
+		}
+	write:
 		if nDst+utf8.RuneLen(r) > len(dst) {
 			err = transform.ErrShortDst
 			break loop
 		}
 		nDst += utf8.EncodeRune(dst[nDst:], r)
 	}
-	if atEOF && err == transform.ErrShortSrc {
-		err = errInvalidShiftJIS
-	}
 	return nDst, nSrc, err
 }
 

vendor/golang.org/x/text/encoding/korean/euckr.go

@@ -5,7 +5,6 @@
 package korean
 
 import (
-	"errors"
 	"unicode/utf8"
 
 	"golang.org/x/text/encoding"
@@ -26,8 +25,6 @@ var eucKR = internal.Encoding{
 	identifier.EUCKR,
 }
 
-var errInvalidEUCKR = errors.New("korean: invalid EUC-KR encoding")
-
 type eucKRDecoder struct{ transform.NopResetter }
 
 func (eucKRDecoder) Transform(dst, src []byte, atEOF bool) (nDst, nSrc int, err error) {
@@ -40,10 +37,15 @@ loop:
 
 		case 0x81 <= c0 && c0 < 0xff:
 			if nSrc+1 >= len(src) {
-				err = transform.ErrShortSrc
-				break loop
+				if !atEOF {
+					err = transform.ErrShortSrc
+					break loop
+				}
+				r, size = utf8.RuneError, 1
+				break
 			}
 			c1 := src[nSrc+1]
+			size = 2
 			if c0 < 0xc7 {
 				r = 178 * rune(c0-0x81)
 				switch {
@@ -54,39 +56,36 @@ loop:
 				case 0x81 <= c1 && c1 < 0xff:
 					r += rune(c1) - (0x81 - 2*26)
 				default:
-					err = errInvalidEUCKR
-					break loop
+					goto decError
 				}
 			} else if 0xa1 <= c1 && c1 < 0xff {
 				r = 178*(0xc7-0x81) + rune(c0-0xc7)*94 + rune(c1-0xa1)
 			} else {
-				err = errInvalidEUCKR
-				break loop
+				goto decError
 			}
 			if int(r) < len(decode) {
 				r = rune(decode[r])
-				if r == 0 {
-					r = '\ufffd'
+				if r != 0 {
+					break
 				}
-			} else {
-				r = '\ufffd'
 			}
-			size = 2
+		decError:
+			r = utf8.RuneError
+			if c1 < utf8.RuneSelf {
+				size = 1
+			}
 
 		default:
-			err = errInvalidEUCKR
-			break loop
+			r, size = utf8.RuneError, 1
+			break
 		}
 
 		if nDst+utf8.RuneLen(r) > len(dst) {
 			err = transform.ErrShortDst
-			break loop
+			break
 		}
 		nDst += utf8.EncodeRune(dst[nDst:], r)
 	}
-	if atEOF && err == transform.ErrShortSrc {
-		err = errInvalidEUCKR
-	}
 	return nDst, nSrc, err
 }
 

vendor/golang.org/x/text/encoding/simplifiedchinese/gbk.go

@@ -5,7 +5,6 @@
 package simplifiedchinese
 
 import (
-	"errors"
 	"unicode/utf8"
 
 	"golang.org/x/text/encoding"
@@ -40,11 +39,6 @@ var gbk18030 = internal.Encoding{
 	identifier.GB18030,
 }
 
-var (
-	errInvalidGB18030 = errors.New("simplifiedchinese: invalid GB18030 encoding")
-	errInvalidGBK     = errors.New("simplifiedchinese: invalid GBK encoding")
-)
-
 type gbkDecoder struct {
 	transform.NopResetter
 	gb18030 bool
@@ -66,8 +60,12 @@ loop:
 
 		case c0 < 0xff:
 			if nSrc+1 >= len(src) {
-				err = transform.ErrShortSrc
-				break loop
+				if !atEOF {
+					err = transform.ErrShortSrc
+					break loop
+				}
+				r, size = utf8.RuneError, 1
+				goto write
 			}
 			c1 := src[nSrc+1]
 			switch {
@@ -77,18 +75,24 @@ loop:
 				c1 -= 0x41
 			case d.gb18030 && 0x30 <= c1 && c1 < 0x40:
 				if nSrc+3 >= len(src) {
-					err = transform.ErrShortSrc
-					break loop
+					if !atEOF {
+						err = transform.ErrShortSrc
+						break loop
+					}
+					// The second byte here is always ASCII, so we can set size
+					// to 1 in all cases.
+					r, size = utf8.RuneError, 1
+					goto write
 				}
 				c2 := src[nSrc+2]
 				if c2 < 0x81 || 0xff <= c2 {
-					err = errInvalidGB18030
-					break loop
+					r, size = utf8.RuneError, 1
+					goto write
 				}
 				c3 := src[nSrc+3]
 				if c3 < 0x30 || 0x3a <= c3 {
-					err = errInvalidGB18030
-					break loop
+					r, size = utf8.RuneError, 1
+					goto write
 				}
 				size = 4
 				r = ((rune(c0-0x81)*10+rune(c1-0x30))*126+rune(c2-0x81))*10 + rune(c3-0x30)
@@ -109,17 +113,13 @@ loop:
 				r -= 189000
 				if 0 <= r && r < 0x100000 {
 					r += 0x10000
-					goto write
-				}
-				err = errInvalidGB18030
-				break loop
-			default:
-				if d.gb18030 {
-					err = errInvalidGB18030
 				} else {
-					err = errInvalidGBK
+					r, size = utf8.RuneError, 1
 				}
-				break loop
+				goto write
+			default:
+				r, size = utf8.RuneError, 1
+				goto write
 			}
 			r, size = '\ufffd', 2
 			if i := int(c0-0x81)*190 + int(c1); i < len(decode) {
@@ -130,12 +130,7 @@ loop:
 			}
 
 		default:
-			if d.gb18030 {
-				err = errInvalidGB18030
-			} else {
-				err = errInvalidGBK
-			}
-			break loop
+			r, size = utf8.RuneError, 1
 		}
 
 	write:
@@ -145,13 +140,6 @@ loop:
 		}
 		nDst += utf8.EncodeRune(dst[nDst:], r)
 	}
-	if atEOF && err == transform.ErrShortSrc {
-		if d.gb18030 {
-			err = errInvalidGB18030
-		} else {
-			err = errInvalidGBK
-		}
-	}
 	return nDst, nSrc, err
 }
 

vendor/golang.org/x/text/encoding/simplifiedchinese/hzgb2312.go

@@ -5,7 +5,6 @@
 package simplifiedchinese
 
 import (
-	"errors"
 	"unicode/utf8"
 
 	"golang.org/x/text/encoding"
@@ -31,8 +30,6 @@ func hzGB2312NewEncoder() transform.Transformer {
 	return new(hzGB2312Encoder)
 }
 
-var errInvalidHZGB2312 = errors.New("simplifiedchinese: invalid HZ-GB2312 encoding")
-
 const (
 	asciiState = iota
 	gbState
@@ -50,14 +47,18 @@ loop:
 	for ; nSrc < len(src); nSrc += size {
 		c0 := src[nSrc]
 		if c0 >= utf8.RuneSelf {
-			err = errInvalidHZGB2312
-			break loop
+			r, size = utf8.RuneError, 1
+			goto write
 		}
 
 		if c0 == '~' {
 			if nSrc+1 >= len(src) {
-				err = transform.ErrShortSrc
-				break loop
+				if !atEOF {
+					err = transform.ErrShortSrc
+					break loop
+				}
+				r = utf8.RuneError
+				goto write
 			}
 			size = 2
 			switch src[nSrc+1] {
@@ -78,8 +79,8 @@ loop:
 			case '\n':
 				continue
 			default:
-				err = errInvalidHZGB2312
-				break loop
+				r = utf8.RuneError
+				goto write
 			}
 		}
 
@@ -87,33 +88,37 @@ loop:
 			r, size = rune(c0), 1
 		} else {
 			if nSrc+1 >= len(src) {
-				err = transform.ErrShortSrc
-				break loop
+				if !atEOF {
+					err = transform.ErrShortSrc
+					break loop
+				}
+				r, size = utf8.RuneError, 1
+				goto write
 			}
+			size = 2
 			c1 := src[nSrc+1]
 			if c0 < 0x21 || 0x7e <= c0 || c1 < 0x21 || 0x7f <= c1 {
-				err = errInvalidHZGB2312
-				break loop
-			}
-
-			r, size = '\ufffd', 2
-			if i := int(c0-0x01)*190 + int(c1+0x3f); i < len(decode) {
+				// error
+			} else if i := int(c0-0x01)*190 + int(c1+0x3f); i < len(decode) {
 				r = rune(decode[i])
-				if r == 0 {
-					r = '\ufffd'
+				if r != 0 {
+					goto write
 				}
 			}
+			if c1 > utf8.RuneSelf {
+				// Be consistent and always treat non-ASCII as a single error.
+				size = 1
+			}
+			r = utf8.RuneError
 		}
 
+	write:
 		if nDst+utf8.RuneLen(r) > len(dst) {
 			err = transform.ErrShortDst
 			break loop
 		}
 		nDst += utf8.EncodeRune(dst[nDst:], r)
 	}
-	if atEOF && err == transform.ErrShortSrc {
-		err = errInvalidHZGB2312
-	}
 	return nDst, nSrc, err
 }
 

vendor/golang.org/x/text/encoding/traditionalchinese/big5.go

@@ -5,7 +5,6 @@
 package traditionalchinese
 
 import (
-	"errors"
 	"unicode/utf8"
 
 	"golang.org/x/text/encoding"
@@ -26,8 +25,6 @@ var big5 = internal.Encoding{
 	identifier.Big5,
 }
 
-var errInvalidBig5 = errors.New("traditionalchinese: invalid Big5 encoding")
-
 type big5Decoder struct{ transform.NopResetter }
 
 func (big5Decoder) Transform(dst, src []byte, atEOF bool) (nDst, nSrc int, err error) {
@@ -40,8 +37,12 @@ loop:
 
 		case 0x81 <= c0 && c0 < 0xff:
 			if nSrc+1 >= len(src) {
-				err = transform.ErrShortSrc
-				break loop
+				if !atEOF {
+					err = transform.ErrShortSrc
+					break loop
+				}
+				r, size = utf8.RuneError, 1
+				goto write
 			}
 			c1 := src[nSrc+1]
 			switch {
@@ -49,9 +50,12 @@ loop:
 				c1 -= 0x40
 			case 0xa1 <= c1 && c1 < 0xff:
 				c1 -= 0x62
+			case c1 < 0x40:
+				r, size = utf8.RuneError, 1
+				goto write
 			default:
-				err = errInvalidBig5
-				break loop
+				r, size = utf8.RuneError, 2
+				goto write
 			}
 			r, size = '\ufffd', 2
 			if i := int(c0-0x81)*157 + int(c1); i < len(decode) {
@@ -80,10 +84,10 @@ loop:
 			}
 
 		default:
-			err = errInvalidBig5
-			break loop
+			r, size = utf8.RuneError, 1
 		}
 
+	write:
 		if nDst+utf8.RuneLen(r) > len(dst) {
 			err = transform.ErrShortDst
 			break loop
@@ -99,9 +103,6 @@ loop:
 		nDst += copy(dst[nDst:], s)
 		continue loop
 	}
-	if atEOF && err == transform.ErrShortSrc {
-		err = errInvalidBig5
-	}
 	return nDst, nSrc, err
 }
 

vendor/golang.org/x/text/internal/format/LICENSE

@@ -0,0 +1,27 @@
+Copyright (c) 2009 The Go Authors. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+   * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+   * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+   * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

vendor/golang.org/x/text/internal/format/format.go

@@ -0,0 +1,41 @@
+// Copyright 2015 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+// Package format contains types for defining language-specific formatting of
+// values.
+//
+// This package is internal now, but will eventually be exposed after the API
+// settles.
+package format // import "golang.org/x/text/internal/format"
+
+import (
+	"fmt"
+
+	"golang.org/x/text/language"
+)
+
+// State represents the printer state passed to custom formatters. It provides
+// access to the fmt.State interface and the sentence and language-related
+// context.
+type State interface {
+	fmt.State
+
+	// Language reports the requested language in which to render a message.
+	Language() language.Tag
+
+	// TODO: consider this and removing rune from the Format method in the
+	// Formatter interface.
+	//
+	// Verb returns the format variant to render, analogous to the types used
+	// in fmt. Use 'v' for the default or only variant.
+	// Verb() rune
+
+	// TODO: more info:
+	// - sentence context such as linguistic features passed by the translator.
+}
+
+// Formatter is analogous to fmt.Formatter.
+type Formatter interface {
+	Format(state State, verb rune)
+}

vendor/golang.org/x/text/internal/format/parser.go

@@ -0,0 +1,358 @@
+// Copyright 2017 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package format
+
+import (
+	"reflect"
+	"unicode/utf8"
+)
+
+// A Parser parses a format string. The result from the parse are set in the
+// struct fields.
+type Parser struct {
+	Verb rune
+
+	WidthPresent bool
+	PrecPresent  bool
+	Minus        bool
+	Plus         bool
+	Sharp        bool
+	Space        bool
+	Zero         bool
+
+	// For the formats %+v %#v, we set the plusV/sharpV flags
+	// and clear the plus/sharp flags since %+v and %#v are in effect
+	// different, flagless formats set at the top level.
+	PlusV  bool
+	SharpV bool
+
+	HasIndex bool
+
+	Width int
+	Prec  int // precision
+
+	// retain arguments across calls.
+	Args []interface{}
+	// retain current argument number across calls
+	ArgNum int
+
+	// reordered records whether the format string used argument reordering.
+	Reordered bool
+	// goodArgNum records whether the most recent reordering directive was valid.
+	goodArgNum bool
+
+	// position info
+	format   string
+	startPos int
+	endPos   int
+	Status   Status
+}
+
+// Reset initializes a parser to scan format strings for the given args.
+func (p *Parser) Reset(args []interface{}) {
+	p.Args = args
+	p.ArgNum = 0
+	p.startPos = 0
+	p.Reordered = false
+}
+
+// Text returns the part of the format string that was parsed by the last call
+// to Scan. It returns the original substitution clause if the current scan
+// parsed a substitution.
+func (p *Parser) Text() string { return p.format[p.startPos:p.endPos] }
+
+// SetFormat sets a new format string to parse. It does not reset the argument
+// count.
+func (p *Parser) SetFormat(format string) {
+	p.format = format
+	p.startPos = 0
+	p.endPos = 0
+}
+
+// Status indicates the result type of a call to Scan.
+type Status int
+
+const (
+	StatusText Status = iota
+	StatusSubstitution
+	StatusBadWidthSubstitution
+	StatusBadPrecSubstitution
+	StatusNoVerb
+	StatusBadArgNum
+	StatusMissingArg
+)
+
+// ClearFlags reset the parser to default behavior.
+func (p *Parser) ClearFlags() {
+	p.WidthPresent = false
+	p.PrecPresent = false
+	p.Minus = false
+	p.Plus = false
+	p.Sharp = false
+	p.Space = false
+	p.Zero = false
+
+	p.PlusV = false
+	p.SharpV = false
+
+	p.HasIndex = false
+}
+
+// Scan scans the next part of the format string and sets the status to
+// indicate whether it scanned a string literal, substitution or error.
+func (p *Parser) Scan() bool {
+	p.Status = StatusText
+	format := p.format
+	end := len(format)
+	if p.endPos >= end {
+		return false
+	}
+	afterIndex := false // previous item in format was an index like [3].
+
+	p.startPos = p.endPos
+	p.goodArgNum = true
+	i := p.startPos
+	for i < end && format[i] != '%' {
+		i++
+	}
+	if i > p.startPos {
+		p.endPos = i
+		return true
+	}
+	// Process one verb
+	i++
+
+	p.Status = StatusSubstitution
+
+	// Do we have flags?
+	p.ClearFlags()
+
+simpleFormat:
+	for ; i < end; i++ {
+		c := p.format[i]
+		switch c {
+		case '#':
+			p.Sharp = true
+		case '0':
+			p.Zero = !p.Minus // Only allow zero padding to the left.
+		case '+':
+			p.Plus = true
+		case '-':
+			p.Minus = true
+			p.Zero = false // Do not pad with zeros to the right.
+		case ' ':
+			p.Space = true
+		default:
+			// Fast path for common case of ascii lower case simple verbs
+			// without precision or width or argument indices.
+			if 'a' <= c && c <= 'z' && p.ArgNum < len(p.Args) {
+				if c == 'v' {
+					// Go syntax
+					p.SharpV = p.Sharp
+					p.Sharp = false
+					// Struct-field syntax
+					p.PlusV = p.Plus
+					p.Plus = false
+				}
+				p.Verb = rune(c)
+				p.ArgNum++
+				p.endPos = i + 1
+				return true
+			}
+			// Format is more complex than simple flags and a verb or is malformed.
+			break simpleFormat
+		}
+	}
+
+	// Do we have an explicit argument index?
+	i, afterIndex = p.updateArgNumber(format, i)
+
+	// Do we have width?
+	if i < end && format[i] == '*' {
+		i++
+		p.Width, p.WidthPresent = p.intFromArg()
+
+		if !p.WidthPresent {
+			p.Status = StatusBadWidthSubstitution
+		}
+
+		// We have a negative width, so take its value and ensure
+		// that the minus flag is set
+		if p.Width < 0 {
+			p.Width = -p.Width
+			p.Minus = true
+			p.Zero = false // Do not pad with zeros to the right.
+		}
+		afterIndex = false
+	} else {
+		p.Width, p.WidthPresent, i = parsenum(format, i, end)
+		if afterIndex && p.WidthPresent { // "%[3]2d"
+			p.goodArgNum = false
+		}
+	}
+
+	// Do we have precision?
+	if i+1 < end && format[i] == '.' {
+		i++
+		if afterIndex { // "%[3].2d"
+			p.goodArgNum = false
+		}
+		i, afterIndex = p.updateArgNumber(format, i)
+		if i < end && format[i] == '*' {
+			i++
+			p.Prec, p.PrecPresent = p.intFromArg()
+			// Negative precision arguments don't make sense
+			if p.Prec < 0 {
+				p.Prec = 0
+				p.PrecPresent = false
+			}
+			if !p.PrecPresent {
+				p.Status = StatusBadPrecSubstitution
+			}
+			afterIndex = false
+		} else {
+			p.Prec, p.PrecPresent, i = parsenum(format, i, end)
+			if !p.PrecPresent {
+				p.Prec = 0
+				p.PrecPresent = true
+			}
+		}
+	}
+
+	if !afterIndex {
+		i, afterIndex = p.updateArgNumber(format, i)
+	}
+	p.HasIndex = afterIndex
+
+	if i >= end {
+		p.endPos = i
+		p.Status = StatusNoVerb
+		return true
+	}
+
+	verb, w := utf8.DecodeRuneInString(format[i:])
+	p.endPos = i + w
+	p.Verb = verb
+
+	switch {
+	case verb == '%': // Percent does not absorb operands and ignores f.wid and f.prec.
+		p.startPos = p.endPos - 1
+		p.Status = StatusText
+	case !p.goodArgNum:
+		p.Status = StatusBadArgNum
+	case p.ArgNum >= len(p.Args): // No argument left over to print for the current verb.
+		p.Status = StatusMissingArg
+		p.ArgNum++
+	case verb == 'v':
+		// Go syntax
+		p.SharpV = p.Sharp
+		p.Sharp = false
+		// Struct-field syntax
+		p.PlusV = p.Plus
+		p.Plus = false
+		fallthrough
+	default:
+		p.ArgNum++
+	}
+	return true
+}
+
+// intFromArg gets the ArgNumth element of Args. On return, isInt reports
+// whether the argument has integer type.
+func (p *Parser) intFromArg() (num int, isInt bool) {
+	if p.ArgNum < len(p.Args) {
+		arg := p.Args[p.ArgNum]
+		num, isInt = arg.(int) // Almost always OK.
+		if !isInt {
+			// Work harder.
+			switch v := reflect.ValueOf(arg); v.Kind() {
+			case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
+				n := v.Int()
+				if int64(int(n)) == n {
+					num = int(n)
+					isInt = true
+				}
+			case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
+				n := v.Uint()
+				if int64(n) >= 0 && uint64(int(n)) == n {
+					num = int(n)
+					isInt = true
+				}
+			default:
+				// Already 0, false.
+			}
+		}
+		p.ArgNum++
+		if tooLarge(num) {
+			num = 0
+			isInt = false
+		}
+	}
+	return
+}
+
+// parseArgNumber returns the value of the bracketed number, minus 1
+// (explicit argument numbers are one-indexed but we want zero-indexed).
+// The opening bracket is known to be present at format[0].
+// The returned values are the index, the number of bytes to consume
+// up to the closing paren, if present, and whether the number parsed
+// ok. The bytes to consume will be 1 if no closing paren is present.
+func parseArgNumber(format string) (index int, wid int, ok bool) {
+	// There must be at least 3 bytes: [n].
+	if len(format) < 3 {
+		return 0, 1, false
+	}
+
+	// Find closing bracket.
+	for i := 1; i < len(format); i++ {
+		if format[i] == ']' {
+			width, ok, newi := parsenum(format, 1, i)
+			if !ok || newi != i {
+				return 0, i + 1, false
+			}
+			return width - 1, i + 1, true // arg numbers are one-indexed and skip paren.
+		}
+	}
+	return 0, 1, false
+}
+
+// updateArgNumber returns the next argument to evaluate, which is either the value of the passed-in
+// argNum or the value of the bracketed integer that begins format[i:]. It also returns
+// the new value of i, that is, the index of the next byte of the format to process.
+func (p *Parser) updateArgNumber(format string, i int) (newi int, found bool) {
+	if len(format) <= i || format[i] != '[' {
+		return i, false
+	}
+	p.Reordered = true
+	index, wid, ok := parseArgNumber(format[i:])
+	if ok && 0 <= index && index < len(p.Args) {
+		p.ArgNum = index
+		return i + wid, true
+	}
+	p.goodArgNum = false
+	return i + wid, ok
+}
+
+// tooLarge reports whether the magnitude of the integer is
+// too large to be used as a formatting width or precision.
+func tooLarge(x int) bool {
+	const max int = 1e6
+	return x > max || x < -max
+}
+
+// parsenum converts ASCII to integer.  num is 0 (and isnum is false) if no number present.
+func parsenum(s string, start, end int) (num int, isnum bool, newi int) {
+	if start >= end {
+		return 0, false, end
+	}
+	for newi = start; newi < end && '0' <= s[newi] && s[newi] <= '9'; newi++ {
+		if tooLarge(num) {
+			return 0, false, end // Overflow; crazy long number most likely.
+		}
+		num = num*10 + int(s[newi]-'0')
+		isnum = true
+	}
+	return
+}

vendor/golang.org/x/text/internal/language/LICENSE

@@ -0,0 +1,27 @@
+Copyright (c) 2009 The Go Authors. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+   * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+   * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+   * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

vendor/golang.org/x/text/internal/language/common.go

@@ -0,0 +1,16 @@
+// Code generated by running "go generate" in golang.org/x/text. DO NOT EDIT.
+
+package language
+
+// This file contains code common to the maketables.go and the package code.
+
+// AliasType is the type of an alias in AliasMap.
+type AliasType int8
+
+const (
+	Deprecated AliasType = iota
+	Macro
+	Legacy
+
+	AliasTypeUnknown AliasType = -1
+)

vendor/golang.org/x/text/internal/language/compact.go

@@ -0,0 +1,29 @@
+// Copyright 2018 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package language
+
+// CompactCoreInfo is a compact integer with the three core tags encoded.
+type CompactCoreInfo uint32
+
+// GetCompactCore generates a uint32 value that is guaranteed to be unique for
+// different language, region, and script values.
+func GetCompactCore(t Tag) (cci CompactCoreInfo, ok bool) {
+	if t.LangID > langNoIndexOffset {
+		return 0, false
+	}
+	cci |= CompactCoreInfo(t.LangID) << (8 + 12)
+	cci |= CompactCoreInfo(t.ScriptID) << 12
+	cci |= CompactCoreInfo(t.RegionID)
+	return cci, true
+}
+
+// Tag generates a tag from c.
+func (c CompactCoreInfo) Tag() Tag {
+	return Tag{
+		LangID:   Language(c >> 20),
+		RegionID: Region(c & 0x3ff),
+		ScriptID: Script(c>>12) & 0xff,
+	}
+}

vendor/golang.org/x/text/internal/language/compact/compact.go

@@ -0,0 +1,61 @@
+// Copyright 2018 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+// Package compact defines a compact representation of language tags.
+//
+// Common language tags (at least all for which locale information is defined
+// in CLDR) are assigned a unique index. Each Tag is associated with such an
+// ID for selecting language-related resources (such as translations) as well
+// as one for selecting regional defaults (currency, number formatting, etc.)
+//
+// It may want to export this functionality at some point, but at this point
+// this is only available for use within x/text.
+package compact // import "golang.org/x/text/internal/language/compact"
+
+import (
+	"sort"
+	"strings"
+
+	"golang.org/x/text/internal/language"
+)
+
+// ID is an integer identifying a single tag.
+type ID uint16
+
+func getCoreIndex(t language.Tag) (id ID, ok bool) {
+	cci, ok := language.GetCompactCore(t)
+	if !ok {
+		return 0, false
+	}
+	i := sort.Search(len(coreTags), func(i int) bool {
+		return cci <= coreTags[i]
+	})
+	if i == len(coreTags) || coreTags[i] != cci {
+		return 0, false
+	}
+	return ID(i), true
+}
+
+// Parent returns the ID of the parent or the root ID if id is already the root.
+func (id ID) Parent() ID {
+	return parents[id]
+}
+
+// Tag converts id to an internal language Tag.
+func (id ID) Tag() language.Tag {
+	if int(id) >= len(coreTags) {
+		return specialTags[int(id)-len(coreTags)]
+	}
+	return coreTags[id].Tag()
+}
+
+var specialTags []language.Tag
+
+func init() {
+	tags := strings.Split(specialTagsStr, " ")
+	specialTags = make([]language.Tag, len(tags))
+	for i, t := range tags {
+		specialTags[i] = language.MustParse(t)
+	}
+}

vendor/golang.org/x/text/internal/language/compact/gen.go

@@ -0,0 +1,64 @@
+// Copyright 2013 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+// +build ignore
+
+// Language tag table generator.
+// Data read from the web.
+
+package main
+
+import (
+	"flag"
+	"fmt"
+	"log"
+
+	"golang.org/x/text/internal/gen"
+	"golang.org/x/text/unicode/cldr"
+)
+
+var (
+	test = flag.Bool("test",
+		false,
+		"test existing tables; can be used to compare web data with package data.")
+	outputFile = flag.String("output",
+		"tables.go",
+		"output file for generated tables")
+)
+
+func main() {
+	gen.Init()
+
+	w := gen.NewCodeWriter()
+	defer w.WriteGoFile("tables.go", "compact")
+
+	fmt.Fprintln(w, `import "golang.org/x/text/internal/language"`)
+
+	b := newBuilder(w)
+	gen.WriteCLDRVersion(w)
+
+	b.writeCompactIndex()
+}
+
+type builder struct {
+	w    *gen.CodeWriter
+	data *cldr.CLDR
+	supp *cldr.SupplementalData
+}
+
+func newBuilder(w *gen.CodeWriter) *builder {
+	r := gen.OpenCLDRCoreZip()
+	defer r.Close()
+	d := &cldr.Decoder{}
+	data, err := d.DecodeZip(r)
+	if err != nil {
+		log.Fatal(err)
+	}
+	b := builder{
+		w:    w,
+		data: data,
+		supp: data.Supplemental(),
+	}
+	return &b
+}

vendor/golang.org/x/text/internal/language/compact/gen_index.go

@@ -0,0 +1,113 @@
+// Copyright 2015 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+// +build ignore
+
+package main
+
+// This file generates derivative tables based on the language package itself.
+
+import (
+	"fmt"
+	"log"
+	"sort"
+	"strings"
+
+	"golang.org/x/text/internal/language"
+)
+
+// Compact indices:
+// Note -va-X variants only apply to localization variants.
+// BCP variants only ever apply to language.
+// The only ambiguity between tags is with regions.
+
+func (b *builder) writeCompactIndex() {
+	// Collect all language tags for which we have any data in CLDR.
+	m := map[language.Tag]bool{}
+	for _, lang := range b.data.Locales() {
+		// We include all locales unconditionally to be consistent with en_US.
+		// We want en_US, even though it has no data associated with it.
+
+		// TODO: put any of the languages for which no data exists at the end
+		// of the index. This allows all components based on ICU to use that
+		// as the cutoff point.
+		// if x := data.RawLDML(lang); false ||
+		// 	x.LocaleDisplayNames != nil ||
+		// 	x.Characters != nil ||
+		// 	x.Delimiters != nil ||
+		// 	x.Measurement != nil ||
+		// 	x.Dates != nil ||
+		// 	x.Numbers != nil ||
+		// 	x.Units != nil ||
+		// 	x.ListPatterns != nil ||
+		// 	x.Collations != nil ||
+		// 	x.Segmentations != nil ||
+		// 	x.Rbnf != nil ||
+		// 	x.Annotations != nil ||
+		// 	x.Metadata != nil {
+
+		// TODO: support POSIX natively, albeit non-standard.
+		tag := language.Make(strings.Replace(lang, "_POSIX", "-u-va-posix", 1))
+		m[tag] = true
+		// }
+	}
+
+	// TODO: plural rules are also defined for the deprecated tags:
+	//    iw mo sh tl
+	// Consider removing these as compact tags.
+
+	// Include locales for plural rules, which uses a different structure.
+	for _, plurals := range b.supp.Plurals {
+		for _, rules := range plurals.PluralRules {
+			for _, lang := range strings.Split(rules.Locales, " ") {
+				m[language.Make(lang)] = true
+			}
+		}
+	}
+
+	var coreTags []language.CompactCoreInfo
+	var special []string
+
+	for t := range m {
+		if x := t.Extensions(); len(x) != 0 && fmt.Sprint(x) != "[u-va-posix]" {
+			log.Fatalf("Unexpected extension %v in %v", x, t)
+		}
+		if len(t.Variants()) == 0 && len(t.Extensions()) == 0 {
+			cci, ok := language.GetCompactCore(t)
+			if !ok {
+				log.Fatalf("Locale for non-basic language %q", t)
+			}
+			coreTags = append(coreTags, cci)
+		} else {
+			special = append(special, t.String())
+		}
+	}
+
+	w := b.w
+
+	sort.Slice(coreTags, func(i, j int) bool { return coreTags[i] < coreTags[j] })
+	sort.Strings(special)
+
+	w.WriteComment(`
+	NumCompactTags is the number of common tags. The maximum tag is
+	NumCompactTags-1.`)
+	w.WriteConst("NumCompactTags", len(m))
+
+	fmt.Fprintln(w, "const (")
+	for i, t := range coreTags {
+		fmt.Fprintf(w, "%s ID = %d\n", ident(t.Tag().String()), i)
+	}
+	for i, t := range special {
+		fmt.Fprintf(w, "%s ID = %d\n", ident(t), i+len(coreTags))
+	}
+	fmt.Fprintln(w, ")")
+
+	w.WriteVar("coreTags", coreTags)
+
+	w.WriteConst("specialTagsStr", strings.Join(special, " "))
+}
+
+func ident(s string) string {
+	return strings.Replace(s, "-", "", -1) + "Index"
+}

vendor/golang.org/x/text/internal/language/compact/gen_parents.go

@@ -0,0 +1,54 @@
+// Copyright 2018 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+// +build ignore
+
+package main
+
+import (
+	"log"
+
+	"golang.org/x/text/internal/gen"
+	"golang.org/x/text/internal/language"
+	"golang.org/x/text/internal/language/compact"
+	"golang.org/x/text/unicode/cldr"
+)
+
+func main() {
+	r := gen.OpenCLDRCoreZip()
+	defer r.Close()
+
+	d := &cldr.Decoder{}
+	data, err := d.DecodeZip(r)
+	if err != nil {
+		log.Fatalf("DecodeZip: %v", err)
+	}
+
+	w := gen.NewCodeWriter()
+	defer w.WriteGoFile("parents.go", "compact")
+
+	// Create parents table.
+	type ID uint16
+	parents := make([]ID, compact.NumCompactTags)
+	for _, loc := range data.Locales() {
+		tag := language.MustParse(loc)
+		index, ok := compact.FromTag(tag)
+		if !ok {
+			continue
+		}
+		parentIndex := compact.ID(0) // und
+		for p := tag.Parent(); p != language.Und; p = p.Parent() {
+			if x, ok := compact.FromTag(p); ok {
+				parentIndex = x
+				break
+			}
+		}
+		parents[index] = ID(parentIndex)
+	}
+
+	w.WriteComment(`
+	parents maps a compact index of a tag to the compact index of the parent of
+	this tag.`)
+	w.WriteVar("parents", parents)
+}

vendor/golang.org/x/text/internal/language/compact/language.go

@@ -0,0 +1,260 @@
+// Copyright 2013 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+//go:generate go run gen.go gen_index.go -output tables.go
+//go:generate go run gen_parents.go
+
+package compact
+
+// TODO: Remove above NOTE after:
+// - verifying that tables are dropped correctly (most notably matcher tables).
+
+import (
+	"strings"
+
+	"golang.org/x/text/internal/language"
+)
+
+// Tag represents a BCP 47 language tag. It is used to specify an instance of a
+// specific language or locale. All language tag values are guaranteed to be
+// well-formed.
+type Tag struct {
+	// NOTE: exported tags will become part of the public API.
+	language ID
+	locale   ID
+	full     fullTag // always a language.Tag for now.
+}
+
+const _und = 0
+
+type fullTag interface {
+	IsRoot() bool
+	Parent() language.Tag
+}
+
+// Make a compact Tag from a fully specified internal language Tag.
+func Make(t language.Tag) (tag Tag) {
+	if region := t.TypeForKey("rg"); len(region) == 6 && region[2:] == "zzzz" {
+		if r, err := language.ParseRegion(region[:2]); err == nil {
+			tFull := t
+			t, _ = t.SetTypeForKey("rg", "")
+			// TODO: should we not consider "va" for the language tag?
+			var exact1, exact2 bool
+			tag.language, exact1 = FromTag(t)
+			t.RegionID = r
+			tag.locale, exact2 = FromTag(t)
+			if !exact1 || !exact2 {
+				tag.full = tFull
+			}
+			return tag
+		}
+	}
+	lang, ok := FromTag(t)
+	tag.language = lang
+	tag.locale = lang
+	if !ok {
+		tag.full = t
+	}
+	return tag
+}
+
+// Tag returns an internal language Tag version of this tag.
+func (t Tag) Tag() language.Tag {
+	if t.full != nil {
+		return t.full.(language.Tag)
+	}
+	tag := t.language.Tag()
+	if t.language != t.locale {
+		loc := t.locale.Tag()
+		tag, _ = tag.SetTypeForKey("rg", strings.ToLower(loc.RegionID.String())+"zzzz")
+	}
+	return tag
+}
+
+// IsCompact reports whether this tag is fully defined in terms of ID.
+func (t *Tag) IsCompact() bool {
+	return t.full == nil
+}
+
+// MayHaveVariants reports whether a tag may have variants. If it returns false
+// it is guaranteed the tag does not have variants.
+func (t Tag) MayHaveVariants() bool {
+	return t.full != nil || int(t.language) >= len(coreTags)
+}
+
+// MayHaveExtensions reports whether a tag may have extensions. If it returns
+// false it is guaranteed the tag does not have them.
+func (t Tag) MayHaveExtensions() bool {
+	return t.full != nil ||
+		int(t.language) >= len(coreTags) ||
+		t.language != t.locale
+}
+
+// IsRoot returns true if t is equal to language "und".
+func (t Tag) IsRoot() bool {
+	if t.full != nil {
+		return t.full.IsRoot()
+	}
+	return t.language == _und
+}
+
+// Parent returns the CLDR parent of t. In CLDR, missing fields in data for a
+// specific language are substituted with fields from the parent language.
+// The parent for a language may change for newer versions of CLDR.
+func (t Tag) Parent() Tag {
+	if t.full != nil {
+		return Make(t.full.Parent())
+	}
+	if t.language != t.locale {
+		// Simulate stripping -u-rg-xxxxxx
+		return Tag{language: t.language, locale: t.language}
+	}
+	// TODO: use parent lookup table once cycle from internal package is
+	// removed. Probably by internalizing the table and declaring this fast
+	// enough.
+	// lang := compactID(internal.Parent(uint16(t.language)))
+	lang, _ := FromTag(t.language.Tag().Parent())
+	return Tag{language: lang, locale: lang}
+}
+
+// returns token t and the rest of the string.
+func nextToken(s string) (t, tail string) {
+	p := strings.Index(s[1:], "-")
+	if p == -1 {
+		return s[1:], ""
+	}
+	p++
+	return s[1:p], s[p:]
+}
+
+// LanguageID returns an index, where 0 <= index < NumCompactTags, for tags
+// for which data exists in the text repository.The index will change over time
+// and should not be stored in persistent storage. If t does not match a compact
+// index, exact will be false and the compact index will be returned for the
+// first match after repeatedly taking the Parent of t.
+func LanguageID(t Tag) (id ID, exact bool) {
+	return t.language, t.full == nil
+}
+
+// RegionalID returns the ID for the regional variant of this tag. This index is
+// used to indicate region-specific overrides, such as default currency, default
+// calendar and week data, default time cycle, and default measurement system
+// and unit preferences.
+//
+// For instance, the tag en-GB-u-rg-uszzzz specifies British English with US
+// settings for currency, number formatting, etc. The CompactIndex for this tag
+// will be that for en-GB, while the RegionalID will be the one corresponding to
+// en-US.
+func RegionalID(t Tag) (id ID, exact bool) {
+	return t.locale, t.full == nil
+}
+
+// LanguageTag returns t stripped of regional variant indicators.
+//
+// At the moment this means it is stripped of a regional and variant subtag "rg"
+// and "va" in the "u" extension.
+func (t Tag) LanguageTag() Tag {
+	if t.full == nil {
+		return Tag{language: t.language, locale: t.language}
+	}
+	tt := t.Tag()
+	tt.SetTypeForKey("rg", "")
+	tt.SetTypeForKey("va", "")
+	return Make(tt)
+}
+
+// RegionalTag returns the regional variant of the tag.
+//
+// At the moment this means that the region is set from the regional subtag
+// "rg" in the "u" extension.
+func (t Tag) RegionalTag() Tag {
+	rt := Tag{language: t.locale, locale: t.locale}
+	if t.full == nil {
+		return rt
+	}
+	b := language.Builder{}
+	tag := t.Tag()
+	// tag, _ = tag.SetTypeForKey("rg", "")
+	b.SetTag(t.locale.Tag())
+	if v := tag.Variants(); v != "" {
+		for _, v := range strings.Split(v, "-") {
+			b.AddVariant(v)
+		}
+	}
+	for _, e := range tag.Extensions() {
+		b.AddExt(e)
+	}
+	return t
+}
+
+// FromTag reports closest matching ID for an internal language Tag.
+func FromTag(t language.Tag) (id ID, exact bool) {
+	// TODO: perhaps give more frequent tags a lower index.
+	// TODO: we could make the indexes stable. This will excluded some
+	//       possibilities for optimization, so don't do this quite yet.
+	exact = true
+
+	b, s, r := t.Raw()
+	if t.HasString() {
+		if t.IsPrivateUse() {
+			// We have no entries for user-defined tags.
+			return 0, false
+		}
+		hasExtra := false
+		if t.HasVariants() {
+			if t.HasExtensions() {
+				build := language.Builder{}
+				build.SetTag(language.Tag{LangID: b, ScriptID: s, RegionID: r})
+				build.AddVariant(t.Variants())
+				exact = false
+				t = build.Make()
+			}
+			hasExtra = true
+		} else if _, ok := t.Extension('u'); ok {
+			// TODO: va may mean something else. Consider not considering it.
+			// Strip all but the 'va' entry.
+			old := t
+			variant := t.TypeForKey("va")
+			t = language.Tag{LangID: b, ScriptID: s, RegionID: r}
+			if variant != "" {
+				t, _ = t.SetTypeForKey("va", variant)
+				hasExtra = true
+			}
+			exact = old == t
+		} else {
+			exact = false
+		}
+		if hasExtra {
+			// We have some variants.
+			for i, s := range specialTags {
+				if s == t {
+					return ID(i + len(coreTags)), exact
+				}
+			}
+			exact = false
+		}
+	}
+	if x, ok := getCoreIndex(t); ok {
+		return x, exact
+	}
+	exact = false
+	if r != 0 && s == 0 {
+		// Deal with cases where an extra script is inserted for the region.
+		t, _ := t.Maximize()
+		if x, ok := getCoreIndex(t); ok {
+			return x, exact
+		}
+	}
+	for t = t.Parent(); t != root; t = t.Parent() {
+		// No variants specified: just compare core components.
+		// The key has the form lllssrrr, where l, s, and r are nibbles for
+		// respectively the langID, scriptID, and regionID.
+		if x, ok := getCoreIndex(t); ok {
+			return x, exact
+		}
+	}
+	return 0, exact
+}
+
+var root = language.Tag{}

vendor/golang.org/x/text/internal/language/compact/parents.go

@@ -0,0 +1,120 @@
+// Code generated by running "go generate" in golang.org/x/text. DO NOT EDIT.
+
+package compact
+
+// parents maps a compact index of a tag to the compact index of the parent of
+// this tag.
+var parents = []ID{ // 775 elements
+	// Entry 0 - 3F
+	0x0000, 0x0000, 0x0001, 0x0001, 0x0000, 0x0004, 0x0000, 0x0006,
+	0x0000, 0x0008, 0x0000, 0x000a, 0x000a, 0x000a, 0x000a, 0x000a,
+	0x000a, 0x000a, 0x000a, 0x000a, 0x000a, 0x000a, 0x000a, 0x000a,
+	0x000a, 0x000a, 0x000a, 0x000a, 0x000a, 0x000a, 0x000a, 0x000a,
+	0x000a, 0x000a, 0x000a, 0x000a, 0x000a, 0x000a, 0x000a, 0x0000,
+	0x0000, 0x0028, 0x0000, 0x002a, 0x0000, 0x002c, 0x0000, 0x0000,
+	0x002f, 0x002e, 0x002e, 0x0000, 0x0033, 0x0000, 0x0035, 0x0000,
+	0x0037, 0x0000, 0x0039, 0x0000, 0x003b, 0x0000, 0x0000, 0x003e,
+	// Entry 40 - 7F
+	0x0000, 0x0040, 0x0040, 0x0000, 0x0043, 0x0043, 0x0000, 0x0046,
+	0x0000, 0x0048, 0x0000, 0x0000, 0x004b, 0x004a, 0x004a, 0x0000,
+	0x004f, 0x004f, 0x004f, 0x004f, 0x0000, 0x0054, 0x0054, 0x0000,
+	0x0057, 0x0000, 0x0059, 0x0000, 0x005b, 0x0000, 0x005d, 0x005d,
+	0x0000, 0x0060, 0x0000, 0x0062, 0x0000, 0x0064, 0x0000, 0x0066,
+	0x0066, 0x0000, 0x0069, 0x0000, 0x006b, 0x006b, 0x006b, 0x006b,
+	0x006b, 0x006b, 0x006b, 0x0000, 0x0073, 0x0000, 0x0075, 0x0000,
+	0x0077, 0x0000, 0x0000, 0x007a, 0x0000, 0x007c, 0x0000, 0x007e,
+	// Entry 80 - BF
+	0x0000, 0x0080, 0x0080, 0x0000, 0x0083, 0x0083, 0x0000, 0x0086,
+	0x0087, 0x0087, 0x0087, 0x0086, 0x0088, 0x0087, 0x0087, 0x0087,
+	0x0086, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0088,
+	0x0087, 0x0087, 0x0087, 0x0087, 0x0088, 0x0087, 0x0088, 0x0087,
+	0x0087, 0x0088, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087,
+	0x0087, 0x0087, 0x0087, 0x0086, 0x0087, 0x0087, 0x0087, 0x0087,
+	0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087,
+	0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0086, 0x0087, 0x0086,
+	// Entry C0 - FF
+	0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087,
+	0x0088, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087,
+	0x0086, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0088, 0x0087,
+	0x0087, 0x0088, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087,
+	0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0086, 0x0086, 0x0087,
+	0x0087, 0x0086, 0x0087, 0x0087, 0x0087, 0x0087, 0x0087, 0x0000,
+	0x00ef, 0x0000, 0x00f1, 0x00f2, 0x00f2, 0x00f2, 0x00f2, 0x00f2,
+	0x00f2, 0x00f2, 0x00f2, 0x00f2, 0x00f1, 0x00f2, 0x00f1, 0x00f1,
+	// Entry 100 - 13F
+	0x00f2, 0x00f2, 0x00f1, 0x00f2, 0x00f2, 0x00f2, 0x00f2, 0x00f1,
+	0x00f2, 0x00f2, 0x00f2, 0x00f2, 0x00f2, 0x00f2, 0x0000, 0x010e,
+	0x0000, 0x0110, 0x0000, 0x0112, 0x0000, 0x0114, 0x0114, 0x0000,
+	0x0117, 0x0117, 0x0117, 0x0117, 0x0000, 0x011c, 0x0000, 0x011e,
+	0x0000, 0x0120, 0x0120, 0x0000, 0x0123, 0x0123, 0x0123, 0x0123,
+	0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123,
+	0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123,
+	0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123,
+	// Entry 140 - 17F
+	0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123,
+	0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123, 0x0123,
+	0x0123, 0x0123, 0x0000, 0x0152, 0x0000, 0x0154, 0x0000, 0x0156,
+	0x0000, 0x0158, 0x0000, 0x015a, 0x0000, 0x015c, 0x015c, 0x015c,
+	0x0000, 0x0160, 0x0000, 0x0000, 0x0163, 0x0000, 0x0165, 0x0000,
+	0x0167, 0x0167, 0x0167, 0x0000, 0x016b, 0x0000, 0x016d, 0x0000,
+	0x016f, 0x0000, 0x0171, 0x0171, 0x0000, 0x0174, 0x0000, 0x0176,
+	0x0000, 0x0178, 0x0000, 0x017a, 0x0000, 0x017c, 0x0000, 0x017e,
+	// Entry 180 - 1BF
+	0x0000, 0x0000, 0x0000, 0x0182, 0x0000, 0x0184, 0x0184, 0x0184,
+	0x0184, 0x0000, 0x0000, 0x0000, 0x018b, 0x0000, 0x0000, 0x018e,
+	0x0000, 0x0000, 0x0191, 0x0000, 0x0000, 0x0000, 0x0195, 0x0000,
+	0x0197, 0x0000, 0x0000, 0x019a, 0x0000, 0x0000, 0x019d, 0x0000,
+	0x019f, 0x0000, 0x01a1, 0x0000, 0x01a3, 0x0000, 0x01a5, 0x0000,
+	0x01a7, 0x0000, 0x01a9, 0x0000, 0x01ab, 0x0000, 0x01ad, 0x0000,
+	0x01af, 0x0000, 0x01b1, 0x01b1, 0x0000, 0x01b4, 0x0000, 0x01b6,
+	0x0000, 0x01b8, 0x0000, 0x01ba, 0x0000, 0x01bc, 0x0000, 0x0000,
+	// Entry 1C0 - 1FF
+	0x01bf, 0x0000, 0x01c1, 0x0000, 0x01c3, 0x0000, 0x01c5, 0x0000,
+	0x01c7, 0x0000, 0x01c9, 0x0000, 0x01cb, 0x01cb, 0x01cb, 0x01cb,
+	0x0000, 0x01d0, 0x0000, 0x01d2, 0x01d2, 0x0000, 0x01d5, 0x0000,
+	0x01d7, 0x0000, 0x01d9, 0x0000, 0x01db, 0x0000, 0x01dd, 0x0000,
+	0x01df, 0x01df, 0x0000, 0x01e2, 0x0000, 0x01e4, 0x0000, 0x01e6,
+	0x0000, 0x01e8, 0x0000, 0x01ea, 0x0000, 0x01ec, 0x0000, 0x01ee,
+	0x0000, 0x01f0, 0x0000, 0x0000, 0x01f3, 0x0000, 0x01f5, 0x01f5,
+	0x01f5, 0x0000, 0x01f9, 0x0000, 0x01fb, 0x0000, 0x01fd, 0x0000,
+	// Entry 200 - 23F
+	0x01ff, 0x0000, 0x0000, 0x0202, 0x0000, 0x0204, 0x0204, 0x0000,
+	0x0207, 0x0000, 0x0209, 0x0209, 0x0000, 0x020c, 0x020c, 0x0000,
+	0x020f, 0x020f, 0x020f, 0x020f, 0x020f, 0x020f, 0x020f, 0x0000,
+	0x0217, 0x0000, 0x0219, 0x0000, 0x021b, 0x0000, 0x0000, 0x0000,
+	0x0000, 0x0000, 0x0221, 0x0000, 0x0000, 0x0224, 0x0000, 0x0226,
+	0x0226, 0x0000, 0x0229, 0x0000, 0x022b, 0x022b, 0x0000, 0x0000,
+	0x022f, 0x022e, 0x022e, 0x0000, 0x0000, 0x0234, 0x0000, 0x0236,
+	0x0000, 0x0238, 0x0000, 0x0244, 0x023a, 0x0244, 0x0244, 0x0244,
+	// Entry 240 - 27F
+	0x0244, 0x0244, 0x0244, 0x0244, 0x023a, 0x0244, 0x0244, 0x0000,
+	0x0247, 0x0247, 0x0247, 0x0000, 0x024b, 0x0000, 0x024d, 0x0000,
+	0x024f, 0x024f, 0x0000, 0x0252, 0x0000, 0x0254, 0x0254, 0x0254,
+	0x0254, 0x0254, 0x0254, 0x0000, 0x025b, 0x0000, 0x025d, 0x0000,
+	0x025f, 0x0000, 0x0261, 0x0000, 0x0263, 0x0000, 0x0265, 0x0000,
+	0x0000, 0x0268, 0x0268, 0x0268, 0x0000, 0x026c, 0x0000, 0x026e,
+	0x0000, 0x0270, 0x0000, 0x0000, 0x0000, 0x0274, 0x0273, 0x0273,
+	0x0000, 0x0278, 0x0000, 0x027a, 0x0000, 0x027c, 0x0000, 0x0000,
+	// Entry 280 - 2BF
+	0x0000, 0x0000, 0x0281, 0x0000, 0x0000, 0x0284, 0x0000, 0x0286,
+	0x0286, 0x0286, 0x0286, 0x0000, 0x028b, 0x028b, 0x028b, 0x0000,
+	0x028f, 0x028f, 0x028f, 0x028f, 0x028f, 0x0000, 0x0295, 0x0295,
+	0x0295, 0x0295, 0x0000, 0x0000, 0x0000, 0x0000, 0x029d, 0x029d,
+	0x029d, 0x0000, 0x02a1, 0x02a1, 0x02a1, 0x02a1, 0x0000, 0x0000,
+	0x02a7, 0x02a7, 0x02a7, 0x02a7, 0x0000, 0x02ac, 0x0000, 0x02ae,
+	0x02ae, 0x0000, 0x02b1, 0x0000, 0x02b3, 0x0000, 0x02b5, 0x02b5,
+	0x0000, 0x0000, 0x02b9, 0x0000, 0x0000, 0x0000, 0x02bd, 0x0000,
+	// Entry 2C0 - 2FF
+	0x02bf, 0x02bf, 0x0000, 0x0000, 0x02c3, 0x0000, 0x02c5, 0x0000,
+	0x02c7, 0x0000, 0x02c9, 0x0000, 0x02cb, 0x0000, 0x02cd, 0x02cd,
+	0x0000, 0x0000, 0x02d1, 0x0000, 0x02d3, 0x02d0, 0x02d0, 0x0000,
+	0x0000, 0x02d8, 0x02d7, 0x02d7, 0x0000, 0x0000, 0x02dd, 0x0000,
+	0x02df, 0x0000, 0x02e1, 0x0000, 0x0000, 0x02e4, 0x0000, 0x02e6,
+	0x0000, 0x0000, 0x02e9, 0x0000, 0x02eb, 0x0000, 0x02ed, 0x0000,
+	0x02ef, 0x02ef, 0x0000, 0x0000, 0x02f3, 0x02f2, 0x02f2, 0x0000,
+	0x02f7, 0x0000, 0x02f9, 0x02f9, 0x02f9, 0x02f9, 0x02f9, 0x0000,
+	// Entry 300 - 33F
+	0x02ff, 0x0300, 0x02ff, 0x0000, 0x0303, 0x0051, 0x00e6,
+} // Size: 1574 bytes
+
+// Total table size 1574 bytes (1KiB); checksum: 895AAF0B

vendor/golang.org/x/text/internal/language/compact/tags.go

@@ -0,0 +1,91 @@
+// Copyright 2013 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package compact
+
+var (
+	und = Tag{}
+
+	Und Tag = Tag{}
+
+	Afrikaans            Tag = Tag{language: afIndex, locale: afIndex}
+	Amharic              Tag = Tag{language: amIndex, locale: amIndex}
+	Arabic               Tag = Tag{language: arIndex, locale: arIndex}
+	ModernStandardArabic Tag = Tag{language: ar001Index, locale: ar001Index}
+	Azerbaijani          Tag = Tag{language: azIndex, locale: azIndex}
+	Bulgarian            Tag = Tag{language: bgIndex, locale: bgIndex}
+	Bengali              Tag = Tag{language: bnIndex, locale: bnIndex}
+	Catalan              Tag = Tag{language: caIndex, locale: caIndex}
+	Czech                Tag = Tag{language: csIndex, locale: csIndex}
+	Danish               Tag = Tag{language: daIndex, locale: daIndex}
+	German               Tag = Tag{language: deIndex, locale: deIndex}
+	Greek                Tag = Tag{language: elIndex, locale: elIndex}
+	English              Tag = Tag{language: enIndex, locale: enIndex}
+	AmericanEnglish      Tag = Tag{language: enUSIndex, locale: enUSIndex}
+	BritishEnglish       Tag = Tag{language: enGBIndex, locale: enGBIndex}
+	Spanish              Tag = Tag{language: esIndex, locale: esIndex}
+	EuropeanSpanish      Tag = Tag{language: esESIndex, locale: esESIndex}
+	LatinAmericanSpanish Tag = Tag{language: es419Index, locale: es419Index}
+	Estonian             Tag = Tag{language: etIndex, locale: etIndex}
+	Persian              Tag = Tag{language: faIndex, locale: faIndex}
+	Finnish              Tag = Tag{language: fiIndex, locale: fiIndex}
+	Filipino             Tag = Tag{language: filIndex, locale: filIndex}
+	French               Tag = Tag{language: frIndex, locale: frIndex}
+	CanadianFrench       Tag = Tag{language: frCAIndex, locale: frCAIndex}
+	Gujarati             Tag = Tag{language: guIndex, locale: guIndex}
+	Hebrew               Tag = Tag{language: heIndex, locale: heIndex}
+	Hindi                Tag = Tag{language: hiIndex, locale: hiIndex}
+	Croatian             Tag = Tag{language: hrIndex, locale: hrIndex}
+	Hungarian            Tag = Tag{language: huIndex, locale: huIndex}
+	Armenian             Tag = Tag{language: hyIndex, locale: hyIndex}
+	Indonesian           Tag = Tag{language: idIndex, locale: idIndex}
+	Icelandic            Tag = Tag{language: isIndex, locale: isIndex}
+	Italian              Tag = Tag{language: itIndex, locale: itIndex}
+	Japanese             Tag = Tag{language: jaIndex, locale: jaIndex}
+	Georgian             Tag = Tag{language: kaIndex, locale: kaIndex}
+	Kazakh               Tag = Tag{language: kkIndex, locale: kkIndex}
+	Khmer                Tag = Tag{language: kmIndex, locale: kmIndex}
+	Kannada              Tag = Tag{language: knIndex, locale: knIndex}
+	Korean               Tag = Tag{language: koIndex, locale: koIndex}
+	Kirghiz              Tag = Tag{language: kyIndex, locale: kyIndex}
+	Lao                  Tag = Tag{language: loIndex, locale: loIndex}
+	Lithuanian           Tag = Tag{language: ltIndex, locale: ltIndex}
+	Latvian              Tag = Tag{language: lvIndex, locale: lvIndex}
+	Macedonian           Tag = Tag{language: mkIndex, locale: mkIndex}
+	Malayalam            Tag = Tag{language: mlIndex, locale: mlIndex}
+	Mongolian            Tag = Tag{language: mnIndex, locale: mnIndex}
+	Marathi              Tag = Tag{language: mrIndex, locale: mrIndex}
+	Malay                Tag = Tag{language: msIndex, locale: msIndex}
+	Burmese              Tag = Tag{language: myIndex, locale: myIndex}
+	Nepali               Tag = Tag{language: neIndex, locale: neIndex}
+	Dutch                Tag = Tag{language: nlIndex, locale: nlIndex}
+	Norwegian            Tag = Tag{language: noIndex, locale: noIndex}
+	Punjabi              Tag = Tag{language: paIndex, locale: paIndex}
+	Polish               Tag = Tag{language: plIndex, locale: plIndex}
+	Portuguese           Tag = Tag{language: ptIndex, locale: ptIndex}
+	BrazilianPortuguese  Tag = Tag{language: ptBRIndex, locale: ptBRIndex}
+	EuropeanPortuguese   Tag = Tag{language: ptPTIndex, locale: ptPTIndex}
+	Romanian             Tag = Tag{language: roIndex, locale: roIndex}
+	Russian              Tag = Tag{language: ruIndex, locale: ruIndex}
+	Sinhala              Tag = Tag{language: siIndex, locale: siIndex}
+	Slovak               Tag = Tag{language: skIndex, locale: skIndex}
+	Slovenian            Tag = Tag{language: slIndex, locale: slIndex}
+	Albanian             Tag = Tag{language: sqIndex, locale: sqIndex}
+	Serbian              Tag = Tag{language: srIndex, locale: srIndex}
+	SerbianLatin         Tag = Tag{language: srLatnIndex, locale: srLatnIndex}
+	Swedish              Tag = Tag{language: svIndex, locale: svIndex}
+	Swahili              Tag = Tag{language: swIndex, locale: swIndex}
+	Tamil                Tag = Tag{language: taIndex, locale: taIndex}
+	Telugu               Tag = Tag{language: teIndex, locale: teIndex}
+	Thai                 Tag = Tag{language: thIndex, locale: thIndex}
+	Turkish              Tag = Tag{language: trIndex, locale: trIndex}
+	Ukrainian            Tag = Tag{language: ukIndex, locale: ukIndex}
+	Urdu                 Tag = Tag{language: urIndex, locale: urIndex}
+	Uzbek                Tag = Tag{language: uzIndex, locale: uzIndex}
+	Vietnamese           Tag = Tag{language: viIndex, locale: viIndex}
+	Chinese              Tag = Tag{language: zhIndex, locale: zhIndex}
+	SimplifiedChinese    Tag = Tag{language: zhHansIndex, locale: zhHansIndex}
+	TraditionalChinese   Tag = Tag{language: zhHantIndex, locale: zhHantIndex}
+	Zulu                 Tag = Tag{language: zuIndex, locale: zuIndex}
+)

vendor/golang.org/x/text/internal/language/compose.go

@@ -0,0 +1,167 @@
+// Copyright 2018 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package language
+
+import (
+	"sort"
+	"strings"
+)
+
+// A Builder allows constructing a Tag from individual components.
+// Its main user is Compose in the top-level language package.
+type Builder struct {
+	Tag Tag
+
+	private    string // the x extension
+	variants   []string
+	extensions []string
+}
+
+// Make returns a new Tag from the current settings.
+func (b *Builder) Make() Tag {
+	t := b.Tag
+
+	if len(b.extensions) > 0 || len(b.variants) > 0 {
+		sort.Sort(sortVariants(b.variants))
+		sort.Strings(b.extensions)
+
+		if b.private != "" {
+			b.extensions = append(b.extensions, b.private)
+		}
+		n := maxCoreSize + tokenLen(b.variants...) + tokenLen(b.extensions...)
+		buf := make([]byte, n)
+		p := t.genCoreBytes(buf)
+		t.pVariant = byte(p)
+		p += appendTokens(buf[p:], b.variants...)
+		t.pExt = uint16(p)
+		p += appendTokens(buf[p:], b.extensions...)
+		t.str = string(buf[:p])
+		// We may not always need to remake the string, but when or when not
+		// to do so is rather tricky.
+		scan := makeScanner(buf[:p])
+		t, _ = parse(&scan, "")
+		return t
+
+	} else if b.private != "" {
+		t.str = b.private
+		t.RemakeString()
+	}
+	return t
+}
+
+// SetTag copies all the settings from a given Tag. Any previously set values
+// are discarded.
+func (b *Builder) SetTag(t Tag) {
+	b.Tag.LangID = t.LangID
+	b.Tag.RegionID = t.RegionID
+	b.Tag.ScriptID = t.ScriptID
+	// TODO: optimize
+	b.variants = b.variants[:0]
+	if variants := t.Variants(); variants != "" {
+		for _, vr := range strings.Split(variants[1:], "-") {
+			b.variants = append(b.variants, vr)
+		}
+	}
+	b.extensions, b.private = b.extensions[:0], ""
+	for _, e := range t.Extensions() {
+		b.AddExt(e)
+	}
+}
+
+// AddExt adds extension e to the tag. e must be a valid extension as returned
+// by Tag.Extension. If the extension already exists, it will be discarded,
+// except for a -u extension, where non-existing key-type pairs will added.
+func (b *Builder) AddExt(e string) {
+	if e[0] == 'x' {
+		if b.private == "" {
+			b.private = e
+		}
+		return
+	}
+	for i, s := range b.extensions {
+		if s[0] == e[0] {
+			if e[0] == 'u' {
+				b.extensions[i] += e[1:]
+			}
+			return
+		}
+	}
+	b.extensions = append(b.extensions, e)
+}
+
+// SetExt sets the extension e to the tag. e must be a valid extension as
+// returned by Tag.Extension. If the extension already exists, it will be
+// overwritten, except for a -u extension, where the individual key-type pairs
+// will be set.
+func (b *Builder) SetExt(e string) {
+	if e[0] == 'x' {
+		b.private = e
+		return
+	}
+	for i, s := range b.extensions {
+		if s[0] == e[0] {
+			if e[0] == 'u' {
+				b.extensions[i] = e + s[1:]
+			} else {
+				b.extensions[i] = e
+			}
+			return
+		}
+	}
+	b.extensions = append(b.extensions, e)
+}
+
+// AddVariant adds any number of variants.
+func (b *Builder) AddVariant(v ...string) {
+	for _, v := range v {
+		if v != "" {
+			b.variants = append(b.variants, v)
+		}
+	}
+}
+
+// ClearVariants removes any variants previously added, including those
+// copied from a Tag in SetTag.
+func (b *Builder) ClearVariants() {
+	b.variants = b.variants[:0]
+}
+
+// ClearExtensions removes any extensions previously added, including those
+// copied from a Tag in SetTag.
+func (b *Builder) ClearExtensions() {
+	b.private = ""
+	b.extensions = b.extensions[:0]
+}
+
+func tokenLen(token ...string) (n int) {
+	for _, t := range token {
+		n += len(t) + 1
+	}
+	return
+}
+
+func appendTokens(b []byte, token ...string) int {
+	p := 0
+	for _, t := range token {
+		b[p] = '-'
+		copy(b[p+1:], t)
+		p += 1 + len(t)
+	}
+	return p
+}
+
+type sortVariants []string
+
+func (s sortVariants) Len() int {
+	return len(s)
+}
+
+func (s sortVariants) Swap(i, j int) {
+	s[j], s[i] = s[i], s[j]
+}
+
+func (s sortVariants) Less(i, j int) bool {
+	return variantIndex[s[i]] < variantIndex[s[j]]
+}

vendor/golang.org/x/text/internal/language/coverage.go

@@ -0,0 +1,28 @@
+// Copyright 2014 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package language
+
+// BaseLanguages returns the list of all supported base languages. It generates
+// the list by traversing the internal structures.
+func BaseLanguages() []Language {
+	base := make([]Language, 0, NumLanguages)
+	for i := 0; i < langNoIndexOffset; i++ {
+		// We included "und" already for the value 0.
+		if i != nonCanonicalUnd {
+			base = append(base, Language(i))
+		}
+	}
+	i := langNoIndexOffset
+	for _, v := range langNoIndex {
+		for k := 0; k < 8; k++ {
+			if v&1 == 1 {
+				base = append(base, Language(i))
+			}
+			v >>= 1
+			i++
+		}
+	}
+	return base
+}

vendor/golang.org/x/text/internal/language/gen.go

@@ -57,7 +57,7 @@ Each 3-letter code is followed by its 1-byte langID.`,
 	`
 altLangIndex is used to convert indexes in altLangISO3 to langIDs.`,
 	`
-langAliasMap maps langIDs to their suggested replacements.`,
+AliasMap maps langIDs to their suggested replacements.`,
 	`
 script is an alphabetically sorted list of ISO 15924 codes. The index
 of the script in the string, divided by 4, is the internal scriptID.`,
@@ -68,7 +68,7 @@ the UN.M49 codes used for groups.)`,
 	`
 regionISO holds a list of alphabetically sorted 2-letter ISO region codes.
 Each 2-letter codes is followed by two bytes with the following meaning:
-    - [A-Z}{2}: the first letter of the 2-letter code plus these two 
+    - [A-Z}{2}: the first letter of the 2-letter code plus these two
                 letters form the 3-letter ISO code.
     - 0, n:     index into altRegionISO3.`,
 	`
@@ -114,13 +114,6 @@ likelyRegionList holds lists info associated with likelyRegion.`,
 likelyScript is a lookup table, indexed by scriptID, for the most likely
 languages and regions given a script.`,
 	`
-matchLang holds pairs of langIDs of base languages that are typically
-mutually intelligible. Each pair is associated with a confidence and
-whether the intelligibility goes one or both ways.`,
-	`
-matchScript holds pairs of scriptIDs where readers of one script
-can typically also read the other. Each is associated with a confidence.`,
-	`
 nRegionGroups is the number of region groups.`,
 	`
 regionInclusion maps region identifiers to sets of regions in regionInclusionBits,
@@ -481,17 +474,17 @@ func (b *builder) writeSliceAddSize(name string, extraSize int, ss interface{})
 	b.p()
 }
 
-type fromTo struct {
-	from, to uint16
+type FromTo struct {
+	From, To uint16
 }
 
 func (b *builder) writeSortedMap(name string, ss *stringSet, index func(s string) uint16) {
 	ss.sortFunc(func(a, b string) bool {
 		return index(a) < index(b)
 	})
-	m := []fromTo{}
+	m := []FromTo{}
 	for _, s := range ss.s {
-		m = append(m, fromTo{index(s), index(ss.update[s])})
+		m = append(m, FromTo{index(s), index(ss.update[s])})
 	}
 	b.writeSlice(name, m)
 }
@@ -665,9 +658,9 @@ func (b *builder) parseIndices() {
 			b.langNoIndex.remove(s)
 		}
 	}
-	b.writeConst("numLanguages", len(b.lang.slice())+len(b.langNoIndex.slice()))
-	b.writeConst("numScripts", len(b.script.slice()))
-	b.writeConst("numRegions", len(b.region.slice()))
+	b.writeConst("NumLanguages", len(b.lang.slice())+len(b.langNoIndex.slice()))
+	b.writeConst("NumScripts", len(b.script.slice()))
+	b.writeConst("NumRegions", len(b.region.slice()))
 
 	// Add dummy codes at the start of each list to represent "unspecified".
 	b.lang.add("---")
@@ -698,8 +691,8 @@ func (b *builder) computeRegionGroups() {
 			b.groups[group] = index(len(b.groups))
 		}
 	}
-	if len(b.groups) > 32 {
-		log.Fatalf("only 32 groups supported, found %d", len(b.groups))
+	if len(b.groups) > 64 {
+		log.Fatalf("only 64 groups supported, found %d", len(b.groups))
 	}
 	b.writeConst("nRegionGroups", len(b.groups))
 }
@@ -729,7 +722,7 @@ func (b *builder) writeLanguage() {
 	// Get language codes that need to be mapped (overlong 3-letter codes,
 	// deprecated 2-letter codes, legacy and grandfathered tags.)
 	langAliasMap := stringSet{}
-	aliasTypeMap := map[string]langAliasType{}
+	aliasTypeMap := map[string]AliasType{}
 
 	// altLangISO3 get the alternative ISO3 names that need to be mapped.
 	altLangISO3 := stringSet{}
@@ -751,7 +744,7 @@ func (b *builder) writeLanguage() {
 		} else if len(a.Type) <= 3 {
 			switch a.Reason {
 			case "macrolanguage":
-				aliasTypeMap[a.Type] = langMacro
+				aliasTypeMap[a.Type] = Macro
 			case "deprecated":
 				// handled elsewhere
 				continue
@@ -759,7 +752,7 @@ func (b *builder) writeLanguage() {
 				if a.Type == "no" {
 					continue
 				}
-				aliasTypeMap[a.Type] = langLegacy
+				aliasTypeMap[a.Type] = Legacy
 			default:
 				log.Fatalf("new %s alias: %s", a.Reason, a.Type)
 			}
@@ -771,14 +764,14 @@ func (b *builder) writeLanguage() {
 	// This can be removed if CLDR adopts this change.
 	langAliasMap.add("nb")
 	langAliasMap.updateLater("nb", "no")
-	aliasTypeMap["nb"] = langMacro
+	aliasTypeMap["nb"] = Macro
 
 	for k, v := range b.registry {
 		// Also add deprecated values for 3-letter ISO codes, which CLDR omits.
 		if v.typ == "language" && v.deprecated != "" && v.preferred != "" {
 			langAliasMap.add(k)
 			langAliasMap.updateLater(k, v.preferred)
-			aliasTypeMap[k] = langDeprecated
+			aliasTypeMap[k] = Deprecated
 		}
 	}
 	// Fix CLDR mappings.
@@ -806,10 +799,10 @@ func (b *builder) writeLanguage() {
 		}
 	}
 
-	// Complete canonialized language tags.
+	// Complete canonicalized language tags.
 	lang.freeze()
 	for i, v := range lang.s {
-		// We can avoid these manual entries by using the IANI registry directly.
+		// We can avoid these manual entries by using the IANA registry directly.
 		// Seems easier to update the list manually, as changes are rare.
 		// The panic in this loop will trigger if we miss an entry.
 		add := ""
@@ -844,12 +837,12 @@ func (b *builder) writeLanguage() {
 	b.writeConst("altLangISO3", tag.Index(altLangISO3.join()))
 	b.writeSlice("altLangIndex", altLangIndex)
 
-	b.writeSortedMap("langAliasMap", &langAliasMap, b.langIndex)
-	types := make([]langAliasType, len(langAliasMap.s))
+	b.writeSortedMap("AliasMap", &langAliasMap, b.langIndex)
+	types := make([]AliasType, len(langAliasMap.s))
 	for i, s := range langAliasMap.s {
 		types[i] = aliasTypeMap[s]
 	}
-	b.writeSlice("langAliasTypes", types)
+	b.writeSlice("AliasTypes", types)
 }
 
 var scriptConsts = []string{
@@ -916,7 +909,7 @@ func (b *builder) writeRegion() {
 			i := b.region.index(s)
 			for _, d := range e.description {
 				if strings.Contains(d, "Private use") {
-					regionTypes[i] = iso3166UserAssgined
+					regionTypes[i] = iso3166UserAssigned
 				}
 			}
 			regionTypes[i] |= bcp47Region
@@ -1073,7 +1066,7 @@ const (
 )
 
 const (
-	iso3166UserAssgined = 1 << iota
+	iso3166UserAssigned = 1 << iota
 	ccTLD
 	bcp47Region
 )
@@ -1361,125 +1354,6 @@ func (b *builder) writeLikelyData() {
 	b.writeSlice("likelyRegionGroup", likelyRegionGroup)
 }
 
-type mutualIntelligibility struct {
-	want, have uint16
-	conf       uint8
-	oneway     bool
-}
-
-type scriptIntelligibility struct {
-	lang       uint16 // langID or 0 if *
-	want, have uint8
-	conf       uint8
-}
-
-type sortByConf []mutualIntelligibility
-
-func (l sortByConf) Less(a, b int) bool {
-	return l[a].conf > l[b].conf
-}
-
-func (l sortByConf) Swap(a, b int) {
-	l[a], l[b] = l[b], l[a]
-}
-
-func (l sortByConf) Len() int {
-	return len(l)
-}
-
-// toConf converts a percentage value [0, 100] to a confidence class.
-func toConf(pct uint8) uint8 {
-	switch {
-	case pct == 100:
-		return 3 // Exact
-	case pct >= 90:
-		return 2 // High
-	case pct > 50:
-		return 1 // Low
-	default:
-		return 0 // No
-	}
-}
-
-// writeMatchData writes tables with languages and scripts for which there is
-// mutual intelligibility. The data is based on CLDR's languageMatching data.
-// Note that we use a different algorithm than the one defined by CLDR and that
-// we slightly modify the data. For example, we convert scores to confidence levels.
-// We also drop all region-related data as we use a different algorithm to
-// determine region equivalence.
-func (b *builder) writeMatchData() {
-	b.writeType(mutualIntelligibility{})
-	b.writeType(scriptIntelligibility{})
-	lm := b.supp.LanguageMatching.LanguageMatches
-	cldr.MakeSlice(&lm).SelectAnyOf("type", "written")
-
-	matchLang := []mutualIntelligibility{}
-	matchScript := []scriptIntelligibility{}
-	// Convert the languageMatch entries in lists keyed by desired language.
-	for _, m := range lm[0].LanguageMatch {
-		// Different versions of CLDR use different separators.
-		desired := strings.Replace(m.Desired, "-", "_", -1)
-		supported := strings.Replace(m.Supported, "-", "_", -1)
-		d := strings.Split(desired, "_")
-		s := strings.Split(supported, "_")
-		if len(d) != len(s) || len(d) > 2 {
-			// Skip all entries with regions and work around CLDR bug.
-			continue
-		}
-		pct, _ := strconv.ParseInt(m.Percent, 10, 8)
-		if len(d) == 2 && d[0] == s[0] && len(d[1]) == 4 {
-			// language-script pair.
-			lang := uint16(0)
-			if d[0] != "*" {
-				lang = uint16(b.langIndex(d[0]))
-			}
-			matchScript = append(matchScript, scriptIntelligibility{
-				lang: lang,
-				want: uint8(b.script.index(d[1])),
-				have: uint8(b.script.index(s[1])),
-				conf: toConf(uint8(pct)),
-			})
-			if m.Oneway != "true" {
-				matchScript = append(matchScript, scriptIntelligibility{
-					lang: lang,
-					want: uint8(b.script.index(s[1])),
-					have: uint8(b.script.index(d[1])),
-					conf: toConf(uint8(pct)),
-				})
-			}
-		} else if len(d) == 1 && d[0] != "*" {
-			if pct == 100 {
-				// nb == no is already handled by macro mapping. Check there
-				// really is only this case.
-				if d[0] != "no" || s[0] != "nb" {
-					log.Fatalf("unhandled equivalence %s == %s", s[0], d[0])
-				}
-				continue
-			}
-			matchLang = append(matchLang, mutualIntelligibility{
-				want:   uint16(b.langIndex(d[0])),
-				have:   uint16(b.langIndex(s[0])),
-				conf:   uint8(pct),
-				oneway: m.Oneway == "true",
-			})
-		} else {
-			// TODO: Handle other mappings.
-			a := []string{"*;*", "*_*;*_*", "es_MX;es_419"}
-			s := strings.Join([]string{desired, supported}, ";")
-			if i := sort.SearchStrings(a, s); i == len(a) || a[i] != s {
-				log.Printf("%q not handled", s)
-			}
-		}
-	}
-	sort.Stable(sortByConf(matchLang))
-	// collapse percentage into confidence classes
-	for i, m := range matchLang {
-		matchLang[i].conf = toConf(m.conf)
-	}
-	b.writeSlice("matchLang", matchLang)
-	b.writeSlice("matchScript", matchScript)
-}
-
 func (b *builder) writeRegionInclusionData() {
 	var (
 		// mm holds for each group the set of groups with a distance of 1.
@@ -1507,7 +1381,7 @@ func (b *builder) writeRegionInclusionData() {
 		}
 	}
 
-	regionContainment := make([]uint32, len(b.groups))
+	regionContainment := make([]uint64, len(b.groups))
 	for _, g := range b.groups {
 		l := containment[g]
 
@@ -1521,15 +1395,14 @@ func (b *builder) writeRegionInclusionData() {
 		for _, v := range l {
 			regionContainment[g] |= 1 << v
 		}
-		// log.Printf("%d: %X", g, regionContainment[g])
 	}
 	b.writeSlice("regionContainment", regionContainment)
 
 	regionInclusion := make([]uint8, len(b.region.s))
-	bvs := make(map[uint32]index)
+	bvs := make(map[uint64]index)
 	// Make the first bitvector positions correspond with the groups.
 	for r, i := range b.groups {
-		bv := uint32(1 << i)
+		bv := uint64(1 << i)
 		for _, g := range mm[r] {
 			bv |= 1 << g
 		}
@@ -1538,7 +1411,7 @@ func (b *builder) writeRegionInclusionData() {
 	}
 	for r := 1; r < len(b.region.s); r++ {
 		if _, ok := b.groups[r]; !ok {
-			bv := uint32(0)
+			bv := uint64(0)
 			for _, g := range mm[r] {
 				bv |= 1 << g
 			}
@@ -1553,9 +1426,9 @@ func (b *builder) writeRegionInclusionData() {
 		}
 	}
 	b.writeSlice("regionInclusion", regionInclusion)
-	regionInclusionBits := make([]uint32, len(bvs))
+	regionInclusionBits := make([]uint64, len(bvs))
 	for k, v := range bvs {
-		regionInclusionBits[v] = uint32(k)
+		regionInclusionBits[v] = uint64(k)
 	}
 	// Add bit vectors for increasingly large distances until a fixed point is reached.
 	regionInclusionNext := []uint8{}
@@ -1634,7 +1507,7 @@ func main() {
 	gen.WriteCLDRVersion(w)
 
 	b.parseIndices()
-	b.writeType(fromTo{})
+	b.writeType(FromTo{})
 	b.writeLanguage()
 	b.writeScript()
 	b.writeRegion()
@@ -1642,7 +1515,6 @@ func main() {
 	// TODO: b.writeLocale()
 	b.computeRegionGroups()
 	b.writeLikelyData()
-	b.writeMatchData()
 	b.writeRegionInclusionData()
 	b.writeParents()
 }

vendor/golang.org/x/text/internal/language/gen_common.go

@@ -8,13 +8,13 @@ package main
 
 // This file contains code common to the maketables.go and the package code.
 
-// langAliasType is the type of an alias in langAliasMap.
-type langAliasType int8
+// AliasType is the type of an alias in AliasMap.
+type AliasType int8
 
 const (
-	langDeprecated langAliasType = iota
-	langMacro
-	langLegacy
+	Deprecated AliasType = iota
+	Macro
+	Legacy
 
-	langAliasTypeUnknown langAliasType = -1
+	AliasTypeUnknown AliasType = -1
 )

vendor/golang.org/x/text/internal/language/language.go

@@ -0,0 +1,596 @@
+// Copyright 2013 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+//go:generate go run gen.go gen_common.go -output tables.go
+
+package language // import "golang.org/x/text/internal/language"
+
+// TODO: Remove above NOTE after:
+// - verifying that tables are dropped correctly (most notably matcher tables).
+
+import (
+	"errors"
+	"fmt"
+	"strings"
+)
+
+const (
+	// maxCoreSize is the maximum size of a BCP 47 tag without variants and
+	// extensions. Equals max lang (3) + script (4) + max reg (3) + 2 dashes.
+	maxCoreSize = 12
+
+	// max99thPercentileSize is a somewhat arbitrary buffer size that presumably
+	// is large enough to hold at least 99% of the BCP 47 tags.
+	max99thPercentileSize = 32
+
+	// maxSimpleUExtensionSize is the maximum size of a -u extension with one
+	// key-type pair. Equals len("-u-") + key (2) + dash + max value (8).
+	maxSimpleUExtensionSize = 14
+)
+
+// Tag represents a BCP 47 language tag. It is used to specify an instance of a
+// specific language or locale. All language tag values are guaranteed to be
+// well-formed. The zero value of Tag is Und.
+type Tag struct {
+	// TODO: the following fields have the form TagTypeID. This name is chosen
+	// to allow refactoring the public package without conflicting with its
+	// Base, Script, and Region methods. Once the transition is fully completed
+	// the ID can be stripped from the name.
+
+	LangID   Language
+	RegionID Region
+	// TODO: we will soon run out of positions for ScriptID. Idea: instead of
+	// storing lang, region, and ScriptID codes, store only the compact index and
+	// have a lookup table from this code to its expansion. This greatly speeds
+	// up table lookup, speed up common variant cases.
+	// This will also immediately free up 3 extra bytes. Also, the pVariant
+	// field can now be moved to the lookup table, as the compact index uniquely
+	// determines the offset of a possible variant.
+	ScriptID Script
+	pVariant byte   // offset in str, includes preceding '-'
+	pExt     uint16 // offset of first extension, includes preceding '-'
+
+	// str is the string representation of the Tag. It will only be used if the
+	// tag has variants or extensions.
+	str string
+}
+
+// Make is a convenience wrapper for Parse that omits the error.
+// In case of an error, a sensible default is returned.
+func Make(s string) Tag {
+	t, _ := Parse(s)
+	return t
+}
+
+// Raw returns the raw base language, script and region, without making an
+// attempt to infer their values.
+// TODO: consider removing
+func (t Tag) Raw() (b Language, s Script, r Region) {
+	return t.LangID, t.ScriptID, t.RegionID
+}
+
+// equalTags compares language, script and region subtags only.
+func (t Tag) equalTags(a Tag) bool {
+	return t.LangID == a.LangID && t.ScriptID == a.ScriptID && t.RegionID == a.RegionID
+}
+
+// IsRoot returns true if t is equal to language "und".
+func (t Tag) IsRoot() bool {
+	if int(t.pVariant) < len(t.str) {
+		return false
+	}
+	return t.equalTags(Und)
+}
+
+// IsPrivateUse reports whether the Tag consists solely of an IsPrivateUse use
+// tag.
+func (t Tag) IsPrivateUse() bool {
+	return t.str != "" && t.pVariant == 0
+}
+
+// RemakeString is used to update t.str in case lang, script or region changed.
+// It is assumed that pExt and pVariant still point to the start of the
+// respective parts.
+func (t *Tag) RemakeString() {
+	if t.str == "" {
+		return
+	}
+	extra := t.str[t.pVariant:]
+	if t.pVariant > 0 {
+		extra = extra[1:]
+	}
+	if t.equalTags(Und) && strings.HasPrefix(extra, "x-") {
+		t.str = extra
+		t.pVariant = 0
+		t.pExt = 0
+		return
+	}
+	var buf [max99thPercentileSize]byte // avoid extra memory allocation in most cases.
+	b := buf[:t.genCoreBytes(buf[:])]
+	if extra != "" {
+		diff := len(b) - int(t.pVariant)
+		b = append(b, '-')
+		b = append(b, extra...)
+		t.pVariant = uint8(int(t.pVariant) + diff)
+		t.pExt = uint16(int(t.pExt) + diff)
+	} else {
+		t.pVariant = uint8(len(b))
+		t.pExt = uint16(len(b))
+	}
+	t.str = string(b)
+}
+
+// genCoreBytes writes a string for the base languages, script and region tags
+// to the given buffer and returns the number of bytes written. It will never
+// write more than maxCoreSize bytes.
+func (t *Tag) genCoreBytes(buf []byte) int {
+	n := t.LangID.StringToBuf(buf[:])
+	if t.ScriptID != 0 {
+		n += copy(buf[n:], "-")
+		n += copy(buf[n:], t.ScriptID.String())
+	}
+	if t.RegionID != 0 {
+		n += copy(buf[n:], "-")
+		n += copy(buf[n:], t.RegionID.String())
+	}
+	return n
+}
+
+// String returns the canonical string representation of the language tag.
+func (t Tag) String() string {
+	if t.str != "" {
+		return t.str
+	}
+	if t.ScriptID == 0 && t.RegionID == 0 {
+		return t.LangID.String()
+	}
+	buf := [maxCoreSize]byte{}
+	return string(buf[:t.genCoreBytes(buf[:])])
+}
+
+// MarshalText implements encoding.TextMarshaler.
+func (t Tag) MarshalText() (text []byte, err error) {
+	if t.str != "" {
+		text = append(text, t.str...)
+	} else if t.ScriptID == 0 && t.RegionID == 0 {
+		text = append(text, t.LangID.String()...)
+	} else {
+		buf := [maxCoreSize]byte{}
+		text = buf[:t.genCoreBytes(buf[:])]
+	}
+	return text, nil
+}
+
+// UnmarshalText implements encoding.TextUnmarshaler.
+func (t *Tag) UnmarshalText(text []byte) error {
+	tag, err := Parse(string(text))
+	*t = tag
+	return err
+}
+
+// Variants returns the part of the tag holding all variants or the empty string
+// if there are no variants defined.
+func (t Tag) Variants() string {
+	if t.pVariant == 0 {
+		return ""
+	}
+	return t.str[t.pVariant:t.pExt]
+}
+
+// VariantOrPrivateUseTags returns variants or private use tags.
+func (t Tag) VariantOrPrivateUseTags() string {
+	if t.pExt > 0 {
+		return t.str[t.pVariant:t.pExt]
+	}
+	return t.str[t.pVariant:]
+}
+
+// HasString reports whether this tag defines more than just the raw
+// components.
+func (t Tag) HasString() bool {
+	return t.str != ""
+}
+
+// Parent returns the CLDR parent of t. In CLDR, missing fields in data for a
+// specific language are substituted with fields from the parent language.
+// The parent for a language may change for newer versions of CLDR.
+func (t Tag) Parent() Tag {
+	if t.str != "" {
+		// Strip the variants and extensions.
+		b, s, r := t.Raw()
+		t = Tag{LangID: b, ScriptID: s, RegionID: r}
+		if t.RegionID == 0 && t.ScriptID != 0 && t.LangID != 0 {
+			base, _ := addTags(Tag{LangID: t.LangID})
+			if base.ScriptID == t.ScriptID {
+				return Tag{LangID: t.LangID}
+			}
+		}
+		return t
+	}
+	if t.LangID != 0 {
+		if t.RegionID != 0 {
+			maxScript := t.ScriptID
+			if maxScript == 0 {
+				max, _ := addTags(t)
+				maxScript = max.ScriptID
+			}
+
+			for i := range parents {
+				if Language(parents[i].lang) == t.LangID && Script(parents[i].maxScript) == maxScript {
+					for _, r := range parents[i].fromRegion {
+						if Region(r) == t.RegionID {
+							return Tag{
+								LangID:   t.LangID,
+								ScriptID: Script(parents[i].script),
+								RegionID: Region(parents[i].toRegion),
+							}
+						}
+					}
+				}
+			}
+
+			// Strip the script if it is the default one.
+			base, _ := addTags(Tag{LangID: t.LangID})
+			if base.ScriptID != maxScript {
+				return Tag{LangID: t.LangID, ScriptID: maxScript}
+			}
+			return Tag{LangID: t.LangID}
+		} else if t.ScriptID != 0 {
+			// The parent for an base-script pair with a non-default script is
+			// "und" instead of the base language.
+			base, _ := addTags(Tag{LangID: t.LangID})
+			if base.ScriptID != t.ScriptID {
+				return Und
+			}
+			return Tag{LangID: t.LangID}
+		}
+	}
+	return Und
+}
+
+// ParseExtension parses s as an extension and returns it on success.
+func ParseExtension(s string) (ext string, err error) {
+	scan := makeScannerString(s)
+	var end int
+	if n := len(scan.token); n != 1 {
+		return "", ErrSyntax
+	}
+	scan.toLower(0, len(scan.b))
+	end = parseExtension(&scan)
+	if end != len(s) {
+		return "", ErrSyntax
+	}
+	return string(scan.b), nil
+}
+
+// HasVariants reports whether t has variants.
+func (t Tag) HasVariants() bool {
+	return uint16(t.pVariant) < t.pExt
+}
+
+// HasExtensions reports whether t has extensions.
+func (t Tag) HasExtensions() bool {
+	return int(t.pExt) < len(t.str)
+}
+
+// Extension returns the extension of type x for tag t. It will return
+// false for ok if t does not have the requested extension. The returned
+// extension will be invalid in this case.
+func (t Tag) Extension(x byte) (ext string, ok bool) {
+	for i := int(t.pExt); i < len(t.str)-1; {
+		var ext string
+		i, ext = getExtension(t.str, i)
+		if ext[0] == x {
+			return ext, true
+		}
+	}
+	return "", false
+}
+
+// Extensions returns all extensions of t.
+func (t Tag) Extensions() []string {
+	e := []string{}
+	for i := int(t.pExt); i < len(t.str)-1; {
+		var ext string
+		i, ext = getExtension(t.str, i)
+		e = append(e, ext)
+	}
+	return e
+}
+
+// TypeForKey returns the type associated with the given key, where key and type
+// are of the allowed values defined for the Unicode locale extension ('u') in
+// http://www.unicode.org/reports/tr35/#Unicode_Language_and_Locale_Identifiers.
+// TypeForKey will traverse the inheritance chain to get the correct value.
+func (t Tag) TypeForKey(key string) string {
+	if start, end, _ := t.findTypeForKey(key); end != start {
+		return t.str[start:end]
+	}
+	return ""
+}
+
+var (
+	errPrivateUse       = errors.New("cannot set a key on a private use tag")
+	errInvalidArguments = errors.New("invalid key or type")
+)
+
+// SetTypeForKey returns a new Tag with the key set to type, where key and type
+// are of the allowed values defined for the Unicode locale extension ('u') in
+// http://www.unicode.org/reports/tr35/#Unicode_Language_and_Locale_Identifiers.
+// An empty value removes an existing pair with the same key.
+func (t Tag) SetTypeForKey(key, value string) (Tag, error) {
+	if t.IsPrivateUse() {
+		return t, errPrivateUse
+	}
+	if len(key) != 2 {
+		return t, errInvalidArguments
+	}
+
+	// Remove the setting if value is "".
+	if value == "" {
+		start, end, _ := t.findTypeForKey(key)
+		if start != end {
+			// Remove key tag and leading '-'.
+			start -= 4
+
+			// Remove a possible empty extension.
+			if (end == len(t.str) || t.str[end+2] == '-') && t.str[start-2] == '-' {
+				start -= 2
+			}
+			if start == int(t.pVariant) && end == len(t.str) {
+				t.str = ""
+				t.pVariant, t.pExt = 0, 0
+			} else {
+				t.str = fmt.Sprintf("%s%s", t.str[:start], t.str[end:])
+			}
+		}
+		return t, nil
+	}
+
+	if len(value) < 3 || len(value) > 8 {
+		return t, errInvalidArguments
+	}
+
+	var (
+		buf    [maxCoreSize + maxSimpleUExtensionSize]byte
+		uStart int // start of the -u extension.
+	)
+
+	// Generate the tag string if needed.
+	if t.str == "" {
+		uStart = t.genCoreBytes(buf[:])
+		buf[uStart] = '-'
+		uStart++
+	}
+
+	// Create new key-type pair and parse it to verify.
+	b := buf[uStart:]
+	copy(b, "u-")
+	copy(b[2:], key)
+	b[4] = '-'
+	b = b[:5+copy(b[5:], value)]
+	scan := makeScanner(b)
+	if parseExtensions(&scan); scan.err != nil {
+		return t, scan.err
+	}
+
+	// Assemble the replacement string.
+	if t.str == "" {
+		t.pVariant, t.pExt = byte(uStart-1), uint16(uStart-1)
+		t.str = string(buf[:uStart+len(b)])
+	} else {
+		s := t.str
+		start, end, hasExt := t.findTypeForKey(key)
+		if start == end {
+			if hasExt {
+				b = b[2:]
+			}
+			t.str = fmt.Sprintf("%s-%s%s", s[:start], b, s[end:])
+		} else {
+			t.str = fmt.Sprintf("%s%s%s", s[:start], value, s[end:])
+		}
+	}
+	return t, nil
+}
+
+// findKeyAndType returns the start and end position for the type corresponding
+// to key or the point at which to insert the key-value pair if the type
+// wasn't found. The hasExt return value reports whether an -u extension was present.
+// Note: the extensions are typically very small and are likely to contain
+// only one key-type pair.
+func (t Tag) findTypeForKey(key string) (start, end int, hasExt bool) {
+	p := int(t.pExt)
+	if len(key) != 2 || p == len(t.str) || p == 0 {
+		return p, p, false
+	}
+	s := t.str
+
+	// Find the correct extension.
+	for p++; s[p] != 'u'; p++ {
+		if s[p] > 'u' {
+			p--
+			return p, p, false
+		}
+		if p = nextExtension(s, p); p == len(s) {
+			return len(s), len(s), false
+		}
+	}
+	// Proceed to the hyphen following the extension name.
+	p++
+
+	// curKey is the key currently being processed.
+	curKey := ""
+
+	// Iterate over keys until we get the end of a section.
+	for {
+		// p points to the hyphen preceding the current token.
+		if p3 := p + 3; s[p3] == '-' {
+			// Found a key.
+			// Check whether we just processed the key that was requested.
+			if curKey == key {
+				return start, p, true
+			}
+			// Set to the next key and continue scanning type tokens.
+			curKey = s[p+1 : p3]
+			if curKey > key {
+				return p, p, true
+			}
+			// Start of the type token sequence.
+			start = p + 4
+			// A type is at least 3 characters long.
+			p += 7 // 4 + 3
+		} else {
+			// Attribute or type, which is at least 3 characters long.
+			p += 4
+		}
+		// p points past the third character of a type or attribute.
+		max := p + 5 // maximum length of token plus hyphen.
+		if len(s) < max {
+			max = len(s)
+		}
+		for ; p < max && s[p] != '-'; p++ {
+		}
+		// Bail if we have exhausted all tokens or if the next token starts
+		// a new extension.
+		if p == len(s) || s[p+2] == '-' {
+			if curKey == key {
+				return start, p, true
+			}
+			return p, p, true
+		}
+	}
+}
+
+// ParseBase parses a 2- or 3-letter ISO 639 code.
+// It returns a ValueError if s is a well-formed but unknown language identifier
+// or another error if another error occurred.
+func ParseBase(s string) (Language, error) {
+	if n := len(s); n < 2 || 3 < n {
+		return 0, ErrSyntax
+	}
+	var buf [3]byte
+	return getLangID(buf[:copy(buf[:], s)])
+}
+
+// ParseScript parses a 4-letter ISO 15924 code.
+// It returns a ValueError if s is a well-formed but unknown script identifier
+// or another error if another error occurred.
+func ParseScript(s string) (Script, error) {
+	if len(s) != 4 {
+		return 0, ErrSyntax
+	}
+	var buf [4]byte
+	return getScriptID(script, buf[:copy(buf[:], s)])
+}
+
+// EncodeM49 returns the Region for the given UN M.49 code.
+// It returns an error if r is not a valid code.
+func EncodeM49(r int) (Region, error) {
+	return getRegionM49(r)
+}
+
+// ParseRegion parses a 2- or 3-letter ISO 3166-1 or a UN M.49 code.
+// It returns a ValueError if s is a well-formed but unknown region identifier
+// or another error if another error occurred.
+func ParseRegion(s string) (Region, error) {
+	if n := len(s); n < 2 || 3 < n {
+		return 0, ErrSyntax
+	}
+	var buf [3]byte
+	return getRegionID(buf[:copy(buf[:], s)])
+}
+
+// IsCountry returns whether this region is a country or autonomous area. This
+// includes non-standard definitions from CLDR.
+func (r Region) IsCountry() bool {
+	if r == 0 || r.IsGroup() || r.IsPrivateUse() && r != _XK {
+		return false
+	}
+	return true
+}
+
+// IsGroup returns whether this region defines a collection of regions. This
+// includes non-standard definitions from CLDR.
+func (r Region) IsGroup() bool {
+	if r == 0 {
+		return false
+	}
+	return int(regionInclusion[r]) < len(regionContainment)
+}
+
+// Contains returns whether Region c is contained by Region r. It returns true
+// if c == r.
+func (r Region) Contains(c Region) bool {
+	if r == c {
+		return true
+	}
+	g := regionInclusion[r]
+	if g >= nRegionGroups {
+		return false
+	}
+	m := regionContainment[g]
+
+	d := regionInclusion[c]
+	b := regionInclusionBits[d]
+
+	// A contained country may belong to multiple disjoint groups. Matching any
+	// of these indicates containment. If the contained region is a group, it
+	// must strictly be a subset.
+	if d >= nRegionGroups {
+		return b&m != 0
+	}
+	return b&^m == 0
+}
+
+var errNoTLD = errors.New("language: region is not a valid ccTLD")
+
+// TLD returns the country code top-level domain (ccTLD). UK is returned for GB.
+// In all other cases it returns either the region itself or an error.
+//
+// This method may return an error for a region for which there exists a
+// canonical form with a ccTLD. To get that ccTLD canonicalize r first. The
+// region will already be canonicalized it was obtained from a Tag that was
+// obtained using any of the default methods.
+func (r Region) TLD() (Region, error) {
+	// See http://en.wikipedia.org/wiki/Country_code_top-level_domain for the
+	// difference between ISO 3166-1 and IANA ccTLD.
+	if r == _GB {
+		r = _UK
+	}
+	if (r.typ() & ccTLD) == 0 {
+		return 0, errNoTLD
+	}
+	return r, nil
+}
+
+// Canonicalize returns the region or a possible replacement if the region is
+// deprecated. It will not return a replacement for deprecated regions that
+// are split into multiple regions.
+func (r Region) Canonicalize() Region {
+	if cr := normRegion(r); cr != 0 {
+		return cr
+	}
+	return r
+}
+
+// Variant represents a registered variant of a language as defined by BCP 47.
+type Variant struct {
+	ID  uint8
+	str string
+}
+
+// ParseVariant parses and returns a Variant. An error is returned if s is not
+// a valid variant.
+func ParseVariant(s string) (Variant, error) {
+	s = strings.ToLower(s)
+	if id, ok := variantIndex[s]; ok {
+		return Variant{id, s}, nil
+	}
+	return Variant{}, NewValueError([]byte(s))
+}
+
+// String returns the string representation of the variant.
+func (v Variant) String() string {
+	return v.str
+}

vendor/golang.org/x/text/internal/language/lookup.go

@@ -17,11 +17,11 @@ import (
 // if it could not be found.
 func findIndex(idx tag.Index, key []byte, form string) (index int, err error) {
 	if !tag.FixCase(form, key) {
-		return 0, errSyntax
+		return 0, ErrSyntax
 	}
 	i := idx.Index(key)
 	if i == -1 {
-		return 0, mkErrInvalid(key)
+		return 0, NewValueError(key)
 	}
 	return i, nil
 }
@@ -32,38 +32,45 @@ func searchUint(imap []uint16, key uint16) int {
 	})
 }
 
-type langID uint16
+type Language uint16
 
 // getLangID returns the langID of s if s is a canonical subtag
 // or langUnknown if s is not a canonical subtag.
-func getLangID(s []byte) (langID, error) {
+func getLangID(s []byte) (Language, error) {
 	if len(s) == 2 {
 		return getLangISO2(s)
 	}
 	return getLangISO3(s)
 }
 
+// TODO language normalization as well as the AliasMaps could be moved to the
+// higher level package, but it is a bit tricky to separate the generation.
+
+func (id Language) Canonicalize() (Language, AliasType) {
+	return normLang(id)
+}
+
 // mapLang returns the mapped langID of id according to mapping m.
-func normLang(id langID) (langID, langAliasType) {
-	k := sort.Search(len(langAliasMap), func(i int) bool {
-		return langAliasMap[i].from >= uint16(id)
+func normLang(id Language) (Language, AliasType) {
+	k := sort.Search(len(AliasMap), func(i int) bool {
+		return AliasMap[i].From >= uint16(id)
 	})
-	if k < len(langAliasMap) && langAliasMap[k].from == uint16(id) {
-		return langID(langAliasMap[k].to), langAliasTypes[k]
+	if k < len(AliasMap) && AliasMap[k].From == uint16(id) {
+		return Language(AliasMap[k].To), AliasTypes[k]
 	}
-	return id, langAliasTypeUnknown
+	return id, AliasTypeUnknown
 }
 
 // getLangISO2 returns the langID for the given 2-letter ISO language code
 // or unknownLang if this does not exist.
-func getLangISO2(s []byte) (langID, error) {
+func getLangISO2(s []byte) (Language, error) {
 	if !tag.FixCase("zz", s) {
-		return 0, errSyntax
+		return 0, ErrSyntax
 	}
 	if i := lang.Index(s); i != -1 && lang.Elem(i)[3] != 0 {
-		return langID(i), nil
+		return Language(i), nil
 	}
-	return 0, mkErrInvalid(s)
+	return 0, NewValueError(s)
 }
 
 const base = 'z' - 'a' + 1
@@ -88,7 +95,7 @@ func intToStr(v uint, s []byte) {
 
 // getLangISO3 returns the langID for the given 3-letter ISO language code
 // or unknownLang if this does not exist.
-func getLangISO3(s []byte) (langID, error) {
+func getLangISO3(s []byte) (Language, error) {
 	if tag.FixCase("und", s) {
 		// first try to match canonical 3-letter entries
 		for i := lang.Index(s[:2]); i != -1; i = lang.Next(s[:2], i) {
@@ -96,7 +103,7 @@ func getLangISO3(s []byte) (langID, error) {
 				// We treat "und" as special and always translate it to "unspecified".
 				// Note that ZZ and Zzzz are private use and are not treated as
 				// unspecified by default.
-				id := langID(i)
+				id := Language(i)
 				if id == nonCanonicalUnd {
 					return 0, nil
 				}
@@ -104,26 +111,26 @@ func getLangISO3(s []byte) (langID, error) {
 			}
 		}
 		if i := altLangISO3.Index(s); i != -1 {
-			return langID(altLangIndex[altLangISO3.Elem(i)[3]]), nil
+			return Language(altLangIndex[altLangISO3.Elem(i)[3]]), nil
 		}
 		n := strToInt(s)
 		if langNoIndex[n/8]&(1<<(n%8)) != 0 {
-			return langID(n) + langNoIndexOffset, nil
+			return Language(n) + langNoIndexOffset, nil
 		}
 		// Check for non-canonical uses of ISO3.
 		for i := lang.Index(s[:1]); i != -1; i = lang.Next(s[:1], i) {
 			if e := lang.Elem(i); e[2] == s[1] && e[3] == s[2] {
-				return langID(i), nil
+				return Language(i), nil
 			}
 		}
-		return 0, mkErrInvalid(s)
+		return 0, NewValueError(s)
 	}
-	return 0, errSyntax
+	return 0, ErrSyntax
 }
 
-// stringToBuf writes the string to b and returns the number of bytes
+// StringToBuf writes the string to b and returns the number of bytes
 // written.  cap(b) must be >= 3.
-func (id langID) stringToBuf(b []byte) int {
+func (id Language) StringToBuf(b []byte) int {
 	if id >= langNoIndexOffset {
 		intToStr(uint(id)-langNoIndexOffset, b[:3])
 		return 3
@@ -140,7 +147,7 @@ func (id langID) stringToBuf(b []byte) int {
 // String returns the BCP 47 representation of the langID.
 // Use b as variable name, instead of id, to ensure the variable
 // used is consistent with that of Base in which this type is embedded.
-func (b langID) String() string {
+func (b Language) String() string {
 	if b == 0 {
 		return "und"
 	} else if b >= langNoIndexOffset {
@@ -157,7 +164,7 @@ func (b langID) String() string {
 }
 
 // ISO3 returns the ISO 639-3 language code.
-func (b langID) ISO3() string {
+func (b Language) ISO3() string {
 	if b == 0 || b >= langNoIndexOffset {
 		return b.String()
 	}
@@ -173,15 +180,24 @@ func (b langID) ISO3() string {
 }
 
 // IsPrivateUse reports whether this language code is reserved for private use.
-func (b langID) IsPrivateUse() bool {
+func (b Language) IsPrivateUse() bool {
 	return langPrivateStart <= b && b <= langPrivateEnd
 }
 
-type regionID uint16
+// SuppressScript returns the script marked as SuppressScript in the IANA
+// language tag repository, or 0 if there is no such script.
+func (b Language) SuppressScript() Script {
+	if b < langNoIndexOffset {
+		return Script(suppressScript[b])
+	}
+	return 0
+}
+
+type Region uint16
 
 // getRegionID returns the region id for s if s is a valid 2-letter region code
 // or unknownRegion.
-func getRegionID(s []byte) (regionID, error) {
+func getRegionID(s []byte) (Region, error) {
 	if len(s) == 3 {
 		if isAlpha(s[0]) {
 			return getRegionISO3(s)
@@ -195,34 +211,34 @@ func getRegionID(s []byte) (regionID, error) {
 
 // getRegionISO2 returns the regionID for the given 2-letter ISO country code
 // or unknownRegion if this does not exist.
-func getRegionISO2(s []byte) (regionID, error) {
+func getRegionISO2(s []byte) (Region, error) {
 	i, err := findIndex(regionISO, s, "ZZ")
 	if err != nil {
 		return 0, err
 	}
-	return regionID(i) + isoRegionOffset, nil
+	return Region(i) + isoRegionOffset, nil
 }
 
 // getRegionISO3 returns the regionID for the given 3-letter ISO country code
 // or unknownRegion if this does not exist.
-func getRegionISO3(s []byte) (regionID, error) {
+func getRegionISO3(s []byte) (Region, error) {
 	if tag.FixCase("ZZZ", s) {
 		for i := regionISO.Index(s[:1]); i != -1; i = regionISO.Next(s[:1], i) {
 			if e := regionISO.Elem(i); e[2] == s[1] && e[3] == s[2] {
-				return regionID(i) + isoRegionOffset, nil
+				return Region(i) + isoRegionOffset, nil
 			}
 		}
 		for i := 0; i < len(altRegionISO3); i += 3 {
 			if tag.Compare(altRegionISO3[i:i+3], s) == 0 {
-				return regionID(altRegionIDs[i/3]), nil
+				return Region(altRegionIDs[i/3]), nil
 			}
 		}
-		return 0, mkErrInvalid(s)
+		return 0, NewValueError(s)
 	}
-	return 0, errSyntax
+	return 0, ErrSyntax
 }
 
-func getRegionM49(n int) (regionID, error) {
+func getRegionM49(n int) (Region, error) {
 	if 0 < n && n <= 999 {
 		const (
 			searchBits = 7
@@ -236,7 +252,7 @@ func getRegionM49(n int) (regionID, error) {
 			return buf[i] >= val
 		})
 		if r := fromM49[int(m49Index[idx])+i]; r&^regionMask == val {
-			return regionID(r & regionMask), nil
+			return Region(r & regionMask), nil
 		}
 	}
 	var e ValueError
@@ -247,13 +263,13 @@ func getRegionM49(n int) (regionID, error) {
 // normRegion returns a region if r is deprecated or 0 otherwise.
 // TODO: consider supporting BYS (-> BLR), CSK (-> 200 or CZ), PHI (-> PHL) and AFI (-> DJ).
 // TODO: consider mapping split up regions to new most populous one (like CLDR).
-func normRegion(r regionID) regionID {
+func normRegion(r Region) Region {
 	m := regionOldMap
 	k := sort.Search(len(m), func(i int) bool {
-		return m[i].from >= uint16(r)
+		return m[i].From >= uint16(r)
 	})
-	if k < len(m) && m[k].from == uint16(r) {
-		return regionID(m[k].to)
+	if k < len(m) && m[k].From == uint16(r) {
+		return Region(m[k].To)
 	}
 	return 0
 }
@@ -264,13 +280,13 @@ const (
 	bcp47Region
 )
 
-func (r regionID) typ() byte {
+func (r Region) typ() byte {
 	return regionTypes[r]
 }
 
 // String returns the BCP 47 representation for the region.
 // It returns "ZZ" for an unspecified region.
-func (r regionID) String() string {
+func (r Region) String() string {
 	if r < isoRegionOffset {
 		if r == 0 {
 			return "ZZ"
@@ -284,7 +300,7 @@ func (r regionID) String() string {
 // ISO3 returns the 3-letter ISO code of r.
 // Note that not all regions have a 3-letter ISO code.
 // In such cases this method returns "ZZZ".
-func (r regionID) ISO3() string {
+func (r Region) ISO3() string {
 	if r < isoRegionOffset {
 		return "ZZZ"
 	}
@@ -301,29 +317,29 @@ func (r regionID) ISO3() string {
 
 // M49 returns the UN M.49 encoding of r, or 0 if this encoding
 // is not defined for r.
-func (r regionID) M49() int {
+func (r Region) M49() int {
 	return int(m49[r])
 }
 
 // IsPrivateUse reports whether r has the ISO 3166 User-assigned status. This
 // may include private-use tags that are assigned by CLDR and used in this
 // implementation. So IsPrivateUse and IsCountry can be simultaneously true.
-func (r regionID) IsPrivateUse() bool {
+func (r Region) IsPrivateUse() bool {
 	return r.typ()&iso3166UserAssigned != 0
 }
 
-type scriptID uint8
+type Script uint8
 
 // getScriptID returns the script id for string s. It assumes that s
 // is of the format [A-Z][a-z]{3}.
-func getScriptID(idx tag.Index, s []byte) (scriptID, error) {
+func getScriptID(idx tag.Index, s []byte) (Script, error) {
 	i, err := findIndex(idx, s, "Zzzz")
-	return scriptID(i), err
+	return Script(i), err
 }
 
 // String returns the script code in title case.
 // It returns "Zzzz" for an unspecified script.
-func (s scriptID) String() string {
+func (s Script) String() string {
 	if s == 0 {
 		return "Zzzz"
 	}
@@ -331,7 +347,7 @@ func (s scriptID) String() string {
 }
 
 // IsPrivateUse reports whether this script code is reserved for private use.
-func (s scriptID) IsPrivateUse() bool {
+func (s Script) IsPrivateUse() bool {
 	return _Qaaa <= s && s <= _Qabx
 }
 
@@ -389,7 +405,7 @@ func grandfathered(s [maxAltTaglen]byte) (t Tag, ok bool) {
 		if v < 0 {
 			return Make(altTags[altTagIndex[-v-1]:altTagIndex[-v]]), true
 		}
-		t.lang = langID(v)
+		t.LangID = Language(v)
 		return t, true
 	}
 	return t, false

vendor/golang.org/x/text/internal/language/match.go

@@ -0,0 +1,226 @@
+// Copyright 2013 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package language
+
+import "errors"
+
+type scriptRegionFlags uint8
+
+const (
+	isList = 1 << iota
+	scriptInFrom
+	regionInFrom
+)
+
+func (t *Tag) setUndefinedLang(id Language) {
+	if t.LangID == 0 {
+		t.LangID = id
+	}
+}
+
+func (t *Tag) setUndefinedScript(id Script) {
+	if t.ScriptID == 0 {
+		t.ScriptID = id
+	}
+}
+
+func (t *Tag) setUndefinedRegion(id Region) {
+	if t.RegionID == 0 || t.RegionID.Contains(id) {
+		t.RegionID = id
+	}
+}
+
+// ErrMissingLikelyTagsData indicates no information was available
+// to compute likely values of missing tags.
+var ErrMissingLikelyTagsData = errors.New("missing likely tags data")
+
+// addLikelySubtags sets subtags to their most likely value, given the locale.
+// In most cases this means setting fields for unknown values, but in some
+// cases it may alter a value.  It returns an ErrMissingLikelyTagsData error
+// if the given locale cannot be expanded.
+func (t Tag) addLikelySubtags() (Tag, error) {
+	id, err := addTags(t)
+	if err != nil {
+		return t, err
+	} else if id.equalTags(t) {
+		return t, nil
+	}
+	id.RemakeString()
+	return id, nil
+}
+
+// specializeRegion attempts to specialize a group region.
+func specializeRegion(t *Tag) bool {
+	if i := regionInclusion[t.RegionID]; i < nRegionGroups {
+		x := likelyRegionGroup[i]
+		if Language(x.lang) == t.LangID && Script(x.script) == t.ScriptID {
+			t.RegionID = Region(x.region)
+		}
+		return true
+	}
+	return false
+}
+
+// Maximize returns a new tag with missing tags filled in.
+func (t Tag) Maximize() (Tag, error) {
+	return addTags(t)
+}
+
+func addTags(t Tag) (Tag, error) {
+	// We leave private use identifiers alone.
+	if t.IsPrivateUse() {
+		return t, nil
+	}
+	if t.ScriptID != 0 && t.RegionID != 0 {
+		if t.LangID != 0 {
+			// already fully specified
+			specializeRegion(&t)
+			return t, nil
+		}
+		// Search matches for und-script-region. Note that for these cases
+		// region will never be a group so there is no need to check for this.
+		list := likelyRegion[t.RegionID : t.RegionID+1]
+		if x := list[0]; x.flags&isList != 0 {
+			list = likelyRegionList[x.lang : x.lang+uint16(x.script)]
+		}
+		for _, x := range list {
+			// Deviating from the spec. See match_test.go for details.
+			if Script(x.script) == t.ScriptID {
+				t.setUndefinedLang(Language(x.lang))
+				return t, nil
+			}
+		}
+	}
+	if t.LangID != 0 {
+		// Search matches for lang-script and lang-region, where lang != und.
+		if t.LangID < langNoIndexOffset {
+			x := likelyLang[t.LangID]
+			if x.flags&isList != 0 {
+				list := likelyLangList[x.region : x.region+uint16(x.script)]
+				if t.ScriptID != 0 {
+					for _, x := range list {
+						if Script(x.script) == t.ScriptID && x.flags&scriptInFrom != 0 {
+							t.setUndefinedRegion(Region(x.region))
+							return t, nil
+						}
+					}
+				} else if t.RegionID != 0 {
+					count := 0
+					goodScript := true
+					tt := t
+					for _, x := range list {
+						// We visit all entries for which the script was not
+						// defined, including the ones where the region was not
+						// defined. This allows for proper disambiguation within
+						// regions.
+						if x.flags&scriptInFrom == 0 && t.RegionID.Contains(Region(x.region)) {
+							tt.RegionID = Region(x.region)
+							tt.setUndefinedScript(Script(x.script))
+							goodScript = goodScript && tt.ScriptID == Script(x.script)
+							count++
+						}
+					}
+					if count == 1 {
+						return tt, nil
+					}
+					// Even if we fail to find a unique Region, we might have
+					// an unambiguous script.
+					if goodScript {
+						t.ScriptID = tt.ScriptID
+					}
+				}
+			}
+		}
+	} else {
+		// Search matches for und-script.
+		if t.ScriptID != 0 {
+			x := likelyScript[t.ScriptID]
+			if x.region != 0 {
+				t.setUndefinedRegion(Region(x.region))
+				t.setUndefinedLang(Language(x.lang))
+				return t, nil
+			}
+		}
+		// Search matches for und-region. If und-script-region exists, it would
+		// have been found earlier.
+		if t.RegionID != 0 {
+			if i := regionInclusion[t.RegionID]; i < nRegionGroups {
+				x := likelyRegionGroup[i]
+				if x.region != 0 {
+					t.setUndefinedLang(Language(x.lang))
+					t.setUndefinedScript(Script(x.script))
+					t.RegionID = Region(x.region)
+				}
+			} else {
+				x := likelyRegion[t.RegionID]
+				if x.flags&isList != 0 {
+					x = likelyRegionList[x.lang]
+				}
+				if x.script != 0 && x.flags != scriptInFrom {
+					t.setUndefinedLang(Language(x.lang))
+					t.setUndefinedScript(Script(x.script))
+					return t, nil
+				}
+			}
+		}
+	}
+
+	// Search matches for lang.
+	if t.LangID < langNoIndexOffset {
+		x := likelyLang[t.LangID]
+		if x.flags&isList != 0 {
+			x = likelyLangList[x.region]
+		}
+		if x.region != 0 {
+			t.setUndefinedScript(Script(x.script))
+			t.setUndefinedRegion(Region(x.region))
+		}
+		specializeRegion(&t)
+		if t.LangID == 0 {
+			t.LangID = _en // default language
+		}
+		return t, nil
+	}
+	return t, ErrMissingLikelyTagsData
+}
+
+func (t *Tag) setTagsFrom(id Tag) {
+	t.LangID = id.LangID
+	t.ScriptID = id.ScriptID
+	t.RegionID = id.RegionID
+}
+
+// minimize removes the region or script subtags from t such that
+// t.addLikelySubtags() == t.minimize().addLikelySubtags().
+func (t Tag) minimize() (Tag, error) {
+	t, err := minimizeTags(t)
+	if err != nil {
+		return t, err
+	}
+	t.RemakeString()
+	return t, nil
+}
+
+// minimizeTags mimics the behavior of the ICU 51 C implementation.
+func minimizeTags(t Tag) (Tag, error) {
+	if t.equalTags(Und) {
+		return t, nil
+	}
+	max, err := addTags(t)
+	if err != nil {
+		return t, err
+	}
+	for _, id := range [...]Tag{
+		{LangID: t.LangID},
+		{LangID: t.LangID, RegionID: t.RegionID},
+		{LangID: t.LangID, ScriptID: t.ScriptID},
+	} {
+		if x, err := addTags(id); err == nil && max.equalTags(x) {
+			t.setTagsFrom(id)
+			break
+		}
+	}
+	return t, nil
+}

vendor/golang.org/x/text/internal/language/parse.go

@@ -0,0 +1,594 @@
+// Copyright 2013 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package language
+
+import (
+	"bytes"
+	"errors"
+	"fmt"
+	"sort"
+
+	"golang.org/x/text/internal/tag"
+)
+
+// isAlpha returns true if the byte is not a digit.
+// b must be an ASCII letter or digit.
+func isAlpha(b byte) bool {
+	return b > '9'
+}
+
+// isAlphaNum returns true if the string contains only ASCII letters or digits.
+func isAlphaNum(s []byte) bool {
+	for _, c := range s {
+		if !('a' <= c && c <= 'z' || 'A' <= c && c <= 'Z' || '0' <= c && c <= '9') {
+			return false
+		}
+	}
+	return true
+}
+
+// ErrSyntax is returned by any of the parsing functions when the
+// input is not well-formed, according to BCP 47.
+// TODO: return the position at which the syntax error occurred?
+var ErrSyntax = errors.New("language: tag is not well-formed")
+
+// ErrDuplicateKey is returned when a tag contains the same key twice with
+// different values in the -u section.
+var ErrDuplicateKey = errors.New("language: different values for same key in -u extension")
+
+// ValueError is returned by any of the parsing functions when the
+// input is well-formed but the respective subtag is not recognized
+// as a valid value.
+type ValueError struct {
+	v [8]byte
+}
+
+// NewValueError creates a new ValueError.
+func NewValueError(tag []byte) ValueError {
+	var e ValueError
+	copy(e.v[:], tag)
+	return e
+}
+
+func (e ValueError) tag() []byte {
+	n := bytes.IndexByte(e.v[:], 0)
+	if n == -1 {
+		n = 8
+	}
+	return e.v[:n]
+}
+
+// Error implements the error interface.
+func (e ValueError) Error() string {
+	return fmt.Sprintf("language: subtag %q is well-formed but unknown", e.tag())
+}
+
+// Subtag returns the subtag for which the error occurred.
+func (e ValueError) Subtag() string {
+	return string(e.tag())
+}
+
+// scanner is used to scan BCP 47 tokens, which are separated by _ or -.
+type scanner struct {
+	b     []byte
+	bytes [max99thPercentileSize]byte
+	token []byte
+	start int // start position of the current token
+	end   int // end position of the current token
+	next  int // next point for scan
+	err   error
+	done  bool
+}
+
+func makeScannerString(s string) scanner {
+	scan := scanner{}
+	if len(s) <= len(scan.bytes) {
+		scan.b = scan.bytes[:copy(scan.bytes[:], s)]
+	} else {
+		scan.b = []byte(s)
+	}
+	scan.init()
+	return scan
+}
+
+// makeScanner returns a scanner using b as the input buffer.
+// b is not copied and may be modified by the scanner routines.
+func makeScanner(b []byte) scanner {
+	scan := scanner{b: b}
+	scan.init()
+	return scan
+}
+
+func (s *scanner) init() {
+	for i, c := range s.b {
+		if c == '_' {
+			s.b[i] = '-'
+		}
+	}
+	s.scan()
+}
+
+// restToLower converts the string between start and end to lower case.
+func (s *scanner) toLower(start, end int) {
+	for i := start; i < end; i++ {
+		c := s.b[i]
+		if 'A' <= c && c <= 'Z' {
+			s.b[i] += 'a' - 'A'
+		}
+	}
+}
+
+func (s *scanner) setError(e error) {
+	if s.err == nil || (e == ErrSyntax && s.err != ErrSyntax) {
+		s.err = e
+	}
+}
+
+// resizeRange shrinks or grows the array at position oldStart such that
+// a new string of size newSize can fit between oldStart and oldEnd.
+// Sets the scan point to after the resized range.
+func (s *scanner) resizeRange(oldStart, oldEnd, newSize int) {
+	s.start = oldStart
+	if end := oldStart + newSize; end != oldEnd {
+		diff := end - oldEnd
+		if end < cap(s.b) {
+			b := make([]byte, len(s.b)+diff)
+			copy(b, s.b[:oldStart])
+			copy(b[end:], s.b[oldEnd:])
+			s.b = b
+		} else {
+			s.b = append(s.b[end:], s.b[oldEnd:]...)
+		}
+		s.next = end + (s.next - s.end)
+		s.end = end
+	}
+}
+
+// replace replaces the current token with repl.
+func (s *scanner) replace(repl string) {
+	s.resizeRange(s.start, s.end, len(repl))
+	copy(s.b[s.start:], repl)
+}
+
+// gobble removes the current token from the input.
+// Caller must call scan after calling gobble.
+func (s *scanner) gobble(e error) {
+	s.setError(e)
+	if s.start == 0 {
+		s.b = s.b[:+copy(s.b, s.b[s.next:])]
+		s.end = 0
+	} else {
+		s.b = s.b[:s.start-1+copy(s.b[s.start-1:], s.b[s.end:])]
+		s.end = s.start - 1
+	}
+	s.next = s.start
+}
+
+// deleteRange removes the given range from s.b before the current token.
+func (s *scanner) deleteRange(start, end int) {
+	s.b = s.b[:start+copy(s.b[start:], s.b[end:])]
+	diff := end - start
+	s.next -= diff
+	s.start -= diff
+	s.end -= diff
+}
+
+// scan parses the next token of a BCP 47 string.  Tokens that are larger
+// than 8 characters or include non-alphanumeric characters result in an error
+// and are gobbled and removed from the output.
+// It returns the end position of the last token consumed.
+func (s *scanner) scan() (end int) {
+	end = s.end
+	s.token = nil
+	for s.start = s.next; s.next < len(s.b); {
+		i := bytes.IndexByte(s.b[s.next:], '-')
+		if i == -1 {
+			s.end = len(s.b)
+			s.next = len(s.b)
+			i = s.end - s.start
+		} else {
+			s.end = s.next + i
+			s.next = s.end + 1
+		}
+		token := s.b[s.start:s.end]
+		if i < 1 || i > 8 || !isAlphaNum(token) {
+			s.gobble(ErrSyntax)
+			continue
+		}
+		s.token = token
+		return end
+	}
+	if n := len(s.b); n > 0 && s.b[n-1] == '-' {
+		s.setError(ErrSyntax)
+		s.b = s.b[:len(s.b)-1]
+	}
+	s.done = true
+	return end
+}
+
+// acceptMinSize parses multiple tokens of the given size or greater.
+// It returns the end position of the last token consumed.
+func (s *scanner) acceptMinSize(min int) (end int) {
+	end = s.end
+	s.scan()
+	for ; len(s.token) >= min; s.scan() {
+		end = s.end
+	}
+	return end
+}
+
+// Parse parses the given BCP 47 string and returns a valid Tag. If parsing
+// failed it returns an error and any part of the tag that could be parsed.
+// If parsing succeeded but an unknown value was found, it returns
+// ValueError. The Tag returned in this case is just stripped of the unknown
+// value. All other values are preserved. It accepts tags in the BCP 47 format
+// and extensions to this standard defined in
+// http://www.unicode.org/reports/tr35/#Unicode_Language_and_Locale_Identifiers.
+func Parse(s string) (t Tag, err error) {
+	// TODO: consider supporting old-style locale key-value pairs.
+	if s == "" {
+		return Und, ErrSyntax
+	}
+	if len(s) <= maxAltTaglen {
+		b := [maxAltTaglen]byte{}
+		for i, c := range s {
+			// Generating invalid UTF-8 is okay as it won't match.
+			if 'A' <= c && c <= 'Z' {
+				c += 'a' - 'A'
+			} else if c == '_' {
+				c = '-'
+			}
+			b[i] = byte(c)
+		}
+		if t, ok := grandfathered(b); ok {
+			return t, nil
+		}
+	}
+	scan := makeScannerString(s)
+	return parse(&scan, s)
+}
+
+func parse(scan *scanner, s string) (t Tag, err error) {
+	t = Und
+	var end int
+	if n := len(scan.token); n <= 1 {
+		scan.toLower(0, len(scan.b))
+		if n == 0 || scan.token[0] != 'x' {
+			return t, ErrSyntax
+		}
+		end = parseExtensions(scan)
+	} else if n >= 4 {
+		return Und, ErrSyntax
+	} else { // the usual case
+		t, end = parseTag(scan)
+		if n := len(scan.token); n == 1 {
+			t.pExt = uint16(end)
+			end = parseExtensions(scan)
+		} else if end < len(scan.b) {
+			scan.setError(ErrSyntax)
+			scan.b = scan.b[:end]
+		}
+	}
+	if int(t.pVariant) < len(scan.b) {
+		if end < len(s) {
+			s = s[:end]
+		}
+		if len(s) > 0 && tag.Compare(s, scan.b) == 0 {
+			t.str = s
+		} else {
+			t.str = string(scan.b)
+		}
+	} else {
+		t.pVariant, t.pExt = 0, 0
+	}
+	return t, scan.err
+}
+
+// parseTag parses language, script, region and variants.
+// It returns a Tag and the end position in the input that was parsed.
+func parseTag(scan *scanner) (t Tag, end int) {
+	var e error
+	// TODO: set an error if an unknown lang, script or region is encountered.
+	t.LangID, e = getLangID(scan.token)
+	scan.setError(e)
+	scan.replace(t.LangID.String())
+	langStart := scan.start
+	end = scan.scan()
+	for len(scan.token) == 3 && isAlpha(scan.token[0]) {
+		// From http://tools.ietf.org/html/bcp47, <lang>-<extlang> tags are equivalent
+		// to a tag of the form <extlang>.
+		lang, e := getLangID(scan.token)
+		if lang != 0 {
+			t.LangID = lang
+			copy(scan.b[langStart:], lang.String())
+			scan.b[langStart+3] = '-'
+			scan.start = langStart + 4
+		}
+		scan.gobble(e)
+		end = scan.scan()
+	}
+	if len(scan.token) == 4 && isAlpha(scan.token[0]) {
+		t.ScriptID, e = getScriptID(script, scan.token)
+		if t.ScriptID == 0 {
+			scan.gobble(e)
+		}
+		end = scan.scan()
+	}
+	if n := len(scan.token); n >= 2 && n <= 3 {
+		t.RegionID, e = getRegionID(scan.token)
+		if t.RegionID == 0 {
+			scan.gobble(e)
+		} else {
+			scan.replace(t.RegionID.String())
+		}
+		end = scan.scan()
+	}
+	scan.toLower(scan.start, len(scan.b))
+	t.pVariant = byte(end)
+	end = parseVariants(scan, end, t)
+	t.pExt = uint16(end)
+	return t, end
+}
+
+var separator = []byte{'-'}
+
+// parseVariants scans tokens as long as each token is a valid variant string.
+// Duplicate variants are removed.
+func parseVariants(scan *scanner, end int, t Tag) int {
+	start := scan.start
+	varIDBuf := [4]uint8{}
+	variantBuf := [4][]byte{}
+	varID := varIDBuf[:0]
+	variant := variantBuf[:0]
+	last := -1
+	needSort := false
+	for ; len(scan.token) >= 4; scan.scan() {
+		// TODO: measure the impact of needing this conversion and redesign
+		// the data structure if there is an issue.
+		v, ok := variantIndex[string(scan.token)]
+		if !ok {
+			// unknown variant
+			// TODO: allow user-defined variants?
+			scan.gobble(NewValueError(scan.token))
+			continue
+		}
+		varID = append(varID, v)
+		variant = append(variant, scan.token)
+		if !needSort {
+			if last < int(v) {
+				last = int(v)
+			} else {
+				needSort = true
+				// There is no legal combinations of more than 7 variants
+				// (and this is by no means a useful sequence).
+				const maxVariants = 8
+				if len(varID) > maxVariants {
+					break
+				}
+			}
+		}
+		end = scan.end
+	}
+	if needSort {
+		sort.Sort(variantsSort{varID, variant})
+		k, l := 0, -1
+		for i, v := range varID {
+			w := int(v)
+			if l == w {
+				// Remove duplicates.
+				continue
+			}
+			varID[k] = varID[i]
+			variant[k] = variant[i]
+			k++
+			l = w
+		}
+		if str := bytes.Join(variant[:k], separator); len(str) == 0 {
+			end = start - 1
+		} else {
+			scan.resizeRange(start, end, len(str))
+			copy(scan.b[scan.start:], str)
+			end = scan.end
+		}
+	}
+	return end
+}
+
+type variantsSort struct {
+	i []uint8
+	v [][]byte
+}
+
+func (s variantsSort) Len() int {
+	return len(s.i)
+}
+
+func (s variantsSort) Swap(i, j int) {
+	s.i[i], s.i[j] = s.i[j], s.i[i]
+	s.v[i], s.v[j] = s.v[j], s.v[i]
+}
+
+func (s variantsSort) Less(i, j int) bool {
+	return s.i[i] < s.i[j]
+}
+
+type bytesSort struct {
+	b [][]byte
+	n int // first n bytes to compare
+}
+
+func (b bytesSort) Len() int {
+	return len(b.b)
+}
+
+func (b bytesSort) Swap(i, j int) {
+	b.b[i], b.b[j] = b.b[j], b.b[i]
+}
+
+func (b bytesSort) Less(i, j int) bool {
+	for k := 0; k < b.n; k++ {
+		if b.b[i][k] == b.b[j][k] {
+			continue
+		}
+		return b.b[i][k] < b.b[j][k]
+	}
+	return false
+}
+
+// parseExtensions parses and normalizes the extensions in the buffer.
+// It returns the last position of scan.b that is part of any extension.
+// It also trims scan.b to remove excess parts accordingly.
+func parseExtensions(scan *scanner) int {
+	start := scan.start
+	exts := [][]byte{}
+	private := []byte{}
+	end := scan.end
+	for len(scan.token) == 1 {
+		extStart := scan.start
+		ext := scan.token[0]
+		end = parseExtension(scan)
+		extension := scan.b[extStart:end]
+		if len(extension) < 3 || (ext != 'x' && len(extension) < 4) {
+			scan.setError(ErrSyntax)
+			end = extStart
+			continue
+		} else if start == extStart && (ext == 'x' || scan.start == len(scan.b)) {
+			scan.b = scan.b[:end]
+			return end
+		} else if ext == 'x' {
+			private = extension
+			break
+		}
+		exts = append(exts, extension)
+	}
+	sort.Sort(bytesSort{exts, 1})
+	if len(private) > 0 {
+		exts = append(exts, private)
+	}
+	scan.b = scan.b[:start]
+	if len(exts) > 0 {
+		scan.b = append(scan.b, bytes.Join(exts, separator)...)
+	} else if start > 0 {
+		// Strip trailing '-'.
+		scan.b = scan.b[:start-1]
+	}
+	return end
+}
+
+// parseExtension parses a single extension and returns the position of
+// the extension end.
+func parseExtension(scan *scanner) int {
+	start, end := scan.start, scan.end
+	switch scan.token[0] {
+	case 'u':
+		attrStart := end
+		scan.scan()
+		for last := []byte{}; len(scan.token) > 2; scan.scan() {
+			if bytes.Compare(scan.token, last) != -1 {
+				// Attributes are unsorted. Start over from scratch.
+				p := attrStart + 1
+				scan.next = p
+				attrs := [][]byte{}
+				for scan.scan(); len(scan.token) > 2; scan.scan() {
+					attrs = append(attrs, scan.token)
+					end = scan.end
+				}
+				sort.Sort(bytesSort{attrs, 3})
+				copy(scan.b[p:], bytes.Join(attrs, separator))
+				break
+			}
+			last = scan.token
+			end = scan.end
+		}
+		var last, key []byte
+		for attrEnd := end; len(scan.token) == 2; last = key {
+			key = scan.token
+			keyEnd := scan.end
+			end = scan.acceptMinSize(3)
+			// TODO: check key value validity
+			if keyEnd == end || bytes.Compare(key, last) != 1 {
+				// We have an invalid key or the keys are not sorted.
+				// Start scanning keys from scratch and reorder.
+				p := attrEnd + 1
+				scan.next = p
+				keys := [][]byte{}
+				for scan.scan(); len(scan.token) == 2; {
+					keyStart, keyEnd := scan.start, scan.end
+					end = scan.acceptMinSize(3)
+					if keyEnd != end {
+						keys = append(keys, scan.b[keyStart:end])
+					} else {
+						scan.setError(ErrSyntax)
+						end = keyStart
+					}
+				}
+				sort.Stable(bytesSort{keys, 2})
+				if n := len(keys); n > 0 {
+					k := 0
+					for i := 1; i < n; i++ {
+						if !bytes.Equal(keys[k][:2], keys[i][:2]) {
+							k++
+							keys[k] = keys[i]
+						} else if !bytes.Equal(keys[k], keys[i]) {
+							scan.setError(ErrDuplicateKey)
+						}
+					}
+					keys = keys[:k+1]
+				}
+				reordered := bytes.Join(keys, separator)
+				if e := p + len(reordered); e < end {
+					scan.deleteRange(e, end)
+					end = e
+				}
+				copy(scan.b[p:], reordered)
+				break
+			}
+		}
+	case 't':
+		scan.scan()
+		if n := len(scan.token); n >= 2 && n <= 3 && isAlpha(scan.token[1]) {
+			_, end = parseTag(scan)
+			scan.toLower(start, end)
+		}
+		for len(scan.token) == 2 && !isAlpha(scan.token[1]) {
+			end = scan.acceptMinSize(3)
+		}
+	case 'x':
+		end = scan.acceptMinSize(1)
+	default:
+		end = scan.acceptMinSize(2)
+	}
+	return end
+}
+
+// getExtension returns the name, body and end position of the extension.
+func getExtension(s string, p int) (end int, ext string) {
+	if s[p] == '-' {
+		p++
+	}
+	if s[p] == 'x' {
+		return len(s), s[p:]
+	}
+	end = nextExtension(s, p)
+	return end, s[p:end]
+}
+
+// nextExtension finds the next extension within the string, searching
+// for the -<char>- pattern from position p.
+// In the fast majority of cases, language tags will have at most
+// one extension and extensions tend to be small.
+func nextExtension(s string, p int) int {
+	for n := len(s) - 3; p < n; {
+		if s[p] == '-' {
+			if s[p+2] == '-' {
+				return p
+			}
+			p += 3
+		} else {
+			p++
+		}
+	}
+	return len(s)
+}

vendor/golang.org/x/text/internal/language/tags.go

@@ -0,0 +1,48 @@
+// Copyright 2013 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package language
+
+// MustParse is like Parse, but panics if the given BCP 47 tag cannot be parsed.
+// It simplifies safe initialization of Tag values.
+func MustParse(s string) Tag {
+	t, err := Parse(s)
+	if err != nil {
+		panic(err)
+	}
+	return t
+}
+
+// MustParseBase is like ParseBase, but panics if the given base cannot be parsed.
+// It simplifies safe initialization of Base values.
+func MustParseBase(s string) Language {
+	b, err := ParseBase(s)
+	if err != nil {
+		panic(err)
+	}
+	return b
+}
+
+// MustParseScript is like ParseScript, but panics if the given script cannot be
+// parsed. It simplifies safe initialization of Script values.
+func MustParseScript(s string) Script {
+	scr, err := ParseScript(s)
+	if err != nil {
+		panic(err)
+	}
+	return scr
+}
+
+// MustParseRegion is like ParseRegion, but panics if the given region cannot be
+// parsed. It simplifies safe initialization of Region values.
+func MustParseRegion(s string) Region {
+	r, err := ParseRegion(s)
+	if err != nil {
+		panic(err)
+	}
+	return r
+}
+
+// Und is the root language.
+var Und Tag

vendor/golang.org/x/text/language/common.go

@@ -1,16 +0,0 @@
-// This file was generated by go generate; DO NOT EDIT
-
-package language
-
-// This file contains code common to the maketables.go and the package code.
-
-// langAliasType is the type of an alias in langAliasMap.
-type langAliasType int8
-
-const (
-	langDeprecated langAliasType = iota
-	langMacro
-	langLegacy
-
-	langAliasTypeUnknown langAliasType = -1
-)

vendor/golang.org/x/text/language/coverage.go

@@ -7,6 +7,8 @@ package language
 import (
 	"fmt"
 	"sort"
+
+	"golang.org/x/text/internal/language"
 )
 
 // The Coverage interface is used to define the level of coverage of an
@@ -44,9 +46,9 @@ type allSubtags struct{}
 // consecutive range, it simply returns a slice of numbers in increasing order.
 // The "undefined" region is not returned.
 func (s allSubtags) Regions() []Region {
-	reg := make([]Region, numRegions)
+	reg := make([]Region, language.NumRegions)
 	for i := range reg {
-		reg[i] = Region{regionID(i + 1)}
+		reg[i] = Region{language.Region(i + 1)}
 	}
 	return reg
 }
@@ -55,9 +57,9 @@ func (s allSubtags) Regions() []Region {
 // consecutive range, it simply returns a slice of numbers in increasing order.
 // The "undefined" script is not returned.
 func (s allSubtags) Scripts() []Script {
-	scr := make([]Script, numScripts)
+	scr := make([]Script, language.NumScripts)
 	for i := range scr {
-		scr[i] = Script{scriptID(i + 1)}
+		scr[i] = Script{language.Script(i + 1)}
 	}
 	return scr
 }
@@ -65,22 +67,10 @@ func (s allSubtags) Scripts() []Script {
 // BaseLanguages returns the list of all supported base languages. It generates
 // the list by traversing the internal structures.
 func (s allSubtags) BaseLanguages() []Base {
-	base := make([]Base, 0, numLanguages)
-	for i := 0; i < langNoIndexOffset; i++ {
-		// We included "und" already for the value 0.
-		if i != nonCanonicalUnd {
-			base = append(base, Base{langID(i)})
-		}
-	}
-	i := langNoIndexOffset
-	for _, v := range langNoIndex {
-		for k := 0; k < 8; k++ {
-			if v&1 == 1 {
-				base = append(base, Base{langID(i)})
-			}
-			v >>= 1
-			i++
-		}
+	bs := language.BaseLanguages()
+	base := make([]Base, len(bs))
+	for i, b := range bs {
+		base[i] = Base{b}
 	}
 	return base
 }
@@ -134,7 +124,7 @@ func (s *coverage) BaseLanguages() []Base {
 		}
 		a := make([]Base, len(tags))
 		for i, t := range tags {
-			a[i] = Base{langID(t.lang)}
+			a[i] = Base{language.Language(t.lang())}
 		}
 		sort.Sort(bases(a))
 		k := 0

vendor/golang.org/x/text/language/display/display.go

@@ -15,8 +15,10 @@
 package display // import "golang.org/x/text/language/display"
 
 import (
+	"fmt"
 	"strings"
 
+	"golang.org/x/text/internal/format"
 	"golang.org/x/text/language"
 )
 
@@ -32,6 +34,65 @@ All fairly low priority at the moment:
     - Consider compressing infrequently used languages and decompress on demand.
 */
 
+// A Formatter formats a tag in the current language. It is used in conjunction
+// with the message package.
+type Formatter struct {
+	lookup func(tag int, x interface{}) string
+	x      interface{}
+}
+
+// Format implements "golang.org/x/text/internal/format".Formatter.
+func (f Formatter) Format(state format.State, verb rune) {
+	// TODO: there are a lot of inefficiencies in this code. Fix it when we
+	// language.Tag has embedded compact tags.
+	t := state.Language()
+	_, index, _ := matcher.Match(t)
+	str := f.lookup(index, f.x)
+	if str == "" {
+		// TODO: use language-specific punctuation.
+		// TODO: use codePattern instead of language?
+		if unknown := f.lookup(index, language.Und); unknown != "" {
+			fmt.Fprintf(state, "%v (%v)", unknown, f.x)
+		} else {
+			fmt.Fprintf(state, "[language: %v]", f.x)
+		}
+	} else {
+		state.Write([]byte(str))
+	}
+}
+
+// Language returns a Formatter that renders the name for lang in the
+// the current language. x may be a language.Base or a language.Tag.
+// It renders lang in the default language if no translation for the current
+// language is supported.
+func Language(lang interface{}) Formatter {
+	return Formatter{langFunc, lang}
+}
+
+// Region returns a Formatter that renders the name for region in the current
+// language. region may be a language.Region or a language.Tag.
+// It renders region in the default language if no translation for the current
+// language is supported.
+func Region(region interface{}) Formatter {
+	return Formatter{regionFunc, region}
+}
+
+// Script returns a Formatter that renders the name for script in the current
+// language. script may be a language.Script or a language.Tag.
+// It renders script in the default language if no translation for the current
+// language is supported.
+func Script(script interface{}) Formatter {
+	return Formatter{scriptFunc, script}
+}
+
+// Script returns a Formatter that renders the name for tag in the current
+// language. tag may be a language.Tag.
+// It renders tag in the default language if no translation for the current
+// language is supported.
+func Tag(tag interface{}) Formatter {
+	return Formatter{tagFunc, tag}
+}
+
 // A Namer is used to get the name for a given value, such as a Tag, Language,
 // Script or Region.
 type Namer interface {
@@ -84,6 +145,10 @@ func Languages(t language.Tag) Namer {
 
 type languageNamer int
 
+func langFunc(i int, x interface{}) string {
+	return nameLanguage(languageNamer(i), x)
+}
+
 func (n languageNamer) name(i int) string {
 	return lookup(langHeaders[:], int(n), i)
 }
@@ -116,6 +181,10 @@ func Scripts(t language.Tag) Namer {
 
 type scriptNamer int
 
+func scriptFunc(i int, x interface{}) string {
+	return nameScript(scriptNamer(i), x)
+}
+
 func (n scriptNamer) name(i int) string {
 	return lookup(scriptHeaders[:], int(n), i)
 }
@@ -140,6 +209,10 @@ func Regions(t language.Tag) Namer {
 
 type regionNamer int
 
+func regionFunc(i int, x interface{}) string {
+	return nameRegion(regionNamer(i), x)
+}
+
 func (n regionNamer) name(i int) string {
 	return lookup(regionHeaders[:], int(n), i)
 }
@@ -162,6 +235,10 @@ func Tags(t language.Tag) Namer {
 
 type tagNamer int
 
+func tagFunc(i int, x interface{}) string {
+	return nameTag(languageNamer(i), scriptNamer(i), regionNamer(i), x)
+}
+
 // Name implements the Namer interface for tag names.
 func (n tagNamer) Name(x interface{}) string {
 	return nameTag(languageNamer(n), scriptNamer(n), regionNamer(n), x)

vendor/golang.org/x/text/language/display/maketables.go

@@ -205,7 +205,13 @@ func (b *builder) generate() {
 	b.setData("lang", func(g *group, loc language.Tag, ldn *cldr.LocaleDisplayNames) {
 		if ldn.Languages != nil {
 			for _, v := range ldn.Languages.Language {
-				tag := tagForm.MustParse(v.Type)
+				lang := v.Type
+				if lang == "root" {
+					// We prefer the data from "und"
+					// TODO: allow both the data for root and und somehow.
+					continue
+				}
+				tag := tagForm.MustParse(lang)
 				if tags.contains(tag) {
 					g.set(loc, tag.String(), v.Data())
 				}