{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "Tce3stUlHN0L"
},
"source": [
"##### Copyright 2020 The TensorFlow Authors."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "tuOe1ymfHZPu"
},
"outputs": [],
"source": [
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qFdPvlXBOdUN"
},
"source": [
"# TF.Text Metrics"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MfBg1C5NB3X0"
},
"source": [
"\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
" \u003ctd\u003e\n",
" \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/text/tutorials/text_similarity\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
" \u003c/td\u003e\n",
" \u003ctd\u003e\n",
" \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/text/blob/master/docs/tutorials/text_similarity.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
" \u003c/td\u003e\n",
" \u003ctd\u003e\n",
" \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/text/blob/master/docs/tutorials/text_similarity.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView on GitHub\u003c/a\u003e\n",
" \u003c/td\u003e\n",
" \u003ctd\u003e\n",
" \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/text/docs/tutorials/text_similarity.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
" \u003c/td\u003e\n",
"\u003c/table\u003e"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xHxb-dlhMIzW"
},
"source": [
"## Overview\n",
"\n",
"TensorFlow Text provides a collection of text-metrics-related classes and ops ready to use with TensorFlow 2.0. The library contains implementations of text-similarity metrics such as ROUGE-L, required for automatic evaluation of text generation models.\n",
"\n",
"The benefit of using these ops in evaluating your models is that they are compatible with TPU evaluation and work nicely with TF streaming metric APIs."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MUXex9ctTuDB"
},
"source": [
"## Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "K_8D_DtQJ0kC"
},
"outputs": [],
"source": [
"!pip install -q tensorflow-text"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "IqR2PQG4ZaZ0"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"import tensorflow_text as text"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QKp40qS-DGEZ"
},
"source": [
"### ROUGE-L\n",
"\n",
"The Rouge-L metric is a score from 0 to 1 indicating how similar two sequences are, based on the length of the longest common subsequence (LCS). In particular, Rouge-L is the weighted harmonic mean (or f-measure) combining the LCS precision (the percentage of the hypothesis sequence covered by the LCS) and the LCS recall (the percentage of the reference sequence covered by the LCS).\n",
"\n",
"Source: https://www.microsoft.com/en-us/research/publication/rouge-a-package-for-automatic-evaluation-of-summaries/\n",
"\n",
"The TF.Text implementation returns the F-measure, Precision, and Recall for each (hypothesis, reference) pair.\n",
"\n",
"Consider the following hypothesis/reference pair:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WUgEkGHRKafG"
},
"outputs": [],
"source": [
"hypotheses = tf.ragged.constant([['captain', 'of', 'the', 'delta', 'flight'],\n",
" ['the', '1990', 'transcript']])\n",
"references = tf.ragged.constant([['delta', 'air', 'lines', 'flight'],\n",
" ['this', 'concludes', 'the', 'transcript']])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qeiXnY-_Khp1"
},
"source": [
"The hypotheses and references are expected to be tf.RaggedTensors of tokens. Tokens are required instead of raw sentences because no single tokenization strategy fits all tasks.\n",
"\n",
"Now we can call text.metrics.rouge_l and get our result back:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LS_NigzqKgtT"
},
"outputs": [],
"source": [
"result = text.metrics.rouge_l(hypotheses, references)\n",
"print('F-Measure: %s' % result.f_measure)\n",
"print('P-Measure: %s' % result.p_measure)\n",
"print('R-Measure: %s' % result.r_measure)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FQoprhImKoD0"
},
"source": [
"ROUGE-L has an additional hyperparameter, alpha, which determines the weight of the harmonic mean used for computing the F-Measure. Values closer to 0 treat Recall as more important and values closer to 1 treat Precision as more important. alpha defaults to .5, which corresponds to equal weight for Precision and Recall."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Q2ZnjOIgKnnS"
},
"outputs": [],
"source": [
"# Compute ROUGE-L with alpha=0\n",
"result = text.metrics.rouge_l(hypotheses, references, alpha=0)\n",
"print('F-Measure (alpha=0): %s' % result.f_measure)\n",
"print('P-Measure (alpha=0): %s' % result.p_measure)\n",
"print('R-Measure (alpha=0): %s' % result.r_measure)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "iYUYiLJhKseb"
},
"outputs": [],
"source": [
"# Compute ROUGE-L with alpha=1\n",
"result = text.metrics.rouge_l(hypotheses, references, alpha=1)\n",
"print('F-Measure (alpha=1): %s' % result.f_measure)\n",
"print('P-Measure (alpha=1): %s' % result.p_measure)\n",
"print('R-Measure (alpha=1): %s' % result.r_measure)"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [
"Tce3stUlHN0L"
],
"name": "text_similarity.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}