CANINE

This model was released on 2021-03-11 and added to Hugging Face Transformers on 2021-06-30.

CANINE

CANINE is a tokenization-free Transformer. It skips the usual step of splitting text into subwords or wordpieces and processes text character by character. That means it works directly with raw Unicode, making it especially useful for languages with complex or inconsistent tokenization rules and even noisy inputs like typos. Since working with characters means handling longer sequences, CANINE uses a smart trick. The model compresses the input early on (called downsampling) so the transformer doesn’t have to process every character individually. This keeps things fast and efficient.

You can find all the original CANINE checkpoints under the Google organization.

Tip

Click on the CANINE models in the right sidebar for more examples of how to apply CANINE to different language tasks.

The example below demonstrates how to generate embeddings with Pipeline, AutoModel, and from the command line.

import torch
from transformers import pipeline

pipeline = pipeline(
    task="feature-extraction",
    model="google/canine-c",
    device=0,
)

pipeline("Plant create energy through a process known as photosynthesis.")

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("google/canine-c")

text = "Plant create energy through a process known as photosynthesis."
input_ids = torch.tensor([[ord(char) for char in text]])

outputs = model(input_ids)
pooled_output = outputs.pooler_output
sequence_output = outputs.last_hidden_state

echo -e "Plant create energy through a process known as photosynthesis." | transformers run --task feature-extraction --model google/canine-c --device 0

Notes

• CANINE skips tokenization entirely — it works directly on raw characters, not subwords. You can use it with or without a tokenizer. For batched inference and training, it is recommended to use the tokenizer to pad and truncate all sequences to the same length.

  from transformers import AutoTokenizer, AutoModel
  
  tokenizer = AutoTokenizer("google/canine-c")
  inputs = ["Life is like a box of chocolates.", "You never know what you gonna get."]
  encoding = tokenizer(inputs, padding="longest", truncation=True, return_tensors="pt")

• CANINE is primarily designed to be fine-tuned on a downstream task. The pretrained model can be used for either masked language modeling or next sentence prediction.

CanineConfig

[[autodoc]] CanineConfig

CanineTokenizer

[[autodoc]] CanineTokenizer - build_inputs_with_special_tokens - get_special_tokens_mask - create_token_type_ids_from_sequences

CANINE specific outputs

[[autodoc]] models.canine.modeling_canine.CanineModelOutputWithPooling

CanineModel

[[autodoc]] CanineModel - forward

CanineForSequenceClassification

[[autodoc]] CanineForSequenceClassification - forward

CanineForMultipleChoice

[[autodoc]] CanineForMultipleChoice - forward

CanineForTokenClassification

[[autodoc]] CanineForTokenClassification - forward

CanineForQuestionAnswering

[[autodoc]] CanineForQuestionAnswering - forward