Google has consistently remained at the forefront of AI research, leveraging its vast resources and employing a substantial number of top-talent engineers. However, in terms of language models, Google’s efforts were late to the game.
With tech giant Microsoft already benefitting from a fruitful partnership with OpenAI, Google had no choice but to catch up.
At this year’s Google I/O conference, the company announced its answer to the generative AI arms race: PaLM 2. Will this new model measure up in performance alongside OpenAI’s GPT-4?
What is PaLM 2?
Google describes PaLM 2 as a state-of-the-art language model that improves on their existing PaLM model first announced in 2022. Similar to other language models, PaLM 2 is able to perform various text generation tasks such as PaLM is capable of a wide range of tasks, including answering questions, translating text, generating code, and much more.
Tests have shown that the PaLM 2 already shows significant improvements, outperforming the PaLM model while using a much lower number of parameters.
PaLM 2 is a Family of Models
Like other language models, the PaLM 2 project is actually a family of models that range in size. Google will provide the PaLM 2 model in four sizes: Gecko, Otter, Bison, and Unicorn.
The variety in sizes makes it easy to deploy PaLM 2 in various use cases. For example, the Gecko model is lightweight enough that the entire model can fit in a mobile device and even run offline.
PaLM 2’s Training Dataset
One of the most important aspects of a successful language model is the training dataset. The training dataset must be diverse enough to allow the model to have a deep understanding of the subject matter it is designed for.
For large language models (LLMs), there is typically no specific topic the model must train on. LLMs are instead built to be general-purpose models that must be fit to perform a wide number of tasks. These models use large textual datasets that capture a large portion of the web as well as published reference material, literature, and even source code.
The main difference between PaLM 2’s training dataset and other models is the inclusion of a higher percentage of non-English data. According to their technical report, expanding the dataset to include non-English texts exposes the model to a wider variety of languages and cultures.
The PaLM 2 model was also trained on parallel multilingual data to help the model gain the ability to translate from one language to another. The data includes pairs of text where one entry is in English and the other is an equivalent text in another language.
The table above shows the language distribution of the multilingual web documents used to train PaLM 2.
PaLM 2 Key Features
Here are some of the main areas that PaLM 2 excels at compared to other language models.
Reasoning
PaLM 2’s dataset includes sources such as scientific papers and web content with mathematical expressions. This gives the model improved capabilities in mathematics, common sense reasoning, and logic.
Researchers tested the model’s mathematical reasoning abilities on grade school and high school math questions where it shows comparable results to GPT-4’s math capabilities.
Coding
PaLM 2’s training data also gives it the ability to generate code in a variety of programming languages. The PALM 2 team created a coding-specific PaLM 2 model called PaLM 2-S* which was trained on a code-heavy multilingual dataset.
Not only is the model capable of code generation, but it is also able to handle tasks that involve multiple languages. For example, you can ask PaLM 2 to create a Python sorting function that adds line-by-line comments in Spanish.
Multilinguality
Since the model was trained on a dataset that includes over 100 languages, PaLM 2 shows proficiency in understanding, generating, and translating text across multiple languages.
To test multilinguality, the researchers tested the model on various language proficiency tests in different languages. The results show that not only does PaLM 2 outperform PaLM but also achieved a passing grade for every evaluated language.
PaLM 2 also shows its multilingual capabilities by its ability to understand idioms in different languages, explaining jokes, fixing typos, and can even learn how to convert formal text to colloquial chat.
PaLM 2 Powers Google Products
Google is already taking advantage of PaLM 2’s advancements by integrating the model with other products.
Bard
The model’s ability to handle multilingual tasks is now powering Google’s Bard experiment as it expands to over 180 countries and territories.
Bard is now also using PaLM 2’s coding capabilities to assist in programming and software development tasks such as code generation and code debugging.
Duet AI for Google Workspace
Google is also planning on adding generative AI features to its Google Workspace group of applications. Gmail and Docs will soon include a feature called Duet AI that will help the user draft their replies and writing using prompts.
Duet AI will also allow users to create custom plans in Google Sheets for tasks and projects based on prompts given by the user.
Conclusion
Google is surely hoping to close the gap in the market of AI language tools with their PaLM 2 language model. While the model’sAPI is not yet publicly available, the results from their research show that the model is competitive enough to match GPT-4’s performance.
With Google’s existing user base, they certainly have the advantage of massive adaption if their AI becomes integrated into their services such as their search engine or their suite of productivity tools.
Leave a Reply