Appendix merged
IM8/SNDGO guidelines on the use of Cloud Services for Gen AI applications
Note: While we aim to share relevant IM8/SNDGO guidelines here, we implore readers to independently verify for themselves before proceeding as guidelines may have been updated since the time of publication.
According to the circular by SNDGO1, large language model based applications (such as RAG) must only ingest data that will not cause damage to an agency (i.e. up to OFFICIAL (CLOSED) / NON-SENSITIVE). If the RAG application requires a knowledge base beyond OFFICIAL (CLOSED) / NON-SENSITIVE, approval from the AI Policy Group (SNDGG) has to be sought. Approval will take into account whether any data in transit for the system are encrypted end-to-end, and whether no data will be logged within overseas servers. In addition, measures must be in place to ensure that failures or disruptions to the RAG system do not cause failures or disruptions to NE-critical, high criticality (CII), or high significance systems (SII). The system should abide by existing IM8 cybersecurity policies on connectivity ot Government Enterprise Networks (GEN) if GEN connectivity is required.
The system should also include the following categories of risk-mitigating measures to address the accuracy and accountability risks inherent in the use of large language models in government applications:
Education The system should include visual UX cues to educate users on proper use. This includes informing users that the agency (owning the RAG system) is piloting the use of large language models, that the results are AI-generated, and that the users should always double check and adapt generated output for use.
Minimise hallucination Steps should be taken to minimise hallucinations for the RAG system. Measures include limiting output to information drawn from a reliable knowledge base, enabling the user to verify the output with relevant citations, and conducting accuracy / robustness tests for a range of commonly asked questions. The RAG system should ensure that the model temperature for the large language model used internally should be set to 0, as this will also reduce the likelihood of hallucinations.
Adverserial testing Tests should be conducted to ensure that the RAG system performs satisfactorily against queries deliberately designed to elicit inappropriate responses. There should be a comprehensive plan to systematically test and measure the system’s performance (read the section on evaluation), recommended to be performed by a red-teaming regime. The RAG system should consider rate limiting queries to deter users from “brute force” approaches to following the system.
Data Residency and Cost for CSP's Solutions on GCC
Vertex AI on GCC
- Generative AI on Vertex AI — which includes PaLM 2, Codey, and Imagen models, as well as the Text Embeddings and Multimodal Embeddings APIs — offers customers data store options for
- North America: US and Canada
- Europe: Netherlands, France, UK, Germany, and Belgium
- Asia: Japan, Singapore, and Korea
- Vertex AI Search offers customers data store options for
- North America: US (multi-region)
- Europe: EU (multi-region)
- Vertex AI Conversation offers customers data store options for
- North America: US (multi-region)
- Europe: EU (multi-region)
For further information, see here
Amazon Q and Bedrock on GCC
Tools for Evaluating Models on GCC
Vertex AI Model Eval and AutoSxS
Vertex AI provides two different tools for evaluating models
- ModelEval evaluates models on certain metrics like AUC, LogLoss and provides results
- AutoSxS compares results from two models and determines which one is the better answer
Note
There are no screenshots for this as currently the only way to invoke this is via code.
AWS Bedrock
Note
Currently only available in selected regions like N. Virginia
AWS Bedrock provides a simple tool to perform model evaluation based on metrics like accuracy and toxicity. It also leverages AWS SageMaker to add in human loops for manual human evaluation, which can be provided by the user or by AWS via AWS Mechanical Turk.
Setting up an evaluation task
Step 1: Select an evaluation task
Step 2: Configure the evaluation parameters (automatic evaluation in this example) Note the available evaluation task types.
Step 3: Create and wait for evaluation to complete
Step 4: View results The web console provides a summary view of the evaluation results for the metrics like accuracy and toxicity.
A more detailed per-entry evaluation result can be found in the output file
For human based evaluation, your humans will get an annotation task to manually provide evaluation results, instead of it being system generated and tabulated
Dataset Requirements
See here for the latest information.
Azure AI Studio
Note that Azure AI Studio is different from Azure OpenAI Studio
AI Studio provides an evaluation service to measure how well your model performs on primarily Q&A use cases.
Setting up an evaluation task
Step 1: Select the type of evaluation task
Step 2: Select the metrics to measure
Step 3: Upload dataset and map columns
Step 4: Run the evaluation
Step 5: View Results
For evaluation on Q&A without context, the tool helps provide an assessment of how well the model answer fits with relevance and groundedness against the source data.
For evaluation on Q&A with context, the tool also provides a summary of the metric scores obtained
There is also the option of setting up a manual evaluation task where users will manually evaluate each entry and provide a thumbs up or thumbs down
Dataset Requirements
The dataset should contain minimally the question, answer, and ground truth columns. A context column is also required depending on the evaluation task selected.
Fine-Tuning Methods
Prompt Tuning
Complexity: High
Background: Prompt tuning, prefix tuning and p-tuning are variants of the same approach - tunable, auxiliary embeddings are concatenated to the original prompt embeddings to engineer a better prompt. It is important to note that these are not human readable prompts - they are added to the input query sequence after sequence embedding is performed. Because these auxiliary embeddings are trainable, they can be fine-tuned to improve prompt quality and LLM output for specific downstream tasks. Prompt tuning can be considered a form of Parameter Efficient Fine-Tuning.
Methods:
Prompt Tuning1
Tunable input embeddings are prepended to embeddings of the original prompt.
Prefix Tuning
Tunable embeddings, fed through a regular MLP, are prepended to both the input layer and transformer layers in the network. Conceptually similar to prompt tuning, but performed in several layers rather than just the input layer.
p-tuning23
Tunable embeddings, fed through an LSTM or MLP, are inserted into the input layer. Conceptually similar to prompt tuning, but auxiliary embeddings are not necessarily prepended. The authors find that this improves stability in LLM outputs to small discrete changes in prompts.
p-tuning V245
The primary difference between p-tuning V2 and the original p-tuning is that p-tuning V2 introduces tunable embeddings to every layer, not just the input layer. Conceptually similar to prefix tuning and p-tuning.
Parameter Efficient Fine-tuning
Complexity: High
Background: Fine-tuning entails updating the weights of a pre-trained LLM using task-specific data to improve its in-domain performance. There are multiple variations to the technique - parameter-efficient fine-tuning only updates a small subset of model weights for computational efficiency, while full fine-tuning requires updating all model weights.
Methods:
LoRA and QLora6
Low-Rank Adaption (LoRA) is a fine-tuning approach stemming from the insight that model weights, which are large matrices, can be decomposed into smaller matrices while still approximately preserving the information they contain7 . QLora is an extension of LoRA that uses quantisation to further improve the efficiency of LoRA.
For instance, a 100x100 matrix decomposed into two separate 100x2 matrices has significantly fewer parameters; 10000 (100x100) vs 400 (100x2x2). The authors of the paper find that, in their tests on GPT-3 175B, using just 0.02% of the total parameters for LoRA produces results superior to full fine-tuning.
More specifically, low-rank decompositions are used to approximate the updates to model weights during fine-tuning, not the weights themselves8. The original model weights are frozen and only the deltas are fine-tuned. These updates can then be added to the model’s original weights. At training, this significantly reduces the computational requirements for fine-tuning a model. At inference, unlike other methods (prefix tuning, adapters, etc) LoRA does not increase latency because there are no additional parameters added to the model.
IA3910
IA311 performs fine-tuning by introducing trainable task-specific vectors to the key, value and feed-forward layers of each transformer block in an encoder-decoder transformer12 . Similar to LoRA, IA3 comes at no additional latency at inference because the total number of parameters in the model are not changed. Additionally, IA3 also allows for mixed-task batches as the activations of each task in the batch can be separately multiplied by its associated learned task vector. Compared to LoRA, IA3 requires even fewer parameters (0.01% vs >0.1% of total parameters in LoRA) while achieving superior performance based on their tests13.
Full Fine-tuning
Complexity: High
Background: In its simplest form, full fine-tuning of a model entails supervised learning over a domain-specific dataset or task. Because all model weights are being optimized, for larger models this entails a tremendous number of gradient calculations and large model checkpoints, making full fine-tuning very costly in both compute and storage. Unless a task or dataset falls significantly outside the domain of a pre-trained LLM, full fine-tuning is unlikely to be necessary.
Methods:
Reinforcement Learning with Human Feedback (RLHF)
RLHF is a fine-tuning technique with the specific goal of aligning models with some framework of human preference. RLHF is used to fine-tune ChatGPT and GPT-4 to ensure they are helpful, honest and harmless. It proceeds in the following steps:
- A set of actual prompts is sampled from the input prompt distribution and manually labeled by human evaluators. This initial dataset is used in a regular supervised learning procedure to fine-tune the model14, which we call the SFT model.
- A new set of prompts is sampled and several outputs are generated from the SFT model. These responses are ranked by human evaluators, and a reward model15 (RM) is trained to predict the higher ranked output between pairs of outputs16.
- A new set of prompts is sampled and outputs are generated from the SFT model and scored by the RM. The PPO algorithm17 is used to iteratively optimize the SFT model, which in reinforcement learning terms is referred to as the ‘policy’.
Dataset Generation
In order to develop, iterate and improve on RAG pipeline performance, a clear framework for experimentation is needed. While well-defined metrics are fundamental to performance benchmarking, equally important is the availability of an appropriate dataset. In the context of applying RAG in government, the ideal dataset should 1) reflect realistic use cases by public officers 2) encapsulate the complexity of public policy.
A Brief Overview of Synthetic Data Generation
As is often the case in ML problems in the real world, large and accurately labeled datasets are a luxury seldom available. In these cases, synthetic data generation is indispensable. When training LLMs, causal language modeling tasks (predicting the next word using only the previous words) and masked language modeling tasks (predicting a randomly masked word within a sentence) are examples of producing labels from unstructured text data.
RAG is fundamentally a question-answering technology, so evaluating RAG pipelines requires valid question-answer pairs that are not easily collated and not straightforward to synthetically generate. Importantly, question-answer pairs must be (mostly) correct, if not it runs the risk of unfair or inaccurate evaluation of RAG pipelines. Unlike causal and masked language modeling where the answer is obvious (we know exactly what word is missing from a sentence), RAG requires a valid ground truth and a well-defined question that asks about that ground truth.
An increasingly popular approach is to reverse-engineer question-answer pairs from documents by leveraging LLMs. The approach is as follows:
- Chunk and index a selected document base
- Sample N contexts from the index
-
For each sampled context, instruct an LLM to generate a question that can be answered entirely by the context
- E.g. A sampled context might be "Singapore's full-year headline inflation was 4.8% in 2023, down from 6.2% in 2022", a valid question might be "What was Singapore's headline inflation in 2023?" and the correct answer is "4.8%"
- Evaluate the RAG against the same document base used to generate the question-answer pairs
One popular package for this is Ragas, which enhances the data generation process by iteratively increasing the complexity of questions using subsequent LLM calls. This is done in two ways:
- When sampling contexts, a filtering step is introduced by prompting LLMs to determine if the sampled context and generated question are suitable, if not they are amended or replaced
-
When generating questions, a base question is first generated then transformed into a more complex question
- E.g. "Context: What was Singapore's headline inflation in 2023 and 2022? Answer: 4.8% in 2023 and 6.2% in 2022" might be transformed into a reasoning question like "Context: How did headline inflation in Singapore change from 2022 to 2023? Answer: Headline inflation decreased from 6.2% in 2022 to 4.8% in 2023"
Figure: Diagram of Ragas’ approach to synthetic dataset generation
Limitations
Documents from Hansard were selected as an evaluation dataset because of its broad coverage of public policy and its realistic government use cases. To manage the scale of experiments, only Hansard documents after 2022 were used.
Additionally, documents from Supreme Court Judgements were also used to generate a separate evaluation dataset from a more structured source. Only documents after 2020 were used.
Several issues emerged from using the Ragas package to generate synthetic data:
-
Ragas has a tendency to generate question-answer pairs that either were contrived or not meaningful.
- E.g. The context "Mr Chua Kheng Wee Louis asked the Minister for Foreign Affairs (a) whether he can provide an update on the Ministry's discussions with China on the bilateral visa-free travel arrangement." might generate questions like "What did Mr Chua Kheng Wee Louis ask the Minister for Foreign Affairs?" (not meaningful) or "Can the Minister for Foreign Affairs provide an update on the Ministry's discussions with China on the bilateral visa-free travel arrangement?" (contrived).
- Similarly in the judiciary dataset, using specific text chunks to generate questions lead to questions that were contrived and lacked context.
-
Ragas' generated questions do not provide the full picture with regards to complex policy issues.
- E.g. The context "To help Singaporeans manage the rising cost-of-living, every Singaporean household will receive $500 CDC Vouchers in 2024" might yield the question "What is the government doing to address the rising cost-of-living?" with the answer "They are issuing $500 in CDC Vouchers in 2024.", which is factually correct but just a single instance of cost-of-living related policies.
- Similarly, Supreme Court Judgements can sometimes be extremely long documents with hundreds of thousands of tokens. Questions generated from small chunks of this document do not have access to the broader context of the case.
-
Ragas' approach to filtering contexts and questions relies on LLMs to give a 0-1 score based on some criteria, which is a task that LLMs are poor at. This also significantly inflates latency and tokens consumed by the Ragas algorithm.
- Anecdotally, we observed inconsistent judging where the same context was sometimes deemed suitable and other times in appropriate
Our Approach
Hansard
The approach Ragas (and similar packages) uses for synthetic data generation might work well in simpler contexts (e.g. laws, rules, manuals, structured data, where the full answer is entailed in a single context), but is not well-suited to datasets like Hansard. Based on our observations, there is also limited merit in the filtering approach that Ragas takes due to the inconsistency of LLMs in providing objective, score-based judgements. Instead of relying on available packages, we take a simpler multi-context approach for question generation based on a custom Langchain pipeline.
- Chunk and index Hansard documents
- Determine a set of common topics in Singapore public policy; for example, "Cost-of-living", "COE", "Smart Nation Initiative"
- For each topic, retrieve N (we used 25) documents from the index
- Based on the retrieved documents, instruct an LLM to generate 5-10 questions that can be answered using multiple retrieved contexts
In total, 196 questions were generated. The following additions also helped to improve question quality:
-
Further preprocessing of Hansard documents to remove redundant text
- The first paragraph of each Hansard document typically has a summary, paraphrasing questions addressed to relevant Ministers. These chunks are semantically similar to queries (because they are also questions), but do not actually contain meaningful information. In testing, they were often retrieved but only served to muddle the quality of questions generated, so they are removed.
-
Ensemble retrieval using document titles and document chunks. The titles of Hansard documents are often extremely informative about their content, this helps to retrieve chunks that contain relevant information but may not be retrieved when using basic semantic search.
- Few-shot prompting with clear instructions in the generation prompt, specifically:
- Strict adherence to no paraphrasing of questions already in the contexts
- Only focusing on statements of fact about public policy
- Clear reference to dates and relevant individuals
- Enhancement of documents with metadata like parliamentary seating date and title
An important lesson here is that our synthetic dataset generation approach benefits just as much from RAG pipeline improvements as a regular RAG pipeline. In order to generate the most relevant questions, the most relevant contexts to a given topic must first be retrieved, along with clearly defined rules for questions to follow.
Judiciary
Question generation in Hansard must be multi-context to be realistic. This is less likely to be the case for supreme court judgements, where meaningful questions can still be generated from individual judgements. Hence, we generate a synthetic dataset based on the judiciary dataset with question-answer pairs coming from single documents, not multiple ones.
This single-context approach makes question generation simpler. However, some judiciary documents are extremely long, going well beyond the context window for GPT-4 32K. The immediate option is to chunk each document then generate questions based on individual chunks. As we observed in the Hansard dataset, questions based on specific chunks have the tendency to be contrived or lack context, particularly when prompting the LLM to use only information from the provided chunk. The following approach is used instead:
-
Apply the following filters:
- Documents after 2022 (to keep same size small)
- Documents with less than 30000 tokens (to keep costs and time down)
-
Documents for cases with only one document
- For example, a specific case, Person A v Person B, might have multiple documents. We want to exclude these because they are more likely to reference information beyond the current document, making evaluation more difficult
-
Sample documents randomly from the document subset
-
For each document, generate a summary using LangChain's summarization chain with the 'refine' option.
- Under the 'refine' method, we first chunk the document, then sequentially take in each chunk in chronological order to create a running summary. This ensures that all information in the document is ingested, and the LLM is prompted to include new information from subsequent chunks to ensure exhaustiveness.
-
For each document summary, prompt an LLM to generate 3 questions based entirely on the summary and metadata.
In total, 90 documents were sampled with 3 questions per document, resulting in 270 questions.
-
https://arxiv.org/pdf/2103.10385.pdf ↩
-
https://github.com/THUDM/P-tuning ↩
-
https://arxiv.org/pdf/2110.07602.pdf ↩
-
https://github.com/THUDM/P-tuning-v2 ↩
-
https://arxiv.org/pdf/2106.09685.pdf ↩
-
https://arxiv.org/pdf/2012.13255.pdf ↩
-
In the paper’s testing, targeting key and query matrices of the transformer layers performed best. ↩
-
https://arxiv.org/pdf/2205.05638.pdf ↩
-
https://huggingface.co/docs/peft/en/conceptual_guides/ia3 ↩
-
As the name suggests, IA3, which stands for Infused Adapter by Inhibiting and Amplifying Inner Activations, is a variation of one of the first PEFT techniques which utilized Adapter modules added to transformer blocks for fine-tuning. ↩
-
In the paper, IA3 is implemented together with modifications to the T0-3B model’s loss functions, which add a length normalization and unlikelihood loss. However, these should be independent of the actual mechanism for IA3. ↩
-
It should be noted that IA3 was tested on T0-3B whereas LoRA was tested on GPT-3 175B (and several other models), and on different datasets. T0-3B is also an encoder-decoder style model whereas popular models like Llama and GPT are decoder only, so it is not clear if results carry over. ↩
-
OpenAI’s RLHF paper notes that, for their first iteration of RLHF, prompts were manually written by labelers rather than sampled. ↩
-
For the RM, OpenAI used an SFT GPT-3 6B with the final unembedding layer removed. ↩
-
Assuming there are 4 generated responses, these are ranked 1>2>3>4. Then, pairs are chosen to train the RM, leading to 4C2 data points. ↩
-
https://huggingface.co/learn/deep-rl-course/unit8/intuition-behind-ppo ↩