Skip to main content
Version: v3.0.0

Running an Optimize:ai Scan

The Trustwise Optimize:ai portal allows you to conduct detailed safety, alignment, and performance scans on your LLMs. Follow these steps to run an Optimize AI scan effectively:

Step 1: Access the Trustwise Control Plane​

Navigate to the Trustwise Control Plane by following this link. This is your starting point for setting up and running performance scans.

Trustwise Control Plane

Step 2: Enter API Keys​

  1. Enter OpenAI API Key: Provide your OpenAI API key in the designated field. This key is required to access models hosted by OpenAI.
  2. Enter Together AI Key (Optional): If you are using Together AI models, enter the Together AI key as well.
Enter API Key

Step 3: Select Dataset​

Choose the dataset from the dropdown that best suits the context of the scan. Current options include:

  • Home Insurance Policy Coverage
  • NHS Covid Pass
  • ACH Treasury
Select Dataset

Step 4: Choose Embedding Model​

Select an embedding model to use for the scan. Current options include:

  • bge-small-en-v1.5
  • openai
Choose Embedding Model

Step 5: Set Policies and Preferences​

  • Cost Policies: Select the appropriate cost policy for the scan. Currently, the only option available is TW-Model-Cost-Policy.
Cost Policies
  • Business and Regulatory Policies: Choose one or more of the available policies. These policies currently include
    • ISO SCI 1.0
    • NIST AI RMF
    • UK FCA Consumer Duty
    • EU AI Act
Business and Regulatory Policies
  • Energy & Emission Maps: Select the appropriate energy and emission map for the scan. Currently, the only option available is TW-Energy-Emission-Map.
Energy & Emission Maps

Step 6: Configure Infrastructure​

  • Infrastructure Provider: Choose one of the cloud providers. Currently, the only option available is gcp.
    • Coming soon: aws, azure
  • Processor Name: Select a processor from the list provided, which will depend on the chosen cloud provider.
  • Cloud Instance Type: Choose an instance type that fits your performance and cost requirements. Available instances will also depend on the chosen provider.
  • Regions: Select the geographic region of the cloud provider that best fits your needs. Available regions will also depend on the chosen provider.
Configure Infrastructure

Step 7: Safety and Alignment Evaluation​

  • Turn on the toggles for Safety Evaluation and Alignment Evaluation to ensure these metrics are included in the scan results. Users can also configure individual thresholds for Safety and Alignment to make the scan better fit for their unique needs.
Safety and Alignment Evaluation

Step 8: Upload Queries and Select LLMs​

  • Add Your Queries: Add queries for which you would like to run the Optimize:ai scan.
Upload Queries
  • Select LLMs: Choose different LLMs to run the performance scan. If you have selected LLMs hosted by Together AI, ensure that the Together AI key is provided.
Select LLMs

Step 9: Submit Scan​

Once all configurations are set, click the "Submit" button to start the scan. You will be able to monitor the progress and eventually analyze the results directly from the Control Plane.

Note​

Ensure all entries are correct, as mistakes may affect the accuracy of the scan results. For any assistance with setting up the scans, please contact our support team at support@trustwise.ai.