Climb the (bed)rock with Python, Javascript and GO



Bedrock is now available in eu-central-1. It’s time to get real and use it in applications. Reading all blog posts about Bedrock, you might get the impression that Python and LangChain is the only way to do it. Quite the opposite! As Bedrock makes calling the models available as AWS API, all AWS SDKs are supported! This post shows how to use Bedrock with Python, Javascript and GO.

A pragmatic view on Bedrock

Bedrock has two parts:

  1. The Amazon Bedrock Runtime to invoke the models which are available
  2. The Amazon Bedrock API

Most things from the bedrock API, like fine-tuning are still in preview (October 2023). Let’s wait after re:Invent to see what is GA.

So, we take what we have now and invoke models! In Frankfurt, we have 2 foundation models available, Titan embedding and Claude V1.2 / V2. We call the AWS API, and AWS calls the foundation model.

bedrock This is the AWS call.

bedrock Then, AWS calls the foundation model. The models’ parameters are defined in the body of the AWS call. The body is not typed in the AWS API. It is a JSON object as a string. In this example, we call a Claude model. The parameters are defined in the Claude API.

Steps to invoke completion

  1. Decide model
  2. Design prompt
  3. Prepare model body
  4. Call API
  5. Decode response

Decide model

Soon, we will have more models available. For now, we have Claude and Titan in Frankfurt. Claude is an all-purpose text model, and Titan is an embedding model. With more choices, you must look at the model scorecards to get information about the model.

Then check:

  • Is the task you want to achieve supported? Like text completion, summarization, translation, question answering, etc.
  • Is the language you want to use supported? Like English, German, French, etc.

Here, we decide anthropic.claude-instant-v1. It supports completion and English/German. In the Bedrock console, you see the “Supported use cases” for each model.

Design the prompt

See Claude API prompt desgin

Prompt engineering is a knowledge domain which is constantly growing. Look for new patterns and best practices.

In the prompt we will use an instruction: “Write the top 3 differences in markdown.” and a document part to create the prompt. The text is: “Compare JSON to XML.”

Prepare model body

See Claude API

type ClaudeV1 struct {
	Prompt            string   `json:"prompt"`
	MaxTokensToSample int      `json:"max_tokens_to_sample"`
	Temperature       float64  `json:"temperature"`
	TopK              int      `json:"top_k"`
	TopP              float64  `json:"top_p"`
	StopSequences     []string `json:"stop_sequences"`
	AnthropicVersion  ModelId  `json:"anthropic_version"`
}

As Python and JavaScript are not typed by default, I use this go struct to show the types. The parameter are descriped in the Claude completion API.

The Bedrock service uses InvokeModel to call the Claude `completion API.

See go-bedrock/claude/claude.go on github.

Call Parameters GO

With the prepared structure we can create the body for the AWS call:

completePrompt := fmt.Sprintf("Human: \n\n%s\n\n\nAssistant:", prompt)

	body := &claude.ClaudeV1{
		Prompt:            completePrompt,
		MaxTokensToSample: 1024,
		Temperature:       0,
		TopK:              250,
		TopP:              0.999,
		StopSequences:     []string{"\n\nHuman:"},
		AnthropicVersion:  "bedrock-2023-05-31",
	}

See go-bedrock/claude/claude.go on github.

With GO we do byteSlice, err := json.Marshal(body) to create the body as JSON string.

Call Parameters Python

First, we define the prompt and the body:

prompt = f"Human: \n\n{instruction}\n\n\nAssistant:"

body = {
	"prompt": prompt,
	"max_tokens_to_sample": 1024,
	"temperature": 0,
	"top_p": 250,
	"top_k": 0.999,
	"stop_sequences": ["\n\nHuman:"],
	"anthropic_version": "bedrock-2023-05-31",
}

Then we do a json.dumps(body) to create the body.

Call Parameters JavaScript

First, create the body:

let instruction = "Compare JSON to XML. Write the top 3 differences in markdown."
let complete_prompt = `Human: \n\n${instruction}\n\n\nAssistant:`;

const body = {
  "prompt": complete_prompt,
  "max_tokens_to_sample": 1024,
  "temperature": 0,
  "top_k": 250,
  "top_p": 0.999,
  "stop_sequences": ["\n\nHuman:"],
  "anthropic_version": "bedrock-2023-05-31",
}

Then we do a JSON.stringify(body) to create the body:

const buffer = Buffer.from(JSON.stringify(body));

Call api

See Bedrock Runtime API.

Calling the API is quite similar in all languages.

GO:

response, err := client.InvokeModel(context.TODO(), &bedrockruntime.InvokeModelInput{
		ModelId:     aws.String(string(claude.ModelIdClaudeInstantV1)),
		ContentType: aws.String("application/json"),
		Accept:      aws.String("*/*"),
		Body:        byteSlice,
	})

Python:

response = client.invoke_model(body=body,
    modelId=modelId,
    accept=accept,
    contentType=contentType
)

JavaScript:

const command = new InvokeModelCommand(input);
try {
  const response = await client.send(command);
}

Decode response

Each model will give a different JSON structure as an answer. Here, we use Claude, so the answer is:

type Response struct {
	Completion string `json:"completion"`
	StopReason string `json:"stop_reason"`
	Model      string `json:"model"`
}

If there is an error, a different JSON structure is returned:

type ResponseError struct {
	Error struct {
		Type    string `json:"type"`
		Message string `json:"message"`
	} `json:"error"`
}

Climb

I have prepared a taskfile, so you can call task run` in each subdirectory to run the code.

GO

With the first time call, the app is built:

[init] go build -o dist/claude main/main.go

Calling run the second time:

cd go-bedrock
time task run                                                                                                                               ─╯
task: Task "init" is up to date
Calling Claude...
Response:
 ### Top 3 Differences Between JSON and XML

1. **Structure** - JSON has a simpler structure than XML. JSON uses key/value pairs and does not require closing tags. XML requires opening and closing tags.

2. **Readability** - JSON is generally considered more human-readable than XML due to its simpler structure. XML tags can sometimes obscure the meaning.

3. **Size** - JSON tends to be more compact than XML since it does not require as much formatting. This makes JSON preferable for data transmission, especially on mobile networks.
task run  0,02s user 0,02s system 1% cpu 2,393 total

Python

With the first time call, the app is built:

cd python-bedrock
task run

This takes some time, because the virtual environment is created and the dependencies are installed:

Requirement already satisfied: pip in ./.venv/lib/python3.11/site-packages (23.0.1)
Collecting pip
  Downloading pip-23.3.1-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 15.2 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.0.1
    Uninstalling pip-23.0.1:
      Successfully uninstalled pip-23.0.1
...Many more lines

Calling run the second time:

time task run                                                                                                                               ─╯
JSON:  {"prompt": "Human: \n\nCompare JSON to XML. Write the top 3 differences in markdown.\n\n\nAssistant:", "max_tokens_to_sample": 1024, "temperature": 0, "top_p": 250, "top_k": 0.999, "stop_sequences": ["\n\nHuman:"], "anthropic_version": "bedrock-2023-05-31"}
Response:  {"completion":" ### Top 3 Differences Between JSON and XML\n\n1. **Structure** - JSON has a simpler structure than XML. JSON uses key/value pairs and does not require closing tags. XML requires opening and closing tags.\n\n2. **Readability** - JSON is generally considered more human-readable than XML due to its simpler structure. XML tags can sometimes obscure the meaning. \n\n3. **Size** - JSON tends to be more compact than XML since it does not require as much markup. This makes JSON lighter and faster to parse. XML documents can become very large with all the required tags.","stop_reason":"stop_sequence"}
task run  0,28s user 0,11s system 14% cpu 2,642 total

Here the full JSON objects are printed.

JavaScript

With the first time call, the app is built:

cd js-bedrock
task run
(#########⠂⠂⠂⠂⠂⠂⠂⠂⠂) ⠋ idealTree:@aws-crypto/sha256-js: timing idealTree:node_modules/@aws-crypto/sha256-js Completed in 1ms

JavaScript has the nicest animation while downloading the depedencies :) .

Calling run the second time:

time task run                                                                                                                               ─╯
Response:

 ### Top 3 Differences Between JSON and XML

1. **Structure** - JSON has a simpler structure than XML. JSON uses key/value pairs and does not require closing tags. XML requires opening and closing tags.

2. **Readability** - JSON is generally considered more human-readable than XML due to its simpler structure. XML tags can sometimes obscure the meaning.

3. **Size** - JSON tends to be more compact than XML since it does not require as much formatting. This makes JSON preferable for data transmission, especially on mobile networks.
Done
task run  0,16s user 0,05s system 9% cpu 2,132 total

As the app itself has not much work to to, javascript is even faster than GO here. This could be the Marshal part. It would be interesting to see how TypeScipt performs here. Anyone want to try out?!

Conclusion

Before Bedrock, I tried to get an Antrophic key for several weeks without success. With Bedrock, this became much easier. Calling the models via Bedrock is more secure than calling them directly. The AWS security methods help us here.

The rest is easy if you understand how to create the body and decode the response. So start in your development language of choice.

Enjoy building!

If you need developers and consulting for your next GenAI project, don’t hesitate to contact the sponsor of this blog, tecRacer.

For more AWS development stuff, follow me on dev https://dev.to/megaproaktiv. Want to learn GO on AWS? GO here

See also

Similar Posts You Might Enjoy

Stop LLM/GenAI hallucination fast: Serverless Kendra RAG with GO

RAG is a way to approach the “hallucination” problem with LLM: A contextual reference increases the accuracy of the answers. Do you want to use RAG (Retrieval Augmented Generation) in production? The Python langchain library may be too slow for your production services. So what about serverless RAG in fast GO Lambda? - by Gernot Glawe

Import existing resources into an AWS CloudFormation Stack

Presentation Deploying resources with infrastructure as code is the recommended way to provision resources in AWS. The native AWS-way of doing it is by using Cloudformation or CDK (Cloud Development Kit), and you should of course do this from day one. But in real world sometimes somebody provisioned resources via the console, or there is a need of refactor your code and split your stack into multiple stacks. Luckily It is not very often we have cases where it’s required to import resources. - by Javier Nuñez

Handling Errors and Retries in StepFunctions

“Everything fails all the time” has been preached to us by Werner Vogels for a few years now. Every engineer working on building and maintaining systems knows this to be true. Distributed systems come with their own kind of challenges, and one of the AWS services that help deal with those is AWS Step Functions. AWS Step Functions allow you to describe workflows as JSON and will execute those workflows for you. In this blog, we’ll explore what happens when things inevitably go wrong and the options the service offers to perform error handling and retries using an example application. - by Maurice Borgmeier