Environments List#
Original tasks#
We use a subset of the original MiniWoB tasks that only involve (1) clicking and (2) typing text from the prompt.
Name |
Description |
|---|---|
Choose an item from a drop down list. |
|
Click on a specific button in a generated form. |
|
Click on buttons in a certain order. |
|
Click desired checkboxes. |
|
Click the specified color. |
|
Click the button to close the dialog box. |
|
Click a specific button in a dialog box. |
|
Click on a specified link in text. |
|
Click option boxes. |
|
Click the shades that match a specified color. |
|
Click on a specific shape. |
|
Click on a tab element. |
|
Click a link inside a specific tab element. |
|
Click on a single button. |
|
Click on one of two buttons. |
|
Click on a specific widget in a generated form. |
|
Count number of shapes. |
|
Navigate through an email inbox and perform some actions. |
|
Use the date input to pick the correct date. |
|
Enter the password into the form. |
|
Enter given text to a textfield. |
|
Enter dynamically generated text to a textfield. |
|
Enter the specified time into the input. |
|
Focus into a text input. |
|
Focus on a specific text input. |
|
Find the Cartesian coordinates on a grid. |
|
Guess the number. |
|
Identify a randomly generated shape. |
|
Enter user login details into the form. |
|
Navigate a file tree to find a specified file or folder. |
|
Search through a bunch of results to find a specified link. |
|
Interact with a social media feed. |
|
Win a game of tic-tac-toe. |
|
Use a spinner to select given number. |
|
Search for flight results. |
|
Learn to operate a date picker tool. |
|
Click a collapsible element to expand it. |
|
Find and click on a specified link, from collapsible elements. |
|
Click items on a pie menu. |
|
Use autocomplete element efficiently. |
|
[book-flight] |
|
[choose-date] |
|
[click-collapsible-2] |
|
[click-collapsible] |
|
[click-pie] |
|
[use-autocomplete] |
Additional tasks#
Some are harder versions of the existing tasks, while some are completely new.
Name |
Description |
|---|---|
[click-checkboxes] Click at least 5 out of up to 12 checkboxes |
|
[click-checkboxes] Paraphrased entries |
|
[click-checkboxes] Train and test on different number of targets |
|
[click-tab-2] Varying number of tabs from 2 to 6 |
|
[login-user] Random popup |
|
Fill in forms of varying layouts. |
|
Fill in forms with shuffled field orderings. |
|
[social-media] Do some action on all matching entries |
|
[social-media] Do some action on some matching entries |
|
[email-inbox-forward] NL instruction (30 templates) |
|
[email-inbox-forward] NL instruction (100 templates) |
|
[email-inbox] NL instruction (100 templates for each subtask) |
Flight search tasks#
These are server-free ports of the FormWoB tasks in the original World of Bits “paper.”,
The prompt is a list of key-value pairs (e.g., Departure City: New York)
If the required fields are not filled, or if the agent navigates away from the page, the reward is “-1.”,
Otherwise, the reward is the fraction of key-value pairs that are satisfied
Name |
Description |
|---|---|
port of Alaska FormWoB |
|
port of Alaska FormWoB but harder |
|
port of American Airlines FormWoB (unused) |
Debug tasks#
These are easier versions of existing tasks. They are used for debugging.
Name |
Description |
|---|---|
[choose-date] December only |
|
[choose-date] December or November only |
|
[click-tab-2] One 1 tab |
|
[click-tab-2] Choose between a link or ‘no match’ |
|
[click-test] Different buttons during train and test |
|
[email-inbox] No scrolling + 1 subtask |
|
[email-inbox] No scrolling + 1 subtask |
|
[email-inbox] No scrolling + 1 subtask |
|
[email-inbox] No scrolling |
|
[email-inbox] No scrolling + 1 subtask |
|
[email-inbox] No scrolling + 2 subtasks |
Clicking non-elements#
These tasks involve clicking at a specific point inside a canvas-like element.
Name |
Description |
|---|---|
Find the line that bisects an angle evenly in two. |
|
Find the center of a circle. |
|
Count the number of sides on a shape. |
|
Find the shortest mid-point of two points. |
|
Draw a given number using checkboxes. |
|
Given two points, add a third point to create a right angle. |
|
Use a color wheel. |
|
Use a color wheel given specific random color. |
Hovering#
These tasks require hovering and moving the mouse cursor.
Name |
Description |
|---|---|
Click menu items. |
|
Find a specific item from a menu. |
Dragging#
These tasks involve dragging.
Name |
Description |
|---|---|
Click multiple items from a scroll list. (also require Shift + click) |
|
Drag the smaller box into the larger box. |
|
Drag an item in a specified direction. |
|
Drag a 3D cube to show a specific face. |
|
Drag items in a list, in a specified direction |
|
Drag items in a 2D grid around. |
|
Drag shapes into a box. |
|
Drag numbers into sorted ascending order. |
|
Highlight all the text. |
|
Highlight the specified paragraph. |
|
Resize a textarea in a given direction. |
|
Scroll through a text area element and enter last word into text area. |
|
Scroll through a text area in a given direction. |
|
Modify a text’s style in a text-editor. |
|
Use a slider to select a particular value. |
|
Use sliders to create a given combination. |
Typing free text#
These tasks involve typing texts that are not substrings of the prompt. Some of these also require advanced reasoning (e.g., solving math problems).
Name |
Description |
|---|---|
Copy text and paste it into an input. |
|
Copy text from a specific textarea and paste it into an input. |
|
Convert given text to upper or lower case. |
|
Find nth word in a block of text. |
|
Read information out from a table. |
|
Read multiple pieces of information out from a table. |
|
Solve for X. |
|
Perform some arithmetic math operations. |
|
Use the terminal to delete a file. |
|
Enter slightly transformed text into a text box. |
|
Count the total number of blocks. |
Timing#
These tasks require the agent to wait for events to happen before acting, and a ‘nodelay’ version is impossible to make.
Name |
Description |
|---|---|
chase-circle |
Keep your mouse inside a moving circle. |
moving-items |
Click moving items before they disappear. |
simon-says |
Push the buttons in the order shown. |
Missing#
These tasks are listed in the original paper but were missing from the OpenAI website.
Name |
Description |
|---|---|
ascending-numbers |
Click on the numbers in ascending order. |
button-delay |
Wait a certain period of time before clicking the second button. |
buy-ticket |
Buy a ticket that matches the requested criteria. |
daily-calendar |
Create an event on a daily calendar. |
drag-shape |
Drag a randomly generated shape in a specified direction. |
drag-shapes-2 |
Drag shapes into boxes, categorized by type. |
draw-circle |
Draw a circle around a marked point. |
draw-line |
Draw a line through a marked point. |
find-greatest |
Find the card with the greatest number. |
form-sequence |
Perform a series of instructions on a form. |
form-sequence-2 |
Perform a series of instructions on a form. |
form-sequence-3 |
Perform a series of instructions on a form. |
generate-number |
Generate a random number that meets certain criteria. |
hot-cold |
Find and click on the hot area. |
hover-shape |
Hover over the colored shape. |
odd-or-even |
Mark each number as odd or even. |
order-food |
Order food items from a menu. |
phone-book |
Find a contact in a phone book. |
sign-agreement |
Sign a user agreement. |
stock-market |
Buy from the stock market below a specified price. |