Predicting Cognitive Performance in Open-ended Dynamic Tasks
A Modeling Comparison Challenge
Organized by: Christian Lebiere, Cleotilde Gonzalez, & Walter Warwick

Menu

:. Home .:
:. Results .:
:. Registration .:
:. The DSF Task .:
:. Method .:
:. Human Performance Data .:
:. References .:
:. FAQ .:
:. Contact Us .:

DSF: The Dynamic Stock & Flows Task

DSF is a generic dynamic decision making task that involves the regulation of a stock (maintain a quantity at a target level) through inflow (increase the stock) and outflow (decrease the stock) rates. DSF was designed to study and explain human dynamic decision making processes. The full capabilities of DSF are described elsewhere (Dutt & Gonzalez, 2007; Gonzalez & Dutt, 2007), therefore we will only give a brief overview of the DSF capabilities relevant for the empirical data and modeling for the purpose of this challenge.

The goal in DSF is to reach and maintain the level of water in a tank at a target level over a number of time periods. The level of water in the tank is the stock that increases with the inflows and decreases with the outflows. There are two types of inflows and outflows in this task: those that are exogenous (outside of the decision maker’s control) and those endogenous (under the decision maker’s control). The exogenous flows are called Environmental Inflow (that increases the level of the stock without the user’s control) and the Environmental Outflow (that decreases the level of stock without user’s control). The endogenous flows are User’s Inflow and Outflow. Setting the UserInFlow and OutFlow are the main decisions made by the user in each time period. In addition, if the system has a delay, defined by number of time periods, the stock level will also depend on the Supply Line (SL) (Sterman, 1989). The SL works as a First-in-First-out (FIFO) queue and contains the User Inflow and/or User Outflow values that have been placed but not yet received in the system.

The stock level at time t can be defined as follows:

Stock (t) = Stock (t-1) + [Environment Inflow (t-1) + User Inflow(t-1)] - [Environment Outflow (t-1) + User Outflow(t-1)] + SL(t-1-delay)

The video below presents the graphical user interface of DSF. At each time period users see the values of Environment Inflow and Outflow, values of User Inflow and Outflow, the amount of water in the tank (stock) and the goal level. At each time period, users submit two values, one for user inflow and another for user outflow. These inputs can take any positive value, including zero. Users may also receive a ‘bonus’ performance monetary incentive in each time period in which they were close enough to the target level.

Click here to download the video of DSF in action.

Download the codec for your operating system from TechSmith if you are having trouble viewing the download video.

Despite its seeming simplicity, controlling the DSF is a slow process for most subjects (Gonzalez & Dutt, 2007; Dutt & Gonzalez, 2007). For example, Cronin, Gonzalez & Sterman (2009) found that in a sample of highly educated graduate students with extensive technical training nearly half were unable to predict the qualitative path of a stock given very simple patterns for its inflow and outflow. Subject learning was slow and ultimately sub-optimal even in the simple conditions of the task, for example, that of controlling the system due to an increasing inflow and zero outflow (Gonzalez & Dutt, 2007; Dutt & Gonzalez, 2007). Moreover, Cronin and Gonzalez (2007) presented subjects with a series of manipulations related to the form of information display, context of the task, incentives and others factors intended to help the subject understand the task, demonstrating that the difficulty in understanding stocks and flows is not due to lack of information or the form of information presentation.

 

Download DSF

The DSF task environment requires a Windows platform and can be run in two modes. First, the DSF can be run as a live experiment so that a human can interact with the same task environment used in human experiments. In this way, modelers can gain hands-on experience with the task and use this experience to inform the development of their own models. Second, the DSF environment can be run as a constructive simulation, without the user interface, in a faster-than-real time mode with the participants' computational models interacting directly with the task environment.

System Requirements

  • Windows XP/Vista Operating System
  • Minimum resolution: 1024 x 1080 pixels
  • While the task environment only runs on Windows OS, participants are free to run their clients on Macs or Linux machines as well as Windows, since the DSF task environment interacts with the cognitive model through a text-based TCP/IP socket communication.

    Click here to download the user interface version of DSF. (Last updated 4/3/09)

    Click here to download the TCP/IP socket version of DSF. (Last updated 4/27/09)

    New! Click here to download the Client Socket Programs for ACT-R 6.0, C/C++, MATLAB, JAVA and VB to connect to the DSF socket version. (Last updated 4/27/09)

    See the DSF documentation below for more information on running and uninstalling DSF.

     

    DSF Documentation

    Instructions for running the user interface version of DSF:
    1. Download the zip file from the website by clicking “Click here to download the user interface version of DSF”
    2. Unzip the contents of the zip file (this will give a folder)
    3. Double click the “setup.exe” file inside the unzipped folder
    4. Simply follow the installation instructions (if you don’t want to change anything simply do Next, Next…. until you are finished with the install)
    5. After installation completes the Dynamic Stocks And Flow program will be installed in Start -> Programs -> Dynamic Stocks And Flow folder (go to this location and click the created executable file which will start DSF with a welcome screen)
    6. Login into the system with any ID and follow the steps that the system presents
    7. You can exit the system by clicking X on the top left hand side or pressing CTRL + X. Sometime the system might ask you an ID and password to log out. The ID is: Varun and Password: Varun

    Instructions for running TCP/IP socket version of DSF:
    The TCP/IP socket version of dynamics stock and flows (DSF) is provided as a zip file on the website. A sample java socket program (DSFClient.java) and its compiled class file (DSFClient.class) have been provided so that you can test TCP/IP socket version of DSF. Due to the versatility and robust availability of sockets in all programming languages, you are encouraged to implement your own socket as a part of the model program in the programming language that you will be using to develop your model.

    1. Download the zip file from the website by clicking “Click here to download the TCP/IP socket version of DSF”
    2. Unzip the contents of the zip file (this will give 2 files and a folder). The two files are java client files for running/testing the simulation (both the .java and compiled .class files are included so that one DOES NOT need to download the java SDK: simply go to the prompt locate the class file on the command prompt and run the class file by writing > java DSFClient (enter)). In case you want to change the IP address etc. for the sample java socket program, you might need to compile the .java file once again by > javac DSFClient.java.
    3. Double click the “setup.exe” file inside the unzipped folder
    4. Simply follow the installation instructions (if you don’t want to change anything simply do Next, Next…. until you are finished with the install)
    5. After install completes the Dynamic Stocks And Flow program will be installed in Start -> Programs -> Dynamic Stocks And Flow folder (go to this location and click the created executable file which will start DSF with a welcome screen)

    Structure of SETUP.TXT File as part TCP/IP socket version of DSF:
    The setup.txt file will be installed as part of TCP/IP socket version of DSF in C:\Program Files\Dynamics Stocks And Flows\data folder (unless you have changed the path during the installation process described above). The default setup variables in Setup.txt file and their existing dummy values have been described below:

    Setup variables
    MaxTrial = 100 (this sets the number of time periods in DSF)
    MaxNumberOfAScenarioType = 100 (this value should always equal MaxTrial parameter)
    InitialAmountInTank = 2 (this determines the initial amount of water in DSF tank at TIME 0)
    GoalLinePosition = 4 (this determines the position of GOAL on the DSF tank)
    InitialBackOrder = 0 (this will determine any backorder (inflow) amount that would enter the DSF tank during the first few time periods in a condition having some delay in the inflow)
    InitialForwardDischarge = 0 (this will determine any forward discharge (outflow) amount that would leave the DSF tank during the first few time periods in a condition having some delay in the outflow)
    inFlowDelayValue = 1 (this creates a delay in inflow for certain number of time periods. A value of 2 would mean that what you will enter now as inflow will come into the DSF tank two time period after)
    inFlowDelayLength = 100 (this is the length in time periods of the inflowDelayValue to hold and work i.e. the inFlowDelayValue will work to provide the delay in DSF for this duration of time. It is recommended to make this parameter equal to the MaxTrial parameter)
    outFlowDelayValue = 1 (this creates a delay in outflow for certain number of time periods. A value of 2 would mean that what you ordered to leave now as outflow will leave out of the DSF tank two time period after)
    outFlowDelayLength = 100 (this is the length in time periods of the outflowDelayValue to hold and work i.e. the outFlowDelayValue will work to provide the delay in DSF for this duration of time. It is recommended to make this parameter equal to the MaxTrial parameter)

    Uninstalling DSF
    To uninstall the DSF (which will be installed in C:\Programs Files by default), go to control panel and follow the normal Windows Procedure of Add/Remove programs.

     

    Socket protocol documentation

    The Dynamic Stocks and Flows (DSF) simulation uses a text-based TCP/IP socket protocol to communicate with any other software package. On double clicking the executable file, a "Welcome" screen is presented and at this time a server socket is opened on IP address of "localhost" (i.e. 127.0.0.1) using port number 9548. Software packages that need to connect to DSF as "clients" need to open a client port after the "Welcome" screen is presented in DSF.

    When you see the welcome screen come up, go to command prompt of your computer (on Windows XP/Vista click start ->Run... -> type "cmd" without quotes). Navigate to the folder where the Java socket client file is stored (DSFClient.java file; using cd <path> command). After this on the command prompt, compile the java file and get its executable (thus, type: javac <filename>.java on command prompt or simply use: java DSFClient which does not need java SDK). Now run the compiled file by typing: java <filename>. This step should give you a success for the socket connection.

    Remember that for any socket client to connect to DSF, the welcome screen should be present prior to running the socket program. After the last step, type the first login command and press enter:

    ENTRY-ID Varun1 VERSION BATCH/NONBATCH ACTR/NONACTR

    (Varun1 is any dummy name and if BATCH is entered then the simulation runs faster and does not show the DSF interface where as when NONBATCH is entered the simulation shows the DSF interface and runs slower; where the last keyword depends upon programming language used for the client). Remember the instruction set for communication on the socket is case sensitive so enter it exactly as it appears here (except for dummies occurring in the instruction text like Varun1). The simulation should start up and you should receive the first message of the state of the simulation. The STATE is just an identifier of the message string and has no purpose. The value following the TIME tells the time period of the simulation. The value following GOAL suggests the goal amount, which is set as 4.000. The value following AMOUNT is the current amount in the DSF tank.

    Similarly, values following EI, EO are the values of environment inflow, environment outflow displayed on the GUI for the current time period TIME. Also UI and UO are the values of user inflow and user outflow that the subject entered in the last time period. As you’ll see for the 0th TIME the EI, EO, UI and UO values are -1 or unknown. Now type a decision.

    For example, DECISION UI 2 UO 4 EI 3 EO 2 (notice that we send not only the UI and UO but also EI and EO from the model) and press enter to send this to the simulation. The simulation would send you the next state after executing the decision. Once a model of the task has been developed, the model should determine the UI and UO values, while the EI and EO values can be set by a program to implement any environmental input/output function.

    Further, you can send SIMULATION RESET anytime to DSF via socket connection, which will cause the simulation to return back to the welcome screen (and end the current participant/model run). The SIMULATION RESET can either be followed by the new login string like:

    ENTRY-ID Varun2 VERSION BATCH/NONBATCH

    (to start a new participants) or QUIT command (to exit DSF). Once the end time period is reached, the simulation would send you END_PROGRAM when it gets a decision string like:

    DECISION UI <value>UO <value> EI <value> EO <value>.

    At this time send: QUIT (to close the socket and DSF) or SIMULATION RESET followed by login instruction ENTRY-ID Varun3 VERSION BATCH/NONBATCH (to run a new participant). The DSF simulation sends an acknowledgement “SIMULATION_RESETTED” after resetting itself. The login instruction with ENTRY-ID should be given after receiving the SIMULATION_RESETTED acknowledgement from DSF.

     

    Organized in part by the Dynamic Decision Making Laboratory, a part of the Social and Decision Sciences Department at Carnegie Mellon University. For updates and comments regarding this website, please email hauyuw@andrew.cmu.edu.