Testing and Debugging

We provide some tests for you, but they are not comprehensive. We strongly recommend writing additional custom tests where possible.

Overview

The testing framework is in test.sh and the tests directory. Test modules are named with the prefix test_:

  • Unit tests check functionality of your subcircuits (e.g. immediate generator, branch comparator). They do not check your entire CPU implementation. Unit tests are prepended with unit-, e.g., the test_alu unit test (module) has tests in the unit-alu directory.
  • Integration tests execute RISC-V instructions (from files) on your entire CPU and compare the outputs to the result of running those instructions on Venus. Integration tests are prepended with integration-, e.g., the test_addi integration test (module) has tests in the integration-addi directory.
  • Custom tests are custom integration tests you make by writing out RISC-V instructions (the testing framework will then create the testing and debugging circuit for you). Custom tests are strongly recommended as our unit tests and integration tests are not comprehensive. All custom tests should be in the test_custom module. Within each test directory, there are a set of tests. Each test will have up to three types of files, all named after the test name itself.

Each test module can have the following components in its directory:

  • .circ files: A .circ Logisim test circuit you can use for debugging. There is one circuit file per test in a test module.
  • out/ directory: Every test module contains an out/ directory where your results (.out) and the reference results (.ref) are stored.
  • in/ files: Integration and custom tests contain an in/ directory where RISC-V instruction .s files are stored. There is one code file per test in an integration or custom test module.

Run a Test

All tests are initiated via the test.sh script in the root directory.

  1. To view all test modules and basic commands, run test.sh with no arguments:

    $ bash test.sh

  2. Unit and integration tests: bash test.sh test_module, e.g.,

    $ bash test.sh test_alu
$ bash test.sh test_addi

  3. Custom tests:

    $ bash test.sh test_custom

Note: You can run bash test.sh part_a or bash test.sh part_b to run all tests for a specific project phase. View all test modules for the full list of commands.

View Output Files

When a test fails, the script will point you to the out/ directory by displaying the difference between your subcircuit output and the reference output.

Use bash test.sh format [path_to_file] to render the hex values of the output into a readable table, e.g.,

$ bash test.sh format tests/unit-alu/out/alu-add.ref   # reference

$ bash test.sh format tests/unit-alu/out/alu-add.out   # your subcircuit

Examine a Debugging Circuit

Each unit test has a .circ test circuit for debugging. Inside the test circuit:

  • Testing Harness: The circuit feeds a sequence of inputs (signals or instructions) from a ROM block to your component. These harness circuits are in harnesses/, but you should access them from the test circuit, which connects your subcircuit to the harness.
  • Subcircuit Access: To view your subcircuit (e.g., the ALU), right-click it and select "View [name]", or click the magnifying glass icon.
  • Probing: Probe wires to access their current values.
  • Simulation:
    • To view later inputs, click Simulate → Manual Tick Full Cycle, which will highlight the next row of the ROM blocks and send this next input into your subcircuit. You can tick cycles while viewing your ALU subcircuit to see later inputs.
    • To reset the simulation, click Simulate → Reset Simulator. You can also close and re-open the debugging circuit.

Important: Avoid making edits in the test circuit, as they may be lost!

Debugging Unit Tests

As an example, let's debug the alu-add unit test on unmodified starter code.

Run Test

$ bash test.sh test_alu

View Output File

  1. View the reference output:

    $ bash test.sh format tests/unit-alu/out/alu-add.ref
Time ALUSel A        B        ALUResult
00   0      00002020 00000f0f 00002f2f 
01   0      ffffdead ffffbeef ffff9d9c 
02   0      00007fff 00000001 00008000

    This shows the inputs (A, B, and ALUSel) sent to your subcircuit at each time step, and the expected output (ALUResult).

  2. View your subcircuit output. Here's the output when the test is run on unmodified starter code:

    $ bash test.sh format tests/unit-alu/out/alu-add.out
Time ALUSel A        B        ALUResult
00   0      00002020 00000f0f UUUUUUUU 
01   0      ffffdead ffffbeef UUUUUUUU 
02   0      00007fff 00000001 UUUUUUUU 
03   0      00000000 00000000 UUUUUUUU

Note that in the example, the inputs to your subcircuit are the same, but the output (ALUResult) of your subcircuit is different (undefined).

Examine Debugging Circuit

Open tests/unit-alu/alu-add.circ, which corresponds to the failed test from the previous section.

  1. View Test Harness: The first thing you'll see in this circuit is the testing harness:

    Annotated screenshot of alu-add.circ.

    This feeds a sequence of inputs (InputA, InputB, and ALUSel) to your ALU.

    The ROM (in the red box) contains a list of inputs to your circuit. The first input (InputA = 0x00002020, InputB = 0x00000f0f, ALUSel = 0b0000) is highlighted in dark gray. You can also see these values being passed into your ALU (in the blue box) with the probes.

  2. Access Subcircuit: In this picture, the ALUResult output from your ALU is undefined (all Us). To see why, we can view our ALU subcircuit to see what logic it's doing. Click into the ALU by doing one of the following:

Right-click the ALU and select 'View alu'

Using the magnifying glass to click into a subcircuit.

<pre data-shortcode><details >
Click the magnifying glass

Right-clicking to click into a subcircuit.

  1. Probe wires: If the output of your subcircuit isn't what you expect, you can probe wires to investigate where the incorrect output is coming from.

    Inside your ALU subcircuit, you can see inputs (A, B, and ALUSel) provided from the harness to your subcircuit. You can click on wires to see the values in those wires:

    Clicking on wires inside alu.circ to debug.

    In the starter circuit, the ALUResult output is undefined. In this case, note that the ALUResult tunnel is undefined, so we probably want to send a value to this tunnel.

    To return to the harness, you can click on main in the Simulate → Active Simulations tab in the top-left corner.

  2. Simulate:

    To view later inputs, click Simulate → Manual Tick Full Cycle, which will highlight the next row of the ROM blocks and send this next input into your subcircuit. You can tick cycles while viewing your ALU subcircuit to see later inputs.

    To reset the simulation, click Simulate → Reset Simulator. You can also close and re-open the debugging circuit.

Debugging Integration Tests

In addition to the example below, please check out this video regarding writing integration tests.

As an example, let's debug the addi-basic test within the test_addi integration test module. Again, we start with unmodified starter code.

Run Test

$ bash test.sh test_addi

If the test doesn't pass, this will print out the difference between your CPU output and the reference output. Before we access this code, we should figure out what instructions this test tried to run on your CPU.

View Test Code

Before we can figure out why this test failed, we should first figure out what code this test tried to run on your CPU. View the addi-basic test code:

$ vim tests/integration-addi/in/addi-basic.s

You should see these instructions:

addi t0, x0, 1
addi t0, x0, 42
addi t0, x0, 256
addi t0, x0, 2047

Reading Test Output

Let's return to the original terminal output from running test_addi. Here's the snippet for the addi-basic test. Each row of output shows the program counter (PC), instruction, and values in the 8 debug registers at each time step of running the input program.

FAIL: tests/integration-addi/addi-basic.circ (Did not match expected output)
             Time PC       Instruc. ra (x1)  sp (x2)  t0 (x5)  t1 (x6)  t2 (x7)  s0 (x8)  s1 (x9)  a0 (x10)
  Reference: 0001 00000004 02a00293 00000000 00000000 00000001 00000000 00000000 00000000 00000000 00000000
  Student:   0001 00000004 02a00293 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  ---
  Reference: 0002 00000008 10000293 00000000 00000000 0000002a 00000000 00000000 00000000 00000000 00000000
  Student:   0002 00000008 10000293 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  ---
  Reference: 0003 0000000c 7ff00293 00000000 00000000 00000100 00000000 00000000 00000000 00000000 00000000
  Student:   0003 0000000c 7ff00293 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  • You might see a lot of rows of output, but let's focus on the first two rows, which shows the first time step when your CPU doesn't match the reference output. In this case, that's time step 1.
  • In the first set of Reference/Student output, look for registers with mismatched values. In the above output, it looks like t0 should hold the value 1, but in your CPU, t0 holds the value 0.

Using the terminal output and the RISC-V code, try to work out what failed on the time step immediately before the first failing time step.

What's wrong?

In this output, since incorrect output showed up at time 1, something must have gone wrong at time 0. At time 0, the RISC-V code executed addi t0, x0, 1 - now we can see why the reference output has a 1 in t0. However, our implementation did not put a 1 in t0, so it looks like the very first addi instruction didn't execute correctly.

In tests like addi-basic, the RISC-V instructions execute in sequence, so it's straightforward to work out which instruction failed. For example, if the first line of terminal output shows time step 0003, you know that the instruction at time step 2 failed, and this instruction must be addi t0, x0, 256 (the second instruction in the testing code, zero-indexed).

Examine Debugging Circuit

Open tests/integration-addi/addi-basic.circ, which corresponds to the failed test addi-basic test from the test_addi module.

  1. View Test Harness:

    Screenshot of addi-basic.circ.

    The top-level harness for each integration test contains a ROM block (bottom half of screenshot) containing the RISC-V instructions for that test, representing IMEM (instruction memory). These instructions are passed into your CPU (the circled cpu_harness block at the top). You can also see the 8 debug register outputs; the testing framework will log their values into the .out file when running the test.

  2. Access Subcircuit: To view your CPU circuit, either

    • right-click the cpu_harness block and select "View cpu_harness", or
    • click the cpu_harness block and click the magnifying glass.

    This takes you into the CPU harness, where your CPU interacts with memory.

    Click another time into the cpu block, and now you should see the CPU you've been wiring.

  3. Simulate: To step through the RISC-V instructions, click Simulate → Manual Tick Full Cycle.

    • In each clock cycle, your CPU will output a new ProgramCounter to the harness, which will use the new PC to select the next instruction for your CPU to execute.
    • In this addi test, the instructions execute in sequence, but when testing branches and jumps later, the CPU could output a different ProgramCounter value (not always adding 4) and execute the instructions in a different order.

    To reset the simulation, click Simulate → Reset Simulator. You can also close and re-open the debugging circuit.

  4. Probe Wires

    At this point, you should have identified which RISC-V instruction is failing on your circuit, the expected register values after that instruction, and your register values after that instruction. You should have also opened a debugging circuit and ticked the clock until the failing instruction. Now, it's time to poke at all the wires in your circuit to see why this instruction is failing.

    Some useful wires you can poke:

    • If the instruction writes back to a register: which RegFile input corresponds to the data we're writing on this cycle?
    • If the instruction uses the ALU for computation: what are the three inputs (A, B, and ALUSel) to the ALU, and are they what you expect? What is the output from the ALU, and is it what you expect?
    • If the instruction is a branch/jump: what value is getting written to the PC register on the next clock cycle? Is that value what you expect? Which control logic signal determines the value written to the PC register? Does that signal have the right value?
    • If the instruction is a store/load instruction: what are the three inputs to DMEM, and are they what you expect? What is the output from DMEM, and is it what you expect?

    Note: If the failing instruction is a load instruction, it could be the case that your store instruction failed to write to DMEM first, and now the load instruction can't read a value that was never written to DMEM. In this case, you might want to stop at store instructions too and check that they're working as you expect.

We recommend inserting a time step counter circuit to count instructions. See below.

Debugging Integration Tests: Time Step Counter

When instructions don't execute in sequence, it can be hard to figure out how many cycles to tick before you reach the first failing instruction. To make that easier, we recommend adding a small counter circuit to your cpu.circ circuit:

Basic circuit that adds 1 at each time step.

  • This is the same logic as the PC circuit in the starter code, except the counter circuit adds 1 at each time step instead of adding 4.
  • The ProgramCounter tunnel has also been replaced with a Probe that lets you view the value of the wire while debugging.
  • This counter circuit is only here for debugging, not for your CPU implementation. When you modify your PC circuit to handle jumps and branches, this counter circuit will still be adding 1 at every time step, allowing you to keep track of when the first failing instruction occurs.

You can use this counter to stop at the first failing instruction. For example, if your terminal output looks like this:

$ bash test.sh test_integration_branch
FAIL: tests/integration-branch/branch-basic.circ (Did not match expected output)
             Time PC       Instruc. ra (x1)  sp (x2)  t0 (x5)  t1 (x6)  t2 (x7)  s0 (x8)  s1 (x9)  a0 (x10)
  Reference: 0006 00000010 fe04cce3 00000000 00000000 00000000 00000000 00000000 ffffffff 00000001 00000000
  Student:   0006 00000028 fe000ce3 00000000 00000000 00000000 00000000 00000000 ffffffff 00000001 00000000
  ---
  Reference: 0007 00000014 fe000ce3 00000000 00000000 00000000 00000000 00000000 ffffffff 00000001 00000000
  Student:   0007 00000020 fe904ce3 00000000 00000000 00000000 00000000 00000000 ffffffff 00000001 00000000
  ---

You know that the instruction at time step 5 failed, so you should go into the debugging circuit and tick the clock until the counter circuit is showing 5. Now, you can start poking around to see what instruction is currently executing, and why it's failing.

Debugging Jump/Branch Tests

When we're running code with jumps and branches, the RISC-V instructions don't execute in sequence, so we have to be more careful to figure out which instruction failed. Consider the output below:

$ bash test.sh test_integration_branch
FAIL: tests/integration-branch/branch-basic.circ (Did not match expected output)
             Time PC       Instruc. ra (x1)  sp (x2)  t0 (x5)  t1 (x6)  t2 (x7)  s0 (x8)  s1 (x9)  a0 (x10)
  Reference: 0005 00000018 fe948ce3 00000000 00000000 00000000 00000000 00000000 ffffffff 00000001 00000000
  Student:   0005 00000024 fe944ce3 00000000 00000000 00000000 00000000 00000000 ffffffff 00000001 00000000
  ---
  Reference: 0006 00000010 fe04cce3 00000000 00000000 00000000 00000000 00000000 ffffffff 00000001 00000000
  Student:   0006 00000028 fe000ce3 00000000 00000000 00000000 00000000 00000000 ffffffff 00000001 00000000
  ---
  Reference: 0007 00000014 fe000ce3 00000000 00000000 00000000 00000000 00000000 ffffffff 00000001 00000000
  Student:   0007 00000020 fe904ce3 00000000 00000000 00000000 00000000 00000000 ffffffff 00000001 00000000
  ---

We know that the instruction at time step 4 failed, but which instruction is that? Let's check the RISC-V code in tests/integration-branch/in/branch-basic.s:

         addi s1, x0, 1
         addi s0, x0, -1
label5:  beq  x0, x0, label1
label6:  bltu x0, s0, label8
label4:  blt  s1, x0, label5
         beq  x0, x0, label6
label3:  beq  s1, s1, label4
label10: beq  s0, s0, end
label2:  blt  x0, s1, label3
label9:  blt  s0, s1, label10
label1:  beq  x0, x0, label2
label8:  bltu s1, s1, label9
         beq  s1, s1, label9
end:     addi s0, x0, 2
  • At time step 0, addi s1 x0 1 executes.
  • At time step 1, addi s0 x0 -1 executes.
  • At time step 2, beq x0 x0 label1 executes, causing a branch to be taken.
  • At time step 3, beq x0 x0 label2 (the line at label1) executes, causing a branch to be taken.
  • At time step 4, blt x0 s1 label3 (the line at label2) executes.
What's wrong?

At time step 4, s1 holds 1, so the branch at time step 4 should be taken. This means at time step 5, we should be executing the instruction at label3. This is the 6th instruction (zero-indexed), which corresponds to PC value 24 (each instruction is 4 bytes), which is 0x18 (the value under PC in the reference output).

However, in our output, the PC is at 0x24, or 36 in decimal. This is the 9th instruction, which is label9 (the line directly after label2). It looks like at time step 4, the branch instruction should have updated the PC by taking a branch, but in our implementation, the PC was updated incorrectly.

Writing Custom Tests

All custom tests are integration tests. All you need to do is write some RISC-V instructions for your CPU to run, and the testing framework will handle the rest.

  1. Navigate to tests/integration-custom/in.
  2. Write a RISC-V test and save it in a filename ending in .s.
  3. Run bash test.sh test_custom.

test_custom compiles your RISC-V test code to a Logisim circuit and runs it. If you want to only compile your test, run bash test.sh create_custom. If you want to only run your test, run bash test.sh run_custom.

To debug your circuits, you can step through the debugging circuits (similar to what you did in Project 3A).

  1. Navigate to the tests folder, then navigate to the folder of the relevant test, e.g. tests/integration-custom.
  2. Open the generated .circ file in Logisim. Click into the circuits you made, and tick full cycles to step through inputs.

Custom Test Tips

Write your test as a simple RISC-V program. You will need to set up any registers with the values you want to use, as these are the only instructions that will execute. There is no larger test skeleton provided by staff.

Tips:

  • The testing framework only checks the values in the 8 debug registers when comparing your CPU output with the reference output, so when writing your own tests, make sure to only use the 8 debug registers.

  • The testing framework doesn't check memory (DMEM) when comparing your CPU with the reference. To check values in memory or a non-debug register, you'll need to put the value back into a debug register. For example, to test if a store works, you'll probably have to load the value back from memory into a debug register to see if the value was successfully stored.

  • IMEM and DMEM are separate in Logisim, but combined in Venus. This means that if you write assembly code that tries to access memory overlapping with instructions, Venus will throw an error. Since counting exactly how many instructions your assembly code requires, and multiplying that by 4 can be annoying, we suggest you load/store using addresses greater than 0x3E8 (leaving space for 1000 bytes/250 instructions), and increase this offset if you have more instructions.

  • Make sure to write RISC-V instructions that behave differently on a working CPU and a buggy CPU.

    For example, consider this test:

    addi t0, x0, 0
addi t1, x0, 0

    This wouldn't be very useful to check for a working CPU, because the output in the debugging registers could be all zeros even if your CPU doesn't work. As another example:

    beq t0, t0, 4
addi t1, x0, 10

    On a working CPU, this would branch to the addi instruction. On a buggy CPU where the branch is incorrectly not taken, this would still execute the addi instruction, so this test doesn't do a very good job of distinguishing working circuits from buggy circuits.

Logisim Tips

This section contains some helpful Logisim tips and pitfalls to avoid.

Wiring

  • If you want to know more details about each component, go to Help -> Library Reference for more information on the component and its inputs and outputs.
  • Use tunnels! They will make your wiring cleaner and easier to follow, and will reduce your chances of encountering crossed wires or unexpected errors.
  • Ensure you name your tunnels correctly. The labels are case sensitive!
  • You can hover your cursor over an input/output on a component to get slightly more information about that input/output.

Wiring Pitfalls

  • Your circuits should always fit in the provided harnesses. This means that you should not edit the provided input/output pins or add new ones. To ensure your circuit fits int he harness, you can open the harnesses in the harnesses folder and check that there are no errors.
  • Don't create new .circ files. You can make additional subcircuits if you want, but they must be in existing files.

Subcircuits

  • Note that if you modify a subcircuit, and another circuit file uses that subcircuit, you will need to close and re-open the outer circuit to load the changes from the subcircuit. For example, if you modify imm-gen.circ, you should close and re-open cpu.circ to load your changes.
  • When modifying a subcircuit, you should always open up the subcircuit file. For example, you should modify imm-gen.circ, not the imm-gen subcircuit in cpu.circ.

Signal Tips

  • The clock input signal (clk) can be sent into subcircuits or attached directly to the clock inputs of memory units in Logisim, but should not otherwise be gated (i.e., do not invert it, do not AND it with anything, etc.).
  • We recommend not using the Enable input on your MUXes. In fact, you can turn that attribute off (Include Enable?). We also recommend that you disable the Three-state? attribute (if the plexer has it).

Banned Circuit Elements

The following circuit elements are not necessary for this project, so please don't use them in your implementation.

  • Pull Resistor
  • Transistor
  • Transmission Gate
  • Power
  • POR
  • Ground
  • Divider
  • Random
  • PLA
  • RAM
  • Random Generator