Code and Results for Chapter 4:

Introduction:
These are results and code for the problems and examples found in Chapter 4 of this famous book.

1. Policy Iteration for the Grid World Example:
   • Reproduces this example using in-place and non in-place policy iteration:
   • iter_poly_gw_not_inplace.m
     
   • iter_poly_gw_inplace.m (converges to the same final state as the result in the book)
     
2. Exercise 4.2 (Adding a state to grid world):
   • ex_4_2_sys_solv.m (This script solve the simple linear system)
     
3. Policy Improvement for the Jacks Car Rental (JCR) Example:
   • jcr_example.m (This script reproduces the results from the book)
     
   • jcr_policy_evaluation.m (the policy evalutation step)
     
   • jcr_policy_improvement.m (policy inprovement step)
     
   • jcr_rhs_state_value_bellman.m (computes the expectation over new states)
     
   • cmpt_P_and_R.m (a utility function for the JCR example)
     
   • sample output using the above code (results obtained when running jcr_example.m)
     
4. Exercise 4.5 (Modifications on the Jacks Car Rental Example):
   • ex_4_5_Script.m (reproduces the results from the book)
     
   • ex_4_5_policy_evaluation.m (the policy evalutation step)
     
   • ex_4_5_policy_improvement.m (policy inprovement step)
     
   • ex_4_5_rhs_state_value_bellman.m (computes the expectation over new states)
     
   • sample output from the above code (results obtained when running gam_Script.m)
     
5. The Gambler's Problem:
   • gam_Script.m (runs value iteration for the gambler problem)
     
   • gam_rhs_state_bellman.m (compute the backup computations for gambler problem)
     
   • some sample output that can be obtained from the above code