MENDELSSOHN: USING MARKOV DECISION MODELS 



se, but rather to provide the decision maker with 

 added insight and reasonable first choices. 



Defining the Model on 

 a Discrete Grid 



In order to make Equation (1.2) amenable to 

 numerical methods, it is necessary to define both 

 the state space and the action space on a discrete 

 grid, and then to redefine the transition prob- 

 abilities, etc., on this grid. Several authors (Fox 

 1973; Bertsekas 1976; Hinderer 1978; Waldmann 

 1978; Whitt 1978; Larraneta^) have suggested 

 techniques to reduce MDP's to a grid and give 

 bounds on the error due to the approximation. I 

 have shown elsewhere (Mendelssohn^) that grid 

 choice can have a significant effect on the analysis. 

 An optimal policy and the value of an optimal 

 policy may not be greatlj' affected by the choice of 

 grid, but the estimated probabilistic behavior of 

 the population dynamics is affected significantly 

 by the choice of grid. 



A first effort then is to find an adequate grid for 

 the problem, a grid fine enough for both the de- 

 sired accuracy and for realistic approximations of 

 observed population sizes and coarse enough for 

 computational efficiency. Increased computa- 

 tional efficiency makes it reasonable to solve 

 many variations of a given model, which allows for 

 a more thorough exploration of the management 

 questions of interest and their sensitivity to key 

 assumptions. 



Several different grids were tried for Equation 

 (1.2) for both the Branch and Wood Rivers. 



To defin e Equation ( 1 .2 ) on a given grid, suppose 

 a grid of k points has been chosen on which to 

 discretize the problem and assume, as is reason- 

 able for this problem, that the reduced action 

 space (how many spawners to leave) is equivalent 

 to the state space (how many recruits are observed 

 at the beginning of the period). From Equation 

 (1.1), letting /?j and /?2 represent the parameters 

 of the Ricker equation 



P(jc,^i<w|y,) = P[(e^)/?iy,exp(-i?2y<)<w] 



= P(d<lnco-lna) 



(2.1) 



^Larraneta, J. C. 1978. Approaches to approximate Mar- 

 kov decision processes. Paper presented at Joint National 

 ORSA/TIMS Meeting, Nov. 13-15, 1978, Los Ang., Calif. 



^Mendelssohn, R. 1978. Theeffectsof grid size and approx- 

 imation techniques on the solutions of Markov decision prob- 



where a = [R^y^expi-R 2y, )]. Let ^ be the sj;andard 

 normal integral for a random variable d = did, 

 and let x,, x^^ be any two adjacent points on the 

 grid. Then: 



P(d<\nxi-\na) = (I>| 



In Xi — In a 



a 



P(d<Inx,-+i -In a) = ^\ 



'In jc,+i —In a 



o 



so that one method of defining the transition prob- 

 abilities on a grid is: 



P{xt^\ = x,+ i \yt) = ^\ 



'in Xi+i —In a 



d,! 



a 



\nxi —In a 



a 



The discrete probability when the action is y^ is 

 equal to the total probability of going to any state 

 in the interval (jc„ x, + j). 



If zero is included as a state, the procedure needs 

 to be modified slightly. Suppose the probability of 

 going to Xj is known for each decision _y. Then an 

 arbitrary fraction of this probability is assigned as 

 going to the zero state. In this paper, one-half of 

 the probability in the interval (Q,x^) is assigned to 

 the zero state. The results have been found not to 

 be sensitive to the value of the fraction; this is 

 because zero is an absorbing state. Either there 

 exists a policy that never reaches (0,a;i) and hence 

 never reaches zero, or else with probability one the 

 population goes to zero in finite time. Hence, it is 

 the size of (0,x^) that most influences the results, 

 not the fraction of this total that is assigned to 

 going to the absorbing state. 



Adding an absorbing state is sensible if the ab- 

 sorbing state is thought of as all states at low 

 enough population levels such that it would take 

 years for the fishery to recover again, if it recovers 

 at all. Without the absorbing state, the models in 

 Equation (1.1a, b) will always recover in fairly 

 short order. Since fisheries can be depleted, the 



lems. SWFC Admin. Rep. 20H, 15 p. Southwest Fish. Cent. 

 Honolulu Lab., Natl. Mar. Fish. Serv. NOAA, Honolulu, HI 

 96812. 



37 



