• [PyTorch][chapter 61][强化学习-免模型学习 off-policy]

    前言：

        蒙特卡罗的学习基本流程：

         Policy Evaluation :          生成动作-状态轨迹,完成价值函数的估计。

         Policy Improvement:       通过价值函数估计来优化policy。

           同策略（one-policy）：产生 采样轨迹的策略 \pi^{'} 和要改善的策略 \pi 相同。

           Policy Evaluation :    通过\epsilon-贪心策略（\pi^{'}）,产生（状态-动作-奖赏）轨迹。

           Policy Improvement:  原始策略也是 \epsilon-贪心策略（\pi^{'}), 通过价值函数优化, \epsilon-贪心策略（\pi^{'})

          异策略(off-policy)：产生采样轨迹的  策略 \pi^{'} 和要改善的策略 \pi 不同。

          Policy Evaluation :   通过\epsilon-贪心策略（\pi^{'}）,产生采样轨迹（状态-动作-奖赏）。

          Policy Improvement:  改进原始策略\pi

        两个优势：

        1： 原始策略不容易采样

        2： 降低方差

    易策略常用的方案为 IR(importance sample) 重要性采样

    Importance sampling is a Monte Carlo method for evaluating properties of a particular distribution, while only having samples generated from a different distribution than the distribution of interest. Its introduction in statistics is generally attributed to a paper by Teun Kloek and Herman K. van Dijk in 1978,[1] but its precursors can be found in statistical physics as early as 1949.[2][3] Importance sampling is also related to umbrella sampling in computational physics. Depending on the application, the term may refer to the process of sampling from this alternative distribution, the process of inference, or both.

    一  importance-samling

        1.1 原理

         原始问题：

          uf=∫xp(z)f(z)dx" role="presentation" style="position: relative;">uf=∫xp(z)f(z)dx

         如果采样N次，得到z1,z2,...zN" role="presentation" style="position: relative;">z1,z2,...zN

           uf≈1N∑zi∼p(z)f(zi)" role="presentation" style="position: relative;">uf≈1N∑zi∼p(z)f(zi)

        问题： p(z)" role="presentation" style="position: relative;">p(z) 很难采样（采样空间很大,很多时候只能采样到一部分）

       引入 q(x) 重要性分布(这也是一个分布,容易被采样)

      w(x)=\frac{p(x)}{q(x)}: 称为importance weight

                u_f =\int q(x)\frac{p(x)}{q(x)}f(x)dx

                 \approx \frac{1}{N}\sum_i w(x_i)f(x_i)(大数定理）

     下面例子，我们需要对w(xi)" role="presentation" style="position: relative;">w(xi),做归一化处理，更清楚的看出来占比

       下面代码进行了归一化处理,方案如下：

         w(xi)=logp(xi)−logq(xi)" role="presentation" style="position: relative;">w(xi)=logp(xi)−logq(xi)

         w^1(x_i)=\frac{e^{w(x_i)}}{\sum_j e^{w(x_i)}}

         w2(xi)=w(xi)−log∑j(ew(xj))" role="presentation" style="position: relative;">w2(xi)=w(xi)−log∑j(ew(xj))

          

    1. # -*- coding: utf-8 -*-
    2. """
    3. Created on Wed Nov  8 16:38:34 2023
    5. @author: chengxf2
    6. """
    7.  
    8. import numpy as np
    9. import matplotlib.pyplot as plt
    10. from scipy.special import logsumexp
    11.  
    12. class pdf:
    13.     
    14.     def __call__(self,x):
    15.         pass
    16.     def sample(self,n):
    17.         
    18.         pass
    19.     
    20. #正太分布的概率密度
    21. class Norm(pdf):
    22.     #返回一组符合高斯分布的概率密度随机数。
    23.     def __init__(self, mu=0, sigma=1):
    24.         
    25.         self.mu = mu
    26.         self.sigma = sigma
    27.         
    28.     def __call__(self, x):
    29.         #log p 功能,去掉前面常数项
    30.          logp = (x-self.mu)**2/(2*self.sigma**2)
    31.          return -logp
    32.      
    33.     
    34.     def sample(self, N):
    35.          #产生N 个点，这些点符合正太分布
    36.          x = np.random.normal(self.mu, self.sigma,N)
    37.          
    38.          return x
    39.  
    40.  
    41.  
    42. class Uniform(pdf):
    43.      #均匀分布的概率密度
    44.       def __init__(self, low, high):
    45.           
    46.           self.low = low
    47.           self.high = high
    48.           
    49.           
    50.       def __call__(self, x):
    51.              #logq 功能
    52.               N = len(x)
    53.               a = np.repeat(-np.log(self.high-self.low), N)
    54.               
    55.               return -a
    56.           
    57.          
    58.       def sample(self, N):
    59.               #产生N 点，这些点符合均匀分布
    60.               x = np.random.uniform(self.low, self.high,N)
    61.               
    62.               return x
    63.           
    64. class ImportanceSampler:
    65.     
    66.     
    67.     def __init__(self, p_dist, q_dist):
    68.         
    69.         self.p_dist = p_dist
    70.         self.q_dist = q_dist
    71.         
    72.     
    73.     def sample(self, N):
    74.         #采样
    75.         samples = self.q_dist.sample(N)
    76.         weights = self.calc_weights(samples)
    77.         
    78.         normal_weights = weights - logsumexp(weights)
    79.         
    80.         return samples, normal_weights
    81.     
    82.     
    83.     def calc_weights(self, samples):
    84.         #log (p/q) =log(p)-log(q)
    85.         return self.p_dist(samples)-self.q_dist(samples)
    86.  
    87.  
    88.  
    89. if __name__ == "__main__":
    90.  
    91.     N = 10000
    92.     p = Norm()
    93.     q = Uniform(-10, 10)  
    94.     
    95.     sampler = ImportanceSampler(p, q)
    96.     
    97.     #samples 从q(x)采样出来的点，weight_sample
    98.     samples,weight_sample= sampler.sample(N)
    99.     #以weight_sample的概率，从samples中抽样 N 个点
    100.     samples = np.random.choice(samples,N, p = np.exp(weight_sample))
    101.     
    102.     plt.hist(samples, bins=100)
    103.         
    104.        
    105.

    二 易策略 off-policy 原理

         target policy \pi： 原始策略 

            x：     这里面代表基于原始策略,得到的轨迹

                      [s0,a0,r1,....sT−1,aT−1,rT,sT]" role="presentation" style="position: relative;">[s0,a0,r1,....sT−1,aT−1,rT,sT]

           p(x):   该轨迹的概率

           f(x):    该轨迹的累积奖赏

          期望的累积奖赏：

                        u_f=\int_{x} f(x)p(x)dx \approx \frac{1}{N}\sum f(x_i)

        behavior policy \pi^{'}: 行为策略

         q(x): 代表各种轨迹的采样概率

        则累积奖赏函数f在概率p 也可以等价的写为：

         u_f=\int_{x}q(x)\frac{p(x)}{q(x)}f(x)dx

         E[f] \approx \frac{1}{m}\sum_{i=1}^{m}\frac{p(x_i)}{q(x_i)}f(x_i)

       

         P_i^{\pi} 和 P^{\pi^{'}} 分别表示两个策略产生i 条轨迹的概率,对于给定的一条轨迹

        [s0,a0,r1,....sT−1,aT−1,rT,sT]" role="presentation" style="position: relative;">[s0,a0,r1,....sT−1,aT−1,rT,sT]：

        原始策略\pi 产生该轨迹的概率：

         P^{\pi}=\prod_{i=0}^{T-1} \pi(s_i,a_i)P_{s_i\rightarrow s_{i+1}}^{a_i}

        P^{\pi^{'}}=\prod_{i=0}^{T-1} \pi^{'}(s_i,a_i)P_{s_i\rightarrow s_{i+1}}^{a_i}

       则

        w(s)=\frac{P^{\pi}}{p^{\pi^{'}}}=\prod_{i=0}^{T-1}\frac{\pi(s_i,a_i)}{\pi^{'}(s_i,a_i)}

      若\pi 为确定性策略，但是\pi^{'} 是\pi的\epsilon -贪心策略：

    原始策略   p_i=\left\{\begin{matrix} \pi(s_i,a_i)=1, if: a_i==\pi(x_i) \\ \pi(s_i,a_i)=0, if: a_i \neq \pi(x_i) \end{matrix}\right.

    行为策略: q_i=\left\{\begin{matrix} \pi^{'}(s_i,a_i)=1-\epsilon+\frac{\epsilon }{|A|} , if: a_i==\pi(x_i) \\ \pi^{'}(s_i,a_i)=\frac{\epsilon }{|A|}, if: a_i \neq \pi(x_i) \end{matrix}\right.

      现在通过行为策略产生的轨迹度量权重w

     理论上应该是连乘的，但是p_i=0, if a_i \neq \pi(x_i),

     考虑到只是概率的比值,上面可以做个替换

     w(s)=\frac{p^{\pi}}{p^{\pi^{'}}}=\prod\frac{e^{p_i}}{e^{q_i}}=\prod e^{p_i-q_i}

    其中： wi=epieqi=epi−qi" role="presentation" style="position: relative;">wi=epieqi=epi−qi（更灵活的利用importance sample）

    其核心是要计算两个概率比值，上面的例子是去log，再归一化

    三  方差影响

    四  代码

    代码里面R的计算方式跟上面是不同的,

    R=\frac{1}{T-t}(\sum_{i=t}^{T-1}r_i)(\prod_{j=t}^{T-1}w_j)

    w_j=e^{p_j-q_j}

    1. # -*- coding: utf-8 -*-
    2. """
    3. Created on Wed Nov  8 11:56:26 2023
    5. @author: chengxf2
    6. """
    7.  
    8. import numpy as ap
    9. # -*- coding: utf-8 -*-
    10. """
    11. Created on Fri Nov  3 09:37:32 2023
    13. @author: chengxf2
    14. """
    15.  
    16. # -*- coding: utf-8 -*-
    17. """
    18. Created on Thu Nov  2 19:38:39 2023
    20. @author: cxf
    21. """
    22. import numpy as np
    23. import random
    24. from enum import Enum
    25.  
    26. class State(Enum):
    27.     #状态空间#
    28.     shortWater =1 #缺水
    29.     health = 2   #健康
    30.     overflow = 3 #溢水
    31.     apoptosis = 4 #凋亡
    32.  
    33. class Action(Enum):
    34.     #动作空间A#
    35.     water = 1 #浇水
    36.     noWater = 2 #不浇水
    37.     
    38. class Env():
    39.     
    40.     def reward(self, state):
    41.         #针对转移到新的环境奖赏    
    42.         r = -100
    43.         if state is State.shortWater:
    44.             r =-1
    45.         elif state is State.health:
    46.             r = 1
    47.         elif state is State.overflow:
    48.             r= -1
    49.         else: # State.apoptosis
    50.             r = -100
    51.         return r
    52.             
    53.     
    54.     def action(self, state, action):
    55.  
    56.          if state is State.shortWater:
    57.          
    58.              if action is Action.water :
    59.                  newState =[State.shortWater, State.health]
    60.                  p =[0.4, 0.6]
    61.              else:
    62.                  newState =[State.shortWater, State.apoptosis]
    63.                  p =[0.4, 0.6]
    64.              
    65.             
    66.          elif state is State.health:
    67.              #健康
    68.              if action is Action.water :
    69.                  newState =[State.health, State.overflow]
    70.                  p =[0.6, 0.4]
    71.              else:
    72.                  newState =[State.shortWater, State.health]
    73.                  p =[0.6, 0.4]
    74.              
    75.          elif state is State.overflow:
    76.              #溢水
    77.              if action is Action.water :
    78.                  newState =[State.overflow, State.apoptosis]
    79.                  p =[0.6, 0.4]
    80.              else:
    81.                  newState =[State.health, State.overflow]
    82.                  p =[0.6, 0.4]
    83.              
    84.          else:  
    85.              #凋亡
    86.                  newState=[State.apoptosis]
    87.                  p =[1.0]
    88.          #print("\n S",S, "\t prob ",proba)
    89.          nextState = random.choices(newState, p)[0]
    90.          r = self.reward(nextState)
    91.          return nextState,r
    92.          
    93.  
    94.      
    95.     def __init__(self):
    96.          
    97.         self.name = "环境空间"
    98.          
    99.     
    100.      
    101.     
    102.     
    103.     
    104. class Agent():
    105.     
    106.     
    107.     def initPolicy(self):
    108.         #初始化累积奖赏
    109.         self.Q ={} #(state,action) 的累积奖赏
    110.         self.count ={} #(state,action) 执行的次数
    111.     
    112.         for state in self.S:
    113.             for action in self.A:
    114.                 self. Q[state, action] = 0.0
    115.                 self.count[state,action]= 0
    116.  
    117.             action = self.randomAction()
    118.             self.policy[state]= Action.noWater #初始化都不浇水
    119.             
    120.     def randomAction(self):
    121.        #随机策略
    122.        action = random.choices(self.A, [0.5,0.5])[0]
    123.        return action
    124.    
    125.     def behaviorPolicy(self):
    126.         
    127.         #使用e-贪心策略
    128.         state = State.shortWater #从缺水开始
    129.         env = Env()
    130.         trajectory ={}#[s0,a0,r0]--[s1,a1,r1]--[sT-1,aT-1,rT-1]
    131.         
    132.         for t in range(self.T):
    133.             #选择策略
    134.             rnd = np.random.rand() #生成随机数
    135.             if rnd <self.epsilon:
    136.                 action =self.randomAction()
    137.             else:
    138.                 #通过原始策略选择action
    139.                 action = self.policy[state] 
    140.             newState,reward = env.action(state, action) 
    141.             trajectory[t]=[state,action,reward]
    142.             state = newState
    143.         
    144.         return trajectory
    145.     
    146.     def calcW(self,trajectory):
    147.         #计算权重
    148.         
    149.         q1 = 1.0-self.epsilon+self.epsilon/2.0 # a== 原始策略
    150.         q2 = self.epsilon/2.0   # a!=原始策略
    151.         
    152.         w ={}
    153.         for t, value in trajectory.items():
    154.             #[state, action,reward]
    155.             action =value[1]
    156.             state = value[0]
    157.             
    158.             if action == self.policy[state]:
    159.                 p = 1
    160.                 q = q1
    161.             else:
    162.                 p = 0
    163.                 q = q2
    164.            
    165.             w[t] = round(np.exp(p-q),3)
    166.         #print("\n w ",w)
    167.         return w
    168.                 
    169.             
    170.         
    171.         
    172.      
    173.  
    174.               
    175.  
    176.            
    177.            
    178.     def getReward(self,t,wDict,trajectory):
    179.         
    180.         p = 1.0
    181.         r=  0
    182.         #=[state,action,reward]
    183.         for i in range(t,self.T):
    184.             r+=trajectory[t][-1]
    185.             w =wDict[t]
    186.             p =p*w
    187.         
    188.         R = p*r
    189.         
    190.         m = self.T-t
    191.         
    192.         return R/m
    193.     
    194.     def  improve(self):
    195.        
    196.         a = Action.noWater
    197.         for state in self.S:
    198.              maxR = self.Q[state, a]
    199.              for action in self.A:
    200.                  R = self.Q[state,action]
    201.                  if R>=maxR:
    202.                      maxR = R
    203.                      self.policy[state]= action
    204.                  
    205.                   
    206.             
    207.             
    208.         
    209.         
    210.     def learn(self):
    211.         
    212.         self.initPolicy()
    213.   
    214.   
    215.         
    216.         for s in range(1,self.maxIter): #采样第S 条轨迹
    217.             
    218.                #通过行为策略(e-贪心策略)产生轨迹
    219.                trajectory =self.behaviorPolicy()
    220.                w = self.calcW(trajectory)
    221.                
    222.                print("\n 迭代次数 %d"%s ,"\t 缺水：",self.policy[State.shortWater].name,
    223.                                          "\t 健康：",self.policy[State.health].name,
    224.                                         "\t 溢水：",self.policy[State.overflow].name,
    225.                                         "\t 凋亡：",self.policy[State.apoptosis].name)
    226.                
    227.                #策略评估
    228.                for t in range(self.T):
    229.                    R = self.getReward(t, w,trajectory)
    230.                    state = trajectory[t][0]
    231.                    action = trajectory[t][1]
    232.                    Q = self.Q[state,action]
    233.                    count  = self.count[state, action]
    234.                    
    235.                    self.Q[state,action] = (Q*count+R)/(count+1)
    236.                    self.count[state, action]=count+1
    237.                    
    238.                 
    239.                
    240.                #获取权重系数
    241.                self.improve() 
    242.  
    243.                         
    244.  
    245.                 
    246.  
    247.     
    248.     def __init__(self):
    249.         
    250.         self.S = [State.shortWater, State.health, State.overflow, State.apoptosis]
    251.         self.A = [Action.water, Action.noWater]
    252.         self.Q ={} #累积奖赏
    253.         self.count ={}
    254.         self.policy ={} #target Policy
    255.         self.maxIter =500
    256.         self.epsilon = 0.2
    257.         self.T = 10
    258.  
    259.  
    260.     
    261. if  __name__ == "__main__":
    262.     
    263.      agent = Agent()
    264.      agent.learn()

    https://img2020.cnblogs.com/blog/1027447/202110/1027447-20211013112906490-1926128536.png

  • 相关阅读:
    UGUI性能优化学习笔记（二）合批
    JAVA面试总结
    JsJavascriptEcma的eval性能测试2208011912
    【教程】Pycharm社区版中打开jupyter的方法
    【Spring】事务
    JetBrain Pycharm的一系列快捷键
    python连接MySQL数据库服务器、使用SQL查询数据表中的所有数据
    【LeetCode】Day127-四数之和 & 组合总和
    衡石科技携手亚马逊云科技、浩方集团，三大势能助推出海业务数字化升级
    go语言中如何实现同步操作呢
  • 原文地址：https://blog.csdn.net/chengxf2/article/details/134202820