OpenShift_052：离线安装 OCP 3.4 之安装与配置 Logging

环境：OCP 3.4

1. 安装与配置 Logging （在 Master 机器上操作）
创建一个 admin 用户，并且赋予权限
htpasswd -b /etc/origin/master/htpasswd admin admin
oadm policy add-cluster-role-to-user admin admin

切换到 logging project
oc project logging

如果不希望 fluentd 爬虫进程收集 infra=yes 节点的日志，执行如下命令，否则不要执行
oc annotate namespace logging openshift.io/node-selector='infra=yes' --overwrite

创建账户
oc new-app logging-deployer-account-template

给 service account 账户赋权
oadm policy add-cluster-role-to-user oauth-editor system:serviceaccount:logging:logging-deployer
oadm policy add-scc-to-user privileged system:serviceaccount:logging:aggregated-logging-fluentd
oadm policy add-cluster-role-to-user cluster-reader system:serviceaccount:logging:aggregated-logging-fluentd
oadm policy add-cluster-role-to-user rolebinding-reader system:serviceaccount:logging:aggregated-logging-elasticsearch

说明：第 4 个赋权，如果不执行，会有这个异常：openshift Discover: [security_exception] no permissions for indices:data/read/msearch

创建 configmap
oc create configmap logging-deployer \
   --from-literal kibana-hostname=kibana.apps.example.com \
   --from-literal public-master-url=https://master.example.com:8443 \
   --from-literal es-cluster-size=3 \
   --from-literal es-instance-ram=4G

确认 configmap 创建成功
oc edit configmap logging-deployer
输出如下：
......
apiVersion: v1
data:
  es-cluster-size: "3"
  es-instance-ram: 4G
  kibana-hostname: kibana.apps.example.com
  public-master-url: https://master.example.com:8443
kind: ConfigMap
metadata:
  creationTimestamp: 2017-02-12T03:07:51Z
  name: logging-deployer
  namespace: logging
  resourceVersion: "6549"
  selfLink: /api/v1/namespaces/logging/configmaps/logging-deployer
  uid: 70d238a1-f0d0-11e6-a94b-080027fc450b

如果以前安装过 logging-deployer，需要先将其卸载；如果没装过，则无需执行
oc new-app logging-deployer-template --param MODE=uninstall --param IMAGE_VERSION=v3.4 --param IMAGE_PREFIX=registry.example.com:5000/openshift3/

部署 EFK Stack
oadm policy add-cluster-role-to-user cluster-admin system:serviceaccount:openshift:logging-deployer

oc new-app logging-deployer-template \
             --param IMAGE_VERSION=v3.4 \
             --param MODE=install \
--param IMAGE_PREFIX=registry.example.com:5000/openshift3/ \
--param KIBANA_HOSTNAME=kibana.apps.example.com \
--param KIBANA_OPS_HOSTNAME=kibana-ops.apps.example.com \
--param PUBLIC_MASTER_URL=https://master.example.com:8443 \
--param ES_INSTANCE_RAM=4G \
--param ES_OPS_INSTANCE_RAM=4G

等待所有 pod 创建并运行成功
oc get pod
最终输出如下：
NAME                          READY     STATUS      RESTARTS   AGE
logging-curator-1-i818h       1/1       Running     0          1m
logging-deployer-39dv4        0/1       Completed   0          2m
logging-es-6uawm5uj-1-fgz78   1/1       Running     0          1m
logging-es-8gt7nc7t-1-x1694   1/1       Running     0          1m
logging-es-wxgw7qs2-1-hpcwz   1/1       Running     0          1m
logging-kibana-1-ddd2n        2/2       Running     0          1m

oc get svc
最终输出如下：
NAME                     CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
logging-es               172.30.172.188   <none>        9200/TCP   12m
logging-es-cluster       172.30.247.145   <none>        9300/TCP   12m
logging-es-ops           172.30.31.22     <none>        9200/TCP   12m
logging-es-ops-cluster   172.30.95.70     <none>        9300/TCP   12m
logging-kibana           172.30.18.121    <none>        443/TCP    12m
logging-kibana-ops       172.30.154.22    <none>        443/TCP    12m

oc get route
最终输出如下：
NAME                 HOST/PORT                     PATH      SERVICES             PORT      TERMINATION
logging-kibana       kibana.apps.example.com                 logging-kibana       <all>     reencrypt
logging-kibana-ops   kibana-ops.apps.example.com             logging-kibana-ops   <all>     reencrypt

让 fluentd 爬虫进程从所有节点收集日志
oc label node --all logging-infra-fluentd=true
输出如下：
node "master.example.com" labeled
node "node1.example.com" labeled
node "node2.example.com" labeled

oc get node --show-labels
输出如下：
NAME                 STATUS                     AGE       LABELS
master.example.com   Ready,SchedulingDisabled   67d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=master.example.com,logging-infra-fluentd=true
node1.example.com    Ready                      67d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,infra=yes,kubernetes.io/hostname=node1.example.com,logging-infra-fluentd=true
node2.example.com    Ready                      67d       app=yes,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=node2.example.com,logging-infra-fluentd=true

oc get pod -o wide
可以看到多了几个  logging-fluentd pod，输出如下：
NAME                          READY     STATUS      RESTARTS   AGE       IP            NODE
logging-curator-1-qyzox       1/1       Running     0          2h        10.128.0.11   node1.example.com
logging-deployer-8er59        0/1       Completed   0          2h        10.129.0.3    node2.example.com
logging-es-gh05jndn-1-p1hfp   1/1       Running     0          2h        10.129.0.7    node2.example.com
logging-es-gh7kvmwl-1-hl9wm   1/1       Running     0          2h        10.128.0.10   node1.example.com
logging-es-o3nrrfcx-1-c2cab   1/1       Running     0          2h        10.129.0.8    node2.example.com
logging-fluentd-230am         1/1       Running     0          2h        10.128.0.13   node1.example.com
logging-fluentd-fite6         1/1       Running     0          2h        10.129.0.9    node2.example.com
logging-fluentd-lgn1f         1/1       Running     0          2h        10.130.0.2    master.example.com
logging-kibana-1-3gj0a        2/2       Running     0          2h        10.128.0.12   node1.example.com

2. 确认是否安装成功
https://kibana.apps.example.com
说明：不是所有 Project 的日志都可以查询，比如 Management Infrastructure Project 中没有 pod，所以就没有信息，界面上一直显示 Searching 状态。

刷新几次应用界面，然后切换到该 Project，会显示一些日志数据：

3. 如果安装配置有问题，执行以下命令清除，然后重做
在 Master 机器上操作：
oc delete sa logging-deployer
oc delete sa aggregated-logging-kibana
oc delete sa aggregated-logging-elasticsearch
oc delete sa aggregated-logging-fluentd
oc delete sa aggregated-logging-curator
oc delete clusterrole oauth-editor
oc delete clusterrole daemonset-admin
oc delete rolebinding logging-deployer-edit-role
oc delete rolebinding logging-elasticsearch-view-role
oc delete clusterrole rolebinding-reader
oc delete rolebinding logging-deployer-dsadmin-role
oc delete configmaps logging-deployer