Overview

This document details configuration options for Elasticsearch logging.

See Logging for details about what information gets sent to these logs.

Logging Services

For logging, YOUnite uses Logstash and Elastic/Kibana. A default implementation of Elastic/Kibana from Opensearch (https://opensearch.org/) is provided, which is pre-configured for YOUnite, however, if a different implementation is desired or different destination for logs is desired, Logstash can be configured to send logs to that destination instead.

The following configuration options are detailed in this document:

  1. Disabling logstash logging

  2. Disabling Elasticsearch Log button in UI

  3. Logging with Docker / Kubernetes

  4. Logging with custom or managed services

Disabling Logstash Logging

If logging is not desired, it can be disabled by adding the following environment variable to the younite-api, younite-notification-service and younite-data-virtualization-service docker images:

LOGGING_LOGSTASH_ENABLED=false

Disabling Elasticsearch Log button in UI

The UI will disable the Log when the younite-api has the following environment variable:

LOGGING_ELASTIC_ENABLED=(true or FALSE)

Logging with Docker / Kubernetes

YOUnite provides pre-configured docker images for Elasticsearch, Logstash and Kibana with all the configuration necessary to work "off the shelf". See the Implementation Guides in the Knowledge Base for more information on starting these services.

These pre-configured services use the Single-Sign on provided (Keycloak) to perform authentication. By default, the built-in admin and dgs users have full control over configuration. These two users have the ability to hand out permissions to other users.

Pros: Easy to use, pre-configured.

Cons: Running services that require persistent storage and fast performance is not always recommended in docker / kubernetes. The managed versions of these services will likely provide better performance, scalability, redundancy, etc.

Logging with custom or managed services (ie from AWS, GKS, etc)

When running YOUnite in the cloud, it is recommended that Elasticsearch is run as a managed service for best performance, scalability and redundancy. You may also want to create your own images for Elasticsearch, Logstash and/or Kibana. These instructions tell you how to configure each service to work with the YOUnite stack.

Amazon AWS - OpenSearch

Amazon AWS provides a managed solution for OpenSearch, which is a fork of Elasticsearch. OpenSearch Dashboards is included (the OpenSearch equivalent to Kibana). You will still need to deploy a logstash image in your stack (either use the pre-configured one or create your own from the instructions below).

Depending on your desired configuration, you may wish to use SAML to authenticate to Keycloak. The guide for that is here: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/saml.html

Custom Image (OpenSearch and OpenSearch Dashboards)

When using the OpenSearch, the following instructions should be used to secure the server: https://opensearch.org/docs/latest/security-plugin/configuration/index/ .

The following steps should be done, at a minimum, based on the guide above:

  1. New certificates should be generated to replace the demo certificates. See https://opensearch.org/docs/latest/security-plugin/configuration/generate-certificates/

  2. The internal users opensearch, kibanaserver and logstash users are required and their default passwords should be replaced. See example yml file below.

  3. The admin and dgs users should be mapped to the all_access role. See example yml file below.

  4. Single sign on should be configured. See example yml file below for Open ID Connect (keycloak).

Example Files for OpenSearch configuration

Certificates:

This guide explains how to generate certificates: https://opensearch.org/docs/latest/security-plugin/configuration/generate-certificates/

The following files should be generated and put in /usr/share/opensearch/config/:

  1. admin.pem

  2. admin-key.pem

  3. node.pem

  4. node-key.pem

  5. root-ca.pem

  6. root-ca-key.pem

Opensearch Configuration File: /usr/share/opensearch/config/opensearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0

#
# This configuration is copied from the "demo" configuration with small modifications
#

# configure certificates / security
plugins.security.ssl.transport.pemcert_filepath: node.pem
plugins.security.ssl.transport.pemkey_filepath: node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node.pem
plugins.security.ssl.http.pemkey_filepath: node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=admin,OU=opensearch,O=YOUnite,L=San Mateo,ST=California,C=US
plugins.security.nodes_dn:
  - CN=node1,OU=opensearch,O=YOUnite,L=San Mateo,ST=California,C=US

plugins.security.audit.type: internal_opensearch
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*"]

Security Configuration File: /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml

Note
 Make sure to update the openid_connect_url to point to the keycloak server 
_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false # important! must be false for Open ID to work!
        authentication_backend:
          type: intern
      openid_auth_domain:
        description: "Authenticate via Open ID Connect"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: preferred_username
            roles_key: roles
            openid_connect_url: "${OPENID_URL}"
        authentication_backend:
          type: noop

Internal users file: /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml

Note
 Make sure to generate hashed passwords using the instructions here (plain text passwords will NOT work): https://opensearch.org/docs/latest/security-plugin/configuration/yaml/ 
# This is the internal user database
# The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh

#
# YOUnite - this is similar to the default internal_users.xml file, but with
# less users and configurable passwords. it also names the admin user "opensearch".
#

_meta:
  type: "internalusers"
  config_version: 2

opensearch:
  hash: "..."
  reserved: true
  backend_roles:
    - "admin"
  description: "opensearch admin user"

kibanaserver:
  hash: "..."
  reserved: true
  description: "kibanaserver user"

logstash:
  hash: "..."
  reserved: false
  backend_roles:
    - "logstash"
  description: "Logstash user"

Roles Mapping File: /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml

# In this file users, backendroles and hosts can be mapped to Open Distro Security roles.
# Permissions for Opendistro roles are configured in roles.yml

#
# YOUnite - this is identical to the default roles_mapping.xml file, but with admin
# and dgs added to the "all_access" role.
#

_meta:
  type: "rolesmapping"
  config_version: 2

all_access:
  reserved: false
  backend_roles:
    - "admin"
  users:
    - "admin"
    - "dgs"
  description: "Maps admin to all_access"

own_index:
  reserved: false
  users:
    - "*"
  description: "Allow full access to an index named like the username"

logstash:
  reserved: false
  backend_roles:
    - "logstash"

kibana_user:
  reserved: false
  backend_roles:
    - "kibanauser"
  description: "Maps kibanauser to kibana_user"

readall:
  reserved: false
  backend_roles:
    - "readall"

manage_snapshots:
  reserved: false
  backend_roles:
    - "snapshotrestore"

kibana_server:
  reserved: true
  users:
    - "kibanaserver"

Custom Logstash Image

OpenSearch has a docker image with the output plugin pre-installed called opensearchproject/logstash-oss-with-opensearch-output-plugin. See here: https://opensearch.org/docs/latest/clients/logstash/index/

Create a folder and place the following files (source code below) in the folder:

  1. Dockerfile

  2. logstash.conf

Dockerfile

FROM opensearchproject/logstash-oss-with-opensearch-output-plugin:7.13.2

ADD --chown=logstash:logstash logstash.conf /usr/share/logstash/pipeline/

logstash.conf

input {
    tcp {
        port => 4560
        codec => json_lines
    }
}
filter {
  if [message][elastic-index] {
    mutate {
        add_field => { "[@metadata][index]" => "%{[message][elastic-index]}"}
        remove_field => [ "[message][elastic-index]" ]
    }
    geoip {
        source => "[message][network-address]"
    }
    ruby {
      code => "
        begin
            # remove everything but @metadata message and geoip
            keys = event.to_hash.keys
            keys.each{|key|
                if ( key != '@metadata' && key != 'message' && key != 'geoip' )
                    event.remove(key)
                end
            }

            # move message fields up
            message = event.get('message')
            event.remove('message')
            if (message != nil)
                message.each{|k, v|
                    event.set(k, v)
                }
            end

        rescue Exception => e
            event.set('logstash_ruby_exception', e.message)
        end
      "
    }
  }
}
output {
  if [@metadata][index] {
    opensearch {
      user => "${ELASTIC_USER}"
      password => "${ELASTIC_PASSWORD}"
      hosts => ["${ELASTIC_HOST}"]
      manage_template => false
      index => "%{[@metadata][index]}"
      # set this to true in PRODUCTION and use valid certificates!
      ssl_certificate_verification => false
    }
  } else {
    opensearch {
      user => "${ELASTIC_USER}"
      password => "${ELASTIC_PASSWORD}"
      hosts => ["${ELASTIC_HOST}"]
      manage_template => false
      index => "serverlog-%{+YYYY.MM.dd}"
      # set this to true in PRODUCTION and use valid certificates!
      ssl_certificate_verification => false
    }
  }
}

Build the image using the command docker build . -t younite/logstash

Custom OpenSearch Dashboards Image

OpenSearch Dashboards needs to be configured to use Open ID Connect. Below is example Dockerfile and support files.

Create a folder and place the following files in the folder:

  1. Dockerfile

  2. opensearch_dashboards.yml

Dockerfile:

FROM opensearchproject/opensearch-dashboards

ADD --chown=opensearch-dashboards:opensearch-dashboards opensearch_dashboards.yml /usr/share/opensearch-dashboards/config/

opensearch_dashboards.yml:

# Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License.
# A copy of the License is located at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# or in the "license" file accompanying this file. This file is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied. See the License for the specific language governing
# permissions and limitations under the License.

# Description:
# Default configuration for OpenSearch Dashboards

server.host: "0"
opensearch.hosts: ["https://localhost:9200"]
opensearch.ssl.verificationMode: none
opensearch.username: "kibanaserver"
opensearch.password: "kibanaserver"
opensearch.requestHeadersWhitelist: [ authorization,securitytenant ]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false

#
# Custom below for YOUnite / Open ID Connect
#

opensearch_security.auth.type: openid
opensearch_security.openid.connect_url: ${OPENID_URL}
opensearch_security.openid.client_id: ${OPENID_CLIENT_ID}
opensearch_security.openid.client_secret: ${OPENID_CLIENT_SECRET}
opensearch_security.openid.base_redirect_url: ${OPENID_REDIRECT_URL}

Build the image using the command docker build . -t younite/opensearch-dashboards.