Opensearch
Chapter 39: Amazon OpenSearch Service - Log Analytics
Section titled “Chapter 39: Amazon OpenSearch Service - Log Analytics”Search and Log Analytics Platform
Section titled “Search and Log Analytics Platform”39.1 Overview
Section titled “39.1 Overview”Amazon OpenSearch Service is a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud.
Amazon OpenSearch Service Overview+------------------------------------------------------------------+| || +------------------------+ || | Amazon OpenSearch | || | Service | || +------------------------+ || | || +---------------------+---------------------+ || | | | | || v v v v || +----------+ +----------+ +----------+ +----------+ || | Search | | Log | | Analytics| | Dashboards| || | | | Analytics| | | | | || | - Full | | - Cloud | | - Real | | - Visual | || | Text | | Watch | | Time | | ize | || | - Index | | - Logs | | - Query | | - Kibana | || +----------+ +----------+ +----------+ +----------+ || |+------------------------------------------------------------------+Key Features
Section titled “Key Features”| Feature | Description |
|---|---|
| Search Engine | Full-text search and indexing |
| Log Analytics | Centralized log management |
| Real-time Analytics | Live data analysis |
| Dashboards | OpenSearch Dashboards (Kibana) |
39.2 OpenSearch Architecture
Section titled “39.2 OpenSearch Architecture”Cluster Architecture
Section titled “Cluster Architecture” OpenSearch Cluster Architecture+------------------------------------------------------------------+| || +------------------------+ || | OpenSearch Cluster | || +------------------------+ || | || +---------------------+---------------------+ || | | | || v v v || +----------+ +----------+ +----------+ || | Master | | Data | | Coordinating || | Nodes | | Nodes | | Nodes | || | | | | | | || | - Cluster| | - Store | | - Query | || | Manage | | Data | | Routing| || | - State | | - Index | | - Load | || | | | | | Balance| || +----------+ +----------+ +----------+ || |+------------------------------------------------------------------+Node Types
Section titled “Node Types” OpenSearch Node Types+------------------------------------------------------------------+| || Master Nodes || +------------------------------------------------------------+ || | | || | - Cluster state management | || | - Index creation/deletion | || | - Node coordination | || | - Recommended: 3 for HA | || | | || +------------------------------------------------------------+ || || Data Nodes || +------------------------------------------------------------+ || | | || | - Store and index data | || | - Execute queries | || | - Handle CRUD operations | || | - Scale horizontally | || | | || +------------------------------------------------------------+ || || Coordinating Nodes || +------------------------------------------------------------+ || | | || | - Route queries to data nodes | || | - Aggregate results | || | - Handle incoming requests | || | | || +------------------------------------------------------------+ || || UltraWarm Nodes || +------------------------------------------------------------+ || | | || | - Cost-effective cold storage | || | - S3-backed storage | || | - For infrequently accessed data | || | | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+39.3 Index Management
Section titled “39.3 Index Management”Index Structure
Section titled “Index Structure” OpenSearch Index Structure+------------------------------------------------------------------+| || Index || +------------------------------------------------------------+ || | | || | logs-2024-01-15 | || | +--------------------------------------------------------+ | || | | Shard 1 | | || | | +----------------------------------------------------+ | | || | | | Document 1: { "timestamp": ..., "message": ... } | | | || | | | Document 2: { "timestamp": ..., "message": ... } | | | || | | +----------------------------------------------------+ | | || | +--------------------------------------------------------+ | || | +--------------------------------------------------------+ | || | | Shard 2 | | || | | +----------------------------------------------------+ | | || | | | Document 3: { "timestamp": ..., "message": ... } | | | || | | +----------------------------------------------------+ | | || | +--------------------------------------------------------+ | || | | || +------------------------------------------------------------+ || || Shards || +------------------------------------------------------------+ || | | || | - Primary shards: Handle write operations | || | - Replica shards: Read scaling and redundancy | || | - Number of shards affects performance | || | | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+Index State Management (ISM)
Section titled “Index State Management (ISM)” Index State Management+------------------------------------------------------------------+| || ISM Policy || +------------------------------------------------------------+ || | | || | States: | || | +------------------------------------------------------+ | || | | Hot --> Warm --> Cold --> Delete | | || | +------------------------------------------------------+ | || | | || | Example Policy: | || | +------------------------------------------------------+ | || | | { | | || | | "policy": { | | || | | "description": "Log retention policy", | | || | | "states": [ | | || | | { | | || | | "name": "hot", | | || | | "transitions": [{ | | || | | "state_name": "warm", | | || | | "conditions": { | | || | | "min_index_age": "7d" | | || | | } | | || | | }] | | || | | }, | | || | | { | | || | | "name": "warm", | | || | | "actions": [{ "replica_count": 1 }], | | || | | "transitions": [{ | | || | | "state_name": "delete", | | || | | "conditions": { "min_index_age": "30d" } | | || | | }] | | || | | }, | | || | | { "name": "delete", "actions": [{ "delete": {} }] } | || | | ] | | || | | } | | || | | } | | || | +------------------------------------------------------+ | || | | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+39.4 Log Ingestion
Section titled “39.4 Log Ingestion”Data Sources
Section titled “Data Sources” OpenSearch Data Ingestion+------------------------------------------------------------------+| || Direct Ingestion || +------------------------------------------------------------+ || | | || | - OpenSearch API (REST) | || | - OpenSearch SDK | || | - Bulk API for high-volume ingestion | || | | || +------------------------------------------------------------+ || || AWS Service Integration || +------------------------------------------------------------+ || | | || | CloudWatch Logs: | || | +------------------------------------------------------+ | || | | - Subscription filter to Lambda --> OpenSearch | | || | | - CloudWatch Logs subscription | | || | +------------------------------------------------------+ | || | | || | Kinesis Data Firehose: | || | +------------------------------------------------------+ | || | | - Direct delivery to OpenSearch | | || | | - Data transformation via Lambda | | || | | - Buffer and batch configuration | | || | +------------------------------------------------------+ | || | | || | Amazon S3: | || | +------------------------------------------------------+ | || | | - S3 event notification --> Lambda --> OpenSearch | | || | | - Batch loading from S3 | | || | +------------------------------------------------------+ | || | | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+Kinesis Data Firehose Integration
Section titled “Kinesis Data Firehose Integration” Kinesis Firehose to OpenSearch+------------------------------------------------------------------+| || Data Flow || +------------------------------------------------------------+ || | | || | +----------+ +----------+ +----------+ | || | | Data | --> | Kinesis | --> | Lambda | | || | | Source | | Firehose | | Transform| | || | +----------+ +----------+ +----------+ | || | | | || | v | || | +----------+ | || | | OpenSearch| | || | | Service | | || | +----------+ | || | | | || | v | || | +----------+ | || | | S3 Backup | | || | +----------+ | || | | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+39.5 OpenSearch Dashboards
Section titled “39.5 OpenSearch Dashboards”Dashboard Features
Section titled “Dashboard Features” OpenSearch Dashboards+------------------------------------------------------------------+| || Visualization Types || +------------------------------------------------------------+ || | | || | - Line, area, and bar charts | || | - Pie and donut charts | || | - Heat maps | || | - Gauge and metric visualizations | || | - Maps (geospatial data) | || | - Tables and data grids | || | - Markdown text widgets | || | | || +------------------------------------------------------------+ || || Discover (Log Search) || +------------------------------------------------------------+ || | | || | - Query logs using Lucene or KQL syntax | || | - Filter by field values | || | - Expand document details | || | - Save searches | || | | || +------------------------------------------------------------+ || || Alerting || +------------------------------------------------------------+ || | | || | - Create monitors for queries | || | - Set trigger conditions | || | - Configure actions (SNS, Slack, etc.) | || | - Track alert history | || | | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+39.6 Security
Section titled “39.6 Security”Security Features
Section titled “Security Features” OpenSearch Security+------------------------------------------------------------------+| || Authentication || +------------------------------------------------------------+ || | | || | - Fine-grained access control (FGAC) | || | - IAM authentication | || | - SAML authentication | || | - Basic authentication (username/password) | || | | || +------------------------------------------------------------+ || || Encryption || +------------------------------------------------------------+ || | | || | - Encryption at rest (AWS KMS) | || | - Encryption in transit (TLS) | || | - Node-to-node encryption | || | | || +------------------------------------------------------------+ || || Access Control || +------------------------------------------------------------+ || | | || | - Role-based access control (RBAC) | || | - Index-level permissions | || | - Document-level security | || | - Field-level security | || | | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+39.7 CLI Commands
Section titled “39.7 CLI Commands”# Create OpenSearch domainaws opensearch create-domain \ --domain-name my-domain \ --engine-version OpenSearch_2.3 \ --cluster-config InstanceType=r6g.large.search,InstanceCount=3,DedicatedMasterEnabled=true,DedicatedMasterType=r6g.large.search,DedicatedMasterCount=3 \ --ebs-options EBSEnabled=true,VolumeType=gp3,VolumeSize=100 \ --node-to-node-encryption-options Enabled=true \ --encryption-at-rest-options Enabled=true \ --domain-endpoint-options EnforceHTTPS=true
# Describe domainaws opensearch describe-domain \ --domain-name my-domain
# List domainsaws opensearch list-domain-names
# Get domain statusaws opensearch describe-domain-health \ --domain-name my-domain
# Update domain configurationaws opensearch update-domain-config \ --domain-name my-domain \ --cluster-config InstanceCount=5
# Delete domainaws opensearch delete-domain \ --domain-name my-domain
# Authorize VPC endpointaws opensearch authorize-vpc-endpoint-access \ --domain-name my-domain \ --account 123456789012
# Create data source (for OpenSearch Serverless)aws opensearch create-data-source \ --name my-data-source \ --type OPENSEARCH_SERVERLESS \ --open-search-serverless-options '{"collectionArn": "arn:aws:aoss:..."}'
# Create access policyaws opensearch create-access-policy \ --name my-access-policy \ --type data \ --policy '{"Rules":[{"Resource":["collection/*"],"Permission":["aoss:DescribeCollection"],"Principal":["arn:aws:iam::..."]}]}'39.8 Best Practices
Section titled “39.8 Best Practices”OpenSearch Best Practices
Section titled “OpenSearch Best Practices” OpenSearch Best Practices+------------------------------------------------------------------+| || 1. Use appropriate instance types || +------------------------------------------------------------+ || | - Storage-optimized for large data | || | - Compute-optimized for search-heavy workloads | || +------------------------------------------------------------+ || || 2. Configure proper shard count || +------------------------------------------------------------+ || | - Aim for 10-50 GB per shard | || | - Avoid oversharding | || +------------------------------------------------------------+ || || 3. Use UltraWarm for cold data || +------------------------------------------------------------+ || | - Reduce costs for infrequently accessed data | || | - Configure ISM policies | || +------------------------------------------------------------+ || || 4. Enable security features || +------------------------------------------------------------+ || | - Fine-grained access control | || | - Encryption at rest and in transit | || +------------------------------------------------------------+ || || 5. Monitor cluster health || +------------------------------------------------------------+ || | - Set up CloudWatch alarms | || | - Monitor JVM memory pressure | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+39.9 Exam Tips
Section titled “39.9 Exam Tips”
Key Exam Points+------------------------------------------------------------------+| || 1. OpenSearch is the successor to Elasticsearch Service || || 2. Use Kinesis Firehose for log ingestion || || 3. UltraWarm provides cost-effective cold storage || || 4. ISM automates index lifecycle management || || 5. Fine-grained access control for security || || 6. OpenSearch Dashboards for visualization || || 7. Master nodes manage cluster state || || 8. Data nodes store and index data || || 9. Shards affect performance and scalability || || 10. Cross-cluster search for multi-region queries || |+------------------------------------------------------------------+39.10 Summary
Section titled “39.10 Summary” Chapter 39 Summary+------------------------------------------------------------------+| || OpenSearch Core Concepts || +------------------------------------------------------------+ || | - Clusters: Managed OpenSearch deployment | || | - Indices: Data containers | || | - Shards: Data partitions | || | - Documents: Individual records | || +------------------------------------------------------------+ || || Key Features || +------------------------------------------------------------+ || | - Full-text search | || | - Log analytics | || | - Real-time dashboards | || | - Alerting | || +------------------------------------------------------------+ || || Integration || +------------------------------------------------------------+ || | - Kinesis Data Firehose | || | - CloudWatch Logs | || | - Lambda for transformation | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+Previous Chapter: Chapter 38: AWS X-Ray - Distributed Tracing Next Chapter: Chapter 40: AWS Health Dashboard & Personal Health Dashboard