XoR/apk-deploy-02
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Добавлен код и промты.

Browse Source
This commit is contained in:
Бекетов Дмитрий Сергеевич
2025-03-28 12:40:01 +03:00
parent e914b770b5
commit 7f8a4b1891
14 changed files with 856 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

119
terraform/README.md Normal file
View File

@@ -0,0 +1,119 @@
# K3s Kubernetes Cluster Terraform Project
This Terraform project allows you to deploy a lightweight K3s Kubernetes cluster on any infrastructure that supports SSH access. The project uses only the null and local providers, performing all operations via SSH connections and local/remote execution provisioners.
## Project Structure
```
terraform/
├── modules/
│ └── k3s-install/
│ ├── main.tf # K3s installation logic
│ ├── variables.tf # Module variables
│ └── outputs.tf # Module outputs
└── infra/
└── dev/
├── main.tf # Main entry point
├── variables.tf # Variable definitions
├── terraform.tfvars # Variable values
├── env.tf # Environment file handling
├── providers.tf # Provider configurations
└── .env # Environment variables (template)
```
## Prerequisites
1. SSH access to all target nodes (server and workers)
2. SSH key pair for authentication
3. Terraform >= 1.0.0
4. Target nodes with:
- A supported operating system (Ubuntu, Debian, CentOS, etc.)
- Sufficient privileges to install system packages
## Configuration
### Setting Up Variables
1. Edit `terraform/infra/dev/terraform.tfvars` to configure your infrastructure:
- Set `server_ip` to the IP address of your K3s server node
- Set `worker_ips` to a list of IP addresses for your worker nodes
- Set `ssh_user` to the SSH username for connecting to the nodes
- Set `ssh_private_key` to the path of your SSH private key
- Customize other parameters as needed
2. Alternatively, you can use environment variables by editing the `.env` file:
- Uncomment and set the desired variables
- Variables defined in `.env` will override those in `terraform.tfvars`
## Usage
### Deploying the Cluster
Navigate to the deployment directory and initialize Terraform:
```shell
cd terraform/infra/dev
terraform init
```
Review the execution plan:
```shell
terraform plan
```
Apply the configuration to deploy the K3s cluster:
```shell
terraform apply
```
### Accessing the Cluster
After deployment, the kubeconfig will be available at the path specified by `kubeconfig_path` (default: `~/.kube/config`). You can use this to interact with your cluster:
```shell
kubectl get nodes
kubectl get pods --all-namespaces
```
### Destroying the Cluster
To tear down the cluster:
```shell
terraform destroy
```
This will uninstall K3s from both server and worker nodes, and clean up all resources.
## Features
- **SSH Configuration**: Creates a secure SSH configuration for connecting to nodes
- **K3s Server Installation**: Installs K3s on the server node with customizable options
- **Kubeconfig Management**: Retrieves and configures the kubeconfig for local access
- **Worker Node Deployment**: Joins worker nodes to the cluster using the node token
- **Clean Uninstallation**: Properly cleans up all components on terraform destroy
- **Flexible Configuration**: Supports both variable files and environment variables
## Customization Options
- **Traefik Dashboard**: Enable with `enable_traefik_dashboard = true`
- **SSL Configuration**: Enable with `enable_ssl = true` and provide certificate paths
- **Custom K3s Version**: Specify with `k3s_version` variable
- **Additional Arguments**: Pass extra arguments to both server and agent installs
## Troubleshooting
If you encounter issues:
1. Check SSH connectivity to all nodes
2. Verify that the SSH user has sufficient permissions
3. Ensure all nodes meet the system requirements for K3s
4. Check the logs on each node at `/var/log/k3s.log` or `/var/log/k3s-agent.log`
## Security Considerations
- The SSH configuration disables strict host key checking for convenience but consider enabling it in production
- Secure your kubeconfig file as it contains authentication information
- Consider using an SSH agent or SSH certificates for improved security

34
terraform/infra/dev/env.tf Normal file
View File

@@ -0,0 +1,34 @@
# This file handles environment variable loading from .env file
locals {
# Parse .env file if it exists and make variables available
env_file_exists = fileexists("${path.module}/.env")
# Read and parse .env file content
env_file_content = local.env_file_exists ? file("${path.module}/.env") : ""
# Convert lines to key-value pairs
env_vars = local.env_file_exists ? {
for line in compact(split("\n", local.env_file_content)) :
trimspace(split("=", line)[0]) => trimspace(join("=", slice(split("=", line), 1, length(split("=", line)))))
if length(split("=", line)) > 1 && !startswith(trimspace(line), "#")
} : {}
# Environment variables that can override Terraform variables
server_ip_override = lookup(local.env_vars, "SERVER_IP", "")
worker_ips_override = lookup(local.env_vars, "WORKER_IPS", "")
ssh_user_override = lookup(local.env_vars, "SSH_USER", "")
ssh_private_key_override = lookup(local.env_vars, "SSH_PRIVATE_KEY", "")
domain_override = lookup(local.env_vars, "DOMAIN", "")
k3s_version_override = lookup(local.env_vars, "K3S_VERSION", "")
}
# Use environment variables if provided, otherwise use terraform.tfvars values
locals {
effective_server_ip = local.server_ip_override != "" ? local.server_ip_override : var.server_ip
effective_worker_ips = local.worker_ips_override != "" ? split(",", local.worker_ips_override) : var.worker_ips
effective_ssh_user = local.ssh_user_override != "" ? local.ssh_user_override : var.ssh_user
effective_ssh_private_key = local.ssh_private_key_override != "" ? local.ssh_private_key_override : var.ssh_private_key
effective_domain = local.domain_override != "" ? local.domain_override : var.domain
effective_k3s_version = local.k3s_version_override != "" ? local.k3s_version_override : var.k3s_version
}

56
terraform/infra/dev/main.tf Normal file
View File

@@ -0,0 +1,56 @@
locals {
# Load environment variables if .env file exists
env_vars = fileexists("${path.module}/.env") ? {
for line in [
for line in split("\n", file("${path.module}/.env"))
: line if length(regexall("^[A-Za-z][A-Za-z0-9_]*=.*$", line)) > 0
] : split("=", line)[0] => join("=", slice(split("=", line), 1, length(split("=", line))))
} : {}
}
module "k3s_cluster" {
source = "../../modules/k3s-install"
# Required parameters
server_ip = var.server_ip
worker_ips = var.worker_ips
ssh_user = var.ssh_user
ssh_private_key = var.ssh_private_key
replace_url = var.replace_url != "" ? var.replace_url : var.server_ip
# Optional parameters with defaults
k3s_version = var.k3s_version
k3s_channel = var.k3s_channel
kubeconfig_path = var.kubeconfig_path
node_token_path = var.node_token_path
# Conditional extra arguments
k3s_extra_server_args = join(" ", compact([
var.enable_traefik_dashboard ? "--set traefik.dashboard.enabled=true" : "",
var.enable_ssl ? "--tls-san=${var.domain}" : "",
var.k3s_extra_server_args
]))
k3s_extra_agent_args = var.k3s_extra_agent_args
}
# Output information about the deployed cluster
output "kubeconfig" {
description = "Path to the kubeconfig file"
value = module.k3s_cluster.kubeconfig_path
}
output "server_ip" {
description = "IP address of the K3s server"
value = module.k3s_cluster.server_ip
}
output "worker_ips" {
description = "IP addresses of K3s workers"
value = module.k3s_cluster.worker_ips
}
output "cluster_ready" {
description = "Indicator that the cluster is ready"
value = module.k3s_cluster.cluster_ready
}

17
terraform/infra/dev/providers.tf Normal file
View File

@@ -0,0 +1,17 @@
terraform {
required_version = ">= 1.0.0"
required_providers {
null = {
source = "hashicorp/null"
version = "~> 3.2.0"
}
local = {
source = "hashicorp/local"
version = "~> 2.4.0"
}
}
}
# Provider configuration
provider "null" {}
provider "local" {}

28
terraform/infra/dev/terraform.tfvars Normal file
View File

@@ -0,0 +1,28 @@
# K3s Server Configuration
server_ip = "192.168.1.100" # Replace with your server IP
worker_ips = [ # Replace with your worker IPs
"192.168.1.101",
"192.168.1.102"
]
ssh_user = "ubuntu" # Replace with your SSH username
ssh_private_key = "~/.ssh/id_rsa" # Path to your private SSH key
replace_url = "" # Optional: URL to replace in kubeconfig, leave empty to use server_ip
# Cluster Configuration
domain = "reg.benadis.org" # Your domain name
k3s_version = "v1.27.3+k3s1" # K3s version to install
k3s_channel = "stable" # K3s release channel
kubeconfig_path = "~/.kube/config" # Where to save kubeconfig
node_token_path = "/tmp/node-token" # Where to save node token
# Optional Features
enable_traefik_dashboard = false # Enable Traefik dashboard
enable_ssl = false # Enable SSL
ssl_cert_path = "" # Path to SSL certificate
ssl_key_path = "" # Path to SSL key
install_argocd = false # Install ArgoCD
enable_ssh_tunnel = false # Enable SSH tunneling
# Advanced Configuration
k3s_extra_server_args = "" # Extra args for K3s server
k3s_extra_agent_args = "" # Extra args for K3s agent

105
terraform/infra/dev/variables.tf Normal file
View File

@@ -0,0 +1,105 @@
variable "server_ip" {
description = "IP address of the K3s server node"
type = string
}
variable "worker_ips" {
description = "List of IP addresses for K3s worker nodes"
type = list(string)
default = []
}
variable "ssh_user" {
description = "SSH username for connecting to nodes"
type = string
default = "root"
}
variable "ssh_private_key" {
description = "Path to the SSH private key for authentication"
type = string
}
variable "replace_url" {
description = "URL to replace in the kubeconfig (usually the server_ip or domain name)"
type = string
default = ""
}
variable "domain" {
description = "Domain name for the cluster"
type = string
default = ""
}
variable "k3s_version" {
description = "Version of K3s to install"
type = string
default = "v1.27.3+k3s1"
}
variable "k3s_channel" {
description = "Release channel of K3s to install (stable, latest, etc.)"
type = string
default = "stable"
}
variable "kubeconfig_path" {
description = "Path where to save the kubeconfig file locally"
type = string
default = "~/.kube/config"
}
variable "node_token_path" {
description = "Path where to save the node token locally"
type = string
default = "/tmp/node-token"
}
variable "enable_traefik_dashboard" {
description = "Whether to enable the Traefik dashboard"
type = bool
default = false
}
variable "enable_ssl" {
description = "Whether to configure SSL for the cluster"
type = bool
default = false
}
variable "ssl_cert_path" {
description = "Path to SSL certificate"
type = string
default = ""
}
variable "ssl_key_path" {
description = "Path to SSL key"
type = string
default = ""
}
variable "install_argocd" {
description = "Whether to install ArgoCD for GitOps"
type = bool
default = false
}
variable "enable_ssh_tunnel" {
description = "Whether to enable SSH tunneling for local access"
type = bool
default = false
}
variable "k3s_extra_server_args" {
description = "Additional arguments to pass to the K3s server installation"
type = string
default = ""
}
variable "k3s_extra_agent_args" {
description = "Additional arguments to pass to the K3s agent installation"
type = string
default = ""
}

270
terraform/modules/k3s-install/main.tf Normal file
View File

@@ -0,0 +1,270 @@
terraform {
required_version = ">= 1.0.0"
required_providers {
null = {
source = "hashicorp/null"
version = "~> 3.2.0"
}
local = {
source = "hashicorp/local"
version = "~> 2.4.0"
}
}
}
locals {
ssh_config_path = "${path.module}/ssh_config"
temporary_dir = "/tmp/k3s-terraform"
server_kubeconfig = "${local.temporary_dir}/k3s.yaml"
node_token_path = "${local.temporary_dir}/node-token"
worker_count = length(var.worker_ips)
}
# Create a temporary SSH config file for secure connections
resource "null_resource" "setup_ssh_config" {
triggers = {
server_ip = var.server_ip
worker_ips = join(",", var.worker_ips)
ssh_user = var.ssh_user
ssh_private_key = var.ssh_private_key
}
provisioner "local-exec" {
command = <<-EOT
mkdir -p ${dirname(local.ssh_config_path)}
cat > ${local.ssh_config_path} << 'EOF'
Host ${var.server_ip}
User ${var.ssh_user}
IdentityFile ${var.ssh_private_key}
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
${join("\n", [
for ip in var.worker_ips : <<-WORKER
Host ${ip}
User ${var.ssh_user}
IdentityFile ${var.ssh_private_key}
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
WORKER
])}
EOF
EOT
interpreter = ["bash", "-c"]
}
# Clean up SSH config on destroy
provisioner "local-exec" {
when = destroy
command = "rm -f ${self.triggers.ssh_config_path}"
interpreter = ["bash", "-c"]
on_failure = continue
}
}
# Install K3s on the server node
resource "null_resource" "install_k3s_server" {
depends_on = [null_resource.setup_ssh_config]
triggers = {
server_ip = var.server_ip
ssh_user = var.ssh_user
ssh_private_key = var.ssh_private_key
ssh_config_path = local.ssh_config_path
k3s_version = var.k3s_version
k3s_extra_args = var.k3s_extra_server_args
}
# Install K3s server
provisioner "remote-exec" {
connection {
host = var.server_ip
user = var.ssh_user
private_key = file(var.ssh_private_key)
agent = false
}
inline = [
"mkdir -p ${local.temporary_dir}",
"curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=${var.k3s_version} INSTALL_K3S_CHANNEL=${var.k3s_channel} sh -s - server ${var.k3s_extra_server_args}",
"until systemctl is-active --quiet k3s; do echo 'Waiting for k3s to start...'; sleep 5; done",
"echo 'K3s server installation complete'"
]
}
# Uninstall K3s server on destroy
provisioner "remote-exec" {
when = destroy
connection {
host = self.triggers.server_ip
user = self.triggers.ssh_user
private_key = file(self.triggers.ssh_private_key)
agent = false
}
inline = [
"/usr/local/bin/k3s-uninstall.sh || true",
"rm -rf ${local.temporary_dir} || true"
]
on_failure = continue
}
}
# Retrieve the K3s kubeconfig from the server
resource "null_resource" "get_k3s_config" {
depends_on = [null_resource.install_k3s_server, null_resource.setup_ssh_config]
triggers = {
server_ip = var.server_ip
ssh_user = var.ssh_user
ssh_private_key = var.ssh_private_key
ssh_config_path = local.ssh_config_path
}
# Copy kubeconfig to a temporary location on server
provisioner "remote-exec" {
connection {
host = var.server_ip
user = var.ssh_user
private_key = file(var.ssh_private_key)
agent = false
}
inline = [
"mkdir -p ${local.temporary_dir}",
"sudo cp /etc/rancher/k3s/k3s.yaml ${local.server_kubeconfig}",
"sudo chmod 644 ${local.server_kubeconfig}"
]
}
# Download kubeconfig to local machine
provisioner "local-exec" {
command = "mkdir -p ${dirname(var.kubeconfig_path)} && scp -F ${local.ssh_config_path} ${var.ssh_user}@${var.server_ip}:${local.server_kubeconfig} ${var.kubeconfig_path}"
}
# Update server URL in kubeconfig if needed
provisioner "local-exec" {
command = <<-EOT
if [ -n "${var.replace_url}" ]; then
sed -i 's|https://127.0.0.1:6443|https://${var.replace_url}:6443|g' ${var.kubeconfig_path}
fi
EOT
interpreter = ["bash", "-c"]
on_failure = continue
}
}
# Retrieve the K3s node token from the server
resource "null_resource" "get_k3s_token" {
depends_on = [null_resource.install_k3s_server, null_resource.setup_ssh_config]
triggers = {
server_ip = var.server_ip
ssh_user = var.ssh_user
ssh_private_key = var.ssh_private_key
ssh_config_path = local.ssh_config_path
}
# Extract node token and save to a temporary file
provisioner "remote-exec" {
connection {
host = var.server_ip
user = var.ssh_user
private_key = file(var.ssh_private_key)
agent = false
}
inline = [
"mkdir -p ${local.temporary_dir}",
"sudo cat /var/lib/rancher/k3s/server/node-token > ${local.node_token_path}",
"sudo chmod 644 ${local.node_token_path}"
]
}
# Download node token to local machine
provisioner "local-exec" {
command = "mkdir -p ${dirname(var.node_token_path)} && scp -F ${local.ssh_config_path} ${var.ssh_user}@${var.server_ip}:${local.node_token_path} ${var.node_token_path}"
}
}
# Copy the node token to each worker node
resource "null_resource" "copy_token_to_workers" {
depends_on = [null_resource.get_k3s_token]
count = local.worker_count
triggers = {
worker_ip = var.worker_ips[count.index]
ssh_user = var.ssh_user
ssh_private_key = var.ssh_private_key
ssh_config_path = local.ssh_config_path
node_token_path = var.node_token_path
}
# Create temporary directory on worker and copy token
provisioner "remote-exec" {
connection {
host = var.worker_ips[count.index]
user = var.ssh_user
private_key = file(var.ssh_private_key)
agent = false
}
inline = [
"mkdir -p ${local.temporary_dir}"
]
}
# Upload token file to worker
provisioner "local-exec" {
command = "scp -F ${local.ssh_config_path} ${var.node_token_path} ${var.ssh_user}@${var.worker_ips[count.index]}:${local.node_token_path}"
}
}
# Install K3s on each worker node
resource "null_resource" "install_k3s_worker" {
depends_on = [null_resource.copy_token_to_workers]
count = local.worker_count
triggers = {
worker_ip = var.worker_ips[count.index]
server_ip = var.server_ip
ssh_user = var.ssh_user
ssh_private_key = var.ssh_private_key
ssh_config_path = local.ssh_config_path
k3s_version = var.k3s_version
k3s_extra_args = var.k3s_extra_agent_args
}
# Install K3s agent on worker
provisioner "remote-exec" {
connection {
host = var.worker_ips[count.index]
user = var.ssh_user
private_key = file(var.ssh_private_key)
agent = false
}
inline = [
"curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=${var.k3s_version} INSTALL_K3S_CHANNEL=${var.k3s_channel} K3S_URL=https://${var.server_ip}:6443 K3S_TOKEN=$(cat ${local.node_token_path}) sh -s - agent ${var.k3s_extra_agent_args}",
"until systemctl is-active --quiet k3s-agent; do echo 'Waiting for k3s-agent to start...'; sleep 5; done",
"echo 'K3s agent installation complete'"
]
}
# Uninstall K3s agent on destroy
provisioner "remote-exec" {
when = destroy
connection {
host = self.triggers.worker_ip
user = self.triggers.ssh_user
private_key = file(self.triggers.ssh_private_key)
agent = false
}
inline = [
"/usr/local/bin/k3s-agent-uninstall.sh || true",
"rm -rf ${local.temporary_dir} || true"
]
on_failure = continue
}
}

24
terraform/modules/k3s-install/outputs.tf Normal file
View File

@@ -0,0 +1,24 @@
output "kubeconfig_path" {
description = "Path to the kubeconfig file"
value = var.kubeconfig_path
}
output "server_ip" {
description = "IP address of the K3s server node"
value = var.server_ip
}
output "worker_ips" {
description = "IP addresses of the K3s worker nodes"
value = var.worker_ips
}
output "cluster_ready" {
description = "Indicator that the cluster is ready to use"
value = "true"
depends_on = [
null_resource.install_k3s_server,
null_resource.get_k3s_config,
null_resource.install_k3s_worker
]
}

69
terraform/modules/k3s-install/variables.tf Normal file
View File

@@ -0,0 +1,69 @@
variable "server_ip" {
description = "IP address of the K3s server node"
type = string
}
variable "worker_ips" {
description = "List of IP addresses of the K3s worker nodes"
type = list(string)
default = []
}
variable "ssh_user" {
description = "SSH username for connecting to nodes"
type = string
default = "root"
}
variable "ssh_private_key" {
description = "Path to the SSH private key for authentication"
type = string
}
variable "replace_url" {
description = "URL to replace in the kubeconfig (usually the server_ip or domain name)"
type = string
default = ""
}
variable "k3s_version" {
description = "Version of K3s to install"
type = string
default = "v1.27.3+k3s1"
}
variable "k3s_channel" {
description = "Release channel of K3s to install (stable, latest, etc.)"
type = string
default = "stable"
}
variable "k3s_extra_server_args" {
description = "Extra arguments to pass to the K3s server"
type = string
default = ""
}
variable "k3s_extra_agent_args" {
description = "Extra arguments to pass to the K3s agent"
type = string
default = ""
}
variable "kubeconfig_path" {
description = "Path where to save the kubeconfig file locally"
type = string
default = "~/.kube/config"
}
variable "ssh_key_algorithm" {
description = "SSH key algorithm used"
type = string
default = "rsa"
}
variable "node_token_path" {
description = "Path where to save the node token locally"
type = string
default = "/tmp/node-token"
}