DevOps-Bash-tools

HariSekhon / DevOps-Bash-tools

一套包含 1200+ 个 DevOps 实用 Shell 脚本的集合，覆盖云平台、容器、数据库、CI/CD 等运维场景。

Shell 开发工具 基础设施 安全 DevOps Shell脚本 运维自动化 CI/CD dotfiles

为什么值得看

编辑点评

脚本数量庞大且覆盖面广，从 AWS/GCP 到 Kubernetes、Kafka、Jenkins 等常见工具链均有涉及，还附带高级 dotfiles 配置。适合需要快速完成自动化任务或寻找运维脚本参考的开发者。注意脚本质量参差不齐，使用时需结合自身环境调整，并留意依赖和权限问题。

Star 趋势

近 7 日
  • Star 总数8,411
  • 今日新增+0
  • 7 日增速-100%
  • Fork1,587

同类项目

同场景 · 开发工具 / 基础设施 / 安全
ECC
affaan-m
JavaScript▲ 487
n8n
n8n-io
TypeScript▲ 100
yt-dlp
yt-dlp
Python▲ 107
Microsoft-Activation-Scripts
massgravel
Batchfile▲ 75

项目文档

来自 GitHub README · master 分支

Hari Sekhon - DevOps Bash Tools

LineCount Cocomo

Codefresh

Semaphore

Azure DevOps Pipeline

git.io/bash-tools

1200+ DevOps Shell Scripts and Advanced Bash environment.

Fast, Advanced Systems Engineering, Automation, APIs, shorter CLIs, etc.

Heavily used in many GitHub repos, dozens of DockerHub builds (Dockerfiles) and 600+ CI builds.

Summary

  • Scripts for many popular DevOps technologies, see Index below for more details
  • Advanced configs for common tools like Git, vim, screen, tmux, PostgreSQL psql etc...
  • CI configs for most major Continuous Integration products (see CI builds page)
  • CI scripts for a drop-in framework of standard checks to run in all CI builds, CI detection, accounting for installation differences across CI environments, root vs user, virtualenvs etc.
  • API scripts auto-handling authentication, tokens and other details to quickly query popular APIs with a few keystrokes just supplying the /path/endpoint
  • Advanced Bash environment - .bashrc + .bash.d/*.sh - aliases, functions, colouring, dynamic Git & shell behaviour enhancements, automatic pathing for installations and major languages like Python, Perl, Ruby, NodeJS, Golang across Linux distributions and Mac. See .bash.d/README.md
  • Installs the best systems packages - AWS CLI, Azure CLI, GCloud SDK, Digital Ocean CLI, Terraform, Terragrunt, GitHub CLI, Kubernetes kubectl & kustomize, Helm, eksctl, Docker-Compose, jq and many others... extensive package lists for servers and desktops for most major Linux distributions package managers and Mac
  • install/ - contains many installation scripts for popular open source software and direct binary downloads from GitHub releases
  • configs/ - contains many dot configs for common technologies like ViM, top, Screen, Tmux, MySQL, PostgreSQL etc.
  • setup/ - contains setup scripts, package lists, extra configs, Mac OS X settings etc.
  • Utility Libraries used by many hundreds of scripts and builds across repos:
  • .bash.d/ - interactive library
  • lib/ - scripting and CI library
  • SQL Scripts - 100+ scripts for PostgreSQL, MySQL, AWS Athena + CloudTrail, Google BigQuery
  • Templates - templates for common programming languages and build configs
  • Kubernetes Configs - Kubernetes YAML configs for most common scenarios, including Production Best Practices, Tips & Tricks

See Also: similar DevOps repos in other languages

Hari Sekhon

Cloud & Big Data Contractor, United Kingdom

(ex-Cloudera, former Hortonworks Consultant)


(you're welcome to connect with me on LinkedIn)

Quick Setup

To bootstrap, install packages and link in to your shell profile to inherit all configs, do:

curl -L https://git.io/bash-bootstrap | sh
  • Adds sourcing to .bashrc/.bash_profile to automatically inherit all .bash.d/*.sh environment enhancements for all technologies (see Inventory below)
  • Symlinks .* config dotfiles to $HOME for git, vim, top, htop, screen, tmux, editorconfig, Ansible, PostgreSQL .psqlrc etc. (only when they don't already exist so there is no conflict with your own configs)
  • Installs OS package dependencies for all scripts (detects the OS and installs the right RPMs, Debs, Apk or Mac HomeBrew packages)
  • Installs Python packages
  • Installs AWS CLI

To only install package dependencies to run scripts, simply cd to the git clone directory and run make:

git clone https://github.com/HariSekhon/DevOps-Bash-tools bash-tools
cd bash-tools
make

make install sets your shell profile to source this repo. See Individual Setup Parts below for more install/uninstall options.

Index

  • Dot Configs - .gitconfig, .vimrc, .screenrc, .tmux.conf, .toprc, .gitignore...
  • Bash Environment & Libraries - .bashrc, .bash.d/ interactive library, lib/ scripting library
  • Installation Scripts for many popular open source technologies
  • Linux & Mac - curl OAuth / JWT, LDAP, find duplicate files, SSL certificate get/validate, URL encoding/decoding, Vagrant
  • Mac & AppleScript - Mac settings and UI automation scripts, send keystrokes, mouse clicks, detect foreground app, switch app, detect locked screen or screensaver, activate screensaver, Hammerspoon system event handlers such as automatically switching audio to be able to Shazam while watching on AirPods
  • Monitoring - Grafana, Prometheus, Node Exporter, scripted collection of common Linux & Mac cli monitoring stats and log locations for quick generation of vendor support tarball bundles both locally and over SSH
  • AWS - Amazon Web Services - AWS account summary, lots of IAM reports, CIS Benchmark config hardening, EC2, ECR, EKS, Spot termination, S3 access logging, KMS key rotation info, SSM, CloudTrail, CloudWatch billing alarm with SNS notification topic and subscription for email alerts
  • GCP - Google Cloud Platform - massive GCP auto-inventory, scripts for GCE, GKE, GCR, Secret Manager, BigQuery, Cloud SQL, Cloud Scheduler, Terraform service account creation
  • Kubernetes - massive Kubernetes auto-inventory, cluster management scripts & tricks
  • Docker - Docker API, Dockerhub API, Quay.io API scripts
  • Databases - fast CLI wrappers, instant Docker sandboxes (PostgreSQL, MySQL, MariaDB, SQLite), SQL scripts, SQL script testers against all versions of a DB, advanced .psqlrc
  • Data - data tools, converters and format validators for Avro, Parquet, CSV, JSON, INI / Properties files (Java), LDAP LDIF, XML, YAML
  • Big Data & NoSQL - Kafka, Hadoop, HDFS, Hive, Impala, ZooKeeper, Cloudera Manager API & Cloudera Navigator API scripts
  • Git - GitHub, GitLab, Bitbucket, Azure DevOps - scripts for Git local & mirror management, GitHub, GitLab & BitBucket APIs
  • Markdown - generate Markdown indexes and debug mdl issues like MD005 inconsistent list indentation in large README.md files
  • CI/CD - Continuous Integration / Continuous Delivery - API scripts & build pipeline configs for most major CI systems:
  • Jenkins, Concourse, GoCD, TeamCity - one-touch boot & build
  • Azure DevOps Pipelines, GitHub Actions Workflows, GitLab CI, BitBucket Pipelines, AppVeyor, BuildKite, Travis CI, Circle CI, Codefresh, CodeShip, Drone.io, Semaphore CI, Shippable ...
  • Terraform Cloud, Octopus Deploy
  • Checkov / Bridgecrew Cloud
  • AI & IPaaS - OpenAI (ChatGPT), Make.com
  • Internet Services - Google Maps, Cloudflare, DataDog, Digital Ocean, Kong API Gateway, GitGuardian, Jira, NGrok, Traefik, Pingdom, Wordpress and various pastebins and file upload sites
  • Java - Java utilies to debug running Java programs or decompile Java JAR code for deeper debugging
  • Python - Python utilities & library management
  • Perl - Perl utilities & library management
  • Golang - Golang utilities
  • Diagrams - scripts to generate diagrams from D2lang, MermaidJS and Python Mingrammer source code used in my HariSekhon/Diagrams-as-Code repo
  • Media - video downloaders & converts, MP3 metadata editing, grouping and ordering of albums and audiobooks, mkv/avi to mp4 converters, 720p video downscaler for posting to social media, download YouTube videos or even entire channels and videos from other social media sites like Twitter / X or Facebook, terminal gif capture
  • Spotify - 40+ Spotify API scripts for backups, managing playlists, track deduplication, URI conversion, search, add/delete, liked tracks, followed artists, top artists, top tracks etc.
  • More Linux & Mac - more systems administration scripts, package installation automation
  • Builds, Languages & Linting - programming language, build system & CI linting
  • Templates - Templates for AWS, GCP, Terraform, Docker, Jenkins, Cloud Build, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml etc.
  • Kubernetes Configs - Kubernetes YAML configs for most common scenarios, including Production Best Practices, Tips & Tricks

Dot Configs

Top-level dotfiles and configs/ directory:

  • .* - dot conf files for lots of common software eg. advanced .vimrc, .gitconfig, massive .gitignore, .editorconfig, .screenrc, .tmux.conf etc.
  • .vimrc - contains many awesome vim tweaks, plus hotkeys for linting lots of different file types in place, including Python, Perl, Bash / Shell, Dockerfiles, JSON, YAML, XML, CSV, INI / Properties files, LDAP LDIF etc without leaving the editor!
  • .screenrc - fancy screen configuration including advanced colour bar, large history, hotkey reloading, auto-blanking etc.
  • .tmux.conf - fancy tmux configuration include advanced colour bar and plugins, settings, hotkey reloading etc.
  • Git:
    • .gitconfig - advanced Git configuration
    • .gitignore - extensive Git ignore of trivial files you shouldn't commit
    • enhanced Git diffs
    • protections against committing AWS secret keys or merge conflict unresolved files

Bash Environment & Libraries

Top-level .bashrc and .bash.d/ directory:

  • .bashrc - shell tuning and sourcing of .bash.d/*.sh
  • .bash.d/*.sh - thousands of lines of advanced bashrc code, aliases, functions and environment variables for:
  • Linux & Mac
  • SCM - Git, Mercurial, Svn
  • AWS
  • GCP
  • Docker
  • Kubernetes
  • Kafka
  • Vagrant
  • automatic GPG and SSH agent handling for handling encrypted private keys without re-entering passwords, and lazy evaluation to only prompt key load the first time SSH is called
  • and lots more - see .bash.d/README for a more detailed list
  • run make bash to link .bashrc/.bash_profile and the .* dot config files to your $HOME directory to auto-inherit everything
  • lib/*.sh - Bash utility libraries full of functions for Docker, environment, CI detection (Travis CI, Jenkins etc), port and HTTP url availability content checks etc. Sourced from all my other GitHub repos to make setting up Dockerized tests easier.

Installation Scripts

Linux & Mac

bin/ directory:

  • login.sh - logs to major Cloud platforms if their credentials are found in the environment, CLIs such as AWS, GCP, Azure, GitHub... Docker registries: DockerHub, GHCR, ECR, GCR, GAR, ACR, Gitlab, Quay...
  • clean_caches.sh - cleans out OS package and programming language caches - useful to save space or reduce Docker image size
  • command_return_to_current_window.sh - runs a Mac or Linux command that opens a window and then switches back to the original foreground window
  • crypto_dice_rolls.sh - generates 100 random dice rolls to test a new crypto hardware wallet's fidelity (do not use this for your real crypto seed as your machine could be infected with malware which steals your seed phrase)
  • delete_duplicate_files.sh - deletes duplicate files with (N) suffixes, commonly caused by web browser downloads, in the given or current directory. Checks they're exact duplicates of a matching basename file without the (N) suffix with the exact same checksum for safety. Prompts to delete per file. To auto-accept deletions, do yes | delete_duplicate_files.sh. This is a fast way of cleaning up your ~/Downloads directory and can be put your user crontab
  • disk_speed_read_sequential_dd.sh - runs a sequential read speed test from the given file using dd and bypassing filesystem cache for a more accurate test
  • disk_speed_read_random_dd.sh - runs a random I/O read speed test from the given file using dd and bypassing filesystem cache for a more accurate test
  • disk_speed_write_sequential_dd.sh - runs a sequential write speed test to a file in the given or current directory using dd and bypassing filesystem cache for a more accurate test
  • disk_speed_read_sequential_fio.sh - runs a sequential read speed test in the current or given directory using fio
  • disk_speed_read_random_fio.sh - runs a random I/O read test in the current or given directory using fio
  • disk_speed_write_sequential_fio.sh - runs a sequential write speed test to the current or given directory using fio
  • disk_speed_write_random_fio.sh - runs a sequential write speed test to the current or given directory using fio
  • download_url_file.sh - downloads a file from a URL using wget with no clobber and continue support, or curl with atomic replacement to avoid race conditions. Used by github/github_download_release_file.sh, github_download_release_jar.sh, and install/download_*_jar.sh
  • curl_auth.sh - shortens curl command by auto-loading your OAuth2 / JWT API token or username & password from environment variables or interactive starred password prompt through a ram file descriptor to avoid placing them on the command line (which would expose your credentials in the process list or OS audit log files). Used by many other adjacent API querying scripts
  • curl_with_cookies.sh - extracts cookies for a given URL from your \$BROWSER's cookie jar and passes them to the curl command along with the rest of the args (workaround for older curl builds and Homebrew builds that don't have the newer `--cookies-from-browser functionality)
  • find_duplicate_files*.sh - finds duplicate files by size and/or checksum in given directory trees. Checksums are only done on files that already have matching byte counts for efficiency
  • find_broken_links.sh - find broken links with delays to avoid tripping defenses
  • find_broken_symlinks.sh - find broken symlinks pointing to non-existent files/directories
  • find_lock.sh - tries to find if a lockfile is used in the given or current working directory by taking snapshots of the file list before and after a prompt in which you should open/close an application
  • foreach_path_bin.sh - runs each binary of the given name found in $PATH with the args given. Useful to find all the installed versions of a program in different paths eg. ~/bin/ vs /usr/local/bin/ eg. foreach_path_bin.sh terraform --version
  • http_duplicate_urls.sh - find duplicate URLs in a given web page
  • htmldecode.sh - decodes HTML encoding. Detects available tools such as Perl, Python or xmlstarlet and uses whatever is available
  • ldapsearch.sh - shortens ldapsearch command by inferring switches from environment variables
  • ldap_user_recurse.sh / ldap_group_recurse.sh - recurse Active Directory LDAP users upwards to find all parent groups, or groups downwards to find all nested users (useful for debugging LDAP integration and group-based permissions)
  • linux_distro_versions.sh - quickly returns the list of major versions for a given Linux distro
  • linux_command_return_to_current_window.sh - runs a Linux command that opens a window and then switches back to the original foreground window
  • mac_command_return_to_current_window.sh - runs a Mac command that opens a window and then switches back to the original foreground window
  • diff_line_threshold.sh - compares two files vs a line count diff threshold to determine if they are radically different. Used to avoid overwriting files which are not mere updates but completely different files
  • mv.sh - moves directory trees resumably and removes the source files as they're copied over. Useful to migrate data from one disk to another, optionally with checksums. Uses rsync and shows the overall % of files transferred and the MB/s data transfer rate
  • network_gateway.sh - get the network gateway IP address on Linux or Mac
  • network_randomize_mac_address.sh - assigns a new random mac address to your network interface
  • open.sh - opens given arg, file or URL using whatever default system opener is available for Linux or Mac
  • organize_downloads.sh - moves files of well-known extensions in the $HOME/Downloads directory older than 1 week to capitalized subdirectories of their type to keep the $HOME/Downloads/ directory tidy
  • copy_to_clipboard.sh - copies stdin or string arg to system clipboard on Linux or Mac
  • paste_from_clipboard.sh - pastes from system clipboard to stdout on Linux or Mac
  • paste_from_clipboard_upon_changes.sh - pastes from system clipboard to stdout on Linux or Mac whenever the clipboard changes
  • paste_diff_settings.sh - takes snapshots of before and after clipboard changes and diffs them to show config changes
  • processes_ram_sum.sh - sums the RAM usage of all processes matching a given regex in GB to one decimal place
  • pldd.sh - parses /proc on Linux to show the runtime .so loaded dynamic shared libraries a program pid is using. Runtime equivalent of the classic static ldd command and because the system pldd command often fails to attach to a process
  • random_select.sh - selects one of given args at random. Useful for sampling, running randomized subsets of large test suites etc.
  • random_number.sh - prints a random integer between two integer arguments (inclusive)
  • random_string.sh - prints a random alphanumeric string of a given length
  • screen_terminal_to_stdout.sh - dumps the GNU Screen terminal output to stdout
  • screen_terminal_to_clipboard.sh - dumps the GNU Screen terminal output to a temp file and copies to clipboard for sharing & debugging purposes
  • shields_embed_logo.sh - base64 encodes a given icon file or url and prints the logo=... url parameter you need to add the shields.io badge url
  • shorten_text_selection.sh - shortens the selected text in the prior window. Replaces and with & and crushes out multiple blank lines. I use this for LinkedIn comments due to the short 1250 character limit
  • shred_file.sh - overwrites a file 7 times to DoD standards before deleting it to prevent recovery of sensitive information
  • shred_free_space.sh - overwrites free space to prevent recovery of sensitive information for files that have already been deleted
  • split.sh - split large files into N parts (defaults to the number of your CPU cores) to parallelize operations on them
  • ssl_get_cert.sh - gets a remote host:port server's SSL cert in a format you can pipe, save and use locally, for example in Java truststores
  • ssl_verify_cert.sh - verifies a remote SSL certificate (battle tested more feature-rich version check_ssl_cert.pl exists in the Advanced Nagios Plugins repo)
  • ssl_verify_cert_by_ip.sh - verifies SSL certificates on specific IP addresses, useful to test SSL source addresses for CDNs, such as Cloudflare Proxied sources before enabling SSL Full-Strict Mode for end-to-end, or Kubernetes ingresses (see also curl_k8s_ingress.sh)
  • text_filter_ending_substrings.sh - for a given patterns file of substring endings, print all lines that match in the following files. Uses awk to safely handle all characters as literals, unlike grep, while also maintaining end anchoring which you cannot do using grep -F. Optimized awk code uses a bucketing hash for performance to not attempt matching lines which are shorter than patterns, reducing the number of match attempts
  • tmux_vertical.sh - launches tmux with N-way vertical shell split or commands given as args in equally balanced vertical panes. Fast way to launch a bunch of shell or commands in an easily reviewable side-by-side way
  • tmux_horizontal.sh - same as above but split horizontally
  • tmux_square.sh - same as above but with 4 panes in a square tiled view
  • urlencode.sh / urldecode.sh - URL encode/decode quickly on the command line, in pipes etc.
  • urlextract.sh - extracts the URLs from a given string arg, file or standard input
  • url_extract_redirects.sh - extracts the URLs from a given string arg, file or standard input, queries each one and outputs the redirected urls instead to stdout
  • url_replace_redirects.sh - extracts the URLs from a given string arg, file or standard input, queries each one and outputs the entire contents to stdout with the urls replaced by the redirected urls
  • urlopen.sh - opens the URL given as an arg, or first URL found from stdin or a given file. Uses the system's default browser
  • vagrant_hosts.sh - generate /etc/hosts output from a Vagrantfile
  • vagrant_total_mb.sh - calculate the RAM committed to VMs in a Vagrantfile

See also Knowledge Base notes for Linux and Mac.

Mac & AppleScript

Mac automation scripts to automate the Mac UI and settings

bin/ directory:

  • mac_diff_settings.sh - takes before and after snapshots of UI setting changes and diffs them to make it easy to find defaults keys to add to setup/mac_settings.sh to save settings
  • mac_restore_file.sh - checks all the backup mount points for the latest backup that has a given file and then restores it
  • mac_backup_du_in_progress.sh - find large files in the currently in-progress Time Machine backup to find out what is taking so long and racking up so many more GB of changes than you expect. This helps discover large but unnecessary files that you might want to exclude using the adjacent script mac_backup_exclude_paths.sh
  • mac_backup_exclude_paths.sh - excludes many common large caches, docker and VM paths from macOS Time Machine backups
  • mac_backup_find_excluded_paths.sh - does a deep search for macOS Time Machine excluded backup paths on file/folder attributes. See HariSekhon/Knowledge-Base Mac page for why
  • mac_command_return_to_current_window.sh - runs a Mac command that opens a window and then switches back to the original foreground window
  • mac_rmdir.sh - safely delete a directory on Mac only if it is empty of actual data, by first removing macOS hidden metadata files and dirs such as .fseventsd/, .Spotlight-V100/ and .DS_Store - straight rmdir fails otherwise
  • mac_iso_to_usb.sh - converts a given ISO file to a USB bootable image and burns it onto a given or detected inserted USB drive
  • mac_ramdisk.sh - creates a mac ramdisk of given MB size. Useful for performance, or even testing disk write scripts such as disk_speed_write_*.sh without wearing out your SSD
  • mac_delete_local_snapshots.sh - deletes local macOS snapshots to free up disk space. When there is a substantial discrepancy between what the df -h command and the Finder UI shows, this is often the cause
  • copy_to_clipboard.sh - copies stdin or string arg to system clipboard on Linux or Mac
  • paste_from_clipboard.sh - pastes from system clipboard to stdout on Linux or Mac
  • paste_from_clipboard_upon_changes.sh - pastes from system clipboard to stdout on Linux or Mac whenever the clipboard changes
  • paste_diff_settings.sh - Takes snapshots of before and after clipboard changes and diffs them to show config changes

applescript/ directory:

  • keystrokes.sh - send N keystroke combinations
  • mouse_clicks.sh - send N mouse click combinations to sequence of screen coordinates
  • get_mouse_coordinates.sh - print the current mouse coordinates - to know what to pass to above script
  • mouse_clicks_remote_desktop.sh - switches to Microsoft Remote Desktop, waits 10 seconds and then clicks the mouse once a minute to prevent the screensaver from coming on. Workaround to Active Directory Group Policies that don't let you disable the screensaver. Point your mouse to an area that will have no mouse click effect, the Cmd-Tab to Terminal and run this
  • get_frontmost_process_title.scpt - detect the frontmost window
  • to detect if you should send keystrokes / mouse clicks)
  • set_frontmost_process.scpt - switch to bring the given app to the foreground to send keystrokes / mouse clicks to it
  • browser_get_default.scpt - get the default configured browser in format passable to Applescript (for above script)
  • is_screen_locked.py - detect if the screen is locked to stop sending keystrokes or mouse clicks
  • is_screensaver_running.scpt - detect if the screensaver is running to stop sending keystrokes or mouse clicks
  • reopen_app.sh - relaunch a given app (used to reload Shazam to detect DB changes after removing tracks programmatically from its DB)
  • spotify_app_search.sh - runs a search in the Spotify App on Mac using Applescript
  • spotify_app_search_filelist.sh - searches the Spotify App for each track in a file, prompting to continue to the next track
  • shazam_app_dump_tracks.sh - dumps artist - track one per line from the Shazam local sqlite DB
  • shazam_app_delete_track.sh - deletes a given "artist" "track" from the Shazam local sqlite DB
  • shazam_search_spotify_then_delete_track.sh - searches for each Shazam'd track in the local Spotify desktop app, then prompts to delete each track from the local Shazam DB once you've saved it in Spotify. Useful to migrate Shazam'd tracks to Spotify after Apple removed the integration
  • screensaver_activate.scpt - activate screensaver
  • shorten_text_selection.scpt - shortens the selected text in the prior window. Replaces and with & and crushes out multiple blank lines. I use this for LinkedIn comments due to the short 1250 character limit
  • start_app_at_login.sh - adds an app to the Login items to auto-start

Hammerspoon code has been moved to its own repo:

Readme Card

See also Mac page in HariSekhon/Knowledge-Base.

Monitoring

monitoring/ directory:

  • dump_stats.sh - dumps common command outputs to text files in a local tarball. Useful to collect support information for vendor support cases
  • grafana_api.sh - queries the Grafana API with authentication
  • log_timestamp_large_intervals.sh - finds log lines whose timestamp intervals exceed the given number of seconds and outputs those log lines with the difference between the last and current timestamps. Useful to find actions that are taking a long time from log files such as CI/CD logs
  • prometheus.sh - starts Prometheus locally, downloading it if not found in $PATH
  • prometheus_docker.sh - starts Prometheus in Docker using docker-compose
  • prometheus_node_exporter.sh - starts Prometheus node_exporter locally, downloading it if not found in $PATH
  • ssh_dump_stats.sh - uses SSH and dump_stats.sh to dump common command outputs from remote servers to a local tarball. Useful for vendor support cases
  • ssh_dump_logs.sh - Uses SSH to dump logs from server to local text files for uploading to vendor support cases

See doc pages in HariSekhon/Knowledge-Base on Grafana, Prometheus, OpenTSDB, InfluxDB etc.

Databases

mysql/, postgres/, sql/ and bin/ directories:

  • sql/ - 100+ SQL scripts for PostgreSQL, MySQL, Google BigQuery and AWS Athena CloudTrail logs integration
  • sqlite.sh - one-touch SQLite, starts sqlite3 shell with sample 'chinook' database loaded
  • mysql*.sh - MySQL scripts:
  • mysql.sh - shortens mysql command to connect to MySQL by auto-populating switches from both standard environment variables like $MYSQL_TCP_PORT, $DBI_USER, $MYSQL_PWD (see doc) and other common environment variables like $MYSQL_HOST / $HOST, $MYSQL_USER / $USER, $MYSQL_PASSWORD / $PASSWORD, $MYSQL_DATABASE / $DATABASE
  • mysql_foreach_table.sh - executes a SQL query against every table, replacing {db} and {table} in each iteration eg. select count(*) from {table}
  • mysql_*.sh - various scripts using mysql.sh for row counts, iterating each table, or outputting clean lists of databases and tables for quick scripting
  • mysqld.sh - one-touch MySQL, boots docker container + drops in to mysql shell, with /sql scripts mounted in container for easy sourcing eg. source /sql/<name>.sql. Optionally loads sample 'chinook' database
  • see also the SQL Scripts repo for many more straight MySQL SQL scripts
  • mariadb.sh - one-touch MariaDB, boots docker container + drops in to mysql shell, with /sql scripts mounted in container for easy sourcing eg. source /sql/<name>.sql. Optionally loads sample 'chinook' database
  • postgres*.sh / psql.sh - PostgreSQL scripts:
  • postgres.sh - one-touch PostgreSQL, boots docker container + drops in to psql shell, with /sql scripts mounted in container for easy sourcing eg. \i /sql/<name>.sql. Optionally loads sample 'chinook' database
  • psql.sh - shortens psql command to connect to PostreSQL by auto-populating switches from environment variables, using both standard postgres supported environment variables like $PG* (see doc) as well as other common environment variables like $POSTGRESQL_HOST / $POSTGRES_HOST / $HOST, $POSTGRESQL_USER / $POSTGRES_USER / $USER, $POSTGRESQL_PASSWORD / $POSTGRES_PASSWORD / $PASSWORD, $POSTGRESQL_DATABASE / $POSTGRES_DATABASE / $DATABASE
  • postgres_foreach_table.sh - executes a SQL query against every table, replacing {db}, {schema} and {table} in each iteration eg. select count(*) from {table}
  • postgres_*.sh - various scripts using psql.sh for row counts, iterating each table, or outputting clean lists of databases, schemas and tables for quick scripting
  • checks/check_sqlfluff.sh - recursively iterates all SQL code files found in the given or current directory and runs SQLFluff linter against them, inferring the different SQL dialects from each path/filename/extension

AWS - Amazon Web Services

aws/ directory:

  • AWS scripts - aws_*.sh:
  • aws_profile.sh - switches to an AWS Profile selected from a convenient interactive menu list of AWS profiles from $AWS_CONFIG_FILE - useful when you have lots of AWS work profiles
  • aws_cli_create_credential.sh - creates an AWS service account user for CI/CD or CLI with Admin permissions (or other group or policy), creates an AWS Access Key, saves a credentials CSV and even prints the shell export commands and aws credentials file config to configure your environment to start using it. Useful trick to avoid CLI reauth to aws sso login every day.
  • aws_terraform_create_credential.sh - creates a AWS terraform service account with Administrator permissions for Terraform Cloud or other CI/CD systems to run Terraform plan and apply, since no CI/CD systems can work with AWS SSO workflows. Stores the access key as both CSV and prints shell export commands and credentials file config as above
  • .envrc-aws - copy to .envrc for direnv to auto-load AWS configuration settings such as AWS Profile, Compute Region, EKS cluster kubectl context etc.
    • calls .envrc-kubernetes to set the kubectl context isolated to current shell to prevent race conditions between shells and scripts caused by otherwise naively changing the global ~/.kube/config context
  • aws_sso_ssh.sh - launches local AWS SSO authentication pop-up (if not already authenticated), then scp's the latest resultant ~/.aws/sso/cache/ file to the remote server and SSH's there so that you can use AWS CLI or kubectl to EKS remotely on that server easily, without having to copy and paste the token from remote aws sso login to your local web browser
  • aws_terraform_create_s3_bucket.sh - creates a Terraform S3 bucket for storing the backend state, locks out public access, enables versioning, encryption, and locks out Power Users role and optionally any given user/group/role ARNs via a bucket policy for safety
  • aws_terraform_create_dynamodb_table.sh - creates a Terraform locking table in DynamoDB for use with the S3 backend, plus custom IAM policy which can be applied to less privileged accounts
  • aws_terraform_create_all.sh - runs all of the above, plus also applies the custom DynamoDB IAM policy to the user to ensure if the account is less privileged it can still get the Terraform lock (useful for GitHub Actions environment secret for a read only user to generate Terraform Plans in Pull Request without needing approval)
  • aws_terraform_iam_grant_s3_dynamodb.sh - creates IAM policies to access any S3 buckets and DynamoDB tables with terraform-state or tf-state in their names, and attaches them to the given user. Useful for limited permissions CI/CD accounts that run Terraform Plan eg. in GitHub Actions pull requests
  • aws_account_summary.sh - prints AWS account summary in key = value pairs for easy viewing / grepping of things like AccountMFAEnabled, AccountAccessKeysPresent, useful for checking whether the root account has MFA enabled and no access keys, comparing number of users vs number of MFA devices etc. (see also check_aws_root_account.py in Advanced Nagios Plugins)
  • aws_billing_alarm.sh - creates a CloudWatch billing alarm and SNS topic with subscription to email you when you incur charges above a given threshold. This is often the first thing you want to do on an account
  • aws_budget_alarm.sh - creates an AWS Budgets billing alarm and SNS topic with subscription to email you when both when you start incurring forecasted charges of over 80% of your budget, and 90% actual usage. This is often the first thing you want to do on an account
  • aws_batch_stale_jobs.sh - lists AWS Batch jobs that are older than N hours in a given queue
  • aws_batch_kill_stale_jobs.sh - finds and kills AWS Batch jobs that are older than N hours in a given queue
  • aws_cloudfront_distribution_for_origin.sh - returns the AWS CloudFront ARN of the distribution which serves origins containing a given substring. Useful for quickly finding the CloudFront ARN needed to give permissions to a private S3 bucket exposed via CloudFront
  • aws_cloudtrails_cloudwatch.sh - lists Cloud Trails and their last delivery to CloudWatch Logs (should be recent)
  • aws_cloudtrails_event_selectors.sh - lists Cloud Trails and their event selectors to check each one has at least one event selector
  • aws_cloudtrails_s3_accesslogging.sh - lists Cloud Trails buckets and their Access Logging prefix and target bucket. Checks S3 access logging is enabled
  • aws_cloudtrails_s3_kms.sh - lists Cloud Trails and whether their S3 buckets are KMS secured
  • aws_cloudtrails_status.sh - lists Cloud Trails status - if logging, multi-region and log file validation enabled
  • aws_config_all_types.sh - lists AWS Config recorders, checking all resource types are supported (should be true) and includes global resources (should be true)
  • aws_config_recording.sh - lists AWS Config recorders, their recording status (should be true) and their last status (should be success)
  • aws_csv_creds.sh - prints AWS credentials from a CSV file as shell export statements. Useful to quickly switch your shell to some exported credentials from a service account for testing permissions or pipe to upload to a CI/CD system via an API (eg. jenkins_cred_add*.sh, github_actions_repo*_set_secret.sh, gitlab_*_set_env_vars.sh, circleci_*_set_env_vars.sh, bitbucket_*_set_env_vars.sh, terraform_cloud_*_set_vars.sh, kubectl_kv_to_secret.sh). Supports new user and new access key csv file formats.
  • aws_codecommit_csv_creds.sh - prints AWS CodeCommit Git credentials from a CSV file as shell export statements. Similar use case and chaining as above
  • aws_ec2_*.sh - AWS EC2 scripts:
    • aws_ec2_instance_*.sh - AWS EC2 Instance scripts:
    • aws_ec2_instance_name_to_id.sh - looks up an EC2 instance ID from an instance name with extra safety checks that only a single instance ID is returned and a reverse lookup on that instance ID to re-verify it matches the name. This level of safety is important when wanting to terminate an EC2 instance by name. If an instance ID is passed, returns it as is for convenience. Used by adjacent scripts
    • aws_ec2_instances.sh - lists AWS EC2 instances, their DNS names and States in an easy to read table output
    • aws_ec2_instance_ip.sh - determines an EC2 instance IP address, trying first for a public IP, or failing that a private IP
    • aws_ec2_instance_clone.sh - clones an AWS EC2 instance by creating an AMI from the original and then booting a new instance from the AMI with the same settings as the original instance. Useful to testing risky things on a separate EC2 instance, such as Server Administrator recovery of Tableau
    • aws_ec2_instance_wait_for_ready.sh - polls an AWS EC2 instance and waits for it to finish initializing to a ready state. Used by adjacent scripts
    • aws_ec2_instance_terminate_by_name.sh - terminate an AWS EC2 instance by name for convenience, resolves its instance ID, verifies unique and then terminates by ID
    • aws_ec2_ami*.sh - AWS EC2 AMI scripts:
    • aws_ec2_amis.sh - list AWS EC2 AMIs belonging to your account in an easy to read table output
    • aws_ec2_ami_ids.sh - lists AWS EC2 AMI IDs only, one per line, to be used in adjacent scripts that creating mapping tables and translate AMI IDs to names in inventory scripts aws_info_ec2*.sh
    • aws_ec2_ami_name_to_id.sh - looks up an EC2 AMI ID from a name with extra safety checks that only a single AMI ID is returned and a reverse lookup on that AMI ID to re-verify it matches the name
    • aws_ec2_ami_boot.sh - boots a personal EC2 instance of a given AMI for testing
    • aws_ec2_ami_boot_ssh.sh - boots a personal EC2 instance of a given AMI, determines the public or private IP, and drops you into an SSH shell
    • aws_ec2_ami_create_from_instance.sh - creates an AWS EC2 AMI from an EC2 instance and waits for it to become available for use
    • aws_ec2_ami_share_to_account.sh - shares an AMI with another AWS account. Can specify AMI by name or id
    • aws_ec2_ebs_*.sh - AWS EC2 EBS scripts:
    • aws_ec2_ebs_volumes.sh - list EC2 instances and their EBS volumes in the current region
    • `aws_ec2_ebs_create_snapshot_and_wait.sh - creates a snapshot of a given EBS volume ID and waits for it to complete with exponential backoff
    • `aws_ec2_ebs_resize_and_wait.sh - resizes an EBS volume and waits for it to complete modifying and optionally optimizing with exponential backoff
    • aws_ec2_ebs_volumes_unattached.sh - list an unattached EBS volumes in a table format
    • aws_ec2_launch_templates_ami_id.sh - for each Launch Template lists the AMI ID of the latest version. Useful to check EKS upgrades of node groups via Terragrunt have taken effect
  • aws_ecr_*.sh - AWS ECR docker image management scripts:
    • aws_ecr_docker_login.sh - authenticates Docker to AWS ECR, inferring the ECR registry from the current AWS Account ID and Region
    • aws_ecr_docker_build_push.sh - builds a docker image and pushes it to ECR with not just the latest docker tag but also the current Git hashref and Git tags
    • aws_ecr_list_repos.sh - lists ECR repos, and their docker image mutability and whether image scanning is enabled
    • aws_ecr_list_tags.sh - lists all the tags for a given ECR docker image
    • aws_ecr_newest_image_tags.sh - lists the tags for the given ECR docker image with the newest creation date (can use this to determine which image version to tag as latest)
    • aws_ecr_alternate_tags.sh - lists all the tags for a given ECR docker image:tag (use arg <image>:latest to see what version / build hashref / date tag has been tagged as latest)
    • aws_ecr_tag_image.sh - tags an ECR image with another tag without pulling and pushing it
    • aws_ecr_tag_image_by_digest.sh - same as above but tags an ECR image found via digest (more accurate as reference by existing tag can be a moving target). Useful to recover images that have become untagged
    • aws_ecr_tag_latest.sh - tags a given ECR docker image:tag as latest without pulling or pushing the docker image
    • aws_ecr_tag_branch.sh - tags a given ECR image:tag with the current Git branch without pulling or pushing the docker image
    • aws_ecr_tag_datetime.sh - tags a given ECR docker image with its creation date and UTC timestamp (when it was uploaded to ECR) without pulling or pushing the docker image
    • aws_ecr_tag_newest_image_as_latest.sh - finds and tags the newest build of a given ECR docker image as latest without pulling or pushing the docker image
    • aws_ecr_tags_timestamps.sh - lists all the tags and their timestamps for a given ECR docker image
    • aws_ecr_tags_old.sh - lists tags older than N days for a given ECR docker image
    • aws_ecr_delete_old_tags.sh - deletes tags older than N days for a given ECR docker image. Lists the image:tags to be deleted and prompts for confirmation safety
  • aws_emr_clusters_last_steps.sh - shows the last N steps executed on each EMR cluster and their EndTime to find idle clusters that should be removed. Also checks CloudWatch for number of steps running within the last few months to catch directly submitted jobs such as Spark, Hive, Glue or Athena which won't show up in the native steps list
  • aws_foreach_profile.sh - executes a templated command across all AWS named profiles configured in AWS CLIv2, replacing {profile} in each iteration. Combine with other scripts for powerful functionality, auditing, setup etc. eg. aws_kube_creds.sh to configure kubectl config to all EKS clusters in all environments
  • aws_foreach_region.sh - executes a templated command against each AWS region enabled for the current account, replacing {region} in each iteration. Combine with AWS CLI or scripts to find resources across regions
  • aws_iam_*.sh - AWS IAM scripts:
    • aws_iam_password_policy.sh - prints AWS password policy in key = value pairs for easy viewing / grepping (used by aws_harden_password_policy.sh before and after to show the differences)
    • aws_iam_harden_password_policy.sh - strengthens AWS password policy according to CIS Foundations Benchmark recommendations
    • aws_iam_replace_access_key.sh - replaces the non-current IAM access key (Inactive, Not Used, longer time since used, or an explicitly given key), outputting the new key as shell export statements (useful for piping to the same tools listed for aws_csv_creds.sh above)
    • aws_iam_policies_attached_to_users.sh - finds AWS IAM policies directly attached to users (anti-best practice) instead of groups
    • aws_iam_policies_granting_full_access.sh - finds AWS IAM policies granting full access (anti-best practice)
    • aws_iam_policies_unattached.sh - lists unattached AWS IAM policies
    • aws_iam_policy_attachments.sh - finds all users, groups and roles where a given IAM policy is attached, so that you can remove all these references in your Terraform code and avoid this error Error: error deleting IAM policy arn:aws:iam::***:policy/mypolicy: DeleteConflict: Cannot delete a policy attached to entities.
    • aws_iam_policy_delete.sh - deletes an IAM policy, by first handling all prerequisite steps of deleting all pr

README 内容较长，此处已截断，完整内容请查看 GitHub 仓库。

文档抓取自 GitHub 仓库 README，版权归原作者所有；已过滤徽章等噪音并经安全消毒后展示。