diff --git a/mongo-2-oracle/README.md b/mongo-2-oracle/README.md new file mode 100644 index 0000000..e83f7f3 --- /dev/null +++ b/mongo-2-oracle/README.md @@ -0,0 +1,161 @@ +# MongoDB to Oracle Database 23ai Migration + +**Summary:** Contained herein is a bash script for demonstrating and managing MongoDB to Oracle Database migration in a containerized environment. The `orclMongoMigration.sh` script provides a full-featured environment for demonstrating how to migrate data from MongoDB to Oracle Database. It handles container management, database setup, MongoDB configuration, and the complete migration process in an interactive, user-friendly way. + +## Key Features + +**Container Management:** +* Start/stop Oracle Database containers +* Automatic configuration and environment setup +* Interactive menu system with color-coded output +* Comprehensive command-line interface for automation +* File transfer between host and container +* Volume and resource management + +**Database Setup and Access:** +* Automatic Oracle Database configuration +* Multiple SQL*Plus access modes (nolog, user, SYSDBA) +* ORDS (Oracle REST Data Services) setup and management +* MongoDB initialization and configuration +* MongoDB API compatibility for Oracle + +**MongoDB and Migration:** +* MongoDB instance setup and configuration +* MongoDB shell access via mongosh +* Oracle Database MongoDB API connectivity +* Complete data migration workflow from MongoDB to Oracle +* Sample data generation and import +* Interactive migration process with step-by-step guidance +Demo Application +* Registration demo app deployment +* Sample data generation with geographic features +* Data visualization and testing + +## Prerequisites +* Podman installed (script can install it if not present) +* macOS or Linux operating system +* Internet connection +* 8GB+ RAM recommended for running the Oracle Database container + +## Installation + +1. Download the script + + ```bash + wget https://github.com/oracle-devrel/devo.tutorials/blob/mongo-2-oracle-migration/mongo-2-oracle/src/orclMongoMigration.sh + ``` + +2. Change permissions to make the script executable + + ```bash + chmod +x orclMongoMigration.sh + ``` + +3. Set the global variables within the script file. + + >NOTE: See [the step by step guide](step-by-step-guide.md) for details on setting global variables. + +4. Run the script without arguments to display a user-friendly menu with the following sections: + + ```bash + ./orclMongoMigration.sh + ``` + + * **Container Management:** - Start/stop Oracle container - Bash and root access - Install utilities and manage volumes - Copy files in and out + * **Database Access & Utilities:** - SQL*Plus connections (nolog, user, SYSDBA) - ORDS setup and management - MongoDB API connectivity + * **MongoDB Operations:** - Start MongoDB instance - mongosh to ORDS or MongoDB + * **Application & Migration:** - Run registration demo app - Add demo data - Migrate data from MongoDB to Oracle + +5. For a comprehensive walkthru, please visit the [step-by-step guide](step-by-step-guide.md). + +## Available Commandline Arguments + +* Container Management: + + ```bash + start - Start Oracle container + stop - Stop Oracle container + restart - Restart Oracle container + bash - Bash access to container + root - Root access to container + remove - Remove Oracle container + utils - Install utilities + copyin - Copy file into container + copyout - Copy file out of container + clean - Clean unused volumes + ``` + +* Database Access: + + ```bash + sqlnolog - SQL*Plus nolog connection + sqluser - SQL*Plus user connection + sqlsys - SQL*Plus SYSDBA connection + setupords - Setup ORDS + ords - Start ORDS service + mongoapi - Check MongoDB API connection + ``` + +* MongoDB Commands: + + ```bash + mongodb - Start MongoDB instance + mongoords - Connect to MongoDB via ORDS + mongo - Connect to MongoDB directly + ``` + +* Application Commands: + + ```bash + demoapp - Run Registration Demo App + demodata - Add demo data + migrate - Migrate data + help - Show help message + ``` + +## Migration Workflow Example +Here’s a typical workflow for demonstrating MongoDB to Oracle migration: +1. Start the container: + + ```bash + ./orclMongoMigration.sh start + ``` +2. Start MongoDB instance: + ```bash + ./orclMongoMigration.sh mongodb + ``` +3. Generate demo data: + ```bash + ./orclMongoMigration.sh demodata + ``` +4. Perform the migration: + ```bash + ./orclMongoMigration.sh migrate + ``` +5. Verify the migrated data in Oracle: (The script will offer to connect you to SQL*Plus after migration) + +## Container Details +* **Image:** Oracle Database Free (container-registry.oracle.com/database/free) +* **Default Credentials:** +* SYS/SYSTEM password: Oradoc_db1 +* Created user: matt with password matt +* **Exposed Ports:** +* 1521: Oracle Database listener +* 3000: Node.js demonstration app +* 5500: Enterprise Manager Express +* 8080/8443: ORDS (Oracle REST Data Services) +* 27017: MongoDB API compatibility +* 23456: MongoDB native port + +## Troubleshooting +* If the Oracle container fails to start, try running ./orclMongoMigration.sh clean to remove unused volumes +* For migration issues, check that both MongoDB and ORDS are running +* If demo data generation fails, ensure Python libraries are installed correctly +* Check container logs for Oracle Database startup issues + + +## Acknowledgments +* Created by Matt DeMarco (matthew.demarco@oracle.com) +* Oracle Database Free images provided by Oracle + + diff --git a/mongo-2-oracle/src/orclMongoMigration.sh b/mongo-2-oracle/src/orclMongoMigration.sh new file mode 100644 index 0000000..99e1448 --- /dev/null +++ b/mongo-2-oracle/src/orclMongoMigration.sh @@ -0,0 +1,1240 @@ +#!/bin/bash +# +# +#=================================================================================== +# +# FILE: orclMongoMigration.sh +# +# USAGE: ./orclMongoMigration.sh [start|stop|restart|bash|root|sql|ords|mongoapi|help] +# +# DESCRIPTION: Migration demo container moving MongoDB to Oracle +# OPTIONS: See menu or command line arguments +# REQUIREMENTS: Podman, internet connection +# AUTHOR: Matt DeMarco (matthew.demarco@oracle.com) +# CREATED: 04.01.2025 +# VERSION: 1.1 +# +#=================================================================================== + +# Copyright (c) 2025 Oracle and/or its affiliates. + +# The Universal Permissive License (UPL), Version 1.0 + +# Subject to the condition set forth below, permission is hereby granted to any +# person obtaining a copy of this software, associated documentation and/or data +# (collectively the "Software"), free of charge and under any and all copyright +# rights in the Software, and any and all patent rights owned or freely +# licensable by each licensor hereunder covering either (i) the unmodified +# Software as contributed to or provided by such licensor, or (ii) the Larger +# Works (as defined below), to deal in both + +# (a) the Software, and +# (b) any piece of software and/or hardware listed in the lrgrwrks.txt file if +# one is included with the Software (each a "Larger Work" to which the Software +# is contributed by such licensors), + +# without restriction, including without limitation the rights to copy, create +# derivative works of, display, perform, and distribute the Software and make, +# use, sell, offer for sale, import, export, have made, and have sold the +# Software and the Larger Work(s), and to sublicense the foregoing rights on +# either these or other terms. + +# This license is subject to the following condition: +# The above copyright notice and either this complete permission notice or at +# a minimum a reference to the UPL must be included in all copies or +# substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +# Global variables +ORACLE_USER="<--->" +ORACLE_PASS="<--->" +ORACLE_SYS_PASS="<--->" +ORACLE_CONTAINER="<--->" +ORACLE_PDB="FREEPDB1" +NETWORK_NAME="demonet" #set network +CONTAINER_PORT_MAP="-p 1521:1521 -p 3000:3000 -p 5902:5902 -p 5500:5500 -p 8000:8000 -p 8080:8080 -p 8443:8443 -p 27017:27017 -p 23456:23456" +MAX_INVALID=3 +INVALID_COUNT=0 + +# Colors for better readability +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +CYAN='\033[0;36m' +NC='\033[0m' # No Color + + + +#=========================== +# Helper Functions +#=========================== + +logInfo() { + echo -e "${GREEN}[INFO]${NC} $1" +} + +logWarning() { + echo -e "${YELLOW}[WARNING]${NC} $1" +} + +logError() { + echo -e "${RED}[ERROR]${NC} $1" +} + +logSuccess() { + echo -e "${GREEN}[SUCCESS]${NC} $1" +} + +# Display formatted menu +displayMenu() { + clear + echo -e "${CYAN}===========================================================================${NC}" + echo -e "${CYAN} Oracle Database & MongoDB Migration Demo Toolkit ${NC}" + echo -e "${CYAN}===========================================================================${NC}" + + echo -e "\n${GREEN}Container Management:${NC}" + echo " 1) Start Oracle container 6) Install utilities" + echo " 2) Stop Oracle container 7) Copy file into container" + echo " 3) Bash access 8) Copy file out of container" + echo " 4) Root access 9) Clean unused volumes" + echo " 5) Remove Oracle container 10) Exit script" + + echo -e "\n${GREEN}Database Access & Utilities:${NC}" + echo " 11) SQL*Plus nolog connection 14) Setup ORDS" + echo " 12) SQL*Plus user connection 15) Start ORDS service" + echo " 13) SQL*Plus SYSDBA connection 16) Check MongoDB API connection" + + echo -e "\n${GREEN}MongoDB Operations:${NC}" + echo " 17) Start MongoDB instance 19) mongosh to MongoDB" + echo " 18) mongosh to MongoDB API" + + echo -e "\n${GREEN}Application & Migration:${NC}" + echo " 20) Run Registration Demo App 22) Migrate data" + echo " 21) Add demo data" + + echo -e "\n${CYAN}===========================================================================${NC}" + read -p "Please enter your choice [1-22]: " menuChoice + export menuChoice=$menuChoice +} + +# Check if podman is installed and running +checkPodman() { + if ! command -v podman > /dev/null 2>&1; then + logError "Podman is not installed on your system." + + read -p "Would you like to install Podman and its dependencies? (y/n): " choice + if [[ "$choice" =~ ^[Yy]$ ]]; then + logInfo "Installing Podman and dependencies..." + + # Install Homebrew if it's not installed + if ! command -v brew > /dev/null 2>&1; then + logInfo "Homebrew is not installed. Installing Homebrew first..." + /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + export PATH="/usr/local/bin:$PATH" # For Intel Macs + export PATH="/opt/homebrew/bin:$PATH" # For Apple Silicon + fi + + # Install Podman, QEMU, and vfkit + brew tap cfergeau/crc + brew install vfkit qemu podman podman-desktop + + # Initialize Podman machine + logInfo "Initializing Podman machine..." + podman machine init --cpus 8 --memory 16384 --disk-size 550 + + logInfo "Starting Podman machine..." + podman machine start + + logSuccess "Podman installation complete!" + else + logError "Podman is required to run this script. Exiting..." + exit 1 + fi + fi + + # Verify Podman is running + if ! podman ps > /dev/null 2>&1; then + logInfo "Podman is installed but not running. Starting Podman machine..." + podman machine start + fi +} + +# Create podman network if it doesn't exist +createPodnet() { + if ! podman network inspect $NETWORK_NAME &>/dev/null; then + logInfo "Creating podman network '$NETWORK_NAME'..." + podman network create -d bridge $NETWORK_NAME + fi +} + +# Get running container ID/name +getContainerId() { + export orclRunning=$(podman ps --no-trunc --format "table {{.ID}}\t {{.Names}}\t" | grep -i $ORACLE_CONTAINER | awk '{print $2}') + echo $orclRunning +} + +# Countdown timer +countDown() { + message=${1:-"Please wait..."} + seconds=${2:-3} + + logInfo "$message" + for (( i=$seconds; i>=1; i-- )); do + echo -ne "\rStarting in $i seconds..." + sleep 1 + done + echo -e "\rStarting now! " +} + +# Handle invalid menu choice +badChoice() { + # Increment the invalid choice counter + ((INVALID_COUNT++)) + + logWarning "Invalid choice, please try again..." + logWarning "Attempt $INVALID_COUNT of $MAX_INVALID." + + # Check if invalid attempts exceed the max allowed + if [ "$INVALID_COUNT" -ge "$MAX_INVALID" ]; then + logError "Too many invalid attempts. Exiting the script..." + exit 1 + fi + + sleep 2 +} + +#=========================== +# Core Functions +#=========================== + +# Exit function +doNothing() { + logWarning "You want to quit...yes?" + read -p "Enter yes or no: " doWhat + if [[ $doWhat = yes ]]; then + logInfo "Bye! ¯\\_(ツ)_/¯" + exit 0 + else + return + fi +} + +# List container ports +listPorts() { + container_id=$(getContainerId) + if [ -n "$container_id" ]; then + logInfo "Container ports:" + podman port "$container_id" + + logInfo "Container IP address:" + podman inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$container_id" + else + logError "No running container found." + fi +} + +# Start Oracle container +startOracle() { + checkPodman + createPodnet + + # Check if container is already running + export orclRunning=$(getContainerId) + export orclPresent=$(podman container ls -a --no-trunc --format "table {{.ID}}\t {{.Names}}\t" | grep -i $ORACLE_CONTAINER | awk '{print $2}') + + if [ "$orclRunning" == "$ORACLE_CONTAINER" ]; then + logWarning "Oracle podman container is already running." + listPorts + return + elif [ "$orclPresent" == "$ORACLE_CONTAINER" ]; then + logInfo "Oracle podman container found, restarting..." + podman restart $orclPresent + countDown "Waiting for Oracle to start" 3 + serveORDS + else + echo "Please choose the Oracle Database container version:" + echo "1. Lite Version (Good for general database development)" + echo "2. Full Version (Required for the MongoDB API)" + read -p "Enter your choice [1/2]: " choice + + case $choice in + 1) + image="container-registry.oracle.com/database/free:23.5.0.0-lite" + ;; + 2) + image="container-registry.oracle.com/database/free:latest" + ;; + *) + logWarning "Invalid choice. Defaulting to Full version." + image="container-registry.oracle.com/database/free:latest" + ;; + esac + + logInfo "Provisioning new Oracle container with image: $image" + podman run -d --network="podmannet" $CONTAINER_PORT_MAP -it --name $ORACLE_CONTAINER $image + + if [ $? -ne 0 ]; then + logError "Failed to start Oracle container." + return 1 + fi + + logSuccess "Oracle container started successfully." + countDown "Waiting for Oracle to initialize" 3 + installUtils + fi + listPorts +} + +# Stop Oracle container +stopOracle() { + checkPodman + export stopOrcl=$(podman ps --no-trunc | grep -i oracle | awk '{print $1}') + + if [ -z "$stopOrcl" ]; then + logWarning "No Oracle containers are running." + return + fi + + for i in $stopOrcl; do + logInfo "Stopping container: $i" + podman stop $i + if [ $? -eq 0 ]; then + logSuccess "Container stopped successfully." + else + logError "Failed to stop container." + fi + done + + cleanVolumes +} + +# Clean unused volumes +cleanVolumes() { + logInfo "Cleaning unused volumes..." + podman volume prune -f + logSuccess "Volumes cleaned." +} + +# Remove container +removeContainer() { + stopOracle + logInfo "Removing Oracle container..." + podman rm $(podman ps -a | grep $ORACLE_CONTAINER | awk '{print $1}') + if [ $? -eq 0 ]; then + logSuccess "Container removed successfully." + else + logError "Failed to remove container." + fi +} + +# Get bash access to container +bashAccess() { + checkPodman + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + logInfo "Opening bash shell in container..." + podman exec -it $orclImage /bin/bash +} + +# Get root access to container +rootAccess() { + checkPodman + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + logInfo "Opening root shell in container..." + podman exec -it -u 0 $orclImage /bin/bash +} + +# Get SQLPlus nolog access +sqlPlusNolog() { + checkPodman + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + logInfo "Opening SQLPlus session (no login)..." + podman exec -it $orclImage bash -c "source /home/oracle/.bashrc; sqlplus /nolog" +} + +# Get SYSDBA access +sysDba() { + checkPodman + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + logInfo "Opening SQLPlus session as SYSDBA..." + podman exec -it $orclImage bash -c "source /home/oracle/.bashrc; /home/oracle/sqlcl/bin/sql sys/$ORACLE_SYS_PASS@'(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=$ORACLE_PDB)))' as sysdba" +} + +# Create user account +createUser() { + checkPodman + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + logInfo "Creating user account..." + podman exec -it $orclImage bash -c "source /home/oracle/.bashrc; /home/oracle/sqlcl/bin/sql sys/$ORACLE_SYS_PASS@'(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=$ORACLE_PDB)))' as sysdba <>/etc/yum.repos.d/mongodb-org-8.0.repo" + + podman exec -i -u 0 $orclImage /usr/bin/yum install -y mongodb-mongosh + logSuccess "MongoDB tools installed successfully." +} + +# Start MongoDB inside the Oracle container +startMongoDB() { + export container_id=$(getContainerId) + + if [ -z "$container_id" ]; then + logError "Oracle container is not running." + return 1 + fi + + logInfo "Checking if MongoDB is already running..." + podman exec -i $container_id pgrep mongod >/dev/null 2>&1 + if [ $? -eq 0 ]; then + logInfo "MongoDB is already running in the container." + return 0 + fi + + logInfo "MongoDB is not running. Installing MongoDB..." + + podman exec -i -u 0 $container_id /usr/bin/yum install -y mongodb-org + + logInfo "Configuring MongoDB on port 23456..." + podman exec -i -u 0 $container_id /usr/bin/bash -c " + sed -i 's|bindIp: 127.0.0.1|bindIp: 0.0.0.0|' /etc/mongod.conf + sed -i 's|port: 27017|port: 23456|' /etc/mongod.conf + " + + logInfo "Starting MongoDB instance..." + + # No Replica set config + podman exec -d -u 0 $orclImage /usr/bin/mongod --bind_ip_all --config /etc/mongod.conf + + # Replica set configuration + #podman exec -d -u 0 $container_id /usr/bin/mongod --bind_ip_all --replSet myReplSet --config /etc/mongod.conf + + # get IP address for setting replica set use + # rsIPAddr=$(podman inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$container_id") + # echo $rsIPAddr + + # 1 member + # rsConfig="{_id: 'myReplSet',members: [{ _id: 0, host: '${rsIPAddr}:23456' }]}" + + # 2 members + # rsConfig="{_id: 'myReplSet',members: [{ _id: 0, host: '${rsIPAddr}:23456' },{ _id: 1, host: 'localhost:23456' }]}" + # echo $rsConfig + + sleep 1 + # start replSet for GG CDC + # podman exec -u 0 $container_id mongosh --port 23456 --eval "rs.initiate(${rsConfig});rs.status();" + + # podman exec -u 0 $container_id mongosh --port 23456 --eval "rs.reconfig({_id: 'myReplSet',members: [{ _id: 0, host: '${rsIPAddr}:23456' }]}, { force: true });rs.status();" + + + # Verify it started + sleep 2 + podman exec -i $container_id pgrep mongod >/dev/null 2>&1 + if [ $? -eq 0 ]; then + logSuccess "MongoDB started successfully." + else + logError "MongoDB failed to start." + return 1 + fi +} + +# Install utilities +installUtils() { + logInfo "Installing useful tools after provisioning container..." + logWarning "Please be patient as this can take time given network latency." + + checkPodman + export container_id=$(getContainerId) + + if [ -z "$container_id" ]; then + logError "Oracle container is not running." + return 1 + fi + + # workaround for ol repo issues + logInfo "Configuring YUM repositories..." + podman exec -it -u 0 $container_id /bin/bash -c "/usr/bin/touch /etc/yum/vars/ociregion" + podman exec -it -u 0 $container_id /bin/bash -c "/usr/bin/echo > /etc/yum/vars/ociregion" + + # Add sudo access for oracle user + podman exec -it -u 0 $container_id /bin/bash -c "/usr/bin/echo 'oracle ALL=(ALL) NOPASSWD: ALL' >>/etc/sudoers" + + # Install EPEL repository + logInfo "Installing EPEL repository..." + podman exec -it -u 0 $container_id /usr/bin/rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm + podman exec -it -u 0 $container_id /usr/bin/yum update -y + + # Install required packages + logInfo "Installing required packages..." + podman exec -it -u 0 $container_id /usr/bin/yum install -y sudo which java-17-openjdk wget htop lsof zip unzip rlwrap git python3.12 python3-pip + + # Install pip for Python 3.12 + logInfo "Installing pip for Python 3.12..." + podman exec -it -u 0 $container_id /bin/bash -c "curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py; python3.12 /tmp/get-pip.py" + + # Install Node.js + logInfo "Installing Node.js..." + podman exec -it -u 0 $container_id bash -c "curl -fsSL https://rpm.nodesource.com/setup_20.x | bash - && yum install -y nodejs" + + # Download and install ORDS + logInfo "Downloading and installing ORDS..." + podman exec $container_id /usr/bin/wget -O /home/oracle/ords.zip https://download.oracle.com/otn_software/java/ords/ords-latest.zip + podman exec $container_id /usr/bin/unzip /home/oracle/ords.zip -d /home/oracle/ords/ + + + # Download and install SQLcl + logInfo "Downloading and installing SQLcl..." + podman exec $container_id /usr/bin/wget -O /home/oracle/sqlcl.zip https://download.oracle.com/otn_software/java/sqldeveloper/sqlcl-latest.zip + podman exec $container_id /usr/bin/unzip /home/oracle/sqlcl.zip -d /home/oracle/ + # add alias for sqlcl to be invoked by sqlplus + podman exec -i -u 0 $container_id /bin/bash -c "echo 'alias sqlplus=/home/oracle/sqlcl/bin/sql' >> /home/oracle/.bashrc" + + + # Install MongoDB tools + installMongoTools + + # Install personal tools + logInfo "Installing personal tools..." + podman exec $container_id /usr/bin/wget -O /tmp/PS1.sh https://raw.githubusercontent.com/mattdee/orclDocker/main/PS1.sh + podman exec $container_id /bin/bash /tmp/PS1.sh + podman exec $container_id /usr/bin/wget -O /opt/oracle/product/23ai/dbhomeFree/sqlplus/admin/glogin.sql https://raw.githubusercontent.com/mattdee/orclDocker/main/glogin.sql + + # Set Oracle password + setOrclPwd + + logSuccess "Utilities installation complete!" +} + +# Copy file into container +copyIn() { + checkPodman + export orclRunning=$(getContainerId) + + if [ -z "$orclRunning" ]; then + logError "Oracle container is not running." + return 1 + fi + + read -p "Please enter the ABSOLUTE PATH to the file you want copied: " thePath + read -p "Please enter the FILE NAME you want copied: " theFile + + logInfo "Copying file: $thePath/$theFile into container..." + podman cp $thePath/$theFile $orclRunning:/tmp + + if [ $? -eq 0 ]; then + logSuccess "File copied successfully to /tmp/$theFile in the container." + else + logError "Failed to copy file into container." + fi +} + +# Copy file out of container +copyOut() { + checkPodman + export orclRunning=$(getContainerId) + + if [ -z "$orclRunning" ]; then + logError "Oracle container is not running." + return 1 + fi + + read -p "Please enter the ABSOLUTE PATH in the CONTAINER to the file you want copied to host: " thePath + read -p "Please enter the FILE NAME in the CONTAINER you want copied: " theFile + + logInfo "Copying file: $orclRunning:$thePath/$theFile to host..." + podman cp $orclRunning:$thePath/$theFile /tmp/ + + if [ $? -eq 0 ]; then + logSuccess "File copied successfully to /tmp/$theFile on your host." + else + logError "Failed to copy file from container." + fi +} + +# Setup ORDS +setupORDS() { + checkPodman + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + # Create temp password file + logInfo "Creating temporary password file..." + podman exec -i -u 0 $orclImage /bin/bash -c "echo '$ORACLE_SYS_PASS' > /tmp/orclpwd" + + logInfo "Configuring ORDS..." + + # Create user for ORDS + createUser + + # ORDS silent setup + logInfo "Installing ORDS..." + podman exec -i $orclImage /bin/bash -c "/home/oracle/ords/bin/ords --config /home/oracle/ords_config install --admin-user SYS --db-hostname localhost --db-port 1521 --db-servicename $ORACLE_PDB --log-folder /tmp/ --feature-sdw true --feature-db-api true --feature-rest-enabled-sql true --password-stdin /dev/null 2>&1; sleep 10" + sleep 5 + + # Verify ORDS is running + podman exec $orclImage /bin/bash -c "/usr/bin/ps -ef | grep -i ords" + + logSuccess "ORDS started successfully!" +} + +# Stop ORDS +stopORDS() { + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + logInfo "Stopping ORDS..." + podman exec $orclImage /bin/bash -c "for i in $(ps -ef | grep ords | awk '{print $2}'); do echo $i; kill -9 $i; done" + logSuccess "ORDS stopped successfully!" +} + +# Check MongoDB API +checkMongoAPI() { + # Test MongoDB connections in the container + logInfo "Checking MongoDB API health..." + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + # run disableTelemetry() + logInfo "Creating test collection..." + podman exec -it $orclImage bash -c "mongosh --tlsAllowInvalidCertificates 'mongodb://$ORACLE_USER:$ORACLE_PASS@127.0.0.1:27017/$ORACLE_USER?authMechanism=PLAIN&ssl=true&retryWrites=false&loadBalanced=true'< matthew.demarco@oracle.com " + echo -e "${BLUE}===========================================================================${NC}" + exit 0 + ;; + "") + logInfo "No args provided. Starting menu interface..." + ;; + *) + logError "Invalid argument: $1" + logInfo "Run '$0 help' for usage information." + exit 1 + ;; +esac + +# Main menu loop +while true; do + displayMenu + + case $menuChoice in + 1) + startOracle + ;; + 2) + stopOracle + ;; + 3) + bashAccess + ;; + 4) + rootAccess + ;; + 5) + removeContainer + ;; + 6) + installUtils + ;; + 7) + copyIn + ;; + 8) + copyOut + ;; + 9) + cleanVolumes + ;; + 10) + doNothing + ;; + 11) + sqlPlusNolog + ;; + 12) + sqlPlusUser + ;; + 13) + sysDba + ;; + 14) + setupORDS + ;; + 15) + serveORDS + ;; + 16) + checkMongoAPI + ;; + 17) + startMongoDB + ;; + 18) + mongoshORDS + ;; + 19) + mongoshMongoDB + ;; + 20) + runDemoApp + ;; + 21) + addData + ;; + 22) + migrateData + ;; + 23) + listPorts + ;; + *) + badChoice + ;; + esac + + # Reset count after a valid choice + if [[ $menuChoice =~ ^[1-9]|1[0-9]|2[0-2]$ ]]; then + INVALID_COUNT=0 + fi + + # Pause after each operation to view results + if [[ $menuChoice != 10 && $menuChoice =~ ^[1-9]|1[0-9]|2[0-2]$ ]]; then + echo + read -p "Press Enter to continue..." dummy + fi +done \ No newline at end of file diff --git a/mongo-2-oracle/step-by-step-guide.md b/mongo-2-oracle/step-by-step-guide.md new file mode 100644 index 0000000..03dde7a --- /dev/null +++ b/mongo-2-oracle/step-by-step-guide.md @@ -0,0 +1,284 @@ +# MongoDB-2-Oracle Migration: Step-by-step runbook + +This detailed runbook will guide you through the process of using the `orclMongoMigration.sh` script to demonstrate migrating data from MongoDB to Oracle Database. + + +## Prerequisites +Before you begin, ensure your system meets these requirements: +* **Operating System**: macOS or Linux +* **Hardware**: + * At least 8GB RAM (16GB recommended) + * At least 20GB free disk space +* **Internet Connection**: Required for downloading container images and dependencies +* **Terminal Access**: You’ll be working in the command line + +## Getting Started + +### Step 1: Make the Script Executable +1. Open your terminal application +2. Navigate to the directory containing the script: + + ```bash + cd + ``` +3. Make the script executable: + + ```bash + chmod +x orclMongoMigration.sh + ``` +### Step 2: Understanding the Script’s Interface and global variables +There are two ways to interact with the script: +1. **Interactive Menu**: Run the script without arguments +2. **Command Line Arguments**: Run specific commands directly + + >NOTE: For this runbook, we’ll primarily use the interactive menu for clarity, with command examples where helpful. + +3. Before running the script, it is important to update the global variables to match your environment. Edit the following items: + + ```bash + # Global variables + ORACLE_USER="" + ORACLE_PASS="" + ORACLE_SYS_PASS="" + ORACLE_CONTAINER="" + ORACLE_PDB="FREEPDB1" + NETWORK_NAME="demonet" #set network + // Do not change the settings below this line + CONTAINER_PORT_MAP="-p 1521:1521 -p 3000:3000 -p 5902:5902 -p 5500:5500 -p 8000:8000 -p 8080:8080 -p 8443:8443 -p 27017:27017 -p 23456:23456" + MAX_INVALID=3 + INVALID_COUNT=0 + ``` + +4. Save the script and mvoe to the next step. + +## Container Management + +### Step 3: Starting the Oracle Container +1. Run the script without arguments to access the menu: + + ```bash + ./orclMongoMigration.sh + ``` + +2. You’ll see a color-coded menu with several sections +3. Press 1 to start the Oracle container +4. If Podman is not installed, the script will offer to install it for you: + * Type y and press Enter if prompted to install Podman + * The script will install Homebrew (if needed) and then Podman + * This process may take several minutes +5. Choose the Oracle Database version: + * Type 2 and press Enter to select the Full Version (required for MongoDB API) +6. The script will download and start the Oracle container: + * This is a large download (several GB) and may take time depending on your internet connection + * Be patient during the initial download + * After downloading, the container will initialize (this takes several minutes) +7. The script will automatically install utilities in the container: + * Various Linux utilities + * Python 3.12 + * Node.js + * MongoDB tools + * ORDS (Oracle REST Data Services) +8. When complete, you’ll see a list of mapped ports and a success message +9. Press Enter to return to the main menu + +### Step 4: Checking Container Status +1. You can verify the container is running with option 3 (Bash access) +2. This will open a shell inside the container +3. Type exit to return to the script menu + +## Database Setup + +### Step 5: Setting Up ORDS (Oracle REST Data Services) +1. From the main menu, press 14 to set up ORDS +2. The script will: + * Create necessary users + * Configure database settings + * Enable the MongoDB API + * Start the ORDS service +3. Wait for the “ORDS setup complete!” message +4. Press Enter to return to the main menu + +### Step 6: Starting MongoDB +1. From the main menu, press 17 to start MongoDB instance +2. The script will: + * Install MongoDB if needed + * Configure MongoDB to listen on port 23456 + * Start the MongoDB service +3. Wait for the “MongoDB started successfully” message +4. Press Enter to return to the main menu + +## Working with MongoDB + +### Step 7: Connecting to MongoDB +1. From the main menu, press 19 to connect to MongoDB directly +2. This will open the MongoDB shell (mongosh) +3. Try some basic MongoDB commands: + + ```bash + // Show databases + show dbs + + // Create a new database + use testdb + + // Create a collection and insert a document + db.testcollection.insertOne({ name: "Test User", email: "test@example.com" }) + + // Query the collection + db.testcollection.find() + ``` + +4. Type exit to exit the MongoDB shell and return to the script menu + +## Running the Demo Application + +### Step 8: Getting the Demo Application +1. From the main menu, press 20 to run the Registration Demo App +2. The script will: + * Clone the application repository if needed + * Install Node.js dependencies + * Prompt the user as to what datastore to use for the application + * 1 will use MongoDB + * 2 will use the Oracle Database API for MongoDB + * 3 will use Autonomous JSON database (still in development) + * Start the application on port 3000 +3. Access the application in your web browser at: + http://localhost:3000 +4. The demo application allows you to register users +5. Press Ctrl+C in your terminal to stop the application and return to the script + +### Step 9: Adding Demo Data +1. From the main menu, press 21 to add demo data +2. The script will: + * Install required Python libraries + * Set up the geodata directory + * Download necessary shapefiles + * Start an interactive session to run the data generation script +3. When prompted: + * Enter 10 for the number of fake records + * Enter 3 for “No images” option (simpler) +4. The script will generate fake data with geospatial coordinates within the shapefiles +5. Press Enter to return to the main menu + +## Data Migration Process + +### Step 10: Migrating Data from MongoDB to Oracle +1. From the main menu, press 22 to start the migration process +2. Read the migration information displayed +3. Press Enter to start the interactive migration tool +4. In the migration tool menu: + * Choose option 3 for “Both Export & Import” +5. For MongoDB export (source) settings: + * Enter localhost:23456/test as your MongoDB endpoint + * Leave the authentication fields blank (press Enter) + * Type no for SSL/TLS connection + * Press Enter to export all collections (or specify a collection) + * Press Enter to use default parallel collections + * Enter /tmp/moveit as the export location +6. For Oracle MongoDB API (target) settings: + * Enter matt as the username + * Enter matt as the password + * Enter localhost as the hostname + * Enter matt as the schema name + * Enter /tmp/moveit as the import location +7. The script will: + * Export data from MongoDB to files + * Import data into Oracle Database via the MongoDB API + * Show progress in real-time +8. When the migration completes, you’ll be asked if you want to verify the migration + * Type y and press Enter to open SQL*Plus for verification +9. In SQL*Plus, run verification commands: + + ```bash + -- List tables + SELECT table_name FROM user_tables; + + -- Count records in the migrated collection + SELECT COUNT(*) FROM testcollection; + + -- View sample data + SELECT * FROM testcollection WHERE ROWNUM <= 5; + + -- Exit SQL*Plus + EXIT + ``` + +10. Press Enter to return to the main + +## Verification and Troubleshooting + +### Step 11: Checking MongoDB API Connection +1. From the main menu, press 16 to check MongoDB API connection +2. The script will: + * Create a test collection + * Insert a test document + * Read back the test document +3. If successful, you’ll see the test document data displayed +4. Press Enter to return to the main menu + +## Step 12: Connecting to Oracle via MongoDB API +1. From the main menu, press 18 to connect to Oracle via MongoDB API (mongosh ORDS) +2. This opens a MongoDB shell connected to Oracle’s MongoDB API +3. Try some commands: + + ```bash + // Show collections + show collections + + // Query the migrated collection + db.testcollection.find() + + // Create a new collection in Oracle via MongoDB API + db.newcollection.insertOne({ name: "Oracle API Test", value: 42 }) + + // Verify it was created + db.newcollection.find() + ``` + +4. Type exit to exit the shell and return to the script menu + +## Shutting Down + +### Step 13: Stopping the Container +1. From the main menu, press 2 to stop the Oracle container +2. The script will stop all running Oracle containers +3. Press Enter to return to the main menu +4. Press 10 to exit the script + +## Command Reference +For advanced users or automation, you can run specific commands directly: + +### Container Management + + ```bash + ./orclMongoMigration.sh start # Start Oracle container + ./orclMongoMigration.sh stop # Stop Oracle container + ./orclMongoMigration.sh bash # Bash access to container + ./orclMongoMigration.sh root # Root access to container + ``` + +### Database Operations + + ``` + ./orclMongoMigration.sh sqluser # SQL*Plus user connection + ./orclMongoMigration.sh sqlsys # SQL*Plus SYSDBA connection + ./orclMongoMigration.sh ords # Start ORDS service + MongoDB Operations + ./orclMongoMigration.sh mongodb # Start MongoDB instance + ./orclMongoMigration.sh mongo # Connect to MongoDB directly + ./orclMongoMigration.sh mongoords # Connect to MongoDB via ORDS + ``` + +### Application and Migration + + ```bash + ./orclMongoMigration.sh demoapp # Run Registration Demo App + ./orclMongoMigration.sh demodata # Add demo data + ./orclMongoMigration.sh migrate # Migrate data + Help and Information + ./orclMongoMigration.sh help # Show all available commands + ``` + +## Summary +________________________________________ +This runbook covers the basic workflow for demonstrating MongoDB to Oracle migration. You can explore additional features and options as you become more familiar with the script.