Installation

Requirements

  • Python: 3.11 or higher

  • MongoDB: 4.0 or higher (for imports)

  • Poetry: For dependency management (recommended)

Installation Methods

From Source

# Clone the repository
git clone https://github.com/jdrumgoole/pyimport.git
cd pyimport

# Install with Poetry
poetry install

# Or install with pip
pip install -e .

Using Poetry (Development)

If you’re developing or contributing to PyImport:

# Clone and enter directory
git clone https://github.com/jdrumgoole/pyimport.git
cd pyimport

# Install dependencies
poetry install

# Run pyimport
poetry run pyimport --help

Verify Installation

Check that PyImport is installed correctly:

pyimport --version
# Output: pyimport 1.10.0

Get help:

pyimport --help

MongoDB Setup

PyImport requires a running MongoDB instance. By default, it connects to mongodb://localhost:27017.

Local MongoDB

macOS (Homebrew):

brew install mongodb-community
brew services start mongodb-community

Ubuntu/Debian:

sudo apt-get install mongodb
sudo systemctl start mongodb

Docker:

docker run -d -p 27017:27017 --name mongodb mongo:latest

MongoDB Atlas

For MongoDB Atlas (cloud), use the --mdburi parameter:

pyimport --mdburi "mongodb+srv://user:pass@cluster.mongodb.net/" \
         --database mydb --collection mycol data.csv

Or set the environment variable:

export MDB_URI="mongodb+srv://user:pass@cluster.mongodb.net/"
pyimport --database mydb --collection mycol data.csv

Optional Dependencies

PostgreSQL Support

PyImport also supports importing to PostgreSQL (experimental):

pip install pyimport[postgres]

Set the PostgreSQL connection using standard PostgreSQL environment variables:

export PGHOST=localhost
export PGPORT=5432
export PGDATABASE=dbname
export PGUSER=user

# Store password in ~/.pgpass for security
echo "localhost:5432:dbname:user:password" >> ~/.pgpass
chmod 600 ~/.pgpass

pyimport --pgtable mytable data.csv

Development Dependencies

For development and testing:

poetry install --with dev

This includes:

  • pytest and pytest-asyncio

  • coverage

  • sphinx (documentation)

  • black, flake8 (code quality)

Configuration File

PyImport can read settings from configuration files in these locations (in order of priority):

  1. ./pyimport.conf (current directory)

  2. ~/.pyimport.conf (home directory)

  3. ~/.config/pyimport.conf (config directory)

Example pyimport.conf:

# MongoDB settings
mdburi = mongodb://localhost:27017
database = mydb
collection = imported

# Import settings
batchsize = 1000
hasheader = True

# Parallel processing
poolsize = 4

Environment Variables

PyImport respects these environment variables:

  • MDB_URI: MongoDB connection string (overrides --mdburi)

  • AUDIT_HOST: MongoDB URI for audit records (overrides --audithost)

  • PGHOST, PGPORT, PGDATABASE, PGUSER: Standard PostgreSQL environment variables

Set them in your shell or .env file:

export MDB_URI="mongodb://localhost:27017"
export AUDIT_HOST="mongodb://localhost:27017"

# PostgreSQL
export PGHOST=localhost
export PGPORT=5432
export PGDATABASE=postgres
export PGUSER=myuser

Security Note: Always store PostgreSQL passwords in ~/.pgpass (not in environment variables).

Quick Start

Once installed, try a basic import:

# Create a test CSV file
echo "name,age,city" > test.csv
echo "Alice,30,NYC" >> test.csv
echo "Bob,25,LA" >> test.csv

# Generate field file (type definitions)
pyimport --genfieldfile test.csv

# Import to MongoDB
pyimport --database testdb --collection people test.csv

# Verify import
mongo testdb --eval "db.people.find().pretty()"

Troubleshooting

Connection Issues

If you get “connection refused” errors:

  1. Verify MongoDB is running:

    # Check if MongoDB is listening
netstat -an | grep 27017

# Or use mongosh/mongo
mongosh --eval "db.version()"

  2. Check your connection string:

    pyimport --mdburi mongodb://localhost:27017 --database test --collection test data.csv

Permission Errors

If you get permission errors during installation:

# Use --user flag
pip install --user pyimport

# Or use a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install pyimport

Import Path Issues

If Python can’t find pyimport:

# Add to PYTHONPATH
export PYTHONPATH=/path/to/pyimport:$PYTHONPATH

# Or reinstall in development mode
pip install -e .

Next Steps