barbietunnie

How to read Parquet files with DuckDB

• Create a new DuckDB in-memory connection using DBeaver, specifying the path as :memory:

• Create a new script to run your query with, using the connection created

  SELECT * FROM parquet_schema('/path/to/parquet/file.parquet');
  
  SELECT * FROM read_parquet('/path/to/parquet/file.parquet');
  

barbietunnie

How to resolve the error "Unable to install or update any libraries ModuleNotFoundError: No module named 'apt_pkg'" on Ubuntu

The issue can be resolved by uninstalling and installing the python3-apt package

sudo apt remove python3-apt

sudo apt install python3-apt

barbietunnie

Install mysqlclient with pip on Ubuntu 22

Without installing pkg-config before attempting to install mysqlclient on Ubuntu 22 fails and subsequently breaks apt-pkg.

To install it without issues, run the following commands:

sudo apt install pkg-config
pip install mysqlclient

barbietunnie

Redshift Notes

1. Find dependent objects for a table or view in Postgresql or Redshift

SELECT dependent_ns.nspname as dependent_schema
, dependent_view.relname as dependent_view 
, source_ns.nspname as source_schema
, source_table.relname as source_table
, pg_attribute.attname as column_name

barbietunnie

Granting privileges in Postgres

Steps

Below are some common ways to grant access to a PostgreSQL user:

1. Grant CONNECT to the database:

GRANT CONNECT ON DATABASE database_name TO username;

barbietunnie

Grouping results and removing unwanted ones

Here we want to scrape product name, price and rating from ebay product pages:

url = 'https://www.ebay.com/itm/Sony-PlayStation-4-PS4-Pro-1TB-4K-Console-Black/203084236670' 

wanted_list = ['Sony PlayStation 4 PS4 Pro 1TB 4K Console - Black', 'US $349.99', '4.8'] 

scraper.build(url, wanted_list)

barbietunnie

FastAPI and Uvicorn Logging

When running FastAPI app, all the logs in console are from Uvicorn and they do not have timestamp and other useful information. As Uvicorn applies python logging module, we can override Uvicorn logging formatter by applying a new logging configuration.

Meanwhile, it's able to unify the your endpoints logging with the Uvicorn logging by configuring all of them in the config file log_conf.yaml.

Before overriding:

uvicorn main:app --reload

barbietunnie

How to Install Let's Encrypt SSL on Ubuntu with Certbot

Install snapd

The apt version of certbot is usually not up-to-date, so it's preferable to install it with snap.

sudo systemctl status snapd

barbietunnie

Docker Scout CLI notes

1. Analyze local files

• Get an at-a-glance vulnerability summary of the source code in the current working directory

  docker scout quickview fs://.
  

• View the details of vulnerabilities found in your local source code

barbietunnie

Find dependent objects for a table or view in Postgresql or Redshift

SELECT dependent_ns.nspname as dependent_schema
, dependent_view.relname as dependent_view 
, source_ns.nspname as source_schema
, source_table.relname as source_table
, pg_attribute.attname as column_name
FROM pg_depend 
JOIN pg_rewrite ON pg_depend.objid = pg_rewrite.oid