SSH agent forward into docker container on macOS

Preface

I am used to git ssh remote URL, in case of public and private repositories either from GitHub or gitlab (those projects are either contributed or authored by me). Beside in my current company (Zitelab ApS), we have our gitlab enterprise edition server which is hosted into our own cloud and most of the repositories are internally accessible by our team exclusively. The main reason for using ssh remote URL is to avoid entering username and password at each time of push, pull (required over https remote URL)

Why SSH agent for docker container

For me as a new macOS user, for the first time, I faced problem while trying to clone some of the repositories from the inside docker container. My idea was to copy my local ssh key pair into container´s .ssh directory (not secure huh!) and help from the great teacher Google found many solutions for me like Pass local machine’s SSH key to docker container, Using SSH keys inside docker container and so on, however honestly speaking I was not able to implement the ssh key pair copying idea (may not be tried so hard, because I found a better and safe idea later?)

Finally took the idea of ssh agent forwarding (which method I used already in the vagrant based machine). In my opinion, this approach is the best fit for fulfilling my purpose at least.

Configure SSH agent forward in Mac Machine

I refer ´host´ as my development machine it-self and ´guest´ is referred to the docker container. Ignore this the step if you have already.

Setup Host Machine´s ssh config (~/.ssh/config) (Optional)

Example wildcard(*) applicable for all host
Host *
     ForwardAgent yes
     IdentityFile ~/.ssh/id_rsa
Example Certain Host
Host gitlab.com
     ForwardAgent yes
     IdentityFile ~/.ssh/id_rsa_gitlab
Example Certain IP ranges
Host 192.168.*
ForwardAgent yes
IdentityFile ~/.ssh/id_rsa_gitlab

Setup in Docker-compose file

docker-compose.yml

version: '3'
services:
  my_service_name:
    build: .
    environment:
      - SSH_AUTH_SOCK="${SSH_AUTH_SOCK}"
    volumes:
      - ${SSH_AUTH_SOCK}:${SSH_AUTH_SOCK}

Enable SSH Forwarding directly from docker command

docker run --rm -t -i  -v $SSH_AUTH_SOCK:${SSH_AUTH_SOCK} -e SSH_AUTH_SOCK=${SSH_AUTH_SOCK}  <your container tag>

Disclaimer

My machine is MacBook pro 15 2109 and with Mojave OS (at the time of writing), However, I saw on the internet some complaints about not working properly but in my case, it is working perfectly.

References

  1. Can we re-use the OSX ssh-agent socket in a container?

MySQL Connector C API (libmysqlclient) install and configure in MacOS

Preface

I am the mad (maybe fool) guy who doesn’t prefer to install services (like MySQL, PostgreSQL and so on) into development machine, instead using docker containers, this way gives me a lot of flexibilities, for example, I could have multiple versions of MySQL servers (each version could be project-specific depends on requirements), besides no unnecessary processes are running in my development machine.

This writing for those concerns who don’t have MySQL server installed into MacOS powered development machine (like MacBook Pro) but need to work with MySQL connection through programming languages (i.e Python in my case). Bellows are focused on ´MySQL-Python´, ´mysqlclient´ installation problems and solutions. If you have MySQL server installed through Homebrew, we don’t expect you are facing any connection related problem.

Prerequisite

As a requirement of my solution, I assume you already have Homebrew installed on your machine, other than going to the official site about to installl it, should be pretty easy.

OpenSSL

Make sure you have OpenSSL installed and the path is added in bash and/or zsh profile. Try (which openssl) to check if already available. Other than installed and configure it.

brew install openssl

echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.bash_profile 

echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.zshrc (if you are using zsh)

Install and Configure MySQL connector C API

Install through Homebrew

brew install mysql-connector-c

Configure mysql_config

  1. Find the file location by using the command ls -la `which mysql_config`
  2. Open the file with sudo privileges by any editor.
  3. Find #Create options and comment in libs="$libs -l "
  4. Add libs="$libs -lmysqlclient -lssl -lcrypto" and save

Update bash and/or zsh profile

echo 'export LDFLAGS="-L/usr/local/opt/openssl/lib"' >> ~/.bash_profile 

echo 'export CPPFLAGS="-I/usr/local/opt/openssl/include"' >> ~/.bash_profile 


# if you have zsh
echo 'export LDFLAGS="-L/usr/local/opt/openssl/lib"' >> ~/.zshrc 

echo 'export CPPFLAGS="-I/usr/local/opt/openssl/include"' >> ~/.zshrc

Some Common errors could be solved

Collecting mysqlclient   Using cached 
https://files.pythonhosted.org/packages/ec/fd/83329b9d3e14f7344d1cb31f128e6dbba70c5975c9e57896815dbb1988ad/mysqlclient-1.3.13.tar.gz
     Complete output from command python setup.py egg_info:     
Traceback (most recent call last):       File "<string>", line 1, in <module>       
File "/private/var/folders/h3/sff7td1d6pg5v5qsm5xf31q80000gn/T/pip-install-ki9z7ln9/mysqlclient/setup.py", line 18, in <module>         metadata, options = get_config()       

File "/private/var/folders/h3/sff7td1d6pg5v5qsm5xf31q80000gn/T/pip-install-ki9z7ln9/mysqlclient/setup_posix.py", line 60, in get_config         libraries = [dequote(i[2:]) for i in libs if i.startswith('-l')]       
File "/private/var/folders/h3/sff7td1d6pg5v5qsm5xf31q80000gn/T/pip-install-ki9z7ln9/mysqlclient/setup_posix.py", line 60, in <listcomp>         libraries = [dequote(i[2:]) for i in libs if i.startswith('-l')]       
File "/private/var/folders/h3/sff7td1d6pg5v5qsm5xf31q80000gn/T/pip-install-ki9z7ln9/mysqlclient/setup_posix.py", line 13, in dequote         
raise Exception("Wrong MySQL configuration: maybe https://bugs.mysql.com/bug.php?id=86971 ?")     
Exception: Wrong MySQL configuration: maybe https://bugs.mysql.com/bug.php?id=86971 ?

References

  1. MySQLClient instal error: “raise Exception(”Wrong MySQL configuration: maybe https://bugs.mysql.com/bug.php?id”
  2. pip3 install mysqlclient fails on macOS
  3. MySQL Bug #86971 mysql_config –libs of mysql-connector/c 6.1.0 on macOS is wrong

Anaconda vs Sublimelinter with Jedi for SublimeText

SublimeText is undoubtedly one of the most popular editor right now over million of software / web developers. The developer is using SublimeText as IDE usually, rather than using as plain text editor,  by making  IDE like features with the help from some great packages  available in free of cost.  For python development environment, Anaconda is all-in-one package for transforming to be IDE, in the other hand Sublimtelinter with Jedi combination one of the best method for enabling sublimetext to provide IDE like features. Both two have pros and cons, i will discuss bellows about my experiences for both.

Anaconda

Pros

  1. Most importantly, this all-in-one kind package and with no dependency, that means you don’t have headache to install multiple packages and separate configurations.
  2. Easy configuration and well documented.
  3. Numbers of active contributors and continuous development, see details
  4. Very faster linting process after introducing json server.
  5. McCabe code complexity checker

Cons

  1. Some time I discovered old problem come again with latest version, that is hanging on my sublimetext (may be one of the cause could be for big file (more than ten thousands of line)).
  2. There is no support for flake8, see discussion here. Although  anaconda linting is working with pyflake and pep8 combinely means likely to be same as flake8, but more just pyflake and pep8, flake8 has lots of cool plugins, you will miss those.
  3. Working with json server might easy, depends on Operating System you are using.

Sublimelinter and Sublime-Jedi

Pros

  1. This combination approach is like that you gather all good packages and using it. Sublimelinter is the best linting framework for SublimeText, in the other hand Jedi is one the best autocomplete library that is welly ported to Sublime, thanks author of SublimeJedi
  2. Jedi provides nice (CTRL+SHIFT+G) explorer (Go Defination) feature, you could go to the reference import very quickly, beside you could easily add extra python package paths those will be auto included to Jedi’s sys path.
  3. Sublimelinter is framework for linting, there are lots of plugins available, for python I would like to use flake8 but you may keen to use pylint, that is also available.
  4. Sublimelinter is working very fast and also has lots of configuration options, so that you could customize as your wish.

Cors

  1. I don’t find so much negative side to using this combination. But I find there is not much development activity on Github.
  2. Some rare case, jedi might not working (not sure if it is caused by Sublimetext or jedi itself)

 

We saw the simple comparison between two approach. In my opinion for novice user, they could go with Anaconda as this is very easy to adapted and configure. But in contrast of beginner user of sublimetext, I would suggest, professional developer should definitely with go with Sublimelinter specially if he/she is a full stack developer as it (Sublimelinter) has variety of plugins for different language/markup i.e css, javascript, html, json, yml and more.

Make Anaconda powered SublimeText as powerful Python IDE for full stack development

Like so many SublimeText lovers, Me as a python developer, it is also one of the favorite editor, I am using but I would like to use  it as full featured IDE. You will find many articles, blogs about how SublimeText can be transformed as pure IDE, but here i will discussed about the most easy way to turn your sublimetext as IDE for full stack python development.

Anaconda  is the single package that is capable to make your editor (SublimeText) as IDE for python development. It has almost all IDE like features, i.e linting, autocomplete, etc. You could say this is all-in-one package, with no third party dependency like SublimeLinter . One of the positive things that it is in continuous development by involvement of many contributors, so you can report any bug/error and the highest possibility, we will get the solution. They have very good documentation as well.

Install and Configuration is easy enough, I would suggest you follow instruction  from their official site, for advanced configuration you could follow here. It is good practice that you make user specific settings those are common for all your projects. Follows are example user settings.

{
    "auto_formatting": true,
    "auto_python_builder_enabled": false,
    "anaconda_linting": true,
    "anaconda_linter_phantoms": false,
    "validate_imports": true,
    "enable_signatures_tooltip": false,
    "merge_signatures_and_doc": false,
    "pep8_max_line_length": 120
}

If you saw the example configuration above, I intentionally set the value of  `pep8_max_line_length` is 120 (default is 80), in the other hand you will see that i don’t mention the path of python interpreter because it should goes on project specific settings (*.sublime-project) if you use individual interpreter(i.e virtualenv) for each project.

{
	"build_systems":
	[
		{
			"file_regex": "^[ ]*File \"(...*?)\", line ([0-9]*)",
			"name": "PRS:: Anaconda Python Builder",
			"selector": "source.python",
			"shell_cmd": "{virtualenv path}/bin/python -u \"$file\""
		}
	],
	"folders":
	[
		{
			"path": "."
		}
	],
	"settings":
	{
		"python_interpreter": "{virtualenv path}/bin/python"
        }
}  

If you are using buildout for your project development, we have good news for you! There is a recipe plone.recipe.sublimetext for anaconda settings, this recipe will do for you to generate project specific settings. Followings are possible buildout settings for you (but you could do add  more custom option)

[buildout]
parts += 
     sublimetext
[sublimetext]
recipe = plone.recipe.sublimetext
eggs = ${instance:eggs}
anaconda-enabled = True
anaconda-linting-enabled = True
anaconda-completion-enabled = True
anaconda-validate-imports = True

 

LRU cache invalidation in python

I am getting sucked during writing one of unit test, because one specific function doesn’t works. After huge time spending, it is revealed that I used lru_cache that’s why it just execute one time (could be by another TestCase).

But in the test environment I need fresh result instead of serving from cache for various reasons.

After googling finally I found out real application of cache invalidation. Bellows are example

from functools import lru_cache
@lru_cache(maxsize=None)
def my_function(arg1, arg2)
   return arg1 + arg2
# Clear Cache before calling
my_function.cache_clear()
my_function(2, 2)