After much todo, trying to get powershell to run julia like it would in bash, I got this working:
& julia -i -e ('println(\"Hello World\")')
& julia -i -e ('include(\"path-to-script.jl\"); function_in_script(); exit()')
& julia -i -e "path-to-script.jl"