Scala Visual Studio Code

Scala Visual Studio Code
Scala Development Visual Studio Code
Vscode Scala Spark
Visual Studio Scala Plugin
Scala Maven Visual Studio Code

Open Visual Studio Code command windows by pressing CTRL+SHIFT+P and run Remote-Containers: Open Folder in Container command. This will initiate build and run of the development Docker container. Running Scala Spark job in Dev Container. Now everything is setup. Scala Syntax (official) Visual Studio Code extension providing syntax highlighting for Scala 2 and Scala 3 source files. This repo also powers the Scala syntax highlighting on GitHub. (It is vendored in github/linguist.) Team. The current maintainers (people who can merge pull requests) are: Nicolas Stucki- @nicolasstucki; Maxime Kjaer - @MaximeKjaer.

The Visual Studio Code Remote - WSL extension lets you use the Windows Subsystem for Linux (WSL) as your full-time development environment right from VS Code. You can develop in a Linux-based environment, use Linux-specific toolchains and utilities, and run and debug your Linux-based applications all from the comfort of Windows.

The extension runs commands and other extensions directly in WSL so you can edit files located in WSL or the mounted Windows filesystem (for example /mnt/c) without worrying about pathing issues, binary compatibility, or other cross-OS challenges.

This lets VS Code provide a local-quality development experience — including full IntelliSense (completions), code navigation, and debugging — regardless of where your code is hosted.

Getting started

Note: After reviewing this topic, you can get started with the introductory WSL tutorial.

Installation

To get started, you need to:

Install the Windows Subsystem for Linux along with your preferred Linux distribution.
Note: WSL 1 does have some known limitations for certain types of development. Also, extensions installed in Alpine Linux may not work due to glibc dependencies in native source code inside the extension. See the Remote Development and Linux article for details.
Install Visual Studio Code on the Windows side (not in WSL).
Note: When prompted to Select Additional Tasks during installation, be sure to check the Add to PATH option so you can easily open a folder in WSL using the code command.
Install the Remote Development extension pack.

Open a remote folder or workspace

From the WSL terminal

Opening a folder inside the Windows Subsystem for Linux in VS Code is very similar to opening up a Windows folder from the command prompt or PowerShell.

Open a WSL terminal window (using the start menu item or by typing wsl from a command prompt / PowerShell).
Navigate to a folder you'd like to open in VS Code (including, but not limited to, Windows filesystem mounts like /mnt/c)
Type code . in the terminal. When doing this for the first time, you should see VS Code fetching components needed to run in WSL. This should only take a short while, and is only needed once.
Note: If this command does not work, you may need to restart your terminal or you may not have added VS Code to your path when it was installed.
After a moment, a new VS Code window will appear, and you'll see a notification that VS Code is opening the folder in WSL.
VS Code will now continue to configure itself in WSL and keep you up to date as it makes progress.
Once finished, you now see a WSL indicator in the bottom left corner, and you'll be able to use VS Code as you would normally!

That's it! Any VS Code operations you perform in this window will be executed in the WSL environment, everything from editing and file operations, to debugging, using terminals, and more.

From VS Code

Alternatively, you can open a Remote WSL window directly from VS Code:

Start VS Code.
Press F1, select Remote-WSL: New Window for the default distro or Remote-WSL: New Window using Distro for a specific distro.
Use the File menu to open your folder.

If you already have a folder open, you can also use the Remote-WSL: Reopen in WSL command. You will be prompted which distro to use.

If you are in a WSL window and want to open the current input in a local window, use Remote-WSL: Reopen in Windows.

From the Windows command prompt

To open a WSL window directly from a Windows prompt use the --remote command line parameter:

code --remote wsl+<distro name> <path in WSL>

for example: code --remote wsl+Ubuntu /home/jim/projects/c

Working with Git

If you are working with the same repository in WSL and Windows, be sure to set up consistent line endings. See tips and tricks for details.

You can also avoid passwords by configuring WSL to use the Windows Git credential manager. See tips and tricks for details.

Managing extensions

VS Code runs extensions in one of two places: locally on the UI / client side, or in WSL. While extensions that affect the VS Code UI, like themes and snippets, are installed locally, most extensions will reside inside WSL.

If you install an extension from the Extensions view, it will automatically be installed in the correct location. Once installed, you can tell where an extension is installed based on the category grouping. There will be Local - Installed category and one for WSL.

Note: If you are an extension author and your extension is not working properly or installs in the wrong place, see Supporting Remote Development for details.

Local extensions that actually need to run remotely will appear dimmed and disabled in the Local - Installed category. Select Install to install an extension on your remote host.

You can also install all locally installed extensions inside WSL by going to the Extensions view and selecting Install Local Extensions in WSL: [Name] using the cloud button at the right of the Local - Installed title bar. This will display a dropdown where you can select which locally installed extensions to install in your WSL instance.

Opening a terminal in WSL

Opening a terminal in WSL from VS Code is simple. Once folder is opened in WSL, any terminal window you open in VS Code (Terminal > New Terminal) will automatically run in WSL rather than locally.

You can also use the code command line from this same terminal window to perform a number of operations such as opening a new file or folder in WSL. Type code --help to see what options are available from the command line.

Debugging in WSL

Once you've opened a folder in WSL, you can use VS Code's debugger in the same way you would when running the application locally. For example, if you select a launch configuration in launch.json and start debugging (F5), the application will start on remote host and attach the debugger to it.

See the debugging documentation for details on configuring VS Code's debugging features in .vscode/launch.json.

WSL specific settings

VS Code's local user settings are also reused when you have opened a folder in WSL. While this keeps your user experience consistent, you may want to vary some of these settings between your local machine and WSL. Fortunately, once you have connected to WSL, you can also set WSL specific settings by running the Preferences: Open Remote Settings command from the Command Palette (F1) or by selecting the Remote tab in the Settings editor. These will override any local settings you have in place whenever you open a folder in WSL.

Advanced: Environment setup script

When VS Code Remote is started in WSL, no shell startup scripts are run. This was done to avoid issues with startup scripts that are tuned for shells. If you want to run additional commands or modify the environment this can be done in a setup script ~/.vscode-server/server-env-setup (Insiders: ~/.vscode-server-insiders/server-env-setup). If present, the script is processed before the server is started.

The script needs to be a valid Bourne shell script. Be aware that an invalid script will prevent the server from starting up. If you end up with a script that prevents the server from starting, you will have to use a regular WSL shell and delete or rename the setup script.

Check the WSL log (Remote WSL: Open WSL Log) for output and errors.

Advanced: Opening a WSL 2 folder in a container

If you are using WSL 2 and Docker Desktop's WSL 2 back-end, you can use the Remote - Containers extension to work with source code stored inside WSL! Just follow these steps:

If you have not already, install and setup Docker Desktop's WSL 2 support.
Tip: Go to Settings > Resources > WSL Integration and enable Docker integration with the WSL distribution you will be using.
If you have not already, install the Remote - Containers extension along with the WSL extension.
Next, open your source code folder in WSL as you would normally.
Once your folder is open in WSL, select Remote-Containers: Reopen Folder in Container from the Command Palette (F1).
If the folder does not have a .devcontainer/devcontainer.json file in it, you'll be asked to pick a starting point from a filterable list or an existing Dockerfile or Docker Compose file (if one exists).
The VS Code window (instance) will reload and start building the dev container. A progress notification provides status updates.
After the build completes, VS Code will automatically connect to the container. You can now work with the your source code from inside the container.

See the Remote - Containers documentation for more information.

Known limitations

This section contains a list of common know issues with WSL. The intent is not to provide a complete list of issues but to highlight some of the common problems seen with WSL.

See here for a list of active issues related to WSL.

I see EACCESS: permission denied error trying to rename a folder in the open workspace in WSL 1

That's a known problem with the WSL file system implementation (Microsoft/WSL#3395, Microsoft/WSL#1956) caused by the file watcher active by VSCode. The issue will only be fixed in WSL 2.

To avoid the issue, set remote.WSL.fileWatcher.polling to true. However, polling based file watching has a performance impact for large workspaces.

For large workspace you want to increase the polling interval: remote.WSL.fileWatcher.pollingInterval and control the folders that are watched: files.watcherExclude.

WSL 2 does not have that file watcher problem is also not affected by the new setting.

Golang in WSL 1

Issue	Existing issues
Delve debugger doesn't work under WSL	go-delve/delve#810, Microsoft/vscode-go#926

Node.js in WSL 1

Issue	Existing issues
NodeJS Error: spawn EACCES (different variants of this error)	Microsoft/WSL#3886
Webpack HMR not working	Microsoft/WSL#2709
Firebase via node unusably slow only on WSL	Microsoft/WSL#2657

Git limitations

If you clone a Git repository using SSH and your SSH key has a passphrase, VS Code's pull and sync features may hang when running remotely. Either use an SSH key without a passphrase, clone using HTTPS, or run git push from the command line to work around the issue.

Docker Extension limitations

While the Docker extension can run both remotely and locally, if it is already installed locally, you will be unable to install on a remote SSH host without first uninstalling it locally. We will address this problem in a future VS Code release.

Extension limitations

Many extensions will work in WSL without modification. However, in some cases, certain features may require changes. If you run into an extension issue, see here for a summary of common problems and solutions that you can mention to the extension author when reporting the issue.

In addition, some extensions installed in an WSL when using an Alpine Linux-based distribution may not work due to glibc dependencies in native code inside the extension. See the Remote Development with Linux article for details.

Common questions

Why am I asked to change the default distro?

When using Remote-WSL: New Window using Distro and running on WSL older than Windows 10, May 2019 Update (version 1903) you will be asked to switch the default distribution as the WSL command can only work on the default distro as it does not support the -d option yet.

You can always manually switch the default distro by using wslconfig.exe.

For example:

You can see which distributions you have installed using:

I'm seeing an error about a missing library or dependency

Some extensions rely on libraries not found in the vanilla install of certain WSL Linux distributions. You can add additional libraries into your Linux distribution by using its package manager. For Ubuntu and Debian based distributions, run sudo apt-get install <package> to install the needed libraries. Check the documentation for your extension or the runtime that is mentioned for additional installation details.

What are the connectivity requirements for the Remote - WSL extension?

The Remote - WSL extension and VS Code Server require outbound HTTPS (port 443) connectivity to:

update.code.visualstudio.com
marketplace.visualstudio.com
vscode.blob.core.windows.net
*.vo.msecnd.net (Azure CDN)
*.gallerycdn.vsassets.io (Azure CDN)

Some extensions (like C#) download secondary dependencies from download.microsoft.com or download.visualstudio.microsoft.com. Others (like Visual Studio Live Share) may have additional connectivity requirements. Consult the extension's documentation for details if you run into trouble.

All other communication between the server and the VS Code client is accomplished through an random local TCP port. You can find a list of locations VS Code itself needs access to in the network connections article.

I'm behind a proxy and have connectivity issues

Proxy settings might be missing on either the Windows or the WSL side.

When a remote window is opened out of VSCode, the Remote - WSL extension tries to download the VSCode server on the Windows side. It therefore uses the Window side proxy configuration:

inherited from the OS settings
as described in Network Connections in Visual Studio Code

When the remote VSCode is started from a WSL terminal, the download is done using wget in the WSL distro. Proxy settings can be configured in:

wget proxy settings: https://stackoverflow.com/questions/11211705/how-to-set-proxy-for-wget
manually in the server setup script

Once the server is up and running the proxy settings on the Remote tab are used.

Can I force an extension to run locally / remotely ?

Extensions are typically designed and tested to either run locally or remotely, not both. However, if an extension supports it, you can force it to run in a particular location in your settings.json file.

For example, the setting below will force the Docker extension to run locally and Debugger for Chrome extension to run remotely instead of their defaults:

A value of 'ui' instead of 'workspace' will force the extension to run on the local UI/client side instead. Typically, this should only be used for testing unless otherwise noted in the extension's documentation since it can break extensions. See the article on Supporting Remote Development for details.

As an extension author, what do I need to do?

The VS Code extension API abstracts away local/remote details so most extensions will work without modification. However, given extensions can use any node module or runtime they want, there are situations where adjustments may need to be made. We recommend you test your extension to be sure that no updates are required. See Supporting Remote Development for details.

Questions or feedback

See Tips and Tricks or the FAQ.
Search on Stack Overflow.
Add a feature requests or report a problem.
Contribute to our documentation or VS Code itself.
See our CONTRIBUTING guide for details.

-->

Learn how to use Spark & Hive Tools for Visual Studio Code to create and submit PySpark scripts for Apache Spark, first we'll describe how to install the Spark & Hive tools in Visual Studio Code and then we'll walk through how to submit jobs to Spark.

Spark & Hive Tools can be installed on platforms that are supported by Visual Studio Code, which include Windows, Linux, and macOS. Below you'll find the prerequisites for different platforms.

Prerequisites

The following items are required for completing the steps in this article:

A SQL Server big data cluster. See SQL Server Big Data Clusters.
Visual Studio Code.
Mono. Mono is only required for Linux and macOS.
Set up PySpark interactive environment for Visual Studio Code.
A local directory named SQLBDCexample. This article uses C:SQLBDCSQLBDCexample.

Install Spark & Hive Tools

After you have completed the prerequisites, you can install Spark & Hive Tools for Visual Studio Code. Complete the following steps to install Spark & Hive Tools:

Open Visual Studio Code.
From the menu bar, navigate to View > Extensions.
In the search box, enter Spark & Hive.
Select Spark & Hive Tools from the search results, and then select Install.
Reload when needed.

Open work folder

Complete the following steps to open a work folder, and create a file in Visual Studio Code:

Scala Visual Studio Code

From the menu bar, navigate to File > Open Folder... > C:SQLBDCSQLBDCexample, then select the Select Folder button. The folder appears in the Explorer view on the left.
From the Explorer view, select the folder, SQLBDCexample, and then the New File icon next to the work folder.
Name the new file with the .py (Spark script) file extension. This example uses HelloWorld.py.
Copy and paste the following code into the script file:

Link a SQL Server big data cluster

Before you can submit scripts to your clusters from Visual Studio Code, you need to link a SQL Server big data cluster.

From the menu bar navigate to View > Command Palette..., and enter Spark / Hive: Link a Cluster.
Select linked cluster type SQL Server Big Data.
Enter SQL Server Big Data endpoint.
Enter SQL Server Big Data Cluster user name.
Enter password for user admin.
Set the display name of the cluster (Optional).
List clusters, review OUTPUT view for verification.

List clusters

From the menu bar navigate to View > Command Palette..., and enter Spark / Hive: List Cluster.
Review the OUTPUT view. The view will show your linked cluster(s).

Set default cluster

Re-Open the folder SQLBDCexample created earlier if closed.
Select the file HelloWorld.py created earlier and it will open in the script editor.
Link a cluster if you haven't yet done so.
Right-click the script editor, and select Spark / Hive: Set Default Cluster.
Select a cluster as the default cluster for the current script file. The tools automatically update the configuration file .VSCodesettings.json.

Submit interactive PySpark queries

You can submit interactive PySpark queries by following the steps below:

Reopen the folder SQLBDCexample created earlier if closed.
Select the file HelloWorld.py created earlier and it will open in the script editor.
Link a cluster if you haven't yet done so.
Choose all the code and right-click the script editor, select Spark: PySpark Interactive to submit the query, or use shortcut Ctrl + Alt + I.
Select the cluster if you haven't specified a default cluster. After a few moments, the Python Interactive results appear in a new tab. The tools also allow you to submit a block of code instead of the whole script file using the context menu.
Enter '%%info', and then press Shift + Enter to view job information. (Optional)
Note
When Python Extension Enabled is unchecked in the settings (The default setting is checked), the submitted pyspark interaction results will use the old window.

Submit PySpark batch job

Reopen the folder SQLBDCexample created earlier if closed.
Select the file HelloWorld.py created earlier and it will open in the script editor.
Link a cluster if you haven't yet done so.
Right-click the script editor, and then select Spark: PySpark Batch, or use shortcut Ctrl + Alt + H.
Select the cluster if you haven't specified a default cluster. After you submit a Python job, submission logs appear in the OUTPUT window in Visual Studio Code. The Spark UI URL and Yarn UI URL are shown as well. You can open the URL in a web browser to track the job status.

Apache Livy configuration

Apache Livy configuration is supported, it can be set at the .VSCodesettings.json in the work space folder. Currently, Livy configuration only supports Python script. More details, see Livy README.

How to trigger Livy configuration

Method 1

From the menu bar, navigate to File > Preferences > Settings.
In the Search settings text box enter HDInsight Job Sumission: Livy Conf.
Select Edit in settings.json for the relevant search result.

Method 2

Submit a file, notice the .vscode folder is added automatically to the work folder. You can find the Livy configuration by clicking .vscodesettings.json.

Scala Development Visual Studio Code

The project settings:

Note

For settings driverMomory and executorMomry, set the value with unit, for example 1g or 1024m.

Vscode Scala Spark

Supported Livy configurations

POST /batches

Request body

name	description	type
file	File containing the application to execute	path (required)
proxyUser	User to impersonate when running the job	string
className	Application Java/Spark main class	string
args	Command line arguments for the application	list of strings
jars	jars to be used in this session	List of string
pyFiles	Python files to be used in this session	List of string
files	files to be used in this session	List of string
driverMemory	Amount of memory to use for the driver process	string
driverCores	Number of cores to use for the driver process	int
executorMemory	Amount of memory to use per executor process	string
executorCores	Number of cores to use for each executor	int
numExecutors	Number of executors to launch for this session	int
archives	Archives to be used in this session	List of string
queue	The name of the YARN queue to which submitted	string
name	The name of this session	string
conf	Spark configuration properties	Map of key=val

Response Body

The created batch object.

Visual Studio Scala Plugin

name	description	type
id	The session id	int
appId	The application id of this session	String
appInfo	The detailed application info	Map of key=val
log	The log lines	list of strings
state	The batch state	string

Note

The assigned Livy config will display in output pane when submit script.

Additional features

Spark & Hive for Visual Studio Code supports the following features:

Scala Maven Visual Studio Code

IntelliSense autocomplete. Suggestions pop up for keyword, methods, variables, and so on. Different icons represent different types of objects.
IntelliSense error marker. The language service underlines the editing errors for the Hive script.
Syntax highlights. The language service uses different colors to differentiate variables, keywords, data type, functions, and so on.

Unlink cluster

From the menu bar navigate to View > Command Palette..., and then enter Spark / Hive: Unlink a Cluster.
Select cluster to unlink.
Review OUTPUT view for verification.

Next steps

For more information on SQL Server big data cluster and related scenarios, See SQL Server Big Data Clusters.