SlideShare a Scribd company logo
1
NuGet beyond Hello World
Maarten Balliauw
@maartenballiauw
3
Agenda
Introduction
Beyond package dependencies
dotnet new templates
Extending tools
Using client SDK
Server API
4
Introduction
5
NuGet?
Project started in 2011 to make dependency management easier & better
Several things
Repositories
Public repository at www.nuget.org
Private repositories everywhere (www.myget.org)
Clients & tools
NuGet.exe, Visual Studio, Rider, Xamarin Studio, Paket, …
dotnet CLI tools
Conventions
SemVer, metadata and symbols
6
Maturity model
No package
manager in
use
Consuming
publicly
available
packages
Building and
consuming
own packages
Proper
SemVer,
promotion
flow, license
and
vulnerability
management
7
Beyond project
dependencies
8
NuGet
More than just a package manager
Extensibility (dotnet new, dotnet CLI)
NuGet is… a protocol!
API for transferring metadata and binaries
Server API
Client-side libraries
Versioning system built-in
9
dotnet new
10
Project/item templates as a package
New MVC project:
dotnet new mvc --auth Individual
New Web.config file:
dotnet new webconfig
11
Project/item templates as a package
Roll your own templates!
https://docs.microsoft.com/en-us/dotnet/core/tutorials/create-custom-template
https://github.com/dotnet/templating/wiki/Available-templates-for-dotnet-new
TL;DR:
Add /.template.config/template.json file in package
Add template files, e.g. /ItemTemplate.cs
Create a NuGet package containing the above
12
Project/item templates as a package
13
dotnet new
project template
demo
14
Extending dotnet CLI
15
Extending dotnet CLI
Why?
Existing examples
dotnet nuget – the NuGet command line, integrated
dotnet ef – Entity Framework command line tools
dotnet watch – Automatically load updates to an ASP.NET Core application
Create tools your team/customers can use (please go build these for me )
dotnet protobuf – Companion tool for creating protobuf models
dotnet optimize-images – Optimize images in a project
dotnet outdated – Scan and show outdated package references
dotnet mirror – Mirror referenced packages to a local folder
16
Extending dotnet CLI
How?
Create a console application
Name it dotnet-<something>
Package it with <PackageType>DotnetCliTool</PackageType>
Install as a CLI tool (or in %PATH%)
(per project in .NET Core <= 2.0, global in .NET Core >= 2.1)
https://github.com/dotnet/docs/blob/master/docs/core/tools/extensibility.md
17
Extending dotnet CLI
demo
https://github.com/maartenba/dotnetcli-init
18
Client SDK
19
Consuming NuGet programatically
Why?
Find package metadata
Download and extract package
Custom build task, custom tool used in CI, …
Distribute applications / plugins as NuGet packages
Transport and versioning come free!
dotnet new, NuGet Package Explorer, R# extension manager, Chocolatey, Octopus Deploy,
Seq, CMS like EPIserver, SiteCore, Orchard, …
20
Consuming NuGet programatically
How?
Old world: NuGet.Core
New world: NuGet v3 client
Series of packages, NuGet.PackageManagement being the main package
Open-source on GitHub
Dave Glick has a good series (part 1, part 2, part 3)
21
Packages to install
NuGet.PackageManagement
Overall management of packages, sources, …
NuGet.Protocol.Core.v2
V2 protocol client (OData)
NuGet.Protocol.Core.v3
V3 protocol client (JSON and randomness)
NuGet.ProjectManagement
Project system (inside IDE and inside file system)
NuGet.Packaging
Read and write .nuspec and .nupkg
NuGet.Versioning
Implements working with versions and ranges
NuGet.Commands (optional)
Command-line commens (spec, pack, publish, ...) implemented as static classes mostly
22
NuGet, via C#
demo
23
A few things to know…
All operations performed via resources (e.g. PackageSearchResource,
ListCommandResource, …)
All operations performed against a project
Folder (note: only supports installing as there is no tracking)
PackagesConfig
MSBuild
Your own implementation
Package manager orchestrates all operations
Package repository holds packages (can be remote or local)
24
SearchPortal
demo
25
Server API
26
NuGet talks to a repository
Can be on disk/network share
Or remote over HTTP(S)
2* API’s
V2 – OData based (used by pretty much all NuGet servers out there)
V3 – JSON based (available on NuGet.org and MyGet.org)
* This is a lie…
27
V2 Protocol
OData, started as “OData-to-LINQ-to-Entities” (V1 protocol)
Optimizations added to reduce # of random DB queries (VS2013+ & NuGet 2.x)
Search – Package manager list/search
FindPackagesById – Package restore (Does it exist? Where to download?)
GetUpdates – Package manager updates
https://www.nuget.org/api/v2 (code in https://github.com/NuGet/NuGetGallery)
28
V3 Protocol
JSON based
More of a “resource provider” (see client SDK) of various endpoints per purpose
Catalog (NuGet.org only) – append-only event log
Registrations – latest incarnation of a package
Flat container - .NET Core package restore (and VS autocompletion)
Report abuse URL template
Statistics
…
https://api.nuget.org/v3/index.json (code in https://github.com/NuGet/NuGet.Services.Metadata)
29
V3 Protocol is… interesting
Service index points to resources
@id, @type (versioned), comment
Client should load-balance on @type occurrences
NuGet.org does this to perform search etc.: RetryingHttpClientWrapper
(also has a request queue)
Many views on the same data
Specific to a scenario in the client
Some compute based, some storage based
Search, autocomplete -> compute
Registrations, flatcontainer -> storage
Catalog -> NuGet.org internal use
30
Server API
demo
31
How does it fit together?
User uploads to NuGet.org
Data added to database
Data added to catalog (append-only data stream)
Various jobs run over catalog using a cursor
Registrations (last state of a package/version), reference catalog entry
Flatcontainer (fast restores)
Search index (search, autocomplete, NuGet Gallery search)
32
How does it fit together?
User uploads to NuGet.org
Data added to database
Data added to catalog (append-only data stream)
Various jobs run over catalog using a cursor
Registrations (last state of a package/version), reference catalog entry
Flatcontainer (fast restores)
Search index (search, autocomplete, NuGet Gallery search)
33
NuGet.org catalog
demo
34
Catalog data
Updates (add/update)
Deletes
Chronological
Can continue where left off
Can restore NuGet.org to a given point in time
Why use it?
Mirror locally
Send notifications when certain packages changed
Build own index, build own datastore based on data
E.g. https://packagesearch.azurewebsites.net/
…
35
Other extensibility
points
36
Other extensibility points
NuGet Credential Provider
https://docs.microsoft.com/en-us/nuget/api/nuget-exe-credential-providers
NuGet package download
https://github.com/NuGet/Home/wiki/NuGet-Package-Download-Plugin
37
Conclusion
38
Conclusion
Beyond package dependencies
dotnet new templates
Extending tools
Using client SDK
Server API
39
Thank you!
http://blog.maartenballiauw.be
@maartenballiauw

More Related Content

ConFoo - NuGet beyond Hello World

  • 1. 1 NuGet beyond Hello World Maarten Balliauw @maartenballiauw
  • 2. 3 Agenda Introduction Beyond package dependencies dotnet new templates Extending tools Using client SDK Server API
  • 4. 5 NuGet? Project started in 2011 to make dependency management easier & better Several things Repositories Public repository at www.nuget.org Private repositories everywhere (www.myget.org) Clients & tools NuGet.exe, Visual Studio, Rider, Xamarin Studio, Paket, … dotnet CLI tools Conventions SemVer, metadata and symbols
  • 5. 6 Maturity model No package manager in use Consuming publicly available packages Building and consuming own packages Proper SemVer, promotion flow, license and vulnerability management
  • 7. 8 NuGet More than just a package manager Extensibility (dotnet new, dotnet CLI) NuGet is… a protocol! API for transferring metadata and binaries Server API Client-side libraries Versioning system built-in
  • 9. 10 Project/item templates as a package New MVC project: dotnet new mvc --auth Individual New Web.config file: dotnet new webconfig
  • 10. 11 Project/item templates as a package Roll your own templates! https://docs.microsoft.com/en-us/dotnet/core/tutorials/create-custom-template https://github.com/dotnet/templating/wiki/Available-templates-for-dotnet-new TL;DR: Add /.template.config/template.json file in package Add template files, e.g. /ItemTemplate.cs Create a NuGet package containing the above
  • 14. 15 Extending dotnet CLI Why? Existing examples dotnet nuget – the NuGet command line, integrated dotnet ef – Entity Framework command line tools dotnet watch – Automatically load updates to an ASP.NET Core application Create tools your team/customers can use (please go build these for me ) dotnet protobuf – Companion tool for creating protobuf models dotnet optimize-images – Optimize images in a project dotnet outdated – Scan and show outdated package references dotnet mirror – Mirror referenced packages to a local folder
  • 15. 16 Extending dotnet CLI How? Create a console application Name it dotnet-<something> Package it with <PackageType>DotnetCliTool</PackageType> Install as a CLI tool (or in %PATH%) (per project in .NET Core <= 2.0, global in .NET Core >= 2.1) https://github.com/dotnet/docs/blob/master/docs/core/tools/extensibility.md
  • 18. 19 Consuming NuGet programatically Why? Find package metadata Download and extract package Custom build task, custom tool used in CI, … Distribute applications / plugins as NuGet packages Transport and versioning come free! dotnet new, NuGet Package Explorer, R# extension manager, Chocolatey, Octopus Deploy, Seq, CMS like EPIserver, SiteCore, Orchard, …
  • 19. 20 Consuming NuGet programatically How? Old world: NuGet.Core New world: NuGet v3 client Series of packages, NuGet.PackageManagement being the main package Open-source on GitHub Dave Glick has a good series (part 1, part 2, part 3)
  • 20. 21 Packages to install NuGet.PackageManagement Overall management of packages, sources, … NuGet.Protocol.Core.v2 V2 protocol client (OData) NuGet.Protocol.Core.v3 V3 protocol client (JSON and randomness) NuGet.ProjectManagement Project system (inside IDE and inside file system) NuGet.Packaging Read and write .nuspec and .nupkg NuGet.Versioning Implements working with versions and ranges NuGet.Commands (optional) Command-line commens (spec, pack, publish, ...) implemented as static classes mostly
  • 22. 23 A few things to know… All operations performed via resources (e.g. PackageSearchResource, ListCommandResource, …) All operations performed against a project Folder (note: only supports installing as there is no tracking) PackagesConfig MSBuild Your own implementation Package manager orchestrates all operations Package repository holds packages (can be remote or local)
  • 25. 26 NuGet talks to a repository Can be on disk/network share Or remote over HTTP(S) 2* API’s V2 – OData based (used by pretty much all NuGet servers out there) V3 – JSON based (available on NuGet.org and MyGet.org) * This is a lie…
  • 26. 27 V2 Protocol OData, started as “OData-to-LINQ-to-Entities” (V1 protocol) Optimizations added to reduce # of random DB queries (VS2013+ & NuGet 2.x) Search – Package manager list/search FindPackagesById – Package restore (Does it exist? Where to download?) GetUpdates – Package manager updates https://www.nuget.org/api/v2 (code in https://github.com/NuGet/NuGetGallery)
  • 27. 28 V3 Protocol JSON based More of a “resource provider” (see client SDK) of various endpoints per purpose Catalog (NuGet.org only) – append-only event log Registrations – latest incarnation of a package Flat container - .NET Core package restore (and VS autocompletion) Report abuse URL template Statistics … https://api.nuget.org/v3/index.json (code in https://github.com/NuGet/NuGet.Services.Metadata)
  • 28. 29 V3 Protocol is… interesting Service index points to resources @id, @type (versioned), comment Client should load-balance on @type occurrences NuGet.org does this to perform search etc.: RetryingHttpClientWrapper (also has a request queue) Many views on the same data Specific to a scenario in the client Some compute based, some storage based Search, autocomplete -> compute Registrations, flatcontainer -> storage Catalog -> NuGet.org internal use
  • 30. 31 How does it fit together? User uploads to NuGet.org Data added to database Data added to catalog (append-only data stream) Various jobs run over catalog using a cursor Registrations (last state of a package/version), reference catalog entry Flatcontainer (fast restores) Search index (search, autocomplete, NuGet Gallery search)
  • 31. 32 How does it fit together? User uploads to NuGet.org Data added to database Data added to catalog (append-only data stream) Various jobs run over catalog using a cursor Registrations (last state of a package/version), reference catalog entry Flatcontainer (fast restores) Search index (search, autocomplete, NuGet Gallery search)
  • 33. 34 Catalog data Updates (add/update) Deletes Chronological Can continue where left off Can restore NuGet.org to a given point in time Why use it? Mirror locally Send notifications when certain packages changed Build own index, build own datastore based on data E.g. https://packagesearch.azurewebsites.net/ …
  • 35. 36 Other extensibility points NuGet Credential Provider https://docs.microsoft.com/en-us/nuget/api/nuget-exe-credential-providers NuGet package download https://github.com/NuGet/Home/wiki/NuGet-Package-Download-Plugin
  • 37. 38 Conclusion Beyond package dependencies dotnet new templates Extending tools Using client SDK Server API

Editor's Notes

  1. https://pixabay.com
  2. https://pixabay.com/en/tires-used-tires-pfu-garbage-1846674/
  3. Level 0 – No package manager in use Level 1 – Consuming publicly available packages Level 2 – Building and consuming own packages
  4. https://pixabay.com/en/tires-used-tires-pfu-garbage-1846674/
  5. Open package in NPE Show people around the files Add as template to Rider Create new project
  6. Show the tool in action! Open Rider, create new class library Edit project file, add <ItemGroup><DotNetCliToolReference Include="DotNetInit" Version="*" /></ItemGroup> Run dotnet restore & dotnet init (in correct folder) Profit How to create such tool? Not going to go through code completely as it’s on GitHub, but in simple terms: Open https://github.com/maartenba/dotnetcli-init See it’s just Program.cs that takes args Can be run standalone Edit project file, show: To be part of dotnet-cli -> name it dotnet-<something> Package type (case sensitive)
  7. Create new Console Application in Rider Explain we will build 2 scenarios: a search which displays package metadata, and an install/extract Add NuGet package: NuGet.PackageManagement (should bring down all dependencies we need) AND NuGet.Protocol.Core.v2 Let’s start with search first. Program.cs -> SearchPackageAndDumpDetails("json"); Code – make sure to explain: ILogger Everything is based around resource providers (like protocol support, search, …) private static async Task SearchPackageAndDumpDetails(string searchTerm) { var logger = new NullLogger(); var providers = new List<Lazy<INuGetResourceProvider>>(); providers.AddRange(Repository.Provider.GetCoreV2()); providers.AddRange(Repository.Provider.GetCoreV3()); var packageSource = new PackageSource("https://api.nuget.org/v3/index.json"); var repository = new SourceRepository(packageSource, providers); var searchResource = await repository.GetResourceAsync<PackageSearchResource>(); var searchResults = await searchResource.SearchAsync(searchTerm, new SearchFilter(includePrerelease: true), 0, 10, logger, CancellationToken.None); foreach (var searchResult in searchResults) { Console.WriteLine(searchResult.Identity.Id + " " + searchResult.Identity.Version); } Console.ReadLine(); } Now let’s also install a package. For this, we need a project context. Mention context provider for logging or use EmptyNuGetProjectContext Code: private static async Task InstallPackage(string packageId, string packageVersion, bool includeDependencies) { // Package we ant to install var packageIdentity = new PackageIdentity(packageId, NuGetVersion.Parse(packageVersion)); // Remote var providers = new List<Lazy<INuGetResourceProvider>>(); providers.AddRange(Repository.Provider.GetCoreV2()); providers.AddRange(Repository.Provider.GetCoreV3()); var packageSource = new PackageSource("https://api.nuget.org/v3/index.json"); var repository = new SourceRepository(packageSource, providers); // Local var projectPath = "c:\\users\\maart\\desktop\\temp"; var packagesPath = "c:\\users\\maart\\desktop\\temp\\packages"; var project = new FolderNuGetProject(projectPath); var settings = Settings.LoadDefaultSettings(projectPath, null, new XPlatMachineWideSetting()); // Create package manager var repositoryProvider = new SourceRepositoryProvider(settings, providers); var packageManager = new NuGetPackageManager(repositoryProvider, settings, packagesPath) { PackagesFolderNuGetProject = project }; // Create dependency resolution context (no prerelease/unlisted) var dependencyBehaviour = includeDependencies ? DependencyBehavior.Lowest : DependencyBehavior.Ignore; var resolutionContext = new ResolutionContext(dependencyBehaviour, false, false, VersionConstraints.None); // Install the package already var projectContext = new LoggingNuGetProjectContext(); await packageManager.InstallPackageAsync(packageManager.PackagesFolderNuGetProject, packageIdentity, resolutionContext, projectContext, repository, new SourceRepository[] {}, CancellationToken.None); }
  8. Open SearchPortal application, demo things (install plugin, see it shows up in portal) In code, walk through PluginManager class Plugins stored in plugins folder ListPlugins -> searches remote and combines with local which is an issue with FolderNuGetProject! (no tracking) SearchPortalNuGetProject -> wraps a packages config project as well to keep track of what is actually installed Install -> just like in previous demo Uninstall -> again using override as FolderNuGetProject has no tracking (will not always work as appdomain needs to be unloaded to remove binaries, could do shadow copy to overcome this but meh, demo purposes)
  9. Open Fiddler files Go through V2 one Go through V3 one and look at various resources, mention @id, @type, comment Mention versions as well (e.g. registrations with/without semver2 support) Open a few resources and try some requests Autocomplete: https://api-v2v3search-0.nuget.org/autocomplete?q=json&supportedFramework=any&prerelease=true Autocomplete versions: https://api-v2v3search-0.nuget.org/autocomplete?id=newtonsoft.json Search: https://api-v2v3search-0.nuget.org/query?q=json&supportedFramework=any&prerelease=true Note only latest version + list of version numbers Note link to catalog and registrations Registration: https://api.nuget.org/v3/registration0/newtonsoft.json/index.json Note metadata Note paging, ranges, ... Registration version: https://api.nuget.org/v3/registration0/newtonsoft.json/10.0.1.json Flat container (templated): https://api.nuget.org/v3/flatcontainer/newtonsoft.json/index.json Flat container download (templated): https://api.nuget.org/v3/flatcontainer/newtonsoft.json/10.0.1/newtonsoft.json.10.0.1.nupkg Catalog: list of commits https://api.nuget.org/v3/catalog0/index.json Catalog: specific page https://api.nuget.org/v3/catalog0/page2318.json
  10. Open collector demo project Show packges installed, most important: NuGet.Services.Metadata.Catalog Contains “collectors” that traverse over the catalog and to which we can subscribe Quite an internal implementation, but created a SimpleCommitCollector Handles OnProcessBatch and does some data transformation Fires either ProcessPackageUpdateAsync or ProcessPackageDeleteAsync View hierarchy (Ctrl+E, Ctrl+H) Mention NuGet implementations galore in NuGet.Services.Metadata, even unused ones (but also the ones that create search index, flatcontainer, ...) Example: EchoCollector Run it Breakpoint, show properties available Example: MirrorCollector Run it See it mirrors packages Dirty way of building download URL, should use service index to get template, but it works anyway