Python Software Foundation News: pypi

Showing posts with label pypi. Show all posts

Thursday, February 11, 2021

Welcoming Google as a Visionary Sponsor of the PSF

Our top sponsors—companies who step forward to make the biggest investment in Python and its community—not only use Python for their own internal development, but also offer Python as a crucial part of the products they offer to their own customers. That is certainly true of Google, the Python Software Foundation’s first Visionary Sponsor.

Google's donations and sponsorship funds will be used to support a number of PSF initiatives, including the first CPython Developer in Residence. The Python Steering Council and Python Software Foundation will work together to contract a developer to help CPython determine what needs to take priority through analytical metrics as well as helping CPython understand how backlog can be addressed. The role will also be responsible for surveying maintainers to paint a better landscape of CPython, which will be used to ensure future funding and volunteer hours are used efficiently and effectively.

In addition, the sponsorship funds will also be used towards critical supply-chain security improvements, including developing productized malware detection for PyPI, a prototype of dynamic analysis infrastructure for distributions, and other foundational tool improvements.

Google has been a Python Software Foundation sponsor since 2010. Our hearts are full of gratitude for their support. You can read more about how Google is supporting the Python ecosystem on their blog.

Key generation and signing ceremony for PyPI

On Friday October 30th at 11:15 AM EDT the Python Software Foundation will be live streaming a remote key generation and signing ceremony to bootstrap The Update Framework for The Python Package Index. You can click here to see what time this is in your local timezone.

This ceremony is one of the first practical steps in deploying The Update Framework to PyPI per PEP 458.

The Python Software Foundation Director of Infrastructure, Ernest W. Durbin III, and Trail of Bits Senior Security Engineer, William Woodruff, will be executing the runbook developed at https://github.com/psf/psf-tuf-runbook.

For transparency purposes a live stream will be hosted from the Python Software Foundation's YouTube channel. Please subscribe to the channel to be notified when the stream is live if you'd like to follow along.

Additionally the recording will be archived on the Python Software Foundation's YouTube channel.

This work is being funded by Facebook Research and was originally announced in late 2018 and a portion of it commenced in 2019 while awaiting PEP 458's acceptance. With PEP 458 in place we announced that work would commence in March.

We appreciate the patience and contributions of the community, Facebook Research, and Trail of Bits in seeing through the implementation of PEP 458.

Additionally volunteers from The Secure Systems Lab at NYU, Datadog, and VMWare have helped to develop the implementation for PyPI but have begun work on client implementations to verify the results in pip.

Friday, April 03, 2020

Announcing a new Sponsorship Program for Python Packaging

The Packaging Working Group of the Python Software Foundation is launching an all-new sponsorship program to sustain and improve Python's packaging ecosystem. Funds raised through this program will go directly towards improving the tools that your company uses every day and sustaining the continued operation of the Python Package Index.

With this program we are asking companies that rely on Python, its ecosystem of packaging tools, and PyPI to help us build a dependable basis to continue our efforts.

Improving the packaging ecosystem

Since 2017, the Packaging Working Group has secured multiple grants, completed one contract, and received a generous gift -- all with the goal of improving the Python packaging ecosystem for all users. Most of these projects were funded by not-for-profit organizations and all of them were one-time awards with specific objectives.

Results from these funded projects include:

The successful relaunch of the Python Package Index, powered by the new 'Warehouse' codebase in 2018
Adding security features to PyPI, including two-factor authentication in 2019
Improving PyPI's web interface for users with disabilities and adding support for multiple locales in 2019
Additional security-focused features for PyPI in 2019 and 2020
Overhauling pip's user experience and dependency resolver in 2020

Companies have asked us how they can help fund the platform they depend on. With this new sponsorship program, the Working Group can sustainably fund packaging improvements not directed by a specific grant or contract and benefit millions of Python users around the world. Greater budget flexibility and a deeper reserve will help us invest in what the community needs.

Sustaining PyPI

As of April 2020, the Python Package Index responds to 800 million requests and delivers 200 million packages totalling 400 terabytes, during the typical day. Our users include hobbyists, scientists, companies, students, governments, nonprofits, and more.

Existing sponsors donate their services, which keeps PyPI free to users and to the PSF, aside from a subset of one staff member's time. Without these donations, the costs to operate PyPI each month would be staggering.

These critical service donations must not be taken for granted. Sponsoring the Packaging Working Group through this new program creates and maintains a stable reserve. We'll need that reserve in the event that we lose any of these in-kind service donations and must pay some or all of PyPI's operating costs.

Show your support!

As a company, your team can review the details of this new sponsorship program in our prospectus. Should you have any questions you can contact us at sponsorship@pypi.org. When you're ready, apply here. We are excited to hear from you!

If your company cannot donate: Even as an individual, your contributions count! No matter the size or frequency, please support us if you are able at donate.pypi.org.

Wednesday, March 04, 2020

An Update on PyPI Funded Work

Originally announced at the end of 2018, a gift from Facebook Research is funding improvements for the security PyPI and its users.

What's been done

After launching a request for information and subsequent request for proposal in the second half of 2019, contractors were selected and work commenced on Milestone 2 of the project in December 2019 and was completed in February 2020.

The result is that PyPI now has tooling in place to implement automated checks that run in response to events such as Project or Release creation or File uploads as well as on schedules. In addition to documentation example checks were also implemented that demonstrate event based and scheduled checks.

Results from checks are made available for PyPI moderators and administrators to review, but will not have any automated responses put in place. As a check suite is developed and refined we hope that these will help to identify malicious uploads and spam that PyPI regularly contends with.

What's next

With the acceptance of PEP 458 on February 15 we're excited to announce that work on implementation of The Update Framework has started.

This work will enable clients like pip to ensure that they have downloaded valid files from PyPI and equip the PyPI administrators to better respond in event of a compromise.

The timeline for this work is currently planned over the coming months, with an initial key signing to be held at PyCon 2020 in Pittsburgh, Pennsylvania and rollout of the services needed to support TUF enabled clients in May or June.

Other PyPI News

For users who have enabled two factor authentication on PyPI, support has been added for Account Recovery codes. These codes are intended for use in the case where you've lost your Webauthn device or TOTP application, allowing you to recover access to your account.

You can generate and store recovery codes now by visiting your account settings and clicking "Generate Recovery Codes".

Friday, January 17, 2020

Start using 2FA and API tokens on PyPI

To increase the security of PyPI downloads, we have added two-factor authentication (2FA) as a login security option, and API tokens for uploading packages. This is thanks to a grant from the Open Technology Fund, coordinated by the Packaging Working Group of the Python Software Foundation.

If you maintain or own a project on the Python Package Index, you should start using these features. Click "help" on PyPI for instructions. (These features are also available on Test PyPI.)

Details and plans for the future:

2FA:

Two-factor authentication (2FA) makes your account more secure by requiring two things in order to log in: something you know and something you own.

In PyPI's case, "something you know" is your username and password, while "something you own" can be an application to generate a temporary code, or a security device (most commonly a USB key).

Why? This will help improve the security of your PyPI user accounts, and thus reduce the risk of vandals, spammers, and thieves gaining account access. Protecting login via the website safeguards against malicious changes to project ownership, deletion of old releases, and account takeovers.

PyPI's implementation of the WebAuthn standard and the TOTP standard mean you can use any TOTP authentication application and/or any 2FA device that meets the FIDO standard. (We launched WebAuthn support last year; this week it comes out of beta.)

Go to your account settings to add a second factor.

Add a second factor in your account settings.

Create a key name in the PyPI interface.

2FA only affects logging in via a web browser, and not (yet) package uploads.

API tokens:

In your Account Settings,
select "Add API token".

API tokens provide an alternative way (instead of username and password) to authenticate when uploading packages to PyPI. (We launched API token support last year; this week it comes out of beta.)

PyPI interface for adding an
API token for package upload.

Immediately after creating the API token,
PyPI gives the user one chance to copy it.

Why? These API tokens can only be used to upload packages to PyPI, and not to log in more generally. This makes it safer to automate package upload and store the credential in the cloud, since a thief who copies the token won't also gain the ability to delete the project, delete old releases, or add or remove collaborators. And, since the token is a long character string (with 32 bytes of entropy and a service identifier) that PyPI has securely generated on the server side, we vastly reduce the potential for credential reuse on other sites and for a bad actor to guess the token.

You can create a token for an entire PyPI user account, in which case, the token will work for all projects associated with that account. Alternatively, you can limit a token's scope to a specific project. That way, if a token is compromised, you can just revoke and recreate that token, instead of having to change your password in lots of automated processes.

PyPI token management interface

Go to your account settings to add an API token.

Future:

In the future, PyPI will set and enforce a policy requiring users with two-factor authentication enabled to use API tokens to upload (rather than just their password, without a second factor). We do not yet know when we will make this policy change. When we do, we'll announce it.

Thanks:

Thanks to the Open Technology Fund for funding this work.

More donor-funded work is in progress on pip and PyPI, via the PSF's Packaging Working Group. Please sign up for the PyPI Announcement Mailing List for future updates.

Wednesday, September 25, 2019

PyPI Security Q4 2019 Request for Proposals period opens.

The Python Software Foundation Packaging Working Group has received a grant from Facebook Research to implement advanced security features for PyPI. These features include cryptographic signing of uploaded artifacts and the infrastructure necessary to implement automated detection of malicious files uploaded to the index.

The Python Package Index (PyPI) is a foundational component of the Python ecosystem and broader computer software and technology landscape. This project aims to improve the security and accessibility of PyPI for all users worldwide, whether they are direct users, like project maintainers and pip installers, or indirect users. The impact of this work will be highly visible and improve crucial features of the service.

We plan to begin the project in Quarter 4 of 2019. Because of the size of the project, funding has been allocated to secure one or more contractors to complete the development, testing, verification, and assist in the rollout of necessary features.

Timeline

Date	Milestone
September 25	Request for Proposal period opened.
October 21	Request for Proposal period closes.
October 29	Date proposals will have received a decision.
December 2	Contract work commences.

What is the Request for Proposals period?

A Request for Proposal (RFP) is a process intended to allow us (The Python Software Foundation) to collect proposals from potential contractors and select contractor(s) best suited to fulfill the specified work.

After the RFP period closes we will evaluate the received proposals based on the evaluation criteria, seek clarification from proposers as necessary, and select one or more contractors to complete the work specified in the scope.

The Request for Proposals period opens today, September 25th, 2019, and is scheduled to close October 21, 2019 AoE.

How do I submit a proposal?

First, please read the full contents of the Request for Proposals here!

You'll find the instructions for submission, evaluation criteria, as well as scope of the project there.

Wednesday, July 31, 2019

PyPI now supports uploading via API token

We're further increasing the security of the Python Package Index with another new beta feature: scoped API tokens for package upload. This is thanks to a grant from the Open Technology Fund, coordinated by the Packaging Working Group of the Python Software Foundation.

Over the last few months, we've added two-factor authentication (2FA) login security methods. We added Time-based One-Time Password (TOTP) support in late May and physical security device support in mid-June. Now, over 1600 users have started using physical security devices or TOTP applications to better secure their accounts. And over the past week, over 7.8% of logins to PyPI.org have been protected by 2FA, up from 3% in the month of June.

Add API token screen, with textarea for token name and dropdown menu to choose token scope

PyPI interface for adding an
API token for package upload

Now, we have another improvement: you can use API tokens to upload packages to PyPI and Test PyPI! And we've designed the token to be a drop-in replacement for the username and password you already use (warning: this is a beta feature that we need your help to test).

How it works: Go to your PyPI account settings and select "Add API token". When you create an API token, you choose its scope: you can create a token that can upload to all the projects you maintain or own, or you can limit its scope to just one project.

API token management interface displays each token's name, scope, date/time created, and date/time last used, and the user can view each token's unique ID or revoke it

PyPI API token management interface

The token management screen shows you when each of your tokens were created, and last used. And you can revoke one token without revoking others, and without having to change your password on PyPI and in configuration files.

Uploading with an API token is currently optional but encouraged; in the future, PyPI will set and enforce a policy requiring users with two-factor authentication enabled to use API tokens to upload (rather than just their password sans second factor). Watch our announcement mailing list for future details.


Immediately after creating the API token, PyPI gives the user one chance to copy it

Why: These API tokens can only be used to upload packages to PyPI, and not to log in more generally. This makes it safer to automate package upload and store the credential in the cloud, since a thief who copies the token won't also gain the ability to delete the project, delete old releases, or add or remove collaborators. And, since the token is a long character string (with 32 bytes of entropy and a service identifier) that PyPI has securely generated on the server side, we vastly reduce the potential for credential reuse on other sites and for a bad actor to guess the token.

Help us test: Please try this out! This is a beta feature and we expect that users will find minor issues over the next few weeks; we ask for your bug reports. If you find any potential security vulnerabilities, please follow our published security policy. (Please don't report security issues in Warehouse via GitHub, IRC, or mailing lists. Instead, please directly email security@python.org.) If you find an issue that is not a security vulnerability, please report it via GitHub.

We'd particularly like testing from:

Organizations that automate uploads using continuous integration
People who save PyPI credentials in a .pypirc file
Windows users
People on mobile devices
People on very slow connections
Organizations where users share an auth token within a group
Projects with 4+ maintainers or owners
People who usually block cookies and JavaScript
People who maintain 20+ projects
People who created their PyPI account 6+ years ago

What's next for PyPI: Next, we'll move on to working on an advanced audit trail of sensitive user actions, plus improvements to accessibility and localization for PyPI (some of which have already started). More details are in our progress reports on Discourse.

Thanks to the Open Technology Fund for funding this work. And please sign up for the PyPI Announcement Mailing List for future updates.

Tuesday, June 18, 2019

PyPI Now Supports Two-Factor Login via WebAuthn

To further increase the security of Python package downloads, we're adding a new beta feature to the Python Package Index: WebAuthn support for U2F compatible hardware security keys as a two-factor authentication (2FA) login security method. This is thanks to a grant from the Open Technology Fund, coordinated by the Packaging Working Group of the Python Software Foundation.

Last month, we added the first 2FA method for users logging into the canonical Python Package Index at PyPI.org and the test site at test.pypi.org. Hundreds of project owners and maintainers have now started using that method (generating a code through a Time-based One-time Password (TOTP) application) to better secure their accounts.

Starting today, PyPI also supports (in beta) WebAuthn (U2F compatible) security keys for a second login factor. A security key (also known as a universal second factor, or U2F compatible key) is hardware device that communicates via USB, NFC, or Bluetooth. Popular keys include Yubikey, Google Titan and Thetis. PyPI supports any FIDO U2F compatible key and follows the WebAuthn standard. Users who have set up this second factor will be prompted to use their key (usually by inserting it into a USB port and pressing a button) when logging in. (This feature requires JavaScript.)

This is a beta feature and we expect that users will find minor issues over the next few weeks; we ask for your bug reports. If you find any potential security vulnerabilities, please follow our published security policy. (Please don't report security issues in Warehouse via GitHub, IRC, or mailing lists. Instead, please directly email one or more of our maintainers.) If you find an issue that is not a security vulnerability, please report it via GitHub.

We encourage project maintainers and owners to log in and go to your Account Settings to add a second factor. This will help improve the security of your PyPI user accounts, and thus reduce the risk of vandals, spammers, and thieves gaining account access. If you're not yet comfortable using a beta feature, you can provision a TOTP application for your second factor.

You'll need to verify your primary email address on your Test PyPI and/or PyPI accounts before setting up 2FA. You can also do that in your Account Settings.

2FA only affects login via the website, which safeguards against malicious changes to project ownership, deletion of old releases, and account takeovers. Package uploads will continue to work without users providing 2FA codes.

But that's just for now. We are working on implementing per-user API keys as an alternative form of multifactor authentication in the setuptools/twine/PyPI auth flows. These will be application-specific tokens scoped to individual users/projects, so that users will be able to use token-based logins to better secure uploads. And we'll move on to working on an advanced audit trail of sensitive user actions, plus improvements to accessibility and localization for PyPI. More details are in our progress reports.

Thanks to the Open Technology Fund for funding this work. And please sign up for the PyPI Announcement Mailing List for future updates.

Thursday, May 30, 2019

Use two-factor auth to improve your PyPI account's security

To increase the security of Python package downloads, we're beginning to introduce two-factor authentication (2FA) as a login security option on the Python Package Index. This is thanks to a grant from the Open Technology Fund; coordinated by the Packaging Working Group of the Python Software Foundation.

Starting today, the canonical Python Package Index at PyPI.org and the test site at test.pypi.org offer 2FA for all users. We encourage project maintainers and owners to log in and go to their Account Settings to add a second factor. This will help improve the security of their PyPI user accounts, and thus reduce the risk of vandals, spammers, and thieves gaining account access.

PyPI's maintainers tested this new feature throughout May and fixed several resulting bug reports; regardless, you might find a new issue. If you find any potential security vulnerabilities, please follow our published security policy. (Please don't report security issues in Warehouse via GitHub, IRC, or mailing lists. Instead, please directly email one or more of our maintainers.) If you find an issue that is not a security vulnerability, please report it via GitHub.

PyPI currently supports a single 2FA method: generating a code through a Time-based One-time Password (TOTP) application. After you set up 2FA on your PyPI account, then you must provide a TOTP (along with your username and password) to log in. Therefore, to use 2FA on PyPI, you'll need to provision an application (usually a mobile phone app) in order to generate authentication codes; see our FAQ for suggestions and pointers.

You'll need to verify your primary email address on your Test PyPI and/or PyPI accounts before setting up 2FA. You can also do that in your Account Settings.

Currently, only TOTP is supported as a 2FA method. Also, 2FA only affects login via the website which safeguards against malicious changes to project ownership, deletion of old releases, and account take overs. Package uploads will continue to work without 2FA codes being provided.

But we're not done! We're currently working on WebAuthn-based multi-factor authentication, which will let you use, for instance, Yubikeys for your second factor. Then we'll add API keys for package upload, then an advanced audit trail of sensitive user actions. More details are in our progress reports.

Thanks to the Open Technology Fund for funding this work. And please sign up for the PyPI Announcement Mailing List for future updates.

Wednesday, May 29, 2019

2018 in review!

Happy New Year from the PSF! We’d like to highlight some of our activities from 2018 and update the community on the initiatives we are working on.

PyCon 2018

PyCon 2018 was held in Cleveland, Ohio, US. The conference brought together 3,389 attendees from 41 countries. We awarded $118,543 in financial aid to 143 attendees. In addition to financial aid, the conference continues to offer childcare for attendees, a newcomer orientation, a PyLadies lunch, and many more events.

Registration is now open for PyCon 2019: https://pycon.blogspot.com/2018/11/pycon-2019-registration-is-open.html .

Community Support

We initiated a Python Software Foundation Meetups Pro network at the end of the year, which supports 37 meetups in 8 countries and further expansion planned. The Sponsorship model allows the PSF to invite existing groups to the Meetup Pro network. The organizers no longer pay for the meetup subscription once they become part of the PSF network. This initiative will save approximately 32 hours of PSF staff time and 21 hours of meetup organizer time.

To help with transparency, the PSF launched its first newsletter in December! If you’d like to receive our next edition, subscribe here: https://www.python.org/psf/newsletter/. You can read our first edition here: https://mailchi.mp/53049c7e2d8b/python-software-foundation-q4-newsletter

This year we formalized our fiscal sponsorship program to better support mission related projects. The PSF has signed fiscal sponsorship agreements with 8 groups including Pallets (Flask), PhillyPUG, PuPPy, PyCascades, PyHawaii, PyMNtos, PyArkansas, and the Python San Diego User Group. Through this effort, the PSF is able to support these projects by handling their accounting and admin work so the projects can concentrate on furthering their goals.

Python Package Index

Thanks to a generous award from the Mozilla Open Source Support program, the all new Python Package Index based on the warehouse codebase rollout was completed in April of 2018.

If you are interested in what the Packaging Group is currently working on, check out their RFP for security and accessibility development: http://pyfound.blogspot.com/2018/12/upcoming-pypi-improvements-for-2019.html.

Grants

The Python Ambassador program helps further the PSF's mission with the help of local Pythonistas. The goal is to perform local outreach and introduce Python to areas where it may not exist yet. In March 2018, the board approved expanding our Python Ambassador program to include East Africa. Kato Joshua and the Afrodjango Initiative have been doing great outreach in universities in Uganda, Rwanda, and Kenya.

In a general overview, $324,000 was paid in grants last year to recipients in 51 different countries. We awarded $59,804 more in grants in 2018 than 2017. That's a 22.6% increase for global community support.

Here is a chart showing the global grant distribution in 2018:

PSF Staff

In June Ernest W. Durbin III was hired as Director of Infrastructure. Ernest will be evaluating and strengthening internal systems, supporting and improving community infrastructure, and developing programs that benefit the Python community worldwide.

In September, the PSF hired Jackie Augustine as Event Manager. Jackie will be working with the team on all facets of PyCon and managing several community resources for regional conferences.

It is with great pleasure that we announce that Ewa Jodlowska will be the PSF's first Executive Director, starting January 1, 2019. Given her years of dedicated service to the PSF from event manager to her current position as Director of Operations, we can think of no one more qualified to fill this role as the PSF continues to grow and develop.

Community Recognition

Through out 2018, we presented several awards to recognize those that go above and beyond in our community. This year we gave out several Community Service Awards, a Distinguished Service Award, and a Frank Willison Memorial Award. To find out more about our awards or how to nominate someone for a Community Service Award, check out: https://www.python.org/community/awards/.

Community Service Awards

Chukwudi Nwachukwu was recognized for his contribution to spreading the growth of Python to the Nigerian community and his dedication and research to the PSF grants work group.

Mario Corchero was awarded a CSA for his leadership of the organization of PyConES, PyLondinium, and the PyCon Charlas track in 2018. His work has been instrumental in promoting the use of Python and fostering Python communities in Spain, Latin America, and the UK.

We also honored our Job Board volunteers: Jon Clements, Melanie Jutras, Rhys Yorke, Martijn Pieters, Patrice Neff, and Marc-Andre Lemburg, who have spent many hours reviewing and managing the hundreds of job postings submitted on an annual basis

Mariatta Wijaya was an awardee for her contributions to CPython, her efforts to improve the workflow of the Python core team, and her work to increase diversity in our community. In addition, her work as co-chair of PyCascades helps spread the growth of Python

Alex Gaynor received an award for his contributions to the Python and Django Communities and the Python Software Foundation. Alex previously served as a PSF Director in 2015-2016. He currently serves as an Infrastructure Staff member and contributes to legacy PyPI and the next generation warehouse and has helped legacy warehouse in security (disabling unsupported OpenID) and cutting bandwidth costs by compressing 404 images.

2018 Distinguished Service Award

The 2018 Distinguished Service Award was presented to Marc-Andre Lemburg for his significant contributions to Python as a core developer, EuroPython chair, PSF board member, and board member of the EuroPython Society.

2018 Frank Willison Memorial Award

The Frank Willison Memorial Award for Contributions to the Python Community was awarded to Audrey Roy Greenfeld and Daniel Roy Greenfeld for their contributions to the development of Python and the global Python community through their speaking, teaching, and writing.

Donations and Sponsorships

We'd like to thank all of our donors and sponsors that continue to support our mission! Donations and fundraisers resulted in $489,152 of revenue. This represents 15% of total 2018 revenue. PSF and PyCon sponsors contributed over $1,071K in revenue!

This year we welcomed 17 new sponsors in 2018 including our first Principal Sponsors, Facebook and Capital One. Thank you for your very generous support.

We welcome your thoughts on how you’d like to see our Foundation involved in Python’s ecosystem and are always interested in hearing from you. Email us!

We wish you a very successful 2019!

Ewa Jodlowska
Executive Director

Betsy Waliszewski
Sponsor Coordinator

Wednesday, March 13, 2019

Commencing Security, Accessibility, and Internationalization Improvements to PyPI for 2019

The Python Software Foundation (PSF) and PSF Packaging Working Group are excited to announce that the first round of slated improvements to PyPI for 2019 are underway. This Open Technology Fund funded project will bring improvements to the accessibility and security of the service. You can read more about the scope of this project in our request for proposals document.

We are excited to have two returning contractors from the MOSS funded work that brought the full rewrite of PyPI into production.

Kabu Creative

Responsible for the user interface and user experience of PyPI, Kabu Creative will be fulfilling those aspects of new features for the project. Additionally, we are excited for their work on auditing and improving the accessibility of PyPI's web user interface.

Changeset Consulting, LLC

With experience in project management, communications, and contributions for projects and teams across the realm of open source, Changeset Consulting will be responsible for helping to lead the project to completion. Along the way Changeset Consulting will also be performing reporting, communications, and outreach to help keep the Python community up to speed on how the project is progressing.

We are also welcoming a new contractor to the team to complete this project.

Trail of Bits

Bringing their experience securing organizations and products, Trail of Bits will be handling the backend development related to the security milestones of the project, as well as the backend development necessary for implementing internationalization of the PyPI user interface.

The PSF looks forward to sharing more here as features are developed and deployed to PyPI. Subscribe to pypi-announce for announcements of big changes to PyPI, and follow this blog for updates as the work progresses.

Thursday, December 20, 2018

Upcoming PyPI Improvements for 2019

The Python Package Index (PyPI) is far and away the largest and most visible service that the Python Software Foundation (PSF) supports for the Python community. Throughout the project’s 16 year history, it has primarily relied on volunteers and donated services to operate as it grew from an empty repository to one hosting more than 1.1 million releases for over 162,000 projects and serving more than 2.2 petabytes in 13.8 billion requests in the last month.

In November 2017, we announced an award from the Mozilla Open Source Support (MOSS) program that made it possible to launch the ground up rewrite of PyPI’s backend in April of 2018. This milestone has offered lower maintenance overhead and helped put the codebase into a much better state to add new features, improved security, and increased accessibility for users.

While some smaller features have already been proposed, designed, submitted, reviewed, and merged by volunteer contributors, other larger improvements warrant paid work. As 2019 approaches, we are excited to look forward to plans that will help deliver important improvements to the security and accessibility of PyPI.

As a grant-giving non-profit, the Python Software Foundation is grateful to the organizations that make funding this work possible. For 2019 we are glad to have two initiatives in the works.

Facebook Gift

We’re excited to announce that Facebook has provided the Python Software Foundation with a monetary gift that will be used to fund the development and deployment of enhanced security features to PyPI. As a major Python user, contributor, and supporter, Facebook was impressed with the success of the MOSS award and is enthusiastically assisting with further enhancements to PyPI with this gift.

The PSF Packaging Working Group plans to use these funds to implement highly requested security features in PyPI such as cryptographic signing and verification of files uploaded and installed from the index. Additionally, systems for the automated detection of malicious uploads will lower the time to response and improve the resiliency of PyPI against attacks such as “pytosquatting”.

This work will be undertaken in the second half of 2019 but planning will begin in the second quarter of the year.

Open Technology Fund

The Open Technology Fund (OTF) supports projects and people that develop open and accessible technologies promoting human rights and open societies and help advance inclusive and safe access to global communications networks.

The PSF Packaging Working Group is delighted to have been awarded a contract through the OTF Core Infrastructure Fund to add key security features to PyPI including API keys, multi-factor authentication, and audit logs. Additionally, accessibility and localization features will be a key focus as we ensure service’s ability to support our global community of users.

Get Involved

We plan to begin this work in the first quarter of 2019, if you’re interested in getting involved, keep reading!

If you’re interested in getting involved, you can do so today by responding to our Request for Proposals to fulfill the OTF contract. This RFP will close January 25th, 2019 AoE. If you’re interested in getting involved at a later date to complete the work planned for the Facebook Gift, keep your eyes on this blog, subscribe to the PSF newsletter, or follow us on Twitter.

Wednesday, December 19, 2018

PyPI Security and Accessibility Q1 2019 Request for Proposals Update

Earlier this year we launched a Request for Information (RFI) followed by the launch of a Request for Proposals (RFP) in November to fulfill a contract for the Open Technology Fund (OTF) Core Infrastructure Fund.

The initial deadline for our RFP was December 14th. We had hoped to begin work with the selected proposers in January 2019, but ultimately fell short of the ability to do so.

What’s holding us back

After the deadline passed there were no proposals submitted for Milestone 2. This leaves us in a position where the project cannot proceed as planned without reassessing the RFP process and extending the deadline.

How we’ll move forward

The RFP document has been updated based on feedback received from those who took part in the initial period to allow for additional flexibility on proposal parameters.

The RFP contained two milestones that could be proposed for independently or as a pair. In responses to our RFP the security development milestone (Milestone 1) received more attention than the accessibility and internationalization milestone (Milestone 2).

Given that we will at least need to extend the RFP period to obtain proposals for Milestone 2, we plan to continue to accept proposals for both. This offers us the best chance to select a proposal that will best utilize the available funds.

Our new deadline for responses will be the end of the day January 31st, 2019 AoE. You can read the full Request for Proposals document here.

If you have any questions, concerns, or feedback about the RFP please contact the Python Software Foundation Director of Infrastructure, Ernest W. Durbin III. Proposers may also discuss the RFP in our community discussion forum at discuss.python.org.