How to use Python Requests to login to website, store cookie, then access another page on the website?

Question

I'm trying to login into website using a Python script, store the cookie I receive, and then use that same cookie to access member-only parts of the website. I've read several posts and answers about this topic, but none of the answers have worked for me.

Here is the HTML code for the website login page I'm trying to access.

<form action="/login?task=user.login" method="post">
    <fieldset>
        <table border="0" cellspacing="0" cellpadding="0">
        <tbody>
                                                        <tr>
            <td width="70" nowrap="">Username&nbsp;&nbsp;</td>
            <td width="260"><input type="text" name="username" id="username" value="" class="validate-username" size="25"/></td>
                    </tr>
                                                                                <tr>
            <td width="70" nowrap="">Password&nbsp;&nbsp;</td>
             <td width="260"><input type="password" name="password" id="password" value="" class="validate-password" size="25"/></td>
         </tr>
                                                        <tr>
             <td colspan="2"><label style="float: left;width: 70%;" for="modlgn_remember">Remember Me</label>
             <input style="float: right;width: 20%;"id="modlgn_remember" type="checkbox" name="remember" class="inputbox" value="yes"/></td>
         </tr>
         <tr>
            <td  colspan="2" width="100%"> <a href="/reset-password"> Forgot your password?</a></td>
        </tr>
        <tr>
            <td  colspan="2" width="100%"> <a href="/username-reminder">Forgot your username?</a></td>
        </tr>
        <tr>
            <td colspan="2"><button type="submit" class="button cta">Log in</button></td>
<!--                            <td colspan="1"><a href="/--><!--">Register Now</a></td>-->
        </tr>
        </tbody>
        </table>

        <input type="hidden" name="return"
               value="aHR0cHM6Ly9maWYuY29tLw=="/>
        <input type="hidden" name="3295f23066f7c6ab53c290c6c022cc4b" value="1" />                    </fieldset>
</form>

Here is my own code that I'm using to attempt a login.

from requests import session

payload = {
     'username': 'MY_USERNAME',
     'password': 'MY_PASSWORD'
}

s = session()
s.post('https://fif.com/login?task=user.login', data=payload)

response = s.get('https://fif.com/tools/capacity')

From everything I have read, this should work, but it doesn't. I've been struggling with this for two days, so if you know the answer, I would love the solution.

For reference, here are all the other StackOverflow posts I have looked at in hopes for an answer:

1. Python Requests and Persistent Sessions
2. Logging into a site using Python Reqeusts
3. Login to website using python
4. How to “log in” to a website using Python's Requests module?
5. Python: Requests Session Login Cookies
6. How to use Python to login to a webpage and retrieve cookies for later usage?
7. cUrl Login then cUrl Download

Answer 1

You should be posting all the required data, you can use bs4 to parse the login page to get the values you need:

from requests import session
from bs4 import BeautifulSoup

data = {
    'username': 'MY_USERNAME',
    'password': 'MY_PASSWORD'
}

head = {"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"}
with  session() as s:
    soup = BeautifulSoup(s.get("https://fif.com/login").content)
    form_data = soup.select("form[action^=/login?task] input")
    data.update({inp["name"]: inp["value"] for inp in form_data if inp["name"] not in data})
    s.post('https://fif.com/login?task=user.login', data=data, headers=head)
    resp = s.get('https://fif.com/tools/capacity')

If you make a requests and look in chrome tools or firebug, the form data looks like:

username:foo
password:bar
return:aW5kZXgucGhwP29wdGlvbj1jb21fdXNlcnMmdmlldz1wcm9maWxl
d68a2b40daf7b6c8eaa3a2f652f7ee62:1