There is no provision for delaying the outgoing president's departure.
It would be possible for the outgoing vice president to resign and allow the incoming president to become vice president prior to the Sabbath. Then the outgoing president could resign and allow the incoming president to take office early. The outgoing president could choose to do that, but the incoming president couldn't force it.
Two presidents, James Monroe and Zachary Taylor, delayed their inaugurations because they fell on Sundays (the Christian Sabbath). From Wikipedia:
As neither Taylor nor Fillmore had taken the oath of office on March 4, some historians and Constitutional scholars have argued that neither of them had any legal authority as President in the interim until they did. They go on to argue that, as both President James K. Polk and Vice President George Dallas ceased to hold their offices at noon on March 4, the executive branch was officially empty, and President pro tempore of the United States Senate David Rice Atchison (who at the time was third in line to the presidency) was Acting President until the inauguration on March 5. At the time, friends jestingly pestered Atchenson for ambassadorships and cabinet positions, and he gleefully refused. Whether or not Atchison was actually Acting President or not has been a subject of debate ever since.
Millard Fillmore was Taylor's vice president.
Rutherford B. Hayes took the oath of office early, on Saturday, March 3rd, 1877. But again, he wouldn't actually become president until March 4th.
It's worth noting that January 20th, 2013 was a Sunday (the Christian Sabbath). Also in 1985 and 1957. So Barack Obama, Ronald Reagan, and Dwight Eisenhower actually faced this situation (albeit when re-elected). March 4th was on a Sunday in 1917, 1877, 1849, and 1821. Zachary Taylor in 1849 was the only initial inauguration.
Assuming the incoming president doesn't take the oath of office on time, the vice president would be president until the president-elect took the oath of office. Should the vice president be unavailable, the next two in the current line of succession are the Speaker of the House and the President pro tempore of the Senate.
Presumably the vice president would also serve as president on every other Sabbath throughout the term. Because being president is a twenty-four/seven job. There are no days off. A president must be available in case of emergency.
Note: I'm ignoring the realities of the religion, which may allow for exceptions to the Sabbath rules. The question assumes that this wouldn't be an exception, so I'm going with that. But if all religions would make this an exception, these contingencies may not be necessary.