Fiscal Responsibility Act of 2004

Fiscal Responsibility Act of 2004 - Provides that, if there is a deficit in a fiscal year, then: (1) any pay adjustment for Members of Congress scheduled to take effect in the succeeding calendar year shall be null and void; and (2) effective as of the first day of the first pay period beginning in such year, rates of pay for Members shall be reduced (from the rate in effect as of the day before the start of such year) by the amount determined in this Act, but not below zero.

Varies the mandatory pay rate reduction from five percent, if it takes effect in a calendar year that does not immediately follow another calendar year in which a reduction was made, to ten percent, if it takes effect in a calendar year that immediately follows another calendar year in which a reduction was made.

Specifies conditions for restoration of the original pay rate, together with otherwise scheduled adjustments.

Vests authority in the Director of the Congressional Budget Office to make determinations of whether or not a deficit exists in any fiscal year.