SYN caching allocates some state on the machine, but even with this reduced state it is possible to encounter resource exhaustion. The code must be prepared to handle state overflows and choose which items to drop in order to preserve fairness.
The initial SYN request carries a collection of options which apply the TCP connection, these commonly include the desired MSS, requested window scaling for the connection, use of timestamps, and various other items. Part of the purpose of the allocated state is to record these options, which are not retransmitted in the return ACK from the client.
SYN cookies do not store any state on the machine, but keep all state regarding the initial TCP connection in the network, treating it as an infinitely deep queue. This is done by use of a cryptographic function to encode all information into a value that is sent to the client with the SYN,ACK and returned to the server in the final portion of the 3 way handshake. While this approach appears attractive, it has the drawback of not being able to encode all the TCP options from the initial SYN into the cookie. These options are then lost, denying the use of certain TCP performance enhancements.
A secondary problem related to cookies is that the TCP protocol requires unacknowledged data to be retransmitted. The server is supposed to retransmit the SYN,ACK before giving up and dropping the connection, whereupon a RST is sent to the client in order to shutdown the connection. When SYN,ACK arrives at a client but the return ACK is lost, this results in a disparity about the established state between the client and server. Ordinarily, this case will be handled by server retransmits, but in the case of SYN cookies, there is no state kept on the server, and a retransmission is not possible.
SYN cookies also have the property that the entire connection establishment is performed by the returning ACK, independent of the preceding SYN and SYN,ACK transmissions. This opens the possibility of flooding the server with ACK requests, in hopes that one will contain the correct value which allows a connection to be established. This also provides an approach to bypass firewalls which restrict external connections by filtering out incoming packets which have the SYN bit set, since initial SYN packet is no longer required to establish a connection.
Another difficulty with cookies is that they are incompatible with transactional TCP. T/TCP works by sending monotonically increasing sequence numbers to the peer in the TCP options field, and uses previously received sequence numbers to establish connections on the initial SYN, eliminating the 3 way handshake. However, use of the T/TCP sequence numbers is mandatory once a TCP connection is initiated, and this requires the server to record the initial sequence number, and whether the T/TCP option was requested.
Thus cookies cannot be used as the normal line of defense in a high performance server. The usual approach is to use a state allocation mechanism, and fall back to using cookies only after a certain amount of state has been allocated. This is the approach taken by the the Linux kernel implementation.