Merge tag 'net-next-6.12' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next

Pull networking updates from Jakub Kicinski:
 "The zero-copy changes are relatively significant, but regression risk
  should be contained. The feature needs to be used to cause trouble.

  Also it feels like we got an order of magnitude more semi-automated
  "refactoring" chaff than usual, I wonder if it's just us.

  Core & protocols:

   - Support Device Memory TCP, ability to zero-copy receive TCP
     payloads to a DMABUF region of memory while packet headers land
     separately in normal kernel buffers, and TCP processes then as
     usual.

   - The ability to read the PTP PHC (Physical Hardware Clock) alongside
     MONOTONIC_RAW timestamps with PTP_SYS_OFFSET_EXTENDED. Previously
     only CLOCK_REALTIME was supported.

   - Allow matching on all bits of IP DSCP for routing decisions.
     Previously we only supported on matching TOS bits in IPv4 which is
     a narrower interpretation of the same header field.

   - Increase the range of weights used for multi-path routing from
     8 bits to 16 bits.

   - Add support for IPv6 PIO p flag in the Prefix Information Option
     per draft-ietf-6man-pio-pflag.

   - IPv6 IOAM6 support for new tunsrc encap mode for better
     performance.

   - Detect destinations which blackhole MPTCP traffic and avoid
     initiating MPTCP connections to them for a certain period of time,
     1h by default.

   - Improve IPsec control path performance by removing the inexact
     policies list.

   - AF_VSOCK: add support for SIOCOUTQ ioctl.

   - Add enum for reasons TCP reset was sent for easier tracing.

   - Add SMC ringbufs usage statistics.

  Drivers:

   - Handle netconsole setup failures more gracefully, don't fail
     loading, retain the specified target as disabled.

   - Extend bonding's IPsec offload pass thru capabilities (ESN, stats).

  Filtering:

   - Add TCP_BPF_SOCK_OPS_CB_FLAGS to bpf_*sockopt() to address the case
     when long-lived sockets miss a chance to set additional callbacks
     if a sockops program was not attached early in their lifetime.

   - Support using BPF skb helpers in tracepoints.

   - Conntrack Netlink: support CTA_FILTER for flush.

   - Improve SCTP support in nfnetlink_queue.

   - Improve performance of large nftables flush transactions.

  Things we sprinkled into general kernel code:

   - selftests: support setting an "interpreter" for script files; make
     it easy to run as separate cases tests where one "interpreter" is
     fed various test descriptions (in our case packet sequences).

  Driver API:

   - Extend core and ethtool APIs to support many PHYs connected to a
     single interface (PHY topologies).

   - Extend cable diagnostics to specify whether Time Domain
     Reflectometry (TDR) or Active Link Cable Diagnostic (ALCD) was
     used.

   - Add library for implementing MAC-PHY Ethernet drivers for SPI
     devices compatible with Open Alliance 10BASE-T1x MAC-PHY Serial
     Interface (TC6) standard.

   - Add helpers to the PHY framework, for PHYs following the Open
     Alliance standards:
       - 1000BaseT1 link settings
       - cable test and diagnostics

   - Support listing / dumping all allocated RSS contexts.

   - Add configuration for frequency Embedded SYNC in DPLL, which
     magically embeds sync pulses into Ethernet signaling.

  Device drivers:

   - Ethernet high-speed NICs:
      - Broadcom (bnxt):
         - use better FW APIs for queue reset
         - support QOS and TPID settings for the SR-IOV VLAN
         - support dynamic MSI-X allocation
      - Intel (100G, ice, idpf):
         - ice: support PCIe subfunctions
         - iavf: add support for TC U32 filters on VFs
         - ice: support Embedded SYNC in DPLL
      - nVidia/Mellanox (mlx5):
         - support HW managed steering tables
         - support PCIe PTM cross timestamping
      - AMD/Pensando:
         - ionic: use page_pool to increase Rx performance
      - Cisco (enic):
         - report per-queue statistics

   - Ethernet virtual:
      - Microsoft vNIC:
         - mana: support configuring ring length
         - netvsc: enable more channels on systems with many CPUs
      - IBM veth:
         - optimize polling to improve TCP_RR performance
         - optimize performance of Tx handling
      - VirtIO net:
         - synchronize the operstate with the admin state to allow a
           lower virtio-net to propagate the link status to an upper
           device like macvlan

   - Ethernet NICs consumer, and embedded:
      - Add driver for Realtek automotive PCIe devices (RTL9054,
        RTL9068, RTL9072, RTL9075, RTL9068, RTL9071)
      - Add driver for Microchip LAN8650/1 10BASE-T1S MAC-PHY.
      - Microchip:
         - lan743x: use phylink - support WOL, EEE, pause, link settings
         - add Wake-on-LAN support for KSZ87xx family
         - add KSZ8895/KSZ8864 switch support
         - factor out FDMA code and use it in sparx5 and lan966x
           (including DCB support in both)
      - Synopsys (stmmac):
         - support frame preemption (configured using TC and ethtool)
         - support Loongson DWMAC (GMAC v3.73)
         - support RockChips RK3576 DWMAC
      - TI:
         - am65-cpsw: add multi queue RX support
         - icssg-prueth: HSR offload support
      - Cadence (macb):
         - enable software (hrtimer based) IRQ coalescing by default
      - Xilinx (axinet):
         - expose HW statistics
         - improve multicast filtering
         - relax Rx checksum offload constraints
      - MediaTek:
         - mt7530: add EN7581 support
      - Aspeed (ftgmac100):
         - report link speed and duplex
      - Intel:
         - igc: add mqprio offload
         - igc: report EEE configuration
      - RealTek (r8169):
         - add support for RTL8126A rev.b
      - Vitesse (vsc73xx):
         - implement FDB add/del/dump operations
      - Freescale (fs_enet):
         - use phylink

   - Ethernet PHYs:
      - vitesse: implement downshift and MDI-X in vsc73xx PHYs
      - microchip: support LAN887x, supporting IEEE 802.3bw (100BASE-T1)
        and IEEE 802.3bp (1000BASE-T1) specifications
      - add Applied Micro QT2025 PHY driver (in Rust)
      - add Motorcomm yt8821 2.5G Ethernet PHY driver

   - CAN:
      - add driver for Rockchip RK3568 CAN-FD controller
      - flexcan: add wakeup support for imx95
      - kvaser_usb: set hardware timestamp on transmitted packets

   - WiFi:
      - mac80211/cfg80211:
         - EHT rate support in AQL airtime fairness
         - handle DFS (radar detection) per link in Multi-Link Operation
      - RealTek (rtw89):
         - support RTL8852BT and 8852BE-VT (WiFi 6)
         - support hardware rfkill
         - support HW encryption in unicast management frames
         - support Wake-on-WLAN with supported network detection
      - RealTek (rtw89):
         - improve Rx performance by using USB frame aggregation
         - support USB 3 with RTL8822CU/RTL8822BU
      - Intel (iwlwifi/mvm):
         - offload RLC/SMPS functionality to firmware
      - Marvell (mwifiex):
         - add host based MLME to enable WPA3

   - Bluetooth:
      - add support for Amlogic HCI UART protocol
      - add support for ISO data/packets to Intel and NXP drivers"

* tag 'net-next-6.12' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next: (1303 commits)
  net/mlx5: HWS, check the correct variable in hws_send_ring_alloc_sq()
  netfilter: nft_socket: Fix a NULL vs IS_ERR() bug in nft_socket_cgroup_subtree_level()
  ice: Fix a NULL vs IS_ERR() check in probe()
  ice: Fix a couple NULL vs IS_ERR() bugs
  net: ethernet: fs_enet: Make the per clock optional
  net: ti: icssg-prueth: Add multicast filtering support in HSR mode
  net: ti: icssg-prueth: Enable HSR Tx duplication, Tx Tag and Rx Tag offload
  net: ti: icssg-prueth: Add support for HSR frame forward offload
  net: ti: icssg-prueth: Stop hardcoding def_inc
  net: ti: icss-iep: Move icss_iep structure
  net: ibm: emac: get rid of wol_irq
  net: ibm: emac: remove all waiting code
  net: ibm: emac: replace of_get_property
  net: ibm: emac: use netdev's phydev directly
  net: ibm: emac: use devm for register_netdev
  net: ibm: emac: remove mii_bus with devm
  net: ibm: emac: use devm for of_iomap
  net: ibm: emac: manage emac_irq with devm
  net: ibm: emac: use devm for alloc_etherdev
  octeontx2-af: debugfs: Add Channel info to RPM map
  ...

1 files changed, 75 insertions, 97 deletions

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 4c9d9badd698..b619a3ec245b 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -39,7 +39,8 @@
 #include "ibmveth.h"
 
 static irqreturn_t ibmveth_interrupt(int irq, void *dev_instance);
-static void ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter);
+static void ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,
+				       bool reuse);
 static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev);
 
 static struct kobj_type ktype_veth_pool;
@@ -226,6 +227,16 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 	for (i = 0; i < count; ++i) {
 		union ibmveth_buf_desc desc;
 
+		free_index = pool->consumer_index;
+		index = pool->free_map[free_index];
+		skb = NULL;
+
+		BUG_ON(index == IBM_VETH_INVALID_MAP);
+
+		/* are we allocating a new buffer or recycling an old one */
+		if (pool->skbuff[index])
+			goto reuse;
+
 		skb = netdev_alloc_skb(adapter->netdev, pool->buff_size);
 
 		if (!skb) {
@@ -235,46 +246,46 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 			break;
 		}
 
-		free_index = pool->consumer_index;
-		pool->consumer_index++;
-		if (pool->consumer_index >= pool->size)
-			pool->consumer_index = 0;
-		index = pool->free_map[free_index];
-
-		BUG_ON(index == IBM_VETH_INVALID_MAP);
-		BUG_ON(pool->skbuff[index] != NULL);
-
 		dma_addr = dma_map_single(&adapter->vdev->dev, skb->data,
 				pool->buff_size, DMA_FROM_DEVICE);
 
 		if (dma_mapping_error(&adapter->vdev->dev, dma_addr))
 			goto failure;
 
-		pool->free_map[free_index] = IBM_VETH_INVALID_MAP;
 		pool->dma_addr[index] = dma_addr;
 		pool->skbuff[index] = skb;
 
-		correlator = ((u64)pool->index << 32) | index;
-		*(u64 *)skb->data = correlator;
-
-		desc.fields.flags_len = IBMVETH_BUF_VALID | pool->buff_size;
-		desc.fields.address = dma_addr;
-
 		if (rx_flush) {
 			unsigned int len = min(pool->buff_size,
-						adapter->netdev->mtu +
-						IBMVETH_BUFF_OH);
+					       adapter->netdev->mtu +
+					       IBMVETH_BUFF_OH);
 			ibmveth_flush_buffer(skb->data, len);
 		}
+reuse:
+		dma_addr = pool->dma_addr[index];
+		desc.fields.flags_len = IBMVETH_BUF_VALID | pool->buff_size;
+		desc.fields.address = dma_addr;
+
+		correlator = ((u64)pool->index << 32) | index;
+		*(u64 *)pool->skbuff[index]->data = correlator;
+
 		lpar_rc = h_add_logical_lan_buffer(adapter->vdev->unit_address,
 						   desc.desc);
 
 		if (lpar_rc != H_SUCCESS) {
+			netdev_warn(adapter->netdev,
+				    "%sadd_logical_lan failed %lu\n",
+				    skb ? "" : "When recycling: ", lpar_rc);
 			goto failure;
-		} else {
-			buffers_added++;
-			adapter->replenish_add_buff_success++;
 		}
+
+		pool->free_map[free_index] = IBM_VETH_INVALID_MAP;
+		pool->consumer_index++;
+		if (pool->consumer_index >= pool->size)
+			pool->consumer_index = 0;
+
+		buffers_added++;
+		adapter->replenish_add_buff_success++;
 	}
 
 	mb();
@@ -282,17 +293,13 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 	return;
 
 failure:
-	pool->free_map[free_index] = index;
-	pool->skbuff[index] = NULL;
-	if (pool->consumer_index == 0)
-		pool->consumer_index = pool->size - 1;
-	else
-		pool->consumer_index--;
-	if (!dma_mapping_error(&adapter->vdev->dev, dma_addr))
+
+	if (dma_addr && !dma_mapping_error(&adapter->vdev->dev, dma_addr))
 		dma_unmap_single(&adapter->vdev->dev,
 		                 pool->dma_addr[index], pool->buff_size,
 		                 DMA_FROM_DEVICE);
-	dev_kfree_skb_any(skb);
+	dev_kfree_skb_any(pool->skbuff[index]);
+	pool->skbuff[index] = NULL;
 	adapter->replenish_add_buff_failure++;
 
 	mb();
@@ -365,7 +372,7 @@ static void ibmveth_free_buffer_pool(struct ibmveth_adapter *adapter,
 
 /* remove a buffer from a pool */
 static void ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,
-					    u64 correlator)
+					    u64 correlator, bool reuse)
 {
 	unsigned int pool  = correlator >> 32;
 	unsigned int index = correlator & 0xffffffffUL;
@@ -376,15 +383,23 @@ static void ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,
 	BUG_ON(index >= adapter->rx_buff_pool[pool].size);
 
 	skb = adapter->rx_buff_pool[pool].skbuff[index];
-
 	BUG_ON(skb == NULL);
 
-	adapter->rx_buff_pool[pool].skbuff[index] = NULL;
+	/* if we are going to reuse the buffer then keep the pointers around
+	 * but mark index as available. replenish will see the skb pointer and
+	 * assume it is to be recycled.
+	 */
+	if (!reuse) {
+		/* remove the skb pointer to mark free. actual freeing is done
+		 * by upper level networking after gro_recieve
+		 */
+		adapter->rx_buff_pool[pool].skbuff[index] = NULL;
 
-	dma_unmap_single(&adapter->vdev->dev,
-			 adapter->rx_buff_pool[pool].dma_addr[index],
-			 adapter->rx_buff_pool[pool].buff_size,
-			 DMA_FROM_DEVICE);
+		dma_unmap_single(&adapter->vdev->dev,
+				 adapter->rx_buff_pool[pool].dma_addr[index],
+				 adapter->rx_buff_pool[pool].buff_size,
+				 DMA_FROM_DEVICE);
+	}
 
 	free_index = adapter->rx_buff_pool[pool].producer_index;
 	adapter->rx_buff_pool[pool].producer_index++;
@@ -411,51 +426,13 @@ static inline struct sk_buff *ibmveth_rxq_get_buffer(struct ibmveth_adapter *ada
 	return adapter->rx_buff_pool[pool].skbuff[index];
 }
 
-/* recycle the current buffer on the rx queue */
-static int ibmveth_rxq_recycle_buffer(struct ibmveth_adapter *adapter)
+static void ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,
+				       bool reuse)
 {
-	u32 q_index = adapter->rx_queue.index;
-	u64 correlator = adapter->rx_queue.queue_addr[q_index].correlator;
-	unsigned int pool = correlator >> 32;
-	unsigned int index = correlator & 0xffffffffUL;
-	union ibmveth_buf_desc desc;
-	unsigned long lpar_rc;
-	int ret = 1;
-
-	BUG_ON(pool >= IBMVETH_NUM_BUFF_POOLS);
-	BUG_ON(index >= adapter->rx_buff_pool[pool].size);
-
-	if (!adapter->rx_buff_pool[pool].active) {
-		ibmveth_rxq_harvest_buffer(adapter);
-		ibmveth_free_buffer_pool(adapter, &adapter->rx_buff_pool[pool]);
-		goto out;
-	}
-
-	desc.fields.flags_len = IBMVETH_BUF_VALID |
-		adapter->rx_buff_pool[pool].buff_size;
-	desc.fields.address = adapter->rx_buff_pool[pool].dma_addr[index];
-
-	lpar_rc = h_add_logical_lan_buffer(adapter->vdev->unit_address, desc.desc);
-
-	if (lpar_rc != H_SUCCESS) {
-		netdev_dbg(adapter->netdev, "h_add_logical_lan_buffer failed "
-			   "during recycle rc=%ld", lpar_rc);
-		ibmveth_remove_buffer_from_pool(adapter, adapter->rx_queue.queue_addr[adapter->rx_queue.index].correlator);
-		ret = 0;
-	}
-
-	if (++adapter->rx_queue.index == adapter->rx_queue.num_slots) {
-		adapter->rx_queue.index = 0;
-		adapter->rx_queue.toggle = !adapter->rx_queue.toggle;
-	}
-
-out:
-	return ret;
-}
+	u64 cor;
 
-static void ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter)
-{
-	ibmveth_remove_buffer_from_pool(adapter, adapter->rx_queue.queue_addr[adapter->rx_queue.index].correlator);
+	cor = adapter->rx_queue.queue_addr[adapter->rx_queue.index].correlator;
+	ibmveth_remove_buffer_from_pool(adapter, cor, reuse);
 
 	if (++adapter->rx_queue.index == adapter->rx_queue.num_slots) {
 		adapter->rx_queue.index = 0;
@@ -1337,6 +1314,7 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
 	unsigned long lpar_rc;
 	u16 mss = 0;
 
+restart_poll:
 	while (frames_processed < budget) {
 		if (!ibmveth_rxq_pending_buffer(adapter))
 			break;
@@ -1346,7 +1324,7 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
 			wmb(); /* suggested by larson1 */
 			adapter->rx_invalid_buffer++;
 			netdev_dbg(netdev, "recycling invalid buffer\n");
-			ibmveth_rxq_recycle_buffer(adapter);
+			ibmveth_rxq_harvest_buffer(adapter, true);
 		} else {
 			struct sk_buff *skb, *new_skb;
 			int length = ibmveth_rxq_frame_length(adapter);
@@ -1379,11 +1357,10 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
 				if (rx_flush)
 					ibmveth_flush_buffer(skb->data,
 						length + offset);
-				if (!ibmveth_rxq_recycle_buffer(adapter))
-					kfree_skb(skb);
+				ibmveth_rxq_harvest_buffer(adapter, true);
 				skb = new_skb;
 			} else {
-				ibmveth_rxq_harvest_buffer(adapter);
+				ibmveth_rxq_harvest_buffer(adapter, false);
 				skb_reserve(skb, offset);
 			}
 
@@ -1420,24 +1397,25 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
 
 	ibmveth_replenish_task(adapter);
 
-	if (frames_processed < budget) {
-		napi_complete_done(napi, frames_processed);
+	if (frames_processed == budget)
+		goto out;
 
-		/* We think we are done - reenable interrupts,
-		 * then check once more to make sure we are done.
-		 */
-		lpar_rc = h_vio_signal(adapter->vdev->unit_address,
-				       VIO_IRQ_ENABLE);
+	if (!napi_complete_done(napi, frames_processed))
+		goto out;
 
-		BUG_ON(lpar_rc != H_SUCCESS);
+	/* We think we are done - reenable interrupts,
+	 * then check once more to make sure we are done.
+	 */
+	lpar_rc = h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_ENABLE);
+	BUG_ON(lpar_rc != H_SUCCESS);
 
-		if (ibmveth_rxq_pending_buffer(adapter) &&
-		    napi_schedule(napi)) {
-			lpar_rc = h_vio_signal(adapter->vdev->unit_address,
-					       VIO_IRQ_DISABLE);
-		}
+	if (ibmveth_rxq_pending_buffer(adapter) && napi_schedule(napi)) {
+		lpar_rc = h_vio_signal(adapter->vdev->unit_address,
+				       VIO_IRQ_DISABLE);
+		goto restart_poll;
 	}
 
+out:
 	return frames_processed;
 }