Jul 20 13:16:45.687150 ip-10-0-132-221 systemd[1]: Starting Kubernetes Kubelet... Jul 20 13:16:46.154763 ip-10-0-132-221 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 13:16:46.154763 ip-10-0-132-221 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 20 13:16:46.154763 ip-10-0-132-221 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 13:16:46.154763 ip-10-0-132-221 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 20 13:16:46.154763 ip-10-0-132-221 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 13:16:46.154763 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.044883 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 20 13:16:46.154763 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048511 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:46.154763 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048522 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:46.154763 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048527 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:46.154763 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048530 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:46.154763 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048532 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:46.154763 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048535 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:46.154763 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048538 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:46.154763 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048540 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:46.107672 ip-10-0-132-221 systemd[1]: Started Kubernetes Kubelet. Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048543 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048546 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048549 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048552 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048555 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048557 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048560 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048562 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048565 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048568 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048571 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048573 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048576 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048579 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048582 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048585 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048587 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048590 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048592 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:46.177203 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048595 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048598 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048600 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048603 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048605 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048608 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048611 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048614 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048618 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048622 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048624 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048627 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048629 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048632 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048634 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048637 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048640 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048642 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048645 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048647 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:46.179413 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048650 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048652 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048656 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048659 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048662 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048664 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048667 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048669 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048672 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048676 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048680 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048683 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048686 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048689 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048691 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048694 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048696 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048699 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048701 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:46.184725 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048704 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048706 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048709 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048712 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048714 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048717 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048720 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048723 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048726 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048730 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048732 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048735 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048737 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048740 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048743 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048761 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048765 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048768 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048771 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048774 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:46.186928 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048777 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048779 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048782 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048785 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048787 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048791 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048795 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048797 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048801 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048803 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048806 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048809 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.048812 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049503 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049510 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049513 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049516 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049519 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049522 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:46.188894 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049525 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049528 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049531 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049534 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049537 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049540 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049543 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049546 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049548 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049551 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049554 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049556 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049559 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049562 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049564 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049566 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049569 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049571 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049574 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:46.189815 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049576 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049579 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049582 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049586 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049589 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049592 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049595 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049598 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049601 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049604 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049606 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049609 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049612 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049615 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049617 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049620 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049624 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049627 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049630 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049633 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:46.194641 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049636 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049638 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049641 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049643 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049645 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049648 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049650 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049653 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049655 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049658 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049660 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049663 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049665 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049668 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049670 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049673 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049675 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049680 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049684 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:46.195271 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049687 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049690 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049693 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049695 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049698 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049700 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049703 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049706 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049708 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049711 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049713 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049716 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049719 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049723 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049726 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049729 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049732 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049735 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049737 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:46.196523 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049740 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049742 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049745 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049747 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049750 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049752 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049755 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049757 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049760 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049762 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049764 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049767 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049769 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049772 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049775 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.049777 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050124 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050129 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050132 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050134 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050137 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:46.197165 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050140 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050143 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050146 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050149 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050152 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050154 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050157 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050159 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050162 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050165 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050167 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050170 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050172 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050175 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050177 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050180 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050182 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050185 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050187 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050191 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:46.198209 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050194 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050197 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050199 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050202 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050205 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050207 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050213 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050215 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050218 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050220 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050223 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050226 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050230 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050233 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050236 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050239 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050245 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050248 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050251 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:46.200790 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050254 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050257 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050260 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050262 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050281 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050285 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050288 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050291 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050294 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050297 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050299 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050302 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050305 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050307 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050310 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050312 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050315 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050318 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050320 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:46.201358 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050324 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050327 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050329 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050332 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050334 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050337 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050339 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050342 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050344 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050347 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050350 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050352 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050356 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050359 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050361 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050363 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050366 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050370 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050372 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:46.201910 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050375 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050377 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050379 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050382 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050385 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050387 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050390 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050392 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050395 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050397 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050400 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050402 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050405 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050408 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050411 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050413 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.050416 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050486 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050501 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050509 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050516 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 20 13:16:46.274885 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050522 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050527 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050531 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050535 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050540 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050543 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050546 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050549 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050552 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050557 2575 flags.go:64] FLAG: --cgroup-root="" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050560 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050563 2575 flags.go:64] FLAG: --client-ca-file="" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050566 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050569 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050573 2575 flags.go:64] FLAG: --cluster-domain="" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050576 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050579 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050582 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050585 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050589 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050592 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050595 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050598 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050601 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 20 13:16:46.295130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050605 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050609 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050612 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050615 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050618 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050622 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050625 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050628 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050631 2575 flags.go:64] FLAG: --enable-server="true" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050634 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050638 2575 flags.go:64] FLAG: --event-burst="100" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050642 2575 flags.go:64] FLAG: --event-qps="50" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050645 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050648 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050651 2575 flags.go:64] FLAG: --eviction-hard="" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050655 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050657 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050661 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050665 2575 flags.go:64] FLAG: --eviction-soft="" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050667 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050670 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050673 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050676 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050679 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050682 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 20 13:16:46.296318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050684 2575 flags.go:64] FLAG: --feature-gates="" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050688 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050691 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050694 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050697 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050700 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050703 2575 flags.go:64] FLAG: --help="false" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050707 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050710 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050713 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050716 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050719 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050722 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050725 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050728 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050731 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050734 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050736 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050740 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050742 2575 flags.go:64] FLAG: --kube-reserved="" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050745 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050748 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050751 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050754 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 20 13:16:46.297427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050758 2575 flags.go:64] FLAG: --lock-file="" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050761 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050764 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050767 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050773 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050775 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050778 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050781 2575 flags.go:64] FLAG: --logging-format="text" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050784 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050787 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050790 2575 flags.go:64] FLAG: --manifest-url="" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050793 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050797 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050800 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050804 2575 flags.go:64] FLAG: --max-pods="110" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050808 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050811 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050814 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050817 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050820 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050823 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050826 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050833 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050835 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050838 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 20 13:16:46.298120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050841 2575 flags.go:64] FLAG: --pod-cidr="" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050844 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050849 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050852 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050855 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050859 2575 flags.go:64] FLAG: --port="10250" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050862 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050866 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c8b136cbb5c8294f" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050881 2575 flags.go:64] FLAG: --qos-reserved="" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050885 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050888 2575 flags.go:64] FLAG: --register-node="true" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050891 2575 flags.go:64] FLAG: --register-with-taints="" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050894 2575 flags.go:64] FLAG: --registry-burst="10" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050897 2575 flags.go:64] FLAG: --registry-qps="5" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050900 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050903 2575 flags.go:64] FLAG: --reserved-memory="" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050909 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050912 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050915 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050918 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050921 2575 flags.go:64] FLAG: --runonce="false" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050924 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050927 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050931 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050934 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 20 13:16:46.298921 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050937 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050940 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050943 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050946 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050948 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050952 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050955 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050957 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050960 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050967 2575 flags.go:64] FLAG: --system-cgroups="" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050970 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050976 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050979 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050982 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050988 2575 flags.go:64] FLAG: --tls-min-version="" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050991 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050994 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050996 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.050999 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.051002 2575 flags.go:64] FLAG: --v="2" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.051006 2575 flags.go:64] FLAG: --version="false" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.051010 2575 flags.go:64] FLAG: --vmodule="" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.051013 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.051016 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051138 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:46.299931 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051142 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051146 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051149 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051152 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051155 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051158 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051161 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051164 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051166 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051169 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051172 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051174 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051177 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051179 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051182 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051184 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051187 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051189 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051192 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051194 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:46.300714 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051197 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051201 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051204 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051206 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051209 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051212 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051214 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051217 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051219 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051222 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051224 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051227 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051229 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051232 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051234 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051237 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051239 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051243 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051246 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051249 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:46.301342 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051251 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051254 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051256 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051259 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051262 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051264 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051266 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051269 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051273 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051276 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051279 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051282 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051285 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051289 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051293 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051295 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051298 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051301 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051303 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051307 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:46.301997 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051309 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051313 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051317 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051319 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051322 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051324 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051327 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051330 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051332 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051337 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051339 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051342 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051344 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051347 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051350 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051353 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051355 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051358 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051360 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:46.302616 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051362 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051365 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051367 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051370 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051372 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051375 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051379 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051381 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051384 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051387 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051391 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051394 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051397 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051400 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051402 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051405 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051408 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051411 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:46.303412 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.051414 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.052013 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.059270 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.059286 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.059304 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.059313 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059364 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059370 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059373 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059375 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059378 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059381 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059384 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059387 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059389 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:46.304009 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059392 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059394 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059397 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059399 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059402 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059405 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059407 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059410 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059412 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059416 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059418 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059421 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059424 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059427 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059434 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059438 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059440 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059443 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059445 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059448 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:46.304513 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059451 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059453 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059456 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059459 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059461 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059463 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059466 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059469 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059472 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059474 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059477 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059479 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059482 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059484 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059487 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059490 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059493 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059496 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059499 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059501 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:46.305132 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059504 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059506 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059509 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059511 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059514 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059517 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059519 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059524 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059527 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059531 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059535 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059538 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059541 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059544 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059547 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059550 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059553 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059555 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059558 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059560 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:46.305707 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059563 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059566 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059568 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059571 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059573 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059577 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059580 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059583 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059586 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059588 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059591 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059594 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059597 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059599 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059602 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059604 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059607 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059609 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059612 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:46.306314 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059616 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059618 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059621 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059623 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059626 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059628 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059631 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059633 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059637 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059640 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059644 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.059649 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059761 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059766 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059769 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 13:16:46.306833 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059772 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059775 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059778 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059780 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059783 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059785 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059788 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059791 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059794 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059799 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059803 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059806 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059808 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059811 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059814 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059816 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059819 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059823 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059836 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059839 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 13:16:46.307346 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059842 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059844 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059847 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059850 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059852 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059857 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059861 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059863 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059866 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059883 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059887 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059891 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059895 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059898 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059900 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059903 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059906 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059909 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059912 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059914 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 13:16:46.308748 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059917 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059919 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059923 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059926 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059928 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059932 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059936 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059939 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059942 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059946 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059949 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059952 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059955 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059958 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059961 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059964 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059966 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059970 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059972 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 13:16:46.309384 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059975 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059977 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059980 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059982 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059985 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059987 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059990 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059992 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059995 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.059997 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060000 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060002 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060005 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060008 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060010 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060012 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060016 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060019 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060021 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060024 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 13:16:46.311057 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060026 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060029 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060032 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060035 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060037 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060040 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060042 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060045 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060047 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060050 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060054 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060056 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060059 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060061 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060064 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060066 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 13:16:46.311661 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:46.060069 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 13:16:46.396473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.060074 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 13:16:46.396473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.060170 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 20 13:16:46.396473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.063405 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 20 13:16:46.396473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.064554 2575 server.go:1019] "Starting client certificate rotation" Jul 20 13:16:46.396473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.064666 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 13:16:46.396473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.064744 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 13:16:46.396473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.066813 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 13:16:46.396473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.067896 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 13:16:46.396473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.078063 2575 log.go:25] "Validated CRI v1 runtime API" Jul 20 13:16:46.396473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.080037 2575 log.go:25] "Validated CRI v1 image API" Jul 20 13:16:46.396473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.080806 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 20 13:16:46.396473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.082836 2575 fs.go:135] Filesystem UUIDs: map[3014c756-3aa9-4d17-b1fa-d898a8423b0b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e18b7ce5-e689-41c8-9c28-d782a9286fcf:/dev/nvme0n1p3] Jul 20 13:16:46.396909 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.082849 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 20 13:16:46.397094 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.089070 2575 manager.go:216] Machine: {Timestamp:2026-07-20 13:16:46.087819593 +0000 UTC m=+0.296215734 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100341 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af3dc42ac97af2958ef8f72da650c SystemUUID:ec2af3dc-42ac-97af-2958-ef8f72da650c BootID:962800e8-249a-4a49-914a-3349e52b3e15 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:30:42:53:0e:25 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:30:42:53:0e:25 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:1c:e4:12:fb:c3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 20 13:16:46.397094 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.089168 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 20 13:16:46.397094 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.089241 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 20 13:16:46.397094 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.091744 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 20 13:16:46.397094 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.091775 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-221.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 20 13:16:46.397094 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.092423 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 13:16:46.397094 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.093770 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.093783 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.093790 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.094366 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.095550 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.098023 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.098134 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.098160 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.098169 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.098181 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.098191 2575 kubelet.go:398] "Adding apiserver pod source" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.098199 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.098496 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.098511 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.100271 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.102039 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.102067 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.103038 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.103051 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.103057 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.103062 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.103070 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.103075 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.103086 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.103093 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.103100 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.103106 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.103114 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.103120 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.103817 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.103836 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.106895 2575 server.go:1268] "Started kubelet" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.107124 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.107203 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.107415 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.108897 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.108920 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.108957 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.108976 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-221.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 20 13:16:46.397523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.109257 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.112233 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.114865 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.114892 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.115384 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.115409 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.115413 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.115436 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.115505 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.115514 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.115865 2575 factory.go:55] Registering systemd factory Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.115897 2575 factory.go:223] Registration of the systemd container factory successfully Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.116083 2575 factory.go:153] Registering CRI-O factory Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.116100 2575 factory.go:223] Registration of the crio container factory successfully Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.116146 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.116163 2575 factory.go:103] Registering Raw factory Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.116173 2575 manager.go:1196] Started watching for new ooms in manager Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.116497 2575 manager.go:349] Starting recovery of all containers Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.116006 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-221.ec2.internal.18c401de2db64310 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-221.ec2.internal,UID:ip-10-0-132-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-221.ec2.internal,},FirstTimestamp:2026-07-20 13:16:46.106845968 +0000 UTC m=+0.315242106,LastTimestamp:2026-07-20 13:16:46.106845968 +0000 UTC m=+0.315242106,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-221.ec2.internal,}" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.126560 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2c6gb" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.128449 2575 manager.go:354] Recovery completed Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.130939 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2c6gb" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.131198 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.131350 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.132725 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.134484 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.134512 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.134525 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.134981 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.134992 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.135004 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.135010 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 20 13:16:46.398601 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.135676 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.135698 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.135708 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.135986 2575 policy_none.go:49] "None policy: Start" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.136003 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.136018 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.136908 2575 policy_none.go:47] "Start" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.216083 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.232508 2575 manager.go:337] "Starting Device Plugin manager" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.232566 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.232576 2575 server.go:87] "Starting device plugin registration server" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.232820 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.232833 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.232926 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.232999 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.233006 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.233715 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.233769 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.301331 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.302524 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.302564 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.302592 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.302632 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.305030 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.333128 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.333889 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.333913 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.333924 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.333948 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.345276 2575 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-221.ec2.internal\" not found" node="ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.346104 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.346122 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-221.ec2.internal\": node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:46.400316 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.382689 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:46.403563 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.402700 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal"] Jul 20 13:16:46.403563 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.402763 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.413344 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.413361 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.413370 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.415414 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.415526 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.415558 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.416085 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.416103 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.416116 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.416132 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.416152 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.416162 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.417432 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ba135fd790f8de9f90cf6e2924b3971-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal\" (UID: \"4ba135fd790f8de9f90cf6e2924b3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.417453 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ba135fd790f8de9f90cf6e2924b3971-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal\" (UID: \"4ba135fd790f8de9f90cf6e2924b3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.417469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0544e6927bc4266c71d4da456a570889-config\") pod \"kube-apiserver-proxy-ip-10-0-132-221.ec2.internal\" (UID: \"0544e6927bc4266c71d4da456a570889\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.418182 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.418213 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.418813 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientMemory" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.418843 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.418857 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientPID" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.438380 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-221.ec2.internal\" not found" node="ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.442617 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-221.ec2.internal\" not found" node="ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.483055 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:46.552680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.517757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ba135fd790f8de9f90cf6e2924b3971-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal\" (UID: \"4ba135fd790f8de9f90cf6e2924b3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.553817 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.517775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ba135fd790f8de9f90cf6e2924b3971-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal\" (UID: \"4ba135fd790f8de9f90cf6e2924b3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.553817 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.517790 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0544e6927bc4266c71d4da456a570889-config\") pod \"kube-apiserver-proxy-ip-10-0-132-221.ec2.internal\" (UID: \"0544e6927bc4266c71d4da456a570889\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.553817 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.517815 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0544e6927bc4266c71d4da456a570889-config\") pod \"kube-apiserver-proxy-ip-10-0-132-221.ec2.internal\" (UID: \"0544e6927bc4266c71d4da456a570889\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.553817 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.517832 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ba135fd790f8de9f90cf6e2924b3971-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal\" (UID: \"4ba135fd790f8de9f90cf6e2924b3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.553817 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.517843 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ba135fd790f8de9f90cf6e2924b3971-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal\" (UID: \"4ba135fd790f8de9f90cf6e2924b3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.672141 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.583522 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:46.741750 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.683820 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:46.741750 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.741371 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.801141 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:46.786536 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" Jul 20 13:16:46.801141 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.786538 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:46.934586 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.887410 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:46.991679 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:46.987700 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:47.064102 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:47.064064 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 20 13:16:47.064181 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:47.064163 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 13:16:47.088291 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:47.088264 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:47.134694 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:47.134667 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 13:11:45 +0000 UTC" deadline="2028-05-02 13:46:27.986547524 +0000 UTC" Jul 20 13:16:47.134694 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:47.134692 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15648h29m40.851857882s" Jul 20 13:16:47.188926 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:47.188907 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:47.232428 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:47.232409 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:47.289859 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:47.289839 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:47.317119 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:47.317064 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba135fd790f8de9f90cf6e2924b3971.slice/crio-beca719e4f78383861157427c9309bbf7181968f51157700f8768157d036447c WatchSource:0}: Error finding container beca719e4f78383861157427c9309bbf7181968f51157700f8768157d036447c: Status 404 returned error can't find the container with id beca719e4f78383861157427c9309bbf7181968f51157700f8768157d036447c Jul 20 13:16:47.320486 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:47.320473 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 13:16:47.341422 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:47.341404 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0544e6927bc4266c71d4da456a570889.slice/crio-f9432fdbdd6a476ffcbdf52044345dbfb09c079d43966e9ec4fe17a0efb0719c WatchSource:0}: Error finding container f9432fdbdd6a476ffcbdf52044345dbfb09c079d43966e9ec4fe17a0efb0719c: Status 404 returned error can't find the container with id f9432fdbdd6a476ffcbdf52044345dbfb09c079d43966e9ec4fe17a0efb0719c Jul 20 13:16:47.380675 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:47.380655 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:47.390827 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:47.390810 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:47.491118 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:47.491102 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 20 13:16:47.573234 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:47.573187 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:47.616145 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:47.616126 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 20 13:16:47.629538 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:47.629524 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 13:16:47.630353 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:47.630343 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" Jul 20 13:16:47.642108 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:47.642096 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 13:16:48.098422 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.098370 2575 apiserver.go:52] "Watching apiserver" Jul 20 13:16:48.107026 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.107002 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 20 13:16:48.107488 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.107468 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-7sjzn","kube-system/konnectivity-agent-lpksh","openshift-dns/node-resolver-h8gtl","openshift-image-registry/node-ca-blhwm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal","openshift-multus/multus-2l6tt","openshift-multus/multus-additional-cni-plugins-pstqq","openshift-network-diagnostics/network-check-target-wgvvv","kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx","openshift-cluster-node-tuning-operator/tuned-mts6s","openshift-multus/network-metrics-daemon-svhpp","openshift-network-operator/iptables-alerter-r7fcq","openshift-ovn-kubernetes/ovnkube-node-khqq5"] Jul 20 13:16:48.109770 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.109752 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.112582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.112385 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.114252 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.114229 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 20 13:16:48.114841 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.114540 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.115165 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.115148 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:16:48.116125 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.115368 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nlwwv\"" Jul 20 13:16:48.116125 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.115467 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 20 13:16:48.116125 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.115780 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 20 13:16:48.116423 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.116408 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kskvr\"" Jul 20 13:16:48.116934 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.116917 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-blhwm" Jul 20 13:16:48.117152 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.117133 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 20 13:16:48.117680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.117362 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 20 13:16:48.117680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.117669 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 20 13:16:48.118903 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.118887 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 20 13:16:48.119106 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.119090 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h8gtl" Jul 20 13:16:48.121006 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.120989 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:48.121098 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.121064 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:16:48.122918 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.122899 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 20 13:16:48.123065 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.123041 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 20 13:16:48.123345 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.123262 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:16:48.123345 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.123315 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:16:48.124244 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.124228 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tznzm\"" Jul 20 13:16:48.124330 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.124273 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 20 13:16:48.124519 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.124505 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 20 13:16:48.124588 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.124527 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tjmd5\"" Jul 20 13:16:48.125053 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.125035 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 20 13:16:48.125283 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.125268 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kfznd\"" Jul 20 13:16:48.125608 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.125509 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.126941 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.126893 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 20 13:16:48.127489 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.127252 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lpksh" Jul 20 13:16:48.129813 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.129796 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:48.129912 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.129892 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:16:48.131181 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.131153 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8h8gr\"" Jul 20 13:16:48.131256 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.131200 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 20 13:16:48.131301 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.131270 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 20 13:16:48.131301 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.131282 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 20 13:16:48.131743 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.131382 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8bpdd\"" Jul 20 13:16:48.131743 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.131531 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 20 13:16:48.131743 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.131688 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 20 13:16:48.131960 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.131940 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r7fcq" Jul 20 13:16:48.134710 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.134692 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.135323 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.135305 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 13:16:48.135323 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.135318 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:16:48.135456 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.135329 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 20 13:16:48.135699 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.135681 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sq28r\"" Jul 20 13:16:48.136025 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.136009 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 20 13:16:48.137763 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.137598 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-slbzm\"" Jul 20 13:16:48.137763 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.137616 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 20 13:16:48.140241 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.140220 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 20 13:16:48.140328 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.140245 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 20 13:16:48.141323 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.141306 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 20 13:16:48.141395 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.141332 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 20 13:16:48.141449 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.141429 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 20 13:16:48.149374 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149353 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57brg\" (UniqueName: \"kubernetes.io/projected/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-kube-api-access-57brg\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.149475 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-systemd\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.149475 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-tuned\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.149475 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149426 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02bda514-3256-495f-8ce5-ae9e7fe50c71-host\") pod \"node-ca-blhwm\" (UID: \"02bda514-3256-495f-8ce5-ae9e7fe50c71\") " pod="openshift-image-registry/node-ca-blhwm" Jul 20 13:16:48.149475 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149445 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-cnibin\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.149475 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149464 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-cni-binary-copy\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.149745 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-multus-socket-dir-parent\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.149745 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-kubernetes\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.149745 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149527 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d0cf929-460c-4fb9-bfd5-c311d138d89d-cnibin\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.149745 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d0cf929-460c-4fb9-bfd5-c311d138d89d-cni-binary-copy\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.149745 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-os-release\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.149745 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149628 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-etc-kubernetes\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.149745 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149652 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-sysconfig\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.149745 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149676 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2lgd\" (UniqueName: \"kubernetes.io/projected/02bda514-3256-495f-8ce5-ae9e7fe50c71-kube-api-access-p2lgd\") pod \"node-ca-blhwm\" (UID: \"02bda514-3256-495f-8ce5-ae9e7fe50c71\") " pod="openshift-image-registry/node-ca-blhwm" Jul 20 13:16:48.149745 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-var-lib-cni-multus\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.149745 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-system-cni-dir\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.149745 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-var-lib-cni-bin\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.150179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-modprobe-d\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.150179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149825 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-sysctl-d\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.150179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-sys\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.150179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149889 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-lib-modules\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.150179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149910 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eac01c81-00c9-45f2-837a-95a88e53dc2b-tmp\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.150179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-multus-cni-dir\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.150179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149941 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-run-k8s-cni-cncf-io\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.150179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.149954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-var-lib-kubelet\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.150179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150004 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-host\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.150179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150020 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d0cf929-460c-4fb9-bfd5-c311d138d89d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.150179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150032 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-var-lib-kubelet\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.150179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150047 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-run-netns\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.150179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-run\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.150179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150098 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg45d\" (UniqueName: \"kubernetes.io/projected/eac01c81-00c9-45f2-837a-95a88e53dc2b-kube-api-access-pg45d\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.150785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d0cf929-460c-4fb9-bfd5-c311d138d89d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.150785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150173 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-hostroot\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.150785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150195 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-multus-conf-dir\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.150785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-multus-daemon-config\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.150785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150241 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d0cf929-460c-4fb9-bfd5-c311d138d89d-system-cni-dir\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.150785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150263 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cm9x\" (UniqueName: \"kubernetes.io/projected/2d0cf929-460c-4fb9-bfd5-c311d138d89d-kube-api-access-6cm9x\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.150785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150287 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02bda514-3256-495f-8ce5-ae9e7fe50c71-serviceca\") pod \"node-ca-blhwm\" (UID: \"02bda514-3256-495f-8ce5-ae9e7fe50c71\") " pod="openshift-image-registry/node-ca-blhwm" Jul 20 13:16:48.150785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-run-multus-certs\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.150785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150339 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-sysctl-conf\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.150785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d0cf929-460c-4fb9-bfd5-c311d138d89d-os-release\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.150785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.150398 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d0cf929-460c-4fb9-bfd5-c311d138d89d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.167140 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.166883 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-66j8r" Jul 20 13:16:48.174014 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.173993 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-66j8r" Jul 20 13:16:48.217269 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.217236 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 20 13:16:48.251463 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251444 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6f801ea8-fd39-44a0-a77e-2b2cfbbee71c-iptables-alerter-script\") pod \"iptables-alerter-r7fcq\" (UID: \"6f801ea8-fd39-44a0-a77e-2b2cfbbee71c\") " pod="openshift-network-operator/iptables-alerter-r7fcq" Jul 20 13:16:48.251576 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251474 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-var-lib-openvswitch\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.251576 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251502 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57brg\" (UniqueName: \"kubernetes.io/projected/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-kube-api-access-57brg\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.251576 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-systemd\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.251576 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-tuned\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.251714 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02bda514-3256-495f-8ce5-ae9e7fe50c71-host\") pod \"node-ca-blhwm\" (UID: \"02bda514-3256-495f-8ce5-ae9e7fe50c71\") " pod="openshift-image-registry/node-ca-blhwm" Jul 20 13:16:48.251714 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-slash\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.251714 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-run-systemd\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.251714 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251622 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-systemd\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.251714 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251638 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-log-socket\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.251714 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251697 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02bda514-3256-495f-8ce5-ae9e7fe50c71-host\") pod \"node-ca-blhwm\" (UID: \"02bda514-3256-495f-8ce5-ae9e7fe50c71\") " pod="openshift-image-registry/node-ca-blhwm" Jul 20 13:16:48.251951 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251807 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-multus-socket-dir-parent\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.251951 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-kubernetes\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.251951 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251862 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 20 13:16:48.251951 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251922 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-kubernetes\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.251951 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-sys-fs\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.252186 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53011d87-65ad-4a8b-a364-5e66e1cea999-agent-certs\") pod \"konnectivity-agent-lpksh\" (UID: \"53011d87-65ad-4a8b-a364-5e66e1cea999\") " pod="kube-system/konnectivity-agent-lpksh" Jul 20 13:16:48.252186 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvr2q\" (UniqueName: \"kubernetes.io/projected/6f801ea8-fd39-44a0-a77e-2b2cfbbee71c-kube-api-access-nvr2q\") pod \"iptables-alerter-r7fcq\" (UID: \"6f801ea8-fd39-44a0-a77e-2b2cfbbee71c\") " pod="openshift-network-operator/iptables-alerter-r7fcq" Jul 20 13:16:48.252186 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.251998 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-multus-socket-dir-parent\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.252186 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-os-release\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.252186 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252040 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdea0d3-dbcd-4be1-bddc-49679ef918f1-tmp-dir\") pod \"node-resolver-h8gtl\" (UID: \"7bdea0d3-dbcd-4be1-bddc-49679ef918f1\") " pod="openshift-dns/node-resolver-h8gtl" Jul 20 13:16:48.252186 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252056 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-os-release\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.252186 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252077 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdq64\" (UniqueName: \"kubernetes.io/projected/7bdea0d3-dbcd-4be1-bddc-49679ef918f1-kube-api-access-jdq64\") pod \"node-resolver-h8gtl\" (UID: \"7bdea0d3-dbcd-4be1-bddc-49679ef918f1\") " pod="openshift-dns/node-resolver-h8gtl" Jul 20 13:16:48.252186 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252109 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-device-dir\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.252186 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252130 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8290b97d-bb82-4de4-b590-b95bd307d4bc-env-overrides\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.252186 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-var-lib-cni-multus\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.252186 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-system-cni-dir\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-var-lib-cni-bin\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252234 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-var-lib-cni-multus\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-sysctl-d\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252258 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-system-cni-dir\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252262 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-sys\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-sys\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-var-lib-cni-bin\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252316 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-sysctl-d\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252326 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvzj9\" (UniqueName: \"kubernetes.io/projected/c805bf30-7720-4274-81e1-26d75eb7ae55-kube-api-access-pvzj9\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-kubelet\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-multus-cni-dir\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-run-k8s-cni-cncf-io\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252402 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-host\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252419 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:48.252582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252444 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-var-lib-kubelet\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252461 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-run-k8s-cni-cncf-io\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252470 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-multus-cni-dir\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-run-netns\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252492 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-host\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252496 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-run\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252506 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-var-lib-kubelet\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252530 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-run\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252536 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg45d\" (UniqueName: \"kubernetes.io/projected/eac01c81-00c9-45f2-837a-95a88e53dc2b-kube-api-access-pg45d\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252558 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-run-netns\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d0cf929-460c-4fb9-bfd5-c311d138d89d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252588 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-run-netns\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8290b97d-bb82-4de4-b590-b95bd307d4bc-ovn-node-metrics-cert\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-hostroot\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252658 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-multus-daemon-config\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.253206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252679 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-hostroot\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.253764 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d0cf929-460c-4fb9-bfd5-c311d138d89d-system-cni-dir\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.253764 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252713 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d0cf929-460c-4fb9-bfd5-c311d138d89d-system-cni-dir\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.253764 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252716 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d0cf929-460c-4fb9-bfd5-c311d138d89d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.253764 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cm9x\" (UniqueName: \"kubernetes.io/projected/2d0cf929-460c-4fb9-bfd5-c311d138d89d-kube-api-access-6cm9x\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.253764 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252767 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:48.253764 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-systemd-units\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.253764 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-run-multus-certs\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.253764 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d0cf929-460c-4fb9-bfd5-c311d138d89d-os-release\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.253764 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-run-ovn\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.253764 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-cni-bin\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.253764 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252916 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-host-run-multus-certs\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.253764 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252986 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d0cf929-460c-4fb9-bfd5-c311d138d89d-os-release\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.253764 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.252937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.253764 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.253383 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8290b97d-bb82-4de4-b590-b95bd307d4bc-ovnkube-script-lib\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.254362 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.253388 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-multus-daemon-config\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.254362 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.253402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmgjw\" (UniqueName: \"kubernetes.io/projected/8290b97d-bb82-4de4-b590-b95bd307d4bc-kube-api-access-kmgjw\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.254362 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.253426 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.254362 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.253448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6n2l\" (UniqueName: \"kubernetes.io/projected/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-kube-api-access-l6n2l\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.254362 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.253498 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8290b97d-bb82-4de4-b590-b95bd307d4bc-ovnkube-config\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.254362 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.253533 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-cnibin\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.254362 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.253558 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-cni-binary-copy\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.254362 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.253581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d0cf929-460c-4fb9-bfd5-c311d138d89d-cnibin\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.254362 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.253594 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-cnibin\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.254362 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.253655 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d0cf929-460c-4fb9-bfd5-c311d138d89d-cnibin\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.254362 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.253895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d0cf929-460c-4fb9-bfd5-c311d138d89d-cni-binary-copy\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.254362 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254229 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7bdea0d3-dbcd-4be1-bddc-49679ef918f1-hosts-file\") pod \"node-resolver-h8gtl\" (UID: \"7bdea0d3-dbcd-4be1-bddc-49679ef918f1\") " pod="openshift-dns/node-resolver-h8gtl" Jul 20 13:16:48.254362 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-etc-kubernetes\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.254362 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254290 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-sysconfig\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.254942 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254322 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2lgd\" (UniqueName: \"kubernetes.io/projected/02bda514-3256-495f-8ce5-ae9e7fe50c71-kube-api-access-p2lgd\") pod \"node-ca-blhwm\" (UID: \"02bda514-3256-495f-8ce5-ae9e7fe50c71\") " pod="openshift-image-registry/node-ca-blhwm" Jul 20 13:16:48.254942 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254354 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vc6r\" (UniqueName: \"kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r\") pod \"network-check-target-wgvvv\" (UID: \"dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e\") " pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:16:48.254942 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f801ea8-fd39-44a0-a77e-2b2cfbbee71c-host-slash\") pod \"iptables-alerter-r7fcq\" (UID: \"6f801ea8-fd39-44a0-a77e-2b2cfbbee71c\") " pod="openshift-network-operator/iptables-alerter-r7fcq" Jul 20 13:16:48.254942 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254407 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-cni-netd\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.254942 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-modprobe-d\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.254942 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-lib-modules\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.254942 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eac01c81-00c9-45f2-837a-95a88e53dc2b-tmp\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.254942 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254522 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d0cf929-460c-4fb9-bfd5-c311d138d89d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.254942 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254552 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-dbus\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:48.254942 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-socket-dir\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.254942 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-run-ovn-kubernetes\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.254942 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254653 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-cni-binary-copy\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.255449 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.254976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-var-lib-kubelet\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.255449 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255008 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53011d87-65ad-4a8b-a364-5e66e1cea999-konnectivity-ca\") pod \"konnectivity-agent-lpksh\" (UID: \"53011d87-65ad-4a8b-a364-5e66e1cea999\") " pod="kube-system/konnectivity-agent-lpksh" Jul 20 13:16:48.255449 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255040 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-etc-openvswitch\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.255449 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255072 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-multus-conf-dir\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.255449 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02bda514-3256-495f-8ce5-ae9e7fe50c71-serviceca\") pod \"node-ca-blhwm\" (UID: \"02bda514-3256-495f-8ce5-ae9e7fe50c71\") " pod="openshift-image-registry/node-ca-blhwm" Jul 20 13:16:48.255449 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255124 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-registration-dir\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.255449 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-etc-selinux\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.255449 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-run-openvswitch\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.255449 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-node-log\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.255449 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255230 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-sysctl-conf\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.255449 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d0cf929-460c-4fb9-bfd5-c311d138d89d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.255449 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255293 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-kubelet-config\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:48.256016 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255474 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d0cf929-460c-4fb9-bfd5-c311d138d89d-cni-binary-copy\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.256016 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255675 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-etc-kubernetes\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.256016 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255723 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-sysconfig\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.256016 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255747 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-sysctl-conf\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.256016 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.255936 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-var-lib-kubelet\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.256016 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.256001 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-multus-conf-dir\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.256320 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.256097 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-lib-modules\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.256320 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.256164 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-modprobe-d\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.256320 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.256179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d0cf929-460c-4fb9-bfd5-c311d138d89d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.256461 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.256392 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eac01c81-00c9-45f2-837a-95a88e53dc2b-etc-tuned\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.256558 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.256539 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02bda514-3256-495f-8ce5-ae9e7fe50c71-serviceca\") pod \"node-ca-blhwm\" (UID: \"02bda514-3256-495f-8ce5-ae9e7fe50c71\") " pod="openshift-image-registry/node-ca-blhwm" Jul 20 13:16:48.256615 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.256587 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d0cf929-460c-4fb9-bfd5-c311d138d89d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.258431 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.258214 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eac01c81-00c9-45f2-837a-95a88e53dc2b-tmp\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.272382 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.272361 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cm9x\" (UniqueName: \"kubernetes.io/projected/2d0cf929-460c-4fb9-bfd5-c311d138d89d-kube-api-access-6cm9x\") pod \"multus-additional-cni-plugins-pstqq\" (UID: \"2d0cf929-460c-4fb9-bfd5-c311d138d89d\") " pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.272552 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.272517 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57brg\" (UniqueName: \"kubernetes.io/projected/4e0ac5e5-38fc-4106-9c04-e4d14dddaa72-kube-api-access-57brg\") pod \"multus-2l6tt\" (UID: \"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72\") " pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.272803 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.272784 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg45d\" (UniqueName: \"kubernetes.io/projected/eac01c81-00c9-45f2-837a-95a88e53dc2b-kube-api-access-pg45d\") pod \"tuned-mts6s\" (UID: \"eac01c81-00c9-45f2-837a-95a88e53dc2b\") " pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.276144 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.276124 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2lgd\" (UniqueName: \"kubernetes.io/projected/02bda514-3256-495f-8ce5-ae9e7fe50c71-kube-api-access-p2lgd\") pod \"node-ca-blhwm\" (UID: \"02bda514-3256-495f-8ce5-ae9e7fe50c71\") " pod="openshift-image-registry/node-ca-blhwm" Jul 20 13:16:48.307581 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.307544 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" event={"ID":"0544e6927bc4266c71d4da456a570889","Type":"ContainerStarted","Data":"f9432fdbdd6a476ffcbdf52044345dbfb09c079d43966e9ec4fe17a0efb0719c"} Jul 20 13:16:48.308638 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.308617 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" event={"ID":"4ba135fd790f8de9f90cf6e2924b3971","Type":"ContainerStarted","Data":"beca719e4f78383861157427c9309bbf7181968f51157700f8768157d036447c"} Jul 20 13:16:48.356182 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:48.356182 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356161 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-systemd-units\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356182 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356176 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-run-ovn\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356354 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356219 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-systemd-units\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356354 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.356263 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:48.356354 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-cni-bin\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356354 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356300 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-cni-bin\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356354 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356336 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356561 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356352 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-run-ovn\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356561 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356356 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356561 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356359 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8290b97d-bb82-4de4-b590-b95bd307d4bc-ovnkube-script-lib\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356561 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.356393 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret podName:715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:48.856373289 +0000 UTC m=+3.064769420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret") pod "global-pull-secret-syncer-7sjzn" (UID: "715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:48.356561 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmgjw\" (UniqueName: \"kubernetes.io/projected/8290b97d-bb82-4de4-b590-b95bd307d4bc-kube-api-access-kmgjw\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356561 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.356561 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6n2l\" (UniqueName: \"kubernetes.io/projected/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-kube-api-access-l6n2l\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.356561 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356518 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.356957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8290b97d-bb82-4de4-b590-b95bd307d4bc-ovnkube-config\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7bdea0d3-dbcd-4be1-bddc-49679ef918f1-hosts-file\") pod \"node-resolver-h8gtl\" (UID: \"7bdea0d3-dbcd-4be1-bddc-49679ef918f1\") " pod="openshift-dns/node-resolver-h8gtl" Jul 20 13:16:48.356957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vc6r\" (UniqueName: \"kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r\") pod \"network-check-target-wgvvv\" (UID: \"dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e\") " pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:16:48.356957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356658 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f801ea8-fd39-44a0-a77e-2b2cfbbee71c-host-slash\") pod \"iptables-alerter-r7fcq\" (UID: \"6f801ea8-fd39-44a0-a77e-2b2cfbbee71c\") " pod="openshift-network-operator/iptables-alerter-r7fcq" Jul 20 13:16:48.356957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356698 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7bdea0d3-dbcd-4be1-bddc-49679ef918f1-hosts-file\") pod \"node-resolver-h8gtl\" (UID: \"7bdea0d3-dbcd-4be1-bddc-49679ef918f1\") " pod="openshift-dns/node-resolver-h8gtl" Jul 20 13:16:48.356957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-cni-netd\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-dbus\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:48.356957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-socket-dir\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.356957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356770 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f801ea8-fd39-44a0-a77e-2b2cfbbee71c-host-slash\") pod \"iptables-alerter-r7fcq\" (UID: \"6f801ea8-fd39-44a0-a77e-2b2cfbbee71c\") " pod="openshift-network-operator/iptables-alerter-r7fcq" Jul 20 13:16:48.356957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356742 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-cni-netd\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356786 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-run-ovn-kubernetes\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356817 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53011d87-65ad-4a8b-a364-5e66e1cea999-konnectivity-ca\") pod \"konnectivity-agent-lpksh\" (UID: \"53011d87-65ad-4a8b-a364-5e66e1cea999\") " pod="kube-system/konnectivity-agent-lpksh" Jul 20 13:16:48.356957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356823 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-run-ovn-kubernetes\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.356957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-etc-openvswitch\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.357572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8290b97d-bb82-4de4-b590-b95bd307d4bc-ovnkube-script-lib\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.357572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356895 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-socket-dir\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.357572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-registration-dir\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.357572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-dbus\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:48.357572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356931 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-registration-dir\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.357572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-etc-selinux\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.357572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-run-openvswitch\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.357572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-node-log\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.357572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-kubelet-config\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:48.357572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357017 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-etc-selinux\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.357572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6f801ea8-fd39-44a0-a77e-2b2cfbbee71c-iptables-alerter-script\") pod \"iptables-alerter-r7fcq\" (UID: \"6f801ea8-fd39-44a0-a77e-2b2cfbbee71c\") " pod="openshift-network-operator/iptables-alerter-r7fcq" Jul 20 13:16:48.357572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357038 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-run-openvswitch\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.357572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357046 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-node-log\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.357572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.356940 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-etc-openvswitch\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357073 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-var-lib-openvswitch\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357093 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-kubelet-config\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357102 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8290b97d-bb82-4de4-b590-b95bd307d4bc-ovnkube-config\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-slash\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357129 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-var-lib-openvswitch\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-run-systemd\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-log-socket\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357164 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-slash\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-sys-fs\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357191 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-run-systemd\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357207 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-log-socket\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53011d87-65ad-4a8b-a364-5e66e1cea999-agent-certs\") pod \"konnectivity-agent-lpksh\" (UID: \"53011d87-65ad-4a8b-a364-5e66e1cea999\") " pod="kube-system/konnectivity-agent-lpksh" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357235 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-sys-fs\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvr2q\" (UniqueName: \"kubernetes.io/projected/6f801ea8-fd39-44a0-a77e-2b2cfbbee71c-kube-api-access-nvr2q\") pod \"iptables-alerter-r7fcq\" (UID: \"6f801ea8-fd39-44a0-a77e-2b2cfbbee71c\") " pod="openshift-network-operator/iptables-alerter-r7fcq" Jul 20 13:16:48.358217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357265 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdea0d3-dbcd-4be1-bddc-49679ef918f1-tmp-dir\") pod \"node-resolver-h8gtl\" (UID: \"7bdea0d3-dbcd-4be1-bddc-49679ef918f1\") " pod="openshift-dns/node-resolver-h8gtl" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdq64\" (UniqueName: \"kubernetes.io/projected/7bdea0d3-dbcd-4be1-bddc-49679ef918f1-kube-api-access-jdq64\") pod \"node-resolver-h8gtl\" (UID: \"7bdea0d3-dbcd-4be1-bddc-49679ef918f1\") " pod="openshift-dns/node-resolver-h8gtl" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357307 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-device-dir\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8290b97d-bb82-4de4-b590-b95bd307d4bc-env-overrides\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357340 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53011d87-65ad-4a8b-a364-5e66e1cea999-konnectivity-ca\") pod \"konnectivity-agent-lpksh\" (UID: \"53011d87-65ad-4a8b-a364-5e66e1cea999\") " pod="kube-system/konnectivity-agent-lpksh" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357354 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzj9\" (UniqueName: \"kubernetes.io/projected/c805bf30-7720-4274-81e1-26d75eb7ae55-kube-api-access-pvzj9\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-kubelet\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357409 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-run-netns\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357422 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8290b97d-bb82-4de4-b590-b95bd307d4bc-ovn-node-metrics-cert\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357516 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-kubelet\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357568 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6f801ea8-fd39-44a0-a77e-2b2cfbbee71c-iptables-alerter-script\") pod \"iptables-alerter-r7fcq\" (UID: \"6f801ea8-fd39-44a0-a77e-2b2cfbbee71c\") " pod="openshift-network-operator/iptables-alerter-r7fcq" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357574 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdea0d3-dbcd-4be1-bddc-49679ef918f1-tmp-dir\") pod \"node-resolver-h8gtl\" (UID: \"7bdea0d3-dbcd-4be1-bddc-49679ef918f1\") " pod="openshift-dns/node-resolver-h8gtl" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357565 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-device-dir\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357637 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8290b97d-bb82-4de4-b590-b95bd307d4bc-host-run-netns\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.358894 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.357699 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:48.359538 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.357763 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs podName:c805bf30-7720-4274-81e1-26d75eb7ae55 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:48.857749882 +0000 UTC m=+3.066146007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs") pod "network-metrics-daemon-svhpp" (UID: "c805bf30-7720-4274-81e1-26d75eb7ae55") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:48.359538 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.357910 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8290b97d-bb82-4de4-b590-b95bd307d4bc-env-overrides\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.359657 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.359642 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8290b97d-bb82-4de4-b590-b95bd307d4bc-ovn-node-metrics-cert\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.360082 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.360066 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53011d87-65ad-4a8b-a364-5e66e1cea999-agent-certs\") pod \"konnectivity-agent-lpksh\" (UID: \"53011d87-65ad-4a8b-a364-5e66e1cea999\") " pod="kube-system/konnectivity-agent-lpksh" Jul 20 13:16:48.369738 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.369722 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdq64\" (UniqueName: \"kubernetes.io/projected/7bdea0d3-dbcd-4be1-bddc-49679ef918f1-kube-api-access-jdq64\") pod \"node-resolver-h8gtl\" (UID: \"7bdea0d3-dbcd-4be1-bddc-49679ef918f1\") " pod="openshift-dns/node-resolver-h8gtl" Jul 20 13:16:48.371761 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.371742 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmgjw\" (UniqueName: \"kubernetes.io/projected/8290b97d-bb82-4de4-b590-b95bd307d4bc-kube-api-access-kmgjw\") pod \"ovnkube-node-khqq5\" (UID: \"8290b97d-bb82-4de4-b590-b95bd307d4bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.373561 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.373540 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6n2l\" (UniqueName: \"kubernetes.io/projected/f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb-kube-api-access-l6n2l\") pod \"aws-ebs-csi-driver-node-tm2gx\" (UID: \"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.374383 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.374358 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:48.374383 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.374382 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:48.374527 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.374394 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6vc6r for pod openshift-network-diagnostics/network-check-target-wgvvv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:48.374527 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.374452 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r podName:dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e nodeName:}" failed. No retries permitted until 2026-07-20 13:16:48.874438299 +0000 UTC m=+3.082834434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6vc6r" (UniqueName: "kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r") pod "network-check-target-wgvvv" (UID: "dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:48.374527 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.374476 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvr2q\" (UniqueName: \"kubernetes.io/projected/6f801ea8-fd39-44a0-a77e-2b2cfbbee71c-kube-api-access-nvr2q\") pod \"iptables-alerter-r7fcq\" (UID: \"6f801ea8-fd39-44a0-a77e-2b2cfbbee71c\") " pod="openshift-network-operator/iptables-alerter-r7fcq" Jul 20 13:16:48.375538 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.375521 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzj9\" (UniqueName: \"kubernetes.io/projected/c805bf30-7720-4274-81e1-26d75eb7ae55-kube-api-access-pvzj9\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:48.424496 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.424473 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2l6tt" Jul 20 13:16:48.430408 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.430389 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mts6s" Jul 20 13:16:48.439753 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.439737 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pstqq" Jul 20 13:16:48.444148 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.444135 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-blhwm" Jul 20 13:16:48.455298 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.455279 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h8gtl" Jul 20 13:16:48.460881 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.460852 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" Jul 20 13:16:48.468174 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.468156 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lpksh" Jul 20 13:16:48.475386 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.475366 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r7fcq" Jul 20 13:16:48.481883 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.481854 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:16:48.608541 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.608486 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:16:48.861342 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.861275 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:48.861483 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.861352 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:48.861483 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.861412 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:48.861483 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.861426 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:48.861483 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.861473 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs podName:c805bf30-7720-4274-81e1-26d75eb7ae55 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:49.861456584 +0000 UTC m=+4.069852705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs") pod "network-metrics-daemon-svhpp" (UID: "c805bf30-7720-4274-81e1-26d75eb7ae55") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:48.861483 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.861486 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret podName:715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:49.861480361 +0000 UTC m=+4.069876481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret") pod "global-pull-secret-syncer-7sjzn" (UID: "715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:48.962167 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:48.962137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vc6r\" (UniqueName: \"kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r\") pod \"network-check-target-wgvvv\" (UID: \"dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e\") " pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:16:48.962312 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.962264 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:48.962312 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.962282 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:48.962312 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.962291 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6vc6r for pod openshift-network-diagnostics/network-check-target-wgvvv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:48.962472 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:48.962332 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r podName:dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e nodeName:}" failed. No retries permitted until 2026-07-20 13:16:49.962321556 +0000 UTC m=+4.170717678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6vc6r" (UniqueName: "kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r") pod "network-check-target-wgvvv" (UID: "dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:49.058667 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:49.058638 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bda514_3256_495f_8ce5_ae9e7fe50c71.slice/crio-1abf2ba765744501a6e14790ada3da6c68e05003a42d97d5096d28c072b65c60 WatchSource:0}: Error finding container 1abf2ba765744501a6e14790ada3da6c68e05003a42d97d5096d28c072b65c60: Status 404 returned error can't find the container with id 1abf2ba765744501a6e14790ada3da6c68e05003a42d97d5096d28c072b65c60 Jul 20 13:16:49.065403 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:49.065376 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0cf929_460c_4fb9_bfd5_c311d138d89d.slice/crio-547a71bbb3319d331519b5aced1b6fa1a05f335d5a893a8cd795012eb8a4b26b WatchSource:0}: Error finding container 547a71bbb3319d331519b5aced1b6fa1a05f335d5a893a8cd795012eb8a4b26b: Status 404 returned error can't find the container with id 547a71bbb3319d331519b5aced1b6fa1a05f335d5a893a8cd795012eb8a4b26b Jul 20 13:16:49.066060 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:49.065964 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f801ea8_fd39_44a0_a77e_2b2cfbbee71c.slice/crio-4b42e61ab542f61100ff58c4173743842041ad685bfc24d0fc6d3ae361cad391 WatchSource:0}: Error finding container 4b42e61ab542f61100ff58c4173743842041ad685bfc24d0fc6d3ae361cad391: Status 404 returned error can't find the container with id 4b42e61ab542f61100ff58c4173743842041ad685bfc24d0fc6d3ae361cad391 Jul 20 13:16:49.067931 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:49.067903 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0ac5e5_38fc_4106_9c04_e4d14dddaa72.slice/crio-88673fb3b262fa68127dfb7b6a103837e01ed0c726fef1d4bfd07a8404d85596 WatchSource:0}: Error finding container 88673fb3b262fa68127dfb7b6a103837e01ed0c726fef1d4bfd07a8404d85596: Status 404 returned error can't find the container with id 88673fb3b262fa68127dfb7b6a103837e01ed0c726fef1d4bfd07a8404d85596 Jul 20 13:16:49.068673 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:49.068653 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78574f0_3f5d_49d9_b8ae_6f2d0d55dcdb.slice/crio-b90efb74d75fab2363a607bf3a6afeb55cbfdd9cc17ea7ad894387ac8e1b1293 WatchSource:0}: Error finding container b90efb74d75fab2363a607bf3a6afeb55cbfdd9cc17ea7ad894387ac8e1b1293: Status 404 returned error can't find the container with id b90efb74d75fab2363a607bf3a6afeb55cbfdd9cc17ea7ad894387ac8e1b1293 Jul 20 13:16:49.087516 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:49.087495 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53011d87_65ad_4a8b_a364_5e66e1cea999.slice/crio-490994b69fa4544c85712c74cf4abff74967e4b14e8cd987c91d75f4c2dfd6e6 WatchSource:0}: Error finding container 490994b69fa4544c85712c74cf4abff74967e4b14e8cd987c91d75f4c2dfd6e6: Status 404 returned error can't find the container with id 490994b69fa4544c85712c74cf4abff74967e4b14e8cd987c91d75f4c2dfd6e6 Jul 20 13:16:49.089089 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:49.089072 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8290b97d_bb82_4de4_b590_b95bd307d4bc.slice/crio-8f3a7a5e15870bacd8f1cf326ca39d70b6d9945f470802773813a0b097f409a1 WatchSource:0}: Error finding container 8f3a7a5e15870bacd8f1cf326ca39d70b6d9945f470802773813a0b097f409a1: Status 404 returned error can't find the container with id 8f3a7a5e15870bacd8f1cf326ca39d70b6d9945f470802773813a0b097f409a1 Jul 20 13:16:49.090007 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:16:49.089989 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdea0d3_dbcd_4be1_bddc_49679ef918f1.slice/crio-abbbaa0492d621fc5cb3f161e2a532e479fc1707a04ea78cee04789a49741f85 WatchSource:0}: Error finding container abbbaa0492d621fc5cb3f161e2a532e479fc1707a04ea78cee04789a49741f85: Status 404 returned error can't find the container with id abbbaa0492d621fc5cb3f161e2a532e479fc1707a04ea78cee04789a49741f85 Jul 20 13:16:49.176811 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.176624 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 13:11:47 +0000 UTC" deadline="2028-02-29 02:55:14.414403963 +0000 UTC" Jul 20 13:16:49.176811 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.176810 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14125h38m25.237598456s" Jul 20 13:16:49.302980 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.302958 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:16:49.303535 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:49.303051 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:16:49.310780 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.310759 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-blhwm" event={"ID":"02bda514-3256-495f-8ce5-ae9e7fe50c71","Type":"ContainerStarted","Data":"1abf2ba765744501a6e14790ada3da6c68e05003a42d97d5096d28c072b65c60"} Jul 20 13:16:49.312373 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.312354 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" event={"ID":"0544e6927bc4266c71d4da456a570889","Type":"ContainerStarted","Data":"62a32aa84f75b7b004a4ef92f9aad3e8dc15b72173aa146ebe0d055e6b2ba0db"} Jul 20 13:16:49.313309 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.313293 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h8gtl" event={"ID":"7bdea0d3-dbcd-4be1-bddc-49679ef918f1","Type":"ContainerStarted","Data":"abbbaa0492d621fc5cb3f161e2a532e479fc1707a04ea78cee04789a49741f85"} Jul 20 13:16:49.314145 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.314123 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lpksh" event={"ID":"53011d87-65ad-4a8b-a364-5e66e1cea999","Type":"ContainerStarted","Data":"490994b69fa4544c85712c74cf4abff74967e4b14e8cd987c91d75f4c2dfd6e6"} Jul 20 13:16:49.315103 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.315088 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pstqq" event={"ID":"2d0cf929-460c-4fb9-bfd5-c311d138d89d","Type":"ContainerStarted","Data":"547a71bbb3319d331519b5aced1b6fa1a05f335d5a893a8cd795012eb8a4b26b"} Jul 20 13:16:49.316355 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.316338 2575 generic.go:356] "Generic (PLEG): container finished" podID="4ba135fd790f8de9f90cf6e2924b3971" containerID="2e6395592cbd22ac399ed46b1097fe1f893c91d86351f2eed0327cff186bb021" exitCode=0 Jul 20 13:16:49.316445 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.316404 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" event={"ID":"4ba135fd790f8de9f90cf6e2924b3971","Type":"ContainerDied","Data":"2e6395592cbd22ac399ed46b1097fe1f893c91d86351f2eed0327cff186bb021"} Jul 20 13:16:49.317393 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.317375 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" event={"ID":"8290b97d-bb82-4de4-b590-b95bd307d4bc","Type":"ContainerStarted","Data":"8f3a7a5e15870bacd8f1cf326ca39d70b6d9945f470802773813a0b097f409a1"} Jul 20 13:16:49.318383 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.318359 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r7fcq" event={"ID":"6f801ea8-fd39-44a0-a77e-2b2cfbbee71c","Type":"ContainerStarted","Data":"4b42e61ab542f61100ff58c4173743842041ad685bfc24d0fc6d3ae361cad391"} Jul 20 13:16:49.319343 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.319327 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" event={"ID":"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb","Type":"ContainerStarted","Data":"b90efb74d75fab2363a607bf3a6afeb55cbfdd9cc17ea7ad894387ac8e1b1293"} Jul 20 13:16:49.320310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.320283 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2l6tt" event={"ID":"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72","Type":"ContainerStarted","Data":"88673fb3b262fa68127dfb7b6a103837e01ed0c726fef1d4bfd07a8404d85596"} Jul 20 13:16:49.321140 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.321124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mts6s" event={"ID":"eac01c81-00c9-45f2-837a-95a88e53dc2b","Type":"ContainerStarted","Data":"2efdb7835df1a7d212066b94fe09954ebc512329139a4526a144345a2026139a"} Jul 20 13:16:49.326581 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.326548 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" podStartSLOduration=2.326538759 podStartE2EDuration="2.326538759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:16:49.326378941 +0000 UTC m=+3.534775085" watchObservedRunningTime="2026-07-20 13:16:49.326538759 +0000 UTC m=+3.534934900" Jul 20 13:16:49.869018 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.868944 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:49.869018 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.869001 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:49.869248 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:49.869142 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:49.869248 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:49.869195 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret podName:715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:51.869179169 +0000 UTC m=+6.077575314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret") pod "global-pull-secret-syncer-7sjzn" (UID: "715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:49.869594 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:49.869573 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:49.869681 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:49.869624 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs podName:c805bf30-7720-4274-81e1-26d75eb7ae55 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:51.869611341 +0000 UTC m=+6.078007466 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs") pod "network-metrics-daemon-svhpp" (UID: "c805bf30-7720-4274-81e1-26d75eb7ae55") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:49.970365 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:49.969812 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vc6r\" (UniqueName: \"kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r\") pod \"network-check-target-wgvvv\" (UID: \"dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e\") " pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:16:49.970365 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:49.969982 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:49.970365 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:49.970001 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:49.970365 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:49.970015 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6vc6r for pod openshift-network-diagnostics/network-check-target-wgvvv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:49.970365 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:49.970064 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r podName:dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e nodeName:}" failed. No retries permitted until 2026-07-20 13:16:51.970048037 +0000 UTC m=+6.178444163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6vc6r" (UniqueName: "kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r") pod "network-check-target-wgvvv" (UID: "dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:50.177575 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:50.177495 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 13:11:47 +0000 UTC" deadline="2028-02-22 12:46:04.828559484 +0000 UTC" Jul 20 13:16:50.177575 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:50.177527 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13967h29m14.651036196s" Jul 20 13:16:50.304401 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:50.303975 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:50.304401 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:50.304096 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:16:50.306638 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:50.304955 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:50.306638 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:50.305048 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:16:50.352785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:50.352366 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" event={"ID":"4ba135fd790f8de9f90cf6e2924b3971","Type":"ContainerStarted","Data":"f32acb627b2b22766afe3ca54fdbbf86646e04522f8e97f685277e788532508e"} Jul 20 13:16:51.304267 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:51.303768 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:16:51.304267 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:51.303903 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:16:51.891180 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:51.891147 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:51.891367 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:51.891190 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:51.891367 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:51.891306 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:51.891507 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:51.891366 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret podName:715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:55.891349699 +0000 UTC m=+10.099745822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret") pod "global-pull-secret-syncer-7sjzn" (UID: "715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:51.891507 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:51.891306 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:51.891507 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:51.891445 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs podName:c805bf30-7720-4274-81e1-26d75eb7ae55 nodeName:}" failed. No retries permitted until 2026-07-20 13:16:55.891431406 +0000 UTC m=+10.099827541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs") pod "network-metrics-daemon-svhpp" (UID: "c805bf30-7720-4274-81e1-26d75eb7ae55") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:51.992056 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:51.992023 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vc6r\" (UniqueName: \"kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r\") pod \"network-check-target-wgvvv\" (UID: \"dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e\") " pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:16:51.992255 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:51.992230 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:51.992255 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:51.992255 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:51.992439 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:51.992268 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6vc6r for pod openshift-network-diagnostics/network-check-target-wgvvv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:51.992439 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:51.992322 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r podName:dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e nodeName:}" failed. No retries permitted until 2026-07-20 13:16:55.992305013 +0000 UTC m=+10.200701150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6vc6r" (UniqueName: "kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r") pod "network-check-target-wgvvv" (UID: "dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:52.303383 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:52.303305 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:52.303551 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:52.303305 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:52.303551 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:52.303446 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:16:52.303551 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:52.303499 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:16:53.303824 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:53.303791 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:16:53.304299 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:53.303932 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:16:54.303247 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:54.303216 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:54.303424 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:54.303349 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:16:54.303697 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:54.303676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:54.303820 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:54.303798 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:16:55.303867 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:55.303341 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:16:55.303867 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:55.303486 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:16:55.925232 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:55.924649 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:55.925232 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:55.924689 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:55.925232 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:55.924818 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:55.925232 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:55.924865 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret podName:715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:03.924849459 +0000 UTC m=+18.133245596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret") pod "global-pull-secret-syncer-7sjzn" (UID: "715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:16:55.925232 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:55.924945 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:55.925232 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:55.924969 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs podName:c805bf30-7720-4274-81e1-26d75eb7ae55 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:03.92496153 +0000 UTC m=+18.133357652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs") pod "network-metrics-daemon-svhpp" (UID: "c805bf30-7720-4274-81e1-26d75eb7ae55") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:16:56.025762 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:56.025668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vc6r\" (UniqueName: \"kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r\") pod \"network-check-target-wgvvv\" (UID: \"dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e\") " pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:16:56.025955 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:56.025815 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:16:56.025955 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:56.025834 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:16:56.025955 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:56.025847 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6vc6r for pod openshift-network-diagnostics/network-check-target-wgvvv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:56.025955 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:56.025920 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r podName:dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e nodeName:}" failed. No retries permitted until 2026-07-20 13:17:04.02590203 +0000 UTC m=+18.234298167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6vc6r" (UniqueName: "kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r") pod "network-check-target-wgvvv" (UID: "dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:16:56.304249 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:56.303805 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:56.304249 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:56.303937 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:16:56.304249 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:56.303982 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:56.304249 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:56.304105 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:16:57.303674 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:57.303634 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:16:57.303860 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:57.303774 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:16:58.303334 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:58.303299 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:16:58.303831 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:58.303424 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:16:58.303831 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:58.303767 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:16:58.303945 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:58.303865 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:16:59.303191 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:16:59.303169 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:16:59.303294 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:16:59.303252 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:17:00.303477 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:00.303445 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:00.303975 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:00.303559 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:17:00.303975 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:00.303588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:00.303975 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:00.303682 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:17:01.303623 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:01.303588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:01.304074 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:01.303715 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:17:02.303234 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:02.303202 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:02.303402 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:02.303347 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:17:02.303462 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:02.303408 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:02.303553 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:02.303528 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:17:03.303531 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:03.303178 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:03.303531 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:03.303290 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:17:03.983633 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:03.983555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:03.983633 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:03.983611 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:03.983889 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:03.983704 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:17:03.983889 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:03.983732 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:17:03.983889 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:03.983778 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs podName:c805bf30-7720-4274-81e1-26d75eb7ae55 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:19.983757401 +0000 UTC m=+34.192153539 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs") pod "network-metrics-daemon-svhpp" (UID: "c805bf30-7720-4274-81e1-26d75eb7ae55") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:17:03.983889 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:03.983801 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret podName:715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:19.983790649 +0000 UTC m=+34.192186780 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret") pod "global-pull-secret-syncer-7sjzn" (UID: "715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:17:04.084659 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:04.084631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vc6r\" (UniqueName: \"kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r\") pod \"network-check-target-wgvvv\" (UID: \"dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e\") " pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:04.084829 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:04.084796 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:17:04.084829 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:04.084817 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:17:04.084829 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:04.084829 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6vc6r for pod openshift-network-diagnostics/network-check-target-wgvvv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:17:04.084965 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:04.084900 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r podName:dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e nodeName:}" failed. No retries permitted until 2026-07-20 13:17:20.084868059 +0000 UTC m=+34.293264182 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6vc6r" (UniqueName: "kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r") pod "network-check-target-wgvvv" (UID: "dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:17:04.303052 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:04.302978 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:04.303211 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:04.303112 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:04.303211 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:04.303193 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:17:04.303395 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:04.303365 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:17:05.303190 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:05.303157 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:05.303671 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:05.303277 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:17:06.304526 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:06.304491 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:06.304958 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:06.304616 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:17:06.304958 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:06.304681 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:06.304958 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:06.304783 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:17:07.303054 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:07.303022 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:07.303261 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:07.303127 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:17:08.303984 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.303687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:08.304626 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.303756 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:08.304626 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:08.304058 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:17:08.304626 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:08.304151 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:17:08.393385 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.393362 2575 generic.go:356] "Generic (PLEG): container finished" podID="2d0cf929-460c-4fb9-bfd5-c311d138d89d" containerID="22424cef132ccaa3f94f31da50110791a8857ff76f3545262711937729d4ef63" exitCode=0 Jul 20 13:17:08.393475 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.393439 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pstqq" event={"ID":"2d0cf929-460c-4fb9-bfd5-c311d138d89d","Type":"ContainerDied","Data":"22424cef132ccaa3f94f31da50110791a8857ff76f3545262711937729d4ef63"} Jul 20 13:17:08.396101 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.396065 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" event={"ID":"8290b97d-bb82-4de4-b590-b95bd307d4bc","Type":"ContainerStarted","Data":"d2dd512561e44c68506d4ef928e495ba6cccbf9bac92f9b0c09778bece97b74b"} Jul 20 13:17:08.396201 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.396131 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" event={"ID":"8290b97d-bb82-4de4-b590-b95bd307d4bc","Type":"ContainerStarted","Data":"0a3b6c82b78820566b542767e52e219349086dd3abfa611f97e749a4355e00e2"} Jul 20 13:17:08.396201 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.396181 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" event={"ID":"8290b97d-bb82-4de4-b590-b95bd307d4bc","Type":"ContainerStarted","Data":"dc366f523dfcfed618f2ab903d4d298c7eaafceb60f3d01fb27026da3f298354"} Jul 20 13:17:08.396275 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.396233 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" event={"ID":"8290b97d-bb82-4de4-b590-b95bd307d4bc","Type":"ContainerStarted","Data":"605de319769b80aa44885cb3a816a417b0ff212fec138a99e5cdb0cb75025b5a"} Jul 20 13:17:08.396324 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.396282 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" event={"ID":"8290b97d-bb82-4de4-b590-b95bd307d4bc","Type":"ContainerStarted","Data":"603400b0f427f3d304edeab2bae493768be225ca84e1350f34b28df4ed37f3f1"} Jul 20 13:17:08.397389 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.397362 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" event={"ID":"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb","Type":"ContainerStarted","Data":"061db1ac375d0db273957d63b37fa29bd680ba59d26f5fad9f51b4665b689187"} Jul 20 13:17:08.398788 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.398767 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2l6tt" event={"ID":"4e0ac5e5-38fc-4106-9c04-e4d14dddaa72","Type":"ContainerStarted","Data":"93af3cb1277e8733ef0c8e0278e3451c830ada27c3ed302ac234e1c83091f008"} Jul 20 13:17:08.400169 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.400143 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mts6s" event={"ID":"eac01c81-00c9-45f2-837a-95a88e53dc2b","Type":"ContainerStarted","Data":"0d7f0c1faa5a2a1c58b8ad1ff67f3cadb7a51013d21cccf7ce90ff2187426011"} Jul 20 13:17:08.401409 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.401390 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-blhwm" event={"ID":"02bda514-3256-495f-8ce5-ae9e7fe50c71","Type":"ContainerStarted","Data":"5558bf8c9440d0d56b2312aaba45aac8eb540f1e414103d8f06be5a02e152535"} Jul 20 13:17:08.402759 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.402735 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h8gtl" event={"ID":"7bdea0d3-dbcd-4be1-bddc-49679ef918f1","Type":"ContainerStarted","Data":"a592dcfea41064d7fdc6f928646691db4ab240681eb6b9f59f0fad9d1329acf5"} Jul 20 13:17:08.404101 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.404081 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lpksh" event={"ID":"53011d87-65ad-4a8b-a364-5e66e1cea999","Type":"ContainerStarted","Data":"c24748235c06506719385ecc71dc7ec3a5783dcd6d96a6af6796de137ba4ec25"} Jul 20 13:17:08.420209 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.420171 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" podStartSLOduration=21.420159681 podStartE2EDuration="21.420159681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:16:50.369000292 +0000 UTC m=+4.577396435" watchObservedRunningTime="2026-07-20 13:17:08.420159681 +0000 UTC m=+22.628555825" Jul 20 13:17:08.435180 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.435128 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-blhwm" podStartSLOduration=3.98293134 podStartE2EDuration="22.435116491s" totalImagesPullingTime="18.452185151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:08.434813989 +0000 UTC m=+22.643210346" watchObservedRunningTime="2026-07-20 13:17:08.435116491 +0000 UTC m=+22.643512635" Jul 20 13:17:08.476981 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.476947 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lpksh" podStartSLOduration=7.866379342 podStartE2EDuration="22.476934757s" totalImagesPullingTime="14.610555415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:08.455587754 +0000 UTC m=+22.663983899" watchObservedRunningTime="2026-07-20 13:17:08.476934757 +0000 UTC m=+22.685330901" Jul 20 13:17:08.477118 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.477093 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2l6tt" podStartSLOduration=4.7867927 podStartE2EDuration="23.477088104s" totalImagesPullingTime="18.690295404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:08.476678819 +0000 UTC m=+22.685074964" watchObservedRunningTime="2026-07-20 13:17:08.477088104 +0000 UTC m=+22.685484249" Jul 20 13:17:08.496651 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.496579 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mts6s" podStartSLOduration=4.828199421 podStartE2EDuration="23.496570435s" totalImagesPullingTime="18.668371014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:08.496150494 +0000 UTC m=+22.704546640" watchObservedRunningTime="2026-07-20 13:17:08.496570435 +0000 UTC m=+22.704966584" Jul 20 13:17:08.514160 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:08.514118 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h8gtl" podStartSLOduration=3.873274021 podStartE2EDuration="22.514102512s" totalImagesPullingTime="18.640828491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:08.513433316 +0000 UTC m=+22.721829461" watchObservedRunningTime="2026-07-20 13:17:08.514102512 +0000 UTC m=+22.722498659" Jul 20 13:17:09.303035 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:09.303011 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:09.303191 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:09.303138 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:17:09.339183 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:09.339158 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 20 13:17:09.409665 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:09.409636 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" event={"ID":"8290b97d-bb82-4de4-b590-b95bd307d4bc","Type":"ContainerStarted","Data":"b6bbdce1744741478bafc8a392a86176b7cd54e8224bac31723368a5fc747a05"} Jul 20 13:17:09.411319 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:09.411290 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r7fcq" event={"ID":"6f801ea8-fd39-44a0-a77e-2b2cfbbee71c","Type":"ContainerStarted","Data":"08390688b4608db5f78763e2c0d8f93c317b7751d8bf7909c0ef5d504077da4e"} Jul 20 13:17:09.413332 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:09.413296 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" event={"ID":"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb","Type":"ContainerStarted","Data":"2117b3a60012f16a65e6a8a1250b0ee4365014570e89a7fc51af90a685288592"} Jul 20 13:17:09.428382 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:09.428344 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r7fcq" podStartSLOduration=4.982845061 podStartE2EDuration="23.428325017s" totalImagesPullingTime="18.445479956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:09.427954157 +0000 UTC m=+23.636350304" watchObservedRunningTime="2026-07-20 13:17:09.428325017 +0000 UTC m=+23.636721163" Jul 20 13:17:10.244254 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:10.244144 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T13:17:09.339178546Z","UUID":"a74f3f1c-0048-4693-b4e6-bc060e5da4cc","Handler":null,"Name":"","Endpoint":""} Jul 20 13:17:10.245865 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:10.245830 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 20 13:17:10.246022 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:10.245887 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 20 13:17:10.302927 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:10.302898 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:10.303067 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:10.303027 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:17:10.303137 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:10.303094 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:10.303223 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:10.303198 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:17:11.062555 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:11.062477 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lpksh" Jul 20 13:17:11.063105 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:11.063084 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lpksh" Jul 20 13:17:11.303396 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:11.303367 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:11.303546 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:11.303460 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:17:11.420230 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:11.420148 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" event={"ID":"f78574f0-3f5d-49d9-b8ae-6f2d0d55dcdb","Type":"ContainerStarted","Data":"f53c833b3c9f82cd6dc721fee28d88d7099a9f7e77c276ec424a2e63f92f3e87"} Jul 20 13:17:11.423780 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:11.423741 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" event={"ID":"8290b97d-bb82-4de4-b590-b95bd307d4bc","Type":"ContainerStarted","Data":"378111610f2015dc12fdc7593aa62d1f594b4770360af5016b183b511ecfe2c7"} Jul 20 13:17:11.442158 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:11.442115 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tm2gx" podStartSLOduration=4.274954768 podStartE2EDuration="25.442099768s" totalImagesPullingTime="21.167145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:11.442003554 +0000 UTC m=+25.650399699" watchObservedRunningTime="2026-07-20 13:17:11.442099768 +0000 UTC m=+25.650495912" Jul 20 13:17:12.303945 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:12.303742 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:12.304345 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:12.303772 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:12.304345 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:12.304070 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:17:12.304345 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:12.304126 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:17:13.303446 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:13.303250 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:13.303574 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:13.303475 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:17:13.429319 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:13.429293 2575 generic.go:356] "Generic (PLEG): container finished" podID="2d0cf929-460c-4fb9-bfd5-c311d138d89d" containerID="de05dc3667474bba65940e2e452bca149622db7c1f6bf20df2f61b8a2fe374c8" exitCode=0 Jul 20 13:17:13.430168 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:13.429378 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pstqq" event={"ID":"2d0cf929-460c-4fb9-bfd5-c311d138d89d","Type":"ContainerDied","Data":"de05dc3667474bba65940e2e452bca149622db7c1f6bf20df2f61b8a2fe374c8"} Jul 20 13:17:13.433274 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:13.433253 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" event={"ID":"8290b97d-bb82-4de4-b590-b95bd307d4bc","Type":"ContainerStarted","Data":"4e522f7489a2a48d5c5908fd0087d9077707338f1a18d9672dec054563cba1a0"} Jul 20 13:17:13.433532 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:13.433514 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:17:13.433611 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:13.433557 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:17:13.448327 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:13.448309 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:17:13.448630 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:13.448617 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:17:13.484641 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:13.484602 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" podStartSLOduration=8.676458481 podStartE2EDuration="27.484589168s" totalImagesPullingTime="18.808130687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:13.484320306 +0000 UTC m=+27.692716450" watchObservedRunningTime="2026-07-20 13:17:13.484589168 +0000 UTC m=+27.692985312" Jul 20 13:17:14.303012 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:14.302978 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:14.303236 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:14.302979 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:14.303236 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:14.303134 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:17:14.303236 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:14.303188 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:17:14.435105 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:14.435022 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 20 13:17:14.522767 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:14.522741 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lpksh" Jul 20 13:17:14.522938 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:14.522863 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 20 13:17:14.523335 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:14.523317 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lpksh" Jul 20 13:17:14.996485 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:14.996416 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:17:15.045829 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:15.045798 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-svhpp"] Jul 20 13:17:15.045991 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:15.045925 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:15.046033 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:15.046018 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:17:15.048009 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:15.047984 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wgvvv"] Jul 20 13:17:15.048130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:15.048065 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:15.048192 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:15.048128 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:17:15.051471 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:15.051451 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7sjzn"] Jul 20 13:17:15.051565 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:15.051541 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:15.051649 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:15.051627 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:17:15.438697 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:15.438619 2575 generic.go:356] "Generic (PLEG): container finished" podID="2d0cf929-460c-4fb9-bfd5-c311d138d89d" containerID="4573a9f576da16769a8cc232eb8f8795781fca9f3345e323b332a8155d83eafc" exitCode=0 Jul 20 13:17:15.438697 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:15.438684 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pstqq" event={"ID":"2d0cf929-460c-4fb9-bfd5-c311d138d89d","Type":"ContainerDied","Data":"4573a9f576da16769a8cc232eb8f8795781fca9f3345e323b332a8155d83eafc"} Jul 20 13:17:16.303483 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:16.303453 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:16.303642 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:16.303528 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:17:16.303642 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:16.303604 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:16.303732 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:16.303694 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:17:17.303110 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:17.302952 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:17.303493 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:17.303179 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:17:17.445650 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:17.445585 2575 generic.go:356] "Generic (PLEG): container finished" podID="2d0cf929-460c-4fb9-bfd5-c311d138d89d" containerID="f33f702f2995469f0bebae1640e7ebded83f80a6157d965718920d4c417b9383" exitCode=0 Jul 20 13:17:17.445650 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:17.445634 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pstqq" event={"ID":"2d0cf929-460c-4fb9-bfd5-c311d138d89d","Type":"ContainerDied","Data":"f33f702f2995469f0bebae1640e7ebded83f80a6157d965718920d4c417b9383"} Jul 20 13:17:18.303264 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:18.303232 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:18.303696 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:18.303365 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:17:18.303696 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:18.303405 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:18.303696 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:18.303485 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:17:19.303251 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:19.303173 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:19.303404 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:19.303359 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7sjzn" podUID="715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0" Jul 20 13:17:20.013176 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.013141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:20.013360 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.013187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:20.013360 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:20.013288 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 13:17:20.013451 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:20.013363 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret podName:715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:52.013342622 +0000 UTC m=+66.221738761 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret") pod "global-pull-secret-syncer-7sjzn" (UID: "715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0") : object "kube-system"/"original-pull-secret" not registered Jul 20 13:17:20.013451 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:20.013288 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:17:20.013451 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:20.013445 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs podName:c805bf30-7720-4274-81e1-26d75eb7ae55 nodeName:}" failed. No retries permitted until 2026-07-20 13:17:52.013427286 +0000 UTC m=+66.221823421 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs") pod "network-metrics-daemon-svhpp" (UID: "c805bf30-7720-4274-81e1-26d75eb7ae55") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 13:17:20.114118 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.114088 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vc6r\" (UniqueName: \"kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r\") pod \"network-check-target-wgvvv\" (UID: \"dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e\") " pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:20.114304 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:20.114254 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 13:17:20.114304 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:20.114270 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 13:17:20.114304 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:20.114279 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6vc6r for pod openshift-network-diagnostics/network-check-target-wgvvv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:17:20.114466 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:20.114330 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r podName:dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e nodeName:}" failed. No retries permitted until 2026-07-20 13:17:52.114312839 +0000 UTC m=+66.322708976 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6vc6r" (UniqueName: "kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r") pod "network-check-target-wgvvv" (UID: "dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 13:17:20.303089 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.303054 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:20.303089 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.303063 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:20.303311 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:20.303197 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:17:20.303311 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:20.303279 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wgvvv" podUID="dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e" Jul 20 13:17:20.580496 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.580422 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeReady" Jul 20 13:17:20.581056 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.580573 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 20 13:17:20.643770 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.643741 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ggrjj"] Jul 20 13:17:20.646368 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.646345 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:20.648243 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.648163 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nf4tc"] Jul 20 13:17:20.649309 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.649286 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 20 13:17:20.649825 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.649804 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nf4tc" Jul 20 13:17:20.652679 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.652651 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-49998\"" Jul 20 13:17:20.653333 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.653313 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 20 13:17:20.653425 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.653370 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gxzfq\"" Jul 20 13:17:20.653655 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.653628 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 20 13:17:20.653746 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.653675 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 20 13:17:20.653746 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.653727 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 20 13:17:20.659941 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.659922 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ggrjj"] Jul 20 13:17:20.663192 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.663172 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nf4tc"] Jul 20 13:17:20.719044 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.719017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-config-volume\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:20.719173 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.719052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95756\" (UniqueName: \"kubernetes.io/projected/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-kube-api-access-95756\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:20.719173 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.719126 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49ea05e3-10a6-4258-b9cb-121fa55007f1-cert\") pod \"ingress-canary-nf4tc\" (UID: \"49ea05e3-10a6-4258-b9cb-121fa55007f1\") " pod="openshift-ingress-canary/ingress-canary-nf4tc" Jul 20 13:17:20.719173 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.719158 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wclb\" (UniqueName: \"kubernetes.io/projected/49ea05e3-10a6-4258-b9cb-121fa55007f1-kube-api-access-8wclb\") pod \"ingress-canary-nf4tc\" (UID: \"49ea05e3-10a6-4258-b9cb-121fa55007f1\") " pod="openshift-ingress-canary/ingress-canary-nf4tc" Jul 20 13:17:20.719337 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.719257 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-tmp-dir\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:20.719391 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.719331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:20.820330 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.820300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49ea05e3-10a6-4258-b9cb-121fa55007f1-cert\") pod \"ingress-canary-nf4tc\" (UID: \"49ea05e3-10a6-4258-b9cb-121fa55007f1\") " pod="openshift-ingress-canary/ingress-canary-nf4tc" Jul 20 13:17:20.820330 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.820334 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wclb\" (UniqueName: \"kubernetes.io/projected/49ea05e3-10a6-4258-b9cb-121fa55007f1-kube-api-access-8wclb\") pod \"ingress-canary-nf4tc\" (UID: \"49ea05e3-10a6-4258-b9cb-121fa55007f1\") " pod="openshift-ingress-canary/ingress-canary-nf4tc" Jul 20 13:17:20.820557 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.820374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-tmp-dir\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:20.820557 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.820434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:20.820557 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.820464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-config-volume\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:20.820557 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.820494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95756\" (UniqueName: \"kubernetes.io/projected/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-kube-api-access-95756\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:20.820557 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:20.820544 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:20.820800 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:20.820603 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls podName:d11f1b29-aa51-45b0-8a5f-7d0887d7435c nodeName:}" failed. No retries permitted until 2026-07-20 13:17:21.320584993 +0000 UTC m=+35.528981133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls") pod "dns-default-ggrjj" (UID: "d11f1b29-aa51-45b0-8a5f-7d0887d7435c") : secret "dns-default-metrics-tls" not found Jul 20 13:17:20.820862 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.820794 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-tmp-dir\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:20.821129 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.821092 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-config-volume\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:20.825314 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.825294 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49ea05e3-10a6-4258-b9cb-121fa55007f1-cert\") pod \"ingress-canary-nf4tc\" (UID: \"49ea05e3-10a6-4258-b9cb-121fa55007f1\") " pod="openshift-ingress-canary/ingress-canary-nf4tc" Jul 20 13:17:20.838753 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.838704 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wclb\" (UniqueName: \"kubernetes.io/projected/49ea05e3-10a6-4258-b9cb-121fa55007f1-kube-api-access-8wclb\") pod \"ingress-canary-nf4tc\" (UID: \"49ea05e3-10a6-4258-b9cb-121fa55007f1\") " pod="openshift-ingress-canary/ingress-canary-nf4tc" Jul 20 13:17:20.839359 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.839339 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95756\" (UniqueName: \"kubernetes.io/projected/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-kube-api-access-95756\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:20.964415 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:20.964388 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nf4tc" Jul 20 13:17:21.303634 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:21.303471 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:21.306974 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:21.306924 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 13:17:21.324422 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:21.324395 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:21.324558 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:21.324491 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:21.324619 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:21.324559 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls podName:d11f1b29-aa51-45b0-8a5f-7d0887d7435c nodeName:}" failed. No retries permitted until 2026-07-20 13:17:22.324541151 +0000 UTC m=+36.532937275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls") pod "dns-default-ggrjj" (UID: "d11f1b29-aa51-45b0-8a5f-7d0887d7435c") : secret "dns-default-metrics-tls" not found Jul 20 13:17:21.390597 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:21.390471 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nf4tc"] Jul 20 13:17:21.393670 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:17:21.393642 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ea05e3_10a6_4258_b9cb_121fa55007f1.slice/crio-8ffff9735d18d06ee17a908c84a88c9dae967176e952819821c9085d302426a3 WatchSource:0}: Error finding container 8ffff9735d18d06ee17a908c84a88c9dae967176e952819821c9085d302426a3: Status 404 returned error can't find the container with id 8ffff9735d18d06ee17a908c84a88c9dae967176e952819821c9085d302426a3 Jul 20 13:17:21.457770 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:21.457669 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nf4tc" event={"ID":"49ea05e3-10a6-4258-b9cb-121fa55007f1","Type":"ContainerStarted","Data":"8ffff9735d18d06ee17a908c84a88c9dae967176e952819821c9085d302426a3"} Jul 20 13:17:21.462151 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:21.462128 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pstqq" event={"ID":"2d0cf929-460c-4fb9-bfd5-c311d138d89d","Type":"ContainerStarted","Data":"2833074672bdded94a9b4d84557ce5408b7ee1e43dfe423fe1794b814236a261"} Jul 20 13:17:22.303622 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:22.303589 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:22.304241 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:22.304185 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:22.310444 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:22.310350 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 13:17:22.310571 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:22.310493 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 13:17:22.312065 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:22.312039 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n79z9\"" Jul 20 13:17:22.312065 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:22.312059 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5q5pd\"" Jul 20 13:17:22.312314 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:22.312298 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 13:17:22.330955 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:22.330932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:22.331066 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:22.331052 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:22.331118 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:22.331105 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls podName:d11f1b29-aa51-45b0-8a5f-7d0887d7435c nodeName:}" failed. No retries permitted until 2026-07-20 13:17:24.331088815 +0000 UTC m=+38.539484941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls") pod "dns-default-ggrjj" (UID: "d11f1b29-aa51-45b0-8a5f-7d0887d7435c") : secret "dns-default-metrics-tls" not found Jul 20 13:17:22.470662 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:22.470625 2575 generic.go:356] "Generic (PLEG): container finished" podID="2d0cf929-460c-4fb9-bfd5-c311d138d89d" containerID="2833074672bdded94a9b4d84557ce5408b7ee1e43dfe423fe1794b814236a261" exitCode=0 Jul 20 13:17:22.470833 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:22.470685 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pstqq" event={"ID":"2d0cf929-460c-4fb9-bfd5-c311d138d89d","Type":"ContainerDied","Data":"2833074672bdded94a9b4d84557ce5408b7ee1e43dfe423fe1794b814236a261"} Jul 20 13:17:23.476181 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:23.476151 2575 generic.go:356] "Generic (PLEG): container finished" podID="2d0cf929-460c-4fb9-bfd5-c311d138d89d" containerID="97e35e3fc4780dcd28997946b2b94d621766d0b12e0c6905683ade4d8a09812b" exitCode=0 Jul 20 13:17:23.476508 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:23.476188 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pstqq" event={"ID":"2d0cf929-460c-4fb9-bfd5-c311d138d89d","Type":"ContainerDied","Data":"97e35e3fc4780dcd28997946b2b94d621766d0b12e0c6905683ade4d8a09812b"} Jul 20 13:17:24.344932 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:24.344740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:24.345106 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:24.344899 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:24.345106 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:24.345010 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls podName:d11f1b29-aa51-45b0-8a5f-7d0887d7435c nodeName:}" failed. No retries permitted until 2026-07-20 13:17:28.344996765 +0000 UTC m=+42.553392888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls") pod "dns-default-ggrjj" (UID: "d11f1b29-aa51-45b0-8a5f-7d0887d7435c") : secret "dns-default-metrics-tls" not found Jul 20 13:17:24.481716 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:24.481678 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pstqq" event={"ID":"2d0cf929-460c-4fb9-bfd5-c311d138d89d","Type":"ContainerStarted","Data":"d8f06e98fc0d14179d9bc3faf3de1515739f462caf6b7684b2f096a18a59f93e"} Jul 20 13:17:24.482960 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:24.482940 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nf4tc" event={"ID":"49ea05e3-10a6-4258-b9cb-121fa55007f1","Type":"ContainerStarted","Data":"10e016bbfe6a23a3fde82a766e3099a0171acb4a9f38299626629386fd1a29ec"} Jul 20 13:17:24.510024 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:24.509979 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pstqq" podStartSLOduration=9.494128138 podStartE2EDuration="38.509962876s" totalImagesPullingTime="29.015834738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:24.508197206 +0000 UTC m=+38.716593363" watchObservedRunningTime="2026-07-20 13:17:24.509962876 +0000 UTC m=+38.718359021" Jul 20 13:17:24.526485 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:24.526449 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nf4tc" podStartSLOduration=2.414727499 podStartE2EDuration="4.526438254s" totalImagesPullingTime="2.111710755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:24.526383419 +0000 UTC m=+38.734779564" watchObservedRunningTime="2026-07-20 13:17:24.526438254 +0000 UTC m=+38.734834399" Jul 20 13:17:28.370233 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:28.370201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:28.370733 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:28.370327 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:28.370733 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:28.370375 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls podName:d11f1b29-aa51-45b0-8a5f-7d0887d7435c nodeName:}" failed. No retries permitted until 2026-07-20 13:17:36.370360749 +0000 UTC m=+50.578756873 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls") pod "dns-default-ggrjj" (UID: "d11f1b29-aa51-45b0-8a5f-7d0887d7435c") : secret "dns-default-metrics-tls" not found Jul 20 13:17:36.416912 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:36.416858 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:36.417446 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:36.416994 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:36.417446 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:36.417064 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls podName:d11f1b29-aa51-45b0-8a5f-7d0887d7435c nodeName:}" failed. No retries permitted until 2026-07-20 13:17:52.417047803 +0000 UTC m=+66.625443929 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls") pod "dns-default-ggrjj" (UID: "d11f1b29-aa51-45b0-8a5f-7d0887d7435c") : secret "dns-default-metrics-tls" not found Jul 20 13:17:46.458994 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:46.458964 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-khqq5" Jul 20 13:17:52.019287 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.019246 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:17:52.019287 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.019289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:52.027046 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.027023 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 13:17:52.027186 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.027074 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 13:17:52.029583 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:52.029562 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 13:17:52.029647 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:52.029635 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs podName:c805bf30-7720-4274-81e1-26d75eb7ae55 nodeName:}" failed. No retries permitted until 2026-07-20 13:18:56.029615886 +0000 UTC m=+130.238012011 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs") pod "network-metrics-daemon-svhpp" (UID: "c805bf30-7720-4274-81e1-26d75eb7ae55") : secret "metrics-daemon-secret" not found Jul 20 13:17:52.033217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.033194 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0-original-pull-secret\") pod \"global-pull-secret-syncer-7sjzn\" (UID: \"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0\") " pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:52.120349 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.120325 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vc6r\" (UniqueName: \"kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r\") pod \"network-check-target-wgvvv\" (UID: \"dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e\") " pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:52.127271 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.127249 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 13:17:52.135203 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.135185 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 13:17:52.143312 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.143288 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vc6r\" (UniqueName: \"kubernetes.io/projected/dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e-kube-api-access-6vc6r\") pod \"network-check-target-wgvvv\" (UID: \"dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e\") " pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:52.219415 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.219394 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7sjzn" Jul 20 13:17:52.326352 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.326314 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5q5pd\"" Jul 20 13:17:52.331818 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.331795 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:52.384369 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.384338 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7sjzn"] Jul 20 13:17:52.422312 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.422286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:17:52.422475 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:52.422454 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:17:52.422535 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:17:52.422519 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls podName:d11f1b29-aa51-45b0-8a5f-7d0887d7435c nodeName:}" failed. No retries permitted until 2026-07-20 13:18:24.422500412 +0000 UTC m=+98.630896536 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls") pod "dns-default-ggrjj" (UID: "d11f1b29-aa51-45b0-8a5f-7d0887d7435c") : secret "dns-default-metrics-tls" not found Jul 20 13:17:52.470846 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.470812 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wgvvv"] Jul 20 13:17:52.474058 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:17:52.474029 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7120c9_5fe7_4bbe_b159_cf277b5a7e1e.slice/crio-8645d54a49db2beeef0e8e00cfa7965ab18d75a4818859fb07f214803f26534d WatchSource:0}: Error finding container 8645d54a49db2beeef0e8e00cfa7965ab18d75a4818859fb07f214803f26534d: Status 404 returned error can't find the container with id 8645d54a49db2beeef0e8e00cfa7965ab18d75a4818859fb07f214803f26534d Jul 20 13:17:52.546366 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.546305 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wgvvv" event={"ID":"dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e","Type":"ContainerStarted","Data":"8645d54a49db2beeef0e8e00cfa7965ab18d75a4818859fb07f214803f26534d"} Jul 20 13:17:52.547297 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:52.547278 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7sjzn" event={"ID":"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0","Type":"ContainerStarted","Data":"76002661030b4e4f83c63e5f62540dd171619ddb0fc261ea45ed0f9294249c2d"} Jul 20 13:17:57.561513 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:57.561472 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wgvvv" event={"ID":"dd7120c9-5fe7-4bbe-b159-cf277b5a7e1e","Type":"ContainerStarted","Data":"8c09f1cd89cdd7b21b0835956d6a39864b6b4b96eb19c1de0a8822211b5ca954"} Jul 20 13:17:57.562055 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:57.561617 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:17:57.563322 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:57.563293 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7sjzn" event={"ID":"715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0","Type":"ContainerStarted","Data":"0c17a60c81ac3a554308b1d7faa2bea822421790e4de70841bcaa746d7db0bcc"} Jul 20 13:17:57.587413 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:57.587363 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wgvvv" podStartSLOduration=67.257259047 podStartE2EDuration="1m11.587347989s" totalImagesPullingTime="4.330088942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:57.58723802 +0000 UTC m=+71.795634164" watchObservedRunningTime="2026-07-20 13:17:57.587347989 +0000 UTC m=+71.795744134" Jul 20 13:17:57.632423 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:17:57.632378 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7sjzn" podStartSLOduration=67.212803836 podStartE2EDuration="1m11.63236537s" totalImagesPullingTime="4.419561534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:17:57.632076009 +0000 UTC m=+71.840472155" watchObservedRunningTime="2026-07-20 13:17:57.63236537 +0000 UTC m=+71.840761573" Jul 20 13:18:10.195267 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.195113 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw"] Jul 20 13:18:10.199015 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.198996 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w"] Jul 20 13:18:10.199140 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.199125 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:18:10.201867 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.201847 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.209710 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.209690 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 20 13:18:10.209849 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.209823 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 20 13:18:10.209976 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.209893 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 20 13:18:10.209976 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.209905 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 20 13:18:10.209976 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.209823 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 20 13:18:10.209976 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.209968 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 20 13:18:10.210194 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.210162 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 20 13:18:10.210249 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.210198 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 20 13:18:10.227075 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.227057 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw"] Jul 20 13:18:10.227766 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.227723 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w"] Jul 20 13:18:10.349370 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.349334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ce76e14c-380c-4138-b859-fb1f54c7cdd2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.349486 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.349395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ce76e14c-380c-4138-b859-fb1f54c7cdd2-ca\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.349486 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.349435 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce76e14c-380c-4138-b859-fb1f54c7cdd2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.349486 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.349476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce76e14c-380c-4138-b859-fb1f54c7cdd2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.349661 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.349530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cggbx\" (UniqueName: \"kubernetes.io/projected/ce76e14c-380c-4138-b859-fb1f54c7cdd2-kube-api-access-cggbx\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.349661 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.349590 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ce76e14c-380c-4138-b859-fb1f54c7cdd2-hub\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.349661 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.349649 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92b09f68-ab3c-4391-bdf4-b10a574e57d7-tmp\") pod \"klusterlet-addon-workmgr-744cc94c65-tpmrw\" (UID: \"92b09f68-ab3c-4391-bdf4-b10a574e57d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:18:10.349812 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.349678 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49d6s\" (UniqueName: \"kubernetes.io/projected/92b09f68-ab3c-4391-bdf4-b10a574e57d7-kube-api-access-49d6s\") pod \"klusterlet-addon-workmgr-744cc94c65-tpmrw\" (UID: \"92b09f68-ab3c-4391-bdf4-b10a574e57d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:18:10.349812 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.349723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/92b09f68-ab3c-4391-bdf4-b10a574e57d7-klusterlet-config\") pod \"klusterlet-addon-workmgr-744cc94c65-tpmrw\" (UID: \"92b09f68-ab3c-4391-bdf4-b10a574e57d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:18:10.450785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.450727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ce76e14c-380c-4138-b859-fb1f54c7cdd2-ca\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.450785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.450757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce76e14c-380c-4138-b859-fb1f54c7cdd2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.450785 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.450776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce76e14c-380c-4138-b859-fb1f54c7cdd2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.450991 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.450800 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cggbx\" (UniqueName: \"kubernetes.io/projected/ce76e14c-380c-4138-b859-fb1f54c7cdd2-kube-api-access-cggbx\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.450991 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.450825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ce76e14c-380c-4138-b859-fb1f54c7cdd2-hub\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.451096 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.451010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92b09f68-ab3c-4391-bdf4-b10a574e57d7-tmp\") pod \"klusterlet-addon-workmgr-744cc94c65-tpmrw\" (UID: \"92b09f68-ab3c-4391-bdf4-b10a574e57d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:18:10.451096 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.451055 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49d6s\" (UniqueName: \"kubernetes.io/projected/92b09f68-ab3c-4391-bdf4-b10a574e57d7-kube-api-access-49d6s\") pod \"klusterlet-addon-workmgr-744cc94c65-tpmrw\" (UID: \"92b09f68-ab3c-4391-bdf4-b10a574e57d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:18:10.451193 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.451094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/92b09f68-ab3c-4391-bdf4-b10a574e57d7-klusterlet-config\") pod \"klusterlet-addon-workmgr-744cc94c65-tpmrw\" (UID: \"92b09f68-ab3c-4391-bdf4-b10a574e57d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:18:10.451193 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.451121 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ce76e14c-380c-4138-b859-fb1f54c7cdd2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.451484 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.451448 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92b09f68-ab3c-4391-bdf4-b10a574e57d7-tmp\") pod \"klusterlet-addon-workmgr-744cc94c65-tpmrw\" (UID: \"92b09f68-ab3c-4391-bdf4-b10a574e57d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:18:10.451576 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.451503 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce76e14c-380c-4138-b859-fb1f54c7cdd2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.453391 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.453371 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ce76e14c-380c-4138-b859-fb1f54c7cdd2-hub\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.453473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.453405 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ce76e14c-380c-4138-b859-fb1f54c7cdd2-ca\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.453632 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.453600 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/92b09f68-ab3c-4391-bdf4-b10a574e57d7-klusterlet-config\") pod \"klusterlet-addon-workmgr-744cc94c65-tpmrw\" (UID: \"92b09f68-ab3c-4391-bdf4-b10a574e57d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:18:10.453796 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.453782 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ce76e14c-380c-4138-b859-fb1f54c7cdd2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.453862 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.453842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce76e14c-380c-4138-b859-fb1f54c7cdd2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.472338 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.472315 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cggbx\" (UniqueName: \"kubernetes.io/projected/ce76e14c-380c-4138-b859-fb1f54c7cdd2-kube-api-access-cggbx\") pod \"cluster-proxy-proxy-agent-687cc4c7b5-drt7w\" (UID: \"ce76e14c-380c-4138-b859-fb1f54c7cdd2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.472588 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.472570 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49d6s\" (UniqueName: \"kubernetes.io/projected/92b09f68-ab3c-4391-bdf4-b10a574e57d7-kube-api-access-49d6s\") pod \"klusterlet-addon-workmgr-744cc94c65-tpmrw\" (UID: \"92b09f68-ab3c-4391-bdf4-b10a574e57d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:18:10.508553 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.508534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:18:10.527693 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.527676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:18:10.649769 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.649741 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw"] Jul 20 13:18:10.652383 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:18:10.652357 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b09f68_ab3c_4391_bdf4_b10a574e57d7.slice/crio-6af5af7013d66d2c80772ebed9d7bf3eea5f42ada2db50090c1e042bf4b8778c WatchSource:0}: Error finding container 6af5af7013d66d2c80772ebed9d7bf3eea5f42ada2db50090c1e042bf4b8778c: Status 404 returned error can't find the container with id 6af5af7013d66d2c80772ebed9d7bf3eea5f42ada2db50090c1e042bf4b8778c Jul 20 13:18:10.666041 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:10.666019 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w"] Jul 20 13:18:10.668648 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:18:10.668626 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce76e14c_380c_4138_b859_fb1f54c7cdd2.slice/crio-45ceb45ffa85cf2942c90c3372d278d58636af30c99dca2fa73eb91602b18ca7 WatchSource:0}: Error finding container 45ceb45ffa85cf2942c90c3372d278d58636af30c99dca2fa73eb91602b18ca7: Status 404 returned error can't find the container with id 45ceb45ffa85cf2942c90c3372d278d58636af30c99dca2fa73eb91602b18ca7 Jul 20 13:18:11.601356 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:11.601275 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" event={"ID":"ce76e14c-380c-4138-b859-fb1f54c7cdd2","Type":"ContainerStarted","Data":"45ceb45ffa85cf2942c90c3372d278d58636af30c99dca2fa73eb91602b18ca7"} Jul 20 13:18:11.602920 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:11.602885 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" event={"ID":"92b09f68-ab3c-4391-bdf4-b10a574e57d7","Type":"ContainerStarted","Data":"6af5af7013d66d2c80772ebed9d7bf3eea5f42ada2db50090c1e042bf4b8778c"} Jul 20 13:18:16.619119 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:16.619080 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" event={"ID":"ce76e14c-380c-4138-b859-fb1f54c7cdd2","Type":"ContainerStarted","Data":"a8293ea8e9fc0be1590f81090182f374ae6ebeb60f2a8164b9187f23b910fdb5"} Jul 20 13:18:16.619525 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:16.619131 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" event={"ID":"ce76e14c-380c-4138-b859-fb1f54c7cdd2","Type":"ContainerStarted","Data":"f86bd8090b6e46b393f3ab69c121acd72fcb83e57ba2f41e650d83da943086a5"} Jul 20 13:18:16.619525 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:16.619149 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" event={"ID":"ce76e14c-380c-4138-b859-fb1f54c7cdd2","Type":"ContainerStarted","Data":"301f55d9421cead33ef9c7ef724a6b1258362dcbc3dd11d4426201b5e5309c81"} Jul 20 13:18:16.620514 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:16.620490 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" event={"ID":"92b09f68-ab3c-4391-bdf4-b10a574e57d7","Type":"ContainerStarted","Data":"fc68172b0eb613df1d95daac271ba5c9c521b71f99fd9b28ec73e9dceeedc4f5"} Jul 20 13:18:16.620698 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:16.620682 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:18:16.622199 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:16.622182 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:18:16.645092 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:16.645053 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" podStartSLOduration=1.12927326 podStartE2EDuration="6.64503962s" totalImagesPullingTime="5.51576636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:18:16.64309152 +0000 UTC m=+90.851487679" watchObservedRunningTime="2026-07-20 13:18:16.64503962 +0000 UTC m=+90.853435743" Jul 20 13:18:16.660924 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:16.660865 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" podStartSLOduration=1.130659447 podStartE2EDuration="6.660855911s" totalImagesPullingTime="5.530196464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:18:16.660136384 +0000 UTC m=+90.868532531" watchObservedRunningTime="2026-07-20 13:18:16.660855911 +0000 UTC m=+90.869252056" Jul 20 13:18:24.444655 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:24.444613 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:18:24.445066 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:18:24.444765 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:18:24.445066 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:18:24.444842 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls podName:d11f1b29-aa51-45b0-8a5f-7d0887d7435c nodeName:}" failed. No retries permitted until 2026-07-20 13:19:28.444824145 +0000 UTC m=+162.653220268 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls") pod "dns-default-ggrjj" (UID: "d11f1b29-aa51-45b0-8a5f-7d0887d7435c") : secret "dns-default-metrics-tls" not found Jul 20 13:18:28.567798 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:28.567762 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wgvvv" Jul 20 13:18:56.049259 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:18:56.049205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:18:56.049751 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:18:56.049357 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 13:18:56.049751 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:18:56.049425 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs podName:c805bf30-7720-4274-81e1-26d75eb7ae55 nodeName:}" failed. No retries permitted until 2026-07-20 13:20:58.049407528 +0000 UTC m=+252.257803652 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs") pod "network-metrics-daemon-svhpp" (UID: "c805bf30-7720-4274-81e1-26d75eb7ae55") : secret "metrics-daemon-secret" not found Jul 20 13:19:23.658095 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:19:23.658053 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ggrjj" podUID="d11f1b29-aa51-45b0-8a5f-7d0887d7435c" Jul 20 13:19:23.816738 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:19:23.816709 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ggrjj" Jul 20 13:19:25.315819 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:19:25.315780 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:19:28.464345 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:19:28.464282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:19:28.464859 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:19:28.464437 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 13:19:28.464859 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:19:28.464518 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls podName:d11f1b29-aa51-45b0-8a5f-7d0887d7435c nodeName:}" failed. No retries permitted until 2026-07-20 13:21:30.464501825 +0000 UTC m=+284.672897948 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls") pod "dns-default-ggrjj" (UID: "d11f1b29-aa51-45b0-8a5f-7d0887d7435c") : secret "dns-default-metrics-tls" not found Jul 20 13:19:38.303289 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:19:38.303252 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:20:16.621441 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:20:16.621399 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" podUID="92b09f68-ab3c-4391-bdf4-b10a574e57d7" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.8:8000/readyz\": dial tcp 10.134.0.8:8000: connect: connection refused" Jul 20 13:20:16.962983 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:20:16.962896 2575 generic.go:356] "Generic (PLEG): container finished" podID="92b09f68-ab3c-4391-bdf4-b10a574e57d7" containerID="fc68172b0eb613df1d95daac271ba5c9c521b71f99fd9b28ec73e9dceeedc4f5" exitCode=1 Jul 20 13:20:16.962983 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:20:16.962965 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" event={"ID":"92b09f68-ab3c-4391-bdf4-b10a574e57d7","Type":"ContainerDied","Data":"fc68172b0eb613df1d95daac271ba5c9c521b71f99fd9b28ec73e9dceeedc4f5"} Jul 20 13:20:16.963248 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:20:16.963234 2575 scope.go:117] "RemoveContainer" containerID="fc68172b0eb613df1d95daac271ba5c9c521b71f99fd9b28ec73e9dceeedc4f5" Jul 20 13:20:17.968190 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:20:17.968142 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" event={"ID":"92b09f68-ab3c-4391-bdf4-b10a574e57d7","Type":"ContainerStarted","Data":"2a8b72fa7d8cbf3220008dbd48d5512aec75131ca78b7ca56f9dc1e58350a116"} Jul 20 13:20:17.968620 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:20:17.968423 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:20:17.969110 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:20:17.969087 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-744cc94c65-tpmrw" Jul 20 13:20:58.146209 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:20:58.146165 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:20:58.146715 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:20:58.146317 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 13:20:58.146715 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:20:58.146380 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs podName:c805bf30-7720-4274-81e1-26d75eb7ae55 nodeName:}" failed. No retries permitted until 2026-07-20 13:23:00.146364947 +0000 UTC m=+374.354761073 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs") pod "network-metrics-daemon-svhpp" (UID: "c805bf30-7720-4274-81e1-26d75eb7ae55") : secret "metrics-daemon-secret" not found Jul 20 13:21:04.920184 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:04.920151 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lslld"] Jul 20 13:21:04.922239 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:04.922222 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:04.925111 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:04.925084 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 20 13:21:04.925740 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:04.925723 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 20 13:21:04.925838 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:04.925763 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 20 13:21:04.926589 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:04.926573 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 20 13:21:04.926680 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:04.926585 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fctt2\"" Jul 20 13:21:04.948615 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:04.948595 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lslld"] Jul 20 13:21:05.089971 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.089943 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3fee729-e0f4-4922-856a-2d806439a2f9-data-volume\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.090097 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.089972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.090097 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.089997 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.090097 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.090019 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3fee729-e0f4-4922-856a-2d806439a2f9-crio-socket\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.090097 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.090072 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l49m5\" (UniqueName: \"kubernetes.io/projected/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-api-access-l49m5\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.190626 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.190563 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3fee729-e0f4-4922-856a-2d806439a2f9-data-volume\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.190626 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.190593 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.190626 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.190616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.190783 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.190641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3fee729-e0f4-4922-856a-2d806439a2f9-crio-socket\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.190783 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.190697 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l49m5\" (UniqueName: \"kubernetes.io/projected/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-api-access-l49m5\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.190783 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:05.190705 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:05.190783 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:05.190733 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:05.190931 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:05.190793 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm podName:f3fee729-e0f4-4922-856a-2d806439a2f9 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:05.690775888 +0000 UTC m=+259.899172011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lslld" (UID: "f3fee729-e0f4-4922-856a-2d806439a2f9") : configmap "kube-rbac-proxy" not found Jul 20 13:21:05.190931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.190798 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3fee729-e0f4-4922-856a-2d806439a2f9-crio-socket\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.190931 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:05.190814 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls podName:f3fee729-e0f4-4922-856a-2d806439a2f9 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:05.690804305 +0000 UTC m=+259.899200443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lslld" (UID: "f3fee729-e0f4-4922-856a-2d806439a2f9") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:05.190931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.190927 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3fee729-e0f4-4922-856a-2d806439a2f9-data-volume\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.203015 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.202988 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l49m5\" (UniqueName: \"kubernetes.io/projected/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-api-access-l49m5\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.694635 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.694605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.694809 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:05.694643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:05.694809 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:05.694774 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:05.694809 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:05.694789 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:05.695016 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:05.694855 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls podName:f3fee729-e0f4-4922-856a-2d806439a2f9 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:06.694831913 +0000 UTC m=+260.903228039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lslld" (UID: "f3fee729-e0f4-4922-856a-2d806439a2f9") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:05.695016 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:05.694897 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm podName:f3fee729-e0f4-4922-856a-2d806439a2f9 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:06.694865728 +0000 UTC m=+260.903261855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lslld" (UID: "f3fee729-e0f4-4922-856a-2d806439a2f9") : configmap "kube-rbac-proxy" not found Jul 20 13:21:06.701493 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:06.701454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:06.701493 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:06.701491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:06.702017 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:06.701569 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:06.702017 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:06.701625 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm podName:f3fee729-e0f4-4922-856a-2d806439a2f9 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:08.70161113 +0000 UTC m=+262.910007254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lslld" (UID: "f3fee729-e0f4-4922-856a-2d806439a2f9") : configmap "kube-rbac-proxy" not found Jul 20 13:21:06.702017 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:06.701629 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:06.702017 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:06.701708 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls podName:f3fee729-e0f4-4922-856a-2d806439a2f9 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:08.701688107 +0000 UTC m=+262.910084265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lslld" (UID: "f3fee729-e0f4-4922-856a-2d806439a2f9") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:07.694647 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:07.694613 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h8gtl_7bdea0d3-dbcd-4be1-bddc-49679ef918f1/dns-node-resolver/0.log" Jul 20 13:21:08.694823 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:08.694738 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-blhwm_02bda514-3256-495f-8ce5-ae9e7fe50c71/node-ca/0.log" Jul 20 13:21:08.712996 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:08.712970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:08.713080 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:08.713007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:08.713137 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:08.713084 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:08.713137 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:08.713130 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:08.713234 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:08.713138 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm podName:f3fee729-e0f4-4922-856a-2d806439a2f9 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:12.713126298 +0000 UTC m=+266.921522421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lslld" (UID: "f3fee729-e0f4-4922-856a-2d806439a2f9") : configmap "kube-rbac-proxy" not found Jul 20 13:21:08.713234 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:08.713207 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls podName:f3fee729-e0f4-4922-856a-2d806439a2f9 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:12.713190168 +0000 UTC m=+266.921586297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lslld" (UID: "f3fee729-e0f4-4922-856a-2d806439a2f9") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:09.695089 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:09.695053 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nf4tc_49ea05e3-10a6-4258-b9cb-121fa55007f1/serve-healthcheck-canary/0.log" Jul 20 13:21:12.739208 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:12.739164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:12.739208 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:12.739212 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:12.739814 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:12.739303 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 13:21:12.739814 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:12.739339 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 13:21:12.739814 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:12.739359 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm podName:f3fee729-e0f4-4922-856a-2d806439a2f9 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:20.739343932 +0000 UTC m=+274.947740055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lslld" (UID: "f3fee729-e0f4-4922-856a-2d806439a2f9") : configmap "kube-rbac-proxy" not found Jul 20 13:21:12.739814 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:12.739453 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls podName:f3fee729-e0f4-4922-856a-2d806439a2f9 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:20.739433408 +0000 UTC m=+274.947829535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lslld" (UID: "f3fee729-e0f4-4922-856a-2d806439a2f9") : secret "insights-runtime-extractor-tls" not found Jul 20 13:21:20.792031 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:20.791993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:20.792031 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:20.792036 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:20.792554 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:20.792523 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3fee729-e0f4-4922-856a-2d806439a2f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:20.794462 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:20.794440 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3fee729-e0f4-4922-856a-2d806439a2f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lslld\" (UID: \"f3fee729-e0f4-4922-856a-2d806439a2f9\") " pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:20.830592 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:20.830568 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lslld" Jul 20 13:21:20.958787 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:20.958759 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lslld"] Jul 20 13:21:20.961922 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:21:20.961898 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3fee729_e0f4_4922_856a_2d806439a2f9.slice/crio-d920df5f150734a2e046e2036b3c27f2ea6a8f00923d2299b300b5d302d60d69 WatchSource:0}: Error finding container d920df5f150734a2e046e2036b3c27f2ea6a8f00923d2299b300b5d302d60d69: Status 404 returned error can't find the container with id d920df5f150734a2e046e2036b3c27f2ea6a8f00923d2299b300b5d302d60d69 Jul 20 13:21:21.150026 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:21.149968 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lslld" event={"ID":"f3fee729-e0f4-4922-856a-2d806439a2f9","Type":"ContainerStarted","Data":"d920df5f150734a2e046e2036b3c27f2ea6a8f00923d2299b300b5d302d60d69"} Jul 20 13:21:26.167449 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:26.167414 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lslld" event={"ID":"f3fee729-e0f4-4922-856a-2d806439a2f9","Type":"ContainerStarted","Data":"abc2c9e96c4a5593761806aa7742dd83e5b65dffe59f198c6a7c8970b37c8085"} Jul 20 13:21:26.817241 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:26.817194 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ggrjj" podUID="d11f1b29-aa51-45b0-8a5f-7d0887d7435c" Jul 20 13:21:27.170683 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:27.170608 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ggrjj" Jul 20 13:21:30.182700 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:30.182666 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lslld" event={"ID":"f3fee729-e0f4-4922-856a-2d806439a2f9","Type":"ContainerStarted","Data":"3e5139d96d95442921bd1ba201951f19839cf49ed42147324c72672d55198540"} Jul 20 13:21:30.560605 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:30.560577 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:21:30.562856 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:30.562831 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d11f1b29-aa51-45b0-8a5f-7d0887d7435c-metrics-tls\") pod \"dns-default-ggrjj\" (UID: \"d11f1b29-aa51-45b0-8a5f-7d0887d7435c\") " pod="openshift-dns/dns-default-ggrjj" Jul 20 13:21:30.774592 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:30.774560 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-49998\"" Jul 20 13:21:30.782170 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:30.782147 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ggrjj" Jul 20 13:21:30.930907 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:30.930813 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ggrjj"] Jul 20 13:21:30.933041 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:21:30.933013 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11f1b29_aa51_45b0_8a5f_7d0887d7435c.slice/crio-c9362d510774f712959b19feb1fe59012dd38d52d43c0eb71f3278c6c265db81 WatchSource:0}: Error finding container c9362d510774f712959b19feb1fe59012dd38d52d43c0eb71f3278c6c265db81: Status 404 returned error can't find the container with id c9362d510774f712959b19feb1fe59012dd38d52d43c0eb71f3278c6c265db81 Jul 20 13:21:31.187734 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:31.187639 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ggrjj" event={"ID":"d11f1b29-aa51-45b0-8a5f-7d0887d7435c","Type":"ContainerStarted","Data":"c9362d510774f712959b19feb1fe59012dd38d52d43c0eb71f3278c6c265db81"} Jul 20 13:21:33.196242 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:33.196213 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lslld" event={"ID":"f3fee729-e0f4-4922-856a-2d806439a2f9","Type":"ContainerStarted","Data":"e7606aae7b96bfa873737b50f2191ff3e9e8536497d0d0de3a9398b6b1724b1c"} Jul 20 13:21:33.221087 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:33.221041 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lslld" podStartSLOduration=17.216909674 podStartE2EDuration="29.221025764s" totalImagesPullingTime="12.00411609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 13:21:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:33.219071531 +0000 UTC m=+287.427467690" watchObservedRunningTime="2026-07-20 13:21:33.221025764 +0000 UTC m=+287.429421907" Jul 20 13:21:35.231453 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:35.231425 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-bqzzp"] Jul 20 13:21:35.234629 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:35.234602 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-bqzzp" Jul 20 13:21:35.237594 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:35.237536 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-z4ss5\"" Jul 20 13:21:35.237734 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:35.237620 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 20 13:21:35.247417 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:35.247395 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-bqzzp"] Jul 20 13:21:35.294019 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:35.293992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/342f3190-251a-4f4d-8924-bd75ec13311d-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-bqzzp\" (UID: \"342f3190-251a-4f4d-8924-bd75ec13311d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-bqzzp" Jul 20 13:21:35.394851 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:35.394810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/342f3190-251a-4f4d-8924-bd75ec13311d-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-bqzzp\" (UID: \"342f3190-251a-4f4d-8924-bd75ec13311d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-bqzzp" Jul 20 13:21:35.395009 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:35.394976 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 20 13:21:35.395074 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:35.395056 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/342f3190-251a-4f4d-8924-bd75ec13311d-tls-certificates podName:342f3190-251a-4f4d-8924-bd75ec13311d nodeName:}" failed. No retries permitted until 2026-07-20 13:21:35.895039396 +0000 UTC m=+290.103435519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/342f3190-251a-4f4d-8924-bd75ec13311d-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-bqzzp" (UID: "342f3190-251a-4f4d-8924-bd75ec13311d") : secret "prometheus-operator-admission-webhook-tls" not found Jul 20 13:21:35.897905 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:35.897794 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/342f3190-251a-4f4d-8924-bd75ec13311d-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-bqzzp\" (UID: \"342f3190-251a-4f4d-8924-bd75ec13311d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-bqzzp" Jul 20 13:21:35.900202 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:35.900182 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/342f3190-251a-4f4d-8924-bd75ec13311d-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-bqzzp\" (UID: \"342f3190-251a-4f4d-8924-bd75ec13311d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-bqzzp" Jul 20 13:21:36.144438 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:36.144390 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-bqzzp" Jul 20 13:21:36.209178 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:36.209144 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ggrjj" event={"ID":"d11f1b29-aa51-45b0-8a5f-7d0887d7435c","Type":"ContainerStarted","Data":"af4b68e44d3e6fd845df01d7ccb22dbb13e286b5c540513fa7706d669d18ddeb"} Jul 20 13:21:36.209316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:36.209206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ggrjj" event={"ID":"d11f1b29-aa51-45b0-8a5f-7d0887d7435c","Type":"ContainerStarted","Data":"2f938d4f68fbb0608d78f0b6586e98d10b6b5607409260fbff322625ad753051"} Jul 20 13:21:36.209316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:36.209295 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ggrjj" Jul 20 13:21:36.239748 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:36.239705 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ggrjj" podStartSLOduration=251.557060938 podStartE2EDuration="4m16.239688608s" totalImagesPullingTime="4.68262767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:36.239154889 +0000 UTC m=+290.447551036" watchObservedRunningTime="2026-07-20 13:21:36.239688608 +0000 UTC m=+290.448084730" Jul 20 13:21:36.271134 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:36.271112 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-bqzzp"] Jul 20 13:21:36.272362 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:21:36.272340 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342f3190_251a_4f4d_8924_bd75ec13311d.slice/crio-95d7331dc796de1ef84085ffbca42ccf224283e769c3a58f833af30304bb63b9 WatchSource:0}: Error finding container 95d7331dc796de1ef84085ffbca42ccf224283e769c3a58f833af30304bb63b9: Status 404 returned error can't find the container with id 95d7331dc796de1ef84085ffbca42ccf224283e769c3a58f833af30304bb63b9 Jul 20 13:21:37.212708 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:37.212672 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-bqzzp" event={"ID":"342f3190-251a-4f4d-8924-bd75ec13311d","Type":"ContainerStarted","Data":"95d7331dc796de1ef84085ffbca42ccf224283e769c3a58f833af30304bb63b9"} Jul 20 13:21:38.217205 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:38.217169 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-bqzzp" event={"ID":"342f3190-251a-4f4d-8924-bd75ec13311d","Type":"ContainerStarted","Data":"edf1b23326d767fb3beb136e61b0f3b4f045140b8e27e98b90317d5fd83033b6"} Jul 20 13:21:38.217649 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:38.217433 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-bqzzp" Jul 20 13:21:38.221924 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:38.221905 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-bqzzp" Jul 20 13:21:38.236961 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:38.236912 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-bqzzp" podStartSLOduration=1.448899484 podStartE2EDuration="3.23689743s" totalImagesPullingTime="1.787997946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:38.235325727 +0000 UTC m=+292.443721898" watchObservedRunningTime="2026-07-20 13:21:38.23689743 +0000 UTC m=+292.445293573" Jul 20 13:21:39.345501 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.345471 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8"] Jul 20 13:21:39.347231 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.347215 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:39.351041 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.351022 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 20 13:21:39.352417 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.352400 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 20 13:21:39.352799 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.352771 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 20 13:21:39.352799 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.352785 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 20 13:21:39.352984 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.352802 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 20 13:21:39.352984 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.352774 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lv8wv\"" Jul 20 13:21:39.365752 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.365730 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8"] Jul 20 13:21:39.421656 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.421632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rzpw8\" (UID: \"b1061ffd-d7cc-45aa-8980-90454b6ad0b7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:39.421753 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.421657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zjh5\" (UniqueName: \"kubernetes.io/projected/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-kube-api-access-7zjh5\") pod \"prometheus-operator-5945b5cfd-rzpw8\" (UID: \"b1061ffd-d7cc-45aa-8980-90454b6ad0b7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:39.421753 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.421684 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-rzpw8\" (UID: \"b1061ffd-d7cc-45aa-8980-90454b6ad0b7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:39.421753 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.421705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-rzpw8\" (UID: \"b1061ffd-d7cc-45aa-8980-90454b6ad0b7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:39.522105 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.522084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-rzpw8\" (UID: \"b1061ffd-d7cc-45aa-8980-90454b6ad0b7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:39.522179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.522136 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rzpw8\" (UID: \"b1061ffd-d7cc-45aa-8980-90454b6ad0b7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:39.522179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.522158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zjh5\" (UniqueName: \"kubernetes.io/projected/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-kube-api-access-7zjh5\") pod \"prometheus-operator-5945b5cfd-rzpw8\" (UID: \"b1061ffd-d7cc-45aa-8980-90454b6ad0b7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:39.522246 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.522194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-rzpw8\" (UID: \"b1061ffd-d7cc-45aa-8980-90454b6ad0b7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:39.522279 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:39.522248 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 20 13:21:39.522325 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:39.522313 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-prometheus-operator-tls podName:b1061ffd-d7cc-45aa-8980-90454b6ad0b7 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:40.022294107 +0000 UTC m=+294.230690247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-rzpw8" (UID: "b1061ffd-d7cc-45aa-8980-90454b6ad0b7") : secret "prometheus-operator-tls" not found Jul 20 13:21:39.522683 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.522659 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-rzpw8\" (UID: \"b1061ffd-d7cc-45aa-8980-90454b6ad0b7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:39.524401 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.524384 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-rzpw8\" (UID: \"b1061ffd-d7cc-45aa-8980-90454b6ad0b7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:39.547093 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:39.547074 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zjh5\" (UniqueName: \"kubernetes.io/projected/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-kube-api-access-7zjh5\") pod \"prometheus-operator-5945b5cfd-rzpw8\" (UID: \"b1061ffd-d7cc-45aa-8980-90454b6ad0b7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:40.025222 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:40.025191 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rzpw8\" (UID: \"b1061ffd-d7cc-45aa-8980-90454b6ad0b7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:40.027387 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:40.027369 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1061ffd-d7cc-45aa-8980-90454b6ad0b7-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rzpw8\" (UID: \"b1061ffd-d7cc-45aa-8980-90454b6ad0b7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:40.255318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:40.255285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" Jul 20 13:21:40.378300 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:40.378270 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8"] Jul 20 13:21:40.381276 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:21:40.381241 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1061ffd_d7cc_45aa_8980_90454b6ad0b7.slice/crio-46039cb4f3d92503937972671c19e015f3bf8c8a1f15b867c85fd3d6975fa703 WatchSource:0}: Error finding container 46039cb4f3d92503937972671c19e015f3bf8c8a1f15b867c85fd3d6975fa703: Status 404 returned error can't find the container with id 46039cb4f3d92503937972671c19e015f3bf8c8a1f15b867c85fd3d6975fa703 Jul 20 13:21:40.528998 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:40.528965 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" podUID="ce76e14c-380c-4138-b859-fb1f54c7cdd2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 20 13:21:41.228944 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:41.228905 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" event={"ID":"b1061ffd-d7cc-45aa-8980-90454b6ad0b7","Type":"ContainerStarted","Data":"46039cb4f3d92503937972671c19e015f3bf8c8a1f15b867c85fd3d6975fa703"} Jul 20 13:21:41.304582 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:41.304548 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-svhpp" podUID="c805bf30-7720-4274-81e1-26d75eb7ae55" Jul 20 13:21:42.238245 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:42.234965 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" event={"ID":"b1061ffd-d7cc-45aa-8980-90454b6ad0b7","Type":"ContainerStarted","Data":"14641fb451c1a7d281e23d557ebada075bb299a9816d05340e5d1d3de7645279"} Jul 20 13:21:42.238245 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:42.235080 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" event={"ID":"b1061ffd-d7cc-45aa-8980-90454b6ad0b7","Type":"ContainerStarted","Data":"7f75790ed406c72c1f6b90990c2ed7e3c331a2706448d30d560f2817de60af7f"} Jul 20 13:21:42.258120 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:42.258075 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rzpw8" podStartSLOduration=2.024645569 podStartE2EDuration="3.25806s" totalImagesPullingTime="1.233414431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:42.257541024 +0000 UTC m=+296.465937169" watchObservedRunningTime="2026-07-20 13:21:42.25806 +0000 UTC m=+296.466456151" Jul 20 13:21:43.726178 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.726144 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7z7ts"] Jul 20 13:21:43.728248 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.728231 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.730981 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.730956 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nzg4g\"" Jul 20 13:21:43.731108 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.731085 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 20 13:21:43.731645 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.731626 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 20 13:21:43.731731 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.731654 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 20 13:21:43.851756 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.851731 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-textfile\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.851938 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.851762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-tls\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.851938 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.851790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29d7dae1-aff7-440c-9d24-6d46c71d84b5-sys\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.851938 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.851832 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p67w\" (UniqueName: \"kubernetes.io/projected/29d7dae1-aff7-440c-9d24-6d46c71d84b5-kube-api-access-9p67w\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.851938 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.851859 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.851938 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.851929 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29d7dae1-aff7-440c-9d24-6d46c71d84b5-metrics-client-ca\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.852096 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.851963 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29d7dae1-aff7-440c-9d24-6d46c71d84b5-root\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.852096 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.851991 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.852096 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.852013 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-wtmp\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953216 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953190 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29d7dae1-aff7-440c-9d24-6d46c71d84b5-metrics-client-ca\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953314 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29d7dae1-aff7-440c-9d24-6d46c71d84b5-root\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953314 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953314 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-wtmp\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953423 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-textfile\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953423 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953394 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29d7dae1-aff7-440c-9d24-6d46c71d84b5-root\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953525 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953507 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-wtmp\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953585 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953542 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-tls\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953650 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29d7dae1-aff7-440c-9d24-6d46c71d84b5-sys\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953650 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p67w\" (UniqueName: \"kubernetes.io/projected/29d7dae1-aff7-440c-9d24-6d46c71d84b5-kube-api-access-9p67w\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953750 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953653 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953750 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953679 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29d7dae1-aff7-440c-9d24-6d46c71d84b5-sys\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953750 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:43.953697 2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 20 13:21:43.953750 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953742 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-textfile\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953962 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:21:43.953755 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-tls podName:29d7dae1-aff7-440c-9d24-6d46c71d84b5 nodeName:}" failed. No retries permitted until 2026-07-20 13:21:44.453738031 +0000 UTC m=+298.662134175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-tls") pod "node-exporter-7z7ts" (UID: "29d7dae1-aff7-440c-9d24-6d46c71d84b5") : secret "node-exporter-tls" not found Jul 20 13:21:43.953962 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953830 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.953962 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.953843 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29d7dae1-aff7-440c-9d24-6d46c71d84b5-metrics-client-ca\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.956232 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.956209 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:43.967986 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:43.967958 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p67w\" (UniqueName: \"kubernetes.io/projected/29d7dae1-aff7-440c-9d24-6d46c71d84b5-kube-api-access-9p67w\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:44.458277 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:44.458248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-tls\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:44.460474 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:44.460454 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29d7dae1-aff7-440c-9d24-6d46c71d84b5-node-exporter-tls\") pod \"node-exporter-7z7ts\" (UID: \"29d7dae1-aff7-440c-9d24-6d46c71d84b5\") " pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:44.637223 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:44.637182 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7z7ts" Jul 20 13:21:44.648184 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:21:44.648158 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d7dae1_aff7_440c_9d24_6d46c71d84b5.slice/crio-36ee8ec643b9bd44946b831106eaf54a621e902bdae6873f626665650889dde7 WatchSource:0}: Error finding container 36ee8ec643b9bd44946b831106eaf54a621e902bdae6873f626665650889dde7: Status 404 returned error can't find the container with id 36ee8ec643b9bd44946b831106eaf54a621e902bdae6873f626665650889dde7 Jul 20 13:21:45.247204 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:45.247163 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7z7ts" event={"ID":"29d7dae1-aff7-440c-9d24-6d46c71d84b5","Type":"ContainerStarted","Data":"36ee8ec643b9bd44946b831106eaf54a621e902bdae6873f626665650889dde7"} Jul 20 13:21:46.144827 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:46.144799 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 20 13:21:46.214655 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:46.214631 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ggrjj" Jul 20 13:21:46.252015 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:46.251985 2575 generic.go:356] "Generic (PLEG): container finished" podID="29d7dae1-aff7-440c-9d24-6d46c71d84b5" containerID="554b4f6fe47c938b05c8a528c6e0df2c59fa4b7b957b80e1e62caebb5bb54d5c" exitCode=0 Jul 20 13:21:46.252322 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:46.252027 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7z7ts" event={"ID":"29d7dae1-aff7-440c-9d24-6d46c71d84b5","Type":"ContainerDied","Data":"554b4f6fe47c938b05c8a528c6e0df2c59fa4b7b957b80e1e62caebb5bb54d5c"} Jul 20 13:21:47.257704 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:47.257669 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7z7ts" event={"ID":"29d7dae1-aff7-440c-9d24-6d46c71d84b5","Type":"ContainerStarted","Data":"1dd5cf553d621b7b85f8f28a530e5b269c2fdbee511f88c6c149f50742316f7c"} Jul 20 13:21:47.258159 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:47.257725 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7z7ts" event={"ID":"29d7dae1-aff7-440c-9d24-6d46c71d84b5","Type":"ContainerStarted","Data":"77f8ebe8ce1bea1a1449da61ed3207d2a04c36f476238a8f461e845bc4c5645a"} Jul 20 13:21:47.303971 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:47.303916 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7z7ts" podStartSLOduration=3.38098545 podStartE2EDuration="4.303895813s" totalImagesPullingTime="861.387308ms" totalInitContainerRuntime="61.523055ms" isStatefulPod=true podCreationTimestamp="2026-07-20 13:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:21:47.303340807 +0000 UTC m=+301.511736951" watchObservedRunningTime="2026-07-20 13:21:47.303895813 +0000 UTC m=+301.512291962" Jul 20 13:21:50.043762 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.043723 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 13:21:50.049595 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.049563 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.057618 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.057598 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 20 13:21:50.057963 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.057944 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 20 13:21:50.058095 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.057944 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 20 13:21:50.058177 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.058152 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 20 13:21:50.058236 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.058197 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 20 13:21:50.058312 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.058294 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 20 13:21:50.058808 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.058669 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 20 13:21:50.058808 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.058774 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 20 13:21:50.059959 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.059179 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a72a2fdrqo31i\"" Jul 20 13:21:50.059959 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.059363 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-55rsm\"" Jul 20 13:21:50.059959 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.059525 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 20 13:21:50.059959 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.059626 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 20 13:21:50.059959 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.059697 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 20 13:21:50.059959 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.059812 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 20 13:21:50.060315 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.060297 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 20 13:21:50.060671 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.060649 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 20 13:21:50.061560 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.061541 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 20 13:21:50.066774 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.066756 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 13:21:50.099232 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099340 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099240 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099340 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099290 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099417 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099335 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3080fe9d-8969-4fda-a115-4e064efb2c62-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099417 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099369 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099480 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099453 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099512 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099482 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099546 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099578 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099549 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-web-config\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099610 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-config\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099642 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099607 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099642 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099704 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099663 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099737 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099714 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099769 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099742 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099769 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099764 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgdhw\" (UniqueName: \"kubernetes.io/projected/3080fe9d-8969-4fda-a115-4e064efb2c62-kube-api-access-cgdhw\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099784 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3080fe9d-8969-4fda-a115-4e064efb2c62-config-out\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099802 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099920 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.099920 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.099905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.200992 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.200963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201116 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201116 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgdhw\" (UniqueName: \"kubernetes.io/projected/3080fe9d-8969-4fda-a115-4e064efb2c62-kube-api-access-cgdhw\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201116 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3080fe9d-8969-4fda-a115-4e064efb2c62-config-out\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201275 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201275 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201216 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201275 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201432 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201399 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201490 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201490 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201604 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3080fe9d-8969-4fda-a115-4e064efb2c62-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201604 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201531 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201604 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201759 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201759 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201759 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-web-config\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201759 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-config\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201759 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.201759 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201753 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.202097 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.201784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.202233 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.202209 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.202318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.202294 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.202774 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.202748 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.203396 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.203370 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.204095 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.204073 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.204411 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.204391 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3080fe9d-8969-4fda-a115-4e064efb2c62-config-out\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.204532 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.204453 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.204832 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.204802 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.205669 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.205131 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.205669 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.205146 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3080fe9d-8969-4fda-a115-4e064efb2c62-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.205669 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.205496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.205669 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.205630 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.206397 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.206356 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.207281 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.207238 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.207281 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.207241 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.207543 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.207523 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-web-config\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.207824 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.207804 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.207868 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.207846 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3080fe9d-8969-4fda-a115-4e064efb2c62-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.208983 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.208960 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3080fe9d-8969-4fda-a115-4e064efb2c62-config\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.210971 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.210304 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgdhw\" (UniqueName: \"kubernetes.io/projected/3080fe9d-8969-4fda-a115-4e064efb2c62-kube-api-access-cgdhw\") pod \"prometheus-k8s-0\" (UID: \"3080fe9d-8969-4fda-a115-4e064efb2c62\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.360344 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.360267 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:21:50.511072 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.511050 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 13:21:50.513187 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:21:50.513160 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3080fe9d_8969_4fda_a115_4e064efb2c62.slice/crio-96c9ace67e795a52ba5d2df39295b37746729869477644cf3cd64a7acdba94b8 WatchSource:0}: Error finding container 96c9ace67e795a52ba5d2df39295b37746729869477644cf3cd64a7acdba94b8: Status 404 returned error can't find the container with id 96c9ace67e795a52ba5d2df39295b37746729869477644cf3cd64a7acdba94b8 Jul 20 13:21:50.515339 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.515322 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 13:21:50.529286 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:50.529257 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" podUID="ce76e14c-380c-4138-b859-fb1f54c7cdd2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 20 13:21:51.272360 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:51.272318 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3080fe9d-8969-4fda-a115-4e064efb2c62","Type":"ContainerStarted","Data":"96c9ace67e795a52ba5d2df39295b37746729869477644cf3cd64a7acdba94b8"} Jul 20 13:21:52.277268 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:52.277233 2575 generic.go:356] "Generic (PLEG): container finished" podID="3080fe9d-8969-4fda-a115-4e064efb2c62" containerID="7ea2a2efdcb3fa0b9d49bd7d7708b50d6f758b6f00645b6ba14fc201004d0a6e" exitCode=0 Jul 20 13:21:52.277696 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:52.277341 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3080fe9d-8969-4fda-a115-4e064efb2c62","Type":"ContainerDied","Data":"7ea2a2efdcb3fa0b9d49bd7d7708b50d6f758b6f00645b6ba14fc201004d0a6e"} Jul 20 13:21:52.303546 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:52.303525 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:21:57.311459 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:57.311428 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3080fe9d-8969-4fda-a115-4e064efb2c62","Type":"ContainerStarted","Data":"a4725541fd60e9c68e0db1f2a12b752cdfe0070367b665c2a5c532008df87ba1"} Jul 20 13:21:57.311802 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:57.311490 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3080fe9d-8969-4fda-a115-4e064efb2c62","Type":"ContainerStarted","Data":"17ef77132d818d26dd928c15ab52c5ca14ecb37f23bc54c6db2e7c06585e6294"} Jul 20 13:21:59.322459 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:21:59.322423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3080fe9d-8969-4fda-a115-4e064efb2c62","Type":"ContainerStarted","Data":"492ff2e40255938a793fb959cc3da3b05a4afade7afbf4721805f352216e9adf"} Jul 20 13:22:00.329631 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:00.329589 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3080fe9d-8969-4fda-a115-4e064efb2c62","Type":"ContainerStarted","Data":"f3ec8e5aa3242f2fa33fe948e5243d24f91d63d6f37ca38c3e1bd3f29aa5ad40"} Jul 20 13:22:00.330075 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:00.329659 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3080fe9d-8969-4fda-a115-4e064efb2c62","Type":"ContainerStarted","Data":"a63e62922ceeba271c3a2a8f26464412897e58b893c909a89b77a67aaba853df"} Jul 20 13:22:00.330075 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:00.329689 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3080fe9d-8969-4fda-a115-4e064efb2c62","Type":"ContainerStarted","Data":"d02b8abe651aff716c7ff50ddcaf9e44e8f36b73b40d6b76878edf1561a606bd"} Jul 20 13:22:00.360809 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:00.360787 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:22:00.365772 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:00.365724 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.623712911 podStartE2EDuration="10.36571157s" totalImagesPullingTime="7.721196419s" totalInitContainerRuntime="20.80224ms" isStatefulPod=false podCreationTimestamp="2026-07-20 13:21:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:22:00.363116264 +0000 UTC m=+314.571512408" watchObservedRunningTime="2026-07-20 13:22:00.36571157 +0000 UTC m=+314.574107715" Jul 20 13:22:00.529477 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:00.529444 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" podUID="ce76e14c-380c-4138-b859-fb1f54c7cdd2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 20 13:22:00.529615 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:00.529505 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" Jul 20 13:22:00.530061 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:00.530032 2575 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"a8293ea8e9fc0be1590f81090182f374ae6ebeb60f2a8164b9187f23b910fdb5"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 20 13:22:00.530105 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:00.530084 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" podUID="ce76e14c-380c-4138-b859-fb1f54c7cdd2" containerName="service-proxy" containerID="cri-o://a8293ea8e9fc0be1590f81090182f374ae6ebeb60f2a8164b9187f23b910fdb5" gracePeriod=30 Jul 20 13:22:01.337341 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:01.337303 2575 generic.go:356] "Generic (PLEG): container finished" podID="ce76e14c-380c-4138-b859-fb1f54c7cdd2" containerID="a8293ea8e9fc0be1590f81090182f374ae6ebeb60f2a8164b9187f23b910fdb5" exitCode=2 Jul 20 13:22:01.337716 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:01.337376 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" event={"ID":"ce76e14c-380c-4138-b859-fb1f54c7cdd2","Type":"ContainerDied","Data":"a8293ea8e9fc0be1590f81090182f374ae6ebeb60f2a8164b9187f23b910fdb5"} Jul 20 13:22:01.337716 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:01.337414 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-687cc4c7b5-drt7w" event={"ID":"ce76e14c-380c-4138-b859-fb1f54c7cdd2","Type":"ContainerStarted","Data":"881e439cdad14c0ea67aa132c67d068bb2d13daebef0b952da6975df11e69c16"} Jul 20 13:22:50.361288 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:50.361211 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:22:50.380901 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:50.380861 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:22:50.520602 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:22:50.520574 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 13:23:00.229354 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:23:00.229318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:23:00.231636 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:23:00.231612 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c805bf30-7720-4274-81e1-26d75eb7ae55-metrics-certs\") pod \"network-metrics-daemon-svhpp\" (UID: \"c805bf30-7720-4274-81e1-26d75eb7ae55\") " pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:23:00.410648 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:23:00.410622 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n79z9\"" Jul 20 13:23:00.415361 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:23:00.415345 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhpp" Jul 20 13:23:00.539495 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:23:00.539470 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-svhpp"] Jul 20 13:23:00.541929 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:23:00.541899 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc805bf30_7720_4274_81e1_26d75eb7ae55.slice/crio-b6a8fd6bc639fed6c69c0086d53fc0e86a25097c7c3d3514530a2241fd021b87 WatchSource:0}: Error finding container b6a8fd6bc639fed6c69c0086d53fc0e86a25097c7c3d3514530a2241fd021b87: Status 404 returned error can't find the container with id b6a8fd6bc639fed6c69c0086d53fc0e86a25097c7c3d3514530a2241fd021b87 Jul 20 13:23:01.544421 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:23:01.544394 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svhpp" event={"ID":"c805bf30-7720-4274-81e1-26d75eb7ae55","Type":"ContainerStarted","Data":"3868a1cd8fab8737df7049acdb7474562ee977da78db83ed1daa669193457bbc"} Jul 20 13:23:01.544757 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:23:01.544440 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svhpp" event={"ID":"c805bf30-7720-4274-81e1-26d75eb7ae55","Type":"ContainerStarted","Data":"f0e54835c70934abd372fd263a7ccf389e8f43f080e14380b508c1a15bcdde7a"} Jul 20 13:23:01.544757 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:23:01.544455 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svhpp" event={"ID":"c805bf30-7720-4274-81e1-26d75eb7ae55","Type":"ContainerStarted","Data":"b6a8fd6bc639fed6c69c0086d53fc0e86a25097c7c3d3514530a2241fd021b87"} Jul 20 13:23:01.564713 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:23:01.564668 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-svhpp" podStartSLOduration=374.751216515 podStartE2EDuration="6m15.564649039s" totalImagesPullingTime="813.432524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:23:01.563294292 +0000 UTC m=+375.771690447" watchObservedRunningTime="2026-07-20 13:23:01.564649039 +0000 UTC m=+375.773045183" Jul 20 13:25:35.386018 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.385976 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr"] Jul 20 13:25:35.389417 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.389397 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.393728 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.393706 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 20 13:25:35.393833 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.393759 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 20 13:25:35.394239 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.394223 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ggvq6\"" Jul 20 13:25:35.394608 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.394587 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 20 13:25:35.394958 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.394945 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 20 13:25:35.408047 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.408021 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr"] Jul 20 13:25:35.482003 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.481974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7pwc\" (UniqueName: \"kubernetes.io/projected/28497bde-8176-4794-a41c-edd6cee61231-kube-api-access-n7pwc\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.482140 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.482022 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28497bde-8176-4794-a41c-edd6cee61231-webhook-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.482140 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.482118 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28497bde-8176-4794-a41c-edd6cee61231-manifests\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.482239 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.482157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28497bde-8176-4794-a41c-edd6cee61231-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.482239 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.482213 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28497bde-8176-4794-a41c-edd6cee61231-tmp\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.582788 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.582762 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7pwc\" (UniqueName: \"kubernetes.io/projected/28497bde-8176-4794-a41c-edd6cee61231-kube-api-access-n7pwc\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.582949 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.582809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28497bde-8176-4794-a41c-edd6cee61231-webhook-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.582949 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.582836 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28497bde-8176-4794-a41c-edd6cee61231-manifests\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.583057 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.582953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28497bde-8176-4794-a41c-edd6cee61231-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.583057 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.582997 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28497bde-8176-4794-a41c-edd6cee61231-tmp\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.583253 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.583230 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28497bde-8176-4794-a41c-edd6cee61231-manifests\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.585104 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.585079 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28497bde-8176-4794-a41c-edd6cee61231-tmp\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.585191 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.585167 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28497bde-8176-4794-a41c-edd6cee61231-webhook-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.585246 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.585219 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28497bde-8176-4794-a41c-edd6cee61231-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.591516 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.591487 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7pwc\" (UniqueName: \"kubernetes.io/projected/28497bde-8176-4794-a41c-edd6cee61231-kube-api-access-n7pwc\") pod \"opendatahub-operator-controller-manager-864d7fc49d-zw5kr\" (UID: \"28497bde-8176-4794-a41c-edd6cee61231\") " pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.699368 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.699312 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:35.844551 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:35.844528 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr"] Jul 20 13:25:35.846538 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:25:35.846513 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28497bde_8176_4794_a41c_edd6cee61231.slice/crio-8fdf2be2a44fe3d1ae99e1de35ae02e2413757b97ce5e9dd694ac9ecee0968ed WatchSource:0}: Error finding container 8fdf2be2a44fe3d1ae99e1de35ae02e2413757b97ce5e9dd694ac9ecee0968ed: Status 404 returned error can't find the container with id 8fdf2be2a44fe3d1ae99e1de35ae02e2413757b97ce5e9dd694ac9ecee0968ed Jul 20 13:25:36.084780 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:36.084744 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" event={"ID":"28497bde-8176-4794-a41c-edd6cee61231","Type":"ContainerStarted","Data":"8fdf2be2a44fe3d1ae99e1de35ae02e2413757b97ce5e9dd694ac9ecee0968ed"} Jul 20 13:25:39.098638 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:39.098573 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" event={"ID":"28497bde-8176-4794-a41c-edd6cee61231","Type":"ContainerStarted","Data":"e8acbe3b6c0e9e3357b20dc34b4816fde2be3f07722d230934261ec56f74578b"} Jul 20 13:25:40.103699 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:40.103663 2575 generic.go:356] "Generic (PLEG): container finished" podID="28497bde-8176-4794-a41c-edd6cee61231" containerID="e8acbe3b6c0e9e3357b20dc34b4816fde2be3f07722d230934261ec56f74578b" exitCode=0 Jul 20 13:25:40.104089 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:40.103750 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" event={"ID":"28497bde-8176-4794-a41c-edd6cee61231","Type":"ContainerDied","Data":"e8acbe3b6c0e9e3357b20dc34b4816fde2be3f07722d230934261ec56f74578b"} Jul 20 13:25:41.109389 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:41.109359 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" event={"ID":"28497bde-8176-4794-a41c-edd6cee61231","Type":"ContainerStarted","Data":"54cfdd92e67ef80c671781db6c20a6fb14b4643255be7338a375886cd7dcb331"} Jul 20 13:25:41.109817 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:41.109490 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:25:41.164546 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:41.164500 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" podStartSLOduration=1.5931880569999999 podStartE2EDuration="6.164482884s" totalImagesPullingTime="3.5253831s" totalInitContainerRuntime="1.045911727s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:25:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:25:41.162293033 +0000 UTC m=+535.370689178" watchObservedRunningTime="2026-07-20 13:25:41.164482884 +0000 UTC m=+535.372879028" Jul 20 13:25:52.115597 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:25:52.115526 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-864d7fc49d-zw5kr" Jul 20 13:26:03.206329 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.206297 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s"] Jul 20 13:26:03.220368 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.220348 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.224175 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.224155 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 20 13:26:03.224325 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.224307 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 20 13:26:03.224811 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.224790 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-s4zk9\"" Jul 20 13:26:03.224954 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.224932 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 20 13:26:03.225056 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.225005 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 20 13:26:03.225056 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.225005 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 20 13:26:03.225382 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.225368 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 20 13:26:03.245900 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.245861 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s"] Jul 20 13:26:03.392387 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.392354 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.392539 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.392393 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.392539 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.392411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.392539 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.392479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfzvp\" (UniqueName: \"kubernetes.io/projected/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-kube-api-access-wfzvp\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.392652 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.392570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.392652 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.392599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.392652 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.392621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.493993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.493920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.493993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.493952 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.493993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.493984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfzvp\" (UniqueName: \"kubernetes.io/projected/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-kube-api-access-wfzvp\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.494223 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.494049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.494223 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.494081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.494223 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.494111 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.494223 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.494156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.495183 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.495154 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.496177 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.496152 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.496446 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.496421 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.496493 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.496473 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.496493 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.496485 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.505416 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.505394 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.505582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.505565 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfzvp\" (UniqueName: \"kubernetes.io/projected/b79abb9c-cf1a-413e-90c8-bcd61c9482cc-kube-api-access-wfzvp\") pod \"istiod-openshift-gateway-94bb8fbfd-rmv5s\" (UID: \"b79abb9c-cf1a-413e-90c8-bcd61c9482cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.529372 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.529352 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:03.669016 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:03.668991 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s"] Jul 20 13:26:03.671860 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:26:03.671832 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79abb9c_cf1a_413e_90c8_bcd61c9482cc.slice/crio-7da6411511431755f26ff51d2e6491d78386680320da920ae13841b1825849e5 WatchSource:0}: Error finding container 7da6411511431755f26ff51d2e6491d78386680320da920ae13841b1825849e5: Status 404 returned error can't find the container with id 7da6411511431755f26ff51d2e6491d78386680320da920ae13841b1825849e5 Jul 20 13:26:04.204630 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:04.204589 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" event={"ID":"b79abb9c-cf1a-413e-90c8-bcd61c9482cc","Type":"ContainerStarted","Data":"7da6411511431755f26ff51d2e6491d78386680320da920ae13841b1825849e5"} Jul 20 13:26:05.625597 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.625563 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-548487689c-n5v9s"] Jul 20 13:26:05.632549 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.632530 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:05.639544 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.639500 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 20 13:26:05.640207 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.640184 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:26:05.640318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.640199 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 20 13:26:05.640318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.640239 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fwswp\"" Jul 20 13:26:05.640318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.640300 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 20 13:26:05.641259 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.640841 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 20 13:26:05.704790 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.704766 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-548487689c-n5v9s"] Jul 20 13:26:05.711673 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.711649 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62p64\" (UniqueName: \"kubernetes.io/projected/509d57b7-3242-438d-970e-3a9593523ee9-kube-api-access-62p64\") pod \"lws-controller-manager-548487689c-n5v9s\" (UID: \"509d57b7-3242-438d-970e-3a9593523ee9\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:05.711809 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.711692 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/509d57b7-3242-438d-970e-3a9593523ee9-cert\") pod \"lws-controller-manager-548487689c-n5v9s\" (UID: \"509d57b7-3242-438d-970e-3a9593523ee9\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:05.711809 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.711716 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/509d57b7-3242-438d-970e-3a9593523ee9-metrics-cert\") pod \"lws-controller-manager-548487689c-n5v9s\" (UID: \"509d57b7-3242-438d-970e-3a9593523ee9\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:05.711958 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.711800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/509d57b7-3242-438d-970e-3a9593523ee9-manager-config\") pod \"lws-controller-manager-548487689c-n5v9s\" (UID: \"509d57b7-3242-438d-970e-3a9593523ee9\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:05.812920 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.812882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/509d57b7-3242-438d-970e-3a9593523ee9-manager-config\") pod \"lws-controller-manager-548487689c-n5v9s\" (UID: \"509d57b7-3242-438d-970e-3a9593523ee9\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:05.813134 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.813098 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62p64\" (UniqueName: \"kubernetes.io/projected/509d57b7-3242-438d-970e-3a9593523ee9-kube-api-access-62p64\") pod \"lws-controller-manager-548487689c-n5v9s\" (UID: \"509d57b7-3242-438d-970e-3a9593523ee9\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:05.813215 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.813163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/509d57b7-3242-438d-970e-3a9593523ee9-cert\") pod \"lws-controller-manager-548487689c-n5v9s\" (UID: \"509d57b7-3242-438d-970e-3a9593523ee9\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:05.813215 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.813198 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/509d57b7-3242-438d-970e-3a9593523ee9-metrics-cert\") pod \"lws-controller-manager-548487689c-n5v9s\" (UID: \"509d57b7-3242-438d-970e-3a9593523ee9\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:05.813739 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.813710 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/509d57b7-3242-438d-970e-3a9593523ee9-manager-config\") pod \"lws-controller-manager-548487689c-n5v9s\" (UID: \"509d57b7-3242-438d-970e-3a9593523ee9\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:05.815971 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.815950 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/509d57b7-3242-438d-970e-3a9593523ee9-cert\") pod \"lws-controller-manager-548487689c-n5v9s\" (UID: \"509d57b7-3242-438d-970e-3a9593523ee9\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:05.816118 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.816097 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/509d57b7-3242-438d-970e-3a9593523ee9-metrics-cert\") pod \"lws-controller-manager-548487689c-n5v9s\" (UID: \"509d57b7-3242-438d-970e-3a9593523ee9\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:05.842086 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.842058 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62p64\" (UniqueName: \"kubernetes.io/projected/509d57b7-3242-438d-970e-3a9593523ee9-kube-api-access-62p64\") pod \"lws-controller-manager-548487689c-n5v9s\" (UID: \"509d57b7-3242-438d-970e-3a9593523ee9\") " pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:05.944537 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:05.944455 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:06.132385 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:06.132345 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:26:06.132473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:06.132413 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 13:26:06.206317 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:26:06.206288 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509d57b7_3242_438d_970e_3a9593523ee9.slice/crio-6260ab90c1bc3c45f3d0a809fc7390352b40ddc67a401e0dfbde7264ad5efb08 WatchSource:0}: Error finding container 6260ab90c1bc3c45f3d0a809fc7390352b40ddc67a401e0dfbde7264ad5efb08: Status 404 returned error can't find the container with id 6260ab90c1bc3c45f3d0a809fc7390352b40ddc67a401e0dfbde7264ad5efb08 Jul 20 13:26:06.208087 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:06.208058 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-548487689c-n5v9s"] Jul 20 13:26:06.213177 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:06.213141 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" event={"ID":"b79abb9c-cf1a-413e-90c8-bcd61c9482cc","Type":"ContainerStarted","Data":"ba493110f086a6c86dc38ee40766fb6754512cdff7aadc732d9ed3f654a38560"} Jul 20 13:26:06.213272 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:06.213234 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:06.214826 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:06.214806 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" event={"ID":"509d57b7-3242-438d-970e-3a9593523ee9","Type":"ContainerStarted","Data":"6260ab90c1bc3c45f3d0a809fc7390352b40ddc67a401e0dfbde7264ad5efb08"} Jul 20 13:26:06.301058 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:06.301009 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" podStartSLOduration=0.842818188 podStartE2EDuration="3.300994547s" totalImagesPullingTime="2.458176359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:26:06.294133227 +0000 UTC m=+560.502529371" watchObservedRunningTime="2026-07-20 13:26:06.300994547 +0000 UTC m=+560.509390691" Jul 20 13:26:07.221378 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:07.221319 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rmv5s" Jul 20 13:26:09.236306 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:09.236272 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" event={"ID":"509d57b7-3242-438d-970e-3a9593523ee9","Type":"ContainerStarted","Data":"61ab00d66c51d4393c3d0f4c91819bd13cfbb27a001a0eddfdd52f9ce3f96e01"} Jul 20 13:26:09.236726 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:09.236401 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:26:09.261343 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:09.261302 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" podStartSLOduration=1.8325590090000001 podStartE2EDuration="4.261290517s" totalImagesPullingTime="2.428731508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:26:09.2603453 +0000 UTC m=+563.468741444" watchObservedRunningTime="2026-07-20 13:26:09.261290517 +0000 UTC m=+563.469686661" Jul 20 13:26:20.243458 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:26:20.243426 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-548487689c-n5v9s" Jul 20 13:28:41.772574 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:41.772498 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:28:41.775732 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:41.775713 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-bppwn" Jul 20 13:28:41.778192 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:41.778164 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 20 13:28:41.778301 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:41.778169 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mrxcc\"" Jul 20 13:28:41.778573 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:41.778553 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 13:28:41.779376 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:41.779357 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 13:28:41.786109 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:41.786089 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:28:41.802268 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:41.802249 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:28:41.857673 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:41.857646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghj88\" (UniqueName: \"kubernetes.io/projected/e018dae8-c0db-4060-9f8d-c7d3e0f35223-kube-api-access-ghj88\") pod \"limitador-limitador-54d6d988d5-bppwn\" (UID: \"e018dae8-c0db-4060-9f8d-c7d3e0f35223\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bppwn" Jul 20 13:28:41.857762 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:41.857674 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e018dae8-c0db-4060-9f8d-c7d3e0f35223-config-file\") pod \"limitador-limitador-54d6d988d5-bppwn\" (UID: \"e018dae8-c0db-4060-9f8d-c7d3e0f35223\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bppwn" Jul 20 13:28:41.958990 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:41.958961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghj88\" (UniqueName: \"kubernetes.io/projected/e018dae8-c0db-4060-9f8d-c7d3e0f35223-kube-api-access-ghj88\") pod \"limitador-limitador-54d6d988d5-bppwn\" (UID: \"e018dae8-c0db-4060-9f8d-c7d3e0f35223\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bppwn" Jul 20 13:28:41.959078 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:41.958996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e018dae8-c0db-4060-9f8d-c7d3e0f35223-config-file\") pod \"limitador-limitador-54d6d988d5-bppwn\" (UID: \"e018dae8-c0db-4060-9f8d-c7d3e0f35223\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bppwn" Jul 20 13:28:41.959629 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:41.959610 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e018dae8-c0db-4060-9f8d-c7d3e0f35223-config-file\") pod \"limitador-limitador-54d6d988d5-bppwn\" (UID: \"e018dae8-c0db-4060-9f8d-c7d3e0f35223\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bppwn" Jul 20 13:28:41.967819 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:41.967792 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghj88\" (UniqueName: \"kubernetes.io/projected/e018dae8-c0db-4060-9f8d-c7d3e0f35223-kube-api-access-ghj88\") pod \"limitador-limitador-54d6d988d5-bppwn\" (UID: \"e018dae8-c0db-4060-9f8d-c7d3e0f35223\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bppwn" Jul 20 13:28:42.087112 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:42.087062 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-bppwn" Jul 20 13:28:42.210234 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:42.210212 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:28:42.212780 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:28:42.212753 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode018dae8_c0db_4060_9f8d_c7d3e0f35223.slice/crio-d8b011ba5640e75db1989584e62678e0259f6a8ea8b4f4646a0b81472f6d539d WatchSource:0}: Error finding container d8b011ba5640e75db1989584e62678e0259f6a8ea8b4f4646a0b81472f6d539d: Status 404 returned error can't find the container with id d8b011ba5640e75db1989584e62678e0259f6a8ea8b4f4646a0b81472f6d539d Jul 20 13:28:42.214690 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:42.214675 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 13:28:42.791606 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:42.791517 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-bppwn" event={"ID":"e018dae8-c0db-4060-9f8d-c7d3e0f35223","Type":"ContainerStarted","Data":"d8b011ba5640e75db1989584e62678e0259f6a8ea8b4f4646a0b81472f6d539d"} Jul 20 13:28:44.801309 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:44.801273 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-bppwn" event={"ID":"e018dae8-c0db-4060-9f8d-c7d3e0f35223","Type":"ContainerStarted","Data":"bbfd2286f826a0717f0dc27ff80e7de0af827eab9b8702d0e5d8ae1e5516e634"} Jul 20 13:28:44.801668 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:44.801336 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-bppwn" Jul 20 13:28:44.819891 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:44.819832 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-bppwn" podStartSLOduration=1.4341916000000001 podStartE2EDuration="3.819816942s" totalImagesPullingTime="2.385625342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:28:44.819063976 +0000 UTC m=+719.027460133" watchObservedRunningTime="2026-07-20 13:28:44.819816942 +0000 UTC m=+719.028213119" Jul 20 13:28:55.806433 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:28:55.806404 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-bppwn" Jul 20 13:29:32.124122 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:29:32.124079 2575 controller.go:195] "Failed to update lease" err="Put \"https://a4531ea4de1eb4f7494ea6858ebefd1d-6a6f32eb2d45c69a.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-221.ec2.internal?timeout=10s\": context deadline exceeded" Jul 20 13:29:41.349456 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:29:41.349408 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T13:29:31Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T13:29:31Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T13:29:31Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-20T13:29:31Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-132-221.ec2.internal\": Patch \"https://a4531ea4de1eb4f7494ea6858ebefd1d-6a6f32eb2d45c69a.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-221.ec2.internal/status?timeout=10s\": context deadline exceeded" Jul 20 13:29:42.125261 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:29:42.125217 2575 controller.go:195] "Failed to update lease" err="Put \"https://a4531ea4de1eb4f7494ea6858ebefd1d-6a6f32eb2d45c69a.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-221.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 20 13:29:51.350656 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:29:51.350611 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-221.ec2.internal\": Get \"https://a4531ea4de1eb4f7494ea6858ebefd1d-6a6f32eb2d45c69a.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-221.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 20 13:29:52.126408 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:29:52.126322 2575 controller.go:195] "Failed to update lease" err="Put \"https://a4531ea4de1eb4f7494ea6858ebefd1d-6a6f32eb2d45c69a.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-221.ec2.internal?timeout=10s\": context deadline exceeded" Jul 20 13:29:57.657113 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657037 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a72a2fdrqo31i\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657113 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657074 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657113 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657077 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657113 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657104 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657113 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657118 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8h8gr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657113 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657102 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657148 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657159 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657162 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657153 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657190 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657195 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657194 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657232 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657242 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657253 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657271 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657278 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657284 2575 reflector.go:571] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657279 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.657828 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657293 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657320 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657386 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gxzfq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657398 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657405 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657422 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fwswp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657444 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:29:57.657495 2575 controller.go:195] "Failed to update lease" err="Put \"https://a4531ea4de1eb4f7494ea6858ebefd1d-6a6f32eb2d45c69a.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-221.ec2.internal?timeout=10s\": http2: client connection lost" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657509 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657541 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657565 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657581 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n79z9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657609 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-55rsm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657623 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-49998\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657649 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-z4ss5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657664 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ggvq6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657690 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657739 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657783 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657794 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657793 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657799 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657853 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657857 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657854 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657897 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657913 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-slbzm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657928 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657944 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657967 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.658993 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.657973 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658001 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658019 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658035 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658051 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658063 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mrxcc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658032 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658080 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658083 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658095 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658102 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tznzm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658108 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fctt2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658124 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658126 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658131 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-dockercfg-kskvr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659437 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658151 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658153 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658178 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658181 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658188 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658186 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658183 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658207 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658206 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658227 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-s4zk9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658240 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658247 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658241 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658239 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658268 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.659863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658273 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658272 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658267 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nzg4g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658293 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658298 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658258 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658313 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658322 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658323 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tjmd5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658345 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658349 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658349 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658364 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658368 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658376 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660310 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658385 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658388 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658390 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658406 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5q5pd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658409 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658416 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"default-dockercfg-8bpdd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658416 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658431 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658443 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658443 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lv8wv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658442 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658451 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sq28r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658457 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658462 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658462 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kfznd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.660852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658467 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.661325 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658475 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.661325 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658482 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.661325 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658491 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.661325 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658481 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nlwwv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.661325 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658498 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:57.661325 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:57.658503 2575 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 20 13:29:58.462367 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.462340 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 20 13:29:58.477829 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.477802 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 20 13:29:58.477829 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.477802 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 20 13:29:58.478027 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.477913 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 20 13:29:58.480187 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.480163 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 20 13:29:58.484487 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.484471 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 20 13:29:58.499188 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.499167 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ggvq6\"" Jul 20 13:29:58.500339 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.500327 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 20 13:29:58.507941 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.507922 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:29:58.511571 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.511556 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 20 13:29:58.529646 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.529624 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-55rsm\"" Jul 20 13:29:58.529720 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.529634 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 20 13:29:58.539309 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.539290 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 20 13:29:58.543491 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.543477 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:29:58.548383 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.548366 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 20 13:29:58.548474 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.548456 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kfznd\"" Jul 20 13:29:58.555749 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.555734 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 20 13:29:58.566384 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.566360 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 20 13:29:58.567255 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.567238 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 20 13:29:58.576769 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.576749 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nlwwv\"" Jul 20 13:29:58.583305 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.583286 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fwswp\"" Jul 20 13:29:58.588937 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.588917 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 20 13:29:58.591298 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.591283 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 20 13:29:58.608475 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.608455 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 20 13:29:58.608703 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.608688 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nzg4g\"" Jul 20 13:29:58.611329 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.611316 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:29:58.611403 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.611390 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 20 13:29:58.614594 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.614579 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 20 13:29:58.624202 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.624187 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 13:29:58.627632 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.627614 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:29:58.631243 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.631227 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 20 13:29:58.633476 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.633464 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 20 13:29:58.659723 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.659705 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 13:29:58.669059 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.669039 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 13:29:58.669128 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.669061 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8bpdd\"" Jul 20 13:29:58.681207 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.681186 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 20 13:29:58.681483 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.681469 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 20 13:29:58.696335 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.696315 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 20 13:29:58.712553 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.712496 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 20 13:29:58.713718 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.713705 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 20 13:29:58.717210 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.717195 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tznzm\"" Jul 20 13:29:58.721973 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.721958 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mrxcc\"" Jul 20 13:29:58.727743 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.727727 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8h8gr\"" Jul 20 13:29:58.734381 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.734364 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 20 13:29:58.734472 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.734430 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 20 13:29:58.746052 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.746037 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a72a2fdrqo31i\"" Jul 20 13:29:58.750699 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.750684 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 20 13:29:58.759314 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.759296 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-slbzm\"" Jul 20 13:29:58.760273 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.760259 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 13:29:58.778060 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.778043 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 20 13:29:58.782928 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.782914 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 20 13:29:58.786490 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.786475 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 20 13:29:58.792035 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.792022 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 13:29:58.798302 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.798286 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 20 13:29:58.807503 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.807487 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 20 13:29:58.821966 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.821947 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 20 13:29:58.822121 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.822109 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 20 13:29:58.827046 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.827030 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 20 13:29:58.853022 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.853004 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 20 13:29:58.871060 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.871043 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 20 13:29:58.873779 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.873764 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-z4ss5\"" Jul 20 13:29:58.881585 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.881570 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 20 13:29:58.886191 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.886175 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gxzfq\"" Jul 20 13:29:58.890936 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.890920 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 20 13:29:58.891011 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.890960 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-s4zk9\"" Jul 20 13:29:58.898623 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.898604 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n79z9\"" Jul 20 13:29:58.919707 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.919689 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 20 13:29:58.936648 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.936631 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 20 13:29:58.937558 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.937543 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fctt2\"" Jul 20 13:29:58.946233 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.946220 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 20 13:29:58.957718 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.957700 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 20 13:29:58.976056 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.976013 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 20 13:29:58.985397 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.985381 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 20 13:29:58.995107 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.995090 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 20 13:29:58.997632 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.997618 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sq28r\"" Jul 20 13:29:58.997681 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.997603 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 20 13:29:58.997722 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:58.997703 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kskvr\"" Jul 20 13:29:59.002222 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.002207 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 20 13:29:59.009211 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.009194 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5q5pd\"" Jul 20 13:29:59.012579 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.012566 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 20 13:29:59.013569 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.013553 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 20 13:29:59.022205 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.022186 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 20 13:29:59.025710 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.025696 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 20 13:29:59.026918 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.026905 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 20 13:29:59.034532 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.034517 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tjmd5\"" Jul 20 13:29:59.035603 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.035586 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 20 13:29:59.045364 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.045351 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 20 13:29:59.050016 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.049999 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 20 13:29:59.051170 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.051154 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 20 13:29:59.068951 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.068929 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-49998\"" Jul 20 13:29:59.084562 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.084534 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 13:29:59.085832 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.085816 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 20 13:29:59.088194 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.088178 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 20 13:29:59.092966 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.092950 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 13:29:59.093260 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.093244 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 20 13:29:59.096586 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.096572 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 20 13:29:59.116793 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.116776 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 20 13:29:59.123762 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.123741 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 20 13:29:59.127547 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.127529 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 20 13:29:59.127622 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.127600 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 20 13:29:59.138247 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.138233 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 20 13:29:59.143977 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.143959 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:29:59.145019 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.145005 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 20 13:29:59.148447 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.148433 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 20 13:29:59.154073 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.154057 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lv8wv\"" Jul 20 13:29:59.154992 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.154979 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 20 13:29:59.163762 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.163749 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 20 13:29:59.164626 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.164612 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 20 13:29:59.193316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.193292 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 20 13:29:59.210717 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.210701 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 20 13:29:59.211681 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.211669 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 20 13:29:59.223219 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.223205 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 20 13:29:59.231322 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.231284 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 20 13:29:59.250131 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.250113 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 20 13:29:59.250852 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.250839 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 20 13:29:59.267072 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:29:59.267036 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 20 13:30:21.357740 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.357657 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66799fd458-skgmc"] Jul 20 13:30:21.360689 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.360669 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" Jul 20 13:30:21.362845 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.362827 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 13:30:21.363456 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.363434 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 20 13:30:21.363538 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.363457 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4lqjv\"" Jul 20 13:30:21.365928 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.365910 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 20 13:30:21.380866 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.380846 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66799fd458-skgmc"] Jul 20 13:30:21.464867 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.464839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/711464d6-27b3-4e26-8012-5698e594ed4f-openshift-service-ca\") pod \"maas-api-66799fd458-skgmc\" (UID: \"711464d6-27b3-4e26-8012-5698e594ed4f\") " pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" Jul 20 13:30:21.465016 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.464912 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9fwk\" (UniqueName: \"kubernetes.io/projected/711464d6-27b3-4e26-8012-5698e594ed4f-kube-api-access-p9fwk\") pod \"maas-api-66799fd458-skgmc\" (UID: \"711464d6-27b3-4e26-8012-5698e594ed4f\") " pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" Jul 20 13:30:21.465016 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.464946 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/711464d6-27b3-4e26-8012-5698e594ed4f-maas-api-tls\") pod \"maas-api-66799fd458-skgmc\" (UID: \"711464d6-27b3-4e26-8012-5698e594ed4f\") " pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" Jul 20 13:30:21.566108 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.566078 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/711464d6-27b3-4e26-8012-5698e594ed4f-openshift-service-ca\") pod \"maas-api-66799fd458-skgmc\" (UID: \"711464d6-27b3-4e26-8012-5698e594ed4f\") " pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" Jul 20 13:30:21.566217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.566120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9fwk\" (UniqueName: \"kubernetes.io/projected/711464d6-27b3-4e26-8012-5698e594ed4f-kube-api-access-p9fwk\") pod \"maas-api-66799fd458-skgmc\" (UID: \"711464d6-27b3-4e26-8012-5698e594ed4f\") " pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" Jul 20 13:30:21.566217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.566149 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/711464d6-27b3-4e26-8012-5698e594ed4f-maas-api-tls\") pod \"maas-api-66799fd458-skgmc\" (UID: \"711464d6-27b3-4e26-8012-5698e594ed4f\") " pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" Jul 20 13:30:21.566710 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.566685 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/711464d6-27b3-4e26-8012-5698e594ed4f-openshift-service-ca\") pod \"maas-api-66799fd458-skgmc\" (UID: \"711464d6-27b3-4e26-8012-5698e594ed4f\") " pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" Jul 20 13:30:21.568687 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.568663 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/711464d6-27b3-4e26-8012-5698e594ed4f-maas-api-tls\") pod \"maas-api-66799fd458-skgmc\" (UID: \"711464d6-27b3-4e26-8012-5698e594ed4f\") " pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" Jul 20 13:30:21.574856 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.574824 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9fwk\" (UniqueName: \"kubernetes.io/projected/711464d6-27b3-4e26-8012-5698e594ed4f-kube-api-access-p9fwk\") pod \"maas-api-66799fd458-skgmc\" (UID: \"711464d6-27b3-4e26-8012-5698e594ed4f\") " pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" Jul 20 13:30:21.671068 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.671008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" Jul 20 13:30:21.803512 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.803393 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66799fd458-skgmc"] Jul 20 13:30:21.806573 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:30:21.806543 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711464d6_27b3_4e26_8012_5698e594ed4f.slice/crio-cb7288a4f2caff27717fb858457db3675bd57aa47d0a9b86acae0cf7f971473f WatchSource:0}: Error finding container cb7288a4f2caff27717fb858457db3675bd57aa47d0a9b86acae0cf7f971473f: Status 404 returned error can't find the container with id cb7288a4f2caff27717fb858457db3675bd57aa47d0a9b86acae0cf7f971473f Jul 20 13:30:21.898538 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:21.898509 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:30:22.165156 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:22.165124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" event={"ID":"711464d6-27b3-4e26-8012-5698e594ed4f","Type":"ContainerStarted","Data":"cb7288a4f2caff27717fb858457db3675bd57aa47d0a9b86acae0cf7f971473f"} Jul 20 13:30:25.178890 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:25.178837 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" event={"ID":"711464d6-27b3-4e26-8012-5698e594ed4f","Type":"ContainerStarted","Data":"2019d78bcf79ceeba668f543a0f476d4d5c5f0f31369c369fabbb3a7ae5b6c9f"} Jul 20 13:30:25.179254 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:25.178928 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" Jul 20 13:30:25.198513 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:25.198466 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" podStartSLOduration=1.458214037 podStartE2EDuration="4.19845396s" totalImagesPullingTime="2.740239923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:30:25.196335481 +0000 UTC m=+819.404731627" watchObservedRunningTime="2026-07-20 13:30:25.19845396 +0000 UTC m=+819.406850104" Jul 20 13:30:31.188079 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:31.188050 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66799fd458-skgmc" Jul 20 13:30:40.159525 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.159493 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns"] Jul 20 13:30:40.162853 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.162834 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.164717 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.164697 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 20 13:30:40.164978 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.164959 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 20 13:30:40.165043 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.164972 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f779n\"" Jul 20 13:30:40.165043 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.164972 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 20 13:30:40.172572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.172551 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns"] Jul 20 13:30:40.302744 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.302710 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p4gk\" (UniqueName: \"kubernetes.io/projected/66828029-4406-4251-a84a-ca40ff61c891-kube-api-access-5p4gk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.302863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.302752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66828029-4406-4251-a84a-ca40ff61c891-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.302863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.302777 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66828029-4406-4251-a84a-ca40ff61c891-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.302863 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.302843 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66828029-4406-4251-a84a-ca40ff61c891-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.303064 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.302896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66828029-4406-4251-a84a-ca40ff61c891-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.303064 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.302971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66828029-4406-4251-a84a-ca40ff61c891-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.403768 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.403736 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66828029-4406-4251-a84a-ca40ff61c891-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.403891 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.403779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4gk\" (UniqueName: \"kubernetes.io/projected/66828029-4406-4251-a84a-ca40ff61c891-kube-api-access-5p4gk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.403891 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.403798 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66828029-4406-4251-a84a-ca40ff61c891-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.403891 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.403815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66828029-4406-4251-a84a-ca40ff61c891-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.403891 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.403842 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66828029-4406-4251-a84a-ca40ff61c891-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.403891 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.403856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66828029-4406-4251-a84a-ca40ff61c891-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.404285 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.404257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66828029-4406-4251-a84a-ca40ff61c891-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.404285 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.404275 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66828029-4406-4251-a84a-ca40ff61c891-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.404465 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.404338 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66828029-4406-4251-a84a-ca40ff61c891-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.405809 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.405786 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66828029-4406-4251-a84a-ca40ff61c891-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.406113 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.406097 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66828029-4406-4251-a84a-ca40ff61c891-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.411829 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.411772 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p4gk\" (UniqueName: \"kubernetes.io/projected/66828029-4406-4251-a84a-ca40ff61c891-kube-api-access-5p4gk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns\" (UID: \"66828029-4406-4251-a84a-ca40ff61c891\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.474322 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.474300 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:40.601053 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.601030 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns"] Jul 20 13:30:40.603672 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:30:40.603643 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66828029_4406_4251_a84a_ca40ff61c891.slice/crio-d266881ecaf241c21e2935abb7cafd9821730fe86f6904ec3d46d6f127693584 WatchSource:0}: Error finding container d266881ecaf241c21e2935abb7cafd9821730fe86f6904ec3d46d6f127693584: Status 404 returned error can't find the container with id d266881ecaf241c21e2935abb7cafd9821730fe86f6904ec3d46d6f127693584 Jul 20 13:30:40.654106 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:40.654081 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:30:41.244521 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:41.244486 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" event={"ID":"66828029-4406-4251-a84a-ca40ff61c891","Type":"ContainerStarted","Data":"d266881ecaf241c21e2935abb7cafd9821730fe86f6904ec3d46d6f127693584"} Jul 20 13:30:42.250250 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:42.250215 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" event={"ID":"66828029-4406-4251-a84a-ca40ff61c891","Type":"ContainerStarted","Data":"0bc615de17e85b314b9dfd1d3d4f851b264c9bafe245fdf37413528213677342"} Jul 20 13:30:42.269450 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:42.269404 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" podStartSLOduration=1.094965802 podStartE2EDuration="2.26938794s" totalImagesPullingTime="1.174422138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:30:42.267718624 +0000 UTC m=+836.476114768" watchObservedRunningTime="2026-07-20 13:30:42.26938794 +0000 UTC m=+836.477784087" Jul 20 13:30:50.474935 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:50.474897 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:50.475349 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:50.474952 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:50.488229 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:50.488200 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:51.298152 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:51.298118 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns" Jul 20 13:30:52.447759 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:52.447719 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:30:57.653282 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:30:57.653246 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:31:09.844578 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:31:09.844543 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:31:19.873580 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:31:19.873542 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:31:30.244379 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:31:30.244343 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:33:28.151714 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:33:28.151679 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:33:39.349372 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:33:39.349340 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:33:50.451547 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:33:50.451513 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:34:49.053174 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.053094 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-vpb9x"] Jul 20 13:34:49.056141 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.056126 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" Jul 20 13:34:49.058098 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.058074 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 20 13:34:49.058516 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.058499 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mhvsf\"" Jul 20 13:34:49.058600 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.058515 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 20 13:34:49.066233 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.066210 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-vpb9x"] Jul 20 13:34:49.161195 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.161164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2282c397-f503-458f-bbf8-4f875cd03354-webhook-cert\") pod \"maas-controller-6f55c6d57f-vpb9x\" (UID: \"2282c397-f503-458f-bbf8-4f875cd03354\") " pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" Jul 20 13:34:49.161358 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.161208 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5g2r\" (UniqueName: \"kubernetes.io/projected/2282c397-f503-458f-bbf8-4f875cd03354-kube-api-access-w5g2r\") pod \"maas-controller-6f55c6d57f-vpb9x\" (UID: \"2282c397-f503-458f-bbf8-4f875cd03354\") " pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" Jul 20 13:34:49.262478 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.262448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2282c397-f503-458f-bbf8-4f875cd03354-webhook-cert\") pod \"maas-controller-6f55c6d57f-vpb9x\" (UID: \"2282c397-f503-458f-bbf8-4f875cd03354\") " pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" Jul 20 13:34:49.262613 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.262494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5g2r\" (UniqueName: \"kubernetes.io/projected/2282c397-f503-458f-bbf8-4f875cd03354-kube-api-access-w5g2r\") pod \"maas-controller-6f55c6d57f-vpb9x\" (UID: \"2282c397-f503-458f-bbf8-4f875cd03354\") " pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" Jul 20 13:34:49.264788 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.264753 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2282c397-f503-458f-bbf8-4f875cd03354-webhook-cert\") pod \"maas-controller-6f55c6d57f-vpb9x\" (UID: \"2282c397-f503-458f-bbf8-4f875cd03354\") " pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" Jul 20 13:34:49.270671 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.270648 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5g2r\" (UniqueName: \"kubernetes.io/projected/2282c397-f503-458f-bbf8-4f875cd03354-kube-api-access-w5g2r\") pod \"maas-controller-6f55c6d57f-vpb9x\" (UID: \"2282c397-f503-458f-bbf8-4f875cd03354\") " pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" Jul 20 13:34:49.366321 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.366248 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" Jul 20 13:34:49.493266 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.493239 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-vpb9x"] Jul 20 13:34:49.495090 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:34:49.495066 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2282c397_f503_458f_bbf8_4f875cd03354.slice/crio-a4ab60f1f25ef02eae0b6537ffe68dbeb097fc059969b82718d65f93f6c3cf21 WatchSource:0}: Error finding container a4ab60f1f25ef02eae0b6537ffe68dbeb097fc059969b82718d65f93f6c3cf21: Status 404 returned error can't find the container with id a4ab60f1f25ef02eae0b6537ffe68dbeb097fc059969b82718d65f93f6c3cf21 Jul 20 13:34:49.496275 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:49.496258 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 13:34:50.239565 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:50.239528 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" event={"ID":"2282c397-f503-458f-bbf8-4f875cd03354","Type":"ContainerStarted","Data":"a4ab60f1f25ef02eae0b6537ffe68dbeb097fc059969b82718d65f93f6c3cf21"} Jul 20 13:34:51.245794 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:51.245703 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" event={"ID":"2282c397-f503-458f-bbf8-4f875cd03354","Type":"ContainerStarted","Data":"efc6a967ebb7486c06c58625625c5f86932b5af018e2e46f3ba470114ed0568c"} Jul 20 13:34:51.245794 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:51.245779 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" Jul 20 13:34:51.261975 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:34:51.261895 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" podStartSLOduration=0.805317439 podStartE2EDuration="2.261864597s" totalImagesPullingTime="1.456547158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:34:51.261752511 +0000 UTC m=+1085.470148656" watchObservedRunningTime="2026-07-20 13:34:51.261864597 +0000 UTC m=+1085.470260744" Jul 20 13:35:02.253026 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:02.252995 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" Jul 20 13:35:06.916096 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:06.916059 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-vpb9x"] Jul 20 13:35:06.916451 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:06.916237 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" podUID="2282c397-f503-458f-bbf8-4f875cd03354" containerName="manager" containerID="cri-o://efc6a967ebb7486c06c58625625c5f86932b5af018e2e46f3ba470114ed0568c" gracePeriod=30 Jul 20 13:35:07.155636 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.155614 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" Jul 20 13:35:07.311017 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.310992 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5g2r\" (UniqueName: \"kubernetes.io/projected/2282c397-f503-458f-bbf8-4f875cd03354-kube-api-access-w5g2r\") pod \"2282c397-f503-458f-bbf8-4f875cd03354\" (UID: \"2282c397-f503-458f-bbf8-4f875cd03354\") " Jul 20 13:35:07.311162 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.311029 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2282c397-f503-458f-bbf8-4f875cd03354-webhook-cert\") pod \"2282c397-f503-458f-bbf8-4f875cd03354\" (UID: \"2282c397-f503-458f-bbf8-4f875cd03354\") " Jul 20 13:35:07.313179 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.313145 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2282c397-f503-458f-bbf8-4f875cd03354-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2282c397-f503-458f-bbf8-4f875cd03354" (UID: "2282c397-f503-458f-bbf8-4f875cd03354"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:35:07.313304 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.313216 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2282c397-f503-458f-bbf8-4f875cd03354-kube-api-access-w5g2r" (OuterVolumeSpecName: "kube-api-access-w5g2r") pod "2282c397-f503-458f-bbf8-4f875cd03354" (UID: "2282c397-f503-458f-bbf8-4f875cd03354"). InnerVolumeSpecName "kube-api-access-w5g2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:35:07.315732 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.315707 2575 generic.go:356] "Generic (PLEG): container finished" podID="2282c397-f503-458f-bbf8-4f875cd03354" containerID="efc6a967ebb7486c06c58625625c5f86932b5af018e2e46f3ba470114ed0568c" exitCode=0 Jul 20 13:35:07.315827 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.315750 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" Jul 20 13:35:07.315827 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.315763 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" event={"ID":"2282c397-f503-458f-bbf8-4f875cd03354","Type":"ContainerDied","Data":"efc6a967ebb7486c06c58625625c5f86932b5af018e2e46f3ba470114ed0568c"} Jul 20 13:35:07.315827 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.315801 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f55c6d57f-vpb9x" event={"ID":"2282c397-f503-458f-bbf8-4f875cd03354","Type":"ContainerDied","Data":"a4ab60f1f25ef02eae0b6537ffe68dbeb097fc059969b82718d65f93f6c3cf21"} Jul 20 13:35:07.315827 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.315822 2575 scope.go:117] "RemoveContainer" containerID="efc6a967ebb7486c06c58625625c5f86932b5af018e2e46f3ba470114ed0568c" Jul 20 13:35:07.324273 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.324254 2575 scope.go:117] "RemoveContainer" containerID="efc6a967ebb7486c06c58625625c5f86932b5af018e2e46f3ba470114ed0568c" Jul 20 13:35:07.324533 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:35:07.324512 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc6a967ebb7486c06c58625625c5f86932b5af018e2e46f3ba470114ed0568c\": container with ID starting with efc6a967ebb7486c06c58625625c5f86932b5af018e2e46f3ba470114ed0568c not found: ID does not exist" containerID="efc6a967ebb7486c06c58625625c5f86932b5af018e2e46f3ba470114ed0568c" Jul 20 13:35:07.324581 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.324541 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc6a967ebb7486c06c58625625c5f86932b5af018e2e46f3ba470114ed0568c"} err="failed to get container status \"efc6a967ebb7486c06c58625625c5f86932b5af018e2e46f3ba470114ed0568c\": rpc error: code = NotFound desc = could not find container \"efc6a967ebb7486c06c58625625c5f86932b5af018e2e46f3ba470114ed0568c\": container with ID starting with efc6a967ebb7486c06c58625625c5f86932b5af018e2e46f3ba470114ed0568c not found: ID does not exist" Jul 20 13:35:07.335915 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.335894 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-vpb9x"] Jul 20 13:35:07.339398 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.339379 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-vpb9x"] Jul 20 13:35:07.412280 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.412244 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5g2r\" (UniqueName: \"kubernetes.io/projected/2282c397-f503-458f-bbf8-4f875cd03354-kube-api-access-w5g2r\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:35:07.412280 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:07.412279 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2282c397-f503-458f-bbf8-4f875cd03354-webhook-cert\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:35:08.170583 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.170549 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-s8q2w"] Jul 20 13:35:08.170973 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.170833 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2282c397-f503-458f-bbf8-4f875cd03354" containerName="manager" Jul 20 13:35:08.170973 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.170843 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2282c397-f503-458f-bbf8-4f875cd03354" containerName="manager" Jul 20 13:35:08.170973 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.170924 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2282c397-f503-458f-bbf8-4f875cd03354" containerName="manager" Jul 20 13:35:08.174767 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.174746 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f55c6d57f-s8q2w" Jul 20 13:35:08.176948 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.176928 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mhvsf\"" Jul 20 13:35:08.177259 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.177242 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 20 13:35:08.177381 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.177364 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 20 13:35:08.187973 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.187951 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-s8q2w"] Jul 20 13:35:08.308404 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.308371 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2282c397-f503-458f-bbf8-4f875cd03354" path="/var/lib/kubelet/pods/2282c397-f503-458f-bbf8-4f875cd03354/volumes" Jul 20 13:35:08.317821 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.317798 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0383138e-0b70-4738-b321-b984e3da082d-webhook-cert\") pod \"maas-controller-6f55c6d57f-s8q2w\" (UID: \"0383138e-0b70-4738-b321-b984e3da082d\") " pod="opendatahub/maas-controller-6f55c6d57f-s8q2w" Jul 20 13:35:08.317955 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.317853 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k477p\" (UniqueName: \"kubernetes.io/projected/0383138e-0b70-4738-b321-b984e3da082d-kube-api-access-k477p\") pod \"maas-controller-6f55c6d57f-s8q2w\" (UID: \"0383138e-0b70-4738-b321-b984e3da082d\") " pod="opendatahub/maas-controller-6f55c6d57f-s8q2w" Jul 20 13:35:08.418411 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.418383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k477p\" (UniqueName: \"kubernetes.io/projected/0383138e-0b70-4738-b321-b984e3da082d-kube-api-access-k477p\") pod \"maas-controller-6f55c6d57f-s8q2w\" (UID: \"0383138e-0b70-4738-b321-b984e3da082d\") " pod="opendatahub/maas-controller-6f55c6d57f-s8q2w" Jul 20 13:35:08.418540 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.418427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0383138e-0b70-4738-b321-b984e3da082d-webhook-cert\") pod \"maas-controller-6f55c6d57f-s8q2w\" (UID: \"0383138e-0b70-4738-b321-b984e3da082d\") " pod="opendatahub/maas-controller-6f55c6d57f-s8q2w" Jul 20 13:35:08.420685 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.420643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0383138e-0b70-4738-b321-b984e3da082d-webhook-cert\") pod \"maas-controller-6f55c6d57f-s8q2w\" (UID: \"0383138e-0b70-4738-b321-b984e3da082d\") " pod="opendatahub/maas-controller-6f55c6d57f-s8q2w" Jul 20 13:35:08.429518 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.429481 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k477p\" (UniqueName: \"kubernetes.io/projected/0383138e-0b70-4738-b321-b984e3da082d-kube-api-access-k477p\") pod \"maas-controller-6f55c6d57f-s8q2w\" (UID: \"0383138e-0b70-4738-b321-b984e3da082d\") " pod="opendatahub/maas-controller-6f55c6d57f-s8q2w" Jul 20 13:35:08.484326 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.484302 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f55c6d57f-s8q2w" Jul 20 13:35:08.622206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:08.622179 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f55c6d57f-s8q2w"] Jul 20 13:35:08.623050 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:35:08.623019 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0383138e_0b70_4738_b321_b984e3da082d.slice/crio-f7bf090d8187ad2e0451f0a49fb90a624299ca3ee8256854c122c42b31338f6e WatchSource:0}: Error finding container f7bf090d8187ad2e0451f0a49fb90a624299ca3ee8256854c122c42b31338f6e: Status 404 returned error can't find the container with id f7bf090d8187ad2e0451f0a49fb90a624299ca3ee8256854c122c42b31338f6e Jul 20 13:35:09.326009 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:09.325970 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f55c6d57f-s8q2w" event={"ID":"0383138e-0b70-4738-b321-b984e3da082d","Type":"ContainerStarted","Data":"5c9dbe987d614099485f402454f7bd00b98b1609c5030f850c876454a7a2571b"} Jul 20 13:35:09.326397 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:09.326030 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f55c6d57f-s8q2w" event={"ID":"0383138e-0b70-4738-b321-b984e3da082d","Type":"ContainerStarted","Data":"f7bf090d8187ad2e0451f0a49fb90a624299ca3ee8256854c122c42b31338f6e"} Jul 20 13:35:09.326397 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:09.326057 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f55c6d57f-s8q2w" Jul 20 13:35:09.347107 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:09.347054 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f55c6d57f-s8q2w" podStartSLOduration=0.984956457 podStartE2EDuration="1.347038391s" totalImagesPullingTime="362.081934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:35:09.34506383 +0000 UTC m=+1103.553459974" watchObservedRunningTime="2026-07-20 13:35:09.347038391 +0000 UTC m=+1103.555434535" Jul 20 13:35:20.331936 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:20.331898 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f55c6d57f-s8q2w" Jul 20 13:35:31.356117 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:31.356087 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:35:41.746612 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:35:41.746579 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:36:18.956064 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:36:18.955979 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:36:28.049586 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:36:28.049549 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:36:36.451017 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:36:36.450982 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:36:45.847465 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:36:45.847430 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:36:57.548582 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:36:57.548546 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:37:06.658953 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:37:06.658911 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:37:17.176232 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:37:17.176191 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:37:26.849928 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:37:26.849890 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:37:49.650543 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:37:49.650460 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:37:58.945763 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:37:58.945722 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:38:24.942042 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:38:24.942010 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:38:37.245726 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:38:37.245694 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:38:52.849995 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:38:52.849964 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:39:02.154576 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:39:02.154541 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:39:11.894045 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:39:11.893963 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:39:21.254208 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:39:21.254173 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:39:30.750073 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:39:30.750034 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:41:18.648645 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:41:18.648563 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:41:27.748458 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:41:27.748421 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:41:40.454803 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:41:40.454766 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:41:52.152288 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:41:52.152252 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:42:02.653116 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:42:02.653082 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:42:13.160131 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:42:13.160050 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:42:23.165812 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:42:23.165775 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:42:33.756639 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:42:33.756605 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:42:44.063776 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:42:44.063733 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:42:54.660501 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:42:54.660461 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:43:03.751886 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:43:03.751850 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:43:26.856656 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:43:26.856619 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:43:37.753931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:43:37.753896 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:43:55.463145 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:43:55.463108 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:44:28.957108 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:44:28.957068 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:44:42.169661 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:44:42.169625 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:44:51.190933 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:44:51.190893 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:45:00.221707 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:00.221661 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742585-gb5zh"] Jul 20 13:45:00.225254 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:00.225230 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742585-gb5zh" Jul 20 13:45:00.228542 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:00.228506 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t89s5\"" Jul 20 13:45:00.234683 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:00.234658 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742585-gb5zh" Jul 20 13:45:00.239911 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:00.239886 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742585-gb5zh"] Jul 20 13:45:00.374298 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:45:00.374269 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3c87c7_b883_4bfe_bfb7_54caf151341c.slice/crio-a0c5c26bebe0c15428b8194b5efe0525bb3475df730ff7ae0e4208aa8de3e70a WatchSource:0}: Error finding container a0c5c26bebe0c15428b8194b5efe0525bb3475df730ff7ae0e4208aa8de3e70a: Status 404 returned error can't find the container with id a0c5c26bebe0c15428b8194b5efe0525bb3475df730ff7ae0e4208aa8de3e70a Jul 20 13:45:00.374794 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:00.374777 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742585-gb5zh"] Jul 20 13:45:00.376348 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:00.376325 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 13:45:00.715929 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:00.715889 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742585-gb5zh" event={"ID":"cc3c87c7-b883-4bfe-bfb7-54caf151341c","Type":"ContainerStarted","Data":"a0c5c26bebe0c15428b8194b5efe0525bb3475df730ff7ae0e4208aa8de3e70a"} Jul 20 13:45:03.733182 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:03.733082 2575 generic.go:356] "Generic (PLEG): container finished" podID="cc3c87c7-b883-4bfe-bfb7-54caf151341c" containerID="0133ab22c5dc89cd9e72e7c71e34e820b61f8af1ad34cdb2f56549d7a40e543d" exitCode=0 Jul 20 13:45:03.733182 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:03.733140 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742585-gb5zh" event={"ID":"cc3c87c7-b883-4bfe-bfb7-54caf151341c","Type":"ContainerDied","Data":"0133ab22c5dc89cd9e72e7c71e34e820b61f8af1ad34cdb2f56549d7a40e543d"} Jul 20 13:45:04.257335 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:04.257293 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:45:04.868038 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:04.868006 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742585-gb5zh" Jul 20 13:45:05.743255 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:05.743228 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742585-gb5zh" Jul 20 13:45:05.743470 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:05.743255 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742585-gb5zh" event={"ID":"cc3c87c7-b883-4bfe-bfb7-54caf151341c","Type":"ContainerDied","Data":"a0c5c26bebe0c15428b8194b5efe0525bb3475df730ff7ae0e4208aa8de3e70a"} Jul 20 13:45:05.743470 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:05.743295 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0c5c26bebe0c15428b8194b5efe0525bb3475df730ff7ae0e4208aa8de3e70a" Jul 20 13:45:18.258889 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:18.258780 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:45:27.850537 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:27.850501 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:45:40.677727 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:40.677693 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:45:50.061190 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:45:50.061153 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:46:05.963210 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:46:05.963172 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:46:14.855949 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:46:14.855905 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:46:24.257692 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:46:24.257653 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:46:34.152247 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:46:34.152212 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:46:46.652376 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:46:46.652342 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:46:55.344892 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:46:55.344833 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:47:08.961836 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:47:08.961796 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:47:20.751326 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:47:20.751292 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:47:31.960372 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:47:31.960335 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:47:44.578635 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:47:44.578599 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:48:14.951639 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:48:14.951562 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:48:23.754770 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:48:23.754739 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:48:57.557421 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:48:57.557384 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:49:07.076588 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.076559 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr"] Jul 20 13:49:07.076959 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.076912 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc3c87c7-b883-4bfe-bfb7-54caf151341c" containerName="cleanup" Jul 20 13:49:07.076959 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.076932 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc3c87c7-b883-4bfe-bfb7-54caf151341c" containerName="cleanup" Jul 20 13:49:07.077034 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.077014 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc3c87c7-b883-4bfe-bfb7-54caf151341c" containerName="cleanup" Jul 20 13:49:07.080053 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.080037 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:07.081937 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.081916 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d5814763-serving-cert\"" Jul 20 13:49:07.090505 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.090481 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr"] Jul 20 13:49:07.188732 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.188705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc776818-c288-4198-8581-975aadac1c1e-maas-api-tls\") pod \"maas-api-e2e-ait-d5814763-97dfcf687-sdfdr\" (UID: \"bc776818-c288-4198-8581-975aadac1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:07.188854 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.188736 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx5bf\" (UniqueName: \"kubernetes.io/projected/bc776818-c288-4198-8581-975aadac1c1e-kube-api-access-vx5bf\") pod \"maas-api-e2e-ait-d5814763-97dfcf687-sdfdr\" (UID: \"bc776818-c288-4198-8581-975aadac1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:07.188854 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.188760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc776818-c288-4198-8581-975aadac1c1e-openshift-service-ca\") pod \"maas-api-e2e-ait-d5814763-97dfcf687-sdfdr\" (UID: \"bc776818-c288-4198-8581-975aadac1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:07.289270 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.289243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc776818-c288-4198-8581-975aadac1c1e-maas-api-tls\") pod \"maas-api-e2e-ait-d5814763-97dfcf687-sdfdr\" (UID: \"bc776818-c288-4198-8581-975aadac1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:07.289387 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.289277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx5bf\" (UniqueName: \"kubernetes.io/projected/bc776818-c288-4198-8581-975aadac1c1e-kube-api-access-vx5bf\") pod \"maas-api-e2e-ait-d5814763-97dfcf687-sdfdr\" (UID: \"bc776818-c288-4198-8581-975aadac1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:07.289387 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.289304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc776818-c288-4198-8581-975aadac1c1e-openshift-service-ca\") pod \"maas-api-e2e-ait-d5814763-97dfcf687-sdfdr\" (UID: \"bc776818-c288-4198-8581-975aadac1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:07.289933 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.289914 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc776818-c288-4198-8581-975aadac1c1e-openshift-service-ca\") pod \"maas-api-e2e-ait-d5814763-97dfcf687-sdfdr\" (UID: \"bc776818-c288-4198-8581-975aadac1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:07.291755 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.291733 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc776818-c288-4198-8581-975aadac1c1e-maas-api-tls\") pod \"maas-api-e2e-ait-d5814763-97dfcf687-sdfdr\" (UID: \"bc776818-c288-4198-8581-975aadac1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:07.297428 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.297386 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx5bf\" (UniqueName: \"kubernetes.io/projected/bc776818-c288-4198-8581-975aadac1c1e-kube-api-access-vx5bf\") pod \"maas-api-e2e-ait-d5814763-97dfcf687-sdfdr\" (UID: \"bc776818-c288-4198-8581-975aadac1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:07.390778 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.390724 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:07.518971 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.518946 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr"] Jul 20 13:49:07.521328 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:49:07.521292 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc776818_c288_4198_8581_975aadac1c1e.slice/crio-ae736c16eb9fe12b86f0238fe695d47f2d36978956ab074858748f2ff0952614 WatchSource:0}: Error finding container ae736c16eb9fe12b86f0238fe695d47f2d36978956ab074858748f2ff0952614: Status 404 returned error can't find the container with id ae736c16eb9fe12b86f0238fe695d47f2d36978956ab074858748f2ff0952614 Jul 20 13:49:07.759973 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:07.759896 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" event={"ID":"bc776818-c288-4198-8581-975aadac1c1e","Type":"ContainerStarted","Data":"ae736c16eb9fe12b86f0238fe695d47f2d36978956ab074858748f2ff0952614"} Jul 20 13:49:08.765912 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:08.765807 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" event={"ID":"bc776818-c288-4198-8581-975aadac1c1e","Type":"ContainerStarted","Data":"600cb6f0bd75b1980002f405cf7755879f9248cbbd019ba21a2879ed340f21ac"} Jul 20 13:49:08.766291 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:08.765940 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:08.784548 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:08.784500 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" podStartSLOduration=0.981824718 podStartE2EDuration="1.784487017s" totalImagesPullingTime="802.662299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:49:08.782296319 +0000 UTC m=+1942.990692464" watchObservedRunningTime="2026-07-20 13:49:08.784487017 +0000 UTC m=+1942.992883161" Jul 20 13:49:11.151621 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:11.151583 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:49:14.775755 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:14.775726 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:32.519547 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:32.519509 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx"] Jul 20 13:49:32.523385 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:32.523366 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx" Jul 20 13:49:32.525342 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:32.525318 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t89s5\"" Jul 20 13:49:32.529083 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:32.529058 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx"] Jul 20 13:49:32.685939 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:32.685896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0abb59-71ff-4ddf-b551-20557594b112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5814763-fptcx\" (UID: \"9d0abb59-71ff-4ddf-b551-20557594b112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx" Jul 20 13:49:32.786572 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:32.786548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0abb59-71ff-4ddf-b551-20557594b112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5814763-fptcx\" (UID: \"9d0abb59-71ff-4ddf-b551-20557594b112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx" Jul 20 13:49:32.787198 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:32.787179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0abb59-71ff-4ddf-b551-20557594b112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5814763-fptcx\" (UID: \"9d0abb59-71ff-4ddf-b551-20557594b112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx" Jul 20 13:49:32.837684 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:32.837661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx" Jul 20 13:49:32.964718 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:32.964695 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx"] Jul 20 13:49:32.966220 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:49:32.966197 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0abb59_71ff_4ddf_b551_20557594b112.slice/crio-abdc656641e82384a8c5e52d6022f3b193902d33ca1b53730fc64ad767c9d41c WatchSource:0}: Error finding container abdc656641e82384a8c5e52d6022f3b193902d33ca1b53730fc64ad767c9d41c: Status 404 returned error can't find the container with id abdc656641e82384a8c5e52d6022f3b193902d33ca1b53730fc64ad767c9d41c Jul 20 13:49:33.874978 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:33.874902 2575 generic.go:356] "Generic (PLEG): container finished" podID="9d0abb59-71ff-4ddf-b551-20557594b112" containerID="4efe94c2d371ef70a13752cddd94d5150f3265b5b8cc9c07740d5cd651729450" exitCode=0 Jul 20 13:49:33.875324 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:33.874969 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx" event={"ID":"9d0abb59-71ff-4ddf-b551-20557594b112","Type":"ContainerDied","Data":"4efe94c2d371ef70a13752cddd94d5150f3265b5b8cc9c07740d5cd651729450"} Jul 20 13:49:33.875324 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:33.875031 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx" event={"ID":"9d0abb59-71ff-4ddf-b551-20557594b112","Type":"ContainerStarted","Data":"abdc656641e82384a8c5e52d6022f3b193902d33ca1b53730fc64ad767c9d41c"} Jul 20 13:49:35.004946 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:35.004926 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx" Jul 20 13:49:35.103535 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:35.103509 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0abb59-71ff-4ddf-b551-20557594b112-maas-api-service-ca\") pod \"9d0abb59-71ff-4ddf-b551-20557594b112\" (UID: \"9d0abb59-71ff-4ddf-b551-20557594b112\") " Jul 20 13:49:35.103840 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:35.103817 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0abb59-71ff-4ddf-b551-20557594b112-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d0abb59-71ff-4ddf-b551-20557594b112" (UID: "9d0abb59-71ff-4ddf-b551-20557594b112"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:49:35.204576 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:35.204506 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0abb59-71ff-4ddf-b551-20557594b112-maas-api-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:49:35.884245 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:35.884207 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx" event={"ID":"9d0abb59-71ff-4ddf-b551-20557594b112","Type":"ContainerDied","Data":"abdc656641e82384a8c5e52d6022f3b193902d33ca1b53730fc64ad767c9d41c"} Jul 20 13:49:35.884245 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:35.884241 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx" Jul 20 13:49:35.884448 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:35.884260 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abdc656641e82384a8c5e52d6022f3b193902d33ca1b53730fc64ad767c9d41c" Jul 20 13:49:43.139392 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.139356 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr"] Jul 20 13:49:43.139748 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.139562 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" podUID="bc776818-c288-4198-8581-975aadac1c1e" containerName="maas-api" containerID="cri-o://600cb6f0bd75b1980002f405cf7755879f9248cbbd019ba21a2879ed340f21ac" gracePeriod=30 Jul 20 13:49:43.382950 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.382928 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:43.570105 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.570077 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc776818-c288-4198-8581-975aadac1c1e-maas-api-tls\") pod \"bc776818-c288-4198-8581-975aadac1c1e\" (UID: \"bc776818-c288-4198-8581-975aadac1c1e\") " Jul 20 13:49:43.570270 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.570115 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc776818-c288-4198-8581-975aadac1c1e-openshift-service-ca\") pod \"bc776818-c288-4198-8581-975aadac1c1e\" (UID: \"bc776818-c288-4198-8581-975aadac1c1e\") " Jul 20 13:49:43.570270 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.570149 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx5bf\" (UniqueName: \"kubernetes.io/projected/bc776818-c288-4198-8581-975aadac1c1e-kube-api-access-vx5bf\") pod \"bc776818-c288-4198-8581-975aadac1c1e\" (UID: \"bc776818-c288-4198-8581-975aadac1c1e\") " Jul 20 13:49:43.570457 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.570434 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc776818-c288-4198-8581-975aadac1c1e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bc776818-c288-4198-8581-975aadac1c1e" (UID: "bc776818-c288-4198-8581-975aadac1c1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:49:43.572210 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.572182 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc776818-c288-4198-8581-975aadac1c1e-kube-api-access-vx5bf" (OuterVolumeSpecName: "kube-api-access-vx5bf") pod "bc776818-c288-4198-8581-975aadac1c1e" (UID: "bc776818-c288-4198-8581-975aadac1c1e"). InnerVolumeSpecName "kube-api-access-vx5bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:49:43.572306 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.572237 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc776818-c288-4198-8581-975aadac1c1e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bc776818-c288-4198-8581-975aadac1c1e" (UID: "bc776818-c288-4198-8581-975aadac1c1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:49:43.671272 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.671246 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc776818-c288-4198-8581-975aadac1c1e-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:49:43.671272 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.671271 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc776818-c288-4198-8581-975aadac1c1e-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:49:43.671422 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.671281 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx5bf\" (UniqueName: \"kubernetes.io/projected/bc776818-c288-4198-8581-975aadac1c1e-kube-api-access-vx5bf\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:49:43.925586 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.925510 2575 generic.go:356] "Generic (PLEG): container finished" podID="bc776818-c288-4198-8581-975aadac1c1e" containerID="600cb6f0bd75b1980002f405cf7755879f9248cbbd019ba21a2879ed340f21ac" exitCode=0 Jul 20 13:49:43.925586 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.925568 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" Jul 20 13:49:43.925743 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.925589 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" event={"ID":"bc776818-c288-4198-8581-975aadac1c1e","Type":"ContainerDied","Data":"600cb6f0bd75b1980002f405cf7755879f9248cbbd019ba21a2879ed340f21ac"} Jul 20 13:49:43.925743 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.925645 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr" event={"ID":"bc776818-c288-4198-8581-975aadac1c1e","Type":"ContainerDied","Data":"ae736c16eb9fe12b86f0238fe695d47f2d36978956ab074858748f2ff0952614"} Jul 20 13:49:43.925743 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.925668 2575 scope.go:117] "RemoveContainer" containerID="600cb6f0bd75b1980002f405cf7755879f9248cbbd019ba21a2879ed340f21ac" Jul 20 13:49:43.934638 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.934619 2575 scope.go:117] "RemoveContainer" containerID="600cb6f0bd75b1980002f405cf7755879f9248cbbd019ba21a2879ed340f21ac" Jul 20 13:49:43.934908 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:49:43.934864 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600cb6f0bd75b1980002f405cf7755879f9248cbbd019ba21a2879ed340f21ac\": container with ID starting with 600cb6f0bd75b1980002f405cf7755879f9248cbbd019ba21a2879ed340f21ac not found: ID does not exist" containerID="600cb6f0bd75b1980002f405cf7755879f9248cbbd019ba21a2879ed340f21ac" Jul 20 13:49:43.935006 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.934918 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600cb6f0bd75b1980002f405cf7755879f9248cbbd019ba21a2879ed340f21ac"} err="failed to get container status \"600cb6f0bd75b1980002f405cf7755879f9248cbbd019ba21a2879ed340f21ac\": rpc error: code = NotFound desc = could not find container \"600cb6f0bd75b1980002f405cf7755879f9248cbbd019ba21a2879ed340f21ac\": container with ID starting with 600cb6f0bd75b1980002f405cf7755879f9248cbbd019ba21a2879ed340f21ac not found: ID does not exist" Jul 20 13:49:43.946130 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.946104 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr"] Jul 20 13:49:43.950026 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:43.950005 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5814763-97dfcf687-sdfdr"] Jul 20 13:49:44.311403 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:44.311371 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc776818-c288-4198-8581-975aadac1c1e" path="/var/lib/kubelet/pods/bc776818-c288-4198-8581-975aadac1c1e/volumes" Jul 20 13:49:46.050816 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:46.050785 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:49:47.642630 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:47.642598 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx"] Jul 20 13:49:47.655772 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:47.655745 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5814763-fptcx"] Jul 20 13:49:48.308146 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:48.308114 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d0abb59-71ff-4ddf-b551-20557594b112" path="/var/lib/kubelet/pods/9d0abb59-71ff-4ddf-b551-20557594b112/volumes" Jul 20 13:49:51.422746 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.422713 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s"] Jul 20 13:49:51.423099 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.423051 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc776818-c288-4198-8581-975aadac1c1e" containerName="maas-api" Jul 20 13:49:51.423099 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.423063 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc776818-c288-4198-8581-975aadac1c1e" containerName="maas-api" Jul 20 13:49:51.423099 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.423073 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0abb59-71ff-4ddf-b551-20557594b112" containerName="revoke-keys" Jul 20 13:49:51.423099 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.423080 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0abb59-71ff-4ddf-b551-20557594b112" containerName="revoke-keys" Jul 20 13:49:51.423258 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.423145 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc776818-c288-4198-8581-975aadac1c1e" containerName="maas-api" Jul 20 13:49:51.423258 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.423154 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d0abb59-71ff-4ddf-b551-20557594b112" containerName="revoke-keys" Jul 20 13:49:51.425987 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.425961 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:49:51.429470 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.429451 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5dcc91bd-serving-cert\"" Jul 20 13:49:51.446326 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.446301 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s"] Jul 20 13:49:51.530001 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.529978 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ff603b-fa44-43c0-89a4-1ef36f470af6-openshift-service-ca\") pod \"maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s\" (UID: \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:49:51.530134 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.530033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ff603b-fa44-43c0-89a4-1ef36f470af6-maas-api-tls\") pod \"maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s\" (UID: \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:49:51.530183 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.530135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mpqg\" (UniqueName: \"kubernetes.io/projected/e7ff603b-fa44-43c0-89a4-1ef36f470af6-kube-api-access-4mpqg\") pod \"maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s\" (UID: \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:49:51.630554 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.630511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ff603b-fa44-43c0-89a4-1ef36f470af6-openshift-service-ca\") pod \"maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s\" (UID: \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:49:51.630703 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.630589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ff603b-fa44-43c0-89a4-1ef36f470af6-maas-api-tls\") pod \"maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s\" (UID: \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:49:51.630703 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.630638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mpqg\" (UniqueName: \"kubernetes.io/projected/e7ff603b-fa44-43c0-89a4-1ef36f470af6-kube-api-access-4mpqg\") pod \"maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s\" (UID: \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:49:51.630820 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:49:51.630760 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-serving-cert: secret "maas-api-e2e-migrate-5dcc91bd-serving-cert" not found Jul 20 13:49:51.630903 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:49:51.630847 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7ff603b-fa44-43c0-89a4-1ef36f470af6-maas-api-tls podName:e7ff603b-fa44-43c0-89a4-1ef36f470af6 nodeName:}" failed. No retries permitted until 2026-07-20 13:49:52.130824882 +0000 UTC m=+1986.339221020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e7ff603b-fa44-43c0-89a4-1ef36f470af6-maas-api-tls") pod "maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" (UID: "e7ff603b-fa44-43c0-89a4-1ef36f470af6") : secret "maas-api-e2e-migrate-5dcc91bd-serving-cert" not found Jul 20 13:49:51.631185 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.631167 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ff603b-fa44-43c0-89a4-1ef36f470af6-openshift-service-ca\") pod \"maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s\" (UID: \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:49:51.640899 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:51.640861 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mpqg\" (UniqueName: \"kubernetes.io/projected/e7ff603b-fa44-43c0-89a4-1ef36f470af6-kube-api-access-4mpqg\") pod \"maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s\" (UID: \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:49:52.133736 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:52.133705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ff603b-fa44-43c0-89a4-1ef36f470af6-maas-api-tls\") pod \"maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s\" (UID: \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:49:52.136076 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:52.136055 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ff603b-fa44-43c0-89a4-1ef36f470af6-maas-api-tls\") pod \"maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s\" (UID: \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:49:52.335687 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:52.335653 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:49:52.462472 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:52.462445 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s"] Jul 20 13:49:52.464345 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:49:52.464316 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ff603b_fa44_43c0_89a4_1ef36f470af6.slice/crio-3c718f89dae7a361f39bac76495dab25a93c29ebdb3e8020e5ff72ac489d2286 WatchSource:0}: Error finding container 3c718f89dae7a361f39bac76495dab25a93c29ebdb3e8020e5ff72ac489d2286: Status 404 returned error can't find the container with id 3c718f89dae7a361f39bac76495dab25a93c29ebdb3e8020e5ff72ac489d2286 Jul 20 13:49:52.964751 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:52.964714 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" event={"ID":"e7ff603b-fa44-43c0-89a4-1ef36f470af6","Type":"ContainerStarted","Data":"9a0a5d8e9adb070d2b04dc397aa14142a30d5db9312babba34cab1ab5d82f43a"} Jul 20 13:49:52.964939 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:52.964769 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" event={"ID":"e7ff603b-fa44-43c0-89a4-1ef36f470af6","Type":"ContainerStarted","Data":"3c718f89dae7a361f39bac76495dab25a93c29ebdb3e8020e5ff72ac489d2286"} Jul 20 13:49:52.964939 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:52.964830 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:49:52.985904 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:52.985835 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" podStartSLOduration=1.7019616690000001 podStartE2EDuration="1.98581751s" totalImagesPullingTime="283.855841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:49:52.985625737 +0000 UTC m=+1987.194021962" watchObservedRunningTime="2026-07-20 13:49:52.98581751 +0000 UTC m=+1987.194213657" Jul 20 13:49:54.153486 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:54.153448 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:49:58.974489 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:49:58.974460 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:50:16.244411 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:16.244371 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm"] Jul 20 13:50:16.247896 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:16.247852 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm" Jul 20 13:50:16.250046 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:16.250026 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t89s5\"" Jul 20 13:50:16.255848 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:16.255826 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm"] Jul 20 13:50:16.316958 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:16.316928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49c0eae-640a-4d0a-aadc-78239c2cafe9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm\" (UID: \"a49c0eae-640a-4d0a-aadc-78239c2cafe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm" Jul 20 13:50:16.417976 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:16.417947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49c0eae-640a-4d0a-aadc-78239c2cafe9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm\" (UID: \"a49c0eae-640a-4d0a-aadc-78239c2cafe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm" Jul 20 13:50:16.418524 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:16.418502 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49c0eae-640a-4d0a-aadc-78239c2cafe9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm\" (UID: \"a49c0eae-640a-4d0a-aadc-78239c2cafe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm" Jul 20 13:50:16.559813 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:16.559793 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm" Jul 20 13:50:16.689604 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:16.689581 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm"] Jul 20 13:50:16.691952 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:50:16.691919 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49c0eae_640a_4d0a_aadc_78239c2cafe9.slice/crio-4c856d87adde1f880e49dc423ecb010e86741accb9c72915a3cfbd022300b9b8 WatchSource:0}: Error finding container 4c856d87adde1f880e49dc423ecb010e86741accb9c72915a3cfbd022300b9b8: Status 404 returned error can't find the container with id 4c856d87adde1f880e49dc423ecb010e86741accb9c72915a3cfbd022300b9b8 Jul 20 13:50:17.071211 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:17.071176 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm" event={"ID":"a49c0eae-640a-4d0a-aadc-78239c2cafe9","Type":"ContainerStarted","Data":"9ea5b2c1dac4ff32d342cf97a52d97909bffacdf8e6724c7cf6c2d26a999bd3a"} Jul 20 13:50:17.071363 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:17.071270 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm" event={"ID":"a49c0eae-640a-4d0a-aadc-78239c2cafe9","Type":"ContainerStarted","Data":"4c856d87adde1f880e49dc423ecb010e86741accb9c72915a3cfbd022300b9b8"} Jul 20 13:50:17.091149 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:17.091100 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm" podStartSLOduration=1.091084376 podStartE2EDuration="1.091084376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:50:17.089592322 +0000 UTC m=+2011.297988469" watchObservedRunningTime="2026-07-20 13:50:17.091084376 +0000 UTC m=+2011.299480518" Jul 20 13:50:18.076905 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:18.076853 2575 generic.go:356] "Generic (PLEG): container finished" podID="a49c0eae-640a-4d0a-aadc-78239c2cafe9" containerID="9ea5b2c1dac4ff32d342cf97a52d97909bffacdf8e6724c7cf6c2d26a999bd3a" exitCode=0 Jul 20 13:50:18.076905 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:18.076910 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm" event={"ID":"a49c0eae-640a-4d0a-aadc-78239c2cafe9","Type":"ContainerDied","Data":"9ea5b2c1dac4ff32d342cf97a52d97909bffacdf8e6724c7cf6c2d26a999bd3a"} Jul 20 13:50:19.204523 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:19.204494 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm" Jul 20 13:50:19.240760 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:19.240732 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49c0eae-640a-4d0a-aadc-78239c2cafe9-maas-api-service-ca\") pod \"a49c0eae-640a-4d0a-aadc-78239c2cafe9\" (UID: \"a49c0eae-640a-4d0a-aadc-78239c2cafe9\") " Jul 20 13:50:19.241063 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:19.241043 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49c0eae-640a-4d0a-aadc-78239c2cafe9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a49c0eae-640a-4d0a-aadc-78239c2cafe9" (UID: "a49c0eae-640a-4d0a-aadc-78239c2cafe9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:50:19.341507 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:19.341437 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49c0eae-640a-4d0a-aadc-78239c2cafe9-maas-api-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:50:20.087372 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:20.087344 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm" Jul 20 13:50:20.087372 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:20.087359 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm" event={"ID":"a49c0eae-640a-4d0a-aadc-78239c2cafe9","Type":"ContainerDied","Data":"4c856d87adde1f880e49dc423ecb010e86741accb9c72915a3cfbd022300b9b8"} Jul 20 13:50:20.087568 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:20.087400 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c856d87adde1f880e49dc423ecb010e86741accb9c72915a3cfbd022300b9b8" Jul 20 13:50:26.841028 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:26.840993 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s"] Jul 20 13:50:26.841537 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:26.841224 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" podUID="e7ff603b-fa44-43c0-89a4-1ef36f470af6" containerName="maas-api" containerID="cri-o://9a0a5d8e9adb070d2b04dc397aa14142a30d5db9312babba34cab1ab5d82f43a" gracePeriod=30 Jul 20 13:50:27.089217 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.089196 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:50:27.119180 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.119122 2575 generic.go:356] "Generic (PLEG): container finished" podID="e7ff603b-fa44-43c0-89a4-1ef36f470af6" containerID="9a0a5d8e9adb070d2b04dc397aa14142a30d5db9312babba34cab1ab5d82f43a" exitCode=0 Jul 20 13:50:27.119284 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.119198 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" Jul 20 13:50:27.119284 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.119210 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" event={"ID":"e7ff603b-fa44-43c0-89a4-1ef36f470af6","Type":"ContainerDied","Data":"9a0a5d8e9adb070d2b04dc397aa14142a30d5db9312babba34cab1ab5d82f43a"} Jul 20 13:50:27.119284 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.119262 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s" event={"ID":"e7ff603b-fa44-43c0-89a4-1ef36f470af6","Type":"ContainerDied","Data":"3c718f89dae7a361f39bac76495dab25a93c29ebdb3e8020e5ff72ac489d2286"} Jul 20 13:50:27.119375 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.119294 2575 scope.go:117] "RemoveContainer" containerID="9a0a5d8e9adb070d2b04dc397aa14142a30d5db9312babba34cab1ab5d82f43a" Jul 20 13:50:27.127356 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.127340 2575 scope.go:117] "RemoveContainer" containerID="9a0a5d8e9adb070d2b04dc397aa14142a30d5db9312babba34cab1ab5d82f43a" Jul 20 13:50:27.127642 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:50:27.127624 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0a5d8e9adb070d2b04dc397aa14142a30d5db9312babba34cab1ab5d82f43a\": container with ID starting with 9a0a5d8e9adb070d2b04dc397aa14142a30d5db9312babba34cab1ab5d82f43a not found: ID does not exist" containerID="9a0a5d8e9adb070d2b04dc397aa14142a30d5db9312babba34cab1ab5d82f43a" Jul 20 13:50:27.127707 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.127652 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0a5d8e9adb070d2b04dc397aa14142a30d5db9312babba34cab1ab5d82f43a"} err="failed to get container status \"9a0a5d8e9adb070d2b04dc397aa14142a30d5db9312babba34cab1ab5d82f43a\": rpc error: code = NotFound desc = could not find container \"9a0a5d8e9adb070d2b04dc397aa14142a30d5db9312babba34cab1ab5d82f43a\": container with ID starting with 9a0a5d8e9adb070d2b04dc397aa14142a30d5db9312babba34cab1ab5d82f43a not found: ID does not exist" Jul 20 13:50:27.197112 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.197091 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ff603b-fa44-43c0-89a4-1ef36f470af6-maas-api-tls\") pod \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\" (UID: \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\") " Jul 20 13:50:27.197210 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.197130 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ff603b-fa44-43c0-89a4-1ef36f470af6-openshift-service-ca\") pod \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\" (UID: \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\") " Jul 20 13:50:27.197210 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.197158 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mpqg\" (UniqueName: \"kubernetes.io/projected/e7ff603b-fa44-43c0-89a4-1ef36f470af6-kube-api-access-4mpqg\") pod \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\" (UID: \"e7ff603b-fa44-43c0-89a4-1ef36f470af6\") " Jul 20 13:50:27.197424 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.197401 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ff603b-fa44-43c0-89a4-1ef36f470af6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e7ff603b-fa44-43c0-89a4-1ef36f470af6" (UID: "e7ff603b-fa44-43c0-89a4-1ef36f470af6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:50:27.199023 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.199001 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7ff603b-fa44-43c0-89a4-1ef36f470af6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e7ff603b-fa44-43c0-89a4-1ef36f470af6" (UID: "e7ff603b-fa44-43c0-89a4-1ef36f470af6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:50:27.199171 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.199152 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ff603b-fa44-43c0-89a4-1ef36f470af6-kube-api-access-4mpqg" (OuterVolumeSpecName: "kube-api-access-4mpqg") pod "e7ff603b-fa44-43c0-89a4-1ef36f470af6" (UID: "e7ff603b-fa44-43c0-89a4-1ef36f470af6"). InnerVolumeSpecName "kube-api-access-4mpqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:50:27.298603 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.298572 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mpqg\" (UniqueName: \"kubernetes.io/projected/e7ff603b-fa44-43c0-89a4-1ef36f470af6-kube-api-access-4mpqg\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:50:27.298603 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.298602 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ff603b-fa44-43c0-89a4-1ef36f470af6-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:50:27.298603 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.298611 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ff603b-fa44-43c0-89a4-1ef36f470af6-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:50:27.439637 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.439616 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s"] Jul 20 13:50:27.446135 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:27.446113 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5dcc91bd-586d4c649f-wf46s"] Jul 20 13:50:28.307635 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:28.307599 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7ff603b-fa44-43c0-89a4-1ef36f470af6" path="/var/lib/kubelet/pods/e7ff603b-fa44-43c0-89a4-1ef36f470af6/volumes" Jul 20 13:50:29.844891 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:29.844838 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:50:31.330407 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:31.330374 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm"] Jul 20 13:50:31.332365 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:31.332342 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5dcc91bd-tkdpm"] Jul 20 13:50:32.308420 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:32.308388 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a49c0eae-640a-4d0a-aadc-78239c2cafe9" path="/var/lib/kubelet/pods/a49c0eae-640a-4d0a-aadc-78239c2cafe9/volumes" Jul 20 13:50:34.487685 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.487656 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q"] Jul 20 13:50:34.488049 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.487998 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49c0eae-640a-4d0a-aadc-78239c2cafe9" containerName="revoke-keys" Jul 20 13:50:34.488049 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.488009 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49c0eae-640a-4d0a-aadc-78239c2cafe9" containerName="revoke-keys" Jul 20 13:50:34.488049 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.488019 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ff603b-fa44-43c0-89a4-1ef36f470af6" containerName="maas-api" Jul 20 13:50:34.488049 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.488024 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ff603b-fa44-43c0-89a4-1ef36f470af6" containerName="maas-api" Jul 20 13:50:34.488192 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.488083 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49c0eae-640a-4d0a-aadc-78239c2cafe9" containerName="revoke-keys" Jul 20 13:50:34.488192 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.488093 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7ff603b-fa44-43c0-89a4-1ef36f470af6" containerName="maas-api" Jul 20 13:50:34.492341 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.492319 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:50:34.497157 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.497135 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-358b8783-serving-cert\"" Jul 20 13:50:34.503042 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.503021 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q"] Jul 20 13:50:34.557698 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.557676 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g74bn\" (UniqueName: \"kubernetes.io/projected/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-kube-api-access-g74bn\") pod \"maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q\" (UID: \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:50:34.557821 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.557732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-openshift-service-ca\") pod \"maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q\" (UID: \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:50:34.557916 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.557837 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-maas-api-tls\") pod \"maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q\" (UID: \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:50:34.658224 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.658201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-maas-api-tls\") pod \"maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q\" (UID: \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:50:34.658350 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.658238 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g74bn\" (UniqueName: \"kubernetes.io/projected/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-kube-api-access-g74bn\") pod \"maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q\" (UID: \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:50:34.658350 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.658266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-openshift-service-ca\") pod \"maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q\" (UID: \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:50:34.658454 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:50:34.658346 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-serving-cert: secret "maas-api-e2e-ait-358b8783-serving-cert" not found Jul 20 13:50:34.658454 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:50:34.658415 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-maas-api-tls podName:d3ab5ecf-7f1f-40d8-9e1d-325656f33494 nodeName:}" failed. No retries permitted until 2026-07-20 13:50:35.158398911 +0000 UTC m=+2029.366795033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-maas-api-tls") pod "maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" (UID: "d3ab5ecf-7f1f-40d8-9e1d-325656f33494") : secret "maas-api-e2e-ait-358b8783-serving-cert" not found Jul 20 13:50:34.658851 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.658833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-openshift-service-ca\") pod \"maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q\" (UID: \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:50:34.672829 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:34.672803 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g74bn\" (UniqueName: \"kubernetes.io/projected/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-kube-api-access-g74bn\") pod \"maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q\" (UID: \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:50:35.162101 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:35.162064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-maas-api-tls\") pod \"maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q\" (UID: \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:50:35.164570 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:35.164544 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-maas-api-tls\") pod \"maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q\" (UID: \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:50:35.402476 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:35.402449 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:50:35.539336 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:35.539314 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q"] Jul 20 13:50:35.541084 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:50:35.541052 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ab5ecf_7f1f_40d8_9e1d_325656f33494.slice/crio-5651c3ddff72e4a2145484aaf3d0c072317b3acaec2e4d3c5ea9856e6023b9d9 WatchSource:0}: Error finding container 5651c3ddff72e4a2145484aaf3d0c072317b3acaec2e4d3c5ea9856e6023b9d9: Status 404 returned error can't find the container with id 5651c3ddff72e4a2145484aaf3d0c072317b3acaec2e4d3c5ea9856e6023b9d9 Jul 20 13:50:35.542662 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:35.542641 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 13:50:36.159276 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:36.159194 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" event={"ID":"d3ab5ecf-7f1f-40d8-9e1d-325656f33494","Type":"ContainerStarted","Data":"4a929d35da356b325ad034fced1cfe28d6351957b99eeefbbafa2a8f81256227"} Jul 20 13:50:36.159276 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:36.159246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" event={"ID":"d3ab5ecf-7f1f-40d8-9e1d-325656f33494","Type":"ContainerStarted","Data":"5651c3ddff72e4a2145484aaf3d0c072317b3acaec2e4d3c5ea9856e6023b9d9"} Jul 20 13:50:36.159441 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:36.159277 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:50:36.195152 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:36.195074 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" podStartSLOduration=1.839416355 podStartE2EDuration="2.195058803s" totalImagesPullingTime="355.642448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:50:36.19465681 +0000 UTC m=+2030.403052957" watchObservedRunningTime="2026-07-20 13:50:36.195058803 +0000 UTC m=+2030.403454948" Jul 20 13:50:41.246514 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:41.246479 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:50:42.170297 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:50:42.170268 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:51:00.280912 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:00.280866 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr"] Jul 20 13:51:00.284133 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:00.284117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr" Jul 20 13:51:00.286244 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:00.286220 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t89s5\"" Jul 20 13:51:00.292371 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:00.292349 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr"] Jul 20 13:51:00.447801 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:00.447763 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0727f787-d216-49e7-a52c-0e48de6dfc89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-358b8783-gl7fr\" (UID: \"0727f787-d216-49e7-a52c-0e48de6dfc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr" Jul 20 13:51:00.548953 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:00.548924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0727f787-d216-49e7-a52c-0e48de6dfc89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-358b8783-gl7fr\" (UID: \"0727f787-d216-49e7-a52c-0e48de6dfc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr" Jul 20 13:51:00.549508 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:00.549473 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0727f787-d216-49e7-a52c-0e48de6dfc89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-358b8783-gl7fr\" (UID: \"0727f787-d216-49e7-a52c-0e48de6dfc89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr" Jul 20 13:51:00.594743 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:00.594706 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr" Jul 20 13:51:00.720272 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:00.720245 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr"] Jul 20 13:51:00.722219 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:51:00.722193 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0727f787_d216_49e7_a52c_0e48de6dfc89.slice/crio-c401dcb742c2d28bdec577ad8220308db99cc0d34f97d0967d116a6585e25a38 WatchSource:0}: Error finding container c401dcb742c2d28bdec577ad8220308db99cc0d34f97d0967d116a6585e25a38: Status 404 returned error can't find the container with id c401dcb742c2d28bdec577ad8220308db99cc0d34f97d0967d116a6585e25a38 Jul 20 13:51:01.270103 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:01.270018 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr" event={"ID":"0727f787-d216-49e7-a52c-0e48de6dfc89","Type":"ContainerStarted","Data":"1b56315bcd95909ae537ae2fbc494469f77e4bc9ddfd97a85cd48e29322fe0e5"} Jul 20 13:51:01.270103 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:01.270068 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr" event={"ID":"0727f787-d216-49e7-a52c-0e48de6dfc89","Type":"ContainerStarted","Data":"c401dcb742c2d28bdec577ad8220308db99cc0d34f97d0967d116a6585e25a38"} Jul 20 13:51:01.291360 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:01.291308 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr" podStartSLOduration=1.291294232 podStartE2EDuration="1.291294232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:51:01.289218085 +0000 UTC m=+2055.497614228" watchObservedRunningTime="2026-07-20 13:51:01.291294232 +0000 UTC m=+2055.499690375" Jul 20 13:51:01.370317 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:51:01.370286 2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0727f787_d216_49e7_a52c_0e48de6dfc89.slice/crio-1b56315bcd95909ae537ae2fbc494469f77e4bc9ddfd97a85cd48e29322fe0e5.scope\": RecentStats: unable to find data in memory cache]" Jul 20 13:51:02.275901 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:02.275853 2575 generic.go:356] "Generic (PLEG): container finished" podID="0727f787-d216-49e7-a52c-0e48de6dfc89" containerID="1b56315bcd95909ae537ae2fbc494469f77e4bc9ddfd97a85cd48e29322fe0e5" exitCode=0 Jul 20 13:51:02.276080 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:02.275923 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr" event={"ID":"0727f787-d216-49e7-a52c-0e48de6dfc89","Type":"ContainerDied","Data":"1b56315bcd95909ae537ae2fbc494469f77e4bc9ddfd97a85cd48e29322fe0e5"} Jul 20 13:51:03.404145 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:03.404125 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr" Jul 20 13:51:03.573191 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:03.573164 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0727f787-d216-49e7-a52c-0e48de6dfc89-maas-api-service-ca\") pod \"0727f787-d216-49e7-a52c-0e48de6dfc89\" (UID: \"0727f787-d216-49e7-a52c-0e48de6dfc89\") " Jul 20 13:51:03.573473 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:03.573453 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0727f787-d216-49e7-a52c-0e48de6dfc89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0727f787-d216-49e7-a52c-0e48de6dfc89" (UID: "0727f787-d216-49e7-a52c-0e48de6dfc89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:51:03.674155 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:03.674130 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0727f787-d216-49e7-a52c-0e48de6dfc89-maas-api-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:51:04.286076 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:04.286049 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr" Jul 20 13:51:04.286256 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:04.286043 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr" event={"ID":"0727f787-d216-49e7-a52c-0e48de6dfc89","Type":"ContainerDied","Data":"c401dcb742c2d28bdec577ad8220308db99cc0d34f97d0967d116a6585e25a38"} Jul 20 13:51:04.286256 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:04.286190 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c401dcb742c2d28bdec577ad8220308db99cc0d34f97d0967d116a6585e25a38" Jul 20 13:51:10.747279 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:10.747158 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q"] Jul 20 13:51:10.747717 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:10.747475 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" podUID="d3ab5ecf-7f1f-40d8-9e1d-325656f33494" containerName="maas-api" containerID="cri-o://4a929d35da356b325ad034fced1cfe28d6351957b99eeefbbafa2a8f81256227" gracePeriod=30 Jul 20 13:51:10.995443 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:10.995420 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:51:11.129827 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.129791 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-openshift-service-ca\") pod \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\" (UID: \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\") " Jul 20 13:51:11.130038 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.129833 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-maas-api-tls\") pod \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\" (UID: \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\") " Jul 20 13:51:11.130038 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.129945 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g74bn\" (UniqueName: \"kubernetes.io/projected/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-kube-api-access-g74bn\") pod \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\" (UID: \"d3ab5ecf-7f1f-40d8-9e1d-325656f33494\") " Jul 20 13:51:11.130252 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.130220 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3ab5ecf-7f1f-40d8-9e1d-325656f33494" (UID: "d3ab5ecf-7f1f-40d8-9e1d-325656f33494"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:51:11.131957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.131933 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-kube-api-access-g74bn" (OuterVolumeSpecName: "kube-api-access-g74bn") pod "d3ab5ecf-7f1f-40d8-9e1d-325656f33494" (UID: "d3ab5ecf-7f1f-40d8-9e1d-325656f33494"). InnerVolumeSpecName "kube-api-access-g74bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:51:11.131957 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.131949 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3ab5ecf-7f1f-40d8-9e1d-325656f33494" (UID: "d3ab5ecf-7f1f-40d8-9e1d-325656f33494"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:51:11.230484 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.230462 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g74bn\" (UniqueName: \"kubernetes.io/projected/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-kube-api-access-g74bn\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:51:11.230484 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.230483 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:51:11.230632 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.230495 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ab5ecf-7f1f-40d8-9e1d-325656f33494-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:51:11.314931 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.314900 2575 generic.go:356] "Generic (PLEG): container finished" podID="d3ab5ecf-7f1f-40d8-9e1d-325656f33494" containerID="4a929d35da356b325ad034fced1cfe28d6351957b99eeefbbafa2a8f81256227" exitCode=0 Jul 20 13:51:11.315047 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.314986 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" Jul 20 13:51:11.315047 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.315013 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" event={"ID":"d3ab5ecf-7f1f-40d8-9e1d-325656f33494","Type":"ContainerDied","Data":"4a929d35da356b325ad034fced1cfe28d6351957b99eeefbbafa2a8f81256227"} Jul 20 13:51:11.315163 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.315069 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q" event={"ID":"d3ab5ecf-7f1f-40d8-9e1d-325656f33494","Type":"ContainerDied","Data":"5651c3ddff72e4a2145484aaf3d0c072317b3acaec2e4d3c5ea9856e6023b9d9"} Jul 20 13:51:11.315163 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.315103 2575 scope.go:117] "RemoveContainer" containerID="4a929d35da356b325ad034fced1cfe28d6351957b99eeefbbafa2a8f81256227" Jul 20 13:51:11.325760 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.324968 2575 scope.go:117] "RemoveContainer" containerID="4a929d35da356b325ad034fced1cfe28d6351957b99eeefbbafa2a8f81256227" Jul 20 13:51:11.325760 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:51:11.325745 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a929d35da356b325ad034fced1cfe28d6351957b99eeefbbafa2a8f81256227\": container with ID starting with 4a929d35da356b325ad034fced1cfe28d6351957b99eeefbbafa2a8f81256227 not found: ID does not exist" containerID="4a929d35da356b325ad034fced1cfe28d6351957b99eeefbbafa2a8f81256227" Jul 20 13:51:11.325856 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.325771 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a929d35da356b325ad034fced1cfe28d6351957b99eeefbbafa2a8f81256227"} err="failed to get container status \"4a929d35da356b325ad034fced1cfe28d6351957b99eeefbbafa2a8f81256227\": rpc error: code = NotFound desc = could not find container \"4a929d35da356b325ad034fced1cfe28d6351957b99eeefbbafa2a8f81256227\": container with ID starting with 4a929d35da356b325ad034fced1cfe28d6351957b99eeefbbafa2a8f81256227 not found: ID does not exist" Jul 20 13:51:11.335422 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.335398 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q"] Jul 20 13:51:11.338790 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:11.338771 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-358b8783-7bd644cf8c-h7f6q"] Jul 20 13:51:12.308174 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:12.308142 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3ab5ecf-7f1f-40d8-9e1d-325656f33494" path="/var/lib/kubelet/pods/d3ab5ecf-7f1f-40d8-9e1d-325656f33494/volumes" Jul 20 13:51:13.748629 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:13.748594 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:51:15.363462 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:15.363434 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr"] Jul 20 13:51:15.365809 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:15.365785 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-358b8783-gl7fr"] Jul 20 13:51:16.308403 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:16.308372 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0727f787-d216-49e7-a52c-0e48de6dfc89" path="/var/lib/kubelet/pods/0727f787-d216-49e7-a52c-0e48de6dfc89/volumes" Jul 20 13:51:19.883583 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:19.883549 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs"] Jul 20 13:51:19.883974 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:19.883865 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0727f787-d216-49e7-a52c-0e48de6dfc89" containerName="revoke-keys" Jul 20 13:51:19.883974 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:19.883898 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0727f787-d216-49e7-a52c-0e48de6dfc89" containerName="revoke-keys" Jul 20 13:51:19.883974 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:19.883914 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ab5ecf-7f1f-40d8-9e1d-325656f33494" containerName="maas-api" Jul 20 13:51:19.883974 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:19.883919 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ab5ecf-7f1f-40d8-9e1d-325656f33494" containerName="maas-api" Jul 20 13:51:19.883974 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:19.883975 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3ab5ecf-7f1f-40d8-9e1d-325656f33494" containerName="maas-api" Jul 20 13:51:19.884127 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:19.883984 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0727f787-d216-49e7-a52c-0e48de6dfc89" containerName="revoke-keys" Jul 20 13:51:19.888231 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:19.888213 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:19.891054 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:19.891031 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b79b63cc-serving-cert\"" Jul 20 13:51:19.903249 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:19.903226 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs"] Jul 20 13:51:19.987755 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:19.987719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh5g7\" (UniqueName: \"kubernetes.io/projected/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-kube-api-access-xh5g7\") pod \"maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs\" (UID: \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:19.987920 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:19.987762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-maas-api-tls\") pod \"maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs\" (UID: \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:19.987920 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:19.987827 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-openshift-service-ca\") pod \"maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs\" (UID: \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:20.088357 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:20.088318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-maas-api-tls\") pod \"maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs\" (UID: \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:20.088513 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:20.088397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-openshift-service-ca\") pod \"maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs\" (UID: \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:20.088513 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:20.088430 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh5g7\" (UniqueName: \"kubernetes.io/projected/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-kube-api-access-xh5g7\") pod \"maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs\" (UID: \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:20.088513 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:51:20.088461 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-serving-cert: secret "maas-api-e2e-derive-b79b63cc-serving-cert" not found Jul 20 13:51:20.088652 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:51:20.088525 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-maas-api-tls podName:bc6d1d50-bbb9-4860-92bc-dc28e2786ea9 nodeName:}" failed. No retries permitted until 2026-07-20 13:51:20.58850926 +0000 UTC m=+2074.796905383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-maas-api-tls") pod "maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" (UID: "bc6d1d50-bbb9-4860-92bc-dc28e2786ea9") : secret "maas-api-e2e-derive-b79b63cc-serving-cert" not found Jul 20 13:51:20.089190 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:20.089168 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-openshift-service-ca\") pod \"maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs\" (UID: \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:20.099381 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:20.099355 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh5g7\" (UniqueName: \"kubernetes.io/projected/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-kube-api-access-xh5g7\") pod \"maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs\" (UID: \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:20.591668 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:20.591634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-maas-api-tls\") pod \"maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs\" (UID: \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:20.594153 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:20.594028 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-maas-api-tls\") pod \"maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs\" (UID: \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:20.798045 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:20.798016 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:20.953037 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:20.953006 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs"] Jul 20 13:51:20.956062 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:51:20.956038 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6d1d50_bbb9_4860_92bc_dc28e2786ea9.slice/crio-8e95394029a00954d79680846e921dc37048e715f9219493309542416b431a15 WatchSource:0}: Error finding container 8e95394029a00954d79680846e921dc37048e715f9219493309542416b431a15: Status 404 returned error can't find the container with id 8e95394029a00954d79680846e921dc37048e715f9219493309542416b431a15 Jul 20 13:51:21.356484 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:21.356449 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" event={"ID":"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9","Type":"ContainerStarted","Data":"8e95394029a00954d79680846e921dc37048e715f9219493309542416b431a15"} Jul 20 13:51:22.362511 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:22.362477 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" event={"ID":"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9","Type":"ContainerStarted","Data":"6689c6e00cd6e567c8b9d36c56dc12f151aa6a4bf6680c5ae5c355139c4268d3"} Jul 20 13:51:22.362945 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:22.362616 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:22.382327 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:22.382277 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" podStartSLOduration=2.74034571 podStartE2EDuration="3.382261179s" totalImagesPullingTime="641.915469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:51:22.381488623 +0000 UTC m=+2076.589884750" watchObservedRunningTime="2026-07-20 13:51:22.382261179 +0000 UTC m=+2076.590657325" Jul 20 13:51:23.370613 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:23.370581 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:51:28.373510 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:28.373473 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:44.642448 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:44.642406 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz"] Jul 20 13:51:44.646322 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:44.646298 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz" Jul 20 13:51:44.648262 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:44.648239 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t89s5\"" Jul 20 13:51:44.649984 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:44.649964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc6974e-6d46-489f-9b7c-ca7a17cdfd49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz\" (UID: \"2fc6974e-6d46-489f-9b7c-ca7a17cdfd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz" Jul 20 13:51:44.653966 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:44.653692 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz"] Jul 20 13:51:44.750730 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:44.750696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc6974e-6d46-489f-9b7c-ca7a17cdfd49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz\" (UID: \"2fc6974e-6d46-489f-9b7c-ca7a17cdfd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz" Jul 20 13:51:44.751327 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:44.751299 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc6974e-6d46-489f-9b7c-ca7a17cdfd49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz\" (UID: \"2fc6974e-6d46-489f-9b7c-ca7a17cdfd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz" Jul 20 13:51:44.957734 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:44.957653 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz" Jul 20 13:51:45.092216 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:45.092088 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz"] Jul 20 13:51:45.094480 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:51:45.094451 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc6974e_6d46_489f_9b7c_ca7a17cdfd49.slice/crio-a48c140417dc015870d4c0d024b0c200ff78749241c45e7804f57c5f118ff322 WatchSource:0}: Error finding container a48c140417dc015870d4c0d024b0c200ff78749241c45e7804f57c5f118ff322: Status 404 returned error can't find the container with id a48c140417dc015870d4c0d024b0c200ff78749241c45e7804f57c5f118ff322 Jul 20 13:51:45.461887 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:45.461850 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz" event={"ID":"2fc6974e-6d46-489f-9b7c-ca7a17cdfd49","Type":"ContainerStarted","Data":"eb914b9c6f0f54f8bab0b80e5b3c7b1a7c2b65c11e32d4fd8dd0b8777197d3e7"} Jul 20 13:51:45.462065 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:45.461911 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz" event={"ID":"2fc6974e-6d46-489f-9b7c-ca7a17cdfd49","Type":"ContainerStarted","Data":"a48c140417dc015870d4c0d024b0c200ff78749241c45e7804f57c5f118ff322"} Jul 20 13:51:45.476783 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:45.476733 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz" podStartSLOduration=1.476718876 podStartE2EDuration="1.476718876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:51:45.475629112 +0000 UTC m=+2099.684025251" watchObservedRunningTime="2026-07-20 13:51:45.476718876 +0000 UTC m=+2099.685115020" Jul 20 13:51:46.467458 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:46.467422 2575 generic.go:356] "Generic (PLEG): container finished" podID="2fc6974e-6d46-489f-9b7c-ca7a17cdfd49" containerID="eb914b9c6f0f54f8bab0b80e5b3c7b1a7c2b65c11e32d4fd8dd0b8777197d3e7" exitCode=0 Jul 20 13:51:46.467842 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:46.467501 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz" event={"ID":"2fc6974e-6d46-489f-9b7c-ca7a17cdfd49","Type":"ContainerDied","Data":"eb914b9c6f0f54f8bab0b80e5b3c7b1a7c2b65c11e32d4fd8dd0b8777197d3e7"} Jul 20 13:51:47.619752 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:47.619727 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz" Jul 20 13:51:47.680299 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:47.680268 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc6974e-6d46-489f-9b7c-ca7a17cdfd49-maas-api-service-ca\") pod \"2fc6974e-6d46-489f-9b7c-ca7a17cdfd49\" (UID: \"2fc6974e-6d46-489f-9b7c-ca7a17cdfd49\") " Jul 20 13:51:47.680596 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:47.680569 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc6974e-6d46-489f-9b7c-ca7a17cdfd49-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2fc6974e-6d46-489f-9b7c-ca7a17cdfd49" (UID: "2fc6974e-6d46-489f-9b7c-ca7a17cdfd49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:51:47.781268 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:47.781207 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc6974e-6d46-489f-9b7c-ca7a17cdfd49-maas-api-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:51:48.477241 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:48.477202 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz" event={"ID":"2fc6974e-6d46-489f-9b7c-ca7a17cdfd49","Type":"ContainerDied","Data":"a48c140417dc015870d4c0d024b0c200ff78749241c45e7804f57c5f118ff322"} Jul 20 13:51:48.477416 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:48.477246 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz" Jul 20 13:51:48.477416 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:48.477255 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a48c140417dc015870d4c0d024b0c200ff78749241c45e7804f57c5f118ff322" Jul 20 13:51:55.118998 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.114160 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs"] Jul 20 13:51:55.118998 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.114509 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" podUID="bc6d1d50-bbb9-4860-92bc-dc28e2786ea9" containerName="maas-api" containerID="cri-o://6689c6e00cd6e567c8b9d36c56dc12f151aa6a4bf6680c5ae5c355139c4268d3" gracePeriod=30 Jul 20 13:51:55.358634 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.358610 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:55.442006 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.441928 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-openshift-service-ca\") pod \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\" (UID: \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\") " Jul 20 13:51:55.442151 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.442003 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-maas-api-tls\") pod \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\" (UID: \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\") " Jul 20 13:51:55.442151 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.442092 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh5g7\" (UniqueName: \"kubernetes.io/projected/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-kube-api-access-xh5g7\") pod \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\" (UID: \"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9\") " Jul 20 13:51:55.442303 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.442277 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bc6d1d50-bbb9-4860-92bc-dc28e2786ea9" (UID: "bc6d1d50-bbb9-4860-92bc-dc28e2786ea9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:51:55.443901 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.443864 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bc6d1d50-bbb9-4860-92bc-dc28e2786ea9" (UID: "bc6d1d50-bbb9-4860-92bc-dc28e2786ea9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:51:55.444036 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.444016 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-kube-api-access-xh5g7" (OuterVolumeSpecName: "kube-api-access-xh5g7") pod "bc6d1d50-bbb9-4860-92bc-dc28e2786ea9" (UID: "bc6d1d50-bbb9-4860-92bc-dc28e2786ea9"). InnerVolumeSpecName "kube-api-access-xh5g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:51:55.508392 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.508368 2575 generic.go:356] "Generic (PLEG): container finished" podID="bc6d1d50-bbb9-4860-92bc-dc28e2786ea9" containerID="6689c6e00cd6e567c8b9d36c56dc12f151aa6a4bf6680c5ae5c355139c4268d3" exitCode=0 Jul 20 13:51:55.508503 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.508425 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" Jul 20 13:51:55.508503 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.508453 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" event={"ID":"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9","Type":"ContainerDied","Data":"6689c6e00cd6e567c8b9d36c56dc12f151aa6a4bf6680c5ae5c355139c4268d3"} Jul 20 13:51:55.508615 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.508511 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs" event={"ID":"bc6d1d50-bbb9-4860-92bc-dc28e2786ea9","Type":"ContainerDied","Data":"8e95394029a00954d79680846e921dc37048e715f9219493309542416b431a15"} Jul 20 13:51:55.508615 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.508535 2575 scope.go:117] "RemoveContainer" containerID="6689c6e00cd6e567c8b9d36c56dc12f151aa6a4bf6680c5ae5c355139c4268d3" Jul 20 13:51:55.516317 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.516303 2575 scope.go:117] "RemoveContainer" containerID="6689c6e00cd6e567c8b9d36c56dc12f151aa6a4bf6680c5ae5c355139c4268d3" Jul 20 13:51:55.516606 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:51:55.516585 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6689c6e00cd6e567c8b9d36c56dc12f151aa6a4bf6680c5ae5c355139c4268d3\": container with ID starting with 6689c6e00cd6e567c8b9d36c56dc12f151aa6a4bf6680c5ae5c355139c4268d3 not found: ID does not exist" containerID="6689c6e00cd6e567c8b9d36c56dc12f151aa6a4bf6680c5ae5c355139c4268d3" Jul 20 13:51:55.516728 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.516614 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6689c6e00cd6e567c8b9d36c56dc12f151aa6a4bf6680c5ae5c355139c4268d3"} err="failed to get container status \"6689c6e00cd6e567c8b9d36c56dc12f151aa6a4bf6680c5ae5c355139c4268d3\": rpc error: code = NotFound desc = could not find container \"6689c6e00cd6e567c8b9d36c56dc12f151aa6a4bf6680c5ae5c355139c4268d3\": container with ID starting with 6689c6e00cd6e567c8b9d36c56dc12f151aa6a4bf6680c5ae5c355139c4268d3 not found: ID does not exist" Jul 20 13:51:55.531522 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.531502 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs"] Jul 20 13:51:55.535411 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.535370 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b79b63cc-7fdccc475f-9psqs"] Jul 20 13:51:55.543646 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.543629 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:51:55.543726 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.543650 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh5g7\" (UniqueName: \"kubernetes.io/projected/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-kube-api-access-xh5g7\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:51:55.543726 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:55.543660 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:51:56.308376 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:56.308339 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc6d1d50-bbb9-4860-92bc-dc28e2786ea9" path="/var/lib/kubelet/pods/bc6d1d50-bbb9-4860-92bc-dc28e2786ea9/volumes" Jul 20 13:51:58.145963 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:58.145925 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:51:59.718834 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:59.718799 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz"] Jul 20 13:51:59.721316 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:51:59.721291 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b79b63cc-cfpqz"] Jul 20 13:52:00.307952 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:52:00.307917 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fc6974e-6d46-489f-9b7c-ca7a17cdfd49" path="/var/lib/kubelet/pods/2fc6974e-6d46-489f-9b7c-ca7a17cdfd49/volumes" Jul 20 13:53:25.946412 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:25.946380 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:53:35.350196 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:35.350156 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:53:41.314100 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.314069 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g"] Jul 20 13:53:41.314465 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.314353 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc6d1d50-bbb9-4860-92bc-dc28e2786ea9" containerName="maas-api" Jul 20 13:53:41.314465 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.314363 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc6d1d50-bbb9-4860-92bc-dc28e2786ea9" containerName="maas-api" Jul 20 13:53:41.314465 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.314382 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fc6974e-6d46-489f-9b7c-ca7a17cdfd49" containerName="revoke-keys" Jul 20 13:53:41.314465 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.314388 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fc6974e-6d46-489f-9b7c-ca7a17cdfd49" containerName="revoke-keys" Jul 20 13:53:41.314465 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.314449 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc6d1d50-bbb9-4860-92bc-dc28e2786ea9" containerName="maas-api" Jul 20 13:53:41.314465 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.314456 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fc6974e-6d46-489f-9b7c-ca7a17cdfd49" containerName="revoke-keys" Jul 20 13:53:41.317477 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.317450 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:53:41.319155 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.319130 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ffcdeab1-serving-cert\"" Jul 20 13:53:41.327413 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.327390 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g"] Jul 20 13:53:41.396349 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.396324 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bcxm\" (UniqueName: \"kubernetes.io/projected/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-kube-api-access-9bcxm\") pod \"maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g\" (UID: \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:53:41.396491 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.396374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-openshift-service-ca\") pod \"maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g\" (UID: \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:53:41.396557 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.396511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-maas-api-tls\") pod \"maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g\" (UID: \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:53:41.496954 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.496915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-maas-api-tls\") pod \"maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g\" (UID: \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:53:41.497135 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.497014 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bcxm\" (UniqueName: \"kubernetes.io/projected/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-kube-api-access-9bcxm\") pod \"maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g\" (UID: \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:53:41.497135 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.497069 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-openshift-service-ca\") pod \"maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g\" (UID: \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:53:41.497853 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.497810 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-openshift-service-ca\") pod \"maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g\" (UID: \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:53:41.499668 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.499645 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-maas-api-tls\") pod \"maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g\" (UID: \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:53:41.506454 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.506432 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bcxm\" (UniqueName: \"kubernetes.io/projected/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-kube-api-access-9bcxm\") pod \"maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g\" (UID: \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:53:41.627788 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.627728 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:53:41.755427 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.755402 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g"] Jul 20 13:53:41.757653 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:53:41.757617 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b956ae6_3aa0_4b5e_8eef_bd64212f28ca.slice/crio-9b7010089968cbdc8eb4cef5c19d9d29e76e28e943c6c6f40db921fb732b2f8f WatchSource:0}: Error finding container 9b7010089968cbdc8eb4cef5c19d9d29e76e28e943c6c6f40db921fb732b2f8f: Status 404 returned error can't find the container with id 9b7010089968cbdc8eb4cef5c19d9d29e76e28e943c6c6f40db921fb732b2f8f Jul 20 13:53:41.982240 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:41.982167 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" event={"ID":"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca","Type":"ContainerStarted","Data":"9b7010089968cbdc8eb4cef5c19d9d29e76e28e943c6c6f40db921fb732b2f8f"} Jul 20 13:53:42.988145 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:42.988111 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" event={"ID":"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca","Type":"ContainerStarted","Data":"ea2e3316d93dbc5c6735ce4aeb7c52b0e56488b006bf0a29fa36c7a534c839c8"} Jul 20 13:53:42.988511 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:42.988248 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:53:43.010214 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:43.010164 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" podStartSLOduration=1.558177371 podStartE2EDuration="2.010145917s" totalImagesPullingTime="451.968546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:53:43.008214426 +0000 UTC m=+2217.216610571" watchObservedRunningTime="2026-07-20 13:53:43.010145917 +0000 UTC m=+2217.218542076" Jul 20 13:53:46.352814 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:46.352777 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:53:48.999280 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:48.999248 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:53:53.225920 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:53.225865 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9"] Jul 20 13:53:53.229355 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:53.229336 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9" Jul 20 13:53:53.231381 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:53.231362 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t89s5\"" Jul 20 13:53:53.236177 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:53.236150 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9"] Jul 20 13:53:53.285113 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:53.285084 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa2d125-e6fb-46f0-93de-b09acd42e668-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9\" (UID: \"5fa2d125-e6fb-46f0-93de-b09acd42e668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9" Jul 20 13:53:53.386175 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:53.386148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa2d125-e6fb-46f0-93de-b09acd42e668-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9\" (UID: \"5fa2d125-e6fb-46f0-93de-b09acd42e668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9" Jul 20 13:53:53.386724 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:53.386706 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa2d125-e6fb-46f0-93de-b09acd42e668-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9\" (UID: \"5fa2d125-e6fb-46f0-93de-b09acd42e668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9" Jul 20 13:53:53.542763 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:53.542738 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9" Jul 20 13:53:53.673831 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:53.673795 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9"] Jul 20 13:53:53.677759 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:53:53.677729 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa2d125_e6fb_46f0_93de_b09acd42e668.slice/crio-f00b30032356667258a11cc342e9e0715ffb06905e22490637c0b30e5d5adc16 WatchSource:0}: Error finding container f00b30032356667258a11cc342e9e0715ffb06905e22490637c0b30e5d5adc16: Status 404 returned error can't find the container with id f00b30032356667258a11cc342e9e0715ffb06905e22490637c0b30e5d5adc16 Jul 20 13:53:54.038140 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:54.038102 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9" event={"ID":"5fa2d125-e6fb-46f0-93de-b09acd42e668","Type":"ContainerStarted","Data":"c86e6cdbc1f259235c62527419b62dfd28697b815d6f07826d9e8d8696c8eba3"} Jul 20 13:53:54.038318 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:54.038166 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9" event={"ID":"5fa2d125-e6fb-46f0-93de-b09acd42e668","Type":"ContainerStarted","Data":"f00b30032356667258a11cc342e9e0715ffb06905e22490637c0b30e5d5adc16"} Jul 20 13:53:54.053088 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:54.053038 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9" podStartSLOduration=1.053020164 podStartE2EDuration="1.053020164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:53:54.052021383 +0000 UTC m=+2228.260417528" watchObservedRunningTime="2026-07-20 13:53:54.053020164 +0000 UTC m=+2228.261416309" Jul 20 13:53:55.043959 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:55.043925 2575 generic.go:356] "Generic (PLEG): container finished" podID="5fa2d125-e6fb-46f0-93de-b09acd42e668" containerID="c86e6cdbc1f259235c62527419b62dfd28697b815d6f07826d9e8d8696c8eba3" exitCode=0 Jul 20 13:53:55.043959 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:55.043962 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9" event={"ID":"5fa2d125-e6fb-46f0-93de-b09acd42e668","Type":"ContainerDied","Data":"c86e6cdbc1f259235c62527419b62dfd28697b815d6f07826d9e8d8696c8eba3"} Jul 20 13:53:56.175592 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:56.175567 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9" Jul 20 13:53:56.208161 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:56.208136 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa2d125-e6fb-46f0-93de-b09acd42e668-maas-api-service-ca\") pod \"5fa2d125-e6fb-46f0-93de-b09acd42e668\" (UID: \"5fa2d125-e6fb-46f0-93de-b09acd42e668\") " Jul 20 13:53:56.208443 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:56.208422 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa2d125-e6fb-46f0-93de-b09acd42e668-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5fa2d125-e6fb-46f0-93de-b09acd42e668" (UID: "5fa2d125-e6fb-46f0-93de-b09acd42e668"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:53:56.309553 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:56.309492 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa2d125-e6fb-46f0-93de-b09acd42e668-maas-api-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:53:57.054558 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:57.054522 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9" event={"ID":"5fa2d125-e6fb-46f0-93de-b09acd42e668","Type":"ContainerDied","Data":"f00b30032356667258a11cc342e9e0715ffb06905e22490637c0b30e5d5adc16"} Jul 20 13:53:57.054558 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:57.054553 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9" Jul 20 13:53:57.054753 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:53:57.054574 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f00b30032356667258a11cc342e9e0715ffb06905e22490637c0b30e5d5adc16" Jul 20 13:54:03.679992 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:03.679955 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g"] Jul 20 13:54:03.680438 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:03.680182 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" podUID="7b956ae6-3aa0-4b5e-8eef-bd64212f28ca" containerName="maas-api" containerID="cri-o://ea2e3316d93dbc5c6735ce4aeb7c52b0e56488b006bf0a29fa36c7a534c839c8" gracePeriod=30 Jul 20 13:54:03.926359 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:03.926335 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:54:03.968887 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:03.968786 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-maas-api-tls\") pod \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\" (UID: \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\") " Jul 20 13:54:03.969025 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:03.968897 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bcxm\" (UniqueName: \"kubernetes.io/projected/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-kube-api-access-9bcxm\") pod \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\" (UID: \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\") " Jul 20 13:54:03.969025 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:03.968931 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-openshift-service-ca\") pod \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\" (UID: \"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca\") " Jul 20 13:54:03.969271 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:03.969244 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b956ae6-3aa0-4b5e-8eef-bd64212f28ca" (UID: "7b956ae6-3aa0-4b5e-8eef-bd64212f28ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:54:03.971041 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:03.971011 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-kube-api-access-9bcxm" (OuterVolumeSpecName: "kube-api-access-9bcxm") pod "7b956ae6-3aa0-4b5e-8eef-bd64212f28ca" (UID: "7b956ae6-3aa0-4b5e-8eef-bd64212f28ca"). InnerVolumeSpecName "kube-api-access-9bcxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:54:03.971041 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:03.971011 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b956ae6-3aa0-4b5e-8eef-bd64212f28ca" (UID: "7b956ae6-3aa0-4b5e-8eef-bd64212f28ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:54:04.069796 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:04.069753 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:54:04.069796 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:04.069789 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bcxm\" (UniqueName: \"kubernetes.io/projected/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-kube-api-access-9bcxm\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:54:04.069796 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:04.069798 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:54:04.086588 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:04.086552 2575 generic.go:356] "Generic (PLEG): container finished" podID="7b956ae6-3aa0-4b5e-8eef-bd64212f28ca" containerID="ea2e3316d93dbc5c6735ce4aeb7c52b0e56488b006bf0a29fa36c7a534c839c8" exitCode=0 Jul 20 13:54:04.086729 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:04.086596 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" event={"ID":"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca","Type":"ContainerDied","Data":"ea2e3316d93dbc5c6735ce4aeb7c52b0e56488b006bf0a29fa36c7a534c839c8"} Jul 20 13:54:04.086729 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:04.086639 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" event={"ID":"7b956ae6-3aa0-4b5e-8eef-bd64212f28ca","Type":"ContainerDied","Data":"9b7010089968cbdc8eb4cef5c19d9d29e76e28e943c6c6f40db921fb732b2f8f"} Jul 20 13:54:04.086729 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:04.086661 2575 scope.go:117] "RemoveContainer" containerID="ea2e3316d93dbc5c6735ce4aeb7c52b0e56488b006bf0a29fa36c7a534c839c8" Jul 20 13:54:04.086729 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:04.086601 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g" Jul 20 13:54:04.095755 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:04.095737 2575 scope.go:117] "RemoveContainer" containerID="ea2e3316d93dbc5c6735ce4aeb7c52b0e56488b006bf0a29fa36c7a534c839c8" Jul 20 13:54:04.096053 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:54:04.096034 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2e3316d93dbc5c6735ce4aeb7c52b0e56488b006bf0a29fa36c7a534c839c8\": container with ID starting with ea2e3316d93dbc5c6735ce4aeb7c52b0e56488b006bf0a29fa36c7a534c839c8 not found: ID does not exist" containerID="ea2e3316d93dbc5c6735ce4aeb7c52b0e56488b006bf0a29fa36c7a534c839c8" Jul 20 13:54:04.096133 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:04.096062 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2e3316d93dbc5c6735ce4aeb7c52b0e56488b006bf0a29fa36c7a534c839c8"} err="failed to get container status \"ea2e3316d93dbc5c6735ce4aeb7c52b0e56488b006bf0a29fa36c7a534c839c8\": rpc error: code = NotFound desc = could not find container \"ea2e3316d93dbc5c6735ce4aeb7c52b0e56488b006bf0a29fa36c7a534c839c8\": container with ID starting with ea2e3316d93dbc5c6735ce4aeb7c52b0e56488b006bf0a29fa36c7a534c839c8 not found: ID does not exist" Jul 20 13:54:04.106841 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:04.106819 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g"] Jul 20 13:54:04.112547 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:04.112523 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ffcdeab1-6db488bc6d-6z79g"] Jul 20 13:54:04.308308 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:04.308268 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b956ae6-3aa0-4b5e-8eef-bd64212f28ca" path="/var/lib/kubelet/pods/7b956ae6-3aa0-4b5e-8eef-bd64212f28ca/volumes" Jul 20 13:54:06.544983 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:06.544949 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:54:08.309078 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:08.309006 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9"] Jul 20 13:54:08.309078 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:08.309040 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffcdeab1-5p8p9"] Jul 20 13:54:10.308587 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:10.308557 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fa2d125-e6fb-46f0-93de-b09acd42e668" path="/var/lib/kubelet/pods/5fa2d125-e6fb-46f0-93de-b09acd42e668/volumes" Jul 20 13:54:38.390859 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.390820 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924"] Jul 20 13:54:38.391346 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.391325 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fa2d125-e6fb-46f0-93de-b09acd42e668" containerName="revoke-keys" Jul 20 13:54:38.391390 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.391351 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fa2d125-e6fb-46f0-93de-b09acd42e668" containerName="revoke-keys" Jul 20 13:54:38.391390 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.391376 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b956ae6-3aa0-4b5e-8eef-bd64212f28ca" containerName="maas-api" Jul 20 13:54:38.391390 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.391385 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b956ae6-3aa0-4b5e-8eef-bd64212f28ca" containerName="maas-api" Jul 20 13:54:38.391481 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.391461 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b956ae6-3aa0-4b5e-8eef-bd64212f28ca" containerName="maas-api" Jul 20 13:54:38.391481 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.391473 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fa2d125-e6fb-46f0-93de-b09acd42e668" containerName="revoke-keys" Jul 20 13:54:38.393783 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.393756 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:54:38.395610 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.395583 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6a45cad8-serving-cert\"" Jul 20 13:54:38.403626 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.403603 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924"] Jul 20 13:54:38.460176 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.460141 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a71d09e-e669-479a-9bdb-a0e61299bc3e-openshift-service-ca\") pod \"maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924\" (UID: \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:54:38.460330 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.460186 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz8wt\" (UniqueName: \"kubernetes.io/projected/0a71d09e-e669-479a-9bdb-a0e61299bc3e-kube-api-access-dz8wt\") pod \"maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924\" (UID: \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:54:38.460330 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.460218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a71d09e-e669-479a-9bdb-a0e61299bc3e-maas-api-tls\") pod \"maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924\" (UID: \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:54:38.561609 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.561577 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a71d09e-e669-479a-9bdb-a0e61299bc3e-maas-api-tls\") pod \"maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924\" (UID: \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:54:38.561768 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.561673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a71d09e-e669-479a-9bdb-a0e61299bc3e-openshift-service-ca\") pod \"maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924\" (UID: \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:54:38.561768 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.561716 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz8wt\" (UniqueName: \"kubernetes.io/projected/0a71d09e-e669-479a-9bdb-a0e61299bc3e-kube-api-access-dz8wt\") pod \"maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924\" (UID: \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:54:38.562314 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.562293 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a71d09e-e669-479a-9bdb-a0e61299bc3e-openshift-service-ca\") pod \"maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924\" (UID: \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:54:38.563964 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.563947 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a71d09e-e669-479a-9bdb-a0e61299bc3e-maas-api-tls\") pod \"maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924\" (UID: \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:54:38.569341 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.569319 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz8wt\" (UniqueName: \"kubernetes.io/projected/0a71d09e-e669-479a-9bdb-a0e61299bc3e-kube-api-access-dz8wt\") pod \"maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924\" (UID: \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:54:38.705935 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.705837 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:54:38.834964 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:38.834937 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924"] Jul 20 13:54:38.836697 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:54:38.836668 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a71d09e_e669_479a_9bdb_a0e61299bc3e.slice/crio-48694a4c268a7e48582f458fff84c1b8e0b10c416e0d905d9da90696333c986c WatchSource:0}: Error finding container 48694a4c268a7e48582f458fff84c1b8e0b10c416e0d905d9da90696333c986c: Status 404 returned error can't find the container with id 48694a4c268a7e48582f458fff84c1b8e0b10c416e0d905d9da90696333c986c Jul 20 13:54:39.244192 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:39.244155 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" event={"ID":"0a71d09e-e669-479a-9bdb-a0e61299bc3e","Type":"ContainerStarted","Data":"48694a4c268a7e48582f458fff84c1b8e0b10c416e0d905d9da90696333c986c"} Jul 20 13:54:40.249656 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:40.249621 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" event={"ID":"0a71d09e-e669-479a-9bdb-a0e61299bc3e","Type":"ContainerStarted","Data":"3ddb2e668853f348c8ff3efcdc5913af70f0fd818191c1f6683970ae3b3645a0"} Jul 20 13:54:40.250076 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:40.249708 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:54:40.266397 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:40.266335 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" podStartSLOduration=1.785704824 podStartE2EDuration="2.266315597s" totalImagesPullingTime="480.610773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:54:40.266018211 +0000 UTC m=+2274.474414354" watchObservedRunningTime="2026-07-20 13:54:40.266315597 +0000 UTC m=+2274.474711757" Jul 20 13:54:44.943372 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:44.943335 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:54:46.258191 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:46.258162 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:54:53.251824 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:54:53.251789 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:55:07.353782 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:07.353749 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:55:10.946351 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:10.946316 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82"] Jul 20 13:55:10.949701 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:10.949676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82" Jul 20 13:55:10.951565 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:10.951548 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t89s5\"" Jul 20 13:55:10.957359 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:10.957340 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82"] Jul 20 13:55:11.019664 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:11.019628 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18592fc-4a64-447c-9fc5-658c8b6e38e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a45cad8-btq82\" (UID: \"f18592fc-4a64-447c-9fc5-658c8b6e38e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82" Jul 20 13:55:11.120326 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:11.120287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18592fc-4a64-447c-9fc5-658c8b6e38e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a45cad8-btq82\" (UID: \"f18592fc-4a64-447c-9fc5-658c8b6e38e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82" Jul 20 13:55:11.120950 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:11.120927 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18592fc-4a64-447c-9fc5-658c8b6e38e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a45cad8-btq82\" (UID: \"f18592fc-4a64-447c-9fc5-658c8b6e38e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82" Jul 20 13:55:11.261460 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:11.261401 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82" Jul 20 13:55:11.387162 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:11.387134 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82"] Jul 20 13:55:11.389068 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:55:11.389037 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18592fc_4a64_447c_9fc5_658c8b6e38e2.slice/crio-f742da905a510261ac91cc034bedda45bcea2d1eb2e65b94a2ccba782d488e81 WatchSource:0}: Error finding container f742da905a510261ac91cc034bedda45bcea2d1eb2e65b94a2ccba782d488e81: Status 404 returned error can't find the container with id f742da905a510261ac91cc034bedda45bcea2d1eb2e65b94a2ccba782d488e81 Jul 20 13:55:12.394935 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:12.394836 2575 generic.go:356] "Generic (PLEG): container finished" podID="f18592fc-4a64-447c-9fc5-658c8b6e38e2" containerID="4a70e8fa221edf5ad0e73c2fe53626d49f4aa1b8a5e4c26dafa14911f30b7de5" exitCode=0 Jul 20 13:55:12.394935 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:12.394908 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82" event={"ID":"f18592fc-4a64-447c-9fc5-658c8b6e38e2","Type":"ContainerDied","Data":"4a70e8fa221edf5ad0e73c2fe53626d49f4aa1b8a5e4c26dafa14911f30b7de5"} Jul 20 13:55:12.395327 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:12.394956 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82" event={"ID":"f18592fc-4a64-447c-9fc5-658c8b6e38e2","Type":"ContainerStarted","Data":"f742da905a510261ac91cc034bedda45bcea2d1eb2e65b94a2ccba782d488e81"} Jul 20 13:55:13.525264 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:13.525238 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82" Jul 20 13:55:13.638002 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:13.637975 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18592fc-4a64-447c-9fc5-658c8b6e38e2-maas-api-service-ca\") pod \"f18592fc-4a64-447c-9fc5-658c8b6e38e2\" (UID: \"f18592fc-4a64-447c-9fc5-658c8b6e38e2\") " Jul 20 13:55:13.638334 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:13.638311 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18592fc-4a64-447c-9fc5-658c8b6e38e2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f18592fc-4a64-447c-9fc5-658c8b6e38e2" (UID: "f18592fc-4a64-447c-9fc5-658c8b6e38e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:55:13.738922 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:13.738855 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18592fc-4a64-447c-9fc5-658c8b6e38e2-maas-api-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:55:14.404682 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:14.404655 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82" Jul 20 13:55:14.404682 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:14.404667 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82" event={"ID":"f18592fc-4a64-447c-9fc5-658c8b6e38e2","Type":"ContainerDied","Data":"f742da905a510261ac91cc034bedda45bcea2d1eb2e65b94a2ccba782d488e81"} Jul 20 13:55:14.404855 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:14.404708 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f742da905a510261ac91cc034bedda45bcea2d1eb2e65b94a2ccba782d488e81" Jul 20 13:55:25.150393 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:25.150354 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:55:26.450230 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:26.450201 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924"] Jul 20 13:55:26.450654 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:26.450405 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" podUID="0a71d09e-e669-479a-9bdb-a0e61299bc3e" containerName="maas-api" containerID="cri-o://3ddb2e668853f348c8ff3efcdc5913af70f0fd818191c1f6683970ae3b3645a0" gracePeriod=30 Jul 20 13:55:26.697490 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:26.697459 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:55:26.835049 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:26.835018 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz8wt\" (UniqueName: \"kubernetes.io/projected/0a71d09e-e669-479a-9bdb-a0e61299bc3e-kube-api-access-dz8wt\") pod \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\" (UID: \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\") " Jul 20 13:55:26.835212 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:26.835057 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a71d09e-e669-479a-9bdb-a0e61299bc3e-openshift-service-ca\") pod \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\" (UID: \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\") " Jul 20 13:55:26.835212 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:26.835139 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a71d09e-e669-479a-9bdb-a0e61299bc3e-maas-api-tls\") pod \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\" (UID: \"0a71d09e-e669-479a-9bdb-a0e61299bc3e\") " Jul 20 13:55:26.835444 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:26.835420 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a71d09e-e669-479a-9bdb-a0e61299bc3e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a71d09e-e669-479a-9bdb-a0e61299bc3e" (UID: "0a71d09e-e669-479a-9bdb-a0e61299bc3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:55:26.837176 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:26.837154 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a71d09e-e669-479a-9bdb-a0e61299bc3e-kube-api-access-dz8wt" (OuterVolumeSpecName: "kube-api-access-dz8wt") pod "0a71d09e-e669-479a-9bdb-a0e61299bc3e" (UID: "0a71d09e-e669-479a-9bdb-a0e61299bc3e"). InnerVolumeSpecName "kube-api-access-dz8wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:55:26.837280 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:26.837207 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a71d09e-e669-479a-9bdb-a0e61299bc3e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a71d09e-e669-479a-9bdb-a0e61299bc3e" (UID: "0a71d09e-e669-479a-9bdb-a0e61299bc3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:55:26.936625 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:26.936600 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a71d09e-e669-479a-9bdb-a0e61299bc3e-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:55:26.936625 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:26.936625 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz8wt\" (UniqueName: \"kubernetes.io/projected/0a71d09e-e669-479a-9bdb-a0e61299bc3e-kube-api-access-dz8wt\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:55:26.936790 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:26.936634 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a71d09e-e669-479a-9bdb-a0e61299bc3e-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:55:27.467302 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:27.467272 2575 generic.go:356] "Generic (PLEG): container finished" podID="0a71d09e-e669-479a-9bdb-a0e61299bc3e" containerID="3ddb2e668853f348c8ff3efcdc5913af70f0fd818191c1f6683970ae3b3645a0" exitCode=0 Jul 20 13:55:27.467717 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:27.467320 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" Jul 20 13:55:27.467717 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:27.467356 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" event={"ID":"0a71d09e-e669-479a-9bdb-a0e61299bc3e","Type":"ContainerDied","Data":"3ddb2e668853f348c8ff3efcdc5913af70f0fd818191c1f6683970ae3b3645a0"} Jul 20 13:55:27.467717 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:27.467396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924" event={"ID":"0a71d09e-e669-479a-9bdb-a0e61299bc3e","Type":"ContainerDied","Data":"48694a4c268a7e48582f458fff84c1b8e0b10c416e0d905d9da90696333c986c"} Jul 20 13:55:27.467717 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:27.467416 2575 scope.go:117] "RemoveContainer" containerID="3ddb2e668853f348c8ff3efcdc5913af70f0fd818191c1f6683970ae3b3645a0" Jul 20 13:55:27.476524 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:27.476507 2575 scope.go:117] "RemoveContainer" containerID="3ddb2e668853f348c8ff3efcdc5913af70f0fd818191c1f6683970ae3b3645a0" Jul 20 13:55:27.476770 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:55:27.476752 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ddb2e668853f348c8ff3efcdc5913af70f0fd818191c1f6683970ae3b3645a0\": container with ID starting with 3ddb2e668853f348c8ff3efcdc5913af70f0fd818191c1f6683970ae3b3645a0 not found: ID does not exist" containerID="3ddb2e668853f348c8ff3efcdc5913af70f0fd818191c1f6683970ae3b3645a0" Jul 20 13:55:27.476816 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:27.476778 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ddb2e668853f348c8ff3efcdc5913af70f0fd818191c1f6683970ae3b3645a0"} err="failed to get container status \"3ddb2e668853f348c8ff3efcdc5913af70f0fd818191c1f6683970ae3b3645a0\": rpc error: code = NotFound desc = could not find container \"3ddb2e668853f348c8ff3efcdc5913af70f0fd818191c1f6683970ae3b3645a0\": container with ID starting with 3ddb2e668853f348c8ff3efcdc5913af70f0fd818191c1f6683970ae3b3645a0 not found: ID does not exist" Jul 20 13:55:27.487615 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:27.487595 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924"] Jul 20 13:55:27.491808 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:27.491790 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a45cad8-59f54bdffd-ps924"] Jul 20 13:55:28.308630 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:28.308597 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a71d09e-e669-479a-9bdb-a0e61299bc3e" path="/var/lib/kubelet/pods/0a71d09e-e669-479a-9bdb-a0e61299bc3e/volumes" Jul 20 13:55:31.032013 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:31.031980 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82"] Jul 20 13:55:31.033788 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:31.033768 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a45cad8-btq82"] Jul 20 13:55:32.308436 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:32.308399 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f18592fc-4a64-447c-9fc5-658c8b6e38e2" path="/var/lib/kubelet/pods/f18592fc-4a64-447c-9fc5-658c8b6e38e2/volumes" Jul 20 13:55:33.052018 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:33.051978 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:55:46.358416 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:55:46.358349 2575 scope.go:117] "RemoveContainer" containerID="4efe94c2d371ef70a13752cddd94d5150f3265b5b8cc9c07740d5cd651729450" Jul 20 13:56:39.048567 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:56:39.048529 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:56:46.382357 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:56:46.382329 2575 scope.go:117] "RemoveContainer" containerID="9ea5b2c1dac4ff32d342cf97a52d97909bffacdf8e6724c7cf6c2d26a999bd3a" Jul 20 13:56:48.650133 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:56:48.650100 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:57:34.197569 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.197537 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf"] Jul 20 13:57:34.198384 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.197837 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a71d09e-e669-479a-9bdb-a0e61299bc3e" containerName="maas-api" Jul 20 13:57:34.198384 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.197847 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a71d09e-e669-479a-9bdb-a0e61299bc3e" containerName="maas-api" Jul 20 13:57:34.198384 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.197860 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f18592fc-4a64-447c-9fc5-658c8b6e38e2" containerName="revoke-keys" Jul 20 13:57:34.198384 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.197865 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f18592fc-4a64-447c-9fc5-658c8b6e38e2" containerName="revoke-keys" Jul 20 13:57:34.198384 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.197942 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a71d09e-e669-479a-9bdb-a0e61299bc3e" containerName="maas-api" Jul 20 13:57:34.198384 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.197952 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f18592fc-4a64-447c-9fc5-658c8b6e38e2" containerName="revoke-keys" Jul 20 13:57:34.200725 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.200698 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:57:34.202558 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.202538 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-06fdf5-serving-cert\"" Jul 20 13:57:34.219333 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.219311 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf"] Jul 20 13:57:34.262643 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.262620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a59598-ac37-400b-9492-bf431450dd31-openshift-service-ca\") pod \"maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf\" (UID: \"82a59598-ac37-400b-9492-bf431450dd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:57:34.262765 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.262673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl2xs\" (UniqueName: \"kubernetes.io/projected/82a59598-ac37-400b-9492-bf431450dd31-kube-api-access-vl2xs\") pod \"maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf\" (UID: \"82a59598-ac37-400b-9492-bf431450dd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:57:34.262765 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.262741 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a59598-ac37-400b-9492-bf431450dd31-maas-api-tls\") pod \"maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf\" (UID: \"82a59598-ac37-400b-9492-bf431450dd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:57:34.363636 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.363608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl2xs\" (UniqueName: \"kubernetes.io/projected/82a59598-ac37-400b-9492-bf431450dd31-kube-api-access-vl2xs\") pod \"maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf\" (UID: \"82a59598-ac37-400b-9492-bf431450dd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:57:34.363752 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.363651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a59598-ac37-400b-9492-bf431450dd31-maas-api-tls\") pod \"maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf\" (UID: \"82a59598-ac37-400b-9492-bf431450dd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:57:34.363752 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.363673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a59598-ac37-400b-9492-bf431450dd31-openshift-service-ca\") pod \"maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf\" (UID: \"82a59598-ac37-400b-9492-bf431450dd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:57:34.363827 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:57:34.363765 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-serving-cert: secret "maas-api-e2e-inf-a-06fdf5-serving-cert" not found Jul 20 13:57:34.363860 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:57:34.363837 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82a59598-ac37-400b-9492-bf431450dd31-maas-api-tls podName:82a59598-ac37-400b-9492-bf431450dd31 nodeName:}" failed. No retries permitted until 2026-07-20 13:57:34.863818334 +0000 UTC m=+2449.072214460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82a59598-ac37-400b-9492-bf431450dd31-maas-api-tls") pod "maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" (UID: "82a59598-ac37-400b-9492-bf431450dd31") : secret "maas-api-e2e-inf-a-06fdf5-serving-cert" not found Jul 20 13:57:34.364350 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.364333 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a59598-ac37-400b-9492-bf431450dd31-openshift-service-ca\") pod \"maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf\" (UID: \"82a59598-ac37-400b-9492-bf431450dd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:57:34.378765 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.378738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl2xs\" (UniqueName: \"kubernetes.io/projected/82a59598-ac37-400b-9492-bf431450dd31-kube-api-access-vl2xs\") pod \"maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf\" (UID: \"82a59598-ac37-400b-9492-bf431450dd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:57:34.868098 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.868060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a59598-ac37-400b-9492-bf431450dd31-maas-api-tls\") pod \"maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf\" (UID: \"82a59598-ac37-400b-9492-bf431450dd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:57:34.870438 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:34.870414 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a59598-ac37-400b-9492-bf431450dd31-maas-api-tls\") pod \"maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf\" (UID: \"82a59598-ac37-400b-9492-bf431450dd31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:57:35.111743 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:35.111707 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:57:35.263348 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:35.263324 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf"] Jul 20 13:57:35.268812 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:35.268792 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 13:57:36.011948 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:36.011910 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" event={"ID":"82a59598-ac37-400b-9492-bf431450dd31","Type":"ContainerStarted","Data":"a6c1fba704ce8254cc5b71772c7f00ccfe24d5d3383a32016638b892e58c2729"} Jul 20 13:57:36.012114 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:36.011984 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" event={"ID":"82a59598-ac37-400b-9492-bf431450dd31","Type":"ContainerStarted","Data":"43f90a19174cbc17fcf3e545a417ec16487563104d371ccbfb00db3602716703"} Jul 20 13:57:36.012114 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:36.012010 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:57:36.028896 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:36.028837 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" podStartSLOduration=1.454536435 podStartE2EDuration="2.028822135s" totalImagesPullingTime="574.2857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:57:36.02639127 +0000 UTC m=+2450.234787416" watchObservedRunningTime="2026-07-20 13:57:36.028822135 +0000 UTC m=+2450.237218278" Jul 20 13:57:40.252155 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:40.252123 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:57:42.021967 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:42.021940 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:57:42.599564 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:42.599531 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9"] Jul 20 13:57:42.603118 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:42.603095 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 13:57:42.604867 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:42.604848 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1af9e1-serving-cert\"" Jul 20 13:57:42.612726 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:42.612700 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9"] Jul 20 13:57:42.728466 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:42.728437 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkdps\" (UniqueName: \"kubernetes.io/projected/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-kube-api-access-lkdps\") pod \"maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9\" (UID: \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 13:57:42.728581 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:42.728472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-maas-api-tls\") pod \"maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9\" (UID: \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 13:57:42.728581 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:42.728543 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9\" (UID: \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 13:57:42.829151 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:42.829124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9\" (UID: \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 13:57:42.829269 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:42.829173 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkdps\" (UniqueName: \"kubernetes.io/projected/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-kube-api-access-lkdps\") pod \"maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9\" (UID: \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 13:57:42.829309 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:42.829291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-maas-api-tls\") pod \"maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9\" (UID: \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 13:57:42.829434 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:57:42.829420 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-serving-cert: secret "maas-api-e2e-inf-b-1af9e1-serving-cert" not found Jul 20 13:57:42.829494 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:57:42.829485 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-maas-api-tls podName:f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c nodeName:}" failed. No retries permitted until 2026-07-20 13:57:43.329470258 +0000 UTC m=+2457.537866381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-maas-api-tls") pod "maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" (UID: "f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c") : secret "maas-api-e2e-inf-b-1af9e1-serving-cert" not found Jul 20 13:57:42.829734 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:42.829715 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9\" (UID: \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 13:57:42.840406 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:42.840385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkdps\" (UniqueName: \"kubernetes.io/projected/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-kube-api-access-lkdps\") pod \"maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9\" (UID: \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 13:57:43.333891 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:43.333831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-maas-api-tls\") pod \"maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9\" (UID: \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 13:57:43.336257 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:43.336235 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-maas-api-tls\") pod \"maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9\" (UID: \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 13:57:43.513630 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:43.513595 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 13:57:43.645988 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:43.645960 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9"] Jul 20 13:57:43.647746 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:57:43.647720 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31fd8a3_2a9f_4639_baa9_2c1cdc54a49c.slice/crio-9532d1c6f0c3d3949a4559b9957d06f998e3a8b4df9382ba2b04ee41ac07adb9 WatchSource:0}: Error finding container 9532d1c6f0c3d3949a4559b9957d06f998e3a8b4df9382ba2b04ee41ac07adb9: Status 404 returned error can't find the container with id 9532d1c6f0c3d3949a4559b9957d06f998e3a8b4df9382ba2b04ee41ac07adb9 Jul 20 13:57:44.051074 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:44.051049 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" event={"ID":"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c","Type":"ContainerStarted","Data":"9532d1c6f0c3d3949a4559b9957d06f998e3a8b4df9382ba2b04ee41ac07adb9"} Jul 20 13:57:45.057304 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:45.057263 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" event={"ID":"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c","Type":"ContainerStarted","Data":"849afb59252e7a9e71694855023128aa06a2cf4b9f6b7652b632d109e7e76fa2"} Jul 20 13:57:45.057803 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:45.057404 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 13:57:45.077712 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:45.077660 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" podStartSLOduration=2.7108735299999998 podStartE2EDuration="3.077643206s" totalImagesPullingTime="366.769676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 13:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 13:57:45.076582745 +0000 UTC m=+2459.284978890" watchObservedRunningTime="2026-07-20 13:57:45.077643206 +0000 UTC m=+2459.286039351" Jul 20 13:57:46.403972 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:46.403942 2575 scope.go:117] "RemoveContainer" containerID="1b56315bcd95909ae537ae2fbc494469f77e4bc9ddfd97a85cd48e29322fe0e5" Jul 20 13:57:46.411621 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:46.411606 2575 scope.go:117] "RemoveContainer" containerID="eb914b9c6f0f54f8bab0b80e5b3c7b1a7c2b65c11e32d4fd8dd0b8777197d3e7" Jul 20 13:57:51.067850 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:51.067816 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 13:57:54.045941 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:57:54.045902 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:58:04.648630 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:58:04.648593 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:58:40.440440 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:58:40.440367 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:58:54.946932 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:58:54.946891 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:59:34.035896 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:34.035843 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7"] Jul 20 13:59:34.039073 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:34.039052 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7" Jul 20 13:59:34.040746 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:34.040725 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t89s5\"" Jul 20 13:59:34.045952 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:34.045931 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7"] Jul 20 13:59:34.104028 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:34.104003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4011aa-baa6-40fd-8538-230c7f283f8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7\" (UID: \"bd4011aa-baa6-40fd-8538-230c7f283f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7" Jul 20 13:59:34.204925 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:34.204895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4011aa-baa6-40fd-8538-230c7f283f8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7\" (UID: \"bd4011aa-baa6-40fd-8538-230c7f283f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7" Jul 20 13:59:34.205469 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:34.205444 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4011aa-baa6-40fd-8538-230c7f283f8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7\" (UID: \"bd4011aa-baa6-40fd-8538-230c7f283f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7" Jul 20 13:59:34.350258 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:34.350206 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7" Jul 20 13:59:34.475158 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:34.475127 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7"] Jul 20 13:59:34.476341 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:59:34.476314 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4011aa_baa6_40fd_8538_230c7f283f8a.slice/crio-03fb6a8fc89deb3383a1debc0b03ee1108d3f6bf13d9c5ef8ddd5e650245de34 WatchSource:0}: Error finding container 03fb6a8fc89deb3383a1debc0b03ee1108d3f6bf13d9c5ef8ddd5e650245de34: Status 404 returned error can't find the container with id 03fb6a8fc89deb3383a1debc0b03ee1108d3f6bf13d9c5ef8ddd5e650245de34 Jul 20 13:59:34.528073 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:34.528045 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7" event={"ID":"bd4011aa-baa6-40fd-8538-230c7f283f8a","Type":"ContainerStarted","Data":"03fb6a8fc89deb3383a1debc0b03ee1108d3f6bf13d9c5ef8ddd5e650245de34"} Jul 20 13:59:35.533804 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:35.533776 2575 generic.go:356] "Generic (PLEG): container finished" podID="bd4011aa-baa6-40fd-8538-230c7f283f8a" containerID="db2f471a0c606e385a88453a067a925cc5f1688379df6409fad6f6213f89ca15" exitCode=0 Jul 20 13:59:35.534143 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:35.533821 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7" event={"ID":"bd4011aa-baa6-40fd-8538-230c7f283f8a","Type":"ContainerDied","Data":"db2f471a0c606e385a88453a067a925cc5f1688379df6409fad6f6213f89ca15"} Jul 20 13:59:36.663890 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:36.663844 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7" Jul 20 13:59:36.731693 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:36.731663 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4011aa-baa6-40fd-8538-230c7f283f8a-maas-api-service-ca\") pod \"bd4011aa-baa6-40fd-8538-230c7f283f8a\" (UID: \"bd4011aa-baa6-40fd-8538-230c7f283f8a\") " Jul 20 13:59:36.732030 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:36.732011 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd4011aa-baa6-40fd-8538-230c7f283f8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd4011aa-baa6-40fd-8538-230c7f283f8a" (UID: "bd4011aa-baa6-40fd-8538-230c7f283f8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:59:36.833199 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:36.833141 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4011aa-baa6-40fd-8538-230c7f283f8a-maas-api-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:59:37.544247 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:37.544215 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7" event={"ID":"bd4011aa-baa6-40fd-8538-230c7f283f8a","Type":"ContainerDied","Data":"03fb6a8fc89deb3383a1debc0b03ee1108d3f6bf13d9c5ef8ddd5e650245de34"} Jul 20 13:59:37.544422 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:37.544262 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03fb6a8fc89deb3383a1debc0b03ee1108d3f6bf13d9c5ef8ddd5e650245de34" Jul 20 13:59:37.544422 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:37.544222 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7" Jul 20 13:59:38.039651 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:38.039619 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:59:44.692968 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:44.692934 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf"] Jul 20 13:59:44.693361 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:44.693164 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" podUID="82a59598-ac37-400b-9492-bf431450dd31" containerName="maas-api" containerID="cri-o://a6c1fba704ce8254cc5b71772c7f00ccfe24d5d3383a32016638b892e58c2729" gracePeriod=30 Jul 20 13:59:44.938611 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:44.938588 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:59:44.998399 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:44.998270 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a59598-ac37-400b-9492-bf431450dd31-maas-api-tls\") pod \"82a59598-ac37-400b-9492-bf431450dd31\" (UID: \"82a59598-ac37-400b-9492-bf431450dd31\") " Jul 20 13:59:44.998549 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:44.998472 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a59598-ac37-400b-9492-bf431450dd31-openshift-service-ca\") pod \"82a59598-ac37-400b-9492-bf431450dd31\" (UID: \"82a59598-ac37-400b-9492-bf431450dd31\") " Jul 20 13:59:44.998549 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:44.998503 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl2xs\" (UniqueName: \"kubernetes.io/projected/82a59598-ac37-400b-9492-bf431450dd31-kube-api-access-vl2xs\") pod \"82a59598-ac37-400b-9492-bf431450dd31\" (UID: \"82a59598-ac37-400b-9492-bf431450dd31\") " Jul 20 13:59:44.998791 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:44.998771 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a59598-ac37-400b-9492-bf431450dd31-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "82a59598-ac37-400b-9492-bf431450dd31" (UID: "82a59598-ac37-400b-9492-bf431450dd31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:59:45.000206 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:45.000180 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a59598-ac37-400b-9492-bf431450dd31-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "82a59598-ac37-400b-9492-bf431450dd31" (UID: "82a59598-ac37-400b-9492-bf431450dd31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 13:59:45.000430 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:45.000406 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a59598-ac37-400b-9492-bf431450dd31-kube-api-access-vl2xs" (OuterVolumeSpecName: "kube-api-access-vl2xs") pod "82a59598-ac37-400b-9492-bf431450dd31" (UID: "82a59598-ac37-400b-9492-bf431450dd31"). InnerVolumeSpecName "kube-api-access-vl2xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 13:59:45.099842 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:45.099800 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a59598-ac37-400b-9492-bf431450dd31-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:59:45.099842 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:45.099836 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl2xs\" (UniqueName: \"kubernetes.io/projected/82a59598-ac37-400b-9492-bf431450dd31-kube-api-access-vl2xs\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:59:45.099842 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:45.099850 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a59598-ac37-400b-9492-bf431450dd31-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:59:45.575861 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:45.575831 2575 generic.go:356] "Generic (PLEG): container finished" podID="82a59598-ac37-400b-9492-bf431450dd31" containerID="a6c1fba704ce8254cc5b71772c7f00ccfe24d5d3383a32016638b892e58c2729" exitCode=0 Jul 20 13:59:45.576021 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:45.575890 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" Jul 20 13:59:45.576021 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:45.575916 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" event={"ID":"82a59598-ac37-400b-9492-bf431450dd31","Type":"ContainerDied","Data":"a6c1fba704ce8254cc5b71772c7f00ccfe24d5d3383a32016638b892e58c2729"} Jul 20 13:59:45.576021 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:45.575973 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf" event={"ID":"82a59598-ac37-400b-9492-bf431450dd31","Type":"ContainerDied","Data":"43f90a19174cbc17fcf3e545a417ec16487563104d371ccbfb00db3602716703"} Jul 20 13:59:45.576021 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:45.575996 2575 scope.go:117] "RemoveContainer" containerID="a6c1fba704ce8254cc5b71772c7f00ccfe24d5d3383a32016638b892e58c2729" Jul 20 13:59:45.583864 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:45.583847 2575 scope.go:117] "RemoveContainer" containerID="a6c1fba704ce8254cc5b71772c7f00ccfe24d5d3383a32016638b892e58c2729" Jul 20 13:59:45.584155 ip-10-0-132-221 kubenswrapper[2575]: E0720 13:59:45.584137 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c1fba704ce8254cc5b71772c7f00ccfe24d5d3383a32016638b892e58c2729\": container with ID starting with a6c1fba704ce8254cc5b71772c7f00ccfe24d5d3383a32016638b892e58c2729 not found: ID does not exist" containerID="a6c1fba704ce8254cc5b71772c7f00ccfe24d5d3383a32016638b892e58c2729" Jul 20 13:59:45.584226 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:45.584161 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c1fba704ce8254cc5b71772c7f00ccfe24d5d3383a32016638b892e58c2729"} err="failed to get container status \"a6c1fba704ce8254cc5b71772c7f00ccfe24d5d3383a32016638b892e58c2729\": rpc error: code = NotFound desc = could not find container \"a6c1fba704ce8254cc5b71772c7f00ccfe24d5d3383a32016638b892e58c2729\": container with ID starting with a6c1fba704ce8254cc5b71772c7f00ccfe24d5d3383a32016638b892e58c2729 not found: ID does not exist" Jul 20 13:59:45.595654 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:45.595627 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf"] Jul 20 13:59:45.597531 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:45.597512 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06fdf5-59ff86fb8f-cj9pf"] Jul 20 13:59:46.310976 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:46.310943 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82a59598-ac37-400b-9492-bf431450dd31" path="/var/lib/kubelet/pods/82a59598-ac37-400b-9492-bf431450dd31/volumes" Jul 20 13:59:49.145275 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:49.145241 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7"] Jul 20 13:59:49.147097 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:49.147076 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06fdf5-f2wn7"] Jul 20 13:59:50.308771 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:50.308739 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd4011aa-baa6-40fd-8538-230c7f283f8a" path="/var/lib/kubelet/pods/bd4011aa-baa6-40fd-8538-230c7f283f8a/volumes" Jul 20 13:59:53.553929 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:53.553895 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 13:59:56.332850 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.332812 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr"] Jul 20 13:59:56.333280 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.333161 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd4011aa-baa6-40fd-8538-230c7f283f8a" containerName="revoke-keys" Jul 20 13:59:56.333280 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.333176 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd4011aa-baa6-40fd-8538-230c7f283f8a" containerName="revoke-keys" Jul 20 13:59:56.333280 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.333198 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a59598-ac37-400b-9492-bf431450dd31" containerName="maas-api" Jul 20 13:59:56.333280 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.333204 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a59598-ac37-400b-9492-bf431450dd31" containerName="maas-api" Jul 20 13:59:56.333280 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.333252 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a59598-ac37-400b-9492-bf431450dd31" containerName="maas-api" Jul 20 13:59:56.333280 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.333261 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd4011aa-baa6-40fd-8538-230c7f283f8a" containerName="revoke-keys" Jul 20 13:59:56.336088 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.336071 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr" Jul 20 13:59:56.341935 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.341915 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t89s5\"" Jul 20 13:59:56.360831 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.360808 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr"] Jul 20 13:59:56.489432 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.489405 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f31350c-9897-421b-863e-d80a3fb9e16b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr\" (UID: \"5f31350c-9897-421b-863e-d80a3fb9e16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr" Jul 20 13:59:56.590333 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.590251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f31350c-9897-421b-863e-d80a3fb9e16b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr\" (UID: \"5f31350c-9897-421b-863e-d80a3fb9e16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr" Jul 20 13:59:56.590857 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.590836 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f31350c-9897-421b-863e-d80a3fb9e16b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr\" (UID: \"5f31350c-9897-421b-863e-d80a3fb9e16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr" Jul 20 13:59:56.645570 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.645544 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr" Jul 20 13:59:56.772259 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:56.772230 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr"] Jul 20 13:59:56.773496 ip-10-0-132-221 kubenswrapper[2575]: W0720 13:59:56.773468 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f31350c_9897_421b_863e_d80a3fb9e16b.slice/crio-0c15b24d7ab4ee34ddf1abb099b59e85163ae3ffb6aa0af6883eb5a12c2e2d42 WatchSource:0}: Error finding container 0c15b24d7ab4ee34ddf1abb099b59e85163ae3ffb6aa0af6883eb5a12c2e2d42: Status 404 returned error can't find the container with id 0c15b24d7ab4ee34ddf1abb099b59e85163ae3ffb6aa0af6883eb5a12c2e2d42 Jul 20 13:59:57.628438 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:57.628402 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f31350c-9897-421b-863e-d80a3fb9e16b" containerID="de8fd0622762635d5b6913de08f6aedddf66da5670c922bd637c5e3a226b5765" exitCode=0 Jul 20 13:59:57.628833 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:57.628455 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr" event={"ID":"5f31350c-9897-421b-863e-d80a3fb9e16b","Type":"ContainerDied","Data":"de8fd0622762635d5b6913de08f6aedddf66da5670c922bd637c5e3a226b5765"} Jul 20 13:59:57.628833 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:57.628518 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr" event={"ID":"5f31350c-9897-421b-863e-d80a3fb9e16b","Type":"ContainerStarted","Data":"0c15b24d7ab4ee34ddf1abb099b59e85163ae3ffb6aa0af6883eb5a12c2e2d42"} Jul 20 13:59:58.758201 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:58.758175 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr" Jul 20 13:59:58.908158 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:58.908082 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f31350c-9897-421b-863e-d80a3fb9e16b-maas-api-service-ca\") pod \"5f31350c-9897-421b-863e-d80a3fb9e16b\" (UID: \"5f31350c-9897-421b-863e-d80a3fb9e16b\") " Jul 20 13:59:58.908402 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:58.908373 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f31350c-9897-421b-863e-d80a3fb9e16b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f31350c-9897-421b-863e-d80a3fb9e16b" (UID: "5f31350c-9897-421b-863e-d80a3fb9e16b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 13:59:59.009104 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:59.009078 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f31350c-9897-421b-863e-d80a3fb9e16b-maas-api-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 13:59:59.637706 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:59.637667 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr" event={"ID":"5f31350c-9897-421b-863e-d80a3fb9e16b","Type":"ContainerDied","Data":"0c15b24d7ab4ee34ddf1abb099b59e85163ae3ffb6aa0af6883eb5a12c2e2d42"} Jul 20 13:59:59.637894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:59.637725 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c15b24d7ab4ee34ddf1abb099b59e85163ae3ffb6aa0af6883eb5a12c2e2d42" Jul 20 13:59:59.637894 ip-10-0-132-221 kubenswrapper[2575]: I0720 13:59:59.637681 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr" Jul 20 14:00:00.164467 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.164433 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1af9e1-29742600-b48ql"] Jul 20 14:00:00.164844 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.164769 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f31350c-9897-421b-863e-d80a3fb9e16b" containerName="revoke-keys" Jul 20 14:00:00.164844 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.164779 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f31350c-9897-421b-863e-d80a3fb9e16b" containerName="revoke-keys" Jul 20 14:00:00.164844 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.164835 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f31350c-9897-421b-863e-d80a3fb9e16b" containerName="revoke-keys" Jul 20 14:00:00.167805 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.167782 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1af9e1-29742600-b48ql" Jul 20 14:00:00.169000 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.168975 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742600-v8x2v"] Jul 20 14:00:00.171970 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.171953 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742600-v8x2v" Jul 20 14:00:00.174622 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.174602 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t89s5\"" Jul 20 14:00:00.178395 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.178379 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1af9e1-29742600-b48ql" Jul 20 14:00:00.184824 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.184807 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742600-v8x2v" Jul 20 14:00:00.189154 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.189131 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1af9e1-29742600-b48ql"] Jul 20 14:00:00.211316 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.211288 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742600-v8x2v"] Jul 20 14:00:00.331569 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.331540 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1af9e1-29742600-b48ql"] Jul 20 14:00:00.332542 ip-10-0-132-221 kubenswrapper[2575]: W0720 14:00:00.332510 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb07f6e_d5ea_43e8_9290_5ecd9276fc02.slice/crio-c8409666e13d800d52e727f00f1a69d49dee8335b576ff046f8bb2c22e487f41 WatchSource:0}: Error finding container c8409666e13d800d52e727f00f1a69d49dee8335b576ff046f8bb2c22e487f41: Status 404 returned error can't find the container with id c8409666e13d800d52e727f00f1a69d49dee8335b576ff046f8bb2c22e487f41 Jul 20 14:00:00.355319 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.355300 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742600-v8x2v"] Jul 20 14:00:00.363890 ip-10-0-132-221 kubenswrapper[2575]: W0720 14:00:00.363843 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1b31be_8df4_45c9_95c2_bf2b28ca0728.slice/crio-43a86552a0cd3d47354a540c403baf951e1daf1b70dad921df8824faa462ec8c WatchSource:0}: Error finding container 43a86552a0cd3d47354a540c403baf951e1daf1b70dad921df8824faa462ec8c: Status 404 returned error can't find the container with id 43a86552a0cd3d47354a540c403baf951e1daf1b70dad921df8824faa462ec8c Jul 20 14:00:00.643627 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.643598 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1af9e1-29742600-b48ql" event={"ID":"ddb07f6e-d5ea-43e8-9290-5ecd9276fc02","Type":"ContainerStarted","Data":"874613338dce8c2d9ae3a11ea2008a23ee482476c2fbaf70f281585a4f457a1a"} Jul 20 14:00:00.643743 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.643662 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1af9e1-29742600-b48ql" event={"ID":"ddb07f6e-d5ea-43e8-9290-5ecd9276fc02","Type":"ContainerStarted","Data":"c8409666e13d800d52e727f00f1a69d49dee8335b576ff046f8bb2c22e487f41"} Jul 20 14:00:00.644841 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.644820 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742600-v8x2v" event={"ID":"2a1b31be-8df4-45c9-95c2-bf2b28ca0728","Type":"ContainerStarted","Data":"43a86552a0cd3d47354a540c403baf951e1daf1b70dad921df8824faa462ec8c"} Jul 20 14:00:00.676349 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:00.676308 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1af9e1-29742600-b48ql" podStartSLOduration=0.676295969 podStartE2EDuration="676.295969ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:00:00.675479197 +0000 UTC m=+2594.883875341" watchObservedRunningTime="2026-07-20 14:00:00.676295969 +0000 UTC m=+2594.884692113" Jul 20 14:00:01.650802 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:01.650770 2575 generic.go:356] "Generic (PLEG): container finished" podID="ddb07f6e-d5ea-43e8-9290-5ecd9276fc02" containerID="874613338dce8c2d9ae3a11ea2008a23ee482476c2fbaf70f281585a4f457a1a" exitCode=0 Jul 20 14:00:01.651263 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:01.650852 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1af9e1-29742600-b48ql" event={"ID":"ddb07f6e-d5ea-43e8-9290-5ecd9276fc02","Type":"ContainerDied","Data":"874613338dce8c2d9ae3a11ea2008a23ee482476c2fbaf70f281585a4f457a1a"} Jul 20 14:00:01.652315 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:01.652294 2575 generic.go:356] "Generic (PLEG): container finished" podID="2a1b31be-8df4-45c9-95c2-bf2b28ca0728" containerID="dc23c125575365a940cd8f5f0e94dbb1282202e571e1e5f0fd19ed9742ac9927" exitCode=0 Jul 20 14:00:01.652433 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:01.652340 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742600-v8x2v" event={"ID":"2a1b31be-8df4-45c9-95c2-bf2b28ca0728","Type":"ContainerDied","Data":"dc23c125575365a940cd8f5f0e94dbb1282202e571e1e5f0fd19ed9742ac9927"} Jul 20 14:00:02.808771 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:02.808744 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742600-v8x2v" Jul 20 14:00:02.810809 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:02.810793 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1af9e1-29742600-b48ql" Jul 20 14:00:03.663427 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:03.663397 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1af9e1-29742600-b48ql" Jul 20 14:00:03.663586 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:03.663429 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1af9e1-29742600-b48ql" event={"ID":"ddb07f6e-d5ea-43e8-9290-5ecd9276fc02","Type":"ContainerDied","Data":"c8409666e13d800d52e727f00f1a69d49dee8335b576ff046f8bb2c22e487f41"} Jul 20 14:00:03.663586 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:03.663483 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8409666e13d800d52e727f00f1a69d49dee8335b576ff046f8bb2c22e487f41" Jul 20 14:00:03.664677 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:03.664652 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742600-v8x2v" Jul 20 14:00:03.664677 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:03.664659 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742600-v8x2v" event={"ID":"2a1b31be-8df4-45c9-95c2-bf2b28ca0728","Type":"ContainerDied","Data":"43a86552a0cd3d47354a540c403baf951e1daf1b70dad921df8824faa462ec8c"} Jul 20 14:00:03.664890 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:03.664693 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43a86552a0cd3d47354a540c403baf951e1daf1b70dad921df8824faa462ec8c" Jul 20 14:00:06.980615 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:06.980582 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9"] Jul 20 14:00:06.981173 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:06.980795 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" podUID="f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c" containerName="maas-api" containerID="cri-o://849afb59252e7a9e71694855023128aa06a2cf4b9f6b7652b632d109e7e76fa2" gracePeriod=30 Jul 20 14:00:07.020223 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.020197 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1af9e1-29742600-b48ql"] Jul 20 14:00:07.024314 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.024296 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1af9e1-29742600-b48ql"] Jul 20 14:00:07.226390 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.226367 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 14:00:07.374841 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.374812 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkdps\" (UniqueName: \"kubernetes.io/projected/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-kube-api-access-lkdps\") pod \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\" (UID: \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\") " Jul 20 14:00:07.374991 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.374858 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-maas-api-tls\") pod \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\" (UID: \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\") " Jul 20 14:00:07.374991 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.374940 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-openshift-service-ca\") pod \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\" (UID: \"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c\") " Jul 20 14:00:07.375319 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.375284 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c" (UID: "f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:00:07.376772 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.376746 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-kube-api-access-lkdps" (OuterVolumeSpecName: "kube-api-access-lkdps") pod "f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c" (UID: "f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c"). InnerVolumeSpecName "kube-api-access-lkdps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:00:07.377023 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.377003 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c" (UID: "f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 14:00:07.476274 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.476251 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:00:07.476274 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.476272 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkdps\" (UniqueName: \"kubernetes.io/projected/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-kube-api-access-lkdps\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:00:07.476398 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.476282 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:00:07.683852 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.683789 2575 generic.go:356] "Generic (PLEG): container finished" podID="f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c" containerID="849afb59252e7a9e71694855023128aa06a2cf4b9f6b7652b632d109e7e76fa2" exitCode=0 Jul 20 14:00:07.683852 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.683833 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" Jul 20 14:00:07.684008 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.683836 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" event={"ID":"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c","Type":"ContainerDied","Data":"849afb59252e7a9e71694855023128aa06a2cf4b9f6b7652b632d109e7e76fa2"} Jul 20 14:00:07.684008 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.683964 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9" event={"ID":"f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c","Type":"ContainerDied","Data":"9532d1c6f0c3d3949a4559b9957d06f998e3a8b4df9382ba2b04ee41ac07adb9"} Jul 20 14:00:07.684008 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.683991 2575 scope.go:117] "RemoveContainer" containerID="849afb59252e7a9e71694855023128aa06a2cf4b9f6b7652b632d109e7e76fa2" Jul 20 14:00:07.692992 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.692976 2575 scope.go:117] "RemoveContainer" containerID="849afb59252e7a9e71694855023128aa06a2cf4b9f6b7652b632d109e7e76fa2" Jul 20 14:00:07.693243 ip-10-0-132-221 kubenswrapper[2575]: E0720 14:00:07.693226 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"849afb59252e7a9e71694855023128aa06a2cf4b9f6b7652b632d109e7e76fa2\": container with ID starting with 849afb59252e7a9e71694855023128aa06a2cf4b9f6b7652b632d109e7e76fa2 not found: ID does not exist" containerID="849afb59252e7a9e71694855023128aa06a2cf4b9f6b7652b632d109e7e76fa2" Jul 20 14:00:07.693308 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.693256 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849afb59252e7a9e71694855023128aa06a2cf4b9f6b7652b632d109e7e76fa2"} err="failed to get container status \"849afb59252e7a9e71694855023128aa06a2cf4b9f6b7652b632d109e7e76fa2\": rpc error: code = NotFound desc = could not find container \"849afb59252e7a9e71694855023128aa06a2cf4b9f6b7652b632d109e7e76fa2\": container with ID starting with 849afb59252e7a9e71694855023128aa06a2cf4b9f6b7652b632d109e7e76fa2 not found: ID does not exist" Jul 20 14:00:07.707427 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.707402 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9"] Jul 20 14:00:07.710882 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:07.710852 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1af9e1-5cf685c5f4-9rwt9"] Jul 20 14:00:08.308523 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:08.308451 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddb07f6e-d5ea-43e8-9290-5ecd9276fc02" path="/var/lib/kubelet/pods/ddb07f6e-d5ea-43e8-9290-5ecd9276fc02/volumes" Jul 20 14:00:08.308864 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:08.308787 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c" path="/var/lib/kubelet/pods/f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c/volumes" Jul 20 14:00:10.077587 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:10.077554 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:00:11.464548 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:11.464518 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr"] Jul 20 14:00:11.471589 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:11.471565 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1af9e1-fg7kr"] Jul 20 14:00:12.307904 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:12.307856 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f31350c-9897-421b-863e-d80a3fb9e16b" path="/var/lib/kubelet/pods/5f31350c-9897-421b-863e-d80a3fb9e16b/volumes" Jul 20 14:00:22.225998 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.225962 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db"] Jul 20 14:00:22.226540 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.226393 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c" containerName="maas-api" Jul 20 14:00:22.226540 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.226412 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c" containerName="maas-api" Jul 20 14:00:22.226540 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.226429 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddb07f6e-d5ea-43e8-9290-5ecd9276fc02" containerName="cleanup" Jul 20 14:00:22.226540 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.226438 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddb07f6e-d5ea-43e8-9290-5ecd9276fc02" containerName="cleanup" Jul 20 14:00:22.226540 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.226469 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1b31be-8df4-45c9-95c2-bf2b28ca0728" containerName="cleanup" Jul 20 14:00:22.226540 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.226477 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1b31be-8df4-45c9-95c2-bf2b28ca0728" containerName="cleanup" Jul 20 14:00:22.226846 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.226557 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f31fd8a3-2a9f-4639-baa9-2c1cdc54a49c" containerName="maas-api" Jul 20 14:00:22.226846 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.226571 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddb07f6e-d5ea-43e8-9290-5ecd9276fc02" containerName="cleanup" Jul 20 14:00:22.226846 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.226581 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1b31be-8df4-45c9-95c2-bf2b28ca0728" containerName="cleanup" Jul 20 14:00:22.231063 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.231041 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:22.253701 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.253672 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0f1ced-serving-cert\"" Jul 20 14:00:22.254300 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.254277 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db"] Jul 20 14:00:22.287345 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.287306 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76afb3a-95f7-4931-9282-b3d60211158a-openshift-service-ca\") pod \"maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db\" (UID: \"c76afb3a-95f7-4931-9282-b3d60211158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:22.287613 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.287588 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkzv5\" (UniqueName: \"kubernetes.io/projected/c76afb3a-95f7-4931-9282-b3d60211158a-kube-api-access-xkzv5\") pod \"maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db\" (UID: \"c76afb3a-95f7-4931-9282-b3d60211158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:22.287716 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.287622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76afb3a-95f7-4931-9282-b3d60211158a-maas-api-tls\") pod \"maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db\" (UID: \"c76afb3a-95f7-4931-9282-b3d60211158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:22.387964 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.387933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkzv5\" (UniqueName: \"kubernetes.io/projected/c76afb3a-95f7-4931-9282-b3d60211158a-kube-api-access-xkzv5\") pod \"maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db\" (UID: \"c76afb3a-95f7-4931-9282-b3d60211158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:22.387964 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.387966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76afb3a-95f7-4931-9282-b3d60211158a-maas-api-tls\") pod \"maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db\" (UID: \"c76afb3a-95f7-4931-9282-b3d60211158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:22.388160 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.388060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76afb3a-95f7-4931-9282-b3d60211158a-openshift-service-ca\") pod \"maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db\" (UID: \"c76afb3a-95f7-4931-9282-b3d60211158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:22.388160 ip-10-0-132-221 kubenswrapper[2575]: E0720 14:00:22.388105 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-serving-cert: secret "maas-api-e2e-api-a-0f1ced-serving-cert" not found Jul 20 14:00:22.388288 ip-10-0-132-221 kubenswrapper[2575]: E0720 14:00:22.388190 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76afb3a-95f7-4931-9282-b3d60211158a-maas-api-tls podName:c76afb3a-95f7-4931-9282-b3d60211158a nodeName:}" failed. No retries permitted until 2026-07-20 14:00:22.888169951 +0000 UTC m=+2617.096566086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c76afb3a-95f7-4931-9282-b3d60211158a-maas-api-tls") pod "maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" (UID: "c76afb3a-95f7-4931-9282-b3d60211158a") : secret "maas-api-e2e-api-a-0f1ced-serving-cert" not found Jul 20 14:00:22.388714 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.388692 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76afb3a-95f7-4931-9282-b3d60211158a-openshift-service-ca\") pod \"maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db\" (UID: \"c76afb3a-95f7-4931-9282-b3d60211158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:22.411506 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.411478 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkzv5\" (UniqueName: \"kubernetes.io/projected/c76afb3a-95f7-4931-9282-b3d60211158a-kube-api-access-xkzv5\") pod \"maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db\" (UID: \"c76afb3a-95f7-4931-9282-b3d60211158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:22.892316 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.892283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76afb3a-95f7-4931-9282-b3d60211158a-maas-api-tls\") pod \"maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db\" (UID: \"c76afb3a-95f7-4931-9282-b3d60211158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:22.895154 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:22.895131 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76afb3a-95f7-4931-9282-b3d60211158a-maas-api-tls\") pod \"maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db\" (UID: \"c76afb3a-95f7-4931-9282-b3d60211158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:23.141377 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:23.141343 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:23.278172 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:23.276547 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db"] Jul 20 14:00:23.280146 ip-10-0-132-221 kubenswrapper[2575]: W0720 14:00:23.280112 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76afb3a_95f7_4931_9282_b3d60211158a.slice/crio-f4ed8dca2f7992db39e258b370078deab8ea8dc4b98c89c0e3991accbad69391 WatchSource:0}: Error finding container f4ed8dca2f7992db39e258b370078deab8ea8dc4b98c89c0e3991accbad69391: Status 404 returned error can't find the container with id f4ed8dca2f7992db39e258b370078deab8ea8dc4b98c89c0e3991accbad69391 Jul 20 14:00:23.761671 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:23.761646 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" event={"ID":"c76afb3a-95f7-4931-9282-b3d60211158a","Type":"ContainerStarted","Data":"f4ed8dca2f7992db39e258b370078deab8ea8dc4b98c89c0e3991accbad69391"} Jul 20 14:00:24.767792 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:24.767755 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" event={"ID":"c76afb3a-95f7-4931-9282-b3d60211158a","Type":"ContainerStarted","Data":"e81c21998107d101a31a8f1b990d0c36dd5a83d3dde571c31b49b0c02b0b3054"} Jul 20 14:00:24.768211 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:24.767852 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:24.784015 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:24.783938 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" podStartSLOduration=2.306898019 podStartE2EDuration="2.783924171s" totalImagesPullingTime="477.026152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:00:24.783744348 +0000 UTC m=+2618.992140493" watchObservedRunningTime="2026-07-20 14:00:24.783924171 +0000 UTC m=+2618.992320316" Jul 20 14:00:29.557509 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:29.557474 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:00:30.708665 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:30.708631 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs"] Jul 20 14:00:30.711977 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:30.711947 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:00:30.714088 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:30.714066 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ad282e-serving-cert\"" Jul 20 14:00:30.729732 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:30.729709 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs"] Jul 20 14:00:30.756672 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:30.756648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14dac13f-dc7d-4e15-a760-3264b0ccdcee-maas-api-tls\") pod \"maas-api-e2e-api-b-ad282e-675749d564-gf9vs\" (UID: \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:00:30.756799 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:30.756699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwz85\" (UniqueName: \"kubernetes.io/projected/14dac13f-dc7d-4e15-a760-3264b0ccdcee-kube-api-access-wwz85\") pod \"maas-api-e2e-api-b-ad282e-675749d564-gf9vs\" (UID: \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:00:30.756799 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:30.756742 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14dac13f-dc7d-4e15-a760-3264b0ccdcee-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad282e-675749d564-gf9vs\" (UID: \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:00:30.778707 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:30.778679 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:30.858047 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:30.858015 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz85\" (UniqueName: \"kubernetes.io/projected/14dac13f-dc7d-4e15-a760-3264b0ccdcee-kube-api-access-wwz85\") pod \"maas-api-e2e-api-b-ad282e-675749d564-gf9vs\" (UID: \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:00:30.858229 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:30.858057 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14dac13f-dc7d-4e15-a760-3264b0ccdcee-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad282e-675749d564-gf9vs\" (UID: \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:00:30.858229 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:30.858129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14dac13f-dc7d-4e15-a760-3264b0ccdcee-maas-api-tls\") pod \"maas-api-e2e-api-b-ad282e-675749d564-gf9vs\" (UID: \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:00:30.858439 ip-10-0-132-221 kubenswrapper[2575]: E0720 14:00:30.858414 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-serving-cert: secret "maas-api-e2e-api-b-ad282e-serving-cert" not found Jul 20 14:00:30.858586 ip-10-0-132-221 kubenswrapper[2575]: E0720 14:00:30.858491 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14dac13f-dc7d-4e15-a760-3264b0ccdcee-maas-api-tls podName:14dac13f-dc7d-4e15-a760-3264b0ccdcee nodeName:}" failed. No retries permitted until 2026-07-20 14:00:31.358468494 +0000 UTC m=+2625.566864635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/14dac13f-dc7d-4e15-a760-3264b0ccdcee-maas-api-tls") pod "maas-api-e2e-api-b-ad282e-675749d564-gf9vs" (UID: "14dac13f-dc7d-4e15-a760-3264b0ccdcee") : secret "maas-api-e2e-api-b-ad282e-serving-cert" not found Jul 20 14:00:30.858836 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:30.858792 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14dac13f-dc7d-4e15-a760-3264b0ccdcee-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad282e-675749d564-gf9vs\" (UID: \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:00:30.870865 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:30.870840 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwz85\" (UniqueName: \"kubernetes.io/projected/14dac13f-dc7d-4e15-a760-3264b0ccdcee-kube-api-access-wwz85\") pod \"maas-api-e2e-api-b-ad282e-675749d564-gf9vs\" (UID: \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:00:31.362070 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:31.362038 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14dac13f-dc7d-4e15-a760-3264b0ccdcee-maas-api-tls\") pod \"maas-api-e2e-api-b-ad282e-675749d564-gf9vs\" (UID: \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:00:31.364441 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:31.364412 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14dac13f-dc7d-4e15-a760-3264b0ccdcee-maas-api-tls\") pod \"maas-api-e2e-api-b-ad282e-675749d564-gf9vs\" (UID: \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:00:31.623590 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:31.623510 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:00:31.758683 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:31.758661 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs"] Jul 20 14:00:31.760998 ip-10-0-132-221 kubenswrapper[2575]: W0720 14:00:31.760971 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14dac13f_dc7d_4e15_a760_3264b0ccdcee.slice/crio-a6085aecfed4cc518e266de9b37837673d109ca2632730adae91ced93d3e7c61 WatchSource:0}: Error finding container a6085aecfed4cc518e266de9b37837673d109ca2632730adae91ced93d3e7c61: Status 404 returned error can't find the container with id a6085aecfed4cc518e266de9b37837673d109ca2632730adae91ced93d3e7c61 Jul 20 14:00:31.799031 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:31.799002 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" event={"ID":"14dac13f-dc7d-4e15-a760-3264b0ccdcee","Type":"ContainerStarted","Data":"a6085aecfed4cc518e266de9b37837673d109ca2632730adae91ced93d3e7c61"} Jul 20 14:00:32.804513 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:32.804484 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" event={"ID":"14dac13f-dc7d-4e15-a760-3264b0ccdcee","Type":"ContainerStarted","Data":"2403d053c9558befa5d55cf9cab8cbb644545fc72f6f9957c8782d91ab62d8c0"} Jul 20 14:00:32.804927 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:32.804537 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:00:32.849385 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:32.849338 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" podStartSLOduration=2.508576112 podStartE2EDuration="2.84932351s" totalImagesPullingTime="340.747398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:00:32.848190687 +0000 UTC m=+2627.056586831" watchObservedRunningTime="2026-07-20 14:00:32.84932351 +0000 UTC m=+2627.057719654" Jul 20 14:00:38.814339 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:38.814308 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:00:46.448897 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:46.448853 2575 scope.go:117] "RemoveContainer" containerID="c86e6cdbc1f259235c62527419b62dfd28697b815d6f07826d9e8d8696c8eba3" Jul 20 14:00:47.763990 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:47.763952 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:00:58.361550 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.361518 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db"] Jul 20 14:00:58.362024 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.361773 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" podUID="c76afb3a-95f7-4931-9282-b3d60211158a" containerName="maas-api" containerID="cri-o://e81c21998107d101a31a8f1b990d0c36dd5a83d3dde571c31b49b0c02b0b3054" gracePeriod=30 Jul 20 14:00:58.609664 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.609640 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:58.670661 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.670595 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkzv5\" (UniqueName: \"kubernetes.io/projected/c76afb3a-95f7-4931-9282-b3d60211158a-kube-api-access-xkzv5\") pod \"c76afb3a-95f7-4931-9282-b3d60211158a\" (UID: \"c76afb3a-95f7-4931-9282-b3d60211158a\") " Jul 20 14:00:58.670661 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.670624 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76afb3a-95f7-4931-9282-b3d60211158a-maas-api-tls\") pod \"c76afb3a-95f7-4931-9282-b3d60211158a\" (UID: \"c76afb3a-95f7-4931-9282-b3d60211158a\") " Jul 20 14:00:58.670853 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.670722 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76afb3a-95f7-4931-9282-b3d60211158a-openshift-service-ca\") pod \"c76afb3a-95f7-4931-9282-b3d60211158a\" (UID: \"c76afb3a-95f7-4931-9282-b3d60211158a\") " Jul 20 14:00:58.671077 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.671055 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76afb3a-95f7-4931-9282-b3d60211158a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c76afb3a-95f7-4931-9282-b3d60211158a" (UID: "c76afb3a-95f7-4931-9282-b3d60211158a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:00:58.672583 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.672553 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76afb3a-95f7-4931-9282-b3d60211158a-kube-api-access-xkzv5" (OuterVolumeSpecName: "kube-api-access-xkzv5") pod "c76afb3a-95f7-4931-9282-b3d60211158a" (UID: "c76afb3a-95f7-4931-9282-b3d60211158a"). InnerVolumeSpecName "kube-api-access-xkzv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:00:58.672583 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.672571 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c76afb3a-95f7-4931-9282-b3d60211158a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c76afb3a-95f7-4931-9282-b3d60211158a" (UID: "c76afb3a-95f7-4931-9282-b3d60211158a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 14:00:58.771220 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.771197 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76afb3a-95f7-4931-9282-b3d60211158a-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:00:58.771220 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.771217 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkzv5\" (UniqueName: \"kubernetes.io/projected/c76afb3a-95f7-4931-9282-b3d60211158a-kube-api-access-xkzv5\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:00:58.771351 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.771229 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76afb3a-95f7-4931-9282-b3d60211158a-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:00:58.921653 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.921561 2575 generic.go:356] "Generic (PLEG): container finished" podID="c76afb3a-95f7-4931-9282-b3d60211158a" containerID="e81c21998107d101a31a8f1b990d0c36dd5a83d3dde571c31b49b0c02b0b3054" exitCode=0 Jul 20 14:00:58.921653 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.921611 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" Jul 20 14:00:58.921859 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.921707 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" event={"ID":"c76afb3a-95f7-4931-9282-b3d60211158a","Type":"ContainerDied","Data":"e81c21998107d101a31a8f1b990d0c36dd5a83d3dde571c31b49b0c02b0b3054"} Jul 20 14:00:58.921859 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.921765 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db" event={"ID":"c76afb3a-95f7-4931-9282-b3d60211158a","Type":"ContainerDied","Data":"f4ed8dca2f7992db39e258b370078deab8ea8dc4b98c89c0e3991accbad69391"} Jul 20 14:00:58.921859 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.921808 2575 scope.go:117] "RemoveContainer" containerID="e81c21998107d101a31a8f1b990d0c36dd5a83d3dde571c31b49b0c02b0b3054" Jul 20 14:00:58.931107 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.931070 2575 scope.go:117] "RemoveContainer" containerID="e81c21998107d101a31a8f1b990d0c36dd5a83d3dde571c31b49b0c02b0b3054" Jul 20 14:00:58.931370 ip-10-0-132-221 kubenswrapper[2575]: E0720 14:00:58.931347 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e81c21998107d101a31a8f1b990d0c36dd5a83d3dde571c31b49b0c02b0b3054\": container with ID starting with e81c21998107d101a31a8f1b990d0c36dd5a83d3dde571c31b49b0c02b0b3054 not found: ID does not exist" containerID="e81c21998107d101a31a8f1b990d0c36dd5a83d3dde571c31b49b0c02b0b3054" Jul 20 14:00:58.931440 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.931383 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e81c21998107d101a31a8f1b990d0c36dd5a83d3dde571c31b49b0c02b0b3054"} err="failed to get container status \"e81c21998107d101a31a8f1b990d0c36dd5a83d3dde571c31b49b0c02b0b3054\": rpc error: code = NotFound desc = could not find container \"e81c21998107d101a31a8f1b990d0c36dd5a83d3dde571c31b49b0c02b0b3054\": container with ID starting with e81c21998107d101a31a8f1b990d0c36dd5a83d3dde571c31b49b0c02b0b3054 not found: ID does not exist" Jul 20 14:00:58.944340 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.944320 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db"] Jul 20 14:00:58.949540 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:00:58.949517 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0f1ced-5bd4d89cc8-8d5db"] Jul 20 14:01:00.308732 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:00.308699 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c76afb3a-95f7-4931-9282-b3d60211158a" path="/var/lib/kubelet/pods/c76afb3a-95f7-4931-9282-b3d60211158a/volumes" Jul 20 14:01:03.656692 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:03.656651 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:01:10.243147 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:10.243111 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv"] Jul 20 14:01:10.243629 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:10.243606 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c76afb3a-95f7-4931-9282-b3d60211158a" containerName="maas-api" Jul 20 14:01:10.243629 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:10.243625 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c76afb3a-95f7-4931-9282-b3d60211158a" containerName="maas-api" Jul 20 14:01:10.243749 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:10.243720 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c76afb3a-95f7-4931-9282-b3d60211158a" containerName="maas-api" Jul 20 14:01:10.247849 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:10.247827 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv" Jul 20 14:01:10.249757 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:10.249725 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t89s5\"" Jul 20 14:01:10.253774 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:10.253750 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv"] Jul 20 14:01:10.359253 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:10.359223 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea8f218-5999-46b0-98d2-308c52112b23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ad282e-mftlv\" (UID: \"eea8f218-5999-46b0-98d2-308c52112b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv" Jul 20 14:01:10.460255 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:10.460219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea8f218-5999-46b0-98d2-308c52112b23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ad282e-mftlv\" (UID: \"eea8f218-5999-46b0-98d2-308c52112b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv" Jul 20 14:01:10.460850 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:10.460828 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea8f218-5999-46b0-98d2-308c52112b23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ad282e-mftlv\" (UID: \"eea8f218-5999-46b0-98d2-308c52112b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv" Jul 20 14:01:10.560359 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:10.560334 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv" Jul 20 14:01:10.691823 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:10.691798 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv"] Jul 20 14:01:10.694439 ip-10-0-132-221 kubenswrapper[2575]: W0720 14:01:10.694407 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea8f218_5999_46b0_98d2_308c52112b23.slice/crio-456797976dcf22ecdeb9cb6d3bec26c9f8aaf997dcfdefc9412cc8a4d8a0e4df WatchSource:0}: Error finding container 456797976dcf22ecdeb9cb6d3bec26c9f8aaf997dcfdefc9412cc8a4d8a0e4df: Status 404 returned error can't find the container with id 456797976dcf22ecdeb9cb6d3bec26c9f8aaf997dcfdefc9412cc8a4d8a0e4df Jul 20 14:01:10.977653 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:10.977617 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv" event={"ID":"eea8f218-5999-46b0-98d2-308c52112b23","Type":"ContainerStarted","Data":"456797976dcf22ecdeb9cb6d3bec26c9f8aaf997dcfdefc9412cc8a4d8a0e4df"} Jul 20 14:01:11.984042 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:11.984008 2575 generic.go:356] "Generic (PLEG): container finished" podID="eea8f218-5999-46b0-98d2-308c52112b23" containerID="768980f8a2ca2d12e26b14281289267825a489d9e3bb226339ab34f6471e0978" exitCode=0 Jul 20 14:01:11.984427 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:11.984103 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv" event={"ID":"eea8f218-5999-46b0-98d2-308c52112b23","Type":"ContainerDied","Data":"768980f8a2ca2d12e26b14281289267825a489d9e3bb226339ab34f6471e0978"} Jul 20 14:01:13.115184 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:13.115165 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv" Jul 20 14:01:13.184500 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:13.184475 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea8f218-5999-46b0-98d2-308c52112b23-maas-api-service-ca\") pod \"eea8f218-5999-46b0-98d2-308c52112b23\" (UID: \"eea8f218-5999-46b0-98d2-308c52112b23\") " Jul 20 14:01:13.184809 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:13.184786 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea8f218-5999-46b0-98d2-308c52112b23-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eea8f218-5999-46b0-98d2-308c52112b23" (UID: "eea8f218-5999-46b0-98d2-308c52112b23"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:01:13.285863 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:13.285841 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea8f218-5999-46b0-98d2-308c52112b23-maas-api-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:01:13.994321 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:13.994279 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv" Jul 20 14:01:13.994524 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:13.994283 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv" event={"ID":"eea8f218-5999-46b0-98d2-308c52112b23","Type":"ContainerDied","Data":"456797976dcf22ecdeb9cb6d3bec26c9f8aaf997dcfdefc9412cc8a4d8a0e4df"} Jul 20 14:01:13.994524 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:13.994423 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="456797976dcf22ecdeb9cb6d3bec26c9f8aaf997dcfdefc9412cc8a4d8a0e4df" Jul 20 14:01:25.725767 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:25.725731 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs"] Jul 20 14:01:25.726195 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:25.726014 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" podUID="14dac13f-dc7d-4e15-a760-3264b0ccdcee" containerName="maas-api" containerID="cri-o://2403d053c9558befa5d55cf9cab8cbb644545fc72f6f9957c8782d91ab62d8c0" gracePeriod=30 Jul 20 14:01:25.971530 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:25.971507 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:01:26.046946 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.046916 2575 generic.go:356] "Generic (PLEG): container finished" podID="14dac13f-dc7d-4e15-a760-3264b0ccdcee" containerID="2403d053c9558befa5d55cf9cab8cbb644545fc72f6f9957c8782d91ab62d8c0" exitCode=0 Jul 20 14:01:26.047084 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.046963 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" Jul 20 14:01:26.047084 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.047002 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" event={"ID":"14dac13f-dc7d-4e15-a760-3264b0ccdcee","Type":"ContainerDied","Data":"2403d053c9558befa5d55cf9cab8cbb644545fc72f6f9957c8782d91ab62d8c0"} Jul 20 14:01:26.047084 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.047051 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs" event={"ID":"14dac13f-dc7d-4e15-a760-3264b0ccdcee","Type":"ContainerDied","Data":"a6085aecfed4cc518e266de9b37837673d109ca2632730adae91ced93d3e7c61"} Jul 20 14:01:26.047084 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.047072 2575 scope.go:117] "RemoveContainer" containerID="2403d053c9558befa5d55cf9cab8cbb644545fc72f6f9957c8782d91ab62d8c0" Jul 20 14:01:26.055056 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.055040 2575 scope.go:117] "RemoveContainer" containerID="2403d053c9558befa5d55cf9cab8cbb644545fc72f6f9957c8782d91ab62d8c0" Jul 20 14:01:26.055303 ip-10-0-132-221 kubenswrapper[2575]: E0720 14:01:26.055285 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2403d053c9558befa5d55cf9cab8cbb644545fc72f6f9957c8782d91ab62d8c0\": container with ID starting with 2403d053c9558befa5d55cf9cab8cbb644545fc72f6f9957c8782d91ab62d8c0 not found: ID does not exist" containerID="2403d053c9558befa5d55cf9cab8cbb644545fc72f6f9957c8782d91ab62d8c0" Jul 20 14:01:26.055362 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.055311 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2403d053c9558befa5d55cf9cab8cbb644545fc72f6f9957c8782d91ab62d8c0"} err="failed to get container status \"2403d053c9558befa5d55cf9cab8cbb644545fc72f6f9957c8782d91ab62d8c0\": rpc error: code = NotFound desc = could not find container \"2403d053c9558befa5d55cf9cab8cbb644545fc72f6f9957c8782d91ab62d8c0\": container with ID starting with 2403d053c9558befa5d55cf9cab8cbb644545fc72f6f9957c8782d91ab62d8c0 not found: ID does not exist" Jul 20 14:01:26.085995 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.085978 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14dac13f-dc7d-4e15-a760-3264b0ccdcee-maas-api-tls\") pod \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\" (UID: \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\") " Jul 20 14:01:26.086073 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.086011 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwz85\" (UniqueName: \"kubernetes.io/projected/14dac13f-dc7d-4e15-a760-3264b0ccdcee-kube-api-access-wwz85\") pod \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\" (UID: \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\") " Jul 20 14:01:26.086073 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.086063 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14dac13f-dc7d-4e15-a760-3264b0ccdcee-openshift-service-ca\") pod \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\" (UID: \"14dac13f-dc7d-4e15-a760-3264b0ccdcee\") " Jul 20 14:01:26.086419 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.086391 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14dac13f-dc7d-4e15-a760-3264b0ccdcee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "14dac13f-dc7d-4e15-a760-3264b0ccdcee" (UID: "14dac13f-dc7d-4e15-a760-3264b0ccdcee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:01:26.087963 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.087926 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14dac13f-dc7d-4e15-a760-3264b0ccdcee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "14dac13f-dc7d-4e15-a760-3264b0ccdcee" (UID: "14dac13f-dc7d-4e15-a760-3264b0ccdcee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 14:01:26.088047 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.088024 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14dac13f-dc7d-4e15-a760-3264b0ccdcee-kube-api-access-wwz85" (OuterVolumeSpecName: "kube-api-access-wwz85") pod "14dac13f-dc7d-4e15-a760-3264b0ccdcee" (UID: "14dac13f-dc7d-4e15-a760-3264b0ccdcee"). InnerVolumeSpecName "kube-api-access-wwz85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:01:26.186591 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.186571 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14dac13f-dc7d-4e15-a760-3264b0ccdcee-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:01:26.186591 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.186591 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwz85\" (UniqueName: \"kubernetes.io/projected/14dac13f-dc7d-4e15-a760-3264b0ccdcee-kube-api-access-wwz85\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:01:26.186733 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.186600 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14dac13f-dc7d-4e15-a760-3264b0ccdcee-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:01:26.361906 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.361860 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs"] Jul 20 14:01:26.365489 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:26.365461 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad282e-675749d564-gf9vs"] Jul 20 14:01:28.309000 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:28.308968 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14dac13f-dc7d-4e15-a760-3264b0ccdcee" path="/var/lib/kubelet/pods/14dac13f-dc7d-4e15-a760-3264b0ccdcee/volumes" Jul 20 14:01:30.391836 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:30.391806 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv"] Jul 20 14:01:30.397339 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:30.397313 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ad282e-mftlv"] Jul 20 14:01:30.793978 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:30.793948 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:01:32.308170 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:32.308136 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eea8f218-5999-46b0-98d2-308c52112b23" path="/var/lib/kubelet/pods/eea8f218-5999-46b0-98d2-308c52112b23/volumes" Jul 20 14:01:42.475437 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.475368 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd"] Jul 20 14:01:42.475860 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.475661 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14dac13f-dc7d-4e15-a760-3264b0ccdcee" containerName="maas-api" Jul 20 14:01:42.475860 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.475670 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="14dac13f-dc7d-4e15-a760-3264b0ccdcee" containerName="maas-api" Jul 20 14:01:42.475860 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.475692 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea8f218-5999-46b0-98d2-308c52112b23" containerName="revoke-keys" Jul 20 14:01:42.475860 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.475697 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea8f218-5999-46b0-98d2-308c52112b23" containerName="revoke-keys" Jul 20 14:01:42.475860 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.475750 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="14dac13f-dc7d-4e15-a760-3264b0ccdcee" containerName="maas-api" Jul 20 14:01:42.475860 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.475760 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eea8f218-5999-46b0-98d2-308c52112b23" containerName="revoke-keys" Jul 20 14:01:42.480084 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.480061 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:01:42.482164 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.482145 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-001aab-serving-cert\"" Jul 20 14:01:42.498365 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.498340 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd"] Jul 20 14:01:42.514203 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.514176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fb212f-98c2-4b2a-92c0-5fe24e79217c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd\" (UID: \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:01:42.514297 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.514216 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxk7h\" (UniqueName: \"kubernetes.io/projected/13fb212f-98c2-4b2a-92c0-5fe24e79217c-kube-api-access-bxk7h\") pod \"maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd\" (UID: \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:01:42.514297 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.514235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13fb212f-98c2-4b2a-92c0-5fe24e79217c-maas-api-tls\") pod \"maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd\" (UID: \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:01:42.615599 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.615566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fb212f-98c2-4b2a-92c0-5fe24e79217c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd\" (UID: \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:01:42.615771 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.615606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxk7h\" (UniqueName: \"kubernetes.io/projected/13fb212f-98c2-4b2a-92c0-5fe24e79217c-kube-api-access-bxk7h\") pod \"maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd\" (UID: \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:01:42.615771 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.615626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13fb212f-98c2-4b2a-92c0-5fe24e79217c-maas-api-tls\") pod \"maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd\" (UID: \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:01:42.615771 ip-10-0-132-221 kubenswrapper[2575]: E0720 14:01:42.615732 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-serving-cert: secret "maas-api-e2e-shared-a-001aab-serving-cert" not found Jul 20 14:01:42.616011 ip-10-0-132-221 kubenswrapper[2575]: E0720 14:01:42.615801 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13fb212f-98c2-4b2a-92c0-5fe24e79217c-maas-api-tls podName:13fb212f-98c2-4b2a-92c0-5fe24e79217c nodeName:}" failed. No retries permitted until 2026-07-20 14:01:43.115786666 +0000 UTC m=+2697.324182802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13fb212f-98c2-4b2a-92c0-5fe24e79217c-maas-api-tls") pod "maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" (UID: "13fb212f-98c2-4b2a-92c0-5fe24e79217c") : secret "maas-api-e2e-shared-a-001aab-serving-cert" not found Jul 20 14:01:42.616332 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.616303 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fb212f-98c2-4b2a-92c0-5fe24e79217c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd\" (UID: \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:01:42.628244 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:42.628221 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxk7h\" (UniqueName: \"kubernetes.io/projected/13fb212f-98c2-4b2a-92c0-5fe24e79217c-kube-api-access-bxk7h\") pod \"maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd\" (UID: \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:01:43.119464 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:43.119429 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13fb212f-98c2-4b2a-92c0-5fe24e79217c-maas-api-tls\") pod \"maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd\" (UID: \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:01:43.122148 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:43.122115 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13fb212f-98c2-4b2a-92c0-5fe24e79217c-maas-api-tls\") pod \"maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd\" (UID: \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:01:43.394367 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:43.394281 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:01:43.561382 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:43.561358 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd"] Jul 20 14:01:43.563293 ip-10-0-132-221 kubenswrapper[2575]: W0720 14:01:43.563264 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fb212f_98c2_4b2a_92c0_5fe24e79217c.slice/crio-ef27f76c5565840c66b469794b1c11daf5f11520e0d6cde0baaaff6ccadb69ff WatchSource:0}: Error finding container ef27f76c5565840c66b469794b1c11daf5f11520e0d6cde0baaaff6ccadb69ff: Status 404 returned error can't find the container with id ef27f76c5565840c66b469794b1c11daf5f11520e0d6cde0baaaff6ccadb69ff Jul 20 14:01:44.130453 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:44.130423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" event={"ID":"13fb212f-98c2-4b2a-92c0-5fe24e79217c","Type":"ContainerStarted","Data":"167062b840fd30b4fb2ad76fec3505103648493b4f833e0167bfa82fc1e67ddd"} Jul 20 14:01:44.130613 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:44.130472 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" event={"ID":"13fb212f-98c2-4b2a-92c0-5fe24e79217c","Type":"ContainerStarted","Data":"ef27f76c5565840c66b469794b1c11daf5f11520e0d6cde0baaaff6ccadb69ff"} Jul 20 14:01:44.130613 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:44.130550 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:01:44.153290 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:44.153243 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" podStartSLOduration=1.757664626 podStartE2EDuration="2.153224878s" totalImagesPullingTime="395.560252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:01:44.152464838 +0000 UTC m=+2698.360860995" watchObservedRunningTime="2026-07-20 14:01:44.153224878 +0000 UTC m=+2698.361621023" Jul 20 14:01:46.473542 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:46.473511 2575 scope.go:117] "RemoveContainer" containerID="4a70e8fa221edf5ad0e73c2fe53626d49f4aa1b8a5e4c26dafa14911f30b7de5" Jul 20 14:01:48.781687 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:48.781654 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:01:50.140756 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:50.140725 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:01:51.172901 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:51.172856 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd"] Jul 20 14:01:51.177978 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:51.177952 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:01:51.180551 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:51.180525 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-92659e-serving-cert\"" Jul 20 14:01:51.200862 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:51.200830 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd"] Jul 20 14:01:51.287865 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:51.287821 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd\" (UID: \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:01:51.288052 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:51.287937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-maas-api-tls\") pod \"maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd\" (UID: \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:01:51.288052 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:51.287975 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld8cz\" (UniqueName: \"kubernetes.io/projected/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-kube-api-access-ld8cz\") pod \"maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd\" (UID: \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:01:51.388533 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:51.388502 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd\" (UID: \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:01:51.388687 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:51.388557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-maas-api-tls\") pod \"maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd\" (UID: \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:01:51.388687 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:51.388593 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld8cz\" (UniqueName: \"kubernetes.io/projected/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-kube-api-access-ld8cz\") pod \"maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd\" (UID: \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:01:51.388830 ip-10-0-132-221 kubenswrapper[2575]: E0720 14:01:51.388713 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-serving-cert: secret "maas-api-e2e-shared-b-92659e-serving-cert" not found Jul 20 14:01:51.388830 ip-10-0-132-221 kubenswrapper[2575]: E0720 14:01:51.388779 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-maas-api-tls podName:d5e60cb9-42b4-437d-b3e1-ef6dc763fa23 nodeName:}" failed. No retries permitted until 2026-07-20 14:01:51.888764577 +0000 UTC m=+2706.097160701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-maas-api-tls") pod "maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" (UID: "d5e60cb9-42b4-437d-b3e1-ef6dc763fa23") : secret "maas-api-e2e-shared-b-92659e-serving-cert" not found Jul 20 14:01:51.389222 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:51.389200 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd\" (UID: \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:01:51.403825 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:51.403803 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld8cz\" (UniqueName: \"kubernetes.io/projected/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-kube-api-access-ld8cz\") pod \"maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd\" (UID: \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:01:51.891026 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:51.890988 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-maas-api-tls\") pod \"maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd\" (UID: \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:01:51.893332 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:51.893309 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-maas-api-tls\") pod \"maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd\" (UID: \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:01:52.088993 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:52.088958 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:01:52.232138 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:52.232113 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd"] Jul 20 14:01:52.233583 ip-10-0-132-221 kubenswrapper[2575]: W0720 14:01:52.233555 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e60cb9_42b4_437d_b3e1_ef6dc763fa23.slice/crio-6eeb6f114d1e70144f8121090a44fdf6d3b50e4a5207b780d1001ec421e317de WatchSource:0}: Error finding container 6eeb6f114d1e70144f8121090a44fdf6d3b50e4a5207b780d1001ec421e317de: Status 404 returned error can't find the container with id 6eeb6f114d1e70144f8121090a44fdf6d3b50e4a5207b780d1001ec421e317de Jul 20 14:01:53.172883 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:53.172838 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" event={"ID":"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23","Type":"ContainerStarted","Data":"e53549fbaa42c496820b28f0e1bae4b52e6bc4dca5ad4b493b98121fd99d14ce"} Jul 20 14:01:53.173086 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:53.172900 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" event={"ID":"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23","Type":"ContainerStarted","Data":"6eeb6f114d1e70144f8121090a44fdf6d3b50e4a5207b780d1001ec421e317de"} Jul 20 14:01:53.173086 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:53.172969 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:01:53.203162 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:53.203114 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" podStartSLOduration=1.835954976 podStartE2EDuration="2.203098609s" totalImagesPullingTime="367.143633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:01:53.200296587 +0000 UTC m=+2707.408692748" watchObservedRunningTime="2026-07-20 14:01:53.203098609 +0000 UTC m=+2707.411494751" Jul 20 14:01:59.182861 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:01:59.182833 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:02:02.351404 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:02:02.351367 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:02:15.703728 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:02:15.703694 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:02:41.874680 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:02:41.874640 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:02:57.756822 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:02:57.756781 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:03:12.751610 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:03:12.751522 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:03:29.355674 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:03:29.355637 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:03:44.455346 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:03:44.455307 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:03:58.354302 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:03:58.354268 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:04:16.961397 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:16.961358 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:04:36.645848 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:36.645810 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:04:45.857441 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:45.857359 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q"] Jul 20 14:04:45.860728 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:45.860711 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:45.862826 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:45.862801 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-001aab\"/\"kube-root-ca.crt\"" Jul 20 14:04:45.863445 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:45.863425 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-001aab\"/\"sub-test-model-001aab-kserve-self-signed-certs\"" Jul 20 14:04:45.863602 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:45.863530 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-001aab\"/\"default-dockercfg-7cn7l\"" Jul 20 14:04:45.863602 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:45.863531 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-001aab\"/\"openshift-service-ca.crt\"" Jul 20 14:04:45.869484 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:45.869453 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q"] Jul 20 14:04:45.949801 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:45.949768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-dshm\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:45.949801 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:45.949799 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-tmp-dir\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:45.950072 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:45.949830 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-model-cache\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:45.950072 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:45.949962 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k788g\" (UniqueName: \"kubernetes.io/projected/a3f492ce-1345-4cb6-9858-467037017d73-kube-api-access-k788g\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:45.950072 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:45.949994 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-home\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:45.950072 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:45.950013 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f492ce-1345-4cb6-9858-467037017d73-tls-certs\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:46.050975 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.050945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k788g\" (UniqueName: \"kubernetes.io/projected/a3f492ce-1345-4cb6-9858-467037017d73-kube-api-access-k788g\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:46.051167 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.050987 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-home\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:46.051167 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.051009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f492ce-1345-4cb6-9858-467037017d73-tls-certs\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:46.051167 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.051131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-dshm\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:46.051345 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.051172 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-tmp-dir\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:46.051345 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.051221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-model-cache\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:46.051448 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.051395 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-home\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:46.051508 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.051494 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-model-cache\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:46.051581 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.051560 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-tmp-dir\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:46.053313 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.053281 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-dshm\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:46.053568 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.053551 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f492ce-1345-4cb6-9858-467037017d73-tls-certs\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:46.059536 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.059513 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k788g\" (UniqueName: \"kubernetes.io/projected/a3f492ce-1345-4cb6-9858-467037017d73-kube-api-access-k788g\") pod \"sub-test-model-001aab-kserve-66f9655d77-zdt4q\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:46.171827 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.171754 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-001aab\"/\"default-dockercfg-7cn7l\"" Jul 20 14:04:46.180436 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.180415 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:46.315555 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.315527 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q"] Jul 20 14:04:46.316359 ip-10-0-132-221 kubenswrapper[2575]: W0720 14:04:46.316330 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3f492ce_1345_4cb6_9858_467037017d73.slice/crio-7bfc42cb884ae3d79e0642cde3d0a01f742da0e4e53c09167e9d60ebaab716e8 WatchSource:0}: Error finding container 7bfc42cb884ae3d79e0642cde3d0a01f742da0e4e53c09167e9d60ebaab716e8: Status 404 returned error can't find the container with id 7bfc42cb884ae3d79e0642cde3d0a01f742da0e4e53c09167e9d60ebaab716e8 Jul 20 14:04:46.929974 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.929938 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" event={"ID":"a3f492ce-1345-4cb6-9858-467037017d73","Type":"ContainerStarted","Data":"54238ddfd2b6c7eff2b7bf2b15d9a63ddc9cd9fcc2c8e062eb6ea0c95c50ed69"} Jul 20 14:04:46.930418 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.929984 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" event={"ID":"a3f492ce-1345-4cb6-9858-467037017d73","Type":"ContainerStarted","Data":"7bfc42cb884ae3d79e0642cde3d0a01f742da0e4e53c09167e9d60ebaab716e8"} Jul 20 14:04:46.949780 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:46.949726 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" podStartSLOduration=1.949710923 podStartE2EDuration="1.949710923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:04:46.948293992 +0000 UTC m=+2881.156690148" watchObservedRunningTime="2026-07-20 14:04:46.949710923 +0000 UTC m=+2881.158107068" Jul 20 14:04:53.754232 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:53.754194 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:04:56.181389 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:56.181354 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:56.181770 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:56.181411 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:56.193949 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:56.193920 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:04:56.986015 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:04:56.985987 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:05:19.167737 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:05:19.167695 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:05:39.264546 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:05:39.264510 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:05:46.515641 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:05:46.515608 2575 scope.go:117] "RemoveContainer" containerID="db2f471a0c606e385a88453a067a925cc5f1688379df6409fad6f6213f89ca15" Jul 20 14:05:56.128748 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:05:56.128710 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q"] Jul 20 14:06:02.958510 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:02.958474 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:06:11.148740 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.148647 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" podUID="a3f492ce-1345-4cb6-9858-467037017d73" containerName="main" containerID="cri-o://54238ddfd2b6c7eff2b7bf2b15d9a63ddc9cd9fcc2c8e062eb6ea0c95c50ed69" gracePeriod=45 Jul 20 14:06:11.302310 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.302281 2575 generic.go:356] "Generic (PLEG): container finished" podID="a3f492ce-1345-4cb6-9858-467037017d73" containerID="54238ddfd2b6c7eff2b7bf2b15d9a63ddc9cd9fcc2c8e062eb6ea0c95c50ed69" exitCode=0 Jul 20 14:06:11.302454 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.302334 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" event={"ID":"a3f492ce-1345-4cb6-9858-467037017d73","Type":"ContainerDied","Data":"54238ddfd2b6c7eff2b7bf2b15d9a63ddc9cd9fcc2c8e062eb6ea0c95c50ed69"} Jul 20 14:06:11.389254 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.389232 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:06:11.423415 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.423341 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-home\") pod \"a3f492ce-1345-4cb6-9858-467037017d73\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " Jul 20 14:06:11.423415 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.423386 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k788g\" (UniqueName: \"kubernetes.io/projected/a3f492ce-1345-4cb6-9858-467037017d73-kube-api-access-k788g\") pod \"a3f492ce-1345-4cb6-9858-467037017d73\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " Jul 20 14:06:11.423575 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.423439 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-dshm\") pod \"a3f492ce-1345-4cb6-9858-467037017d73\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " Jul 20 14:06:11.423575 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.423467 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-model-cache\") pod \"a3f492ce-1345-4cb6-9858-467037017d73\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " Jul 20 14:06:11.423690 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.423615 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-home" (OuterVolumeSpecName: "home") pod "a3f492ce-1345-4cb6-9858-467037017d73" (UID: "a3f492ce-1345-4cb6-9858-467037017d73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:06:11.423740 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.423708 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3f492ce-1345-4cb6-9858-467037017d73" (UID: "a3f492ce-1345-4cb6-9858-467037017d73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:06:11.423796 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.423744 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f492ce-1345-4cb6-9858-467037017d73-tls-certs\") pod \"a3f492ce-1345-4cb6-9858-467037017d73\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " Jul 20 14:06:11.423796 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.423788 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-tmp-dir\") pod \"a3f492ce-1345-4cb6-9858-467037017d73\" (UID: \"a3f492ce-1345-4cb6-9858-467037017d73\") " Jul 20 14:06:11.424101 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.424068 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-home\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:06:11.424101 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.424090 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-model-cache\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:06:11.424242 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.424098 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3f492ce-1345-4cb6-9858-467037017d73" (UID: "a3f492ce-1345-4cb6-9858-467037017d73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:06:11.426033 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.426011 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f492ce-1345-4cb6-9858-467037017d73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3f492ce-1345-4cb6-9858-467037017d73" (UID: "a3f492ce-1345-4cb6-9858-467037017d73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 14:06:11.426116 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.426041 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f492ce-1345-4cb6-9858-467037017d73-kube-api-access-k788g" (OuterVolumeSpecName: "kube-api-access-k788g") pod "a3f492ce-1345-4cb6-9858-467037017d73" (UID: "a3f492ce-1345-4cb6-9858-467037017d73"). InnerVolumeSpecName "kube-api-access-k788g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:06:11.426177 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.426159 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-dshm" (OuterVolumeSpecName: "dshm") pod "a3f492ce-1345-4cb6-9858-467037017d73" (UID: "a3f492ce-1345-4cb6-9858-467037017d73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 14:06:11.525248 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.525225 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k788g\" (UniqueName: \"kubernetes.io/projected/a3f492ce-1345-4cb6-9858-467037017d73-kube-api-access-k788g\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:06:11.525248 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.525246 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-dshm\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:06:11.525382 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.525256 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f492ce-1345-4cb6-9858-467037017d73-tls-certs\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:06:11.525382 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:11.525266 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3f492ce-1345-4cb6-9858-467037017d73-tmp-dir\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:06:12.308454 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:12.308424 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" Jul 20 14:06:12.308937 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:12.308514 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q" event={"ID":"a3f492ce-1345-4cb6-9858-467037017d73","Type":"ContainerDied","Data":"7bfc42cb884ae3d79e0642cde3d0a01f742da0e4e53c09167e9d60ebaab716e8"} Jul 20 14:06:12.308937 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:12.308548 2575 scope.go:117] "RemoveContainer" containerID="54238ddfd2b6c7eff2b7bf2b15d9a63ddc9cd9fcc2c8e062eb6ea0c95c50ed69" Jul 20 14:06:12.334344 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:12.334321 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q"] Jul 20 14:06:12.336178 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:12.336162 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-001aab/sub-test-model-001aab-kserve-66f9655d77-zdt4q"] Jul 20 14:06:14.309335 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:14.309306 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3f492ce-1345-4cb6-9858-467037017d73" path="/var/lib/kubelet/pods/a3f492ce-1345-4cb6-9858-467037017d73/volumes" Jul 20 14:06:24.253220 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:24.253185 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:06:38.448116 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:38.448081 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:06:46.533266 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:46.533239 2575 scope.go:117] "RemoveContainer" containerID="874613338dce8c2d9ae3a11ea2008a23ee482476c2fbaf70f281585a4f457a1a" Jul 20 14:06:46.540239 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:46.540218 2575 scope.go:117] "RemoveContainer" containerID="de8fd0622762635d5b6913de08f6aedddf66da5670c922bd637c5e3a226b5765" Jul 20 14:06:53.155214 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:06:53.155173 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:07:08.662560 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:08.662523 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:07:20.379955 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:20.379924 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf"] Jul 20 14:07:20.380394 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:20.380250 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f492ce-1345-4cb6-9858-467037017d73" containerName="main" Jul 20 14:07:20.380394 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:20.380262 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f492ce-1345-4cb6-9858-467037017d73" containerName="main" Jul 20 14:07:20.380394 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:20.380317 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f492ce-1345-4cb6-9858-467037017d73" containerName="main" Jul 20 14:07:20.383195 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:20.383178 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf" Jul 20 14:07:20.385085 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:20.385066 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t89s5\"" Jul 20 14:07:20.391683 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:20.391661 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf"] Jul 20 14:07:20.454175 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:20.454149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf\" (UID: \"e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf" Jul 20 14:07:20.554923 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:20.554866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf\" (UID: \"e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf" Jul 20 14:07:20.555509 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:20.555490 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf\" (UID: \"e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf" Jul 20 14:07:20.695475 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:20.695401 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf" Jul 20 14:07:20.822978 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:20.822901 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf"] Jul 20 14:07:20.825297 ip-10-0-132-221 kubenswrapper[2575]: W0720 14:07:20.825264 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c59a92_7f56_4a34_aa9e_fc3a49b74ed8.slice/crio-e7fac2f97a38b50eff1f81492e33dc7735f0eae9936c56edae309fb37c5cd2cf WatchSource:0}: Error finding container e7fac2f97a38b50eff1f81492e33dc7735f0eae9936c56edae309fb37c5cd2cf: Status 404 returned error can't find the container with id e7fac2f97a38b50eff1f81492e33dc7735f0eae9936c56edae309fb37c5cd2cf Jul 20 14:07:21.598333 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:21.598297 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf" event={"ID":"e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8","Type":"ContainerStarted","Data":"e3c997061bb0273037462a47f1c238c9b149c1f9eb248d9f82ab44453f065736"} Jul 20 14:07:21.598688 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:21.598351 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf" event={"ID":"e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8","Type":"ContainerStarted","Data":"e7fac2f97a38b50eff1f81492e33dc7735f0eae9936c56edae309fb37c5cd2cf"} Jul 20 14:07:21.614362 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:21.614319 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf" podStartSLOduration=1.614305641 podStartE2EDuration="1.614305641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 14:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:07:21.613981017 +0000 UTC m=+3035.822377163" watchObservedRunningTime="2026-07-20 14:07:21.614305641 +0000 UTC m=+3035.822701786" Jul 20 14:07:22.604365 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:22.604326 2575 generic.go:356] "Generic (PLEG): container finished" podID="e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8" containerID="e3c997061bb0273037462a47f1c238c9b149c1f9eb248d9f82ab44453f065736" exitCode=0 Jul 20 14:07:22.604786 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:22.604365 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf" event={"ID":"e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8","Type":"ContainerDied","Data":"e3c997061bb0273037462a47f1c238c9b149c1f9eb248d9f82ab44453f065736"} Jul 20 14:07:23.738198 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:23.738173 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf" Jul 20 14:07:23.778850 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:23.778817 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8-maas-api-service-ca\") pod \"e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8\" (UID: \"e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8\") " Jul 20 14:07:23.779160 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:23.779137 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8" (UID: "e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:07:23.880001 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:23.879926 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8-maas-api-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:07:24.614978 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:24.614888 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf" event={"ID":"e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8","Type":"ContainerDied","Data":"e7fac2f97a38b50eff1f81492e33dc7735f0eae9936c56edae309fb37c5cd2cf"} Jul 20 14:07:24.614978 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:24.614934 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf" Jul 20 14:07:24.614978 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:24.614947 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7fac2f97a38b50eff1f81492e33dc7735f0eae9936c56edae309fb37c5cd2cf" Jul 20 14:07:30.987063 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:30.987032 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd"] Jul 20 14:07:30.987522 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:30.987314 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" podUID="13fb212f-98c2-4b2a-92c0-5fe24e79217c" containerName="maas-api" containerID="cri-o://167062b840fd30b4fb2ad76fec3505103648493b4f833e0167bfa82fc1e67ddd" gracePeriod=30 Jul 20 14:07:31.238049 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.237994 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:07:31.342301 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.342269 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxk7h\" (UniqueName: \"kubernetes.io/projected/13fb212f-98c2-4b2a-92c0-5fe24e79217c-kube-api-access-bxk7h\") pod \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\" (UID: \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\") " Jul 20 14:07:31.342440 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.342319 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fb212f-98c2-4b2a-92c0-5fe24e79217c-openshift-service-ca\") pod \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\" (UID: \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\") " Jul 20 14:07:31.342440 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.342345 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13fb212f-98c2-4b2a-92c0-5fe24e79217c-maas-api-tls\") pod \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\" (UID: \"13fb212f-98c2-4b2a-92c0-5fe24e79217c\") " Jul 20 14:07:31.342685 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.342663 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fb212f-98c2-4b2a-92c0-5fe24e79217c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "13fb212f-98c2-4b2a-92c0-5fe24e79217c" (UID: "13fb212f-98c2-4b2a-92c0-5fe24e79217c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:07:31.344374 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.344352 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13fb212f-98c2-4b2a-92c0-5fe24e79217c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "13fb212f-98c2-4b2a-92c0-5fe24e79217c" (UID: "13fb212f-98c2-4b2a-92c0-5fe24e79217c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 14:07:31.344476 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.344390 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13fb212f-98c2-4b2a-92c0-5fe24e79217c-kube-api-access-bxk7h" (OuterVolumeSpecName: "kube-api-access-bxk7h") pod "13fb212f-98c2-4b2a-92c0-5fe24e79217c" (UID: "13fb212f-98c2-4b2a-92c0-5fe24e79217c"). InnerVolumeSpecName "kube-api-access-bxk7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:07:31.443354 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.443328 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxk7h\" (UniqueName: \"kubernetes.io/projected/13fb212f-98c2-4b2a-92c0-5fe24e79217c-kube-api-access-bxk7h\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:07:31.443354 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.443350 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fb212f-98c2-4b2a-92c0-5fe24e79217c-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:07:31.443513 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.443361 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13fb212f-98c2-4b2a-92c0-5fe24e79217c-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:07:31.645409 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.645379 2575 generic.go:356] "Generic (PLEG): container finished" podID="13fb212f-98c2-4b2a-92c0-5fe24e79217c" containerID="167062b840fd30b4fb2ad76fec3505103648493b4f833e0167bfa82fc1e67ddd" exitCode=0 Jul 20 14:07:31.645551 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.645417 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" event={"ID":"13fb212f-98c2-4b2a-92c0-5fe24e79217c","Type":"ContainerDied","Data":"167062b840fd30b4fb2ad76fec3505103648493b4f833e0167bfa82fc1e67ddd"} Jul 20 14:07:31.645551 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.645429 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" Jul 20 14:07:31.645551 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.645478 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd" event={"ID":"13fb212f-98c2-4b2a-92c0-5fe24e79217c","Type":"ContainerDied","Data":"ef27f76c5565840c66b469794b1c11daf5f11520e0d6cde0baaaff6ccadb69ff"} Jul 20 14:07:31.645551 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.645506 2575 scope.go:117] "RemoveContainer" containerID="167062b840fd30b4fb2ad76fec3505103648493b4f833e0167bfa82fc1e67ddd" Jul 20 14:07:31.654586 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.654563 2575 scope.go:117] "RemoveContainer" containerID="167062b840fd30b4fb2ad76fec3505103648493b4f833e0167bfa82fc1e67ddd" Jul 20 14:07:31.654837 ip-10-0-132-221 kubenswrapper[2575]: E0720 14:07:31.654817 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"167062b840fd30b4fb2ad76fec3505103648493b4f833e0167bfa82fc1e67ddd\": container with ID starting with 167062b840fd30b4fb2ad76fec3505103648493b4f833e0167bfa82fc1e67ddd not found: ID does not exist" containerID="167062b840fd30b4fb2ad76fec3505103648493b4f833e0167bfa82fc1e67ddd" Jul 20 14:07:31.654902 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.654847 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"167062b840fd30b4fb2ad76fec3505103648493b4f833e0167bfa82fc1e67ddd"} err="failed to get container status \"167062b840fd30b4fb2ad76fec3505103648493b4f833e0167bfa82fc1e67ddd\": rpc error: code = NotFound desc = could not find container \"167062b840fd30b4fb2ad76fec3505103648493b4f833e0167bfa82fc1e67ddd\": container with ID starting with 167062b840fd30b4fb2ad76fec3505103648493b4f833e0167bfa82fc1e67ddd not found: ID does not exist" Jul 20 14:07:31.682403 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.682357 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd"] Jul 20 14:07:31.684438 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:31.684420 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-001aab-6b78ff785f-tswwd"] Jul 20 14:07:32.308986 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:32.308949 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13fb212f-98c2-4b2a-92c0-5fe24e79217c" path="/var/lib/kubelet/pods/13fb212f-98c2-4b2a-92c0-5fe24e79217c/volumes" Jul 20 14:07:34.048180 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:34.048143 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:07:35.475854 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:35.475824 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf"] Jul 20 14:07:35.481991 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:35.481968 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-001aab-zcbxf"] Jul 20 14:07:36.308608 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:36.308573 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8" path="/var/lib/kubelet/pods/e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8/volumes" Jul 20 14:07:43.241459 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.241366 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn"] Jul 20 14:07:43.241811 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.241704 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13fb212f-98c2-4b2a-92c0-5fe24e79217c" containerName="maas-api" Jul 20 14:07:43.241811 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.241716 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="13fb212f-98c2-4b2a-92c0-5fe24e79217c" containerName="maas-api" Jul 20 14:07:43.241811 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.241731 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8" containerName="revoke-keys" Jul 20 14:07:43.241811 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.241735 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8" containerName="revoke-keys" Jul 20 14:07:43.241811 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.241784 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="13fb212f-98c2-4b2a-92c0-5fe24e79217c" containerName="maas-api" Jul 20 14:07:43.241811 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.241794 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c59a92-7f56-4a34-aa9e-fc3a49b74ed8" containerName="revoke-keys" Jul 20 14:07:43.245965 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.245949 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn" Jul 20 14:07:43.248489 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.248460 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t89s5\"" Jul 20 14:07:43.323220 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.323190 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn"] Jul 20 14:07:43.333932 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.333902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e0691-0e4c-44a0-ab30-3f0179048b42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn\" (UID: \"2c2e0691-0e4c-44a0-ab30-3f0179048b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn" Jul 20 14:07:43.434485 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.434457 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e0691-0e4c-44a0-ab30-3f0179048b42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn\" (UID: \"2c2e0691-0e4c-44a0-ab30-3f0179048b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn" Jul 20 14:07:43.435093 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.435064 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e0691-0e4c-44a0-ab30-3f0179048b42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn\" (UID: \"2c2e0691-0e4c-44a0-ab30-3f0179048b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn" Jul 20 14:07:43.555743 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.555715 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn" Jul 20 14:07:43.691239 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.691215 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn"] Jul 20 14:07:43.692820 ip-10-0-132-221 kubenswrapper[2575]: W0720 14:07:43.692792 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2e0691_0e4c_44a0_ab30_3f0179048b42.slice/crio-acf1e0bab0555e31c6f2b5f147f35a4db968716361175406c642cbafb613e8f4 WatchSource:0}: Error finding container acf1e0bab0555e31c6f2b5f147f35a4db968716361175406c642cbafb613e8f4: Status 404 returned error can't find the container with id acf1e0bab0555e31c6f2b5f147f35a4db968716361175406c642cbafb613e8f4 Jul 20 14:07:43.705919 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:43.705891 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn" event={"ID":"2c2e0691-0e4c-44a0-ab30-3f0179048b42","Type":"ContainerStarted","Data":"acf1e0bab0555e31c6f2b5f147f35a4db968716361175406c642cbafb613e8f4"} Jul 20 14:07:44.711661 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:44.711576 2575 generic.go:356] "Generic (PLEG): container finished" podID="2c2e0691-0e4c-44a0-ab30-3f0179048b42" containerID="983ba8ac009cd78f5057343404d83abdcf8e5c58548d5bbfe6e88b9184579246" exitCode=0 Jul 20 14:07:44.711661 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:44.711614 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn" event={"ID":"2c2e0691-0e4c-44a0-ab30-3f0179048b42","Type":"ContainerDied","Data":"983ba8ac009cd78f5057343404d83abdcf8e5c58548d5bbfe6e88b9184579246"} Jul 20 14:07:45.841256 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:45.841230 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn" Jul 20 14:07:45.955368 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:45.955339 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e0691-0e4c-44a0-ab30-3f0179048b42-maas-api-service-ca\") pod \"2c2e0691-0e4c-44a0-ab30-3f0179048b42\" (UID: \"2c2e0691-0e4c-44a0-ab30-3f0179048b42\") " Jul 20 14:07:45.955711 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:45.955693 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2e0691-0e4c-44a0-ab30-3f0179048b42-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c2e0691-0e4c-44a0-ab30-3f0179048b42" (UID: "2c2e0691-0e4c-44a0-ab30-3f0179048b42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:07:46.056571 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:46.056547 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e0691-0e4c-44a0-ab30-3f0179048b42-maas-api-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:07:46.562484 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:46.562454 2575 scope.go:117] "RemoveContainer" containerID="768980f8a2ca2d12e26b14281289267825a489d9e3bb226339ab34f6471e0978" Jul 20 14:07:46.720993 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:46.720970 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn" Jul 20 14:07:46.721098 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:46.721014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn" event={"ID":"2c2e0691-0e4c-44a0-ab30-3f0179048b42","Type":"ContainerDied","Data":"acf1e0bab0555e31c6f2b5f147f35a4db968716361175406c642cbafb613e8f4"} Jul 20 14:07:46.721098 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:46.721083 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acf1e0bab0555e31c6f2b5f147f35a4db968716361175406c642cbafb613e8f4" Jul 20 14:07:53.740466 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:53.740424 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd"] Jul 20 14:07:53.741086 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:53.741052 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" podUID="d5e60cb9-42b4-437d-b3e1-ef6dc763fa23" containerName="maas-api" containerID="cri-o://e53549fbaa42c496820b28f0e1bae4b52e6bc4dca5ad4b493b98121fd99d14ce" gracePeriod=30 Jul 20 14:07:53.981767 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:53.981743 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:07:54.114724 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.114690 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ld8cz\" (UniqueName: \"kubernetes.io/projected/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-kube-api-access-ld8cz\") pod \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\" (UID: \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\") " Jul 20 14:07:54.114961 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.114751 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-maas-api-tls\") pod \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\" (UID: \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\") " Jul 20 14:07:54.114961 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.114786 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-openshift-service-ca\") pod \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\" (UID: \"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23\") " Jul 20 14:07:54.115218 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.115191 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5e60cb9-42b4-437d-b3e1-ef6dc763fa23" (UID: "d5e60cb9-42b4-437d-b3e1-ef6dc763fa23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 14:07:54.116949 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.116922 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5e60cb9-42b4-437d-b3e1-ef6dc763fa23" (UID: "d5e60cb9-42b4-437d-b3e1-ef6dc763fa23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 14:07:54.117054 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.116996 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-kube-api-access-ld8cz" (OuterVolumeSpecName: "kube-api-access-ld8cz") pod "d5e60cb9-42b4-437d-b3e1-ef6dc763fa23" (UID: "d5e60cb9-42b4-437d-b3e1-ef6dc763fa23"). InnerVolumeSpecName "kube-api-access-ld8cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 14:07:54.215730 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.215687 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:07:54.215730 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.215721 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld8cz\" (UniqueName: \"kubernetes.io/projected/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-kube-api-access-ld8cz\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:07:54.215730 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.215732 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 20 14:07:54.754252 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.754221 2575 generic.go:356] "Generic (PLEG): container finished" podID="d5e60cb9-42b4-437d-b3e1-ef6dc763fa23" containerID="e53549fbaa42c496820b28f0e1bae4b52e6bc4dca5ad4b493b98121fd99d14ce" exitCode=0 Jul 20 14:07:54.754712 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.754263 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" Jul 20 14:07:54.754712 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.754273 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" event={"ID":"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23","Type":"ContainerDied","Data":"e53549fbaa42c496820b28f0e1bae4b52e6bc4dca5ad4b493b98121fd99d14ce"} Jul 20 14:07:54.754712 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.754314 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd" event={"ID":"d5e60cb9-42b4-437d-b3e1-ef6dc763fa23","Type":"ContainerDied","Data":"6eeb6f114d1e70144f8121090a44fdf6d3b50e4a5207b780d1001ec421e317de"} Jul 20 14:07:54.754712 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.754334 2575 scope.go:117] "RemoveContainer" containerID="e53549fbaa42c496820b28f0e1bae4b52e6bc4dca5ad4b493b98121fd99d14ce" Jul 20 14:07:54.762451 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.762434 2575 scope.go:117] "RemoveContainer" containerID="e53549fbaa42c496820b28f0e1bae4b52e6bc4dca5ad4b493b98121fd99d14ce" Jul 20 14:07:54.762705 ip-10-0-132-221 kubenswrapper[2575]: E0720 14:07:54.762686 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e53549fbaa42c496820b28f0e1bae4b52e6bc4dca5ad4b493b98121fd99d14ce\": container with ID starting with e53549fbaa42c496820b28f0e1bae4b52e6bc4dca5ad4b493b98121fd99d14ce not found: ID does not exist" containerID="e53549fbaa42c496820b28f0e1bae4b52e6bc4dca5ad4b493b98121fd99d14ce" Jul 20 14:07:54.762772 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.762718 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e53549fbaa42c496820b28f0e1bae4b52e6bc4dca5ad4b493b98121fd99d14ce"} err="failed to get container status \"e53549fbaa42c496820b28f0e1bae4b52e6bc4dca5ad4b493b98121fd99d14ce\": rpc error: code = NotFound desc = could not find container \"e53549fbaa42c496820b28f0e1bae4b52e6bc4dca5ad4b493b98121fd99d14ce\": container with ID starting with e53549fbaa42c496820b28f0e1bae4b52e6bc4dca5ad4b493b98121fd99d14ce not found: ID does not exist" Jul 20 14:07:54.771173 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.771150 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd"] Jul 20 14:07:54.775414 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:54.775394 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-92659e-7b8fcf75b9-z5mnd"] Jul 20 14:07:56.307885 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:56.307851 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5e60cb9-42b4-437d-b3e1-ef6dc763fa23" path="/var/lib/kubelet/pods/d5e60cb9-42b4-437d-b3e1-ef6dc763fa23/volumes" Jul 20 14:07:56.751295 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:56.751215 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bppwn"] Jul 20 14:07:58.308545 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:58.308516 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn"] Jul 20 14:07:58.308545 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:07:58.308549 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-92659e-lgnwn"] Jul 20 14:08:00.308674 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:00.308637 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c2e0691-0e4c-44a0-ab30-3f0179048b42" path="/var/lib/kubelet/pods/2c2e0691-0e4c-44a0-ab30-3f0179048b42/volumes" Jul 20 14:08:12.429735 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:12.429704 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f55c6d57f-s8q2w_0383138e-0b70-4738-b321-b984e3da082d/manager/0.log" Jul 20 14:08:13.052783 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:13.052745 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-864d7fc49d-zw5kr_28497bde-8176-4794-a41c-edd6cee61231/manager/0.log" Jul 20 14:08:13.071611 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:13.071587 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-864d7fc49d-zw5kr_28497bde-8176-4794-a41c-edd6cee61231/copy-manifests/0.log" Jul 20 14:08:13.419728 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:13.419659 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-66799fd458-skgmc_711464d6-27b3-4e26-8012-5698e594ed4f/maas-api/0.log" Jul 20 14:08:13.542371 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:13.542340 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29742585-gb5zh_cc3c87c7-b883-4bfe-bfb7-54caf151341c/cleanup/0.log" Jul 20 14:08:13.657821 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:13.657793 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29742600-v8x2v_2a1b31be-8df4-45c9-95c2-bf2b28ca0728/cleanup/0.log" Jul 20 14:08:15.915917 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:15.915863 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-bppwn_e018dae8-c0db-4060-9f8d-c7d3e0f35223/limitador/0.log" Jul 20 14:08:16.528698 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:16.528666 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-rmv5s_b79abb9c-cf1a-413e-90c8-bcd61c9482cc/discovery/0.log" Jul 20 14:08:17.663780 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:17.663746 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-42cns_66828029-4406-4251-a84a-ca40ff61c891/main/0.log" Jul 20 14:08:27.358209 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:27.358178 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7sjzn_715aeb3f-e7fe-44f2-9bd9-ab6fcb3006f0/global-pull-secret-syncer/0.log" Jul 20 14:08:27.629069 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:27.628999 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lpksh_53011d87-65ad-4a8b-a364-5e66e1cea999/konnectivity-agent/0.log" Jul 20 14:08:27.654415 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:27.654392 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-221.ec2.internal_0544e6927bc4266c71d4da456a570889/haproxy/0.log" Jul 20 14:08:35.561977 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:35.561943 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-bppwn_e018dae8-c0db-4060-9f8d-c7d3e0f35223/limitador/0.log" Jul 20 14:08:37.032357 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.032319 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8"] Jul 20 14:08:37.032823 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.032746 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5e60cb9-42b4-437d-b3e1-ef6dc763fa23" containerName="maas-api" Jul 20 14:08:37.032823 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.032764 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5e60cb9-42b4-437d-b3e1-ef6dc763fa23" containerName="maas-api" Jul 20 14:08:37.032823 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.032804 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2e0691-0e4c-44a0-ab30-3f0179048b42" containerName="revoke-keys" Jul 20 14:08:37.032823 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.032813 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2e0691-0e4c-44a0-ab30-3f0179048b42" containerName="revoke-keys" Jul 20 14:08:37.033089 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.032897 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5e60cb9-42b4-437d-b3e1-ef6dc763fa23" containerName="maas-api" Jul 20 14:08:37.033089 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.032912 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2e0691-0e4c-44a0-ab30-3f0179048b42" containerName="revoke-keys" Jul 20 14:08:37.035943 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.035922 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.042781 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.042755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btksl\" (UniqueName: \"kubernetes.io/projected/8140e231-b756-46dd-84c8-4a7f73497469-kube-api-access-btksl\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.042959 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.042821 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8140e231-b756-46dd-84c8-4a7f73497469-podres\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.042959 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.042915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8140e231-b756-46dd-84c8-4a7f73497469-lib-modules\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.043101 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.042977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8140e231-b756-46dd-84c8-4a7f73497469-sys\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.043101 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.043034 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8140e231-b756-46dd-84c8-4a7f73497469-proc\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.047949 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.047924 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lp56k\"/\"default-dockercfg-m47wx\"" Jul 20 14:08:37.047949 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.047944 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lp56k\"/\"kube-root-ca.crt\"" Jul 20 14:08:37.048091 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.047986 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lp56k\"/\"openshift-service-ca.crt\"" Jul 20 14:08:37.064765 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.064742 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8"] Jul 20 14:08:37.144158 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.144128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8140e231-b756-46dd-84c8-4a7f73497469-podres\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.144158 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.144160 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8140e231-b756-46dd-84c8-4a7f73497469-lib-modules\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.144367 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.144187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8140e231-b756-46dd-84c8-4a7f73497469-sys\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.144367 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.144257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8140e231-b756-46dd-84c8-4a7f73497469-proc\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.144367 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.144290 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8140e231-b756-46dd-84c8-4a7f73497469-proc\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.144367 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.144261 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8140e231-b756-46dd-84c8-4a7f73497469-sys\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.144367 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.144315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btksl\" (UniqueName: \"kubernetes.io/projected/8140e231-b756-46dd-84c8-4a7f73497469-kube-api-access-btksl\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.144367 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.144318 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8140e231-b756-46dd-84c8-4a7f73497469-lib-modules\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.144572 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.144311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8140e231-b756-46dd-84c8-4a7f73497469-podres\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.155698 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.155675 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btksl\" (UniqueName: \"kubernetes.io/projected/8140e231-b756-46dd-84c8-4a7f73497469-kube-api-access-btksl\") pod \"perf-node-gather-daemonset-grlg8\" (UID: \"8140e231-b756-46dd-84c8-4a7f73497469\") " pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.350113 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.350085 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.493628 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.493599 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8"] Jul 20 14:08:37.496141 ip-10-0-132-221 kubenswrapper[2575]: W0720 14:08:37.496111 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8140e231_b756_46dd_84c8_4a7f73497469.slice/crio-6fc5637069ea759e03fbf77fd759d481d4e9b1d8a9b72d6c1d5665a52c60b01e WatchSource:0}: Error finding container 6fc5637069ea759e03fbf77fd759d481d4e9b1d8a9b72d6c1d5665a52c60b01e: Status 404 returned error can't find the container with id 6fc5637069ea759e03fbf77fd759d481d4e9b1d8a9b72d6c1d5665a52c60b01e Jul 20 14:08:37.652752 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.652731 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7z7ts_29d7dae1-aff7-440c-9d24-6d46c71d84b5/node-exporter/0.log" Jul 20 14:08:37.701475 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.701447 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7z7ts_29d7dae1-aff7-440c-9d24-6d46c71d84b5/kube-rbac-proxy/0.log" Jul 20 14:08:37.753057 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.753037 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7z7ts_29d7dae1-aff7-440c-9d24-6d46c71d84b5/init-textfile/0.log" Jul 20 14:08:37.947751 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.947684 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" event={"ID":"8140e231-b756-46dd-84c8-4a7f73497469","Type":"ContainerStarted","Data":"795fe89367c960ff71dcab978560d56553ad19ee20df845dc301f6c90f282fd8"} Jul 20 14:08:37.947751 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.947743 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" event={"ID":"8140e231-b756-46dd-84c8-4a7f73497469","Type":"ContainerStarted","Data":"6fc5637069ea759e03fbf77fd759d481d4e9b1d8a9b72d6c1d5665a52c60b01e"} Jul 20 14:08:37.947967 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.947776 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:37.972810 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:37.972766 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" podStartSLOduration=0.972753052 podStartE2EDuration="972.753052ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 14:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 14:08:37.971931265 +0000 UTC m=+3112.180327410" watchObservedRunningTime="2026-07-20 14:08:37.972753052 +0000 UTC m=+3112.181149194" Jul 20 14:08:38.339517 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:38.339494 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3080fe9d-8969-4fda-a115-4e064efb2c62/prometheus/0.log" Jul 20 14:08:38.427026 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:38.427004 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3080fe9d-8969-4fda-a115-4e064efb2c62/config-reloader/0.log" Jul 20 14:08:38.491288 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:38.491270 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3080fe9d-8969-4fda-a115-4e064efb2c62/thanos-sidecar/0.log" Jul 20 14:08:38.552348 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:38.552328 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3080fe9d-8969-4fda-a115-4e064efb2c62/kube-rbac-proxy-web/0.log" Jul 20 14:08:38.606349 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:38.606295 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3080fe9d-8969-4fda-a115-4e064efb2c62/kube-rbac-proxy/0.log" Jul 20 14:08:38.649550 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:38.649530 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3080fe9d-8969-4fda-a115-4e064efb2c62/kube-rbac-proxy-thanos/0.log" Jul 20 14:08:38.699569 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:38.699547 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3080fe9d-8969-4fda-a115-4e064efb2c62/init-config-reloader/0.log" Jul 20 14:08:38.742617 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:38.742593 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-rzpw8_b1061ffd-d7cc-45aa-8980-90454b6ad0b7/prometheus-operator/0.log" Jul 20 14:08:38.795556 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:38.795537 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-rzpw8_b1061ffd-d7cc-45aa-8980-90454b6ad0b7/kube-rbac-proxy/0.log" Jul 20 14:08:38.832936 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:38.832916 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-bqzzp_342f3190-251a-4f4d-8924-bd75ec13311d/prometheus-operator-admission-webhook/0.log" Jul 20 14:08:43.742262 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:43.742231 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ggrjj_d11f1b29-aa51-45b0-8a5f-7d0887d7435c/dns/0.log" Jul 20 14:08:43.775458 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:43.775430 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ggrjj_d11f1b29-aa51-45b0-8a5f-7d0887d7435c/kube-rbac-proxy/0.log" Jul 20 14:08:43.851046 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:43.851023 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h8gtl_7bdea0d3-dbcd-4be1-bddc-49679ef918f1/dns-node-resolver/0.log" Jul 20 14:08:43.963106 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:43.963080 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lp56k/perf-node-gather-daemonset-grlg8" Jul 20 14:08:44.706131 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:44.706098 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-blhwm_02bda514-3256-495f-8ce5-ae9e7fe50c71/node-ca/0.log" Jul 20 14:08:46.007326 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:46.007296 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-rmv5s_b79abb9c-cf1a-413e-90c8-bcd61c9482cc/discovery/0.log" Jul 20 14:08:47.367045 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:47.367012 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nf4tc_49ea05e3-10a6-4258-b9cb-121fa55007f1/serve-healthcheck-canary/0.log" Jul 20 14:08:47.694723 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:47.694642 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-rmv5s_b79abb9c-cf1a-413e-90c8-bcd61c9482cc/discovery/0.log" Jul 20 14:08:48.135905 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:48.135865 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lslld_f3fee729-e0f4-4922-856a-2d806439a2f9/kube-rbac-proxy/0.log" Jul 20 14:08:48.172484 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:48.172463 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lslld_f3fee729-e0f4-4922-856a-2d806439a2f9/exporter/0.log" Jul 20 14:08:48.205377 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:48.205354 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lslld_f3fee729-e0f4-4922-856a-2d806439a2f9/extractor/0.log" Jul 20 14:08:49.213978 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:49.213902 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-rmv5s_b79abb9c-cf1a-413e-90c8-bcd61c9482cc/discovery/0.log" Jul 20 14:08:51.382243 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:51.382212 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f55c6d57f-s8q2w_0383138e-0b70-4738-b321-b984e3da082d/manager/0.log" Jul 20 14:08:52.040527 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:52.040501 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-864d7fc49d-zw5kr_28497bde-8176-4794-a41c-edd6cee61231/manager/0.log" Jul 20 14:08:52.075337 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:52.075315 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-864d7fc49d-zw5kr_28497bde-8176-4794-a41c-edd6cee61231/copy-manifests/0.log" Jul 20 14:08:53.326643 ip-10-0-132-221 kubenswrapper[2575]: I0720 14:08:53.326604 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-548487689c-n5v9s_509d57b7-3242-438d-970e-3a9593523ee9/manager/0.log"